The domain-specific language (DSL) for Log Service provides different types of functions to meet the data transformation and processing needs in different scenarios.

Language modes

The DSL for Log Service is compatible with Python. In standard mode, the DSL can be regarded as a subset of Python. Except for basic data structures and expressions, other syntax rules are orchestrated by using functions. If you want to use user-defined functions (UDFs), submit a ticket.
Category Python syntax Standard mode
Data structure Number, string, and Boolean Supported, except for strings in the """ form.
Tuple, list, set, and dictionary Supported, except for the set structure, such as {1,2,3}.
Object Only built-in extended data structures such as table and datetime objects are supported.
Basic syntax Operators, such as + (addition), - (subtraction), * (multiplication), and / (division) Not supported directly. You must use the operators in functions.
Comments Supported.
Variable assignment Not supported. You must call functions to pass values to variables.
Condition evaluation and loops Not supported directly. You must use built-in functions to implement condition evaluation and loops.
Function Standard built-in functions of Python Not supported. You can use more than 200 built-in functions provided by the DSL for Log Service.
Function calls Supported, except for function calls with parameter unpacking.
UDFs, such as def or lambda Not supported. You can use more than 200 global processing and expression functions provided by the DSL for Log Service. You can also combine these functions to suite your needs.
Module Import and use of the Python standard library Not supported.
Creation of threads and processes Not supported.
Import of third-party libraries Not supported.
External network connection or external command call Supported. The DSL for Log Service provides built-in resource connectors.

Function categories

In standard mode of the DSL for Log Service, all operations are implemented by calling functions. The DSL provides more than 200 built-in functions, which are categorized into global processing functions and expression functions.
  • Global processing functions

    Global processing functions receive, process, and return events. Only global processing functions can be used to construct each step of a transformation rule.

  • Expression functions

    Expression functions are general functions that receive specific parameters and return specific values. Expression functions can be combined and passed to global processing functions as parameters to define more flexible logic.

The following table compares global processing functions with expression functions.
Category Construct a step Receive an event Return results Modify an event Combine functions
Global processing function Supported. Events are automatically received. Zero to multiple events are returned. Supported. Events can be modified in most cases. Supported.
Expression function Not supported. Supported only by a few expression functions. Most expression functions do not directly process events. Specific data structures are returned. Not supported. Supported.

Global processing functions

Global processing functions receive, process, and return events.
Note Only a global processing function can be placed in the first line of each step.
The following syntax is used:
Global Processing Function 1(...Parameters...)
Global Processing Function 2(...Parameters...)
Global Processing Function 3(...Parameters...)
Global Processing Function 4(...Parameters...)
Global processing functions can be further categorized into flow control functions and event processing functions.
Category Description Example
Flow control function This type of function controls processes, receives events, and calls event processing functions based on conditions. e_if, e_switch, and e_if_else
Event processing function This type of function processes events. Zero to multiple events are returned. Examples:
  • e_drop_fields: discards fields of events.
  • e_kv: extracts and adds key-value pairs of events.
  • e_dict_map: enriches events.
Transformation logic:
  • Basic processing
    The data transformation feature reads streaming data from a source Logstore and sends each log event in a dictionary structure to a transformation rule. Then, the feature runs the event processing functions defined in the transformation rule in sequence to process the events and writes the processing results to the destination Logstores.
    Note All the fields and values of an event are transmitted as strings. For example, the log event {"__time__": "1234567", "__topic__": "", "k1": "test"} is processed by the function e_set("f1", 200), which adds the field f1 with a value of 200. After processing, the event {"__time__": "1234567", "__topic__": "", "k1": "test", "f1": "200"} is returned. f1 and 200 are both strings.

    Event processing functions defined in a transformation rule are called in sequence. Each function receives and processes an event and returns a processed event.

    For example, the function e_set("type", "test") adds the field type to each event and sets the value to test. The next function receives the processed event.

  • Condition evaluation
    • e_if: You can call the e_if function to add conditional expressions to process events. If an event does not meet the specified condition, the operation is skipped. The e_if function implements the if logic.

      For example, the function e_if(e_match("status", "200"), e_regex("data", "ret: \d +", "result")) first determines whether the value of the status field is 200. If the value is 200, the function extracts a new field result from the data field by using the regular expression. If the value is not 200, the extract operation is not performed.

    • e_if_else: This function works in a similar way to the if_else function.
  • Processing termination
    • Some steps of a transformation rule may return no events. This indicates that the event is deleted.

      For example, if the function e_if(str_islower(v("result")), e_drop()) is used and the value of each result field in an event is a string consisting of only lowercase characters, the function deletes this event. Then, the subsequent steps are not performed on this event. The system automatically processes the next event.

    • If an event is written to a destination Logstore, the processing is terminated. For example, the e_output function writes an event to a destination Logstore and deletes the event, and the subsequent steps are not performed on this event.
      Note However, the e_coutput function writes an event to a destination Logstore but retains the original event, and the subsequent steps are performed on this event.
  • Event splitting for parallel processing

    Some steps of a transformation rule may return multiple events. This indicates that the original event is split.

    For example, the function e_split(data) splits an event into two events based on the value of the data field. If the value of the data field in the original event is "abc, xyz", the values of the data field in the events generated after splitting are abc and xyz.

    All the events generated after splitting are processed in the subsequent steps.

Expression functions

In addition to global processing functions, the DSL for Log Service provides 200 expression functions that receive specific parameters and return specific values. You can call a single expression function or a combination of expression functions in a global processing function. The following syntax is used:
Global Processing Function 1(Expression Function 1(...), ...)
Global Processing Function 2(..., Expression Function 2(...), Expression Function 3(...), ...)
Expression functions can be categorized into event check functions, resource functions, control functions, and other expression functions.
Category Description Example
Event check function This type of function receives events, extracts specific information, and returns the information. Events are not modified. v: returns the values of fields. e_search and e_match: check whether events meet the specified conditions.
Resource function This type of function accesses on-premises or external resources, receives specific parameters, and returns specific values. The data types of return values include dictionary and table. res_oss_file, res_rds_mysql, and res_log_logstore_pull.
Control function This type of function receives specific parameters and performs logical operations on expressions or condition-based control. This type of function also calls other expression functions to return results. op_and, op_or, op_not, op_if, and op_coalesce.
Other expression function This type of function receives specific parameters or the results of other functions and returns specific values. String functions, date and time functions, and conversion functions.