This topic describes the syntax and parameters of field processing functions. This topic also provides examples on how to use the functions.

Functions

Function Description
v Extracts the value of a field from an event. If multiple field names are passed in the function, the value of the first field that exists is returned.
e_set Adds a field or specifies a new value for an existing field.
e_drop_fields Deletes the log fields that meet a specified condition.
e_keep_fields Retains the log fields that meet a specified condition.
e_pack_fields Encapsulates specified log fields, and then assigns the log fields as a value to a new field.
e_rename Renames the log fields that meet a specified condition.

v

You can use the v function to extract the value of a field from an event.

  • Syntax
    v(Field name, ..., default=None)
  • Parameters
    Parameter Type Required Description
    Field name String Yes The name of the field whose value you want to extract from an event.
    default Arbitrary No The value of this parameter is returned if none of the specified fields exist. Default value: None.
  • Response

    The value of the first field that exists in an event is returned. If none of the specified fields exist, the value of the default parameter is returned.

  • Example
    Assign the value of the content field to the test_content field.
    • Raw log entry:
      content: hello
    • Transformation rule:
      e_set("test_content", v("content"))
    • Result:
      content: hello
      test_content: hello

e_set

You can use the e_set function to add a field or to specify a new value for an existing field.
  • Syntax
    e_set(key1, value1, key2, value2, mode="overwrite")
    Notice
    • The key1 and value1 parameters must be specified in pairs.
    • If you use the e_set function to specify a value for a time field, such as F_TIME or __time__, the value must be a numeric string.
      e_set(F_TIME, "abc")   # Invalid syntax.
      e_set(F_TIME, "12345678")   # Valid syntax.
  • Parameters
    Parameter Type Required Description
    key String Yes The name of a log field. You can set this parameter to an expression that is used to return a string. For more information about how to specify special field names, see Event structure and fields.
    value Arbitrary Yes The new value of a specified field. If the value of this parameter is not a string, the function automatically converts the value to a string. For example, if you set this parameter to a value of the tuple, list, or dictionary type, the function automatically converts the value to a JSON string. For more information about the conversion rule of strings, see Automatic type conversion during assignment.
    Note If you set this parameter to None, the function does not update the original value of the specified field.
    mode String No The overwrite mode of fields. Default value: overwrite. For more information, see Field check and overwrite modes.
  • Response

    The updated log entry is returned.

  • Examples
    • Example 1: Assign a fixed value to a field.
      Add a new field named city and set the value to Shanghai.
      e_set("city", "Shanghai")
    • Example 2: Extract the value of an existing field, and then assign the value to another field.
      Call an expression function to extract the value of an existing field named ret, and then assign the value to a new field named result.
      e_set("result", v("ret"))
    • Example 3: Assign a dynamic value to a field.
      Call multiple expression functions in sequence to obtain the value in lowercase of the first field from specified existing fields and specify the value for the result field.
      e_set("result", str_lower(v("ret", "return")))
    • Example 4: Specify a value for a field multiple times.
      1. Specify a fixed value for the event_type field.
        e_set("event_type", "login event", "event_info", "login host")
      2. If the value of the ret field is fail, set the event_type field to login failed event.
        e_if(e_search('ret==fail'), e_set("event_type", "login failed event" ))

e_drop_fields

You can use the e_drop_fields function to delete the log fields that meet a specified condition.

  • Syntax
    e_drop_fields (field 1, field 2, ....,regex=False)
  • Parameters
    Parameter Type Required Description
    field String Yes The name of a log field. The value of this parameter can be a regular expression. If the field name meets the specified condition, the field is deleted. Otherwise, the field is retained. For more information about regular expressions, see Regular expressions.

    You must specify at least one log field.

    regex Boolean No If you set this parameter to False, regular expressions are not used to match log fields. Default value: True.
  • Example

    If the value of the content field is 123, the content and age fields are deleted.

    • Raw log entry:
      age: 18
      content: 123
      name: twiss
    • Transformation rule:
      e_if(e_search("content==123"), e_drop_fields("content", "age",regex=True))
    • Result:
      name: twiss

e_keep_fields

You can use the e_keep_fields function to retain the log fields that meet a specified condition.
Note Log Service provides built-in meta-fields such as __time__ and __topic__. If you do not retain the __time__ field when you call the e_keep_fields function, the time of the event is reset to the current time. If you do not want to reset the value of a meta-field, add the meta-field to a list in the format of F_TIME, F_META, F_TAGS, "f1", "f2". For more information, see Identifiers.
  • Syntax
    e_keep_fields (field 1, field 2, ....,regex=False)
  • Parameters
    Parameter Type Required Description
    field String Yes The name of a log field. The value of this parameter can be a regular expression. If the field name meets the specified condition, the field is retained. Otherwise, the field is deleted.

    You must specify at least one log field.

    regex Boolean No If you set this parameter to False, regular expressions are not used to match log fields. Default value: True.
  • Example: If the value of the content field is 123, the content and age fields are retained.
    • Raw log entry:
      age: 18
      content: 123
      name: twiss
    • Transformation rule:
      e_if(e_search("content==123"), e_keep_fields("content", "age"))
    • Result:
      age: 18
      content: 123

e_pack_fields

  • Syntax
    e_pack_fields(output_fields,include=".*",exclude=None,drop_packed=True)
  • Parameters
    Parameter Type Required Description
    output_field String Yes The name of the output field. The value of the field is log data in the JSON format.
    include String No The whitelist. Fields that match the specified regular expression are encapsulated. Default value: ".*". This value indicates that all fields are encapsulated. For more information, see Regular expressions.
    exclude String No The blacklist. Fields that match the specified regular expression are not encapsulated. Default value: None. This value indicates that all fields are encapsulated. For more information, see Regular expressions.
    drop_packed Boolean No Specifies whether to delete raw fields after the fields are encapsulated. Default value: True.
    • True: The raw fields that are encapsulated are deleted in the output. This is the default value.
    • False: The raw fields that are encapsulated are not deleted in the output.
  • Response

    The log entry in which specified fields are encapsulated is returned.

  • Examples
    • Example 1: Encapsulate all log fields into a value, and then assign the value to the test field. By default, the raw fields that are encapsulated are deleted.
      • Raw log entry:
        test1:123
        test2:456
        test3:789
      • Transformation rule:
        e_pack_fields("test")
      • Result:
        test:{"test1": "123", "test2": "456", "test3": "789"}
    • Example 2: Encapsulate all log fields into a value, and then assign the value to the test field. The raw fields that are encapsulated are not deleted.
      • Raw log entry:
        test1:123
        test2:456
        test3:789
      • Transformation rule:
        e_pack_fields("test",drop_packed=False)
      • Result:
        test:{"test1": "123", "test2": "456", "test3": "789"}
        test1:123
        test2:456
        test3:789
    • Example 3: Encapsulate the test and abcd fields into a value, and then assign the value to the content field. The raw fields that are encapsulated are not deleted.
      • Raw log entry:
        abcd@#%:123
        test:456
        abcd:789
      • Transformation rule:
        e_pack_fields("content", include="\w+", drop_packed=False)
      • Result:
        abcd:789
        abcd@#%:123
        content:{"test": "456", "abcd": "789"}
        test:456
    • Example 4: Encapsulate log fields that exclude the test and abcd fields into a value and assign the value to the content field. The raw fields that are encapsulated are deleted.
      • Raw log entry:
        abcd@#%:123
        test:456
        abcd:789
      • Transformation rule:
        e_pack_fields("content", exclude="\w+", drop_packed=True)
      • Result:
        abcd:789
        content:{"abcd@#%": "123"}
        test:456

e_rename

You can use the e_rename function to rename the log fields that meet a specified condition.

  • Syntax
    e_rename("field 1", "renamed field 1", "field 2", "renamed field 2", ..., regex=False)
    Note The field and renamed field parameters must be specified in pairs.
  • Parameters
    Parameter Type Required Description
    field String Yes The name of a log field. The value of this parameter can be a regular expression. If the field name meets the specified condition, the field is renamed. For more information about regular expressions, see Regular expressions.

    You must specify at least one log field.

    renamed field String Yes The new name of the field.
    regex Boolean No If you set this parameter to False, regular expressions are not used to match log fields. Default value: True.
  • Response

    The field with the new name is returned.

  • Examples
    • Example 1
      • Raw log entry:
        host: 1006
      • Transformation rule:
        e_rename("host","client_host")
      • Result:
        client_host: 1006
    • Example 2
      • Raw log entry:
        host: 1006
      • Transformation rule:
        e_rename("url","rename_url")
      • Result:
        host: 1006