In Advanced preview mode, Log Service accesses the specified Logstore and reads data from the Logstore to test the data and simulate the data transformation process. This topic describes how to configure the Advanced preview mode.

Procedure

  1. Log on to the Log Service console.
  2. Go to the data transformation page.
    1. In the Projects section, click the name of the project that you want to view.
    2. Choose Log Storage > Logstores. On the Logstores tab, click the Logstore that you want to view.
    3. On the query and analysis page, click Data Transformation.
  3. In the upper-right corner of the page, select a time range for the required log data.
    Make sure that the Raw Logs tab displays log data.
  4. In the edit box, enter a data transformation statement.
    For more information, see Data processing syntax.
    Note You can add comments to the data transformation statement to debug the statement line by line.
  5. Preview data.
    1. In the upper-right corner of the page, click Advanced.
    2. Click Preview Data.
    3. In the Add Preview Settings panel, set the parameters and click OK. The following table describes the parameters.

      When you preview data for the first time, you must set the parameters. After you set the parameters, you can click Modify Preview Settings to modify the parameters.

      Parameter Description
      Authorization Method You can authorize a data transformation task to read data from the current source Logstore by using one of the following methods:
      • Default Role: authorizes the data transformation task to assume the system role AliyunLogETLRole to read data from the source Logstore.

        You must click You must authorize the system role AliyunLogETLRole. Then, you must configure other parameters as prompted to complete the authorization. For more information, see Authorize Log Service to assume a system role.

        Note
        • If you use a RAM user, you must use an Alibaba Cloud account to assign the AliyunLogETLRole role to the user.
        • If you use an Alibaba Cloud account that has assumed the role, you can skip this operation.
      • Custom Role: authorizes the data transformation task to assume a custom role to read data from the source Logstore.

        You must grant the custom role the permissions to read from the source Logstore. Then, you must enter the Alibaba Cloud Resource Name (ARN) of the custom role in the Role ARN field. For more information about authorization, see Authorize Log Service to assume a custom role.

      • AccessKey Pair: authorizes the data transformation task to use the AccessKey pair of an Alibaba Cloud account or a RAM user to read data from the source Logstore.
        • Alibaba Cloud account: The AccessKey pair of an Alibaba Cloud account has permissions to read from the source Logstore. You can directly enter the AccessKey ID and AccessKey secret of the Alibaba Cloud account in the AccessKey ID and AccessKey Secret fields. For more information about how to obtain an AccessKey pair, see AccessKey pair.
        • RAM user: You must grant the RAM user the permissions to read from the source Logstore. Then, you can enter the AccessKey ID and AccessKey secret of the RAM user in the AccessKey ID and AccessKey Secret fields. For more information about authorization, see Configure an AccessKey pair for a RAM user to access a source Logstore and a destination Logstore.
      Advanced Parameter Settings Log Service allows you to set the passwords that are required in the transformation statement in the key-value pair format. For example, you can set a password that is used to connect to a database in the key-value pair format. You can reference passwords in the transformation statement by using the ${key} variable.
      You can click the plus sign (+) to add more key-value pairs. For example, you can add config.vpc.vpc_id.test1:vpc-uf6mskb0b****n9yj, which indicates the ID of the virtual private cloud (VPC) to which an ApsaraDB RDS instance belongs. Advanced Parameter Settings

      After you configure the preview settings, you can preview the data transformation results on the Transformation Results tab.

      • If the data fails to be transformed because the syntax of the transformation statement or the permissions are invalid, troubleshoot the failure as prompted.
      • If the data is transformed as expected, you can save the transformation statement as a rule. For more information, see Create a data transformation task.

Advanced preview example

  • Transformation statement
    # e_set("insert_field", "test_value")
    e_table_map(
        res_rds_mysql(
            address="rm-uf6wjk5****.mysql.rds.aliyuncs.com",
            username="test_username",
            password="****",
            database="test_db",
            table="test_table",
        ),
        "ip",
        ["country", "province"],
    )
  • Raw log entry
    {
      "__source__": "192.0.2.0",
      "__time__": 1624523917,
      "__topic__": "topic",
      "id": "1001",
      "ip": "127.0.0.1"
    }
  • Advanced preview settingsAdvanced preview
  • Transformation resultTransformation result