Log Service allows you to create a data transformation task to read data from a source Logstore and write transformed data to one or more destination Logstores. You can also query and analyze the transformed data to create more value. This topic describes how to create a data transformation task in the Log Service console.

Prerequisites

Procedure

  1. Log on to the Log Service console.
  2. Go to the data transformation page.
    1. In the Projects section, click the name of the project that you want to view.
    2. Choose Log Storage > Logstores. On the Logstores tab, click the Logstore that you want to view.
    3. On the query and analysis page, click Data Transformation.
  3. In the upper-right corner of the page, select a time range for the required log data.
    Make sure that the Raw Logs tab displays log data.
  4. In the editor, enter transformation statements.
    For more information about the statement syntax, see Language introduction.
  5. Preview transformation results.
    1. Select Quick.
      You can select either Quick or Advanced. For more information, see Configure preview modes.
    2. Click Preview Data.

      View the results.

      • If data fails to be transformed because the syntax of the transformation statements or configured permissions are invalid, troubleshoot the failure as prompted.
      • If the transformed data is returned as expected, go to 6.
  6. Create a data transformation task.
    1. Click Save as Transformation Rule.
    2. In the Create Data Transformation Rule panel, configure the following parameters and click OK.
      Parameter Description
      Rule Name The name of the transformation rule.
      Authorization Method The method used to authorize the data transformation task to read data from the source Logstore. Valid values:
      • Default Role: authorizes the data transformation task to assume the system role AliyunLogETLRole to read data from the source Logstore.

        You must click You must authorize the system role AliyunLogETLRole. Then, you must configure other parameters as prompted to complete the authorization. For more information, see Authorize Log Service to assume a system role.

        Note
        • If you use a RAM user, you must use an Alibaba Cloud account to assign the AliyunLogETLRole role to the user.
        • If you use an Alibaba Cloud account that has assumed the role, you can skip this operation.
      • Custom Role: authorizes the data transformation task to assume a custom role to read data from the source Logstore.

        You must grant the custom role the permissions to read from the source Logstore. Then, you must enter the Alibaba Cloud Resource Name (ARN) of the custom role in the Role ARN field. For more information about authorization, see Authorize Log Service to assume a custom role.

      • AccessKey Pair: authorizes the data transformation task to use the AccessKey pair of an Alibaba Cloud account or a RAM user to read data from the source Logstore.
        • Alibaba Cloud account: The AccessKey pair of an Alibaba Cloud account has permissions to read from the source Logstore. You can directly enter the AccessKey ID and AccessKey secret of the Alibaba Cloud account in the AccessKey ID and AccessKey Secret fields. For more information about how to obtain an AccessKey pair, see AccessKey pair.
        • RAM user: You must grant the RAM user the permissions to read from the source Logstore. Then, you can enter the AccessKey ID and AccessKey secret of the RAM user in the AccessKey ID and AccessKey Secret fields. For more information about authorization, see Configure an AccessKey pair for a RAM user to access a source Logstore and a destination Logstore.
      Storage Target
      Target Name The name of the storage destination. Storage Target includes Target Project and Target Logstore.
      You can create multiple storage destinations to store the transformed data in different destination Logstores.
      • You can also use the name parameter of the e_output or e_coutput function in the transformation statements to specify the name of the storage destination. For more information, see e_output and e_coutput.
      • If you do not include the e_output function in the transformation statements, the task writes the transformed data to the Logstore in the storage destination that is numbered 1 by default.

        If you want to configure only one destination Logstore, you do not need to include the e_output function in the transformation statements.

      • If you include the e_output or e_coutput function and set the name, project, and logstore parameters for the function, the task runs based on the parameter settings in the functions even if you configure the Target Project and Target Logstore parameters in this step.
      Target Region The region of the project to which the destination Logstore belongs.

      If you want to perform data transformation across regions, we recommend that you use HTTPS for data transmission. This ensures the privacy of log data.

      For cross-region data transformation, the data is transmitted over the Internet. If the Internet connections are unstable, data transformation latency may occur. You can select DCDN Acceleration to accelerate the cross-region data transmission. Before you can select DCDN Acceleration, make sure that the global acceleration feature is enabled for the project. For more information, see Enable the global acceleration feature.

      Note You are charged for the amount of Internet traffic that is generated when data after compression is transmitted across regions. For more information, see Billable items.
      Target Project The name of the project to which the destination Logstore belongs.
      Target Logstore The name of the destination Logstore.
      Authorization Method The method used to authorize the data transformation task to write transformed data to the destination Logstore. Valid values:
      • Default Role: authorizes the data transformation task to assume the system role AliyunLogETLRole to write transformed data to the destination Logstore.
        You must click You must authorize the system role AliyunLogETLRole. Then, you must configure other parameters as prompted to complete the authorization. For more information, see Authorize Log Service to assume a system role.
        Note
        • If you use a RAM user, you must use an Alibaba Cloud account to assign the AliyunLogETLRole role to the user.
        • If you use an Alibaba Cloud account that has assumed the role, you can skip this operation.
      • Custom Role: authorizes the data transformation task to assume a custom role to write transformed data to the destination Logstore.

        You must grant the custom role the permissions to write to the destination Logstore. Then, you must enter the ARN of the custom role in the Role ARN field. For more information about authorization, see Authorize Log Service to assume a custom role.

      • AccessKey Pair: authorizes the data transformation task to use the AccessKey pair of an Alibaba Cloud account or a RAM user to write transformed data to the destination Logstore.
        • Alibaba Cloud account: The AccessKey pair of an Alibaba Cloud account has permissions to write to the destination Logstore. You can directly enter the AccessKey ID and AccessKey secret of the Alibaba Cloud account in the AccessKey ID and AccessKey Secret fields. For more information about how to obtain an AccessKey pair, see AccessKey pair.
        • RAM user: You must grant the RAM user the permissions to write to the destination Logstore. Then, you can enter the AccessKey ID and AccessKey secret of the RAM user in the AccessKey ID and AccessKey Secret fields. For more information about authorization, see Configure an AccessKey pair for a RAM user to access a source Logstore and a destination Logstore.
      Processing Range
      Time Range The time range within which the data is transformed. Valid values:
      • All: transforms data in the source Logstore from the first log entry until the task is manually stopped.
      • From Specific Time: transforms data in the source Logstore from the log entry that is received at the specified start time until the task is manually stopped.
      • Within Specific Period: transforms data in the source Logstore from the log entry that is received at the specified start time to the log entry that is received at the specified end time.
      Note The value of Time Range is based on the time when logs are received.
      Advanced
      Advanced Parameter Settings You may need to specify passwords, such as database passwords, in transformation statements. Log Service allows you to add a key-value pair to save the passwords. You can specify the ${key} variable in the statements to reference the passwords.
      You can click the plus sign (+) to add more key-value pairs. For example, you can add config.vpc.vpc_id.test1:vpc-uf6mskb0b****n9yj, which indicates the ID of the virtual private cloud (VPC) to which an ApsaraDB RDS instance belongs. Advanced Parameter Settings

What to do next

After the data transformation task is created, you can perform the following operations:
  • On the Data Transformation Overview page, view the details and status of the task. You can also perform other operations, such as modifying or stopping the task. For more information, see Manage a data transformation task.
  • In a destination Logstore, perform query and analysis operations. For more information, see Query logs.