You can authorize a data transformation task to use the AccessKey pair of an Alibaba Cloud account or a Resource Access Management (RAM) user to read data from a source Logstore and write transformed data to one or more destination Logstores. The AccessKey pair of an Alibaba Cloud account has access permissions on Logstores and can be directly used. If you use a RAM user, you must grant the RAM user the access permissions on Logstores. For more information, see the following procedures.

Prerequisites

A RAM user is created. For more information, see Create a RAM user.
Notice
  • When you create the RAM user, select Programmatic Access for Access Mode. Then, record the AccessKey pair of the RAM user.
  • The AccessKey secret is displayed only when you create the RAM user. The AccessKey secret cannot be queried. We recommend that you record the AccessKey secret for subsequent use and keep it confidential.

Grant the RAM user the permissions to read from a source Logstore

After the RAM user is authorized by an Alibaba Cloud account, the RAM user has permissions to read from the source Logstore. When you create a data transformation task, you can enter the AccessKey pair of the RAM user. For more information, see Create a data transformation task.

  1. Log on to the RAM console by using an Alibaba Cloud account.
  2. Create a policy.
    The policy is used to allow the RAM user to read data from the source Logstore.
    1. In the left-side navigation pane, choose Permissions > Policies.
    2. On the Policies page, click Create Policy.
    3. On the Create Custom Policy page, configure the following parameters and click OK.
      Parameter Description
      Policy Name The name of the policy. In this example, enter log-etl-source-reader-1-policy.
      Configuration Mode Select Script.
      Policy Document The content of the policy. Replace the content in the editor with one of the following scripts based on your business requirements.
      • Policy that uses exact match
        The source project name is log-project-prod. The source Logstore name is access_log. Replace the project and Logstore names based on your business requirements.
        {
          "Version": "1",
          "Statement": [
            {
              "Action": [
                "log:ListShards",
                "log:GetCursorOrData",
                "log:GetConsumerGroupCheckPoint",
                "log:UpdateConsumerGroup",
                "log:ConsumerGroupHeartBeat",
                "log:ConsumerGroupUpdateCheckPoint",
                "log:ListConsumerGroup",
                "log:CreateConsumerGroup"
              ],
              "Resource": [
                "acs:log:*:*:project/log-project-prod/logstore/access_log",
                "acs:log:*:*:project/log-project-prod/logstore/access_log/*"
              ],
              "Effect": "Allow"
            }
          ]
        }
      • Policy that uses fuzzy match
        The source project names can be log-project-dev-a, log-project-dev-b, or log-project-dev-c. The source Logstore names can be app_a_log, app_b_log, or app_c_log. Replace the project and Logstore names based on your business requirements.
        {
          "Version": "1",
          "Statement": [
            {
              "Action": [
                "log:ListShards",
                "log:GetCursorOrData",
                "log:GetConsumerGroupCheckPoint",
                "log:UpdateConsumerGroup",
                "log:ConsumerGroupHeartBeat",
                "log:ConsumerGroupUpdateCheckPoint",
                "log:ListConsumerGroup",
                "log:CreateConsumerGroup"
              ],
              "Resource": [
                "acs:log:*:*:project/log-project-dev-*/logstore/app_*_log",
            "acs:log:*:*:project/log-project-dev-*/logstore/app_*_log/*"
              ],
              "Effect": "Allow"
            }
          ]
        }
        For more information about authorization scenarios, see Use custom policies to grant permissions to a RAM user.
  3. Attach the policy to the RAM user.
    1. In the left-side navigation pane, choose Identities > Users.
    2. On the Users page, find the RAM user and click Add Permissions in the Actions column.
    3. In the Select Policy section, click the Custom Policy tab. From the list of custom policies, click the policy that you created in Step 2 and click OK. In this example, the policy is log-etl-source-reader-1-policy.
      Add permissions
    4. Confirm the authorization results. Then, click Complete.

Grant the RAM user the permissions to write to destination Logstores

After the RAM user is authorized by an Alibaba Cloud account, the RAM user has permissions to write to the destination Logstores. When you create a data transformation task, you can enter the AccessKey pair of the RAM user. For more information, see Create a data transformation task.

  1. Log on to the RAM console by using an Alibaba Cloud account.
  2. Create a policy.
    The policy is used to allow the RAM user to write data to the destination Logstores.
    1. In the left-side navigation pane, choose Permissions > Policies.
    2. On the Policies page, click Create Policy.
    3. On the Create Custom Policy page, configure the following parameters and click OK.
      Parameter Description
      Policy Name The name of the policy. In this example, enter log-etl-target-writer-1-policy.
      Configuration Mode Select Script.
      Policy Document The content of the policy. Replace the content in the editor with one of the following scripts based on your business requirements.
      • Policy that uses exact match
        The destination project name is log-project-prod. The destination Logstore name is access_log_output. Replace the project and Logstore names based on your business requirements.
        {
          "Version": "1",
          "Statement": [
            {
              "Action": [
                "log:Post*",
                "log:BatchPost*"
              ],
               "Resource": "acs:log:*:*:project/log-project-prod/logstore/access_log_output",
              "Effect": "Allow"
            }
          ]
        }
      • Policy that uses fuzzy match
        The destination project names can be log-project-dev-a, log-project-dev-b, or log-project-dev-c. The destination Logstore names can be app_a_log_output, app_b_log_output, or app_c_log_output. Replace the project and Logstore names based on your business requirements.
        {
          "Version": "1",
          "Statement": [
            {
              "Action": [
                "log:Post*",
                "log:BatchPost*"
              ],
               "Resource": "acs:log:*:*:project/log-project-dev-*/logstore/app_*_log_output",
              "Effect": "Allow"
            }
          ]
        }
        For more information about authorization scenarios, see Use custom policies to grant permissions to a RAM user.
  3. Attach the policy to the RAM user.
    1. In the left-side navigation pane, choose Identities > Users.
    2. On the Users page, find the RAM user and click Add Permissions in the Actions column.
    3. In the Select Policy section, click the Custom Policy tab. From the list of custom policies, click the policy that you created in Step 2 and click OK. In this example, the policy is log-etl-target-writer-1-policy.
      Write permissions
    4. Confirm the authorization results. Then, click Complete.

What to do next

You can enter the AccessKey pair of the RAM user in a data transformation task. For more information, see Create a data transformation task. Modify a transformation rule