Log Service allows you to use Function Compute to transform streaming data. You can configure an extract-transform-load (ETL) job to detect data updates and call functions. Then the incremental data in a Logstore is consumed and transformed. You can use template functions or custom functions to transform data.
- Data cleansing and transformation
You can use Log Service to collect, transform, and query logs.
- Data shipping
You can use Log Service to ship data to various destinations. In this scenario, Log Service serves as a data channel for big data services in the cloud.
- Function types
- Function calling mechanism
An ETL job is used to call functions. After you create an ETL job for a Logstore in Log Service, a timer is started to poll data from the shards of the Logstore based on the job configurations. If new data is written to the Logstore, a triple data record in the
<shard_id,begin_cursor,end_cursor >format is generated as a function event. Then the ETL function is called.Note If no new data is written to the Logstore and the storage system is updated, the cursor information will change. The ETL function is called for each shard but no data is transformed. In this case, you can use the cursor information to obtain data from the shards. If no data is obtained, the ETL function is called but no data is transformed. You can ignore the function callings. For more information, see Development guide.An ETL job calls functions based on the time mechanism. For example, you set the calling interval in an ETL job to 60 seconds for a Logstore. If data is continuously written to Shard 0, the ETL function is called every 60 seconds to transform data that is located in the cursor range of the last 60 seconds.
- Query trigger logs
You can create an index for trigger logs and view statistics about tasks. For more information, see Enable and configure the index feature for a Logstore.
- View the logs of function callings
You can use the command line tool to view detailed information about function callings. For more information, see log entry.
- Check whether new data is written to the Logstore for which an ETL job is configured. If new data is written to the Logstore, the function will be called.
- Check trigger logs and logs of function callings.