If you want to monitor the metric change trend of a log field, you can use the e_to_metric function to convert the log field to a metric. Then, you can view the change trend of the metric in a Metricstore. This topic describes how to convert logs to metrics. NGINX access logs are used as an example.

Prerequisites

Log data is collected. For more information, see Log collection methods.

Background information

The operational data of applications includes log data, trace data, and metric data. A log indicates a discrete event. A trace indicates an event that contains a call chain. A metric indicates an event that contains numeric measurements. Logs, traces, and metrics are events. A system that can store events can also store the three types of data. Log Service provides the following two types of stores for data storage:
  • Logstore

    A Logstore in Log Service is a unit that is used to collect, store, and query logs. For more information, see Logstore.

  • Metricstore

    A Metricstore in Log Service is a unit that is used to collect, store, and query metrics. Metricstores are optimized based on metrics. You can use the PromQL syntax that is provided by Prometheus to query data. For more information, see Metricstore.

For most applications, a log contains more information than a metric. Metrics can be considered as logs in a specific format. Log Service allows you to convert logs to metrics. You can use one of the following methods to convert logs to metrics:
  • Use the e_to_metric function to convert logs to metrics.

Scenarios

For example, you have created a Logstore named nginx-demo in the China (Hangzhou) region to store NGINX access logs.

You need to monitor the changes of the request time (request_time) and response time (upstream_response_time) of each backend server (host), and then visualize the change trend on a dashboard.

body_bytes_sent:1750
host:www.gk.mock.com
http_referer:www.sw.mock.com
http_user_agent:Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; it-it) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27
http_x_forwarded_for:203.0.113.10
remote_addr:203.0.113.10
remote_user:p288
request_length:13741
request_method:GET
request_time:71
request_uri:/request/path-1/file-1
status:200
time_local:11/Aug/2021:06:52:27
upstream_response_time:0.66

To meet the preceding requirements, you must convert the request_time and upstream_response_time fields in logs to metrics, and then add the host label to the metrics.

Step 1: Create a Metricstore

Create a Metricstore named service-metric to store the time series data that is returned during data transformation.

  1. Log on to the Log Service console.
  2. In the Projects section, click the name of the project that you want to view.
  3. On the Metricstore Storage > Metricstore tab, click the plus sign (+) next to the search box.
  4. In the Create Metricstore pane, set the following parameters.
    Parameter Description
    MetricStore Name The name of the Metricstore. The name must be unique in the project to which the Metricstore belongs. After the Metricstore is created, the name cannot be modified.
    Data Retention Period The retention period of the time series data that is collected by using Log Service in the Metricstore. Valid values: 15 to 3000. Unit: days. When the retention period expires, the time series data is deleted.
    Shards Log Service provides shards to read and write data. Each shard supports a write speed of 5 MB/s, 500 write operations per second, a read speed of 10 MB/s, and 100 read operations per second. You can create up to 10 shards for each Metricstore. You can create up to 200 shards for each project. For more information, see Shard.
  5. Click Confirm.

Step 2: Create a data transformation task

Use the e_to_metric function to create a data transformation task and store transformed data in the service-metric Metricstore that you created in Step 1.

  1. Go to the data transformation page.
    1. In the Projects section, click the name of the project that you want to view.
    2. Choose Log Storage > Logstores. On the Logstores tab, click the Logstore that you want to view.
    3. On the Search & Analysis page, click Data Transformation .
  2. In the upper-right corner of the page, specify a time range for the required log data.
    Make sure that log data exists on the Raw Logs tab.
  3. In the edit box, enter a data transformation statement.
    Rename the request_time field to RequestTime, rename the upstream_response_time field to ResponseTime, and then add the host label.
    e_to_metric(
        names=[("request_time", "RequestTime"), ("upstream_response_time", "ResponseTime")],
        labels=[("host", "hostname")],
    )
    For more information, see e_to_metric.
  4. Click Preview Data .
    previewdata
  5. Create a data transformation task.
    1. Click Save as Transformation Rule .
    2. In the Create Data Transformation Rule panel, set the parameters and click OK . The following table describes the parameters.
      Parameter Description
      Rule Name The name of the transformation rule, for example, log2mectric.
      Authorization Method Select Default Role to read data from the source Logstore.
      Storage Target
      Target Name The name of the storage target, for example, log2mectric.
      Target Region The region where the destination project resides, for example, China (Hangzhou).
      Target Project The name of the destination project to which transformed data is saved.
      Target Logstore The name of the destination Metricstore to which transformed data is saved, for example, service-metric.
      Authorization Method Select Default Role to write transformed data to the destination service-metric Metricstore.
      Processing Range
      Time Range Select All .
    3. In the Result dialog box, click Confirm .

After you perform the preceding steps, Log Service transforms the log data in the source Logstore and writes transformed data to the service-metric Metricstore.

Step 3: Query time series data

  1. In the left-side navigation pane, choose Time Series Storage > Metricstore .
  2. On the Metricstore tab, select the service-metric Metricstore.
  3. In the upper-right corner of the page, click 15 Minutes (Relative) to specify a time range for the query and analysis.
    You can select a relative time or a time frame. You can also specify a custom time range.
    Note The query and analysis results may contain time series data that is generated 1 minute earlier or later than the specified time range.
  4. On the Query Statements tab, select the RequestTime metric or the ReponseTime metric from the Metrics drop-down list and click Preview .
    • The following figure shows the change trend of the RequestTime metric for each host. requesttime
    • The following figure shows the change trend of the ReponseTime metric for each host. reponsetime