After log data is shipped from Log Service to Object Storage Service (OSS), the log data can be stored in different formats. This topic describes the CSV format.

Parameters

When you configure a shipping rule, you can select csv for Storage Format. The following table describes the parameters that you must configure if you select csv. For more information about how to specify the storage format, see Configure a data shipping rule.

CSV fields
For more information, see Common Format and MIME Type for Comma-Separated Values (CSV) Files and PostgreSQL 9.4.26 Documentation.
Parameter Description
CSV Fields The names of the log fields that you want to ship to OSS. You can view log fields on the Raw Logs tab of a Logstore. You must add the log fields in the Key Name column one by one. The log fields that you can ship to OSS include the fields in the log content and the reserved fields such as __time__, __topic__, and __source__. For more information about reserved fields, see Reserved fields.
Note The log fields that you add in the CSV Fields field must be unique.
Delimiter The delimiter used to delimit log fields. You can use commas (,), vertical bars (|), spaces, or tabs to delimit log fields.
Escape Character If a log field contains a delimiter, use an escape character to enclose the field. This way, the field is not delimited.
Invalid Fields If a log field that you specify in the CSV Fields field does not exist in logs, the value that you specify in the Invalid Fields field is shipped.
Shipped Fields If you turn on Shipped Fields, the names of log fields are written to CSV files.

Directories of OSS buckets

The following table describes the directories of OSS buckets that store data shipped from Log Service.

Compression type File extension Example Description
Not compressed .csv oss://oss-shipper-shenzhen/ecs_test/2016/01/26/20/54_1453812893059571256_937.csv You can download the CSV file to your computer and open the files as text files.
Snappy .snappy.csv oss://oss-shipper-shenzhen/ecs_test/2016/01/26/20/54_1453812893059571256_937.snappy.csv For more information about how to open Snappy compressed files, see Decompression tools for Snappy compressed files.

Examples

If you want to use HybridDB for PostgreSQL to consume data in CSV files in OSS, you must configure the parameters based on the following list:
  • Delimiter: Select comma (,).
  • Escape Character: Select double quotation marks (").
  • Invalid Fields: Leave it empty.
  • Shipped Fields: Turn off the switch. By default, no field names are added to the first line of the CSV file in OSS.