You can import Pub/Sub messages from Google Cloud to Log Service to analyze and transform message data. This topic describes how to import Pub/Sub messages from Google Cloud to Log Service.
- A pull subscription is created in Google Cloud Pub/Sub. For more information, see Create and use subscriptions.
- A service account is created in Google Cloud, and the key file of the service account is downloaded. For more information, see Create service accounts.
When you create the service account, you must assign the Google Cloud Pub/Sub Editor role to the service account.
- A project and a Logstore are created. For more information, see Create a project and Create a Logstore.
Create a data import configuration
- Log on to the Log Service console.
- In the Projects section, click the project that you want to view.
- Choose >. Then, move the pointer over and click +. . On the Logstores tab, find the Logstore that you want to manage and click
- In the Import Data dialog box, click Google Cloud Pub/Sub - Data Import.
- In the Configure Import Settings step, create a data import configuration.
- Configure the following parameters.
Parameter Description Configuration Name The name of the data import configuration. Project ID The ID of the Google Cloud project. Subscription ID The ID of the Google Cloud Pub/Sub subscription. Import Credentials File The key file of the service account that is assigned the Google Cloud Pub/Sub Editor role. The file is in the JSON format. Message Format The format of a message that is published to Google Cloud Pub/Sub. JSON and TEXT formats are supported. Time Source The time source of a message. You can specify one of the following time sources for a Google Cloud Pub/Sub message. After a message is imported to Log Service, the message uses the specified time source.
- Use System Time: the system time at which the message is imported to Log Service.
- Use Message Publish Time: the time at which the message is published to Google Cloud Pub/Sub.
- Use Time in Messages: the time that is contained in the message. If you select this option, you must also configure the Time and Time Format parameters.
Time If you set Message Format to json and Time Source to Use Time in Messages, you must set this parameter to the time field that is contained in a Google Cloud Pub/Sub message. Time Field Extraction Regular Expression If you set Message Format to Text and Time Source to Use Time in Messages, you must set this parameter to extract the time from a Google Cloud Pub/Sub message. Time Format If you set Time Source to Use Time in Messages, you must specify a time format that is supported by Java SimpleDateFormat. The time format is used to parse the time field. For more information about the syntax of the time format, see Class SimpleDateFormat. For more information about common time formats, see Time formats.Note The Java SimpleDateFormat class does not support UNIX timestamps. If you want to use UNIX timestamps, set Time Format to epoch. Time Zone If you set Time Source to Use Time in Messages, you must specify a time zone. The time zone is used for the parsed message time.
If the specified time format includes information about the time zone, this parameter does not take effect.
- Click Preview to preview the import result.
- Confirm the result and click Next.
- Configure the following parameters.
- Preview data, configure indexes, and then click Next. By default, full-text indexing is enabled for Log Service. You can also configure field indexes based on collected logs in manual mode or automatic mode. To configure field indexes in automatic mode, click Automatic Index Generation. This way, Log Service automatically creates field indexes. For more information, see Create indexes.Important If you want to query and analyze logs, you must enable full-text indexing or field indexing. If you enable both full-text indexing and field indexing, the system uses only field indexes.
- Click Log Query. You are redirected to the query and analysis page of your Logstore. On the page, check whether the import task of Google Cloud Pub/Sub messages is successful. Wait for approximately 1 minute. If you can query the Google Cloud Pub/Sub messages, the import task is successful.
View the data import configuration
After you create a data import configuration, you can view the configuration and related reports in the Log Service console.
- In the Projects section, click the project to which the data import configuration belongs.
- In the left-side navigation pane, choose . Click the Logstore to which the data import configuration belongs, choose , and then click the name of the data import configuration.
- On the Import Configuration Overview page, view the basic information and reports of the data import configuration.
What to do next
On the Import Configuration Overview page, you can perform the following operations on the data import configuration:
- Modify the data import configuration
To modify the data import configuration, click Modify Settings. For more information, see Create a data import configuration.
- Delete the data import configuration To delete the data import configuration, click Delete Configuration.Warning After a data import configuration is deleted, it cannot be restored. Proceed with caution.
- Stop the data import job
To stop the data import task, click Stop.