You can import Pub/Sub messages from Google Cloud to Log Service to analyze and transform message data. This topic describes how to import Pub/Sub messages from Google Cloud to Log Service.

Prerequisites

Important Alibaba Cloud has proprietary rights to the information in this topic. This topic describes the capabilities of Alibaba Cloud to interact with third-party services. The names of third-party companies and services are used only for reference.
  • A pull subscription is created in Google Cloud Pub/Sub. For more information, see Create and use subscriptions.
  • A service account is created in Google Cloud, and the key file of the service account is downloaded. For more information, see Create service accounts.

    When you create the service account, you must assign the Google Cloud Pub/Sub Editor role to the service account.

  • A project and a Logstore are created. For more information, see Create a project and Create a Logstore.

Create a data import configuration

Important Only the Singapore region is supported.
  1. Log on to the Log Service console.
  2. In the Projects section, click the project that you want to view.
  3. Choose Log Storage > Logstores. On the Logstores tab, find the Logstore that you want to manage and click >. Then, move the pointer over Data Import > Data Import and click +.
  4. In the Import Data dialog box, click Google Cloud Pub/Sub - Data Import.
  5. In the Configure Import Settings step, create a data import configuration.
    1. Configure the following parameters.
      ParameterDescription
      Configuration NameThe name of the data import configuration.
      Project IDThe ID of the Google Cloud project.
      Subscription IDThe ID of the Google Cloud Pub/Sub subscription.
      Import Credentials FileThe key file of the service account that is assigned the Google Cloud Pub/Sub Editor role. The file is in the JSON format.
      Message FormatThe format of a message that is published to Google Cloud Pub/Sub. JSON and TEXT formats are supported.
      Time SourceThe time source of a message. You can specify one of the following time sources for a Google Cloud Pub/Sub message. After a message is imported to Log Service, the message uses the specified time source.
      • Use System Time: the system time at which the message is imported to Log Service.
      • Use Message Publish Time: the time at which the message is published to Google Cloud Pub/Sub.
      • Use Time in Messages: the time that is contained in the message. If you select this option, you must also configure the Time and Time Format parameters.
      TimeIf you set Message Format to json and Time Source to Use Time in Messages, you must set this parameter to the time field that is contained in a Google Cloud Pub/Sub message.
      Time Field Extraction Regular ExpressionIf you set Message Format to Text and Time Source to Use Time in Messages, you must set this parameter to extract the time from a Google Cloud Pub/Sub message.
      Time FormatIf you set Time Source to Use Time in Messages, you must specify a time format that is supported by Java SimpleDateFormat. The time format is used to parse the time field. For more information about the syntax of the time format, see Class SimpleDateFormat. For more information about common time formats, see Time formats.
      Note The Java SimpleDateFormat class does not support UNIX timestamps. If you want to use UNIX timestamps, set Time Format to epoch.
      Time ZoneIf you set Time Source to Use Time in Messages, you must specify a time zone. The time zone is used for the parsed message time.

      If the specified time format includes information about the time zone, this parameter does not take effect.

    2. Click Preview to preview the import result.
    3. Confirm the result and click Next.
  6. Preview data, configure indexes, and then click Next.
    By default, full-text indexing is enabled for Log Service. You can also configure field indexes based on collected logs in manual mode or automatic mode. To configure field indexes in automatic mode, click Automatic Index Generation. This way, Log Service automatically creates field indexes. For more information, see Create indexes.
    Important If you want to query and analyze logs, you must enable full-text indexing or field indexing. If you enable both full-text indexing and field indexing, the system uses only field indexes.
  7. Click Log Query. You are redirected to the query and analysis page of your Logstore. On the page, check whether the import task of Google Cloud Pub/Sub messages is successful.
    Wait for approximately 1 minute. If you can query the Google Cloud Pub/Sub messages, the import task is successful.

View the data import configuration

After you create a data import configuration, you can view the configuration and related reports in the Log Service console.

  1. In the Projects section, click the project to which the data import configuration belongs.
  2. In the left-side navigation pane, choose Log Storage > Logstores. Click the Logstore to which the data import configuration belongs, choose Data Import > Data Import, and then click the name of the data import configuration.
  3. On the Import Configuration Overview page, view the basic information and reports of the data import configuration.

What to do next

On the Import Configuration Overview page, you can perform the following operations on the data import configuration:

  • Modify the data import configuration

    To modify the data import configuration, click Modify Settings. For more information, see Create a data import configuration.

  • Delete the data import configuration
    To delete the data import configuration, click Delete Configuration.
    Warning After a data import configuration is deleted, it cannot be restored. Proceed with caution.
  • Stop the data import job

    To stop the data import task, click Stop.