All Products
Search
Document Center

Function Compute (2.0):Use Loggie Agent in custom runtimes

Last Updated:Feb 03, 2024

Loggie is a Golang-based cloud-native agent that is used to collect logs. It features light weight and high performance. You can use Loggie Agent in a function in a custom runtime to collect logs from files and upload the logs to Simple Log Service for storage and custom analysis.

Before you start

Procedure

Step 1: Create a function in a custom runtime

  1. Log on to the Function Compute console. In the left-side navigation pane, click Services & Functions.

  2. In the top navigation bar, select a region. On the Services page, click the desired service.

  3. On the Functions page, click Create Function.

  4. On the Create Function page, configure the following parameters, use the default values for other parameters, and then click Create. For more information, see Create a function.

    • Creation method: Select Use Custom Runtime.

    • Basic Settings: Enter a function name in the Function Name field and select Event Handler for Handler Type.

    • Code: Configure the runtime environment and code-related information of the function.

      Parameter

      Example

      Runtime

      Python 3.9

      Code Upload Method

      Select Use Folder. The name of the folder to be uploaded is code, and the file in the code directory is app.py. The following sample code shows the content of app.py:

      from flask import Flask
      from flask import request
      import logging
      import os
      
      REQUEST_ID_HEADER = 'x-fc-request-id'
      
      app = Flask(__name__)
      
      format_str = '[%(asctime)s] %(levelname)s in %(module)s: %(message)s'
      logging.basicConfig(filename='/tmp/log/fc-flask.log', filemode='w', 
          format=format_str, encoding='utf-8', level=logging.DEBUG)
      @app.route("/invoke", methods = ["POST"])
      def hello_world():
          rid = request.headers.get(REQUEST_ID_HEADER)
          logger = logging.getLogger()
      
          print("FC Invoke Start RequestId: " + rid)
          logger.info("FC Invoke Start RequestId: " + rid)
      
          data = request.stream.read()
          print(str(data))
          logger.info("receive event: {}".format(str(data)))
          
          print("FC Invoke End RequestId: " + rid)
          logger.info("FC Invoke Start RequestId: " + rid)
          return "Hello, World!"
      
      if __name__ == '__main__':
          app.run(host='0.0.0.0',port=9000)
      Note

      You can change filename='/tmp/log/fc-flask.log' to the actual log type and log location in the code. Make sure that the configuration is consistent with the sources.paths path configured in Step 2.

      Startup Command

      /code/bootstrap

      Note

      The bootstrap file is created in Step 2.

      Listening Port

      9000

Step 2: Create a bootstrap file as the startup command

  1. After the function is created, use WebIDE on the Code tab to create a bootstrap file in the CODE directory.

    The following sample code shows the content of the bootstrap file.

    #!/bin/bash
    
    #1. Create the pipelines.yml file.
    mkdir -p /tmp/log /code/etc
    cat << EOF > /code/etc/pipelines.yml
    pipelines:
      - name: demo
        sources:
          - type: file
            name: fc-demo
            addonMeta: true
            fields:
              topic: "loggie"
            fieldsUnderRoot: true
            paths:
              - "/tmp/log/*.log"
        sink:
          type: sls
          endpoint: ${LOGGIE_SINK_SLS_ENDPOINT}
          accessKeyId: ${LOGGIE_SINK_SLS_ACCESS_ID}
          accessKeySecret: ${LOGGIE_SINK_SLS_ACCESS_SECRET}
          project: ${LOGGIE_SINK_SLS_PROJECT}
          logstore: ${LOGGIE_SINK_SLS_LOGSTORE}
          topic: ${LOGGIE_SINK_SLS_TOPIC}
    EOF
    
    #2. Create the loggie.yml file.
    cat << EOF > /code/etc/loggie.yml
    EOF
    
    #3. Start Loggie Agent and run it as a background process.
    /opt/bin/loggie -config.system=/code/etc/loggie.yml -config.pipeline=/code/etc/pipelines.yml > /tmp/loggie.log 2>&1 &
    
    #4. Start the application.
    exec python app.py

    The script performs the following operations:

    1. Create the pipelines.yml file, which is the pipeline configuration file.

      • sources

        Specifies the type and path of the logs. This example shows how to collect logs from all files that end with .log in the /tmp/log directory.

      • sink

        Specifies information about Simple Log Service. The variables in the script are set in Step 4.

    2. Create the loggie.yml file, which is the Loggie configuration file.

      If this file is empty, the default configurations are used. In this example, the default configurations are used. However, the loggie.yml file must exist. If the file is not empty, see Overview for the detailed configurations.

    3. Start Loggie Agent and run it as a background process. The run logs of Loggie Agent are printed to the /tmp/loggie.log file.

    4. Start the application. In this example, Python is used. In actual scenarios, specify the programming language that you want to use.

  2. Obtain the execution permission on the bootstrap file.

    In WebIDE, choose Terminal > New Terminal and run the chmod 777 bootstrap command to set make the file executable.

Step 3: Add the Loggie Agent official common layer

  1. Click the Configurations tab. In the Layers section, click Modify.

  2. In the panel for modifying layers, choose Add Layer > Add Official Common Layer and configure the Loggie Agent layer.

    The following table provides information about the Loggie Agent common layer.

    Layer name

    Compatible runtime

    Layer version

    ARN

    Loggie Agent

    Custom runtime

    This example uses layer version 1.

    acs:fc:{region}:official:layers/Loggie13x/versions/1

  3. Click OK.

Step 4: Set environment variables

  1. In the Environment Variables section of the Configurations tab, click Modify.

  2. In the panel for modifying environment variables, add the following environment variables. For more information about how to configure environment variables, see Environment variables.

    • Set environment variable FC_EXTENSION_SLS_LOGGIE=true.

      After you add this environment variable, the instance is frozen 10 seconds after the function is invoked. This ensures that Loggie Agent can report logs as expected.

      Important

      You are charged for using the FC_EXTENSION_SLS_LOGGIE=true environment variable. The billing rules of the environment variable are the same as the billing rules of Prefreeze hooks. For more information, see Billing rules.

    • Set environment variables in the pipelines.yml file, including LOGGIE_SINK_SLS_ENDPOINT, LOGGIE_SINK_SLS_ACCESS_ID, LOGGIE_SINK_SLS_ACCESS_SECRET, LOGGIE_SINK_SLS_PROJECT, LOGGIE_SINK_SLS_LOGSTORE, and LOGGIE_SINK_SLS_TOPIC.

  3. Click OK. After the function configurations are updated, the execution logs of the function can be uploaded to Simple Log Service by using Loggie.

Step 4: Verify results

  1. On the Code tab, click Test Function.

    You may experience some delay in log upload if you are testing the function for the first time. We recommend that you test the function several more times.

  1. Log on to the Log Service console and query logs based on the region, project, and Logstore configurations in the pipelines.yml file. The following figure shows an example.

    image.png

    • body: the log.

    • state.*: the metadata of the log collection state. hostname is the ID of the instance where the function runs.

Troubleshooting

Loggie Agent runs independently in a function instance. Function Compute cannot detect whether Loggie Agent is running as expected. Execution of functions is not affected even if Loggie Agent malfunctions.

When you query logs that are related to Loggie Agent in Simple Log Service, latency of seconds may occur. If you cannot find logs that are related to Loggie Agent in Simple Log Service, perform the following operations to troubleshoot the issue:

Function runs as expected

In this case, the instance is alive for several minutes after the function is invoked. You can log on to the instance to view the running status and logs of Loggie Agent. For more information about how to log on to an instance, see Run commands to manage function instances.

  • If logs do not exist, you can start Loggie Agent in the command line.

  • If logs exist, troubleshoot the issue based on the logs.

    • Check whether the pipelines.yml file is correctly configured.

    • Check whether the Simple Log Service sink is successfully started. Logs are similar to pipeline sink(sink/sls)-0 invoke loop start.

    • Check whether the log file is obtained. Logs are similar to start collect file: /tmp/log/fc-flask.log. If no similar log is available, check whether a log file is generated based on the paths configuration in the pipelines.yml file.

Note

The first time you connect to a Simple Log Service Logstore, latency may occur. If no anomaly is found in the logs, you can invoke the function multiple times and wait a few minutes before you query the logs.

Function fails to run

If a function fails to run, you can remove the startup logic from Loggie Agent to check whether the function is running as expected. In most cases, Loggie Agent does not affect the running of functions because Loggie Agent is an external extension. If unexpected exits of processes or execution timeouts occur, you can scale up the memory or increase the CPU specifications.

References

  • For more information about Loggie, visit the Loggie website.

  • In this example, Loggie collects logs and uploads them without any processing. If you want to process logs before you upload them, for example, parse JSON format logs or remove DEBUG logs, you can add Interceptor configurations in the pipelines.yml file. For more information, see Overview.