Simple Log Service:Upload logs by using the Syslog protocol

Limits

• The Syslog protocol must be the standard RFC 5424. If you use other Syslog protocols, we recommend that you see Collect Syslog from Linux systems.

• The maximum size of a log entry is 64 KB.

• To ensure data transmission security, you must transmit data over TCP using Transport Layer Security (TLS) 1.2.

Configuration

To upload logs using the Syslog protocol, you must configure a log upload address in the ProjectName.Endpoint:SyslogPort format. Example: test-project-1.cn-hangzhou-intranet.log.aliyuncs.com:10009. Select an endpoint based on the region where your project is located. For more information, see Endpoints. The Syslog port is 10009. You must also configure parameters in the STRUCTURED-DATA field, such as the Simple Log Service project and LogStore, and your Alibaba Cloud account's AccessKey pair.

Example: Collect logs by using syslog-ng

1. Open the syslog-ng configuration file.

   The configuration file is typically located at /etc/syslog-ng/syslog-ng.conf.

2. Add the following configuration block to the end of the syslog-ng configuration file, and then configure the parameters within it based on your business requirements.

   ### Syslog-ng Logging Config for LogService ### 
   template LogServiceFormat { 
       template("<${PRI}>1 ${ISODATE} ${HOST:--} ${PROGRAM:--} ${PID:--} ${MSGID:--} [logservice project=\"test-project-1\" logstore=\"test-logstore-1\" access-key-id=\"<yourAccessKeyId>\" access-key-secret=\"<yourAccessKeySecret>\"] $MSG\n"); template_escape(no); 
   }; 
   destination d_logservice{ 
        tcp("test-project-1.cn-hangzhou.log.aliyuncs.com" port(10009) 
        tls(peer-verify(required-untrusted)) 
        template(LogServiceFormat)); 
   }; 
   log { 
        source(s_sys); # default use s_sys 
        destination(d_logservice); 
   }; 
   ### END Syslog-ng Logging Config for LogService ###

3. Restart syslog-ng.

   Run the sudo /etc/init.d/syslog-ng restart, sudo service syslog-ng restart, or sudo systemctl restart syslog-ng command to restart syslog-ng.

4. Run the logger command to generate a test log.

   Example: Run the logger hello world! command to generate a log.

Sample log

After you upload logs, you must configure indexes before you can view the logs in the Simple Log Service console. For more information, see Create an index.

03-28 11:01:01      __source__: xxx
                    __topic__: syslog-forwarder
                    _facility_: 3
                    _hostname_: xxx
                    _priority_: 30
                    _program_: systemd
                    _severity_: 6
                    _unixtimestamp_: 1553742061117098000
                    content: Started Session 59532 of user root.
03-28 11:00:15      __source__: mymachine.example.com
                    __topic__: syslog-forwarder
                    _facility_: 4
                    _hostname_: mymachine.example.com
                    _message_id_: ID47
                    _priority_: 34
                    _program_: su
                    _severity_: 2
                    _unixtimestamp_: 1553742015003000000
                    content: this is a test message

FAQ

• Test log uploads

  You can use Netcat to test log uploads to verify network connectivity and ensure that the AccessKey has upload permissions.

  1. Log on to the server on which you want to test log uploads.

  2. Run the following command to install Netcat:

     sudo yum install nmap-ncat

  3. Run the following command to connect to Simple Log Service:

     ncat --ssl <yourProject>.<yourEndpoint> 10009

     Sample command:

     ncat --ssl test-project-1.cn-hangzhou.log.aliyuncs.com 10009

  4. Netcat does not automatically detect network disconnections. Within 30 seconds of running the ncat command, enter the log message and press Enter.

     <34>1 2019-03-28T03:00:15.003Z mymachine.example.com su - ID47 [logservice project="<yourProject>" logstore="<yourLogstore>" access-key-id="<yourAccessKeyID>" access-key-secret="<yourAccessKeySecret>"] this is a test message

     Sample command:

     <34>1 2019-03-28T03:00:15.003Z mymachine.example.com su - ID47 [logservice project="trace-doc-test" logstore="doc-test-001-logs" access-key-id="<yourAccessKeyID>" access-key-secret="<yourAccessKeySecret>"] this is a test message

  5. Preview logs in the Simple Log Service console to check whether the logs are uploaded.

     For more information, see Preview logs.

     A log entry with a timestamp of 2019-03-28 11:00:15 and a source of mymachine.example.com appears in the preview results. The Syslog fields in the content column, including _hostname_, _severity_, _facility_, _message_id_, _unixtimestamp_, _program_, and _priority_, are parsed. This indicates that the log was uploaded and parsed successfully.

• Check network connectivity

  If the ncat command in the previous step produces no output at all (the ncat process exits immediately with no error message), first use telnet to check whether port 10009 is reachable before assuming the network is the problem.

  1. Run the following command to check port connectivity:

     telnet <yourProject>.<yourEndpoint> 10009

     Sample command:

     telnet test-project-1.cn-hangzhou.log.aliyuncs.com 10009

  2. If the output includes Connected to..., the network is reachable. In this case, the issue is likely an incorrect AccessKey or Project configuration. Continue with the ncat test in Test log uploads, or see Troubleshoot syslog upload failures in this section.

  3. If the connection times out or is refused, the network is not reachable. Check your security group, firewall, or network route configuration.

  Note

  Simple Log Service uses a shared access layer. Even if the Project name is incorrect, telnet might still return Connected. telnet only verifies network-layer connectivity. If the network is reachable but log upload still fails, check your AccessKey and Project name.

• Diagnose collection errors

  If you collect Syslog data through the Logtail Syslog plug-in and a manual log upload fails, you can view specific error messages by diagnosing collection errors. For more information, see How to view Logtail collection error information. This diagnostic feature applies only to Logtail collection scenarios. When you upload logs directly by using the Syslog protocol as described in this topic, the data does not go through Logtail, so this diagnostic feature does not apply. See Troubleshoot syslog upload failures in this section instead.

• Troubleshoot syslog upload failures

  If logs sent by using the Syslog protocol are not received by the LogStore, check the following items in order:

  1. Confirm that the SD-ID in STRUCTURED-DATA is fixed to logservice. Any other identifier, such as sls or sls@17055, causes the connection to succeed but the log to be silently dropped.

  2. Confirm that the data is transmitted over TCP using TLS 1.2.

  3. Confirm that the AccessKey pair has write permissions on the target LogStore.

  4. Run ncat --ssl <ProjectName>.<Endpoint> 10009 to verify that the port is reachable.

  5. Go to the LogStore query page in the Simple Log Service console and search for the log to confirm whether it arrived.