Documentation
Home Page Simple Log Service Data Collection Data collection best practices Network scenarios and stability Collect logs from on-premises servers

Collect logs from on-premises servers

Updated at:
Copy as MD
Product

This topic explains how to use Nginx as a forward proxy server and proxy mode to collect logs from on-premises servers into Simple Log Service.

Prerequisites

Background information

To collect logs from on-premises servers that lack public Internet access, use proxy mode. This mode allows a server with public Internet access to act as a forward proxy, forwarding logs from other servers to Simple Log Service.

How it works

Logtail exchanges control, service, and monitoring data with Simple Log Service. Control data includes information such as Logtail configuration delivery and authentication. This communication uses HTTP and HTTPS, with service and monitoring data sent over HTTP by default. Therefore, the proxy server must proxy both HTTP and HTTPS traffic.

Nginx is a high-performance, open source HTTP proxy server. It natively supports proxying HTTP traffic but cannot directly proxy HTTPS traffic due to factors like authentication. To enable HTTPS proxying, you must apply an HTTPS patch to Nginx.

Step 1: Configure the proxy server

Use Nginx to configure an on-premises server that has public Internet access as a forward proxy server.

  1. Log on to the machine that will act as the forward proxy server.

  2. Download Nginx and the HTTPS patch.

    1. Download the HTTPS patch.

      git clone https://github.com/chobits/ngx_http_proxy_connect_module.git

    2. Download and decompress Nginx.

      Replace ${version} with your Nginx version. For the latest version, see nginx: download.

      wget http://nginx.org/download/nginx-${version}.tar.gz
tar -xzvf nginx-${version}.tar.gz
cd nginx-${version}/

  3. Apply the HTTPS patch to Nginx.

    In the command, ${patchfile} is the file path. Select the file that corresponds to your Nginx version. For more information, see Select patch.

    patch -p1 < ../ngx_http_proxy_connect_module/patch/${patchfile}.patch

  4. Install Nginx.

    ./configure --add-module=../ngx_http_proxy_connect_module
make && make install

  5. Add the following configuration to the nginx.conf file.

    Replace ${proxy_server_listening_port} and ${dns_server_address} with your actual values.

    server {
     listen                         ${proxy_server_listening_port};
     resolver                       ${dns_server_address};
     # Used to specify the proxy for non-HTTP requests.
     proxy_connect;
     proxy_connect_allow            443;
     proxy_connect_connect_timeout  10s;
     proxy_connect_data_timeout     10s;
     # Used to specify the proxy for HTTP requests.
     location / {
         proxy_pass http://$host;
         proxy_set_header Host $host;
     }
}

  6. Start the Nginx server.

Step 2: Set proxy-related environment variables

Linux systems

You can use one of two methods to set proxy-related environment variables.

Method

Pros

Cons

Use case

Method 1

The configuration affects only the Logtail process and has minimal impact.

The configuration is relatively complex.

Ideal for users who are not familiar with the server's overall network configuration.

Method 2

The configuration is simple.

This server-wide configuration has a broader impact.

Ideal for server administrators who fully understand the network request patterns of all processes on the server.

Method 1

  1. Log on to an on-premises server.

  2. Open the /etc/init.d/ilogtaild file, add the required environment variable to the start() function, and then save the file.

    For more information about environment variables, see Appendix: Proxy environment variables.

    start()
{
    cd $BIN_DIR
    umask $UMASK
    # Add the proxy-related environment variable before $BIN_DIR/ilogtail.
    # This example uses ALL_PROXY. Assume the proxy server IP address is 192.168.1.0 and the listening port is 9000.
    # The on-premises server communicates with the proxy server over HTTP.
    ALL_PROXY=http://192.168.1.0:9000 $BIN_DIR/ilogtail
    RETVAL=$?
}

  3. Run the following command to restart Logtail.

    /etc/init.d/ilogtaild restart

  4. Repeat steps 1 to 3 to set proxy-related environment variables for other on-premises servers.

Method 2

Important

Use this method only if you want to proxy all network requests on the on-premises server, or if you want to proxy only Logtail requests and you fully understand the network behavior of other processes on the server. Otherwise, use Method 1.

  1. Log on to an on-premises server.

  2. Use the export command to add the proxy-related environment variable to the ~/.bash_profile or /etc/profile startup file.

    For more information about environment variables, see Appendix: Proxy environment variables.

  3. Run the following command to apply the environment variable.

    This example uses the ~/.bash_profile startup file.

    source ~/.bash_profile

  4. Run the following command to restart Logtail.

    /etc/init.d/ilogtaild restart

  5. Repeat steps 1 to 4 to configure proxy-related environment variables for other on-premises servers.

Windows systems

  1. Open the Running window, enter regedit, and then click OK.

  2. In the Registry Editor window, navigate to Computer\HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Services\LogtailDaemon, and then click LogtailDaemon.

  3. Right-click, select Create > Multi-String Value, and then name the new value Environment.

  4. Double-click Environment. In the Value data text box, enter the proxy-related environment variable, and then click OK.

    This example uses ALL_PROXY. The proxy server IP address is 192.168.1.0 and the listening port is 9000. The on-premises server communicates with the proxy server over HTTP. For more information about environment variables, see Appendix: Proxy environment variables. For example, enter ALL_PROXY=http://192.168.1.0:9000.

  5. Open the Running window, enter services.msc, and then click OK.

  6. In the Service window, find the Logtail service.

    • For Logtail 0.x.x.x, select the LogtailWorker service.

    • For Logtail 1.0.0.0 or later, select the LogtailDaemon service.

  7. Right-click the service and select Restart.

Step 3: Verify network connectivity

  1. Log on to an on-premises server.

  2. Run the following commands.

    In the following commands, ${region} is the region of your Project and ${project_name} is the name of your Project. Replace them with the actual values.

    curl http://logtail.${region}.log.aliyuncs.com
curl https://logtail.${region}.log.aliyuncs.com
curl http://${project_name}.${region}.log.aliyuncs.com
curl http://ali-${region}-sls-admin.${region}.log.aliyuncs.com

    If a response similar to the following is returned, the network connection is working correctly.

    {"Error":{"Code":"OLSInvalidMethod","Message":"The script name is invalid : /","RequestId":"62591BC7C08B7BD4AA99FCD4"}}

  3. Repeat steps 1 and 2 to verify the network connection for other on-premises servers.

Appendix: Proxy environment variables

Important

You can specify these environment variables in lowercase, but uppercase variables take precedence.

  • To send all HTTP and HTTPS traffic to the same proxy server, add the ALL_PROXY environment variable.

    ALL_PROXY=${address_of_the_forward_proxy_server}

  • To send HTTP and HTTPS traffic to different proxy servers, add the HTTP_PROXY and HTTPS_PROXY environment variables.

    HTTP_PROXY=${address_of_the_server_that_proxies_HTTP_traffic}
HTTPS_PROXY=${address_of_the_server_that_proxies_HTTPS_traffic}

    The proxy server address must be in the format [protocol://[username:password@]]address[:port].

    • protocol (optional): Specifies the communication protocol between the on-premises server and the proxy server. Valid values are http, https, and socks5. If you do not set this parameter, http is used by default.

    • username and password (optional): The credentials to log on to the proxy server.

    • address (required): The IP address of the proxy server.

    • port (optional): The listening port of the proxy server that you configured in the nginx.conf file. For more information, see step 5 in the "Step 1: Configure the proxy server" section. If this parameter is omitted, port 80 is used by default.

    You can also add the NO_PROXY environment variable to specify destinations that should bypass the proxy server. Separate multiple addresses with commas (,). Supported formats include:

    • IP address

    • A leading period (.) matches the domain and its subdomains.

    • * (disables the proxy for all destinations)

Previous：Access Simple Log Service via PrivateLinkNext：High-availability solutions for client data collection