Logstash common configurations and troubleshooting FAQ - Elasticsearch

How do I access Logstash over the Internet?

Logstash clusters run inside a virtual private cloud (VPC) and have no direct Internet access. To import or export data over the Internet, configure a NAT gateway for the cluster. See Configure a NAT gateway for data transmission over the Internet.

How do I fix Kafka pipeline errors?

Two errors commonly appear when a self-managed Kafka cluster is used as the source or destination of a Logstash pipeline.

"No entry found for connection"

Logstash nodes cannot resolve the Kafka hostname. Add the following lines to the server.properties tab, replacing the IP address and port with your Kafka cluster values:
```
listeners=PLAINTEXT://10.10.10.10:9092
advertised.listeners=PLAINTEXT://10.10.10.10:9092
```
Important
- Make sure your Logstash node IP addresses are in the whitelist of your Kafka cluster.
- We recommend that you use ApsaraMQ for Kafka.
"could not be established. Broker may not be available"

The Kafka cluster does not exist or cannot be connected. Verify that the cluster is running and that the bootstrap_servers value in your pipeline configuration points to the correct address and port.

Does the Logstash JDBC driver support MySQL?

Yes. Upload the mysql-connector-java driver file first. See Configure third-party libraries.

How do I monitor Logstash nodes?

Enable X-Pack Monitoring on your Logstash cluster and associate it with an Elasticsearch cluster. Monitor your Logstash nodes from the Kibana console of the associated Elasticsearch cluster. See Enable the X-Pack Monitoring feature.

Can I upload script files?

No. Use the Logstash configuration file to manage pipelines and data transmission. See Use configuration files to manage pipelines.

Can I use HTTP as the data collection protocol?

Yes. Logstash accepts single-line and multi-line events over HTTP or HTTPS through the Http input plugin.

Note

Alibaba Cloud Logstash cannot connect to the Internet by default. To collect HTTP requests from the Internet, configure a NAT gateway first. See Configure a NAT gateway for data transmission over the Internet.

How do I synchronize data from Simple Log Service to Elasticsearch?

Use the logstash-input-sls plug-in. See Use the logstash-input-sls plug-in.

Does Logstash synchronize data in real time?

Logstash is a near-real-time synchronization tool. It continuously writes data to the destination until the pipeline is stopped or the source has no new data.

How do I fix the `"[routing] is missing for join field [joinField]"` error?

Synchronizing data that contains child documents requires a routing value so that parent and child documents land on the same shard. Set the number of primary shards for the destination index to 1 and add routing =>"0" to the pipeline configuration.

My pipeline is stuck and the cluster update shows no progress. What do I do?

Check the cluster error logs first. See Query logs. The following table lists common causes and solutions.

Cause	Solution
Incorrect pipeline configuration	Pause the cluster update, wait for it to pause, then fix the pipeline configuration to trigger a restart. See View the progress of a cluster task.
High disk usage	Upgrade the cluster configuration. See Upgrade the configuration of a cluster. Then refresh the cluster details page and check the update progress.
Auto Indexing is disabled on the output Elasticsearch cluster	Enable Auto Indexing. See Configure the YML file. Then refresh the cluster details page and check the update progress.
Beats port is outside the 8000--9000 range	Pause the cluster update, wait for it to pause, then change the port to a value between 8000 and 9000 to trigger a restart.
Public IP address used in the pipeline configuration	Change the public IP address to a private IP address, or configure a NAT gateway. See Configure a NAT gateway for data transmission over the Internet. Then refresh the cluster details page and check the update progress.
`file_extend` is specified but the logstash-output-file_extend plug-in is not installed	Install the plug-in (see Use the pipeline configuration debugging feature), or pause the cluster update and remove `file_extend` from the pipeline configuration to trigger a restart.

Can I ping the private IP address of a Logstash cluster?

No. The private IP address does not respond to ping by default. Use telnet to test connectivity instead.

How do I keep mappings consistent when synchronizing between two Elasticsearch clusters?

Alibaba Cloud Elasticsearch disables Auto Indexing by default for security. Logstash creates indexes in the destination cluster by submitting data rather than calling the create index API. Before transferring data, either enable Auto Indexing on the destination cluster or manually create the target index with the correct mappings.

Does Alibaba Cloud Logstash support rubydebug?

No. Use the logstash-output-file_extend plug-in to print debug logs instead. The stdout mode is not recommended and no related APIs are provided.