×
Community Blog How to Install Graylog on Ubuntu 16.04

How to Install Graylog on Ubuntu 16.04

In this tutorial, we will be installing Graylog on an Alibaba Cloud ECS instance with Ubuntu 16.04.

By Ghulam Qadir, Alibaba Cloud Tech Share Author. Tech Share is Alibaba Cloud's incentive program to encourage the sharing of technical knowledge and best practices within the cloud community.

Graylog is a powerful open-source log management platform that aggregates and extracts important data from server logs, which are often sent using the Syslog protocol. It also allows you to search and visualize the logs in a web interface.

Graylog is compatible and works well with Alibaba Cloud Elastic Compute Service (ECS) instances. In this tutorial, we'll install and configure Graylog on Ubuntu 16.04, and set up a simple input that receives system logs.

Prerequisites

  1. You must have Alibaba Cloud Elastic Compute Service (ECS) activated and verified your valid payment method. If you are a new user, you can get a free account in your Alibaba Cloud account. If you don't know about how to setup your ECS instance, you can refer to this tutorial or quick-start guide.
  2. You should set up your server's hostname.
  3. Access to VNC console in your Alibaba Cloud or SSH client installed in your PC.

Your ECS Ubuntu 16.04 server must have at least 2 GB of RAM, private networking enabled, and a non-root user set up.

After completing the prerequisites, log in as root user with your root username & password via SSH client (e.g. Putty) or VNC console available in your Alibaba Cloud account dashboard.

Before you installing Graylog, you'll need:

  1. Oracle JDK 8 installed, which you can do by following the "Installing the Oracle JDK" section of this Java installation article.
  2. Elasticsearch 2.x, which you can install by following Steps 1 and 2 of the Elasticsearch installation tutorial. Certain versions of Graylog only work with certain versions of Elasticsearch. For example, Graylog 2.x does not work with Elasticsearch 5.x. Refer to this Graylog-Elasticsearch version comparison table for the exact version. This tutorial uses Elasticsearch 2.4.4 and Graylog 2.2.
  3. MongoDB, which can be installed by following the MongoDB tutorial.

Installing Default JRE/JDK

The easiest option for installing Java is using the version packaged with Ubuntu. Specifically, this will install OpenJDK 8, the latest and recommended version.

First, update the package index.

 sudo apt-get update

Next, install Java. Specifically, this command will install the Java Runtime Environment (JRE).

 sudo apt-get install default-jre

There is another default Java installation called the JDK (Java Development Kit). The JDK is usually only needed if you are going to compile Java programs or if the software that will use Java specifically requires it.

The JDK does contain the JRE, so there are no disadvantages if you install the JDK instead of the JRE, except for the larger file size.

You can install the JDK with the following command:

 Sudo apt-get install default-jdk

Installing Oracle JDK

If you want to install the Oracle JDK, which is the official version distributed by Oracle, you will need to follow a few more steps.

First, add Oracle's PPA, then update your package repository.

 sudo add-apt-repository ppa:webupd8team/java
 sudo apt-get update

Then, depending on the version you want to install, execute one of the following commands:

Oracle JDK 8

This is the latest stable version of Java at time of writing, and the recommended version to install. You can do so using the following command:

 sudo apt-get install oracle-java8-installer

Oracle JDK 9

This is a developer preview and the general release is scheduled for March 2017. It's not recommended that you use this version because there may still be security issues and bugs. There is more information about Java 9 on the official JDK 9 website.

To install JDK 9, use the following command:

 sudo apt-get install oracle-java9-installer

Managing Java

There can be multiple Java installations on one server. You can configure which version is the default for use in the command line by using update-alternatives, which manages which symbolic links are used for different commands.

 sudo update-alternatives –-config java

The output will look something like the following. In this case, this is what the output will look like with all Java versions mentioned above installed.

Output

 There are 5 choices for the alternative java (providing /usr/bin/java).


 Section    Path                                       Priority   Status
 * 0      /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java   1081   auto mode 
 1        /usr/lib/jvm/java-6-oracle/jre/bin/java          1      manual mode
 2        /usr/lib/jvm/java-7-oracle/jre/bin               2      manual mode
 3        /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java   1081   manual mode
 4        /usr/lib/jvm/java-8-oracle/jre/bin/java          3      manual mode
 5        /usr/lib/jvm/java-9-oracle/bin/java              4      manual mode

 Press <enter> to keep the current choice[*], or type selection number:

You can now choose the number to use as a default. This can also be done for other Java commands, such as the compiler (javac), the documentation generator (javadoc), the JAR signing tool (jarsigner), and more. You can use the following command, filling in the command you want to customize.

 sudo update-alternatives –-config command

Setting the JAVA_HOME Environment Variable

Many programs, such as Java servers, use the JAVA_HOME environment variable to determine the Java installation location. To set this environment variable, we will first need to find out where Java is installed. You can do this by executing the same command as in the previous section:

 sudo update-alternatives –-config java

Copy the path from your preferred installation and then open /etc/environment using nano or your favorite text editor.

 sudo nano /etc/environment

At the end of this file, add the following line, making sure to replace the highlighted path with your own copied path.

/etc/environment
 JAVA_HOME="/usr/lib/jvm/java-8-oracle"

Save and exit the file, and reload it.

 source /etc/environment

You can now test whether the environment variable has been set by executing the following command:

 echo $JAVA_HOME

This will return the path you just set.

Install Elasticsearch

Elasticsearch is one of the main component which requires Graylog to run, acts as a search server, offers a real-time distributed search and analytics with the RESTful web interface. Elasticsearch stores all the logs sent by the Graylog server and displays the messages whenever user request over the built-in web interface.

This guide covers configuration settings that are required for Graylog.

Let's install the Elasticsearch. First download and install GPG signing key.

 wget –q0 – https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key 
 add –

Configure Elasticsearch repository by running below command.

 Echo "deb https://packages.elastic.co/elasticsearch/2.x/debian stable main" 
 | sudo tee-a /etc/apt/sources.list.d/elasticsearch.list

Update repository cache and install Elasticsearch.

 sudo apt-get update && sudo apt-get install –y elasticsearch

Make Elasticsearch to start automatically on the system startup.

 sudo systemctl enable elasticsearch

Configuring Elasticsearch

We need to modify the Elasticsearch configuration file so that the cluster name matches the one set in the Graylog configuration file. To keep things simple, we'll set the Elasticsearch cluster name to the default Graylog name of graylog. You may set it to whatever you wish, but make sure you update the Graylog configuration file to reflect that change.

Open the Elasticsearch configuration file in your editor:

 sudo nano /etc/elasticsearch/elasticsearch.yml

Find the following line:

/etc/elasticsearch/elasticsearch.yml

 cluster.name: <CURRENT CLUSTER NAME>

Change the cluster.name value to graylog:

/etc/elasticsearch/elasticsearch.yml

 cluster.name: graylog

Save the file and exit your editor.

Since we modified the configuration file, we have to restart the service for the changes to take effect.

 sudo systemctl restart elasticsearch

Now that you have configured Elasticsearch, let's move on to installing Graylog on Elasticsearch.

Disable dynamic scripts to avoid remote execution, by adding the following lines to the server.conf.

 script.inline: false
 script.indexed: false
 script.file: false

Restart the Elasticsearch service to read the new configurations.

 sudo service elasticsearch restart

Wait at least a minute to let the Elasticsearch get fully restarted. Elastisearch should be now listening on 9200 for processing HTTP request, use a CURL to check the response.

Ensure that cluster name shows as "graylog"

 curl –X GET http://localhost:9200
 
 {
   "name" : "Marvin Flumm"
   "cluster_name" : "graylog"
   "version" : {
      "number" : "2.3.3",
      "build_hash" : "218bdf10790eef486ff2c41a3df5cfa32dadcfde",
      "build_timestamp" : "2018-07-28T15:40:04Z",
      "build_snapshot" : false,
      "lucene_version" : "5.5.0"
  },
  "tagline" : "You Know, for Search"
 }

Optional: Test the health of Elasticsearch cluster, make sure the output yields the cluster status as "green"

 curl –XGET 'http://localhost:9200/_cluster/health?pretty=true'
 
 {
   "cluster_name" : "graylog"
   "status" : "green",
   "timed_out" : false,
   "number_of_nodes" : 2,
   "number_of_data_nodes" : 1,
   "active_primary_shards" : 1,
   "active_shards" : 1,
   "relocating_shards" : 0,
   "initializing_shards" : 0,
   "unassigned_shards" : 0,
   "delayed_unassigned_shards" : 0,
   "number_of_pending_tasks" : 0,
   "number_of_in_flight_fetch" : 0,
   "task_max_waiting_in_queue_millis" : 0,
   "active_shards_percent_as_number" : 100.0
 }

Install MongoDB 3.2

Download and install the latest MongoDB from the official website. Import public key on the terminal to begin.

 sudo apt-key adv –keyserver hkp://keyserver.ubuntu.com:80 –-recv EA312927

Add mongodb repository by creating the /etc/apt/sources.list.d/mongodb-org.list file using following command.

 echo "deb http://repo.mongodb.org/apt/debian wheezy/mongodb-org/3.2 main" | sudo tee 
 /etc/apt/sources.list.d/mongodb-org.list

Install MongoDB using the following command.

 sudo apt-get update && sudo apt-get install –y mongodb-org

Start the MongoDB and enable it on the system start-up.

 sudo systemctl start mongod
 sudo systemctl enable mongod

Installing Graylog

In this step, we we'll install the Graylog server.

First, download the package file containing the Graylog repository configuration. Visit the Graylog download page to find the current version number. We'll use version 2.2 for this tutorial.

 wget https://packages.graylog2.org/repo/packages/graylog-2.2-   
 repository_latest.deb

Next, install the repository configuration from the .deb package file, again replacing 2.2 with the version you downloaded.

 sudo dpkg –I graylog-2.2-repository_latest.deb

Now that the repository configuration has been updated, we have to fetch the new list of packages. Execute this command:

 sudo apt-get update

Next, install the graylog-server package:

 sudo apt-get install graylog-server

Lastly, start Graylog automatically on system boot with this command:

 sudo systemctl enable graylog-server.service

Graylog is now successfully installed, but it's not started yet. We have to configure it before it will start.

You must set a secret to secure the user passwords, use the pwgen command to the same.

 pwgen – N 1 –s 96 
   OH9wXpsNZVBA8R5vJQSnkhTB1qDOjCxAh3aE3LvXddtfDlZlKYEyGS24BJAiIxI0sbSTSPovTTnhLkkrUvhSSxodTlzDi5gP

If you get an error like "pwgen: command not found", install pwgen using the following command.

 sudo apt-get install pwgen

Place the secret like below.

 password_secret = OH9wXpsNZVBA8R5vJQSnkhTB1qDOjCxAh3aE3LvXddtfDlZlKYEyGS24BJAiIxI0sbSTSPovTTnhLkkrUvhSSxodTlzDi5gP 

Next is to set a hash (sha256) password for the root user (not to be confused with the system user, root user of graylog is admin). You will need this password to login into the web interface, admin's password can't be changed using web interface; you must edit this variable to set.

Replace "yourpassword" with your own.

 echo –n yourpassword | sha256sum  

 e3c652f0ba0b4801205814f8b6bc49672c4c74e25b497770bb89b22cdeb4e951

Place the hash password.

 root_password_sha2 = e3c652f0ba0b4801205814f8b6bc49672c4c74e25b497770bb89b22cdeb4e951

You can setup email address admin user.

 root_email = "datamounts@gmail.com"

Set time zone of root (admin) user.

 root_timezone = UTC

Graylog server will try to find the Elasticsearch nodes automatically by using multicast mode. But when it comes to larger network, it is recommended to use unicast mode which is best suited one for production.

Add the following entry to graylog server.conf file, replace ipaddress with your ipaddress. You can add multiple hosts with comma separated.

 elasticsearch_discovery_zen_ping_unicast_hosts = ipaddress:9300

Set only one master node by defining the below variable, the default setting is true.

If you add any second Graylog node, set it to false to make the node as a slave. Master node does some periodic tasks that slave nodes won't perform.

 is_master = true

Set the number of log messages to keep per index; it is recommended to have several smaller indices instead of larger ones.

 elasticsearch_max_docs_per_index = 20000000

The following parameter defines to have a total number of indices, if this number is reached old index will be deleted.

 elasticsearch_max_number_of_indices = 20

Shards setting rely on the number of nodes in the particular Elasticsearch cluster, if you have only one node, set it as 1.

 elasticsearch_shards = 1

This the number of replicas for your indices, if you have only one node in Elasticsearch cluster; set it as 0.

Install Graylog Web Interface

From the version 2.x, no more extra web interface component, the web interface is being served directly by Graylog server.

Configure Graylog web interface by editing the server.conf file.

 sudo nano /etc/graylog/server/server.conf

Modify the below entries to let Graylog Web Interface to connect to the Graylog server.

 rest_listen_uri = http://your-server-ip:12900/
 web_listen_uri = http://your-server-ip:9000/

Restart Graylog service.

 sudo systemctl daemon-reload
 sudo systemctl restart graylog-server

Make Graylog server to start automatically on system startup.

 sudo systemctl enable graylog-server

You can check out the server startup logs; it will be useful for you to troubleshoot Graylog in case of any issue.

 sudo tailf /var/log/graylog-server/server.log

On the successful start of graylog-server, you should get the following message in the log file.

 2018-07-28T08:21:41.538Z INFO [ServerBootstrap] Graylog server up and running.

Accessing Graylog Web Interface

The web interface will now be listening on port 9000, point your browser to http://ip-add-ress:9000.

Login with username "admin" and the password you configured at root_password_sha2 on server.conf.

1

Once you logged in, you would see the getting started page.

Click on System/Overview to know the status of Graylog server.

2

Click on System/Overview to know the status of Graylog server.

3

Configure Graylog Inputs

Graylog inputs need to be configured to receive the logs from the external source, i.e., Syslog or any logging system.

Click System –> Inputs –> Select Syslog UDP and then click Launch new input. Fill with the values in the screen like below.

4

Once you have created the inputs, configure rsyslog or forward any system logs to your–ip-address:1514

Following screenshot shows the logs received by Graylog (Graylog console –> Search).

5

That's all! You have successfully installed Graylog 2.0.3 on Ubuntu 16.04.

Conclusion

You now have a working Graylog server with an input source that can collect logs from other servers.

Next, you might want to look into setting up dashboards, alerts, and streams. Dashboards provide a quick overview of your logs. Streams categorize messages, which you can monitor with alerts. To learn more about configuring the more advanced features of Graylog, you can find instructions in the Graylog documentation.

1 0 0
Share on

Alibaba Clouder

1,018 posts | 161 followers

You may also like

Comments

5868553025160064 March 19, 2019 at 7:53 pm

You have mistakes in a lot of your commands but this is good article!!!! Examples . "Echo" "tee-a" the "-y (before the install)" the "add - (should replace - with name of file?) etc..

Alibaba Clouder

1,018 posts | 161 followers

Related Products