Write Review

How to Configure and Install Elastic Stack on Ubuntu 18.04

With the advancement of IT infrastructure, the organizations and professionals are using the cloud services. With the increase in cloud-based servers, the amount of log generation also increases. It is very important to analyze these logs for multiple reasons.

The elastic stack is an open source system which combines Elasticsearch, Logstash, and Kibana.

  • Logstash - Passes the logs to the Elasticsearch
  • Elasticsearch - A database which stores all the parsed logs
  • Kibana - UI integrated with the elastic search to query the required fields.

Prerequisites

  • OS: Ubuntu 18.04
  • RAM: 4GB
  • CPU: 2

Step 1:
Perform a System Update

It is recommended to update the system before installing any packages. Open the terminal run the following commands to update the system.

$ sudo apt update
$ sudo apt -y upgrade

Proceed to the next step after the update process is completed.

Step 2:
Install Java

Installing Java is a must for the Elastic stack to work. In this tutorial, we will install Oracle Java.

To install Oracle Java on your Ubuntu system, you will need to add the Oracle Java PPA by running:

$ sudo add-apt-repository ppa:webupd8team/java

Now update the repository information by running:

$ sudo apt update

Now install the Java by using the following command.

$ sudo apt -y install oracle-java8-installer

Accept the license agreement, and also check that java is successfully installed using the following command.

$ java -version

You will see a message similar to this:

user@anyone:~$ java -version
java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)

You can also set the JAVA_HOME and other defaults by installing oracle-java8-set-default. Run:

$sudo apt -y install oracle-java8-set-default

You can now verify if the JAVA_HOME variable is set by running:

$echo "$JAVA_HOME"

Step 3:
Install Elasticsearch

With the help of a package manager, install Elasticsearch using following command.

$wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

If the prompt is hanging,
Create the Elasticsearch source list:

$echo "deb http://packages.elastic.co/elasticsearch/2.x/debian stable main" | sudo tee -a /etc/apt/sources.list.d/elasticsearch-2.x.list

Update your apt package database:

$sudo apt-get update

Install Elasticsearch with this command:

$sudo apt-get -y install elasticsearch

Elasticsearch is now installed. Edit it’s configurations now, using following commands.

$sudo vi /etc/elasticsearch/elasticsearch.yml

You will want to restrict outside access to your Elasticsearch instance (port 9200), so outsiders can't read your data or shutdown your Elasticsearch cluster through the HTTP API. Find the line that specifies network.host, uncomment it, and replace its value with "localhost" so it looks like this:

elasticsearch.yml excerpt (updated)

network.host: localhost

Save and exit elasticsearch.yml.

Now start Elasticsearch:
Then run the following command to start Elasticsearch on startup:

$sudo update-rc.d elasticsearch defaults 95 10

Step 4:
Install Kibana

Kibana can be installed with a package manager.

Create the Kibana source list:

$echo "deb http://packages.elastic.co/kibana/4.5/debian stable main" | sudo tee -a /etc/apt/sources.list.d/kibana-4.5.x.list

Update your apt package database:

$sudo apt-get update

Install Kibana with this command:

$sudo apt-get -y install kibana

Kibana is now installed.

Open the Kibana configuration file for editing:

$sudo vi /opt/kibana/config/kibana.yml

In the Kibana configuration file, find the line that specifies server.host, and replace the IP address ("0.0.0.0" by default) with "localhost":

kibana.yml excerpt (updated)
server.host: "localhost"

Save and exit.

Now enable the Kibana service, and start it:

$sudo update-rc.d kibana defaults 96 9
$sudo service kibana start

Step 5:
Install Logstash

The logstash is installed from the same repository. So now just create the Logstash source list:

$echo 'deb http://packages.elastic.co/logstash/2.2/debian stable main' | sudo tee /etc/apt/sources.list.d/logstash-2.2.x.list

Update your apt package database:

$sudo apt-get update

Install Logstash with this command:

$sudo apt-get install logstash

Logstash is installed now lets configure it.

Step 6 :
Configure Logstash

Logstash configuration files are in the JSON-format.

Create a configuration file called 02-beats-input.conf and set up our "filebeat" input:

$sudo vi /etc/logstash/conf.d/02-beats-input.conf

Insert the following input> configuration:

02-beats-input.conf

input {
beats {
port => 5044
ssl => true
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
}

Save and quit.

Create a configuration file called 10-syslog-filter.conf, where we will add a filter for syslog messages:

$sudo vi /etc/logstash/conf.d/10-syslog-filter.conf

Insert the following syslog filter configuration:

10-syslog-filter.conf

filter {
  if [type] == "syslog" {
  grok {
  	match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
  	add_field => [ "received_at", "%{@timestamp}" ]
   add_field => [ "received_from", "%{host}" ]
    }
  syslog_pri { }
  date {
  	match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ] }
  }}

Save and quit..

Lastly, we will create a configuration file called 30-elasticsearch-output.conf:

$sudo vi /etc/logstash/conf.d/30-elasticsearch-output.conf

Insert the following output configuration:

/etc/logstash/conf.d/30-elasticsearch-output.conf

  output {
  elasticsearch {
  hosts => ["localhost:9200"]
  sniffing => true
  manage_template => false
  index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
  document_type => "%{[@metadata][type]}"
  }
}

Save and exit.

Test your Logstash configuration with this command:

$sudo service logstash configtest

It should display Configuration OK if there are no syntax errors. Otherwise, try and read the error output to see what's wrong with your Logstash configuration.

Restart Logstash, and enable it, to put our configuration changes into effect:

$sudo service logstash restart
$sudo update-rc.d logstash defaults 96 9

Next, we'll load the sample Kibana dashboards.

Step 7:
Load Kibana Dashboards

First, download the sample dashboards archive to your home directory:

$cd ~
$curl -L -O https://download.elastic.co/beats/dashboards/beats-dashboards-1.1.0.zip

Install the unzip package with this command:

$sudo apt-get -y install unzip

Next, extract the contents of the archive:

unzip beats-dashboards-*.zip

And load the sample dashboards, visualizations and Beats index patterns into Elasticsearch with these commands:

$cd beats-dashboards-*
$./load.sh

Elastic Stack is successfully installed and configured.

 

Check out these top 3 Linux hosting services

Was this article helpful?