Installing and Configuring Elastic Stack Components
The Elastic Stack is a suite of innovative, open-source tools used to analyze different sets of data. The Stack includes tools such as Metricbeat and Filebeat which are useful in collecting web server and system logs. The tools then send the logs to Elasticsearch where they are analyzed, searched, and visualized using a browser-based application called Kibana.
On the other hand, Nginx is a robust web server which can serve as a load balancer for (HTTPS, UDP, and TCP), HTTP cache, mail proxy server (POP3, IMAP, and SMTP), or reverse proxy. It’s powerful, open-source software designed to deliver optimal stability and performance.
This tutorial is compiled to help you learn how to install the different components of the Elastic Stack.
Ready to go? Let’s roll!
Pre-installation instructions
- All commands that demand eminent prerogatives are prefixed with sudo.
- Familiarize yourself with setting up your Linode’s timezone and hostname
- Before you embark on the installations, you must configure your web server stack with Nginx on a CentOS server.
- Create a secure, standard user account, remove unnecessary network services, and harden your SSH access.
Once you meet all the above conditions, you can begin the installation. Run the command below to update your system:
$ sudo yum update
Install OpenJDK 8
Next, install OpenJDK 8 application on CentOS 7, since Elasticsearch requires the latest version of Java. Run the command below to institute the headless package of OpenJDK:
$ sudo yum install -y java-1.8.0-openjdk-headless
Check if the version you have installed is the latest version (at least Java 1.8.0). Use the command:
$ java -version
If the installed version is similar to this:
openjdk version "1.8.0_151" OpenJDK Runtime Environment (build 1.8.0_151-b12) OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)
Then, your Java is prime for utilization by Elasticsearch. You can now proceed with the next step.
Install the Elastic Yum Repository
The Elastic Yum Repository is a comprehensive suite that contains all the packages we need for this tutorial.
Use the command below to import the Elastic signing key:
$ sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
Next, develop a Yum repository to utilize the Elastic yum repository:
elastic.repo
[elasticsearch-6.x] name=Elastic repository for 6.x packages baseurl=https://artifacts.elastic.co/packages/6.x/yum gpgcheck=1 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch enabled=1 autorefresh=1 type=rpm-md
Run the command below to update the Yum cache and facilitate the availability of all new packages:
$ sudo yum update
Install the stack components
We intend to use various components of the Elastic Stack for machine metrics and log analysis. Therefore, it’s important you install the following parts:
Elasticsearch.
Elasticsearch stores the metrics and logs received from each beat. For this reason, we’ll install and configure it first to provide the datastore for Kibana and Beats. Run the command below to install the Elasticsearch package:
$ sudo yum install -y elasticsearch
Next, set the Java Virtual Machine heap size to half of the available server memory. Access the /etc/elasticsearch/jvm.options file and modify the Xmx and Xms values. Do not change the other values:
/etc/elasticsearch/jvm.options
-Xms512m -Xmx512m
At this point, we’ll install two very important plugins that enable Filebeat to process and parse certain docs accurately. The first plugin is the ingest-user-agent, which makes it easy for Elasticsearch to flawlessly parse user-agent strings. Run the command below:
$ sudo /usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-user-agent
The second plugin is the Geoip processor.
Note: If the elasticsearch-plugin request for permissions to edit the /etc/elasticsearch path, confirm, then run the command below:
$ sudo /usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-geoip
Next, start and enable Elasticsearch service:
$ sudo systemctl enable elasticsearch $ sudo systemctl start elasticsearch
The service may take some minutes to start, give it time. Then check the available Elasticsearch API:
$ curl localhost:9200
To check if the service started successfully, run the following command to view the latest logs:
$ systemctl status elasticsearch
The Elasticsearch API will deliver a JSON response, such as the one below:
{
"name" : "Q1R2Oz7",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "amcxppmvTkmuApEdTz673A",
"version" : {
"number" : "6.0.0",
"build_hash" : "8f0685b",
"build_date" : "2017-11-10T18:41:22.859Z",
"build_snapshot" : false,
"lucene_version" : "7.0.1",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}Once the Elasticsearch application is successfully installed, you can now install the other components.
Filebeat
Run the command below to install Filebeat:
$ sudo yum install filebeat
Metricbeat
Use this command to install the Metricbeat package:
$ sudo yum install metricbeat
Kibana
Next, install Kibana packae using:
sudo yum install kibana
Configure the Stack
By now you have installed all the necessary component, but they are not properly configured. Next we configure these components
ElasticSearch
For the sake of this tutorial, we need to change the default Elasticsearch settings since we’ve used a single server in the setup.
First, create a makeshift JSON file featuring an index template which directs Elasticsearch to change the number of shards to 1 and not the default number (5). Then the number of replicas should be set to 0 for all the matching index name:
template.json
"template": "*", "settings": "index": "number_of_shards": 1, "number_of_replicas": 0
Next, use curl to generate and index templates featuring the settings that will be used on a the indices created:
$ curl -H'Content-Type: application/json' -XPUT http://localhost:9200/_template/defaults -d @template.json
Elastcisearch will give the output below:
{"acknowledged":true}Kibana
To start and enable Kibana service, use the command:
$ sudo systemctl enable kibana $ sudo systemctl start kibana
We’ll open the web application via the SSH tunnel. Run the following command in a separate terminal window to access Kibana via your local browser:
$ ssh -L 5601:localhost:5601 username@<Linode public IP> -N
The username can be replaced with your ideal Linux username, whereas the <Linode public IP> can be replaced with the public IP address of the Linode.
Filebeat
Here, we utilize the NGINX module so as to handle the most import configuration and give the relevant instructions required by the Stack to process the logs.
Create a /etc/filebeat/filebeat.yml file using your text editor and add the following details:
/etc/filebeat/filebeat.yml
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
setup.kibana:
host: "localhost:5601"
output.elasticsearch:
hosts: ["localhost:9200"]
setup.dashboards.enabled: trueRemove .disabled from the file to enable your Filebeat modules, then enable NGINX module using:
$ sudo mv /etc/filebeat/modules.d/nginx.yml.disabled /etc/filebeat/modules.d/nginx.yml
Run the command below to enable and start Filebeat:
$ sudo systemctl enable filebeat $ sudo systemctl start filebeat
To configure this service, first create /etc/metricbeat/metricbeat.yml and incorporate the content below:
/etc/metricbeat/metricbeat.yml
metricbeat.config.modules:
path: ${path.config}/modules.d/*.yml
setup.kibana:
host: "localhost:5601"
output.elasticsearch:
hosts: ["localhost:9200"]
setup.dashboards.enabled: trueNext, rename the Nginx, Kibana, and Elasticsearch module configuration files:
$ sudo mv /etc/metricbeat/modules.d/elasticsearch.yml.disabled /etc/metricbeat/modules.d/elasticsearch.yml $ sudo mv /etc/metricbeat/modules.d/kibana.yml.disabled /etc/metricbeat/modules.d/kibana.yml $ sudo mv /etc/metricbeat/modules.d/nginx.yml.disabled /etc/metricbeat/modules.d/nginx.yml
Then start and enable your Metricbeat service.
That’s it! You have successfully installed all the components you required to effortlessly monitor Nginx server. Part Two: How to Monitor Nginx using Elastic Stack on CentOS 7 will help you, use these parts to easily monitor your web server.
Special Note: some leading web hosts provide free 1-click installations of other tools for monitoring server performance. At Hostadvice, you can find extensive information the hosting plans and what they include, as well as user and expert reviews on the best web hosting services.
Check out the top 3 Dedicated server hosting services:
- To know further about best VPS hosting, click here.
