How to Monitor CentOS 7 Server with Prometheus
Introduction
SoundCloud created Prometheus, an open source monitoring system. Prometheus, like other monitoring systems like InfluxDB and Graphite, keeps all of its data in a time series database. It does, however, come with a multi-dimensional data model and a strong query language, which allows system administrators to fine-tune the definitions of their metrics as well as generate more accurate results.
Prometheus also provides PromDash (a browser-based tool for creating custom dashboards) and an experimental AlertManager that can issue alerts via e-mail, Flowdock, Slack, HipChat, and other channels.
Prerequisites
You’ll need the following items to complete this tutorial:
One CentOS 7 64-bit
A sudo user who isn’t root, preferably prometheus.
Installation of Prometheus Server
To begin, make a new directory in which to save all of the files downloaded in this guide.
mkdir ~/Downloads
cd ~/Downloads
Download the newest version of the Prometheus server and time-series database from GitHub with curl.
curl -LO "https://github.com/prometheus/prometheus/releases/download/0.16.0/prometheus-0.16.0.linux-amd64.tar.gz"The Prometheus monitoring system is made up of numerous components that must be installed individually. It’s a good idea to have all of the components in one parent directory, so make one with mkdir.
mkdir ~/PrometheusGo to the newly created directory.
cd ~/PrometheusExtract prometheus-0.16.0.linux-amd64.tar.gz with tar.
tar -xvzf ~/Downloads/prometheus-0.16.0.linux-amd64.tar.gzThe Prometheus server installation is now complete. Type the following command to check the installation:
~/Prometheus/prometheus-0.16.0.linux-amd64/prometheus -versionOn your screen, you should see the following message:
prometheus, version 0.16.0 (branch: HEAD, revision: dcb8ba4)
build user: julius@desktop
build date: 20151009-23:51:17
go version: 1.5.1
Installation Node Exporter
Prometheus was created to track down and monitor web services. You should install a tool called Node Exporter to monitor the metrics of your CentOS server. Node Exporter exports a variety of metrics (including disc I/O statistics, CPU load, memory utilisation, network statistics, and more) in a Prometheus-compatible format.
Enter the Downloads directory and use curl to download the most recent version of Node Exporter from GitHub.
cd ~/Downloads && curl -LO https://github.com/prometheus/node_exporter/releases/download/0.11.0/node_exporter-0.11.0.linux-amd64.tar.gzGet into the Prometheus directory and create a new directory called node exporter:
mkdir ~/Prometheus/node_exporter
cd ~/Prometheus/node_exporter
You may now extract node exporter-0.11.0.linux-amd64.tar.gz with the tar command.
tar -xvzf ~/Downloads/node_exporter-0.11.0.linux-amd64.tar.gzNode Exporter as a Service is up and running
Let’s turn Node Exporter into a service to make it easier to start and stop.
Create a unit configuration file called node exporter.service with vi or another text editor.
sudo vi /etc/systemd/system/node_exporter.serviceThis file should provide the path to the node exporter executable as well as the user who should run it. Add the following code as a result:
[Unit]
Description=Node Exporter
[Service]
User=prometheus
ExecStart=/home/prometheus/Prometheus/node_exporter/node_exporter
[Install]
WantedBy=default.target
Save the file after exiting the text editor.
Restart systemd so that it can read the new configuration file.
sudo systemctl daemon-reloadNode Exporter is now available as a service that can be controlled with the systemctl command. Allow it to start automatically when the computer boots up.
sudo systemctl enable node_exporter.serviceYou may now either reboot your server or manually start the service with the following command:
sudo systemctl start node_exporter.serviceUse a browser to access Node Exporter’s web interface at http://your server ip:9100/metrics once it has started. A page with a lot of text should appear:
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0.00023853100000000002
go_gc_duration_seconds{quantile="0.25"} 0.00023998700000000002
go_gc_duration_seconds{quantile="0.5"} 0.00028122
. . .
Installation Prometheus Server
Enter the path to the Prometheus server installation directory:
cd ~/Prometheus/prometheus-0.16.0.linux-amd64You must first build a prometheus.yml configuration file before you can use Prometheus.
vi ~/Prometheus/prometheus-0.16.0.linux-amd64/prometheus.ymlIn the file, paste the following code.
scrape_configs:
- job_name: "node"
scrape_interval: "15s"
target_groups:
- targets: ['localhost:9100']
This adds a scrape configs section to the file and defines the node job. In its array of targets, it contains the URL of your Node Exporter’s web interface. Prometheus scrapes the metrics every fifteen seconds since the scrape interval is configured to fifteen seconds.
You can title your job whatever you like, but “node” allows you to use Node Exporter’s default console templates.
Exit after saving the file.
As a background procedure, start the Prometheus server.
nohup ./prometheus > prometheus.log 2>&1 &The output of the Prometheus server was forwarded to a file called prometheus.log. Using the tail command, you can see the last few lines of the file:
tail ~/Prometheus/prometheus-0.16.0.linux-amd64/prometheus.logThe following messages will appear in the file once the server is ready:
INFO[0000] Starting target manager... file=targetmanager.go line=75
INFO[0000] Listening on :9090 file=web.go line=118
Visit Prometheus’ homepage at http://your server ip:9090 using a browser. The following is the homepage.
Click the Graph tab at the top of the page to confirm that Prometheus is scraping data from Node Exporter. In the text field labelled Expression on the next page, put the name of a metric (for example, node procs running). After that, click the blue Execute button. You should get a graph for that measure if you click Graph (next to Console) just below:
Console templates in Prometheus allow you to examine graphs of a few regularly used metrics. These console templates are only available if the value of job name in Prometheus’ configuration is set to node.
To access the Node Console, go to http://your server ip:9090/consoles/node.html and click on your server, localhost:9100, to see its metrics:
