Setting Up Your Debian Server 4: Prometheus & Grafana

Prometheus is an open-source systems monitoring and alerting toolkit originally built at SoundCloud. Since its inception in 2012, many companies and organizations have adopted Prometheus, and the project has a very active developer and user community. It is now a standalone open-source project and is maintained independently of any company. To emphasize this, and to clarify the project's governance structure, Prometheus joined the Cloud Native Computing Foundation in 2016 as the second hosted project, after Kubernetes.

Grafana is a multi-platform open source analytics and interactive visualization web application. It provides charts, graphs, and alerts for the web when connected to supported data sources. There is also a licensed Grafana Enterprise version with additional capabilities available as a self-hosted installation or an account on the Grafana Labs cloud service. The enterprise version is free, and we will install it on our own server instead of using the cloud service.

node_exporter is a Prometheus exporter for hardware and OS metrics exposed by *NIX kernels, written in Go with pluggable metric collectors.

Installing Prometheus and node_exporter

Instead of installing Prometheus under our current user, we create a new system user to limit some rights:

sudo groupadd --system prometheus
sudo useradd -s /sbin/nologin --system -g prometheus prometheus

Then we create the folders where we will save the binaries required to save Prometheus, and download the binaries from the official website to our downloads folder:

sudo mkdir /var/lib/prometheus
curl -s https://api.github.com/repos/prometheus/prometheus/releases/latest|grep browser_download_url|grep linux-amd64|cut -d '"' -f 4|wget -qi -

We extract the downloaded file, rename it, and move the binaries to the etc folder:

tar -xvf prometheus-2.32.1.linux-amd64.tar.gz
mv prometheus-2.32.1.linux-amd64 prometheus
sudo mv prometheus /etc/

Next, we need to give the ownership of these folders to the Prometheus user we created:

sudo chown prometheus:prometheus /etc/prometheus
sudo chown prometheus:prometheus /var/lib/prometheus
sudo chown -R prometheus:prometheus /etc/prometheus/consoles
sudo chown -R prometheus:prometheus /etc/prometheus/console_libraries

Then we copy some of its scripts to the /usr/local/bin folder:

sudo cp /etc/prometheus/prometheus /usr/local/bin/
sudo cp /etc/prometheus/promtool /usr/local/bin/

Now we need to create the configuration file for Prometheus and add the configuration:

sudo vi /etc/systemd/system/prometheus.service

We add the following lines to the file:

[Unit]
Description=Prometheus
Documentation=https://prometheus.io/docs/introduction/overview/
Wants=network-online.target
After=network-online.target
[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
--config.file /etc/prometheus/prometheus.yml \
--storage.tsdb.path /var/lib/prometheus/ \
--web.console.templates=/etc/prometheus/consoles \
--web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target

Then we reload the daemon, start the service, and check its status:

sudo systemctl daemon-reload
sudo systemctl enable --now prometheus
systemctl status prometheus

Prometheus uses port 9090 by default, so we need to open it:

sudo ufw allow 9090

Now we go to http://your_server_ip:9090 and we should see the Prometheus dashboard.

We will close port 9090 since we will not use it directly; instead, we will set up node_exporter:

sudo ufw deny 9090

We download the node_exporter binary to our download folder and move it to the /usr/local/bin folder:

curl -s https://api.github.com/repos/prometheus/node_exporter/releases/latest| grep browser_download_url|grep linux-amd64|cut -d '"' -f 4|wget -qi -
tar -xvf node_exporter*.tar.gz
cd  node_exporter*/
sudo cp node_exporter /usr/local/bin

We can check the installation by running the following command:

node_exporter --version

We create a new service file for node_exporter by sudo vi /etc/systemd/system/node_exporter.service and add the following lines:

[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=default.target

Then we reload the daemon, start the service, and check its status:

sudo systemctl daemon-reload
sudo systemctl start node_exporter
sudo systemctl enable node_exporter
systemctl status node_exporter.service

By default, node_exporter uses the port 9100, but we are not going to open it since we will use it with Grafana. But first, we need to add the node_exporter to the Prometheus configuration file by sudo vi /etc/prometheus/prometheus.yml and add the following lines:

scrape_configs:
  # The job name is added as a label `job=` to any timeseries scraped from this config.
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]
  - job_name: 'node_exporter'
    static_configs:
      - targets: ['localhost:9100']

Then we restart the Prometheus service:

sudo systemctl restart prometheus

If an external service consumes the outputs, ports 9090 and 9100 should be opened. But since we will use Grafana locally, we will not open them.

Installing Grafana Enterprise

We will install Grafana Enterprise since it is also free. The optimal way is to install with apt-get:

sudo apt-get update
sudo apt-get install -y apt-transport-https
sudo apt-get install -y software-properties-common wget
sudo wget -q -O /usr/share/keyrings/grafana.key https://apt.grafana.com/gpg.key
sudo apt-get install grafana-enterprise

Then we need to start the service and enable it, and since it uses port 3000 by default, we need to open it:

sudo systemctl daemon-reload
sudo systemctl start grafana-server
sudo systemctl status grafana-server
sudo systemctl enable grafana-server.service
sudo ufw allow 3000

Now we can go to http://your_server_ip:3000 and we should see the Grafana dashboard. The default username and password is admin. The GUI will prompt you to change the password, and the username can be changed later from the Account Settings section.

On the Grafana page, create a new connection to http://localhost:9090 and click Save & Test. If it is successful, you should see a green box with the message "Data source is working". Then import a dashboard from the Grafana website. For example, you can import the Node Exporter Server Metrics dashboard. You can also create your own dashboards.

Setting up Domain Name and HTTPS

By default, Grafana does not use HTTPS and serves from the IP. We will set up a domain name and HTTPS. First, we need to redirect our dashboard subdomain to our server IP with a DNS A record. Then we create a standalone certificate with certbot (as we have installed it in the previous part):

sudo certbot certonly --standalone

This requires port 80 to be available, therefore it may be necessary to stop the Apache server momentarily, by sudo systemctl disable apache2. Then we create our certificate for subdomain.mysite.com. Once the certificate is ready, we can restart the Apache server and enable it again by sudo systemctl enable apache2. Then we set up the symlinks for Grafana and adjust the permissions:

sudo ln -s /etc/letsencrypt/live/subdomain.mysite.com/privkey.pem /etc/grafana/grafana.key
sudo ln -s /etc/letsencrypt/live/subdomain.mysite.com/fullchain.pem /etc/grafana/grafana.crt
sudo chgrp -R grafana /etc/letsencrypt/*
sudo chmod -R g+rx /etc/letsencrypt/*
sudo chgrp -R grafana /etc/grafana/grafana.crt /etc/grafana/grafana.key
sudo chmod 400 /etc/grafana/grafana.crt /etc/grafana/grafana.key

Then we need to edit the Grafana configuration file by sudo vi /etc/grafana/grafana.ini and add the following lines:

[server]
http_addr =
http_port = 3000
domain = mysite.com
root_url = https://subdomain.mysite.com:3000
cert_key = /etc/grafana/grafana.key
cert_file = /etc/grafana/grafana.crt
enforce_domain = False
protocol = https

Then we restart the Grafana service by sudo systemctl restart grafana-server. Now we can go to https://subdomain.mysite.com:3000 and we should see the Grafana dashboard.

Note that we still need to type port 3000 into our dashboard address. To handle this, we create a redirection rule for the subdomain.mysite.com so that any requests to the ports 80 and 443 are forwarded to the 3000. We can do this by editing the Apache configuration file by sudo vi /etc/apache2/sites-available/subdomain.mysite.com.conf and adding the following lines:

<VirtualHost *:80>
    ServerName subdomain.mysite.com
    ServerAlias subdomain.mysite.com
    Redirect permanent / https://subdomain.mysite.com:3000
</VirtualHost>
<VirtualHost *:443>
    ServerName subdomain.mysite.com
    ServerAlias subdomain.mysite.com
    Redirect permanent / https://subdomain.mysite.com:3000
    SSLCertificateFile /etc/letsencrypt/live/subdomain.mysite.com/fullchain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/subdomain.mysite.com/privkey.pem
    Include /etc/letsencrypt/options-ssl-apache.conf
</VirtualHost>

Then we restart the Apache server by sudo systemctl restart apache2. Now we can go to https://subdomain.mysite.com and we should see the Grafana dashboard.

In the next part, we will install Syncthing to synchronize files between our devices without using a cloud service.