Prometheus is a monitoring system that collects time series metrics from targets at regular intervals. Those metrics are stored with labels, queried with PromQL (Prometheus's custom query language), and graphed in tools like Grafana. In a typical setup, Prometheus scrapes metrics from exporters running beside the systems you care about: a node exporter that exports system metrics like CPU, memory and network utilization, a postgresql exporter that makes pg_stat* data available, or an application endpoint that exposes custom metrics.
That model is powerful, but it mostly answers questions from inside the system. The blackbox exporter answers a different question: can a host or service be reached externally? This can be used to probe a remote endpoint to see if a host is up, how long DNS took to resolve the name, how many days are left on the TLS certificate, and the response received from an HTTP GET. This makes it a great fit for monitoring websites, load balancers, TLS endpoints, DNS resolution, and basic network reachability.
The blackbox exporter can run on any host Prometheus can scrape, or on multiple systems if you want to test how services look from different locations.
Setting Up The Blackbox Exporter
The blackbox exporter is distributed as a single Go binary. You can download the release that matches your operating system and CPU architecture from the blackbox exporter releases page, extract it, and place the binary in your favorite bin location.
To get the blackbox_exporter up and running it is advisable to create a non-privileged system user to run the daemon as:
$ sudo useradd --system --no-create-home --shell /usr/sbin/nologin blackbox
$ sudo install -o root -g blackbox -m 0750 -d /usr/local/etc/blackbox_exporter
The blackbox exporter configuration is built around modules. A module tells the
exporter how to probe a target: use HTTP, open a TCP connection, perform a DNS
lookup, send an ICMP echo request, etc. The target itself is not stored in this
file. Prometheus passes the target at scrape time with a URL such as
/probe?target=https://example.com&module=http_2xx. This describes the host
(e.g., in this example) you want to scrape and
the module to use (e.g., http_2xx).
To enable one more more modules you will need to create a blackbox exporter config file. Here is a sample annotated config that enables my favorite modules:
# Probe modules. Each module describes how blackbox_exporter probes a target;
# the target itself is passed as a URL parameter by Prometheus when scraping
# (e.g. /probe?target=https://example.com&module=http_2xx).
modules:
# HTTP / HTTPS probe. Captures DNS lookup time, HTTP status, response time,
# and (for HTTPS targets) TLS handshake time and certificate expiry.
# Use for: most website availability checks.
http_2xx:
prober: http
timeout: 5s
http:
preferred_ip_protocol: ip4
ip_protocol_fallback: true
valid_status_codes: []
method: GET
follow_redirects: true
fail_if_ssl: false
fail_if_not_ssl: false
tls_config:
insecure_skip_verify: false
# Stricter HTTPS probe -- fails if the target is not served over TLS.
https_2xx:
prober: http
timeout: 5s
http:
preferred_ip_protocol: ip4
valid_status_codes: []
method: GET
follow_redirects: true
fail_if_not_ssl: true
tls_config:
insecure_skip_verify: false
# TCP + TLS handshake probe. Scrape with target=host:port (e.g. example.com:443).
# Captures probe_ssl_earliest_cert_expiry and probe_tls_version_info even when
# the service does not speak HTTP.
tls_connect:
prober: tcp
timeout: 5s
tcp:
tls: true
tls_config:
insecure_skip_verify: false
# DNS A-record lookup. Scrape with target=<dns-server-ip> and module=dns_a;
# set the name to look up via __param_query_name in relabel_configs (the
# default 'example.com' is a placeholder).
dns_a:
prober: dns
timeout: 5s
dns:
transport_protocol: udp
preferred_ip_protocol: ip4
query_name: example.com
query_type: A
# ICMP ping probe (IPv4). Scrape with target=<host-or-ip>.
# Requires the blackbox_exporter process to have CAP_NET_RAW (set via the
# systemd service unit) or to run as root.
icmp:
prober: icmp
timeout: 5s
icmp:
preferred_ip_protocol: ip4
ip_protocol_fallback: true
The http_2xx module is the general-purpose website check. It sends an HTTP
GET, follows redirects, accepts any 2xx status code, and validates TLS
certificates when the target URL uses HTTPS. The preferred_ip_protocol: ip4
setting tells the exporter to try IPv4 first, while ip_protocol_fallback: true
allows it to fall back if the preferred protocol cannot be used.
The https_2xx module is stricter because fail_if_not_ssl: true requires the
target to be served over TLS. Use this when a site should never answer over
plain HTTP. It also validates the certificate chain and hostname because
insecure_skip_verify: false, so a bad or mismatched certificate causes the
probe to fail.
The tls_connect module is for TLS services that do not speak HTTP. For
example, you can point tls_connect at example.com:443 or an internal TLS
load balancer and collect certificate metadata from the handshake.
The dns_a module asks a DNS server for an A record. In this example
query_name: example.com is a placeholder. In a production scrape config you
would usually set the DNS server as the target and override the name being
looked up through
Prometheus relabeling
The icmp module performs ping-style reachability checks over IPv4. ICMP uses
raw sockets, so the exporter needs the CAP_NET_RAW Linux capability or it needs
to run as root. The systemd unit below grants only the needed capability instead
of running the whole exporter as root.
Running the blackbox_exporter under systemd
systemd is the service manager used by most modern Linux distributions. It is
responsible for starting services at boot, restarting them when policy says it
should, ordering services around network and filesystem readiness, and giving
you a consistent way to inspect service state and logs with systemctl and
journalctl.
A systemd service definition is called a unit file. Unit files installed by
packages usually live under distribution-managed directories such as
/usr/lib/systemd/system or /lib/systemd/system. Files under
/etc/systemd/system are for local administrator-managed units and custom
overrides. Since this example installs the blackbox exporter by hand,
I will add the following unit file to
/etc/systemd/system/blackbox_exporter.service.
[Unit]
Description=Prometheus Blackbox Exporter
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User=blackbox
Group=blackbox
ExecStart=/usr/local/bin/blackbox_exporter \
--config.file=/usr/local/etc/blackbox_exporter/blackbox.yml \
--web.listen-address=127.0.0.1:9115
Restart=on-failure
RestartSec=5s
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=full
# Required for ICMP (ping) probes -- grants raw-socket access without root.
AmbientCapabilities=CAP_NET_RAW
CapabilityBoundingSet=CAP_NET_RAW
[Install]
WantedBy=multi-user.target
The unit file starts the exporter as the blackbox user, reads the configuration
from /usr/local/etc/blackbox_exporter/blackbox.yml, and listens on port
9115. The Restart=on-failure and RestartSec=5s settings make systemd
restart the process after a crash. NoNewPrivileges, PrivateTmp, and
ProtectSystem reduce what the process can do if it is compromised. The
capability lines grant raw socket access for ICMP probes without giving the
service full root privileges.
The service listens on 127.0.01:9115, which limits inbound connections
to localhost. If the blackbox_exporter needs to be accessible for
remote scraping you can change this address to use the wildcard address
0.0.0.0:9115. If Prometheus scrapes the blackbox_exporter from
across the network, keep the listener reachable
but restrict access with a host firewall or network ACL.
To start the black_box exporter we can reload the unit files, enable the service and start the daemon:
$ sudo systemctl daemon-reload
$ sudo systemctl enable --now blackbox_exporter
$ sudo systemctl status blackbox_exporter
If the service fails, the journal is the fastest place to look for problems:
$ sudo journalctl -u blackbox_exporter -n 50 --no-pager
You can also run a manual probe before touching Prometheus:
$ curl "http://127.0.0.1:9115/probe?target=https://example.com&module=http_2xx"
That command should return Prometheus-formatted metrics such as
probe_success, probe_duration_seconds, probe_http_status_code,
probe_dns_lookup_time_seconds, and, for HTTPS targets,
probe_ssl_earliest_cert_expiry.
Configuring Prometheus
Prometheus does not scrape the website directly. It scrapes the blackbox
exporter's /probe endpoint and passes the real website URL as a query
parameter. That is why relabeling matters: the original target needs to become
__param_target, and the actual scrape address needs to become the exporter.
For HTTP and HTTPS website checks, add a job like this to your Prometheus configuration:
- job_name: blackbox_http
metrics_path: /probe
params:
module:
- http_2xx
relabel_configs:
- source_labels:
- __address__
target_label: __param_target
- regex: (?:https?://)?([^:/]+)(?::\d+)?(?:/.*)?
replacement: ${1}
source_labels:
- __param_target
target_label: instance
- replacement: 127.0.0.1:9115
target_label: __address__
scrape_interval: 60s
static_configs:
- targets:
- https://example.com
The metrics_path: /probe setting tells Prometheus to call the
blackbox_exporter probe
endpoint instead of the default /metrics path. The params block adds
module=http_2xx to the scrape URL, so the exporter knows which module from
blackbox.yml to use.
The first relabel rule copies the original target, such as
https://example.com, into __param_target. Prometheus turns that internal
label into the target=https://example.com query parameter. The second relabel
rule rewrites the visible instance label to the hostname. This keeps graphs
and alerts cleaner than labeling the series with the full URL. The third relabel
rule replaces the actual scrape address with 127.0.0.1:9115, which is where
the blackbox exporter is listening in this example.
After relabeling, Prometheus effectively scrapes this URL:
http://127.0.0.1:9115/probe?module=http_2xx&target=https://example.com
For ICMP checks, use the same pattern with the icmp module:
- job_name: blackbox_icmp
scrape_interval: 60s
metrics_path: /probe
params:
module: [icmp]
static_configs:
- targets:
- "example.com"
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: "127.0.0.1:9115"
The ICMP job uses example.com instead of a full URL because ping checks a host
or IP address, not an HTTP resource. The first relabel rule still passes that
host as the target query parameter. The second rule keeps instance equal to
the probed host, and the final rule sends the scrape to the exporter.
Once Prometheus reloads, the most important metric is probe_success. A value
of 1 means the probe worked, and 0 means it failed. From there you can look
at the supporting metrics to understand why: DNS lookup time, TLS handshake
time, HTTP status code, total probe duration, ICMP round trip time, and
certificate expiration.
Viewing Metrics With Grafana
After Prometheus is collecting metrics from the blackbox_exporter you can build your own Grafana dashboard or import an existing one such as the website monitoring dashboard. The useful panels usually answer a few direct questions: is the endpoint up, is it using SSL, how many days are left on the certificate, which TLS version is in use, how long did DNS resolution take, how many redirects occurred, and how long did it take to retrieve the URI.
Here is a Grafana dashboard screenshot that shows the HTTP status, SSL state, certificate age, DNS lookup time, HTTP version, redirects, ICMP reachability, availability, and probe duration:
The next dashboard screenshot breaks the probe down by phase. This is where the blackbox exporter becomes especially useful for troubleshooting. If total response time jumps, you can see whether the delay came from DNS resolution, TCP connect time, TLS negotiation, server processing, content transfer, or ICMP round trip time:
That phase data is what turns a simple uptime check into something more useful.
If you see probe_success == 0 you know the endpoint failed. Seeing
that DNS lookup time spiked, or that the TLS certificate has only a few days
left, points directly at the next thing to investigate.
Conclusion
The blackbox exporter is a small tool with a lot of operational value. It checks the things users actually feel: whether a site answers, whether TLS works, whether certificates are still valid, whether DNS is fast, whether redirects are behaving, and whether a host is reachable via ICMP. It also fits cleanly into a normal Prometheus setup because Prometheus still scrapes metrics the same way; the exporter simply performs the tests on its behalf.
I want to send a huge thank you to the Prometheus developers as well as the Prometheus community members. Prometheus has become a GO TO for a ton of organizations, and this wouldn't be possible without their dedication and hard work. Viva la Prom!