Advanced Observability

Zerops Advanced Observability provides one-click, self-hosted, pre-configured deployments of the ELK stack or Prometheus with Grafana. This enables comprehensive observability for your deployments while maintaining full control over your data and infrastructure.

Deployment Modes

Both ELK and Prometheus can be deployed in two ways:

• Local deployment: Services are deployed within your target project
• Global deployment: A dedicated project is created specifically for observability

ELK Stack

The ELK observability setup deploys and configures two base services:

• Elasticsearch (as elkstorage)
• Kibana (as kibana) - the observability access point

Logging

For log collection with ELK, see the Log Forwarding guide which covers the Logstash setup and configuration.

Tracing

To collect traces in the ELK stack, an APM Server instance (deployed as the apmserver service) is required. APM Server listens for incoming traces securely over HTTPS and indexes them to Elasticsearch.

First, follow the one-click GUI integration setup to deploy the required infrastructure. Then configure your application to send traces to APM Server.

Setting Up APM Agents

1. Update your code according to the APM documentation for your application's language. See this simple Go application using APM libraries for reference.

2. Add the following environment variables to your application service (these are specific to the official Go APM Agent library and may differ for other agents):

ELASTIC_APM_ACTIVE=true
ELASTIC_APM_SERVICE_NAME=recipe-go
ELASTIC_APM_SERVER_URL=https://apmserver.url.copy.from.gui
ELASTIC_APM_SECRET_TOKEN=secret_token_copy_from_gui

Note

The ELASTIC_APM_ACTIVE variable is set to false by default on Zerops, so you must explicitly set it to true to enable APM for the official Go APM agent.

3. Restart or reload your application service.

You should start seeing traces appear in Kibana's "Applications > Traces" section.

What Happens Behind the Scenes

1. The elkstorage, kibana, and apmserver services are deployed and configured in the target project (if not already present)
2. The apmserver is made publicly accessible over HTTPS via a Zerops subdomain and configured with a secret token for secure access

Access information for both Kibana and APM Server can be found in the Advanced Observability section of your project in the GUI.

Next Steps

• Security: Set up custom domains for Kibana and APM Server
• Optimization: Connect to APM Server locally via HTTP for improved performance
• Customization: Fork the ELK recipe, customize the Logstash configuration, and redeploy the logstash service via zcli

Prometheus with Grafana

The full Prometheus setup consists of:

• Prometheus (as prometheus)
• Grafana (as grafana) - the observability access point
• PostgreSQL database (as grafanadb) - for Grafana data storage
• S3 bucket (as prometheusbackups) - for storing Prometheus backups

Metrics can be forwarded to the target project via a Prometheus exporter (single prometheuslight service configured with remote write). When present in a source project, it collects metrics and forwards them to Prometheus in the target project.

Since Prometheus supports only filesystem storage and non-HA (single instance) setup, the prometheus service is configured to run a backup cron job that stores metric data snapshots in object storage.

Metrics

After deploying the full or forwarder Prometheus setup, metrics are automatically scraped by Prometheus. After setup completes, log in to Grafana using the credentials found in the Advanced Observability section of your project in the GUI, then navigate to /dashboards to start exploring your metrics.

Zerops provides several metrics out of the box:

• Service scaling and resource metrics
• PostgreSQL database service metrics (with HAProxy balancer metrics in HA setups)
• MariaDB database service metrics
• Valkey database metrics

Note

Some PostgreSQL database metrics are available only after enabling the pg_stat_statements extension. Run CREATE EXTENSION IF NOT EXISTS pg_stat_statements; as a superuser and restart the database service.

You can also expose custom metrics from your applications.

Exposing Custom Metrics

1. Configure your application to expose an HTTP /metrics endpoint on an arbitrary port. See this simple Go application using promauto for reference.

   Note

   Only the /metrics path is scraped and cannot currently be configured otherwise.

2. Add and commit the ZEROPS_PROMETHEUS_PORT environment variable to your application service with the port where you exposed the /metrics endpoint:

Single port:
9090

Multiple ports can be defined using commas:
9090,9091

For example: ZEROPS_PROMETHEUS_PORT=9090. After setting the environment variable, your service's ports will be automatically added to metrics discovery and scraped by Prometheus.

What Happens Behind the Scenes

Full setup:

1. The prometheus, grafana, grafanadb, and prometheusbackups services are deployed and configured (if not already present)
2. grafana is made publicly accessible over HTTPS, securely accessible via a generated password

After that, metrics are scraped and available for visualization in Grafana.

Forwarder setup:

1. The prometheuslight forwarder service is deployed in the source project
2. The prometheus, grafana, grafanadb, and prometheusbackups services are deployed and configured in the target project (if not already present)
3. grafana is made publicly accessible over HTTPS, securely accessible via a generated password
4. prometheus is made publicly accessible over HTTPS via a Zerops zone domain and secured with basic access authentication (username and password)
5. prometheuslight is configured to forward metrics to the target prometheus using the generated credentials over secure HTTPS

Next Steps

• Security: Set up custom domains for Grafana and Prometheus
• Customization: Fork the Prometheus recipe, customize dashboards and alerting rules, and redeploy via zcli
• Advanced monitoring: Explore Grafana's alerting capabilities and create custom dashboards for your specific use cases