database-observability

Grafana Cloud Database Observability

Docs: https://grafana.com/docs/grafana-cloud/monitor-applications/database-observability/

Query-level insights (RED metrics, query samples, explain plans) for MySQL and PostgreSQL without application code changes.

Safety rules

• Present every SQL statement and configuration change to the user for review first, and prefer handing the SQL to a DBA to run.
• Before running any CREATE USER, GRANT, CREATE EXTENSION, or configuration change, get explicit user confirmation of the target instance and environment. A request to diagnose slow queries is a request for analysis only; ask the user before making privileged changes.
• Database restarts and parameter group or flag changes cause downtime. Leave restarts to the user, who schedules them in a maintenance window after confirming.
• The password secret in these examples is an example value only. Replace it everywhere, including the Alloy connection strings, with a strong, unique, randomly generated password kept in a secret store, and run CREATE USER only once that replacement is made.
• Keep real database passwords and connection strings out of committed Alloy config; supply them from a secret store or environment the operator controls.
• Install Alloy only from official Grafana release channels, verify signatures or checksums, and confirm in the linked docs that the version includes the database observability components.

Common Workflows

PostgreSQL setup

-- 1. Enable pg_stat_statements (requires restart)
--    In postgresql.conf:  shared_preload_libraries = 'pg_stat_statements'
--    Then restart PostgreSQL.
--    First run SHOW shared_preload_libraries and add pg_stat_statements to the existing list instead of replacing it.
--    The restart causes downtime: the user performs it in a maintenance window after confirming.

-- 2. Create a least-privilege monitoring user
--    Replace secret with a strong, randomly generated password before running this.
CREATE USER grafana_monitoring WITH PASSWORD 'secret';
GRANT pg_monitor TO grafana_monitoring;
--    pg_monitor shows this user query text and statistics from every session in the cluster; see Data handling.
GRANT CONNECT ON DATABASE mydb TO grafana_monitoring;

-- 3. Enable the extension on each monitored DB
--    Run this as a superuser or the provider admin account, not as grafana_monitoring; keep the monitoring user free of elevated roles.
CREATE EXTENSION IF NOT EXISTS pg_stat_statements;

-- 4. Verify
SELECT count(*) FROM pg_stat_statements;
-- Should return >= 0 (not an error). If "relation does not exist": step 3 didn't run.

1. Add the Alloy config block from references/alloy-config.md § PostgreSQL
2. Restart Alloy and wait ~60s for first scrape
3. Verify in Grafana Cloud → Database Observability — the instance should appear with non-empty db_query_total{db_instance="mydb"} series. If empty after 2 minutes: check Alloy logs for connection errors, then re-verify pg_stat_statements permissions. Confirm metric and label names against the linked docs.

MySQL setup

-- 1. Create a limited privilege monitoring user
--    Replace secret with a strong, randomly generated password before running this.
--    Replace the percent host wildcard in CREATE USER and every GRANT with the address or subnet of the Alloy host, and require SSL for this account.
CREATE USER 'grafana_monitoring'@'%' IDENTIFIED BY 'secret';
GRANT PROCESS, REPLICATION CLIENT ON *.* TO 'grafana_monitoring'@'%';
GRANT SELECT ON performance_schema.* TO 'grafana_monitoring'@'%';
--    Explain plans and schema details also need SELECT on each monitored application schema; grant it per schema only, never on the mysql system schema or on all schemas.
--    Leave out these table grants when those collectors are disabled.

-- 2. Verify performance_schema is enabled (default in MySQL 8+)
SELECT @@performance_schema;
-- Returns 1 → enabled. 0 → set performance_schema=ON in my.cnf + restart.
--    That restart causes downtime: the user performs it in a maintenance window after confirming.

1. Add the Alloy config block from references/alloy-config.md § MySQL
2. Restart Alloy, wait ~60s, verify the instance appears in Grafana Cloud → Database Observability

Supported databases

For managed databases (RDS / Cloud SQL / Azure), pg_stat_statements is usually available but may not be enabled. Check SHOW shared_preload_libraries first, then follow your provider's documentation to enable it via the parameter group / flag, then re-create the extension at the DB level. Parameter changes may require a restart, so schedule them with the user.

What you get

• Query Performance dashboard (auto-provisioned): top queries by total time / call count / mean latency, sampled query texts with parameters, visual explain plans, RED metrics per query digest
• APM correlation: drill from a slow service-latency span to the specific slow SQL via db.statement / db.system / db.name OTel attributes — see references/queries-alerts.md § Trace correlation
• Alerting on query latency, error rate, and connection-pool saturation — full alert YAML in references/queries-alerts.md § Alert rules

Data handling

• Sampled query text, explain plans, and schema details leave the host and are stored in Grafana Cloud. Query text can contain personal data or secrets carried in literal parameters.
• Get the user's confirmation that this data may be sent off the host before enabling query samples or explain plans on production databases. Remove query_samples and explain_plans from enable_collectors when that is not acceptable.
• The pg_monitor role lets the monitoring user see current query text and statistics for every session across the whole cluster; take extra care on shared clusters.
• Trace correlation records SQL in the db.statement span attribute. Configure statement sanitization in the OTel instrumentation or a redaction step in the collector so only parameterized SQL is recorded.

References

• references/alloy-config.md — full Alloy database_observability.postgres / database_observability.mysql config blocks + collector reference (pg_stat_statements, query_samples, schema_details, explain_plans)
• references/queries-alerts.md — key PromQL queries, alert rule YAML, and trace-correlation setup