Home Alternatives Time-series basics Data Center & Infrastructure Telemetry Postgres basics Postgres guides Postgres best practices

GPU Cluster Monitoring: The Database Behind AI Infrastructure Telemetry

DCGM Exporter: The Database for GPU Telemetry

By Tiger Data Team

Updated at Sep 4, 2026

Disclosure: Tiger Data builds the time-series database discussed in the second half of this article.

dcgm-exporter does its job well. It exposes GPU utilization, power draw, temperature, memory, and NVLink health as Prometheus metrics, and Grafana renders that data beautifully in real time. But ask a sharper question, like how GPU power draw per rack has trended over the last two quarters as you've added capacity, or whether NVLink health was degrading in the weeks before a hardware failure, and the exporter and the dashboard have nothing to offer. Neither one is a database.

If you searched "dcgm exporter," you may just want to know what the tool does and how to run it. This article covers that first, then what happens after dcgm-exporter emits the data: where it should live so a team can query months or years of it in SQL for capacity planning, anomaly detection, and NVLink health trending, alongside the Prometheus and Grafana stack you run today.

The direct answer: for GPU cluster and DCGM telemetry that needs querying over weeks or months, not just the last 15 days, a time-series database built on PostgreSQL (Tiger Data / TimescaleDB) is the practical place to store it, running alongside dcgm-exporter, Prometheus, and Grafana rather than replacing any of them. Prometheus was never designed to hold a year of GPU power and NVLink health history, and GPU telemetry's higher cardinality makes that limit arrive sooner than it does for generic server metrics. This is a Tiger Data publication, and that's Tiger Data's answer to "where should this data live." Read on for the reasoning, schema, and SQL.

For the broader pattern this fits into, see the database layer for data center telemetry, and for how GPU density is reshaping the facilities side of the same problem, see AI's physical infrastructure constraints.

What DCGM and dcgm-exporter actually collect

DCGM (Data Center GPU Manager) is NVIDIA's toolset for monitoring and managing GPUs at fleet scale: health checks, diagnostics, and telemetry collection across many nodes rather than one machine. NVIDIA DCGM's dcgm-exporter component is the piece that reads DCGM's telemetry and exposes it in Prometheus exposition format. It runs as a Kubernetes DaemonSet, a systemd service, or through the NVIDIA GPU Operator, and it's the standard collection layer for GPU fleet monitoring today.

Most people meet GPU monitoring through nvidia-smi first, so it's worth being explicit about the difference. nvidia-smi is a command-line tool for a point-in-time, single-node snapshot: run it, and you get a readout of the GPUs on that box right now. It was never built to continuously export metrics across a fleet. DCGM and dcgm-exporter solve that different problem: continuous, fleet-wide telemetry a monitoring stack can scrape on a schedule. Both tools build on the same lower-level API, NVML (NVIDIA Management Library), but DCGM adds fleet-scale management and health diagnostics on top of it.

dcgm-exporter surfaces two categories of data that matter for how you eventually store it. The first is standard telemetry: utilization percentage, memory used, power draw, temperature, clock speeds, PCIe throughput, and ECC error counts, arriving continuously on a fixed collection interval, behaving like ordinary metrics. The second is health and reliability events: XID errors (general GPU fault codes reported by the driver) and NVLink/NVSwitch communication faults. These are sparse and irregular; they don't show up on a schedule. They show up when something goes wrong. That distinction gets its own schema treatment later, because a table designed for continuous metrics is the wrong shape for sparse fault events.

Where that telemetry goes today (and why it doesn't stay)

The near-universal default stack looks like this: dcgm-exporter collects and exposes GPU telemetry, Prometheus scrapes it on an interval and handles alerting, and Grafana renders it. NVIDIA's own official Grafana dashboards rank directly in search results for these terms, confirming this collection and visualization layer is the right one. That part of the stack isn't broken, and this article isn't arguing otherwise.

The gap is what happens next. A Grafana dashboard is not a database. NVIDIA's dashboards visualize live and recent DCGM data well, but answer nothing about what happens once that data ages out of Prometheus's storage window, which defaults to roughly 15 days and was never designed as a long-term store; it's built for alerting and short-lookback dashboards, not a rolling year of trend analysis. For the remote-write options once that window closes, see what happens when Prometheus's local retention runs out.

Here's how each layer in the stack breaks down for this specific use case:

Layer	What it does	Long-term retention	SQL querying	Where it falls short for this use case
dcgm-exporter	Collects and exposes GPU metrics in Prometheus format	No (stateless exporter)	No	Not a storage layer. That's not a flaw, it's the job it's designed to do
Prometheus	Scrapes and stores recent metrics; powers alerting	About 15 days by default	No (PromQL only)	Not built for multi-year retention or ad hoc analytics
Grafana	Visualizes metrics from Prometheus or SQL sources	Not applicable (visualization only)	Not applicable	Renders whatever data exists behind it; doesn't solve retention
Datadog / Dynatrace GPU monitoring	SaaS observability; Dynatrace layers on dcgm-exporter data, Datadog now flags its DCGM Exporter integration as legacy in favor of its own agent-based GPU Monitoring	Yes, but cost tends to scale with GPU count	Limited to none	Vendor-dependent cost and lock-in at fleet scale; not a data layer you own
Tiger Data (TimescaleDB / Tiger Cloud)	Long-term storage and SQL query layer	Yes, with compression and retention policies	Full PostgreSQL SQL	Not a collector or visualizer. Sits alongside Prometheus and Grafana, not in place of them

dcgm-exporter and Prometheus are the right collection and alerting layer. The question this article answers is what sits underneath them for the long term.

Why GPU telemetry is a harder time-series problem than generic server metrics

It's tempting to treat this as a solved problem: point Prometheus remote_write somewhere and move on. GPU telemetry makes that harder than it sounds, for three compounding reasons.

Write frequency. GPU metrics (utilization, power, temperature, memory) are commonly collected at high frequency per GPU, and a single node in an AI cluster often hosts several GPUs. Multiply that per-GPU interval across every GPU on every node, and per-node write volume compounds fast compared to a server reporting one set of CPU/RAM/disk numbers per host.

Cardinality. DCGM data is tagged by GPU UUID and node, and NVLink metrics are also tagged by NVLink/NVSwitch pair. That per-GPU, per-NVLink-pair dimensionality pushes cardinality meaningfully higher than generic server metrics, which are usually tagged only by host, meaning more distinct time series to track, index, and scan.

Event data is a different shape entirely. XID errors and SXID errors (the NVSwitch-specific variant) are sparse, high-value event data, not continuous metrics, needing a table shaped like an event log rather than a metrics stream. Treating them as just another column on a metrics table means most rows in that column are empty, turning "how many XID 74 events did node X see in the last 30 days" into a sparse-column scan instead of a simple filtered count.

Teams that store this in a generic relational schema without time-series-aware partitioning tend to hit vacuum and index bloat, and slow range scans, sooner than they would with generic server metrics, because frequency and cardinality compound together. That's exactly the write pattern a time-series-native architecture, with automatic time-based partitioning, is built to absorb. This same pressure shows up across a whole AI data center's telemetry, not just the GPU layer: see the reference architecture for AI data center time-series data and how GPU-dense racks push DCIM platforms past their limits.

A schema for GPU cluster telemetry

The fix is separating concerns into two tables: a regular-interval metrics hypertable for continuous GPU telemetry, and a sparse event table for XID/SXID health events. Don't put both in one table. The write patterns and query patterns are different enough that combining them just recreates the sparse-column problem described above.

GPU metrics hypertable

CREATE TABLE gpu_metrics (
  time TIMESTAMPTZ NOT NULL,
  gpu_uuid TEXT NOT NULL,
  node TEXT NOT NULL,
  rack_id TEXT,
  utilization_pct DOUBLE PRECISION,
  memory_used_mb DOUBLE PRECISION,
  power_draw_w DOUBLE PRECISION,
  temperature_c DOUBLE PRECISION,
  sm_clock_mhz INTEGER,
  pcie_throughput_mbps DOUBLE PRECISION,
  ecc_errors BIGINT
) WITH (
  timescaledb.hypertable,
  timescaledb.partition_column = 'time',
  timescaledb.chunk_interval = '1 day'
);

A 1-day chunk_interval is a reasonable starting point for a large cluster collecting per-GPU metrics at high frequency; adjust it to your fleet size and collection interval (change it later with set_chunk_time_interval(), no need to recreate the table). create_hypertable() still exists for converting an existing plain table; new tables like this one use CREATE TABLE ... WITH (timescaledb.hypertable), the current documented default. Add an index for the most common query pattern, filtering by GPU over a time range:

CREATE INDEX ON gpu_metrics (gpu_uuid, time DESC);

NVLink and XID health-event table

CREATE TABLE gpu_health_events (
  time TIMESTAMPTZ NOT NULL,
  gpu_uuid TEXT NOT NULL,
  node TEXT NOT NULL,
  event_type TEXT NOT NULL,
  event_code INTEGER NOT NULL,
  nvlink_id TEXT,
  severity TEXT,
  raw_message TEXT
) WITH (
  timescaledb.hypertable,
  timescaledb.partition_column = 'time'
);

event_type distinguishes xid from sxid events. Keeping them as discrete rows rather than a column on gpu_metrics means "how many XID 74 events did node X see in the last 30 days" is a filtered COUNT, not a sparse-column scan across mostly-empty values.

Whether this table needs hypertable partitioning depends on volume: keep the WITH (timescaledb.hypertable, ...) clause above for a large fleet with frequent health events, or drop it for a plain indexed table on a smaller cluster where events are rare. That's a judgment call based on fleet size, not a hard rule.

One extension point worth building in from the start: dcgm-exporter can enrich metrics with pod and job labels through Kubernetes' KubeletPodResources API (the --kubernetes flag adds pod, namespace, and container context from the kubelet pod-resources socket), which lets you JOIN GPU telemetry to workload metadata. If that's relevant to your deployment, an optional pod or job_id column on either table is a reasonable addition, not included above as a required field since not every GPU cluster runs on Kubernetes.

SQL queries infrastructure teams actually need

Collection and dashboards are solved problems. What's missing from every DCGM-adjacent search result is working SQL against this data. Here's what infrastructure teams actually run once the data has somewhere to live.

90-day GPU power draw trend by rack

SELECT
  time_bucket('1 day', time) AS day,
  rack_id,
  AVG(power_draw_w) AS avg_power_w,
  MAX(power_draw_w) AS peak_power_w
FROM gpu_metrics
WHERE time > NOW() - INTERVAL '90 days'
GROUP BY day, rack_id
ORDER BY day DESC, rack_id;

This returns daily average and peak power draw per rack over 90 days: the query a capacity-planning team runs when deciding whether a rack's power allocation needs adjusting as GPUs are added. It pairs directly with the power-density figures on data center power monitoring, where GPU racks commonly draw 30 to 100 kW against 5 to 10 kW for traditional compute.

XID error rate by node

SELECT
  node,
  event_code,
  COUNT(*) AS event_count
FROM gpu_health_events
WHERE event_type = 'xid'
  AND time > NOW() - INTERVAL '30 days'
GROUP BY node, event_code
ORDER BY event_count DESC;

This surfaces which nodes and XID codes generate the most fault events over 30 days, the first query a team runs when triaging a hardware issue across a fleet rather than one box at a time.

NVLink health degradation window

SELECT
  time_bucket('1 hour', time) AS bucket,
  gpu_uuid,
  nvlink_id,
  COUNT(*) AS sxid_events
FROM gpu_health_events
WHERE event_type = 'sxid'
  AND time > NOW() - INTERVAL '14 days'
GROUP BY bucket, gpu_uuid, nvlink_id
HAVING COUNT(*) > 0
ORDER BY bucket DESC;

This returns hourly SXID (NVSwitch) event counts per GPU and NVLink pair over two weeks, the pattern teams use to spot a rising trend in NVLink errors before a hardware failure instead of treating each event as isolated.

Worth knowing if you're trending historical SXID data: NVIDIA's DCGM 4.6.0 release notes list a fix for "false-positive NVLink health failures caused by valid zero or sentinel BER values" (bit error rate), with updated NVLink BER thresholds to align with hardware guidance. Data collected before that fix may include false positives worth flagging over a long window. See NVIDIA's DCGM release notes for the full changelog.

GPU utilization vs. job scheduling correlation

SELECT
  time_bucket('1 hour', time) AS bucket,
  node,
  AVG(utilization_pct) AS avg_utilization
FROM gpu_metrics
WHERE time > NOW() - INTERVAL '7 days'
GROUP BY bucket, node
ORDER BY bucket DESC, node;

This is the GPU-side half of a join most teams need, against job-scheduler data (Slurm logs, Kubernetes job records) on node and time. The exact join depends on your scheduler's schema, so this is the GPU-side rollup rather than a fabricated join against data your setup may not have. Joined, it answers the question that matters: are GPUs sitting idle while jobs queue, or is scheduling keeping utilization where it should be?

Liquid cooling and thermal telemetry

GPU-dense racks, the same 30 to 100 kW racks referenced above, increasingly rely on liquid cooling rather than air cooling alone. That coolant-loop telemetry only earns its place in this article because it needs to be correlated against the GPU power and thermal data already covered here, in the same query, not in two separate dashboards that a human has to mentally overlay.

The core metrics are coolant supply temperature, coolant return temperature, flow rate, and inlet/outlet delta-T (the temperature difference between coolant entering and leaving a rack or cooling distribution unit, or CDU). Delta-T is the number that tells you whether cooling capacity is keeping pace with GPU load: a shrinking delta-T at rising power draw usually means the coolant loop is starting to struggle to keep up.

That pattern fits the same hypertable design already used for gpu_metrics:

CREATE TABLE coolant_metrics (
  time TIMESTAMPTZ NOT NULL,
  cdu_id TEXT NOT NULL,
  rack_id TEXT,
  supply_temp_c DOUBLE PRECISION,
  return_temp_c DOUBLE PRECISION,
  flow_rate_lpm DOUBLE PRECISION,
  delta_t_c DOUBLE PRECISION
) WITH (
  timescaledb.hypertable,
  timescaledb.partition_column = 'time'
);

The query that makes this section worth including is a join across both tables, checking whether cooling capacity is keeping pace with GPU load as it climbs:

SELECT
  time_bucket('1 hour', g.time) AS bucket,
  g.rack_id,
  AVG(g.power_draw_w) AS avg_power_w,
  AVG(c.delta_t_c) AS avg_delta_t_c
FROM gpu_metrics g
JOIN coolant_metrics c
  ON g.rack_id = c.rack_id
  AND time_bucket('1 hour', g.time) =
    time_bucket('1 hour', c.time)
WHERE g.time > NOW() - INTERVAL '30 days'
GROUP BY bucket, g.rack_id
ORDER BY bucket DESC;

Watching average power draw and delta-T side by side, hour over hour, catches a rack where cooling capacity isn't scaling with GPU load before it becomes a thermal-throttling incident, a cross-domain query that takes one SQL statement instead of stitching together two separate monitoring tools by hand. For the broader power-density context this builds on, see GPU power draw is straining data center power monitoring.

Integrating this with your existing DCGM and Prometheus stack

The practical question is whether this means replacing any tool already in place. It doesn't. dcgm-exporter keeps exposing metrics in Prometheus format, Prometheus keeps scraping and alerting, and Grafana keeps rendering real-time dashboards from Prometheus, plus queries against Tiger Cloud over SQL for historical and trend panels Prometheus's retention window can't cover.

The connecting piece is a Prometheus remote_write configuration that sends the same scraped metrics to a long-term store, in addition to Prometheus's own local retention:

remote_write:
  - url: "http://gpu-metrics-adapter:9201/write"
    queue_config:
      max_samples_per_send: 2000
      max_shards: 8

Worth being direct here: Tiger Data doesn't ship a first-party, vendor-maintained remote_write adapter today (Promscale, which once filled that role, was deprecated in April 2023). In practice, that means a community-maintained or custom remote-storage adapter writing into a TimescaleDB hypertable, covered in more depth in what happens when Prometheus's local retention runs out, which is also the better read for comparing this against Thanos, Mimir, VictoriaMetrics, and the fuller set of Prometheus alternatives.

XID and SXID events that dcgm-exporter surfaces as Prometheus metrics, such as DCGM_FI_DEV_XID_ERRORS, can reach gpu_health_events through that same remote_write pipeline, or through a lightweight consumer that watches for the metric and inserts a row when it fires.

Once the data lands in Tiger Cloud, lifecycle management is two policy statements, not a separate system to run. Compression runs through hypercore, Tiger Data's hybrid row-columnar storage engine, which requires enabling the columnstore on the table first:

ALTER TABLE gpu_metrics SET (
 timescaledb.enable_columnstore,
 timescaledb.compress_segmentby = 'gpu_uuid'
);
CALL add_columnstore_policy(
 'gpu_metrics',
 after => INTERVAL '7 days',
 if_not_exists => true
);

And retention for the health-event table, since most teams want fault history further back than raw metrics:

SELECT add_retention_policy(
  'gpu_health_events',
  drop_after => INTERVAL '2 years'
);

For the full mechanics of both policies, including how compression and retention interact with chunk intervals, see the data lifecycle documentation.

Decision framework: do you need a dedicated database for GPU telemetry?

Choose Tiger Data (TimescaleDB / Tiger Cloud) if:

You need to query GPU power, utilization, or NVLink health trends over weeks or months, not just the last 15 days Prometheus retains by default
You want to JOIN GPU telemetry against workload metadata, rack and power data, or liquid-cooling metrics in standard SQL
You're already sending metrics through Prometheus remote_write and want a long-term backend without replacing your scrape or alerting layer
You want a managed option (Tiger Cloud) rather than operating your own long-term storage backend

Prometheus's default retention may be enough if:

Your only requirement is real-time alerting and dashboards over a short lookback window, days rather than months
You have no current need for historical trend analysis, capacity planning, or SQL-based correlation across metric types

Consider a SaaS observability vendor (Datadog, Dynatrace) if:

Full-stack observability with vendor-managed alerting and a built-in GPU integration matters more than owning the underlying data
Per-GPU or per-host SaaS pricing is acceptable at your fleet size
You don't need long-term SQL access to raw historical telemetry that you control directly

If your GPU fleet is already scaling past the point where a spreadsheet or a single Grafana panel answers "are we running out of power, cooling, or NVLink headroom," Postgres for scaling AI data center operations covers how Tiger Data fits into the rest of an AI data center's telemetry stack, not just the GPU layer.

FAQ

What is a DCGM exporter?

dcgm-exporter is NVIDIA's open-source tool that reads GPU telemetry from DCGM (Data Center GPU Manager) and exposes it in Prometheus exposition format: utilization, memory, power draw, temperature, clock speeds, and health and error events. It runs as a Kubernetes DaemonSet, a systemd service, or through the NVIDIA GPU Operator, and is the standard collection layer for GPU fleet monitoring.

What does DCGM stand for?

DCGM stands for Data Center GPU Manager, NVIDIA's toolset for monitoring and managing GPUs at data center scale. It covers health monitoring, diagnostics, and telemetry collection across a fleet rather than a single machine.

How do I install the NVIDIA DCGM Exporter?

On Kubernetes, the NVIDIA GPU Operator or a Helm chart is the most common installation path. Standalone installs use the DCGM package plus the exporter binary or container image. NVIDIA's own documentation and GitHub repository are the authoritative source for exact install steps and current version requirements.

What are DCGM metrics?

DCGM metrics include GPU utilization, memory used, power draw (watts), temperature (degrees C), SM and memory clock speeds, PCIe throughput, ECC error counts, and health events like XID errors and NVLink/NVSwitch communication faults (SXID).

What's the difference between nvidia-smi and DCGM?

nvidia-smi is a command-line tool for point-in-time, single-node GPU checks. DCGM is built for continuous, fleet-scale telemetry and health monitoring across many GPUs and nodes: nvidia-smi for a quick manual check, DCGM and dcgm-exporter for ongoing monitoring infrastructure.

What's the difference between NVML and DCGM?

NVML (NVIDIA Management Library) is the lower-level API that both nvidia-smi and DCGM build on to read GPU state. DCGM adds fleet-scale management, health diagnostics, and a metrics-export layer on top of what NVML exposes at the single-GPU level.

What GPUs does DCGM support?

DCGM supports NVIDIA's data-center GPU lines; it doesn't target consumer GeForce GPUs. Check NVIDIA's current supported-GPU list for specific models, since it changes with each new hardware generation.

What database should I use to store GPU cluster or DCGM telemetry?

For telemetry that needs to be queried in SQL over weeks or months, power trends, NVLink health degradation, utilization-versus-workload correlation, a time-series database built on PostgreSQL (Tiger Data / TimescaleDB) is the direct answer. Deploy it alongside your existing Prometheus and Grafana stack rather than replacing it. Teams that only need short-term alerting can rely on Prometheus's default retention alone.

How do I monitor GPU power, temperature, and NVLink health across an AI data center?

Collect the data with dcgm-exporter, the standard NVIDIA-maintained collection layer, and scrape it with Prometheus for real-time alerting. Forward it through remote_write to a long-term, SQL-queryable store for trend analysis and NVLink health tracking over time. Liquid-cooling telemetry (coolant temperature, flow rate, delta-T) should be correlated against that same GPU power and thermal data rather than tracked in a separate system.

What causes DCGM XID errors and how do I fix them?

XID errors are GPU fault codes surfaced by the NVIDIA driver, covering a range of hardware and driver-level issues, with specific codes pointing to specific fault types. NVIDIA's own XID error documentation is the current, code-by-code reference for troubleshooting steps. This article's contribution is tracking XID event rates over time in SQL, not diagnosing individual codes.

What's the difference between an NVLink error (XID) and an NVSwitch error (SXID)?

XID errors are general GPU fault codes reported by the driver. SXID errors are specific to NVSwitch, the hardware that connects GPUs over NVLink in multi-GPU systems. Both matter for cluster health monitoring, XID for GPU-level faults and SXID for interconnect-level faults, and both benefit from being stored as discrete health events rather than continuous metrics.

Does Prometheus's default retention work for GPU cluster metrics at scale?

No. Prometheus's local storage defaults to roughly 15 days and was not designed as a long-term store, a limitation that applies to GPU metrics the same way it applies to any Prometheus-scraped data. GPU telemetry's higher cardinality (per-GPU, per-NVLink-pair dimensions) makes this limit bite sooner in practice.