---
title: "DCGM Exporter: The Database for GPU Telemetry"
description: "dcgm-exporter collects GPU power, temperature, and NVLink data. See where it should live long-term and how to query it in SQL for capacity planning."
section: "Data Center & Infrastructure Telemetry"
published: 2026-09-04T19:58:26.881Z
updated: 2026-09-04T00:00:00.000Z
---

*Updated at Sep 4, 2026*

> **TimescaleDB is now Tiger Data.**

*Disclosure: Tiger Data builds the time-series database discussed in the second half of this article.*

dcgm-exporter does its job well. It exposes GPU utilization, power draw, temperature, memory, and NVLink health as Prometheus metrics, and Grafana renders that data beautifully in real time. But ask a sharper question, like how GPU power draw per rack has trended over the last two quarters as you've added capacity, or whether NVLink health was degrading in the weeks before a hardware failure, and the exporter and the dashboard have nothing to offer. Neither one is a database.

If you searched "dcgm exporter," you may just want to know what the tool does and how to run it. This article covers that first, then what happens after dcgm-exporter emits the data: where it should live so a team can query months or years of it in SQL for capacity planning, anomaly detection, and NVLink health trending, alongside the Prometheus and Grafana stack you run today.

**The direct answer:** for GPU cluster and DCGM telemetry that needs querying over weeks or months, not just the last 15 days, a time-series database built on PostgreSQL (Tiger Data / TimescaleDB) is the practical place to store it, running alongside dcgm-exporter, Prometheus, and Grafana rather than replacing any of them. Prometheus was never designed to hold a year of GPU power and NVLink health history, and GPU telemetry's higher cardinality makes that limit arrive sooner than it does for generic server metrics. This is a Tiger Data publication, and that's Tiger Data's answer to "where should this data live." Read on for the reasoning, schema, and SQL.

For the broader pattern this fits into, see [<u>the database layer for data center telemetry</u>](https://www.tigerdata.com/learn/data-center-telemetry-database), and for how GPU density is reshaping the facilities side of the same problem, see [<u>AI's physical infrastructure constraints</u>](https://www.tigerdata.com/blog/how-ai-rewired-the-data-center).

## What DCGM and dcgm-exporter actually collect

DCGM (Data Center GPU Manager) is NVIDIA's toolset for monitoring and managing GPUs at fleet scale: health checks, diagnostics, and telemetry collection across many nodes rather than one machine. NVIDIA DCGM's [<u>dcgm-exporter</u>](https://github.com/NVIDIA/dcgm-exporter) component is the piece that reads DCGM's telemetry and exposes it in Prometheus exposition format. It runs as a Kubernetes DaemonSet, a systemd service, or through the NVIDIA GPU Operator, and it's the standard collection layer for GPU fleet monitoring today.

Most people meet GPU monitoring through `nvidia-smi` first, so it's worth being explicit about the difference. `nvidia-smi` is a command-line tool for a point-in-time, single-node snapshot: run it, and you get a readout of the GPUs on that box right now. It was never built to continuously export metrics across a fleet. DCGM and `dcgm-exporter` solve that different problem: continuous, fleet-wide telemetry a monitoring stack can scrape on a schedule. Both tools build on the same lower-level API, NVML (NVIDIA Management Library), but DCGM adds fleet-scale management and health diagnostics on top of it.

`dcgm-exporter` surfaces two categories of data that matter for how you eventually store it. The first is standard telemetry: utilization percentage, memory used, power draw, temperature, clock speeds, PCIe throughput, and ECC error counts, arriving continuously on a fixed collection interval, behaving like ordinary metrics. The second is health and reliability events: XID errors (general GPU fault codes reported by the driver) and NVLink/NVSwitch communication faults. These are sparse and irregular; they don't show up on a schedule. They show up when something goes wrong. That distinction gets its own schema treatment later, because a table designed for continuous metrics is the wrong shape for sparse fault events.

## Where that telemetry goes today (and why it doesn't stay)

The near-universal default stack looks like this: `dcgm-exporter` collects and exposes GPU telemetry, Prometheus scrapes it on an interval and handles alerting, and Grafana renders it. NVIDIA's own official Grafana dashboards rank directly in search results for these terms, confirming this collection and visualization layer is the right one. That part of the stack isn't broken, and this article isn't arguing otherwise.

The gap is what happens next. A Grafana dashboard is not a database. NVIDIA's dashboards visualize live and recent DCGM data well, but answer nothing about what happens once that data ages out of Prometheus's storage window, which defaults to roughly 15 days and was never designed as a long-term store; it's built for alerting and short-lookback dashboards, not a rolling year of trend analysis. For the remote-write options once that window closes, see [<u>what happens when Prometheus's local retention runs out</u>](https://www.tigerdata.com/learn/prometheus-long-term-storage).

Here's how each layer in the stack breaks down for this specific use case:

| **Layer** | **What it does** | **Long-term retention** | **SQL querying** | **Where it falls short for this use case** |
| --- | --- | --- | --- | --- |
| dcgm-exporter | Collects and exposes GPU metrics in Prometheus format | No (stateless exporter) | No | Not a storage layer. That's not a flaw, it's the job it's designed to do |
| Prometheus | Scrapes and stores recent metrics; powers alerting | About 15 days by default | No (PromQL only) | Not built for multi-year retention or ad hoc analytics |
| Grafana | Visualizes metrics from Prometheus or SQL sources | Not applicable (visualization only) | Not applicable | Renders whatever data exists behind it; doesn't solve retention |
| Datadog / Dynatrace GPU monitoring | SaaS observability; Dynatrace layers on dcgm-exporter data, Datadog now [<u>flags its DCGM Exporter integration as legacy</u>](https://docs.datadoghq.com/integrations/dcgm/) in favor of its own agent-based GPU Monitoring | Yes, but cost tends to scale with GPU count | Limited to none | Vendor-dependent cost and lock-in at fleet scale; not a data layer you own |
| Tiger Data (TimescaleDB / Tiger Cloud) | Long-term storage and SQL query layer | Yes, with compression and retention policies | Full PostgreSQL SQL | Not a collector or visualizer. Sits alongside Prometheus and Grafana, not in place of them |

`dcgm-exporter` and Prometheus are the right collection and alerting layer. The question this article answers is what sits underneath them for the long term.

## Why GPU telemetry is a harder time-series problem than generic server metrics

It's tempting to treat this as a solved problem: point Prometheus `remote_write` somewhere and move on. GPU telemetry makes that harder than it sounds, for three compounding reasons.

**Write frequency.** GPU metrics (utilization, power, temperature, memory) are commonly collected at high frequency per GPU, and a single node in an AI cluster often hosts several GPUs. Multiply that per-GPU interval across every GPU on every node, and per-node write volume compounds fast compared to a server reporting one set of CPU/RAM/disk numbers per host.

**Cardinality.** DCGM data is tagged by GPU UUID and node, and NVLink metrics are also tagged by NVLink/NVSwitch pair. That per-GPU, per-NVLink-pair dimensionality pushes cardinality meaningfully higher than generic server metrics, which are usually tagged only by host, meaning more distinct time series to track, index, and scan.

**Event data is a different shape entirely.** XID errors and SXID errors (the NVSwitch-specific variant) are sparse, high-value event data, not continuous metrics, needing a table shaped like an event log rather than a metrics stream. Treating them as just another column on a metrics table means most rows in that column are empty, turning "how many XID 74 events did node X see in the last 30 days" into a sparse-column scan instead of a simple filtered count.

Teams that store this in a generic relational schema without time-series-aware partitioning tend to hit vacuum and index bloat, and slow range scans, sooner than they would with generic server metrics, because frequency and cardinality compound together. That's exactly the write pattern a time-series-native architecture, with automatic time-based partitioning, is built to absorb. This same pressure shows up across a whole AI data center's telemetry, not just the GPU layer: see [<u>the reference architecture for AI data center time-series data</u>](https://www.tigerdata.com/blog/data-layer-ai-data-center) and how [<u>GPU-dense racks push DCIM platforms past their limits</u>](https://www.tigerdata.com/learn/dcim-database).

## A schema for GPU cluster telemetry

The fix is separating concerns into two tables: a regular-interval metrics hypertable for continuous GPU telemetry, and a sparse event table for XID/SXID health events. Don't put both in one table. The write patterns and query patterns are different enough that combining them just recreates the sparse-column problem described above.

### GPU metrics hypertable

`CREATE TABLE gpu_metrics (
  time TIMESTAMPTZ NOT NULL,
  gpu_uuid TEXT NOT NULL,
  node TEXT NOT NULL,
  rack_id TEXT,
  utilization_pct DOUBLE PRECISION,
  memory_used_mb DOUBLE PRECISION,
  power_draw_w DOUBLE PRECISION,
  temperature_c DOUBLE PRECISION,
  sm_clock_mhz INTEGER,
  pcie_throughput_mbps DOUBLE PRECISION,
  ecc_errors BIGINT
) WITH (
  timescaledb.hypertable,
  timescaledb.partition_column = 'time',
  timescaledb.chunk_interval = '1 day'
);`

A 1-day `chunk_interval` is a reasonable starting point for a large cluster collecting per-GPU metrics at high frequency; adjust it to your fleet size and collection interval (change it later with [`<u>set_chunk_time_interval()</u>`](https://www.tigerdata.com/docs/reference/timescaledb/hypertables/set_chunk_time_interval), no need to recreate the table). `create_hypertable()` still exists for converting an existing plain table; new tables like this one use `CREATE TABLE ... WITH (timescaledb.hypertable)`, the current documented default. Add an index for the most common query pattern, filtering by GPU over a time range:

`CREATE INDEX ON gpu_metrics (gpu_uuid, time DESC);`

### NVLink and XID health-event table

`CREATE TABLE gpu_health_events (
  time TIMESTAMPTZ NOT NULL,
  gpu_uuid TEXT NOT NULL,
  node TEXT NOT NULL,
  event_type TEXT NOT NULL,
  event_code INTEGER NOT NULL,
  nvlink_id TEXT,
  severity TEXT,
  raw_message TEXT
) WITH (
  timescaledb.hypertable,
  timescaledb.partition_column = 'time'
);`

`event_type` distinguishes `xid` from `sxid` events. Keeping them as discrete rows rather than a column on `gpu_metrics` means "how many XID 74 events did node X see in the last 30 days" is a filtered `COUNT`, not a sparse-column scan across mostly-empty values.

Whether this table needs hypertable partitioning depends on volume: keep the `WITH (timescaledb.hypertable, ...)` clause above for a large fleet with frequent health events, or drop it for a plain indexed table on a smaller cluster where events are rare. That's a judgment call based on fleet size, not a hard rule.

One extension point worth building in from the start: `dcgm-exporter` can enrich metrics with pod and job labels through Kubernetes' KubeletPodResources API (the `--kubernetes` flag adds pod, namespace, and container context from the kubelet pod-resources socket), which lets you JOIN GPU telemetry to workload metadata. If that's relevant to your deployment, an optional `pod` or `job_id` column on either table is a reasonable addition, not included above as a required field since not every GPU cluster runs on Kubernetes.

## SQL queries infrastructure teams actually need

Collection and dashboards are solved problems. What's missing from every DCGM-adjacent search result is working SQL against this data. Here's what infrastructure teams actually run once the data has somewhere to live.

### 90-day GPU power draw trend by rack

`SELECT
  time_bucket('1 day', time) AS day,
  rack_id,
  AVG(power_draw_w) AS avg_power_w,
  MAX(power_draw_w) AS peak_power_w
FROM gpu_metrics
WHERE time > NOW() - INTERVAL '90 days'
GROUP BY day, rack_id
ORDER BY day DESC, rack_id;`

This returns daily average and peak power draw per rack over 90 days: the query a capacity-planning team runs when deciding whether a rack's power allocation needs adjusting as GPUs are added. It pairs directly with the power-density figures on [<u>data center power monitoring</u>](https://www.tigerdata.com/learn/data-center-power-monitoring), where GPU racks commonly draw 30 to 100 kW against 5 to 10 kW for traditional compute.

### XID error rate by node

`SELECT
  node,
  event_code,
  COUNT(*) AS event_count
FROM gpu_health_events
WHERE event_type = 'xid'
  AND time > NOW() - INTERVAL '30 days'
GROUP BY node, event_code
ORDER BY event_count DESC;`

This surfaces which nodes and XID codes generate the most fault events over 30 days, the first query a team runs when triaging a hardware issue across a fleet rather than one box at a time.

### NVLink health degradation window

`SELECT
  time_bucket('1 hour', time) AS bucket,
  gpu_uuid,
  nvlink_id,
  COUNT(*) AS sxid_events
FROM gpu_health_events
WHERE event_type = 'sxid'
  AND time > NOW() - INTERVAL '14 days'
GROUP BY bucket, gpu_uuid, nvlink_id
HAVING COUNT(*) > 0
ORDER BY bucket DESC;`

This returns hourly SXID (NVSwitch) event counts per GPU and NVLink pair over two weeks, the pattern teams use to spot a rising trend in NVLink errors before a hardware failure instead of treating each event as isolated.

Worth knowing if you're trending historical SXID data: NVIDIA's DCGM 4.6.0 release notes list a fix for "false-positive NVLink health failures caused by valid zero or sentinel BER values" (bit error rate), with updated NVLink BER thresholds to align with hardware guidance. Data collected before that fix may include false positives worth flagging over a long window. See [<u>NVIDIA's DCGM release notes</u>](https://docs.nvidia.com/datacenter/dcgm/latest/release-notes/changelog.html) for the full changelog.

### GPU utilization vs. job scheduling correlation

`SELECT
  time_bucket('1 hour', time) AS bucket,
  node,
  AVG(utilization_pct) AS avg_utilization
FROM gpu_metrics
WHERE time > NOW() - INTERVAL '7 days'
GROUP BY bucket, node
ORDER BY bucket DESC, node;`

This is the GPU-side half of a join most teams need, against job-scheduler data (Slurm logs, Kubernetes job records) on `node` and `time`. The exact join depends on your scheduler's schema, so this is the GPU-side rollup rather than a fabricated join against data your setup may not have. Joined, it answers the question that matters: are GPUs sitting idle while jobs queue, or is scheduling keeping utilization where it should be?

## Liquid cooling and thermal telemetry

GPU-dense racks, the same 30 to 100 kW racks referenced above, increasingly rely on liquid cooling rather than air cooling alone. That coolant-loop telemetry only earns its place in this article because it needs to be correlated against the GPU power and thermal data already covered here, in the same query, not in two separate dashboards that a human has to mentally overlay.

The core metrics are coolant supply temperature, coolant return temperature, flow rate, and inlet/outlet delta-T (the temperature difference between coolant entering and leaving a rack or cooling distribution unit, or CDU). Delta-T is the number that tells you whether cooling capacity is keeping pace with GPU load: a shrinking delta-T at rising power draw usually means the coolant loop is starting to struggle to keep up.

That pattern fits the same hypertable design already used for `gpu_metrics`:

`CREATE TABLE coolant_metrics (
  time TIMESTAMPTZ NOT NULL,
  cdu_id TEXT NOT NULL,
  rack_id TEXT,
  supply_temp_c DOUBLE PRECISION,
  return_temp_c DOUBLE PRECISION,
  flow_rate_lpm DOUBLE PRECISION,
  delta_t_c DOUBLE PRECISION
) WITH (
  timescaledb.hypertable,
  timescaledb.partition_column = 'time'
);`

The query that makes this section worth including is a join across both tables, checking whether cooling capacity is keeping pace with GPU load as it climbs:

`SELECT
  time_bucket('1 hour', g.time) AS bucket,
  g.rack_id,
  AVG(g.power_draw_w) AS avg_power_w,
  AVG(c.delta_t_c) AS avg_delta_t_c
FROM gpu_metrics g
JOIN coolant_metrics c
  ON g.rack_id = c.rack_id
  AND time_bucket('1 hour', g.time) =
    time_bucket('1 hour', c.time)
WHERE g.time > NOW() - INTERVAL '30 days'
GROUP BY bucket, g.rack_id
ORDER BY bucket DESC;`

Watching average power draw and delta-T side by side, hour over hour, catches a rack where cooling capacity isn't scaling with GPU load before it becomes a thermal-throttling incident, a cross-domain query that takes one SQL statement instead of stitching together two separate monitoring tools by hand. For the broader power-density context this builds on, see [<u>GPU power draw is straining data center power monitoring</u>](https://www.tigerdata.com/learn/data-center-power-monitoring).

## Integrating this with your existing DCGM and Prometheus stack

The practical question is whether this means replacing any tool already in place. It doesn't. `dcgm-exporter` keeps exposing metrics in Prometheus format, Prometheus keeps scraping and alerting, and Grafana keeps rendering real-time dashboards from Prometheus, plus queries against Tiger Cloud over SQL for historical and trend panels Prometheus's retention window can't cover.

The connecting piece is a Prometheus `remote_write` configuration that sends the same scraped metrics to a long-term store, in addition to Prometheus's own local retention:

`remote_write:
  - url: "http://gpu-metrics-adapter:9201/write"
    queue_config:
      max_samples_per_send: 2000
      max_shards: 8`

Worth being direct here: Tiger Data doesn't ship a first-party, vendor-maintained `remote_write` adapter today (Promscale, which once filled that role, was deprecated in April 2023). In practice, that means a community-maintained or custom remote-storage adapter writing into a TimescaleDB hypertable, covered in more depth in [<u>what happens when Prometheus's local retention runs out</u>](https://www.tigerdata.com/learn/prometheus-long-term-storage), which is also the better read for comparing this against Thanos, Mimir, VictoriaMetrics, and the fuller set of [<u>Prometheus alternatives</u>](https://www.tigerdata.com/learn/prometheus-alternatives).

XID and SXID events that `dcgm-exporter` surfaces as Prometheus metrics, such as `DCGM_FI_DEV_XID_ERRORS`, can reach `gpu_health_events` through that same `remote_write` pipeline, or through a lightweight consumer that watches for the metric and inserts a row when it fires.

Once the data lands in Tiger Cloud, lifecycle management is two policy statements, not a separate system to run. Compression runs through hypercore, Tiger Data's hybrid row-columnar storage engine, which requires enabling the columnstore on the table first:

`ALTER TABLE gpu_metrics SET (
 timescaledb.enable_columnstore,
 timescaledb.compress_segmentby = 'gpu_uuid'
);
CALL add_columnstore_policy(
 'gpu_metrics',
 after => INTERVAL '7 days',
 if_not_exists => true
);`

And retention for the health-event table, since most teams want fault history further back than raw metrics:

`SELECT add_retention_policy(
  'gpu_health_events',
  drop_after => INTERVAL '2 years'
);`

For the full mechanics of both policies, including how compression and retention interact with chunk intervals, see the [<u>data lifecycle documentation</u>](https://www.tigerdata.com/docs/learn/data-management/data-lifecycle).

## Decision framework: do you need a dedicated database for GPU telemetry?

### Choose Tiger Data (TimescaleDB / Tiger Cloud) if:

- You need to query GPU power, utilization, or NVLink health trends over weeks or months, not just the last 15 days Prometheus retains by default
- You want to JOIN GPU telemetry against workload metadata, rack and power data, or liquid-cooling metrics in standard SQL
- You're already sending metrics through Prometheus remote_write and want a long-term backend without replacing your scrape or alerting layer
- You want a managed option (Tiger Cloud) rather than operating your own long-term storage backend

### Prometheus's default retention may be enough if:

- Your only requirement is real-time alerting and dashboards over a short lookback window, days rather than months
- You have no current need for historical trend analysis, capacity planning, or SQL-based correlation across metric types

### Consider a SaaS observability vendor (Datadog, Dynatrace) if:

- Full-stack observability with vendor-managed alerting and a built-in GPU integration matters more than owning the underlying data
- Per-GPU or per-host SaaS pricing is acceptable at your fleet size
- You don't need long-term SQL access to raw historical telemetry that you control directly

If your GPU fleet is already scaling past the point where a spreadsheet or a single Grafana panel answers "are we running out of power, cooling, or NVLink headroom," [<u>Postgres for scaling AI data center operations</u>](https://www.tigerdata.com/data-centers) covers how Tiger Data fits into the rest of an AI data center's telemetry stack, not just the GPU layer.