When PostgreSQL isn't the right fit, the signs don't announce themselves clearly. Postgres is the right database for roughly 90% of workloads, such as SaaS backends, CRUD applications, and transactional systems with mixed read/write access on shared rows. But there's a narrow 10% where those same strengths become overhead: high-frequency append-only ingestion, time-ordered data accumulating at sustained rates, analytical scans over hundreds of millions of rows. If that sounds like your system, this post is for you.

What you will learn

If you've added indexes, implemented partitioning, tuned autovacuum, and upgraded hardware only to watch performance degrade again on the same trajectory, the problem likely isn't your configuration. By the end of this post, you'll know whether your workload is in Postgres's 10%, how to confirm it with a single diagnostic query, and what the first concrete step toward the right architecture looks like.

Why it matters

An optimization problem and an architecture problem look identical in the early stages. Both show up as slow queries. Both respond to the same fixes: indexes, partitioning, autovacuum tuning, hardware upgrades. The divergence happens later, when the fixes stop holding and performance degrades on the same trajectory regardless of what you change.

This is what’s known as the optimization treadmill: a predictable sequence of phases that each buy three to six months of relief without changing the underlying trajectory. MVCC overhead, row-oriented storage, B-tree index maintenance, WAL volume. These aren't bugs. They're architectural tradeoffs that work well for 90% of workloads and work poorly for the 10%.

Knowing which problem you have determines whether you should keep tuning or make a different decision.

What Postgres was designed for

Postgres's architecture is built around concurrent access to shared rows. Multiple transactions read and write the same data at the same time, and MVCC handles the isolation. B-tree indexes find specific rows by key. Row-oriented storage assumes that when you retrieve a row, you want most of the columns in it.

For an e-commerce backend, a user authentication system, or a multi-tenant SaaS product, these are exactly the right tradeoffs. Transactions need isolation. Point lookups by user ID are the dominant query pattern. Write rates track user activity, which gives the database natural breathing room between peaks. The question isn't whether Postgres is good. It's whether the workload you're running matches the patterns its architecture was designed to serve.

The workload that breaks the match

Three characteristics, when they appear together, put a workload outside what Postgres handles well.

Append-only or append-heavy writes. Rows are written once and never, or almost never, updated. Sensor readings, financial transactions, log entries, event streams. Every row still pays the full MVCC cost: a 23-byte tuple header tracking transaction visibility, hint-bit dirtying on reads, and autovacuum running continuously to freeze tuples and update the visibility map. None of that overhead produces value on data that will never be touched again.

Sustained high write rates. Not burst traffic that settles. Continuous ingestion at thousands to hundreds of thousands of rows per second, around the clock. The table grows without pause, B-tree index maintenance adds overhead with every insert, and that cost compounds with row volume, so there is no quiet window for autovacuum to catch up.

Analytical query patterns. The queries are aggregations over time ranges: averages, counts, percentiles, GROUP BY time bucket. Row-oriented storage forces Postgres to read all columns of every matching row even when the query needs two. On a 30-column table, that's fifteen times the I/O a columnar layout would require.

Any one of these is manageable. All three together is the combination that Postgres handles well at one million rows and struggles with at one hundred million.

The optimization treadmill in practice

The pattern is predictable. Queries slow down as the table grows. You add indexes, and reads get faster. Write performance drops because index maintenance scales with row volume. You upgrade the instance. Performance stabilizes and costs go up. You implement partitioning. Recent-data queries get faster. Partition management becomes its own maintenance burden. You tune autovacuum settings. Things stabilize for a while. Data volume increases. The cycle repeats.

Each step is individually correct. The problem is that the sequence never ends. You're working around an architectural mismatch instead of running a workload the architecture was designed to serve.

The engineering cost accumulates in ways that are harder to see on a dashboard. The senior engineer spending a week on partition strategy is not shipping product features. The on-call rotation starts treating "database is slow again" as a recurring incident category. Quarterly planning includes a database scalability line item, every quarter.

How to know which 10% you're in

The answer is already in your table statistics. Not in EXPLAIN plans or monitoring dashboards, but in the counters tracking exactly how rows have been written, updated, and cleaned up over the table's lifetime. Run this against your highest-traffic tables:

SELECT
    relname AS table_name,
    N
_live_tup,
    n_dead_tup,
    n_tup_ins,
    n_tup_upd,
    ROUND(n_tup_upd::numeric / NULLIF(n_tup_ins, 0) * 100, 2) AS update_pct,
    last_autovacuum,
    last_autoanalyze
FROM pg_stat_user_tables
WHERE schemaname = 'public'
ORDER BY n_tup_ins DESC
LIMIT 10;

Here's an example of what a flagged table looks like next to a healthy one:

device_metrics is in the 10%: 847 million inserts, near-zero updates, and autovacuum fired three minutes ago on a table that has never had a meaningful UPDATE run against it. user_accounts is not: nearly half its rows are updated, and autovacuum runs only when it actually needs to.

Look for update_pct under 5% and last_autovacuum timestamps within the last few minutes on tables with near-zero deletes. That's the overhead the companion piece documents in detail: a cleanup process running non-stop on data you never modify, because the storage engine generates that work regardless of your intent.

Pair those numbers against the broader pattern. Your sustained write rate exceeds 10,000 rows per second. Your most common queries aggregate over time ranges, not point lookups by row identifier. You added partitioning specifically to control table size. You upgraded your instance specifically for query performance, not connection headroom.

Three or more of those conditions, and you're in the 10%. The optimization treadmill will keep running, but the trajectory won't change.

What the 10% actually needs

If you've confirmed you're in the 10%, migrating your highest-traffic table starts with a single function call:

SELECT create_hypertable('device_metrics', by_range('ts'));

This converts the table to a TimescaleDB hypertable, which does automatic time-based chunking without cron jobs or partition management scripts. From there, you can enable columnar storage on your chunks. This format reads only the columns a query requests, not full rows, and compresses historical data by 10 to 20x, bringing time-range aggregation performance in line with what the workload demands. The migration post walks through the full process, including zero-downtime options for production tables.

You keep the same SQL, the same connection strings, the same ecosystem tooling. This isn't a replacement for Postgres. It's Postgres with the storage primitives your specific workload actually needs.

Conclusion

Postgres is not the problem. Running the wrong workload class through an architecture designed for a different problem is. The distinction matters because one has a tuning fix and the other has a structural fix, and those two paths look identical for the first several months.

The most expensive version of this recognition happens after 18 months of optimization effort. The cheapest version happens now.

Run the diagnostic query above. If the numbers land where you expect, read the full architectural breakdown. If you're ready to test on your own data, start a free Tiger Data trial today.

Here's a query that runs on most time-series tables:

SELECT time_bucket('1 hour', ts) AS hour,
       avg(temperature),
       max(temperature)
FROM sensor_readings
WHERE ts > now() - interval '7 days'
GROUP BY hour
ORDER BY hour;

The query needs two columns: ts and temperature. The table has 15 columns. Postgres reads all 15 columns for every row that matches the WHERE clause.

That's not a bug. It's how row-oriented storage works. Each row is stored as a contiguous block of bytes on disk, called a heap tuple, and Postgres reads the entire tuple to access any column within it. For point lookups on individual records, this is efficient. You want the whole row, and it's stored together. For analytical scans over millions of rows where you need two columns out of fifteen, it's the dominant source of wasted I/O.

In Understanding Postgres Performance Limits for Analytics on Live Data, row-oriented storage was identified as one of four architectural constraints that compound under high-frequency ingestion. That whitepaper maps the pattern at a system level. This post goes deeper on the physical mechanism: exactly how pages work, how read amplification accumulates, and why the usual fixes don't reach it.

What You Will Learn

By the end of this post, you'll have a concrete diagnostic formula: the read amplification ratio. It tells you whether your storage layout is the dominant I/O bottleneck for analytical queries on any table you own. You'll also understand why indexes can't fix this class of problem and how a hybrid row-columnar storage layout changes the math. This post assumes working familiarity with Postgres page layout and B-tree indexes.

How Row Storage Actually Works in Postgres

Postgres stores data in 8KB pages. Each page holds multiple heap tuples. Each tuple contains every column value for that row, stored sequentially, preceded by a 23-byte header that carries transaction visibility metadata.

A table with 15 columns averaging 200 bytes per row fits roughly 35 to 40 rows per page, after accounting for headers, alignment padding, and page overhead.

When Postgres runs a sequential scan, it reads pages from disk in order. Each page load brings all the rows on that page into shared_buffers, with all 15 columns per row intact. The executor then evaluates the WHERE clause and pulls the needed columns from what was already loaded into memory.

The I/O cost is proportional to total table size, not to the size of the queried columns. A query that needs 12 bytes of data per row still reads 200 bytes from disk. The remaining 188 bytes load into the buffer cache and get discarded.

The Read Amplification Math

The number that makes this concrete is the read amplification ratio: total row width divided by the width of the columns the query actually needs.

For sensor_readings, the calculation is direct. The ts column is a timestamptz at 8 bytes. The temperature column is a float4 at 4 bytes. Together they represent 12 bytes of useful data per row. The full row is 200 bytes.

Read amplification ratio: 200 ÷ 12 = 16.7x

For every byte the query uses, Postgres reads 16.7 bytes from disk.

At 100 million rows covering seven days, that ratio stops being abstract. The query needs 100M x 12 bytes = 1.14 GB. Postgres reads 100M x 200 bytes = 18.6 GB. At a 500 MB/sec sequential read rate, the scan takes approximately 38 seconds. Reading only the needed columns would take roughly 2.3 seconds. That 16x gap is pure storage model overhead.

No index changes this number. No configuration setting changes it. Partitioning reduces scope. Fewer pages get scanned by cutting the time range, but within each partition the same per-row read cost applies. The storage layout determines the I/O, and the storage layout is fixed.

Try This Now: Measure Your Read Amplification

You can calculate the ratio for any table you own. Run these two queries to get the byte widths you need:

-- Full row weight
SELECT pg_column_size(t.*) AS row_bytes
FROM sensor_readings t
LIMIT 1;

-- Queried column weight
SELECT pg_column_size(ts) + pg_column_size(temperature) AS queried_bytes
FROM sensor_readings
LIMIT 1;

Divide row_bytes by queried_bytes. If the ratio is above 5x, the storage model is your largest I/O bottleneck for analytical queries on that table. No index or configuration change will close that gap.

Why Indexes Don’t Solve This

When a query is slow, the instinctive response is to add an index. For OLTP workloads, that instinct is correct. B-tree indexes excel at row selection: they find specific rows in O(log n) time, and for a lookup like SELECT * FROM users WHERE id = 123, the index locates the target row in microseconds.

For analytical queries that touch millions of rows, row selection is not the bottleneck. Finding the rows is fast. Reading the data from those rows is slow. An index scan on a million-row result set still reads the full heap tuple for every matching row to extract the needed columns.

The one exception is a covering index, which stores column values inside the index itself so Postgres can satisfy the query without touching the heap. But covering indexes for analytical queries become impractical at scale. When queries involve aggregations across high-frequency writes, wide covering indexes impose substantial write overhead, compounding exactly the index maintenance costs described in the optimization treadmill post.

B-tree indexes optimize for row selection (which rows to read). Analytical query performance is dominated by row width (how much data per row). These are different problems, and solving one leaves the other intact. For a broader look at what this means for your schema design, see Best Practices for PostgreSQL Data Analysis. 

How Columnar Storage Changes the Equation

In columnar storage, data is organized by column instead of by row. All values for ts live together in one stream on disk. All values for temperature live together in another. When the query needs those two columns, it reads two streams. The other 13 columns are never touched.

Same query, same 100 million rows: data read drops to 100M x 12 bytes = 1.14 GB. With typical 10 to 20x compression for time-series data, that compresses to approximately 60 to 120 MB. At 500 MB/sec, the same scan completes in roughly 0.12 to 0.24 seconds.

The compression benefit stacks on top of the I/O reduction. Because all values in a column share the same data type, compression algorithms work far more effectively. Sequential timestamps delta-encode to near-zero storage overhead. Floating-point sensor values compress with XOR-based techniques derived from Facebook's Gorilla algorithm. Row-oriented heap storage can't apply any of these because values from different columns are interleaved on every page. There's no contiguous column stream to compress.

Hypercore: Row and Columnar in One Table

The tradeoff with pure columnar storage is write performance. Every new row appends to each column file separately, which adds overhead for high-frequency ingestion. You get the read benefit but give up write throughput. Tiger Data's Hypercore solves this with a hybrid layout that keeps both.

Recent data stays in row-oriented storage for fast ingestion. Older data converts automatically to columnar format based on a compression policy you configure. The application writes standard SQL to one table. The storage format changes by age without any application-layer involvement.

-- Enable Hypercore on a hypertable with a 7-day row storage window
ALTER TABLE sensor_readings SET (
    timescaledb.compress,
    timescaledb.compress_segmentby = 'device_id',
    timescaledb.compress_orderby = 'ts DESC'
);

SELECT add_compression_policy('sensor_readings', INTERVAL '7 days');

New rows land in row format and ingest quickly. Data older than seven days converts to columnar chunks. To verify the behavior immediately without waiting for the policy schedule, compress a chunk manually:

SELECT compress_chunk(c) FROM show_chunks('sensor_readings') c LIMIT 1;

Then run EXPLAIN (ANALYZE, BUFFERS) on the aggregation query to see the difference in buffer reads (representative output on a 100M-row dataset):

-- Before: row storage sequential scan
Seq Scan on sensor_readings
  Buffers: shared read=2375000    -- 18.6 GB read from disk
  Execution Time: 38142.2 ms

-- After: Hypercore columnar scan
Custom Scan (ColumnarScan) on sensor_readings
  Buffers: shared read=10240      -- 80 MB read from disk
  Execution Time: 196.4 ms

The same SELECT statement works against both storage formats. The query planner handles the difference transparently.

Conclusion

Row storage reads every column to access any column. For analytical queries that scan millions of rows and need only a few, this is the largest source of I/O overhead. It doesn't yield to index tuning, partitioning, or hardware upgrades.

Calculate the read amplification ratio for your most common analytical queries using the pg_column_size queries above. If the ratio is above 5x, Hypercore is the direct fix. Start a free Tiger Data trial today to enable the hybrid storage model on your tables.

The database bill went up 40% last quarter, and everybody noticed. Finance noticed. You had a meeting about it. Somebody made a slide with a trend line that looked bad enough to earn its own agenda item.

The other database bill probably did not make the deck. It lives on the engineering calendar: weekly database reviews, monthly capacity checks, Slack threads about replica lag, onboarding sessions where a senior engineer explains why the partitioning scheme is the way it is, and the quarterly planning meeting where "database scalability" shows up again. I think of this as "calendar debt".

Calendar debt is what happens when an architectural mismatch stops showing up only as performance pain and starts changing how the team spends its time. The work is legitimate. Debugging write latency, tuning autovacuum, and reviewing partition migrations all require real engineering judgment.

The problem is that this work keeps coming back. At some point, the database is not just consuming CPU, storage, and cloud budget. It is consuming the attention of the people who are supposed to be building the product.

Run a 90-day calendar audit

Before arguing about whether the database needs a different architecture, look at the last 90 days of work. You do not need perfect time tracking. You need a useful approximation.

Pull the incident tickets with database root causes: slow queries, replica lag, connection pool exhaustion, WAL growth, autovacuum backlog, failed partition jobs, index bloat, storage pressure. Count the response time and the follow-up work. The incident is rarely the whole cost, because the real time usually shows up in the cleanup, the postmortem, the monitoring tweak, and the "we should make sure this never happens again" task that becomes someone's afternoon.

Search Slack for the terms that usually mean real work is happening: autovacuum, partition, vacuum, replica lag, WAL, bloat, statistics, index rebuild, capacity. Look at sprint work and meeting titles. Count query tuning, partition management, retention cleanup, schema migration support, capacity planning, runbook updates, and recurring reviews with "database" in the title.

Then ask the senior engineers directly: how much time last week went to database work that was not directly building product? Ask it as a systems question, not a performance question. Nobody needs a tiny productivity court hiding inside a database conversation. The point is whether the architecture is creating recurring work.

That recurrence is the signal. One slow query is an incident. A standing meeting about slow queries is architecture becoming process. One partition failure is a bug. A recurring partition review is lifecycle management leaking onto the calendar.

If the same names keep showing up in the tickets, Slack threads, sprint tasks, and meeting invites, the infrastructure bill is only part of what you are paying. You are carrying calendar debt too.

What the debt looks like

On a high-ingest Postgres workload, calendar debt usually starts small. One slow dashboard query becomes an index review. One retention problem becomes a partitioning discussion. One write latency spike becomes an autovacuum tuning session. One schema migration gets delayed because it touches too many partitions and nobody wants to find out in production that the migration plan was optimistic.

All of that is reasonable work. That is why it is easy to miss. It becomes a standing category of work.

Partition management means creating future partitions, checking for gaps, validating the automation, and handling the incident when a missing partition breaks ingestion at the least convenient possible hour. If you have been on the receiving end of that alert, you already know this is not an abstract problem.

Autovacuum tuning means watching pg_stat_activity, changing per-table settings as data volume changes, and figuring out whether a write latency spike is actually I/O contention from vacuum activity. Index maintenance means tracking bloat, rebuilding indexes, and debating whether a new read-path index is worth the extra write amplification on a table that already takes continuous inserts.

Replication management means watching lag, tuning max_wal_size, and dealing with the WAL accumulation alert when a replica falls behind during a write peak. Capacity planning means projecting data growth, modeling the next vertical scaling event, writing the infrastructure ticket, and explaining why the database needs more money after it got more money last quarter.

Individually, these tasks feel too small to count. Twenty minutes here. An hour there. A planning meeting. A follow-up review. A "quick" Slack thread that is somehow still active after lunch.

Aggregated across a week, they become a day. For senior engineers on teams deep into the Optimization Treadmill, 20-30% of their time can disappear this way. That number sounds high until you actually count the work instead of remembering it.

Memory rounds down. Calendars do not.

The calendar is the leading indicator

The cloud bill tells you what happened after the workload grew. The calendar tells you what is going to keep happening if nothing changes.

A one-time tuning sprint might be normal. A recurring tuning sprint is a strategy, even if nobody meant to make it one. The same goes for recurring capacity reviews, recurring partition checks, recurring autovacuum investigations, recurring schema migration reviews, and recurring "can we ship this dashboard without making the database sad?" conversations.

Each one is a small admission that the system requires ongoing human coordination to stay acceptable. The invoice can tell you the instance got more expensive. It cannot tell you the team has accepted a permanent tax on planning, onboarding, incident response, and senior engineering attention.

This is where the decision gets harder, because the people who understand the database path are the same people you need to change it. When their calendar is already full of maintenance, the work that would reduce the maintenance keeps moving out by a quarter.

That is the loop. The current architecture creates recurring work. The recurring work consumes the time needed to change the architecture. The data keeps growing while everyone waits for a clean window that never arrives. Not great.

Onboarding is where the debt becomes obvious

Existing teams normalize their own weirdness. The partition naming convention makes sense because everyone remembers the incident that created it. The autovacuum thresholds make sense because someone tuned them six months ago after a write peak. The runbook makes sense because the people reading it already know the missing context.

Then a new engineer joins, and suddenly the team has to explain all of it from scratch. Why the partitions are named that way. How the pg_partman automation works and what happens when it fails. Which autovacuum alerts are noisy until the day they are not. How schema migrations work across hundreds of partitions. Which replica is safe to query. Which incident from two years ago explains the one thing in the runbook that otherwise looks completely unhinged.

This is operational folklore, not product knowledge. It lives in runbooks, Slack history, and the heads of the two or three engineers who were there when the decisions were made.

So onboarding takes three or four weeks before someone can safely operate the database path. During that time, the new hire is less productive and the senior engineers are doing support work. Every hire pays that cost again.

The runbooks also have their own bill. Writing them takes time. Keeping them current takes time. When the partitioning scheme changes, someone has to know to update the docs, find the docs, and then actually update the docs. Documentation debt on operational procedures accumulates the same way technical debt does. It just looks more respectable because it has headings.

The debt compounds with data volume

The shape of the problem matters. At 100 million rows, the partitioning scheme may be manageable: maybe 50 partitions, a few runbooks, and one engineer who really understands the sharp edges. Database operations might take 10% of engineering time.

At 500 million rows, the partition count has grown. Autovacuum tuning is more complicated. A few incidents have added new alerts, new checklists, and new exceptions. The original expert has either become the bottleneck or has left enough knowledge behind to make everyone nervous. Now the work is closer to 20%.

At a billion rows and beyond, the scheme is embedded in how the team operates. Schema migrations are multi-day projects. Onboarding has a dedicated database section. Quarterly planning became monthly planning without anyone formally deciding that should happen. At that point, 30% is not a dramatic estimate. It is the floor on a bad quarter.

The growth is not linear because operational surface area does not grow one-to-one with data volume. Each threshold creates new work: more partitions, more monitoring, more migration caution, more review paths, more tribal knowledge.

Meanwhile, product work slows down in the most annoying possible way: gradually. Nobody flips a table. Features just take a little longer because database changes require more review. Releases get a little more careful because the partition scheme adds risk. The roadmap gets a quiet asterisk on every data-heavy feature: check with the database people first. That is how you know the architecture has become a product constraint.

What changes when the architecture matches the workload

This is the part where vendor content usually gets hand-wavy, so let's be specific. Database work does not become zero. You still operate a database. You still care about schema design, query behavior, retention, capacity, and reliability. The useful question is which calendar items should stop existing.

Take the recurring partition review. If that meeting exists because the team has to create future partitions, check for gaps, validate automation, and explain pg_partman failure modes to every new engineer, that is lifecycle work sitting in a meeting invite. Hypertables move time-based partitioning into the table abstraction. Chunks are created automatically as data arrives, so the partition creation job and the gap-monitoring ritual stop being monthly team activities.

Take the retention cleanup thread. If engineers are debating row deletes, manual partition drops, and cleanup windows every time data ages out, retention has become process. A retention policy turns that into database behavior. Expired chunks can be dropped by policy rather than by a quarterly cleanup project everyone swears will be simple this time.

Take the autovacuum investigation that keeps coming back. If the team is repeatedly tuning vacuum behavior around older high-volume data, the storage model is making historical data operationally expensive. Hypercore moves older chunks into a columnar format. Vacuum does not disappear from Postgres, but the recurring work created by high-ingest row churn on data that is no longer actively modified gets smaller.

Take the schema migration review. If every migration requires a special conversation because the table is really hundreds of manually managed partitions wearing a trench coat, the abstraction is leaking. With Hypertables, the application still sees a table. The migration discussion gets smaller because the lifecycle machinery is not scattered across a partition tree the team has to reason about by hand.

The calendar changes because whole categories of recurring work shrink or disappear. No partition creation review next month, no gap-monitoring script to babysit, fewer autovacuum conversations about old high-volume data, and less onboarding time spent explaining why the lifecycle machinery works the way it does.

Same Postgres ecosystem. Different operational shape. That is the actual value.

Bring your calendar to the architecture conversation

The cloud bill is visible. It shows up in the budget report with a trend line and a year-over-year comparison. The engineering calendar usually does not.

That is why teams undercount database cost. The work is distributed across incident tickets, sprint tasks, Slack threads, onboarding sessions, planning meetings, and "quick reviews" that are never quite quick.

If you want to know whether database optimization is still the right path, start with the calendar. Count the incidents. Count the meetings. Count the onboarding time. Count the senior engineer hours that went to keeping the database acceptable instead of moving the product forward.

Then ask the better question: is this optimization work buying us a better architecture, or is it paying interest on the current one? At 50 million rows, changing direction might take a week. At a billion rows, it can take months. Waiting does not make the work cheaper. It usually adds more runbooks.

If you want the mechanical side of why this happens, read Understanding Postgres Performance Limits for Analytics on Live Data. It explains the Optimization Treadmill and the architectural constraints behind it.

Bring your calendar when you read it. That is where the real bill is hiding.

Vector search in Postgres usually starts simply. You add an embedding column, run a nearest-neighbor query, and order by distance.

SELECT content
FROM documents
ORDER BY embedding <=> '[0.1, 0.2, ...]'
LIMIT 10;

For a while, that is enough.

That simplicity breaks down as the workload becomes real. The table grows, filters become part of the query path, and recall starts affecting user experience. The index still has to stay fast while new data keeps arriving.

That is when vector search stops being a query pattern and becomes an index design problem.

Most vector search advice starts with algorithms: HNSW, IVFFlat, DiskANN, recall, latency. That is useful, but incomplete once vector search lives inside Postgres. Postgres developers do not choose algorithms in the abstract. They choose indexes under constraints: memory, recall, write volume, filter selectivity, and the operational cost of adding another system.

The right index is not the best ANN algorithm in isolation. It is the index that fits the constraint your workload hits first: memory, recall, writes, or filters.

This article maps those constraints to real Postgres index choices: what each one costs, when it becomes the binding variable, and which index type it points to.

When exact search stops being enough

Exact k-nearest neighbor search compares the query vector against every vector in the table. It gives perfect recall because it does not approximate the result set. It also scales linearly with the number of rows.

That tradeoff is fine early on. Exact search is the right starting point when the dataset is small, the query rate is low or you are still validating whether embeddings work for your application. It also gives you a useful baseline because the results are not affected by index tuning.

The problem shows up when the table grows into millions or tens of millions of vectors, or when users expect low latency. At that point, scanning every vector for every query becomes too expensive.

Approximate nearest neighbor search, or ANN search, exists for this moment. ANN indexes organize vectors ahead of time so the database can search only the most promising candidates instead of scanning the full table. The index gives up a small, controlled amount of accuracy in exchange for much lower query latency.

That is the first tradeoff: ANN is not magic. You are deciding how much recall you can afford to exchange for speed, memory efficiency, and lower infrastructure cost.

The four constraints behind every vector index

The right vector index is usually decided by four constraints: whether the working set fits in memory, how much recall the application needs, how often the data changes and how selective the surrounding filters are.

Memory

Memory is fast and low-latency, but expensive. SSDs are cheaper and can still work well for many workloads. Object storage is cheaper still, but its higher latency makes it a poor fit for index designs that require many small random reads.

Vector indexes do not all touch storage the same way. Graph-based indexes follow connections between vectors through the index. That access pattern works very well when the graph is in memory and becomes more expensive when each hop risks a disk read. Partitioning-based indexes group vectors into regions and scan the most promising ones, which can be more memory efficient but usually requires more tuning.

In Postgres, the practical question is whether the index working set fits comfortably in shared_buffers and the operating system page cache. If it does, an in-memory graph index can perform very well. If it does not, the storage access pattern starts to dominate the design.

Recall

Recall measures how close approximate search gets to exact search. Higher recall usually costs more because the index has to inspect more candidates, traverse more of a graph or scan more partitions.

For some applications, slightly lower recall is acceptable if latency improves dramatically. For others, especially RAG systems where missing the right document leads to a bad answer, recall is part of product quality.

The honest way to set this tradeoff is to measure against your own data. Embedding model, dimensionality, filters, and query distribution all affect the result.

Writes

Some vector workloads are mostly read-heavy. You build the index, query it many times, and update it occasionally. Other workloads change constantly. New documents arrive, old ones are deleted, embeddings are regenerated.

A structure optimized for high-recall reads may have higher write or maintenance costs. A lighter-weight index may be easier to update but require more tuning to reach the same recall.

Filters

Real Postgres queries rarely search vectors alone. A query might ask for the nearest vectors, but only within a specific customer, time range, tenant or category.

Those predicates change the shape of the search problem. If a filter is highly selective, it may be cheaper to narrow the rows first and then search. If the filter is broad, it may be better to use the vector index first and apply the filter after. The right plan depends on the data distribution, the selectivity of the filter, and the index available to the planner.

That is one reason vector benchmarks can vary so much. Vector search without filters is not the same workload as vector search inside a real application query.

That is why there is no universal best vector index. There is only the index that best matches the shape of your workload.

The ANN algorithms behind Postgres index choices

The point of understanding ANN algorithms is not to memorize every paper. It is to understand why each index behaves differently as your workload changes. Most of the indexes discussed below fall into two broad patterns.

Graph-based indexes, such as HNSW and DiskANN-style designs, search by moving through connections between nearby vectors. Spatial partitioning indexes, such as IVFFlat and SPANN-style designs, divide the vector space into regions and search the most promising ones.

That distinction matters because graph-based indexes tend to optimize for high recall when the working set is hot, while partitioning-based indexes often trade more tuning for lower memory and maintenance overhead.

Each algorithm below is best understood as a response to a specific pressure: memory, write cost, disk access, or update churn.

HNSW: When the index fits in memory

Your dataset fits in memory and you need high recall at high query throughput. HNSW is built for this.

Hierarchical Navigable Small Worlds organizes vectors as a layered graph where each node connects to nearby vectors across multiple levels of granularity. A query enters at the top layer, moves toward the target neighborhood, then descends to finer layers until it converges on the best candidates.

The layered structure is what gives HNSW its speed-recall profile. The upper layers help the search move quickly across the vector space. The lower layers refine the candidate set around the target neighborhood. When the graph is in memory, that traversal can be fast and accurate.

The tradeoffs show up on the write side and at scale. Each node stores multiple edge pointers, so the index carries a higher memory footprint than simpler partitioning-based alternatives. Inserts and deletes require maintaining graph structure, which makes writes more expensive. And when the index grows beyond available memory, latency can climb.

In pgvector, HNSW is often the first ANN index Postgres developers try when query latency and recall matter most. For a practical look at how it performs, see Vector Database Basics: HNSW.

IVFFlat: When memory and writes matter more

Your write throughput matters, or your index cannot comfortably fit in memory. IVFFlat is worth considering.

IVF stands for inverted file. The basic idea is to partition the vector space into lists, then search only the most promising lists at query time. In pgvector, this index type is exposed as ivfflat.

Compared with HNSW, IVFFlat is usually lighter to build and maintain. Inserts are simpler because adding a vector means assigning it to a list rather than updating a graph of neighboring nodes.

The tradeoff is that recall is more sensitive to tuning. If you create 1,000 lists and set probes = 10, the query searches a small fraction of the partitioned index. Increasing probes gives the query more chances to find the true nearest neighbors, but it also pushes the query closer to a broader scan. IVFFlat tuning is about finding the lowest probes value that still meets your recall target.

That is the core IVFFlat tradeoff: lower memory and maintenance overhead, but more responsibility for tuning lists and probes against your workload.

DiskANN: When the index needs to live partly on disk

HNSW assumes the graph fits comfortably in memory. At tens of millions of high-dimensional vectors, that often stops being practical.

DiskANN, developed at Microsoft Research, was built for this case. It is a graph-based algorithm designed for datasets too large to fit entirely in RAM. At a high level, it keeps enough compressed information in memory to guide the search while storing more of the full index and vector data on SSD.

The lesson for Postgres developers is the storage pattern. A vector index that works well in RAM may behave very differently when the query path depends on repeated disk reads. Disk-aware indexes are designed around that constraint instead of treating it as an afterthought.

DiskANN still carries higher update costs than many partitioning-based approaches. But for read-heavy workloads on large datasets, it explains the shape of the problem that disk-aware Postgres vector indexing is trying to solve. See Understanding DiskANN for a deeper look.

SPFresh: The update problem at scale

Large vector indexes create another problem: updates.

Many ANN systems handle inserts and deletes by buffering changes, maintaining secondary structures, or periodically rebuilding parts of the index. Those approaches can work, but at very large scale they require either accepting stale index state or paying an increasingly expensive maintenance cost to keep the index current.

SPFresh, from Microsoft Research, is one such direction. It builds on partitioning-oriented ideas to reduce the need for global rebuilds, incrementally rebalancing partitions as vectors are inserted, deleted, or updated. Partition assignments are not fixed. They can drift and be corrected over time.

SPFresh is not implemented in Postgres today. But it is not purely academic either. The ideas behind it have already shaped how production vector systems outside Postgres are being designed. Turbopuffer is one example: an object-storage-first vector search service whose architecture is built around centroid-based indexing and minimizing storage round trips. Turbopuffer is not a Postgres system. But the tradeoffs it navigates (high-update workloads, disk-based search, incremental index maintenance without global rebuilds) are real problems the Postgres ecosystem will need to address as vector workloads become more dynamic.

This is worth tracking because the maintenance cost of a vector index is not static. It grows with update frequency and dataset size. For read-heavy workloads on stable datasets, this is not a near-term concern. For teams with high insert and delete rates (documents being added, embeddings regenerated, records retired), it is worth understanding now, before the index becomes the bottleneck.

The Postgres vector search stack

The algorithms above map to real problems Postgres developers run into. HNSW is useful for in-memory performance, IVFFlat for lighter-weight indexing and write-sensitive workloads, and DiskANN-style designs for larger datasets where memory becomes the constraint.

Here is how the Postgres ecosystem addresses those problems today.

pgvector

pgvector is the starting point. It adds a native vector column type to Postgres and supports both HNSW and IVFFlat indexes directly.

An HNSW index looks like this:

CREATE INDEX ON documents
USING hnsw (embedding vector_cosine_ops);

For IVFFlat, you define the number of lists and tune the number of probes:

CREATE INDEX ON documents
USING ivfflat (embedding vector_cosine_ops)
WITH (lists = 1000);
SET ivfflat.probes = 10;

The query planner can use these indexes for nearest-neighbor queries, and you can combine vector search with standard SQL filters, joins and CTEs in the same query. For many teams already running Postgres, this can remove the need to operate a separate vector database.

pgvector can start to show limits at larger scale, especially with high-dimensional embeddings at tens of millions of rows and indexes that no longer fit comfortably in memory. That is the problem pgvectorscale was built to address.

pgvectorscale

The DiskANN section above describes a specific problem: vector workloads that have grown too large to keep the working index in memory. For Postgres, pgvectorscale addresses that directly. It introduces a StreamingDiskANN index type that keeps a compressed representation in memory to guide search while storing the full index on disk.

On a Tiger Data benchmark of 50 million Cohere embeddings at 768 dimensions, Postgres with pgvector and pgvectorscale achieved 28x lower p95 latency and 16x higher query throughput compared to Pinecone's storage-optimized index at 99% recall. This was a vendor-run benchmark. Treat it as directionally useful, not universally predictive. Results will vary with embedding model, dimensionality, filters, recall target, and hardware.

The relevant point is that pgvectorscale stays inside the Postgres operational model. It remains composable with pgvector data types and standard SQL patterns. If your index has outgrown memory, you do not need a different system. You need a different index type.

pg_textsearch and ParadeDB

Vector similarity handles the semantic side of search well, but it is not the whole retrieval problem. Keyword-based retrieval still matters. It catches exact matches that embeddings miss, and for many queries, users know precisely what they are looking for.

This is where pg_textsearch and ParadeDB come in.

pg_textsearch, also from Tiger Data, brings BM25-based search into Postgres. BM25 accounts for term frequency saturation and document length normalization, which is why it is often a stronger ranking model for keyword search than simple term matching.

ParadeDB takes a related position as a Postgres distribution, bundling pg_search for BM25-based full-text search and pg_analytics for analytical query execution. If you want Elasticsearch-style search quality and are open to running a Postgres distribution rather than adding individual extensions, ParadeDB belongs on your evaluation list. When you are operating a small dataset, BM25 relevance ranking may not be a key requirement and pg_search will suffice. However, pg_textsearch is a better option when you need true BM25 relevance ranking with term saturation (how many times a term appears) or document length normalization to match the experience of Lucene (that powers Elasticsearch) or the algorithms that power Google.

The real payoff of having both vector search and BM25 inside Postgres is hybrid search: combining vector similarity and keyword scoring in a single query. For many RAG applications, this is often a stronger retrieval pattern than vector search alone because each approach covers the other's blind spots. Vector search captures semantic meaning. BM25 catches exact matches.

A simple hybrid search pattern in SQL

One common way to merge vector and keyword results is Reciprocal Rank Fusion, or RRF.

RRF avoids averaging scores across different scales. Instead, it combines rank positions. A result that appears near the top of either list gets a boost.

The exact syntax depends on which BM25 extension you use, but the query shape looks like this:

WITH keyword_results AS (
  SELECT
    id,
    content,
    paradedb.score(id) AS bm25_score,
    ROW_NUMBER() OVER (ORDER BY paradedb.score(id) DESC) AS keyword_rank
  FROM documents
  WHERE content @@@ 'vector search'
  LIMIT 60
),
vector_results AS (
  SELECT
    id,
    content,
    1 - (embedding <=> '[0.1, 0.2, ...]') AS similarity_score,
    ROW_NUMBER() OVER (ORDER BY embedding <=> '[0.1, 0.2, ...]') AS vector_rank
  FROM documents
  LIMIT 60
),
combined AS (
  SELECT
    COALESCE(k.id, v.id) AS id,
    COALESCE(k.content, v.content) AS content,
    COALESCE(1.0 / (60 + k.keyword_rank), 0) +
    COALESCE(1.0 / (60 + v.vector_rank), 0) AS rrf_score
  FROM keyword_results k
  FULL OUTER JOIN vector_results v ON k.id = v.id
)
SELECT id, content
FROM combined
ORDER BY rrf_score DESC
LIMIT 10;

This retrieves candidates from both systems, ranks them separately, and merges the ranked lists.

This is one of the strongest reasons to keep search in Postgres. Your embeddings, documents, metadata filters, joins, keyword search, and application data can live in one query model.

Learn more: how to build Hybrid Search in Postgres using pg_textsearch and pgvectorscale, and why hybrid search outperforms vector-only retrieval.

What this guide does not decide for you

No article can tell you the right vector index without your data.

Embedding model, dimensionality, filter selectivity, recall target, update rate, hardware, concurrency, and query distribution all change the answer. Even two datasets with the same number of rows can behave differently if their vectors cluster differently or their filters have different selectivity.

The point of this guide is not to replace benchmarking. It is to help you know what to benchmark first. Start with the simplest index that matches the shape of your workload. Measure it against exact search where possible. Tune recall and latency together. Then move to a more specialized index only when the workload gives you a reason.

Which Postgres vector index should you use?

The path usually looks like this: start with exact search while the dataset is small, move to HNSW when latency requires ANN, consider IVFFlat when memory or write cost matters more, evaluate disk-aware indexing when the working set outgrows memory, and add BM25 when retrieval quality needs more than semantic similarity alone.

Where things stand and where they are going

The practical rule is simple: benchmark the workload you actually run, not the cleanest version of vector search.

Start with exact search while the dataset is small. Move to HNSW when latency requires ANN. Consider IVFFlat when memory or write cost matters more. Evaluate StreamingDiskANN when the working set outgrows memory. Add BM25 when retrieval quality needs more than semantic similarity.

The one gap that remains is what SPFresh points toward: high-update workloads at scale without global index rebuilds. That capability is not yet in Postgres, but it is already showing up in production vector systems outside the Postgres ecosystem.

Whether it eventually appears as an extension, a fork or something nobody has named yet, the pattern is familiar: a hard problem gets real and someone in this community builds the thing.

Want to dig in further? Look at Tiger Data docs for pgvectorscale and pg_textsearch.

You just doubled the RAM on your database server to handle a climb in p95 latency. You expect the extra memory to absorb your growing dataset and bring those 45ms spikes back down to 8ms. Instead, the dashboard shows minimal improvement. Write latency remains high, and query response times stay variable.

The problem isn’t that you added too little RAM. It’s that you gave most of it to the wrong layer.

PostgreSQL and your operating system both cache data independently. When you over-allocate memory to Postgres, the OS loses the RAM it needs to do its own caching. Both layers end up storing identical data blocks simultaneously, a condition known as double buffering, while your system spends CPU cycles shuffling data between two pools instead of serving queries. At scale, this pattern becomes a vicious cycle: you add resources, the database absorbs them, performance recovers briefly, and then degrades again as the dataset grows.

This guide explains the double buffering mechanism, gives you the tuning rule that breaks the cycle, and shows you how to diagnose whether your current configuration is already caught in it. By the end, you will know how to calculate the correct shared_buffers value for your server, run a query to identify which tables are crowding out your buffer cache, and interpret the results to decide what to do next.

The Two Layers of Database Memory

To manage memory effectively, you need to understand the differences between the two independent caches that operate simultaneously on every Postgres server.

The internal buffer cache is defined by the shared_buffers configuration parameter. When a query needs a data block, Postgres checks here first. Ideally, it finds the data block so it can avoid a system call entirely. This cache is where your hot data lives.

The OS page cache lives in whatever RAM the operating system has not allocated elsewhere. When Postgres requests a block that is not in shared_buffers, it issues a file system call. If the OS has that block in its page cache, it serves the data immediately. If not, the OS falls through to a physical disk read.

It’s important to note that Postgres does not manage the OS page cache at all. Instead, the kernel manages the cache on its own, including allocating space and moving data into and out of the cache. Regardless, the OS page cache is a required part of Postgres, and not just a backup option for the internal buffer cache.

The Double Buffering Problem

Double buffering happens because neither cache knows what the other holds. Postgres does not inspect the OS page cache before storing a block in shared_buffers. The OS does not inspect shared_buffers before caching a file page. Both layers frequently hold copies of the same data at the same time.

This is wasteful at any size, but at scale it becomes actively harmful.

When shared_buffers is set too high (e.g. 80% of total RAM), the OS page cache is confined to the remaining 20%. Under a write-heavy workload, the OS needs that headroom to manage checkpoint writes, background writer activity, and WAL file flushes that grow proportionally with data volume. When the OS cache is too small, the kernel is forced to evict useful data pages to make room for incoming writes. Postgres then misses in both caches and falls through to disk, even if you have plenty of RAM.

This creates a vicious cycle. Adding more RAM to shared_buffers temporarily absorbs the working set, but as the dataset grows the same pressure returns. Each tuning cycle buys less time than the one before it.

Using The 25% Rule

The standard recommendation for Postgres is to set shared_buffers to 25% of total system RAM. By leaving 75% of memory to the OS, you give the kernel the headroom it needs to cache active data files, manage writes, and handle I/O bursts without evicting pages that Postgres will immediately need again.

To apply this, open postgresql.conf and update the parameter:

# For a server with 64GB RAM: 25% = 16GB
shared_buffers = '16GB'

This parameter requires a full server restart. A configuration reload is not sufficient.

Large Memory Servers

On systems with 512GB or more of RAM, 25% works out to 128GB. Beyond this point, the overhead of managing the internal buffer mapping can decrease performance rather than improve it. For very large memory systems, many teams cap shared_buffers at 128GB to 256GB and let the OS page cache handle the rest. Treat 128GB as your starting ceiling and benchmark from there.

Additional Settings

Changing shared_buffers in isolation can produce misleading results if these settings are not also configured correctly:

• effective_cache_size: Tells the query planner how much total cache (shared_buffers plus OS page cache combined) it can expect to use. Set this to 50-75% of total RAM. It does not allocate memory, but rather informs planning decisions and affects whether the planner chooses index scans over sequential scans.
• work_mem: Controls per-operation memory for sorts and hash joins. Too high, and concurrent queries can exhaust available RAM; too low, and sort operations spill to disk. A conservative starting point is total RAM divided by (max_connections x 2). On a 64GB server with 200 max_connections, that works out to roughly 163MB per operation, a reasonable baseline to start from and adjust under load.
• checkpoint_completion_target: Set to 0.9 to spread checkpoint writes across a longer window, reducing the I/O spikes that compete with the OS page cache during heavy write periods.

Diagnosing Your Current Configuration

Once you apply the 25% rule, the pg_buffercache extension shows you exactly which tables and indexes are occupying your buffer cache right now.

SELECT
  c.relname AS table_name,
  count(*) AS buffered_pages,
  pg_size_pretty(count(*) * 8192) AS buffer_size,
  round(100.0 * count(*) /
    (SELECT setting FROM pg_settings WHERE name = 'shared_buffers')::integer, 2
  ) AS percent_of_cache
FROM pg_buffercache b
INNER JOIN pg_class c ON b.relid = c.oid
INNER JOIN pg_namespace n ON n.oid = c.relnamespace
WHERE n.nspname NOT IN ('pg_catalog', 'information_schema', 'pg_toast')
GROUP BY c.relname
ORDER BY buffered_pages DESC
LIMIT 10;

Interpreting Your Results

A healthy result shows no single object above 15-20% of the cache.

If any single table or index exceeds 30% of the cache, treat it as a signal that one object is crowding out everything else. Do not respond by increasing shared_buffers. If the object is already larger than your current allocation, giving Postgres more memory will only delay the problem until the table grows again. Instead, ask yourself the following questions:

• Can the table be partitioned by time or key range so that queries touch only a recent, smaller slice of the data?
• Can the queries driving the cache pressure be rewritten to use more selective indexes rather than scanning large portions of the table?

Addressing Index Bloat

A separate but related problem is index bloat. When index entries dominate the output over table entries, your indexes have likely grown faster than your access patterns have changed. Use this query to identify indexes that are consuming cache but receiving no scans:

SELECT
  schemaname,
  relname as tablename,
  indexrelname as indexname,
  idx_scan AS scans,
  pg_size_pretty(pg_relation_size(indexrelid)) AS index_size
FROM pg_stat_user_indexes
WHERE idx_scan = 0
ORDER BY pg_relation_size(indexrelid) DESC;

Any index returned here is a candidate for removal. Dropping unused indexes directly reduces buffer pressure and frees cache space for objects that are actually serving queries.

Re-run the pg_buffercache query after any significant data volume increase or schema change to catch concentration drift before it affects query performance.

When Tuning Reaches Its Limit

The 25% rule and the diagnostics above will recover significant performance for most Postgres deployments. But when your working dataset is larger than the memory you can reasonably allocate to either cache layer, buffer management stops being the constraint. Instead, the data volume itself is the problem.

You can see this in pg_buffercache directly. If your largest table is 60GB and shared_buffers is 16GB, the table will never be fully cached regardless of how the allocation is tuned. percent_of_cache for that object will always approach 100% as the query workload pulls it in, leaving nothing for everything else:

At this point, adding more RAM extends the runway but does not change the slope. The next doubling of your dataset will return you to this same result. Columnar storage changes the equation by compressing data aggressively before it ever reaches the cache, reducing the volume that needs to be buffered in the first place.

You can test whether your workload would benefit from this approach by running the same pg_buffercache checks on a Tiger Data instance. Start a free trial today to optimize your database and internal buffer cache without affecting production.

Your indexes are perfect, your CPU is healthy, but your p99 latency is spiking. When you look at the logs, you see a standard IN clause fetching data for a few thousand users. It looks harmless, but for high-growth databases, this single query pattern becomes a significant performance burden. This guide explains how switching to ANY(ARRAY[]) reduces query planning time and prevents your query planner from stalling as you scale.

What You Will Learn

Large IN clauses act as a hidden tax on your PostgreSQL database. Passing a massive list of IDs from your application to a query creates a performance bottleneck that simple indexing cannot fix. In this article, you will learn:

• The hidden planning tax PostgreSQL pays to parse large lists of constants.
• How to implement ANY(ARRAY[...]) as a more efficient alternative for batch lookups.
• How to diagnose query planner bottlenecks using EXPLAIN ANALYZE.

Why It Matters

Object-Relational Mappers (ORMs) are the primary culprits behind oversized IN clauses. When an application needs to fetch metadata for a list of 5,000 users, the ORM typically generates a single query like the following example:

SELECT * FROM users WHERE id IN (1, 2, 3, ... 5000)

While this appears to be a standard batch operation, it forces the database to parse each item individually. Large IN lists can cause the query planner to spend more time analyzing the query than executing it. This overhead breaks plan caching, where latency creeps up even without traffic spikes. If you are managing billions of rows, the cost of moving these lists between your application and the database can consume your entire CPU budget.

​Parsing vs. Executing

To a developer, an IN clause is just a list. To the PostgreSQL parser, it is a complex expression tree.

When you send a query with 5,000 IDs in an IN clause, the parser must:

1. Tokenize every single constant in the string.
2. Assign types to every element (e.g. ensuring they are all integers).
3. Build a tree node for each item.

This creates a large internal structure that the query planner needs to traverse. In contrast, using an array parameter wraps all 5,000 IDs into a single object. The parser sees one parameter, assigns it one type (int[]), and moves straight to the planning phase. This reduces the memory usage of the query string and prevents the expression tree from becoming unmanageable.

Query Planner Diagnostics

To determine if your IN clauses are hurting performance, you need to look beyond total latency and instead look at a breakdown of your query’s lifecycle. Luckily, most of these diagnostics are available via EXPLAIN ANALYZE.

Identifying the Planning Tax

When running EXPLAIN ANALYZE, pay close attention to the Planning Time. If it exceeds 50% of the total time, the database is struggling to parse your constant list. For example, a query that takes 40 ms to plan but only 2 ms to execute is a prime candidate for refactoring.

Analyzing Scan Types

Check how the engine navigates your data structures within the EXPLAIN output. A standard Index Scan is highly efficient for small IN lists because the overhead of looking up a few individual keys is relatively low. However, as the list grows, the planner typically switches to a Bitmap Index Scan, which is commonly used with the ANY(ARRAY[]) syntax. In this scenario, the database constructs a map of all matching rows in memory before initiating the fetch. This process is far more efficient for large batches than jumping back and forth on an index for 5,000 separate values.

​Benchmarking the Planning Tax

To see the performance gains in your own environment, use EXPLAIN (ANALYZE, BUFFERS) to monitor the planning and execution metrics of your queries. These examples demonstrate how to measure the planning tax of a standard IN clause, and the gains you will see with the more efficient ANY(ARRAY[]) pattern, for a typical device_metrics table.

Standard IN Clause

A standard IN clause with 5,000 IDs forces the database to tokenize each element individually, resulting in high Planning Time. The problem is that the parser needs to parse each constant and build a massive expression tree before execution starts.

EXPLAIN (ANALYZE, BUFFERS)
SELECT reading_value
FROM device_metrics
WHERE device_id IN (101, 102, 103, ... 5000);

ANY(ARRAY[])

When you use ANY(ARRAY[]) syntax, you tell PostgreSQL to treat the thousands of IDs as a single object rather than thousands of individual constants. This drastically reduces the query's structural complexity.

EXPLAIN (ANALYZE, BUFFERS)
SELECT reading_value
FROM device_metrics
WHERE device_id = ANY(ARRAY[101, 102, 103, ... 5000]::int[]);

Results

In high-volume environments, moving from the standard IN clause to an array parameter often yields the following improvements:

• Planning Time: Can drop from 40+ ms to less than 2 ms.
• Memory Usage: Significant reduction in the memory required to store the query string and the resulting expression tree.
• Scan Strategy: The planner is more likely to use a Bitmap Index Scan, which fetches data in larger, more efficient batches than individual index lookups.

Next Steps

Switching from a standard IN clause to array parameters can be one of the most impactful changes you can make for database performance. Here are some steps you can take to implement this in your own database.

1. Enable pg_stat_statements: Ensure this extension is enabled in your shared_preload_libraries to track historical query performance at scale.
2. Find top performance offenders: Identify queries with large parameter counts using pg_stat_statements, using the following query.

SELECT query, calls, mean_exec_time
FROM pg_stat_statements
WHERE query ILIKE '%IN (%'
ORDER BY mean_exec_time DESC LIMIT 5;

3. Audit ORMs: Check your ORM configuration to see if it supports "Array Mode" to automate the use of the ANY(ARRAY[]) pattern.
4. Benchmark: EXPLAIN (ANALYZE, BUFFERS) on your largest batch queries. If planning time exceeds 10 ms or 50% of total latency, apply the refactor immediately.

ANY(ARRAY[]) and all these tools come with Tiger Cloud by default. Start a free Tiger Cloud trial today and see these performance benefits for yourself.

"We can fix the performance issue with better indexes, smarter partitioning, and some vacuum tuning. It's cheaper than switching."

You've heard this sentence. You may have said this sentence.

The optimization wasn't cheap. It just felt like it was.

"Cheaper than what" is the question nobody asks. The optimization doesn't show up on an invoice. It costs engineering time. And engineering time has a rate: the fully-loaded cost of the senior engineers doing the work, plus whatever those engineers aren't building while they're doing it. Most teams have never actually added up their database optimization spend. When they do, the number is larger than expected. And it comes back every quarter.

This problem is specific to a particular class of workload: high-frequency, append-heavy data. Telemetry, metrics, events, anything where timestamps are how you think about your data and the table only ever gets bigger. If that describes your system, keep reading. If you're running a CRUD app with predictable write volume, this isn't your problem.

Why optimization doesn't fix this

Here's what most teams figure out a year or two in: optimization isn't the wrong thing to try. It's just solving the wrong problem.

Tuning vanilla Postgres for a high-frequency append workload is a bit like upgrading the engine on a pickup truck because you want to haul more freight. You can make the truck faster and it feels productive. But at some point, you're limited by what the vehicle fundamentally is. The problem isn't the mechanic. It's the vehicle.

When your workload is structurally mismatched to your database architecture, the optimization treadmill is inevitable. Every index you add, every partition scheme you design, every autovacuum you tune: it's solving for a data volume you'll outgrow in months. The gap between "current optimization" and "needed optimization" widens every quarter. Not because you're falling behind. Because the data compounds faster than the fixes do.

A realistic year

Here's what that looks like. A year of Postgres optimization for a high-volume append workload.

Q1. Queries are slowing down. A senior engineer spends two weeks analyzing query plans, adding targeted indexes, and rewriting three critical queries. Performance improves. Write throughput drops roughly 15% because of new index maintenance overhead. (These numbers are illustrative. Your Q1 will have its own version of this tradeoff.)

Q2. Table size is causing partition-related issues. The team implements time-based partitioning. Two engineers spend three weeks on it: designing the partition scheme, migrating existing data, updating application queries that assumed a single table, and fixing the CI/CD pipeline that didn't account for partition management.

Q3. Autovacuum is competing with production writes during peak hours. One engineer spends a week tuning autovacuum parameters, adjusting cost delays, and setting up monitoring for vacuum lag. A follow-up incident two weeks later, when a vacuum job blocks a schema migration, costs another three days.

Q4. Storage costs are climbing. The team evaluates compression options, considers archiving old data to cold storage, and ultimately decides to upgrade the instance size to buy headroom for Q1 of next year. The upgrade takes a day. The evaluation and planning took two weeks.

Total: 12 to 16 engineer-weeks across the year. At fully-loaded senior engineer cost (call it $150K to $200K/year), that's $35K to $60K in direct labor. You bought time, not a solution. And the bill comes back next year.

The opportunity cost (the real number)

The $35K to $60K understates it.

12 to 16 engineer-weeks is a feature. It's a product launch. For a team of 10, that's 3 to 4% of total engineering output spent keeping the database at "acceptable." Not advancing it. Just treading water against a growing dataset.

Ask your engineering manager: if you reclaimed those 12 to 16 weeks, what would you build? That's the true cost of optimization. Not the hours. The roadmap you didn't ship.

And it compounds. Year two has all the same optimization needs plus new ones as data grows, but now you're also maintaining the partitioning scheme from Q2 and the vacuum configuration from Q3. The baseline maintenance burden grows even as new problems arrive.

Flogistix, who runs high-frequency oil and gas telemetry, reported 66% monthly cost savings after moving to Tiger Cloud, and their engineering team said the freed time directly increased roadmap velocity. That's what the other side of this decision looks like.

The hidden costs nobody tracks

These don't show up in sprint planning.

Incident response. Database performance incidents pull engineers off planned work. A slow query that triggers alerts at 2am costs the on-call engineer a night of sleep and a mostly useless next day. These incidents increase in frequency as the gap between "current optimization" and "needed optimization" widens. And the gap always widens.

Knowledge concentration. Database optimization work accumulates in one or two senior engineers who understand the schema, the query patterns, and enough Postgres internals to make changes safely. This is your single point of failure. When that engineer is on vacation or leaves, optimization work stalls or gets done slowly by someone learning as they go. Trust me, I've seen this play out in ways that aren't fun for anyone involved.

Context switching. Engineers don't work on database optimization in clean, uninterrupted blocks. They get pulled in for an afternoon here, a day there, to diagnose a regression or review a partition change. Context switching is expensive because it disrupts both the database work and whatever they were doing before. You're not just paying for the time spent on the database. You're paying for the interrupt tax on everything else.

All three are part of the platform tax: the invisible engineering cost of maintaining infrastructure that doesn't quite fit the workload. It doesn't show up on an invoice either.

Calculate your own number

Track for one month. Count hours spent on: query optimization and explain plan analysis; partition management and creation; autovacuum tuning, monitoring, and incident response; database-related incident response (slow query alerts, replication lag, connection pool exhaustion); and meetings discussing performance, capacity planning, or migration timing.

Multiply the monthly total by 12. Multiply that by the fully-loaded hourly rate of the engineers involved. That's your annual optimization cost.

Compare it against the one-time cost of migrating to a system designed for the workload (typically 2 to 8 engineer-weeks depending on data volume), plus ongoing maintenance that scales with workload complexity rather than with data growth.

For most teams, the breakeven is within the first year. Often within the first quarter. Do the math before assuming migration is the expensive option.

What the alternative looks like

After migrating to TimescaleDB (the open-source Postgres extension that powers Tiger Cloud), the engineering time picture looks different.

Migration cost: one-time, typically 1 to 4 weeks for a single engineer depending on data volume and schema complexity. Most of that time is data backfill, not application changes. TimescaleDB is still Postgres. Your SQL, your tooling, your team's existing knowledge stays intact.

Ongoing costs: not zero, but different in kind. The categories of work that consumed engineering time on vanilla Postgres shift significantly. Automatic partitioning via Hypertables removes partition management as a recurring quarterly project. The database handles it. Compression policies run automatically in the background. Autovacuum pressure on historical data drops because Hypercore converts older chunks to columnar format: instead of accumulating MVCC dead tuples on row-level records, that data is stored as compressed column arrays that don't generate the same vacuum workload. You still tune a database. You just stop tuning the same problems every quarter.

What was being spent on keeping vanilla Postgres at "acceptable" is now available for product work. Not because the database is magic. Because the architecture fits the workload.

The decision you keep deferring

The true cost of database optimization is not the cloud bill. It's the engineering time: senior engineers spending weeks per quarter on maintenance that keeps the system at "acceptable" rather than moving it forward.

If the annual optimization cost exceeds the one-time migration cost (and it usually does, often within the first year), the economic case writes itself. The harder question is whether the team can keep deferring the decision, knowing that each quarter of optimization increases the total spend without changing the trajectory.

Run the numbers. Then decide.

If you've done the math and want to understand what migration looks like at your data scale, The Best Time to Migrate Was at 10M Rows. The Second Best Time Is Now. is a good next read. And when you're ready to move, the migration guide covers the mechanics.

When your Shared Buffer Hits cross 100,000 for a simple dashboard refresh, you have hit the wall of relational scaling. A simple query that used to return in 100ms now takes 1.5 seconds. You’ve added indexes and tuned the buffer pool, but the performance gains are fleeting. Your largest tables crossed 500M rows, and the joins that once felt elegant are now a weight dragging down your entire system.

What You Will Learn

High-frequency data and deep relational hierarchies eventually collide. By following this guide, you will be able to:

• Identify the metadata tax in your current query plans using buffer metrics.
• Implement a flattened schema to bypass the Join Explosion.
• Automate data synchronization between relational and flattened tables.
• Compare the performance gap between relational and denormalized architectures.

Why It Matters

Relational schemas work during small-scale pilots but struggle when dashboards must stitch ten tables together for every refresh, hundreds of times per second, against billions of rows. Complex joins consume excessive CPU cycles, leading to the query speed degradation typical of the optimization treadmill.

Each join forces the database to navigate multiple B-tree indexes and load disparate pages into memory. By flattening data, you reduce the database's computational burden and enable much higher query concurrency.

The Join Explosion Problem

A normalized schema fragments telemetry into separate tables for readings, hardware IDs, and site data. While efficient for storage, this creates a “join explosion” at scale. On tables with a billion rows, the query planner has to navigate multiple B-tree indexes and perform nested loops, or hash joins, just to assemble a single dashboard view. This forces the database to jump between disparate disk locations, incurring a heavy computational tax every time a user refreshes their screen.

The solution is to move the join cost from read-time to write-time. Instead of stitching tables together during every query, you flatten the data by pre-joining the metadata to the raw reading during ingestion.

Moving From Relational to Flattened

Moving from relational to flattened means transitioning from a passive storage model to an active processing model. Instead of a lean ingestion phase followed by expensive reads, you perform the join logic exactly once when the data enters the system. Storing the result in a single, wide table allows the database to perform a single index scan rather than a multi-way join, reducing I/O load and restoring dashboard responsiveness.

The following sections illustrate how to move to a flattened data model, with example SQL commands for a smart building management system. In this scenario, we will move the computational cost of joining sensor data (e.g. temperature, air quality) with building and regional metadata from query-time to write-time.

Step 1: Define the Flattened Structure

Start by creating a table that includes the metadata as native columns. This removes the need for downstream joins.

This example query creates the flattened table and a composite index so the database can locate time-series readings and their metadata without expensive lookups across separate tables. This structure allows the database to instantly locate specific data, like North region telemetry, without performing costly table scans.

CREATE TABLE readings_flattened (
   ts TIMESTAMPTZ NOT NULL,
   sensor_name TEXT,
   building_name TEXT,
   region TEXT,
   value DOUBLE PRECISION
);
CREATE INDEX idx_flattened_ts_region ON readings_flattened (ts DESC, region);

Step 2: Backfill Existing Data in Batches

Moving a billion rows at once will lock your database. Use a batch approach to migrate data from your relational tables.

This example query creates a path to migrate historical records from fragmented tables into the new wide format one window at a time, starting with the past thirty days. By pre-joining the data during this migration, you eliminate the need for the database ever to stitch these specific records together again.

INSERT INTO readings_flattened (ts, sensor_name, building_name, region, value)
SELECT
   r.ts,
   s.sensor_name,
   l.building_name,
   l.region,
   r.value
FROM readings r
JOIN sensors s ON r.sensor_id = s.id
JOIN locations l ON s.location_id = l.id
WHERE r.ts > now() - interval '30 days';

Step 3: Automate Maintenance via Triggers

To keep the flattened table up to date without changing your application code, use a database trigger. This ensures that every new reading is pre-joined as it enters the system.

This example query creates an automated trigger that flattens every incoming sensor reading in real time. This ensures that, as new sensor data arrives, it is immediately linked to its building and regional context before being stored.

CREATE OR REPLACE FUNCTION flatten_reading_trigger()
RETURNS TRIGGER AS $$
BEGIN
   INSERT INTO readings_flattened (ts, sensor_name, building_name, region, value)
   SELECT
       NEW.ts, s.sensor_name, l.building_name, l.region, NEW.value
   FROM sensors s
   JOIN locations l ON s.location_id = l.id
   WHERE s.id = NEW.sensor_id;
   RETURN NEW;
END;
​
CREATE TRIGGER trg_flatten_reading
AFTER INSERT ON readings
FOR EACH ROW EXECUTE FUNCTION flatten_reading_trigger();

​Measuring the Performance Gap

To see the mechanical benefits of your new flattened table, you need to move beyond measuring execution time, which can fluctuate based on concurrent load. Instead, use the BUFFERS metric in your query plan to observe the physical I/O the database performs. This provides a stable, repeatable measure of the work required to retrieve your data.

The Relational Approach (Join at Read)

To identify the join tax in your system, establish a performance baseline using your existing relational schema. This example query uses the EXPLAIN (ANALYZE, BUFFERS) command to measure the physical memory and I/O work required by a standard multi-way join.

In our example scenario, this query tracks how many data blocks your database engine must touch to assemble the North region dashboard.

EXPLAIN (ANALYZE, BUFFERS)
SELECT r.ts, s.sensor_name, l.building_name, r.value
FROM readings r
JOIN sensors s ON r.sensor_id = s.id
JOIN locations l ON s.location_id = l.id
WHERE r.ts > now() - interval '1 hour' AND l.region = 'North';

Focus on the shared hit count. Each hit represents one 8KB block of data the database had to find and load. If you see tens of thousands of hits for one hour of data, your I/O is the bottleneck.

The Flattened Approach (Pre-Joined)

Validate the architectural shift by running a comparative diagnostic against the new flattened structure. This example query proves the efficiency of data co-location by measuring shared buffer hits. Unlike the relational model, which hunts across multiple indexes, this approach allows the engine to find all necessary data within a single storage layer, significantly reducing the number of blocks touched.

In our example scenario, this query will show the efficiency of your database engine in assembling the North region dashboard with a flattened structure.

EXPLAIN (ANALYZE, BUFFERS)
SELECT ts, sensor_name, building_name, value
FROM readings_flattened
WHERE ts > now() - interval '1 hour' AND region = 'North';

​Look for a 70–90% drop in shared hits — the foundation of any real-time analytics workload. This reduction in blocks touched is what allows your dashboard to scale to hundreds of concurrent users without CPU saturation. 

Trade-offs: Consistency vs. Speed

Flattening introduces redundancy. If a building is renamed, the flattened table will still contain the old name for historical rows. However, for telemetry and IIoT, metadata is usually static. The massive gain in query concurrency and dashboard speed almost always outweighs the cost of a rare historical update.

​Next Step

Identify the slowest query in your dashboard. Test a flattened version of that dataset using a temporary table to see the I/O savings immediately:

CREATE TEMPORARY TABLE test_flattened AS
SELECT r.ts, s.sensor_name, l.building_name, l.region, r.value
FROM readings r
JOIN sensors s ON r.sensor_id = s.id
JOIN locations l ON s.location_id = l.id
LIMIT 1000000;
EXPLAIN (ANALYZE, BUFFERS) SELECT * FROM test_flattened WHERE region = 'North';

Compare the shared hit count of this test against your production query. If the savings are significant, it is time to move toward a denormalized architecture. For workloads exceeding billions of rows, see how Tiger Data does this flattening for you with hybrid row-columnar storage. Start a Tiger Cloud free trial today to see these savings for yourself.

Every database has bad days. A slow query after a schema change. A spike in replication lag during a traffic surge. An autovacuum job that runs long enough to make you nervous.

Those are tuning problems. They have tuning solutions. A senior engineer can fix them in a day, sometimes in an hour, and the fix holds. You move on.

But there's a different category of symptom. The kind that comes back. The kind where each solution introduces the need for the next solution, where performance degrades not because of a specific bad query but because the workload has grown into territory the database wasn't designed for.

Here's the thing that makes this category different: growth changes the nature of the problem, not just the severity. When your workload is fundamentally append-heavy and analytically queried, you're running it on a storage engine built for point lookups and updates. That mismatch doesn't show up at small scale. It compounds. And eventually no amount of configuration adjusts for it.

I've watched teams burn two quarters optimizing their way around this before realizing the optimization itself was the trap. The five signs below are how you recognize it before that happens.

If you're seeing three or more, the problem isn't your configuration.

Sign 1: Your optimization work is cyclical, not cumulative

Tuning that sticks is cumulative. You add an index, queries get faster, they stay faster. You adjust work_mem, hash joins improve, the improvement is permanent. Done.

The warning sign: you're doing the same work again six months later. You partition the table, and after a while you need to re-partition because the data distribution shifted. You optimize a query, and three months later the same query is slow again because the table grew 40%. You upgrade the instance, and the headroom is gone in a quarter.

When optimization is cyclical, the underlying cause is growth, not misconfiguration. Growth doesn't respond to configuration. It responds to architecture.

How to measure: look at your sprint backlog. How many database-related tasks do you have this quarter versus last quarter versus the one before that? If the number is flat or growing, your optimization is running in place.

Sign 2: Autovacuum is running constantly on tables with low update rates

Autovacuum on a heavily updated table makes sense. Dead tuples need cleaning, that's the job.

The warning sign: autovacuum running persistently on tables where fewer than 5% of rows are ever updated. This usually means one of two things: transaction ID wraparound prevention driving vacuum on high-insert tables, or hint bit maintenance generating constant background I/O.

Both are MVCC overhead on append-only data. The visibility tracking machinery runs at full cost on rows that will never be modified. Tuning autovacuum parameters (scale_factor, cost_delay, max_workers) adjusts how the work is distributed, not whether it has to happen. The work still has to happen.

How to measure: check pg_stat_user_tables for your largest tables. Compare n_dead_tup to autovacuum_count. If autovacuum runs frequently but dead tuple counts are consistently low, the vacuum is maintenance overhead, not cleanup. You're paying the full MVCC tax on data that never gets modified.

Sign 3: Query performance degrades linearly with data volume

Postgres query performance should be sublinear with data volume when indexes are working correctly. A B-tree index lookup is O(log n). Adding 10x more data should add a constant factor to indexed lookups, not 10x more time.

The warning sign: queries that get proportionally slower as the table grows. This happens when the dominant pattern is sequential scans or index scans over wide ranges rather than point lookups. Aggregations over time ranges, GROUP BY on large result sets, analytical scans touching millions of rows. These scale with data volume, not with the index.

Partitioning helps by limiting scans to relevant partitions. It's also a manual process that requires ongoing management, and partition-level sequential scans still scale linearly within each partition.

How to measure: run your five slowest queries against progressively larger time ranges. If execution time scales proportionally with the range, the queries are scan-bound. More data will always mean slower queries, and no index is going to change that trajectory.

Sign 4: Index strategy has become a tradeoff negotiation

In a well-matched architecture, indexes are straightforward. Add them where you query, the read performance benefit outweighs the write cost, everyone's happy.

The warning sign: adding indexes to improve read performance measurably degrades write throughput, and the team is now making deliberate tradeoffs. "We can't add that index, it would slow inserts below our SLA." That conversation is the sign.

This is specific to high-throughput insert workloads. A SaaS app doing 1K inserts/sec can add ten indexes without noticing. A telemetry pipeline doing 100K inserts/sec feels every additional index in write amplification. The math is just different.

When index strategy becomes a negotiation between read and write performance, the workload has outgrown what B-tree indexing on row-oriented storage can serve without tradeoffs. You can't index your way out.

How to measure: benchmark write throughput with your current index set, then with one additional index. If the drop is more than 10-15%, you're in the tradeoff zone. Whoops.

Sign 5: Storage costs are growing faster than data value

Postgres stores data at the row level, uncompressed (unless you're using TOAST for large values). For time-series and event data, raw storage cost is proportional to row count with no automatic lifecycle management built in.

The warning sign: storage costs growing linearly with data volume, and the team having conversations about retention not because old data is worthless, but because storing it is expensive. You archive to cold storage or drop old partitions to manage costs, even though you'd rather keep the data online.

When storage cost forces data lifecycle decisions, the storage model isn't efficient for the data type. Columnar compression (10-20x for typical time-series data) and automatic data tiering change the economics so that keeping years of history queryable is practical rather than a monthly budget conversation.

How to measure: calculate your cost per million rows stored. Compare that number to the analytical value of the data. If you're deleting or archiving data you'd rather be querying, storage cost is constraining your product. That's the sign.

The diagnostic

Count how many of these apply to your system.

0-1: Tuning is the right move. Your workload fits the architecture, specific optimizations will address the symptoms. The usual playbook works.

2-3: You're approaching the boundary. Current optimizations will buy months, not years. Start evaluating architectural options now, while you still have time to choose rather than react.

4-5: The workload has outgrown the architecture. Further optimization has diminishing returns. Migration to a system designed for this workload will be less expensive than continued optimization within 6-12 months. Not a fun conclusion, but better to reach it now than after another year of sprint-backlog churn.

What this actually means

Tuning problems have solutions that stick. Architectural mismatches have solutions that buy time. The goal of this whole exercise is to figure out which one you're dealing with before you've spent another quarter on the latter.

If the diagnostic puts you in the 4-5 range, the answer isn't to abandon Postgres. The answer is to stop running a workload it wasn't designed for on vanilla storage primitives. What you actually need is automatic time-based partitioning, hybrid row/columnar storage for analytical scans, and lifecycle management that doesn't require you to babysit it. You need Postgres extended for this workload, not replaced.

That's what TimescaleDB does. Same database, same SQL, same toolchain your team already knows. Different storage engine underneath.

If you want to go deeper on why vanilla Postgres hits this wall and what the underlying mechanics look like, Understanding Postgres Performance Limits for Analytics on Live Data covers the architecture side in detail. And if some of these signs felt familiar but you're not sure you're fully in the architectural category yet, Six Signs That Postgres Tuning Won't Fix Your Performance Problems has a different angle on the same diagnostic.

You open pg_stat_activity during a write peak and there it is. Autovacuum. Again.

You've tuned this three times. You wrote a runbook for it. At this point it has its own section in the quarterly database review.

The table it's working on hasn't seen a single UPDATE in six months. Every write is an INSERT. Old data gets dropped by partition, not deleted row by row. By every reasonable intuition about what autovacuum is for, this table should basically run itself.

But there it is.

The intuition isn't wrong. It's incomplete. Autovacuum exists to clean up after concurrent row modifications, and your workload doesn't do that. What your workload does generate is a steady stream of other work that lands in the same process: dead tuples from aborted transactions, hint bits that need setting, transaction IDs that need freezing before the counter wraps. None of it is the problem autovacuum was designed to solve. All of it runs through the same mechanism.

This post isn't about how to tune autovacuum. It's about understanding what it's actually doing on your tables, why tuning helps at the margin but not at the root, and what it means that a process built for row modification cleanup is your most persistent background worker on a table that never modifies rows.

What autovacuum is actually for

Postgres MVCC keeps old row versions alive as long as any active transaction might need to see them. When a row gets updated, the old version stays on the heap page, marked dead. When a row gets deleted, same thing. These dead tuples accumulate until something cleans them up. That something is autovacuum.

Without it, dead tuples pile up permanently. Table bloat grows without bound. Heap pages that hold dead tuples can't be reused. Query performance degrades as scans trip over dead rows.

And then there's the harder problem: XID wraparound. Transaction IDs are 32-bit counters. About 2 billion transactions in, Postgres loses the ability to distinguish old from new. Rows from before the wraparound point become invisible. This isn't theoretical. It has happened in production. Autovacuum's freeze pass exists to prevent it by marking old tuples as frozen before the counter laps them.

For a standard OLTP workload with concurrent reads and updates on shared rows, this is essential infrastructure. Dead tuple accumulation is a direct consequence of normal operation. The cleanup cost is proportional to the update and delete rate. Makes sense.

The confusing part is what happens when your workload never updates or deletes anything.

Three reasons autovacuum runs on append-only tables

Each of these is a different mechanism. None of them are the same problem.

Aborted transactions leave dead tuples. Not every INSERT commits. Connection drops mid-transaction. Application errors trigger rollbacks. Explicit transaction management has bugs. At high insert rates, even a small abort rate produces a steady trickle of dead tuples. A 0.1% abort rate at 50,000 inserts per second is 50 dead tuples per second. Autovacuum has to find and mark them, even though those rows were never part of a committed write.

This is real dead tuple work. Just not from updates. You can see it directly in n_dead_tup in pg_stat_user_tables. Tuning autovacuum to run more aggressively cleans it up faster. There's no way to eliminate it without eliminating aborted transactions, which isn't realistic.

Hint bits require page dirtying. This one surprises most people who haven't dug deep into Postgres internals. When a row is first read after being written, Postgres doesn't just hand you the data. It verifies the writing transaction committed. It checks pg_xact. Once confirmed, it sets a hint bit in t_infomask to cache that result so future reads don't have to hit pg_xact again.

Setting that hint bit modifies the tuple header. A modified tuple header dirties the page. A dirty page needs writing back to disk.

So: your append-only table with immutable rows is generating I/O from reads. Not writes. Reads. The rows don't change. The headers do. This affects checkpoint pressure and the overall I/O budget autovacuum has to compete for.

Insert volume alone triggers autovacuum for freezing. Since PostgreSQL 13, autovacuum_vacuum_insert_threshold and autovacuum_vacuum_insert_scale_factor control a separate trigger: autovacuum fires based on insert count, not just dead tuple count. The reason is XID wraparound prevention. At high insert rates, unfrozen tuples accumulate fast. Postgres needs to freeze them before the counter laps. So autovacuum runs a freeze pass continuously on high-insert tables, regardless of whether any rows have been updated or deleted.

Go check vacuum_count and autovacuum_count in pg_stat_user_tables on your busiest append-only partition. They're climbing. n_dead_tup might be low. The freeze passes are happening anyway. This BPFtrace walkthrough shows exactly which passes and when.

What tuning actually does

Most autovacuum tuning falls into two categories: make it run more aggressively, or make it yield more to writes.

Running it more aggressively means lowering autovacuum_vacuum_scale_factor, increasing autovacuum_max_workers, reducing autovacuum_naptime. Dead tuples get cleaned before they affect query plans. Freeze passes complete before XID pressure builds. Real improvement.

Yielding more to writes means increasing autovacuum_vacuum_cost_delay, lowering autovacuum_vacuum_cost_limit. Write latency stabilizes. The tradeoff is that vacuum falls further behind and bloat accumulates more between cycles.

Here's what neither category does: reduce the amount of work autovacuum needs to do. Every configuration choice is about how the work gets distributed across time and how aggressively it competes with your actual workload. You're tuning the scheduler. Not the workload.

Per-table overrides are the right implementation of this: more aggressive settings on active partitions, letting older ones vacuum on a slower cycle. Good practice. It's still adjusting the tax rate, not the taxable activity.

The operational cost people don't account for

Autovacuum tuning isn't free engineering time.

Someone has to write the per-table ALTER statements. Someone has to monitor whether the settings are actually working. At 500 partitions, "monitor autovacuum lag" is a real job that runs on a recurring schedule. New partitions inherit defaults unless automation creates them with the right settings. That automation needs maintaining.

The monitoring surface for autovacuum lag spans three separate system views and requires correlating timestamps across them. It’s not a dashboard that lights up. It’s a debugging session.

When autovacuum falls behind, the symptom usually isn't an autovacuum alert. It's query latency regression. Write performance degradation. The connection between autovacuum backlog and query performance is real but indirect. Connecting the two is senior engineer work. Hours, not a quick look at a graph. Sigh.

If you've tracked where your team actually spends time on database operations, a meaningful slice of it is this: watching autovacuum, tuning autovacuum, debugging incidents that turn out to be autovacuum-adjacent, and writing runbooks for autovacuum behavior on new partition types.

All of that cost is real. None of it is the kind of cost an append-only workload should be paying.

Why append-only data shouldn't work this way

An append-only system has a simple storage contract. Data arrives. It gets written. It ages. Eventually it gets dropped, wholesale, by time window.

Nothing is ever modified in place. There is no concurrent row modification to manage. Reads never block writes and writes never block reads because there's no contention to prevent. The MVCC guarantee is irrelevant to the workload.

Postgres MVCC is the right model for workloads that need that concurrency guarantee. For workloads that don't, it's overhead. Autovacuum running continuously on your append-only table isn't Postgres failing. It's Postgres correctly maintaining the MVCC invariants on a workload that doesn't benefit from them.

The cost is real. The benefit isn't.

Tuning autovacuum makes the cost more manageable. It doesn't make the benefit appear.

What changes when the storage model matches the workload

Hypercore is built for exactly that contract. It abandons the per-tuple MVCC model for compressed column segments. Up to 1,000 row versions get batched into a single compressed segment before writing. Dead tuples in the traditional sense don't accumulate because the storage model doesn't create them.

When rows land in a Hypercore segment, transaction visibility is tracked at the segment level, not the tuple level. There is no per-tuple t_xmin to freeze. No hint bit to set on first read. The three mechanisms that drive autovacuum on your append-only heap table have nothing to work with here because the storage model does not create the conditions they require.

The result: autovacuum pressure drops proportionally. Postgres doesn't disappear. There's still housekeeping work. But the continuous autovacuum activity on high-insert partitions goes away. So does the I/O competition during write peaks, the tuning overhead, the monitoring surface for vacuum lag. The conditions that produced all of it no longer exist.

vacuum_count stops climbing on tables that nobody updates. pg_stat_activity stops showing vacuum workers at 3am. The runbook section on autovacuum tuning gets shorter.

Same SQL. Same wire protocol. Different storage contract underneath, matched to the workload that's actually running.

Conclusion

Autovacuum isn't broken. It isn't misconfigured. It's working correctly given what it's been handed.

The problem is that "working correctly" on an append-only high-insert table means running constantly, competing with writes, requiring ongoing tuning, and consuming real engineering time. All to manage overhead that a purpose-built append system wouldn't generate in the first place.

That's the tax. The settings determine how you pay it. The workload determines that you owe it.

If autovacuum is in your top processes by CPU and I/O on a table that nobody updates, that's not a Postgres problem to fix. It's a signal about the relationship between your workload and your storage model. If your write pattern is append-only, the question worth asking is not how to tune autovacuum better. It is whether your storage model is matched to your workload at all.

Understanding Postgres Performance Limits for Analytics on Live Data goes deeper on where that mismatch shows up across the system, and what the path forward looks like at different data volumes.

This post is a practical companion to It's 2026, Just Use Postgres. That post makes the architectural case for consolidating on Postgres. This one shows you how.

Below are working SQL examples for each use case. Every extension listed here is available on Tiger Cloud with no additional setup. If you're self-hosting, each section links to the extension's repo.

What you'll be able to do after reading this: Set up Postgres extensions for full-text search, vector search, time-series, caching, message queues, document storage, geospatial queries, and scheduled jobs. Each section is self-contained, so you can skip to what you need.

Enable Everything

Here's the full set. You probably don't need all of them. Pick the ones that match your workload.

CREATE EXTENSION pg_textsearch;   -- BM25 full-text search
CREATE EXTENSION vector;          -- Vector search (pgvector)
CREATE EXTENSION vectorscale;     -- DiskANN index for vectors
CREATE EXTENSION ai;              -- AI embeddings and RAG workflows
CREATE EXTENSION timescaledb;     -- Time-series
CREATE EXTENSION pgmq;            -- Message queues
CREATE EXTENSION pg_cron;         -- Scheduled jobs
CREATE EXTENSION postgis;         -- Geospatial

Full-Text Search (Replace Elasticsearch)

Extension: pg_textsearch (true BM25 ranking)

What you're replacing: Elasticsearch (separate JVM cluster, complex mappings, sync pipelines), Solr, or Algolia ($1 per 1,000 searches).

What you get: The same BM25 algorithm that powers Elasticsearch, running natively in Postgres. No separate cluster. No sync jobs. No data drift.

CREATE TABLE articles (
  id SERIAL PRIMARY KEY,
  title TEXT,
  content TEXT
);

-- Create a BM25 index
CREATE INDEX idx_articles_bm25 ON articles USING bm25(content)
  WITH (text_config = 'english');

-- Search with BM25 scoring
SELECT title, -(content <@> 'database optimization') AS score
FROM articles
ORDER BY content <@> 'database optimization'
LIMIT 10;

Deep dive: You Don't Need Elasticsearch: BM25 is Now in Postgres

Vector Search (Replace Pinecone)

Extensions: pgvector + pgvectorscale

What you're replacing: Pinecone ($70/month minimum, separate infrastructure, data sync), Qdrant, Milvus, or Weaviate.

What you get: pgvectorscale uses the DiskANN algorithm (from Microsoft Research). On a 50M vector benchmark, it achieved 28x lower p95 latency and 16x higher throughput than Pinecone at 99% recall.

CREATE EXTENSION vector;
CREATE EXTENSION vectorscale CASCADE;

CREATE TABLE documents (
  id SERIAL PRIMARY KEY,
  content TEXT,
  embedding vector(1536)
);

-- High-performance DiskANN index
CREATE INDEX idx_docs_embedding ON documents USING diskann(embedding);

-- Find similar documents
SELECT content, embedding <=> '[0.1, 0.2, ...]'::vector AS distance
FROM documents
ORDER BY embedding <=> '[0.1, 0.2, ...]'::vector
LIMIT 10;

Auto-sync embeddings with pgai

No more manual embedding pipelines. pgai regenerates embeddings automatically on every INSERT and UPDATE.

SELECT ai.create_vectorizer(
  'documents'::regclass,
  loading => ai.loading_column(column_name => 'content'),
  embedding => ai.embedding_openai(
    model => 'text-embedding-3-small',
    dimensions => '1536'
  )
);

Every row stays in sync. No batch jobs. No drift.

Hybrid Search: BM25 + Vectors in One Query

This is where Postgres consolidation pays off immediately. Combining keyword search and semantic search in other stacks requires two API calls, result merging, failure handling, and double the latency. In Postgres, it's one query.

Simple weighted hybrid

SELECT
  title,
  -(content <@> 'database optimization') AS bm25_score,
  embedding <=> query_embedding AS vector_distance,
  0.7 * (-(content <@> 'database optimization')) +
  0.3 * (1 - (embedding <=> query_embedding)) AS hybrid_score
FROM articles
ORDER BY hybrid_score DESC
LIMIT 10;

Reciprocal Rank Fusion (for RAG applications)

WITH bm25 AS (
  SELECT id, ROW_NUMBER() OVER (ORDER BY content <@> $1) AS rank
  FROM documents LIMIT 20
),
vectors AS (
  SELECT id, ROW_NUMBER() OVER (ORDER BY embedding <=> $2) AS rank
  FROM documents LIMIT 20
)
SELECT d.*,
  1.0 / (60 + COALESCE(b.rank, 1000)) +
  1.0 / (60 + COALESCE(v.rank, 1000)) AS score
FROM documents d
LEFT JOIN bm25 b ON d.id = b.id
LEFT JOIN vectors v ON d.id = v.id
WHERE b.id IS NOT NULL OR v.id IS NOT NULL
ORDER BY score DESC LIMIT 10;

One query. One transaction. One result set.

Time-Series (Replace InfluxDB)

Extension: TimescaleDB (21K+ GitHub stars)

What you're replacing: InfluxDB (separate database, Flux or limited SQL), Prometheus (metrics only, not application data).

What you get: Automatic time-based partitioning, compression up to 95%, continuous aggregates for fast dashboards, and full SQL. Your time-series data lives alongside your relational data with JOINs and ACID guarantees.

CREATE EXTENSION timescaledb;

CREATE TABLE metrics (
  time TIMESTAMPTZ NOT NULL,
  device_id TEXT,
  temperature DOUBLE PRECISION
);

-- Convert to a hypertable (automatic time partitioning)
SELECT create_hypertable('metrics', 'time');

-- Query with time buckets
SELECT time_bucket('1 hour', time) AS hour,
       AVG(temperature)
FROM metrics
WHERE time > NOW() - INTERVAL '24 hours'
GROUP BY hour;

Lifecycle automation

TimescaleDB handles retention and compression policies so you don't have to build cron jobs for data management.

-- Automatically drop data older than 30 days
SELECT add_retention_policy('metrics', INTERVAL '30 days');

-- Compress data older than 7 days (up to 95% storage reduction)
ALTER TABLE metrics SET (timescaledb.compress);
SELECT add_compression_policy('metrics', INTERVAL '7 days');

Case study: Plexigrid went from 4 databases to 1 and got 350x faster queries.

Caching (Replace Redis)

Feature: UNLOGGED tables + JSONB (built into Postgres, no extension needed)

What you're replacing: Redis for simple key-value caching scenarios.

What you get: In-memory-speed storage without WAL overhead. Good for session data, temporary lookups, and simple caches. No separate service to operate.

When to keep Redis: If you need pub/sub, sorted sets, Lua scripting, or complex data structures, Redis is still the better tool for those specific jobs.

-- UNLOGGED = no WAL overhead, faster writes
CREATE UNLOGGED TABLE cache (
  key TEXT PRIMARY KEY,
  value JSONB,
  expires_at TIMESTAMPTZ
);

-- Set with expiration
INSERT INTO cache (key, value, expires_at)
VALUES ('user:123', '{"name": "Alice"}', NOW() + INTERVAL '1 hour')
ON CONFLICT (key) DO UPDATE SET value = EXCLUDED.value;

-- Get
SELECT value FROM cache
WHERE key = 'user:123' AND expires_at > NOW();

-- Schedule cleanup with pg_cron
SELECT cron.schedule('cache_cleanup', '0 * * * *',
  $$DELETE FROM cache WHERE expires_at < NOW()$$);

Message Queues (Replace Kafka)

Extension: pgmq

What you're replacing: Kafka or RabbitMQ for task queues and simple event processing.

What you get: A lightweight message queue inside Postgres. Send, receive with visibility timeouts, and delete after processing. Transactional with the rest of your data.

When to keep Kafka: If you need high-throughput event streaming across dozens of services, consumer groups, exactly-once semantics, or multi-datacenter replication, Kafka is purpose-built for that.

CREATE EXTENSION pgmq;
SELECT pgmq.create('my_queue');

-- Send a message
SELECT pgmq.send('my_queue', '{"event": "signup", "user_id": 123}');

-- Receive (with 30-second visibility timeout)
SELECT * FROM pgmq.read('my_queue', 30, 5);

-- Delete after processing
SELECT pgmq.delete('my_queue', msg_id);

Alternative: SKIP LOCKED pattern (no extension needed)

For simple job queues, Postgres has a built-in pattern using FOR UPDATE SKIP LOCKED:

CREATE TABLE jobs (
  id SERIAL PRIMARY KEY,
  payload JSONB,
  status TEXT DEFAULT 'pending'
);

-- Worker claims a job atomically
UPDATE jobs SET status = 'processing'
WHERE id = (
  SELECT id FROM jobs WHERE status = 'pending'
  FOR UPDATE SKIP LOCKED LIMIT 1
) RETURNING *;

Documents (Replace MongoDB)

Feature: Native JSONB (built into Postgres since 2014)

What you're replacing: MongoDB for document storage.

What you get: Schemaless document storage with GIN indexing, plus everything Postgres gives you: ACID transactions, relational JOINs, and SQL. No separate database for your "document-shaped" data.

CREATE TABLE users (
  id SERIAL PRIMARY KEY,
  data JSONB
);

-- Insert a nested document
INSERT INTO users (data) VALUES ('{
  "name": "Alice",
  "profile": {"bio": "Developer", "links": ["github.com/alice"]}
}');

-- Query nested fields
SELECT data->>'name', data->'profile'->>'bio'
FROM users
WHERE data->'profile'->>'bio' LIKE '%Developer%';

-- Index specific JSON fields for fast lookups
CREATE INDEX idx_users_email ON users ((data->>'email'));

Geospatial (Replace Specialized GIS)

Extension: PostGIS (the industry standard since 2001)

What you're replacing: Nothing, really. PostGIS is what most specialized GIS tools are built on. It powers OpenStreetMap and has been in production for 24 years.

CREATE EXTENSION postgis;

CREATE TABLE stores (
  id SERIAL PRIMARY KEY,
  name TEXT,
  location GEOGRAPHY(POINT, 4326)
);

-- Find stores within 5km
SELECT name,
  ST_Distance(location, ST_MakePoint(-122.4, 37.78)::geography) AS meters
FROM stores
WHERE ST_DWithin(location, ST_MakePoint(-122.4, 37.78)::geography, 5000);

Scheduled Jobs (Replace External Cron)

Extension: pg_cron

What you're replacing: External cron jobs, Kubernetes CronJobs, or Lambda scheduled triggers for database maintenance tasks.

What you get: Cron scheduling inside Postgres. Useful for cache cleanup, materialized view refreshes, data retention, and periodic aggregation.

CREATE EXTENSION pg_cron;

-- Run cache cleanup every hour
SELECT cron.schedule('cleanup', '0 * * * *',
  $$DELETE FROM cache WHERE expires_at < NOW()$$);

-- Refresh a materialized view every night at 2 AM
SELECT cron.schedule('rollup', '0 2 * * *',
  $$REFRESH MATERIALIZED VIEW CONCURRENTLY daily_stats$$);

Fuzzy Search (Typo Tolerance)

Extension: pg_trgm (built into Postgres)

CREATE EXTENSION pg_trgm;

CREATE INDEX idx_name_trgm ON products USING GIN (name gin_trgm_ops);

-- Finds "PostgreSQL" even when typed as "posgresql"
SELECT name FROM products
WHERE name % 'posgresql'
ORDER BY similarity(name, 'posgresql') DESC;

What's Next

If you want the architectural argument for why consolidating on Postgres matters (especially in the AI era), read It's 2026, Just Use Postgres.

All of these extensions come pre-configured on Tiger Cloud. Create a free database and start building.

Further reading:

• pg_textsearch documentation
• pgvectorscale on GitHub
• TimescaleDB documentation
• pgmq on GitHub
• PostGIS
• How Plexigrid replaced InfluxDB and got 350x faster queries

There are two kinds of database performance problems.

The first kind responds to optimization. You add an index, queries speed up, the improvement holds. Done.

The second kind responds to optimization temporarily. You add an index, queries speed up, then slow down again as data grows. You partition the table, scans get faster, then the partition count becomes its own management burden. You upgrade the instance, headroom appears, then it's consumed within a quarter. You're back in the same meeting, staring at the same graphs trending in the same wrong direction.

Both feel identical in the moment. A slow query is a slow query. The difference shows up after the fix. Optimization problems stay fixed. Architectural problems come back. The difference isn't a matter of degree; it's structural. Operational problems respond to tuning. Architectural problems compound with every additional row you write.

This distinction matters most for a specific class of workload: high-volume, append-heavy data that gets queried analytically. Time-series telemetry. Financial tick data. Operational metrics. Any system where data accumulates continuously and queries shift from point lookups to scans and aggregates on live data. If that describes your system, this post gives you a framework for figuring out which kind of problem you're dealing with, before you've spent six months finding out the hard way.

What optimization can fix

Optimization is genuinely powerful. Worth being clear about that before talking about its limits.

Configuration mismatches are real problems with real fixes. shared_buffers set too low. work_mem too small for hash joins. effective_cache_size not reflecting actual available memory. You fix these once, they stay fixed. These aren't minor tweaks.

Missing indexes are the same story. A query doing a sequential scan when an index scan would serve is O(n) when it could be O(log n). Add the right index, the improvement is permanent. The fix works at 1 million rows and at 100 million rows.

Inefficient queries, connection management, autovacuum tuning, PgBouncer config. All real, all fixable, all improvements that don't interact with data volume in ways that eventually undo your work.

The common thread: these problems don't compound with scale. A missing index is a missing index. You find it, you fix it, you move on.

Why certain workloads hit a wall

Here's where it gets tricky.

Some problems look like optimization problems on the surface. You apply the standard tools. Performance improves. You close the ticket. Three months later, you're back.

That pattern is the signal.

The underlying issue isn't misconfiguration. It's a mismatch between what your architecture provides and what your workload actually requires. For high-volume, append-heavy, analytically-queried data, that mismatch runs three layers deep, and each layer is worse than the last.

Layer one: you're reading data you'll never use. When your dominant query pattern is analytical scans and aggregations across time ranges, and your storage model packs all columns together in rows, no index strategy resolves that. Row storage reads the full row on every access. For transactional workloads, fine. For a query that needs to scan 50 million rows and pull two columns from each one, you're reading every other column on every single row, all the way through. That's not a configuration problem. That's a physics problem.

Layer two: you're paying for features you never use. MVCC exists so concurrent reads and writes stay correct. Valuable for transactional data. But if you're inserting 50,000 rows per second of data that will never be updated, you're paying the full cost of that concurrency model on every insert. Each row carries 23 bytes of MVCC transaction metadata. At 50K inserts per second, that's overhead on 4.3 billion rows per day that will never be modified. Autovacuum runs constantly, cleaning up dead tuples that were never created through updates. No configuration setting removes that structural cost. You're paying correctness guarantees on data that will never be written again.

Layer three: the maintenance never stops growing. Autovacuum, ANALYZE, background statistics collection. As data volume grows, the time these tasks take grows with them. Tuning parameters adjusts priority, not necessity. The work still has to happen, and it will compete with your production workload in ways that compound indefinitely. The data keeps accumulating, and the maintenance scales with it.

Operational problems respond to tuning. Architectural problems compound with every additional row you write.

The recurrence test

The question is whether you've already crossed that line. Here's how to tell.

Apply the standard fix for whatever symptom you're seeing. Better index. Config change. Query rewrite. Partitioning. Then watch what happens.

A fix that holds for a week is probably an operational problem. A fix that holds for a month could be either; watch for regression. A fix that holds for a quarter, and then the same metrics start climbing again: that's the architectural signal.

But the most reliable test isn't the persistence of any individual fix. It's the pattern across fixes. Over the last 12 months:

• Are you solving different problems each time, or variations of the same problem?
• Are the fixes holding, or buying progressively less time?
• Is the interval between optimization cycles getting shorter?

Diverse symptoms, targeted fixes, improvements that hold = healthy optimization. You're in the right architecture.

Same symptoms recurring, each fix buying less runway than the last, the cycle accelerating = you've crossed the architectural boundary. The optimization is working. It's just fighting a battle it can't win.

If you answered yes to all three of those last questions, you already know the answer. The recurrence test just gives you the language to say it out loud in a meeting.

The cost of getting this wrong

Both failure modes are expensive. Worth saying plainly, because the bias in engineering is almost always toward optimization over migration.

Treating an optimization problem as architectural: You migrate when tuning would have been sufficient. The cost is a disruptive migration project (two to eight weeks is realistic), team bandwidth redirected from product work to infrastructure, and the risk of introducing new complexity into a system that didn't need it. This mistake is less common because the default is to keep optimizing. Teams usually get pushed toward migration by external pressure, not internal conviction.

Treating an architectural problem as optimization: This one compounds. You keep tuning when the architecture is the constraint. The cost is cumulative engineering time (teams working through this pattern typically lose months per year to ongoing optimization cycles), a growing operational burden, and a deferred migration that gets more expensive the longer you wait. At 10 million rows, migration takes days. At 500 million, weeks. At a billion plus, you're looking at months.

Sigh.

Every quarter you spend on optimizations that aren't sticking, you're paying twice: once for the engineering time, and once in the form of increased migration cost when you eventually get there. The diagnostic matters because catching it early changes the math significantly.

What the right architecture actually changes

If you ran the recurrence test and landed where the evidence points, here's what changes.

If your workload is high-volume, append-heavy, and primarily analytical, the architectural answer doesn't require leaving Postgres. It requires extending Postgres with primitives designed for that specific workload pattern.

TimescaleDB addresses each of the three failure modes directly.

Layer one (reading data you'll never use): Hypercore. TimescaleDB's hybrid row/columnar storage engine keeps recent data in row format for writes and automatically converts older data to columnar format for analytical scans. A query that needs to scan millions of rows and extract two columns reads only those columns, from compressed columnar chunks. You stop paying row-storage costs on data you'll only scan.

Layer two (paying for features you never use): Hypertables and chunk exclusion. Hypertables partition your data automatically by time. Chunk exclusion lets the query planner skip entire time partitions without scanning them. Query latency stays bounded as data grows, not because you've tuned something but because the planner knows which chunks are irrelevant. Compressed chunks also dramatically reduce autovacuum load: there's nothing to vacuum in a chunk where data is already frozen and compressed.

Analytical performance on top of all of it: Continuous aggregates. Incremental materialized views that refresh in the background, updating only what changed since the last refresh. Dashboards and aggregations stay fast without batch jobs or stale results. Precomputed rollups combine with raw recent data at query time, so results are current without the full scan cost.

The optimization work that was recurring becomes unnecessary because the architecture handles it at the right layer. You're not doing the same work in a faster system. You're doing less work in a system built for the load.

And it's still Postgres. Same SQL. Same extensions. Same tooling your team already knows. What changes is what happens underneath. That's the only thing that needed to change.

The decision

Optimization and architecture solve different categories of problems. The skill is knowing which category you're in before you've invested months in the wrong one.

If your optimization work is targeted, diverse, and the fixes hold, you're in the right architecture. Keep going. The standard playbook works.

If the recurrence test comes back positive (same symptoms, shrinking runway, accelerating cycles), the architecture is the constraint. One more optimization buys a quarter. The right architectural change buys years.

That gap is the decision.

For the full diagnostic on what this workload pattern looks like in a real system, Six Signs That Postgres Tuning Won't Fix Your Performance Problems walks through six specific characteristics with concrete examples. For the deeper mechanical explanation of why vanilla Postgres hits these limits at the architecture level, Understanding Postgres Performance Limits for Analytics on Live Data covers that ground. Start with whichever question is more pressing.

Standard relational normalization is the bedrock of database design. It prevents data duplication and maintains integrity by splitting information into specialized tables. However, as your tables approach the 500 million row mark, the very joins that keep your data clean begin to degrade performance. You may notice p95 latencies creeping up even though your queries remain logically sound. This guide explains how to identify when relational purity is holding you back and how to use data flattening and columnar compression to reclaim your speed.

What You Will Learn

This guide breaks down the mechanics of read amplification and join overhead in high-volume systems. You will learn:

• How normalized schemas create latency through expensive nested loop joins.
• When to flatten data into wide tables to reduce query-time computation.
• The way columnar compression turns denormalized tables into high-performance assets.
• Techniques to reduce read amplification by minimizing the data your database must touch.

Why It Matters

Standard normalization relies on joins to reconstruct data at query time. On small datasets, this is efficient. On tables with billions of rows, every join adds a layer of I/O and CPU overhead that compounds as your data volume grows. Row-based storage forces the database to read every column in a row, even if your query only needs two, leading to massive read amplification.

Flattening your schema, moving frequently joined metadata directly into your main ingestion table, removes the need for these expensive joins. While this usually increases storage size in a row-based system, specialized columnar storage allows for aggressive compression of this redundant data. By choosing the right schema architecture, you can significantly boost the responsiveness of your real-time dashboards and analytics.

Audit Your p95 Latency

Start by identifying the queries that drive your latency spikes. Use the pg_stat_statements extension to find queries with high total execution time. Run EXPLAIN (ANALYZE, BUFFERS) on these queries to see how the database interacts with the storage layer.

Look for "Nested Loop" or "Hash Join" operations where the "Shared Hit Blocks" are high. If the database spends 80% of its time matching keys between a 500M row metrics table and a 10k row device table, your architecture is likely CPU-bound by join coordination. A high number of "read" buffers in the output indicates that the database is hunting through indexes and table pages to find metadata that isn't stored locally with the record.

When you see a query plan where the join cost increases linearly with the size of the primary table, you have identified a "join tax" that cannot be solved with more memory or faster disks. At this stage, the overhead of managing the relationship between tables is outweighing the benefits of relational purity.

Migrate High-Access Metadata

Identify the metadata columns that appear most often in your WHERE and JOIN clauses.

• Good Candidates: Low-cardinality strings like region, site_id, or device_type. These compress effectively in a columnar format because the engine only needs to store a single value and a list of row counts.
• Poor Candidates: High-cardinality unique identifiers like session_uuid or frequently updated timestamps. These values vary for almost every row, which prevents the compression engine from reducing the storage footprint and can lead to table bloat.

Then, add the new denormalized column to your high-frequency ingestion table:

ALTER TABLE device_metrics ADD COLUMN region TEXT;

Next, backfill the existing rows. On tables with hundreds of millions of rows, a single UPDATE statement can cause massive transaction log bloat and table locks. Instead, backfill in batches or use a join-based update to pull metadata from your reference table:

UPDATE device_metrics m
SET region = d.region
FROM devices d
WHERE m.device_id = d.id
  AND m.region IS NULL;

This migration eliminates the join at query time, allowing the database to filter and aggregate in a single pass. By localizing the data, you ensure the database engine no longer needs to load secondary table pages into the buffer pool just to check a filter condition.

Configure Columnar Compression

After flattening your table, you must enable columnar storage to handle the redundant metadata. In Tiger Data, this is achieved through a hybrid storage engine that partitions data into "chunks" and then compresses those chunks into a columnar format. This storage engine groups data by column rather than row, allowing the database to ignore unused columns during a scan and reducing the physical I/O required for every query.

To implement this, you first define your compression policy. You must choose a segmentby column, typically a device ID or primary key, and an orderby column, usually a timestamp. The segmentby column is the most important for performance; it determines how data is grouped within the compressed segments.

-- 1. Enable compression on the hypertable
ALTER TABLE device_metrics SET (
  timescaledb.compress = true,
  timescaledb.compress_orderby = 'ts DESC',
  timescaledb.compress_segmentby = 'device_id, region'
);

-- 2. Add a policy to compress data older than 7 days
SELECT add_compression_policy('device_metrics', INTERVAL '7 days');

When the compression policy runs, the database transforms the row-based data into compressed columnar batches. For a flattened table, this is where the performance "magic" happens. If you have 100 million rows where the region is 'north_east', a row-based engine stores that string 100 million times. The columnar engine stores it once along with a metadata bitmask, reducing the storage footprint of that column by up to 99%.

You can verify the effectiveness of your compression and the reduction in read amplification by querying the compression statistics:

SELECT 
    total_chunks,
    number_compressed_chunks,
    pg_size_pretty(before_compression_total_bytes) AS before_size,
pg_size_pretty(after_compression_total_bytes) AS after_size
FROM hypertable_columnstore_stats('device_metrics');

This architectural shift moves your database from being I/O bound to being CPU efficient. By reducing the "Shared Hit Blocks" (the number of 8KB pages the database must load into memory), you free up the buffer pool for other critical operations and effectively raise the performance ceiling of your entire system.

Measuring the Join Tax

You can measure the performance gap between a normalized join and a flattened table by comparing their execution costs. Consider a system tracking millions of IoT sensors that needs to find the average reading for a specific region.

The Normalized Approach (Slow)

This query must join two tables, forcing the database to match foreign keys for every row in the time range.

SELECT avg(m.value), d.region
FROM device_metrics m
JOIN devices d ON m.device_id = d.id
WHERE m.ts > now() - interval '1 hour.'
 AND d.region = 'north_east'
GROUP BY d.region;

​The Flattened and Compressed Approach (Fast)

By moving the region column into the device_metrics table and using columnar compression, the join disappears. The database only reads the value and region columns from disk, ignoring the rest of the row.

SELECT avg(value), region
FROM device_metrics
WHERE ts > now() - interval '1 hour.'
 AND region = 'north_east'
GROUP BY region;

Next Step

Identify your most expensive join query using pg_stat_statements. Create a denormalized version of that table that includes the necessary metadata, and test query performance against a compressed columnar version. This test will help you determine if your system is hitting a structural ceiling that only an architectural change can fix.

To try this on a fully managed instance, start a free Tiger Cloud trial. You can compare and contrast query performance on real data without impacting your production database.

Picture this: you're searching through your company's internal knowledge base. You know the answer is in there. You typed in what you thought were the right keywords. And yet, the top result is somebody's 2019 onboarding doc about the coffee machine.

Sound familiar?

I've watched this scene play out in every company I've worked at, across every tool that claimed to have "search." The problem isn't that search itself is broken. It's that we've been asking it to do two completely different jobs with one rigid toolset, and calling it solved.

Here's what most people miss: search doesn't have a modality problem. It has an architecture problem. And the fix isn't a bigger stack. It's a smarter query, running in the database you already trust.

The two halves of "finding stuff"

We use "search" as though it's one thing. It isn't.

When someone types a query into a search bar, they're usually doing one of two things:

1. Looking for a specific word or phrase they remember. "Find me that doc with 'SOC 2 audit' in it." This is keyword search. It rewards precision and specificity.
2. Trying to describe a concept, not quote it. "Find me docs about how we handle customer data during compliance reviews." This is semantic search. It rewards understanding.

Traditional full-text search, powered by algorithms like BM25, is excellent at the first job. It's stemmed, ranked, battle-tested, and has been the backbone of search engines for decades. It finds "imposter syndrome" when you type "imposter."

Vector search, the newer kid on the block, is excellent at the second job. It embeds your text into a high-dimensional space where meaning lives closer together than words do. It finds "imposter syndrome" when you type "feeling like a fraud at work," even though literally none of the words overlap.

Now here's the punchline: neither of these systems is wrong, and neither of them is enough.

If you only use BM25, you miss every user who doesn't know the exact vocabulary your docs use. If you only use vector search, you miss the person who typed a part number, a legal citation, or "episode 100." Vector search has no particular opinion about the number 100.

The answer is not to pick a side. The answer is to run both and combine the results. That's hybrid search, and it turns out you can do all of it in one database, in one query, against data you're already storing there.

Why this matters now

Two years ago, hybrid search in Postgres was a demo. Now it's a production architecture. Here's what changed.

BM25 actually runs fast in Postgres now. pg_textsearch 1.0 built the full search engine in C, on top of Postgres's storage layer. Not a wrapper around an external library. Not ts_rank with extra steps. Block-Max WAND optimization, native. For a lot of workloads, you don't need Elasticsearch anymore.

Vector search in Postgres stopped being a party trick. pgvectorscale's DiskANN-inspired index outperforms specialized vector databases on cost and latency at 75% less cost. The index lives on disk, so you're not RAM-capped. Filtered search actually works under production RAG conditions, which is historically where specialized vector databases fall apart.

Every team I talk to is consolidating. LLM apps don't need five databases. They need one system that can store source content, embeddings, chat history, user context, and audit trails, all in the same transactional boundary. The teams winning on infrastructure right now are the ones who realized Postgres is that system and stopped building around it. (My colleagues have been saying this loudly for a while: it's 2026, just use Postgres.)

That third one is the one I keep coming back to. Five databases to answer one question is an architecture that made sense once and then quietly stopped making sense. Most teams are a quarter or two behind on noticing.

Put those three together and hybrid search in Postgres went from "neat prototype" to a real production architecture. That's the shift this post is about.

Why Postgres is where this belongs

I've built the multi-system search stack. Elastic for BM25, Pinecone for vectors, a custom merge service, Redis for caching, and the ritual of explaining the whole thing to every new hire who asks "but why can't we just query the database?"

I've also cussed out that architecture at 2am when the sync job fell behind and results went stale and nobody could figure out which system was wrong.

Every piece of that stack is something you provision, secure, back up, upgrade, monitor, and carry on-call. Every hop between systems is a place where results drift, eventual consistency bites you, and your latency budget gets eaten. Every six months, one of those systems has a breaking change, an incompatible update, or a pricing restructure, and the whole Jenga tower needs attention.

The pragmatic answer: your data is almost certainly already in Postgres. Your users, your content, your metadata, your access controls, your audit logs. So why ship all of it to three other systems just to search it?

You don't have to. Postgres now has what you need:

• pg_textsearch for BM25 keyword search — proper ranked retrieval, not the old ts_rank approximation
• pgvectorscale for vector similarity search with a StreamingDiskANN index that doesn't require your entire index to fit in RAM
• Everything else you already know: joins, filters, transactions, row-level security, time-based partitioning, and the SQL your whole team can read

Put those together and you get a database that can rank by keyword, rank by meaning, filter by any column in your schema, respect your access rules, and return a single fused result set. In one query. Against one system.

That's not a compromise. That's a better architecture.

How hybrid search actually works (the short version)

The full step-by-step is in the Tiger Data docs, including a walkthrough using podcast transcripts from Conduit: 12 episodes, real embeddings, about 30 minutes. Swap "podcast episodes" for "support tickets" and the pattern is identical.

Strip it down to the bones: one table, two indexes, two queries, a little math, one answer.

1. Store your content and its embedding side by side. A single episodes table with a description column and a vector(1536) column. No sync jobs. No reconciliation.
2. Index both. A BM25 index on the text column. A StreamingDiskANN index on the embedding column.
3. Run both searches. BM25 returns a ranked list of keyword matches. Vector search returns a ranked list of semantic matches. Each produces, say, the top 20 candidates.
4. Fuse the rankings with RRF. Reciprocal Rank Fusion scores each result by 1 / (k + rank) across both lists and sums the scores. An item that's #1 in both lists crushes an item that's only in one. An item that's #15 in one and missing from the other still gets a say. (The math here is simpler than it sounds; you're basically rewarding documents that placed well in both races.)
5. Return the fused list. One query. One result set. Ranked by a score that rewards showing up in both.

The SQL for step 4:

WITH bm25_results AS (
  SELECT id, ROW_NUMBER() OVER (
    ORDER BY description <@> 'mental health boundaries'
  ) AS rank
  FROM episodes
  ORDER BY description <@> 'mental health boundaries'
  LIMIT 20
),
vector_results AS (
  SELECT id, ROW_NUMBER() OVER (
    ORDER BY embedding <=> $1
  ) AS rank
  FROM episodes
  ORDER BY embedding <=> $1
  LIMIT 20
)
SELECT
  d.id, d.title,
  COALESCE(1.0 / (60 + b.rank), 0)
    + COALESCE(1.0 / (60 + v.rank), 0) AS rrf_score
FROM episodes d
LEFT JOIN bm25_results b ON d.id = b.id
LEFT JOIN vector_results v ON d.id = v.id
WHERE b.id IS NOT NULL OR v.id IS NOT NULL
ORDER BY rrf_score DESC
LIMIT 10;

That's hybrid search. No sidecar service, no merge layer, no second database, no "enterprise search platform" subscription. Your finance team can thank you later.

Who should be building this right now

The teams I'd push hardest toward this pattern are the ones whose search is quietly embarrassing them without anyone saying it out loud.

• Internal knowledge and support teams. Your agents need to find "ticket #48291" (a keyword query) and "something similar to this customer issue" (a semantic query) in the same tool. Building two separate search surfaces for the same data is a solvable problem, and you're paying for that complexity every sprint.
• RAG pipelines. If your retrieval layer is only doing vector search, you're leaving quality on the table. Hybrid retrieval consistently outperforms pure semantic retrieval, especially for queries that include specific identifiers, product names, or technical terms the embedding model treats as just more tokens. Worth fixing before you spend another month tuning your prompts.
• Product catalogs and discovery. A user searching "red running shoes under $80 that won't fall apart" is giving you structured filters, keyword cues, and a vibe. BM25 handles the keyword cues, vectors handle the vibe, and a SQL WHERE clause handles the filter. All three, one query.

A few honest caveats

pg_textsearch 1.0 doesn't support native phrase queries... yet. There's an over-fetch plus ILIKE workaround in the tutorial, and AND/OR/NOT operators are on the roadmap. BM25 indexes are single-column by default; you use a generated column to search across title + body. And no, a 100-million-row table isn't going to be free to embed. Cost and batching are still real things you have to think through.

But those are tuning problems, not architectural ones. A tuning problem inside one database is a much better problem than an integration problem across four. (If you'd rather not hand-roll the embedding pipeline, pgai Vectorizer automates most of that cost-and-batching work.)

Try it yourself

The step-by-step tutorial walks through the whole thing: Build hybrid search with BM25 and vector similarity. Real podcast transcripts, about 30 minutes end to end, works on Tiger Cloud, Docker, or a local Postgres install.

If you want to go straight to code, the companion repo is here. Clone it, point it at a Postgres instance with pg_textsearch and pgvectorscale installed, and you have a working hybrid search system one psql -f setup.sql away. And if you want a deeper implementation walkthrough — including pgai auto-sync so embeddings stay current without a separate pipeline — this post covers that pattern end to end.

If you want the deeper "why," these are the posts I'd read next:

• From ts_rank to BM25: introducing pg_textsearch
• pg_textsearch 1.0: how we built a BM25 search engine on Postgres
• Hybrid search with TimescaleDB: vector, keyword, and temporal filtering
• Combining semantic search and full-text search in Postgres with Cohere, pgvector, and pgai
• Understanding DiskANN

The takeaway

Search is hard. It's hard because humans are imprecise, language is slippery, and the things we want are rarely the things we type. No computer is reading our minds yet (and honestly, I'm not sure I want one to). One retrieval method will never be enough.

The solution isn't a bigger stack. It's a smarter query.

Postgres, with pg_textsearch and pgvectorscale, lets you run keyword search and semantic search against the same table, fuse them with a few lines of SQL, and ship one system instead of four. That's not just operationally cheaper. It's a better developer experience, a better user experience, and a better night of sleep for whoever's on call (you can thank me later)

If your search is frustrating your users (or worse, quietly frustrating them without anyone saying anything), this is the version I'd build next.

Have you built hybrid search on Postgres? What interesting queries are you running? I want to hear about it.

Find me at erin@tigerdata.com, or drop a note on LinkedIn or Bluesky.

There's a pattern that plays out across almost every team running high-volume append workloads on vanilla Postgres. I've watched it happen enough times that I can practically set a timer.

At 10M rows, everything is fine. Queries are fast. The team is shipping features. Nobody is thinking about the database.

At 50M, queries start getting slow. Someone opens a Jira ticket about dashboard latency. The fix is usually an index or two. Takes an afternoon.

At 100M, someone proposes partitioning. Or read replicas. Or bumping the instance size. These are reasonable ideas. They work for a while.

At 500M, the team is spending one to two days per sprint on database performance work. Not building product. Tuning autovacuum_vacuum_cost_delay and rewriting queries and having meetings about whether to re-partition.

At every stage, migration to a purpose-built solution feels like it can wait. The current optimizations are working. The pain is manageable. Next quarter has fewer deadlines. (Next quarter never has fewer deadlines.)

Then the table hits a billion rows, and migration is now a project, not a task. What would have been a weekend of CREATE TABLE ... USING hypertable and a data backfill is now a phased migration plan with rollback strategies and a project manager.

This post makes the case for migrating earlier than feels necessary. Not because your current setup is broken, but because the cost of waiting (what I'll call the optimization tax) compounds in ways that aren't visible until you're deep in them.

The migration cost curve

Let's put real numbers on this. The technical steps of a migration don't change much as data grows. What changes is how long they take and how many people you need in the room.

At 10M rows: Migration is essentially CREATE TABLE, convert to hypertable, backfill data. A single engineer, a weekend. Downtime measured in minutes if you use a blue-green approach. Risk is near zero because you can run both instances in parallel and compare results before cutting over.

-- This is the scary migration at 10M rows
SELECT create_hypertable('sensor_data', 'time');

INSERT INTO sensor_data_new SELECT * FROM sensor_data_old;
-- Go get coffee. You'll be done before it's cool enough to drink.

At 100M rows: Backfill takes hours, not minutes. You need to plan for write continuity during migration. Indexes need rebuilding. Compression policies need configuring. Testing is more involved because edge cases in the data are now visible. One engineer, one to two weeks.

At 500M+ rows: Full migration project. Data transfer takes days. You need a parallel write strategy (dual-write or CDC). Query compatibility testing across the application layer. A rollback plan. Stakeholder communication. Two to four engineers, four to eight weeks.

The work itself doesn't get harder. There's just more of it, and the blast radius of a mistake gets larger. The migration at 10M rows and the migration at 500M rows are the same technical steps. The difference is entirely in scale and coordination overhead.

That's worth repeating: you're not doing something different at 500M rows. You're doing the same thing, slower, with more people watching.

The optimization tax (the part people don't calculate)

Migration has a visible cost. You can put it on a roadmap. Estimate it. Schedule it. Argue about it in sprint planning.

Staying has an invisible cost. I call it the optimization tax: the cumulative engineering time, infrastructure spend, and incident burden you pay every month to keep vanilla Postgres performing on a workload it wasn't designed for. And unlike migration, the optimization tax never stops. It just goes up.

Engineering time on optimization. How many hours per sprint does your team spend on query tuning, partition management, autovacuum configuration, index strategy reviews? Track it for a month. I've seen teams burning 15-20% of their engineering capacity on database maintenance work that wouldn't exist on a purpose-built system. Most of them didn't realize it until they measured. (If you're wondering whether your team has signs that tuning won't fix the problem, I wrote about that too.)

Instance cost escalation. The progression from db.r6g.xlarge to db.r6g.4xlarge to db.r6g.8xlarge happens gradually enough that nobody raises a flag. Each upgrade is individually justified. "We need more memory for the working set." "The CPU is pegged during dashboard queries." "Read replicas need a bigger instance too." The aggregate cost curve is something else entirely. I wrote a whole post about why vertical scaling buys time you can't afford. The short version: each instance upgrade gets you less headroom than the last.

Opportunity cost. Every sprint hour spent on database maintenance is a sprint hour not spent on product features. This compounds. The team spending 15% of engineering time on database operations ships 15% fewer features than the team that doesn't. Over 12 months, that gap is visible to customers.

Incident burden. Slow query alerts at 2 AM. Autovacuum blocking production writes. Replication lag during write spikes. These aren't catastrophic. They're erosive. They train the team to accept degraded baseline performance as normal. "Oh, the dashboard is slow on Mondays because of the weekly aggregation job." That sentence should make you uncomfortable.

Add these up over 12 months. That's your optimization tax. Compare it to the migration cost at your current data volume. The math almost always favors migrating now over migrating later. And the longer you wait, the more lopsided that comparison gets.

Why teams delay (and why those reasons stop holding)

I want to be fair about this. The reasons teams wait are legitimate. I've used most of them myself. But each one has a shelf life.

"We don't have time right now." This is the most common one, and it contains a cruel irony. Migration time increases with data volume. Delaying to find a better window means the work itself gets bigger. The window never gets better. The task gets worse. The team that "doesn't have time" for a weekend migration at 10M rows will somehow need to find time for an eight-week migration at 500M rows.

"The current optimizations are working." They're working now. Each optimization has a ceiling. When you hit it, you need the next one. The sequence is predictable: indexes, then partitioning, then read replicas, then instance upgrades, then custom vacuum tuning. Each step buys less time than the last. You're running up a down escalator.

"Migration is risky." This is the one that sounds the most reasonable and holds up the least. Migration from Postgres to TimescaleDB is lower risk than most database migrations because TimescaleDB is Postgres. Same SQL. Same wire protocol. Same drivers. Same pg_dump. Your application code changes are minimal. The risk profile is closer to "adding an extension" than "switching databases." You're not leaving Postgres. You're giving it better tools.

-- Your existing queries still work. This isn't a rewrite.
SELECT time_bucket('1 hour', ts) AS hour,
       avg(temperature),
       max(temperature)
FROM sensor_data
WHERE ts > now() - interval '7 days'
GROUP BY hour
ORDER BY hour;

"We'd need to convince stakeholders." Quantify your optimization tax. That's the whole pitch. "We stop spending X hours per month on database maintenance and reclaim that for product work." If your team is spending two days per sprint on database performance, that's roughly 20% of engineering capacity. Put a dollar figure on it. The conversation gets short.

Why migration risk is lower than you think

This is the part where I'm supposed to list seven migration steps and make you feel calm about them. I'll skip the list and give you the summary instead: install the extension, create hypertables, backfill data, configure compression and retention, update connection strings, validate. The migration guide walks through each one. The live migration tool handles the hard parts at scale.

The core point is this: the steps are the same whether you have 10M rows or 500M rows. What changes is the logistics around them. At 10M rows, the backfill is a single INSERT...SELECT that finishes while you're refilling your coffee. At 500M rows, it's a parallel COPY job that runs for days and needs monitoring, dual-write strategies, and a rollback plan.

The steps don't scale. The logistics do. And that's exactly why doing it earlier is the move.

The compound benefit of migrating early

Everything above has been about the cost of staying. But there's a positive version of this story too.

You skip the treadmill entirely. The team that migrates at 10M rows never learns what autovacuum tuning feels like at 500M rows. They never have the "should we add ClickHouse?" meeting. They never build the CDC pipeline. They never debug replication lag during write spikes. Those problems simply don't exist in their world. That's the real dividend.

Compression from the start. TimescaleDB's native compression typically achieves 90-95% compression ratios on time-series data. One of our customers, Latitude, saves $12,000 per month on database costs from compression alone. Your storage costs grow at 5-10% of the raw data rate instead of 100%. Over 12 months on a high-ingest workload, that's a line item your finance team will notice.

Fast dashboards without the custom plumbing. Continuous aggregates give you incrementally-updating materialized views that combine precomputed rollups with the newest raw data. FlightAware went from 6.4-second query times to 30 milliseconds. Your dashboards are fast from day one, not "fast after we spent two weeks building and maintaining custom materialized view refresh jobs." Hypertables handle partitioning automatically in the background, so you also never write another CREATE TABLE sensor_data_2026_q2 PARTITION OF ... DDL statement. One less thing.

That's the real cost of waiting. The migration effort is the obvious part. The intermediate suffering you accumulate between now and when you eventually migrate anyway? That's the part nobody budgets for.

So, about that timing

The best time to migrate was when the table was small and the effort was trivial. If that window has passed, the second best time is now, before the data volume makes the migration larger and the optimization tax makes the ROI case embarrassingly obvious.

Every month of delay increases both costs: the migration gets bigger and the tax keeps compounding. The crossover point where "just optimize" costs more than "migrate and stop optimizing" is earlier than most teams think.

If you're reading this and nodding, you probably already know what you need to do. The question isn't whether to migrate. It's whether you do it this quarter while it's a task, or next year when it's a project.

Get started with the migration guide →

You added a read replica and something real happened. Dashboard queries stopped competing with ingestion. The primary's CPU dropped. Write latency came down. You watched the metrics and thought: that worked.

It did work. For a while.

Now the replica is lagging during write peaks. The primary is accumulating WAL because the replica can't keep up. Write latency is climbing again, despite the replica handling every read you can throw at it. You're managing two Postgres instances instead of one, and the problem you solved is back.

Read replicas are a real solution to a real problem. You did the right thing. The right thing ran out.

What you have now is a write bottleneck. Read replicas solve a read bottleneck. These sound similar. The mechanics are completely different.

What read replicas actually solve

Let's be fair about this. The win is real.

Read replicas fix resource contention on the primary. When expensive analytical queries run on the same instance that handles ingestion, they compete directly for CPU, I/O, and buffer cache. A dashboard query scanning 200M rows is doing real work. It blocks autovacuum, slows the write path, and evicts hot data from shared buffers. Moving that query to a replica removes it from the primary entirely.

The immediate result: primary CPU drops, write latency improves, buffer cache stops getting thrashed by analytical scans. For a system where reads were the dominant load, this can be transformative. A SaaS application with thousands of concurrent users reading data that a small number of writers produce? Read replicas are the right call, and they solve the problem completely.

The mechanism is streaming replication. WAL generated by the primary gets shipped to the replica and applied in order. The replica stays current. Reads on the replica are reads against real data.

That's the win. The primary has less to do because reads moved elsewhere. Full stop.

What read replicas don't touch

Here's where it gets mechanical. Each of these is specific, and none of them changes when you add a replica.

The write path is identical. Every INSERT on the primary still goes through the full Postgres write path: heap tuple with MVCC header, B-tree index insertions for every index on the table, WAL record generation for heap and indexes, autovacuum triggered by insert volume for freezing. Nothing about streaming replication changes any of this. The replica receives the WAL and applies it. The WAL is generated at the same rate it always was, with the same per-row overhead.

Write amplification at 50K inserts/sec with five indexes is still 300K write operations per second on the primary. Adding a replica doesn't change that number. It adds the same write operations on the replica as well, applied from WAL.

Autovacuum still runs on the primary. The maintenance work that drives continuous autovacuum activity on high-insert tables (hint bit setting, freeze passes, dead tuple cleanup from aborted transactions) still happens on the primary. Routing reads to a replica doesn't reduce the insert rate that triggers autovacuum. It doesn't reduce dead tuple accumulation from aborted transactions. It doesn't change the XID freeze schedule.

Autovacuum workers still show up in pg_stat_activity at 3am. They still compete with writes during peaks. The tuning conversation still happens every quarter.

WAL volume increases. Here's the counterintuitive part. Adding replicas increases the total WAL-related workload, not decreases it. The primary now has to ship WAL to every replica in addition to writing it locally. At 50-100MB/sec sustained WAL generation, that's 50-100MB/sec of outbound data per replica. With two replicas, you're managing twice that outbound load.

This is usually invisible when replicas are keeping up. It becomes visible the moment a replica falls behind.

The replica lag problem

This is where things get interesting. And by interesting, I mean expensive.

Streaming replication works by applying WAL on the replica fast enough to stay current. Under normal conditions this is fine. WAL apply is sequential and fast. Replicas keep up.

High sustained write volume changes this. At 50-100MB/sec of WAL generation, the replica is continuously applying writes. If the replica hits any slowdown (a large analytical query competing for I/O, a vacuum cycle, a momentary resource spike) it falls behind. A small lag at high WAL volume becomes a large lag fast.

Here's the self-reinforcing part, and it's the thing most people don't see coming.

A lagging replica can't be used for real-time dashboards. So reads start routing back to the primary. The primary is handling reads again. More resource contention on the primary. Write latency climbs. You're back where you started, except now you're also managing a lagging replica.

Meanwhile, the primary has to retain unprocessed WAL in pg_wal until the replica catches up. The further behind the replica falls, the more disk the primary uses holding WAL. On a system generating 100MB/sec of WAL, a replica that falls 30 minutes behind represents 180GB of retained WAL on the primary. That's not a theoretical edge case. That's a disk alert at 4am.

max_wal_size and checkpoint frequency become critical tuning parameters that they weren't before. Another surface to monitor. Another thing to tune. Another runbook entry.

The operational load nobody budgets for

Adding replicas means running multiple Postgres instances. The cost of that gets absorbed into engineering time, and it's rarely accounted for when the decision is made.

Each replica needs its own monitoring: replication lag alerts, vacuum state, connection pooling, failover procedures. pg_stat_replication on the primary becomes a permanent fixture in the ops dashboard. Lag thresholds need calibration. Alerts need tuning to avoid false positives during expected spikes.

Connection routing adds its own complexity. pgbouncer or pgpool sitting in front of the cluster, routing reads to replicas and writes to the primary. The routing layer needs its own monitoring, and it has its own failure modes. A misconfigured connection router that sends writes to a replica produces cryptic application errors that take a while to diagnose. That incident happens once on every team that adds replicas. (Ask me how I know.)

Schema migrations now touch multiple instances. An ALTER TABLE on the primary replicates to replicas, but the timing matters. Long-running schema changes that lock tables on the primary lock them on replicas too, which means reads route back to the primary, which defeats the purpose of having replicas in the first place.

New engineers need to understand the topology. Which instance handles writes. Which handles reads. What happens when a replica lags. When to promote a replica. The replica architecture that seemed like a contained infrastructure change has tendrils into application code, deployment procedures, and incident response.

None of this is unusual or unreasonable. It's the actual cost of the solution.

Isolation vs. solution

Step back from the mechanics for a second.

There are two different things that can happen when you add capacity to a struggling system. You can solve the problem, meaning the root cause goes away and performance improves permanently. Or you can isolate the problem, meaning you move it further from the things it was affecting. You buy time without changing trajectory.

Read replicas are isolation.

The write bottleneck on the primary doesn't go away. The autovacuum tax doesn't go away. The WAL volume doesn't decrease. What changes is that reads no longer compete with these things. The primary's problems are contained to the primary.

That's not nothing. Isolation buys real headroom, and it's the right call in the right situation. A system where reads actually were the bottleneck is solved, not just isolated. The problem class matters.

For continuous high-frequency ingestion, the bottleneck isn't reads stealing write resources. It's write overhead at the storage level: MVCC headers on every tuple, B-tree index maintenance on every insert, WAL records per row per index, autovacuum competing for the same I/O budget. Routing reads elsewhere reduces resource contention on the primary but doesn't change the write overhead mechanics at all.

Six months after adding replicas, the primary is slower than it was when you added them. The data volume grew. The write overhead grew with it. The ceiling you isolated is now back in frame.

What actually addresses the write bottleneck

The write bottleneck in this workload pattern is architectural. It's in the storage model: per-row MVCC overhead, per-insert WAL records, and the maintenance machinery built around row-level concurrency management. These are fixed costs baked into vanilla Postgres heap storage. They don't go away through replication topology changes.

What changes them is a different storage model.

Columnar storage batches 1,000 rows into compressed segments before writing. Instead of 1,000 individual heap inserts, each generating its own WAL record plus WAL records for every index entry, you get one segment write and one WAL record. The per-row MVCC headers that added 23 bytes to every tuple get amortized across the batch. Autovacuum pressure drops because the storage engine isn't creating row-level churn that needs cleaning up.

Do the math on WAL alone. At 100K inserts/sec in vanilla Postgres with five indexes, you're generating a WAL record for each heap tuple and each index insertion: roughly 600K WAL entries per second. Columnar batching at 1,000 rows per segment reduces that to ~600 segment-level WAL entries per second. WAL volume goes from 50-100MB/sec to roughly 5-15MB/sec. Replica lag stops being a crisis because replicas can apply WAL at the rate it arrives. The primary stops retaining large WAL backlogs. The monitoring surface shrinks. The replication architecture that was struggling to keep up suddenly has room to breathe.

The replicas don't go away. They just stop being the bottleneck.

Where this leaves you

Read replicas were the right call. The write performance improvement was real. The isolation of read traffic from ingestion was real. You did the correct thing given what you knew.

What it doesn't do is change the underlying write cost. That cost is in the storage model, and it scales with data volume. Adding replicas doesn't touch it. Tuning replication doesn't touch it. The ceiling that read replicas pushed into the background is still there, and it moves closer as ingestion continues.

The write bottleneck gets solved at the storage level or it doesn't get solved. Replication topology is orthogonal to that problem.

If continuous high-frequency ingestion describes what you're running and you're already managing replica lag, the full picture of the optimization treadmill covers what's left on the path and what the off-ramp looks like.

MongoDB gets a bad reputation in certain engineering circles that it doesn't entirely deserve. It ships fast. Schema flexibility is real. The developer experience for document-shaped data is good. A lot of teams made a reasonable call when they chose it.

But there's a version of this story that ends badly, and it follows a recognizable pattern. The team picks MongoDB for a new system. The system works. Then the data starts looking less like documents and more like a stream of timestamped events. Queries start filtering by time range. Write volume climbs. Performance degrades in ways that feel familiar if you've read about this problem, and deeply confusing if you haven't.

This post isn't here to relitigate the MongoDB decision. It's here to help you figure out whether the pain you're feeling is a MongoDB problem, a document database problem, or a workload problem that would follow you to Postgres.

The answer matters because the fix is different in each case.

What MongoDB is actually good at

Flexible schema for variable data that's actually variable. Product catalogs where every SKU has different attributes. User profiles where fields vary by account type. Content management where article structure differs by category. These are real document shapes, and MongoDB handles them without the ceremony Postgres requires.

Rapid iteration without migration overhead. Early-stage products change their data model constantly. In Postgres, every schema change is an ALTER TABLE. In MongoDB, you just write different fields. For teams that are still figuring out the shape of their data, this is a real advantage.

Nested and hierarchical data. Some data is naturally a tree. A purchase order with line items with sub-components. A configuration object with nested sections. Postgres can model this with JSONB, but MongoDB's native document model fits it more naturally and queries it more cleanly.

Horizontal scaling for document reads. MongoDB's sharding model was designed for document workloads. For read-heavy document access at scale, it's a mature and well-understood architecture.

These aren't consolation prizes. They're real reasons MongoDB is the right choice for a lot of workloads.

The trouble starts when the data changes shape.

What time-series data actually looks like

Time-series data has a specific shape, and it's not a document shape. Every row is a measurement. It has a timestamp, a source identifier, and a value or set of values. The schema doesn't vary between rows. There's nothing hierarchical about it. The document model isn't adding anything.

What time-series data has instead: enormous volume, strict ordering requirements, queries that almost always filter by time range, and retention policies that drop entire time windows at once.

A wind turbine sensor reporting every five seconds doesn't produce documents. It produces a flat stream of readings: timestamp, sensor ID, RPM, temperature, vibration. A financial trade feed isn't a document store. It's a sequence of immutable events. An APM platform collecting metrics from a distributed system is generating hundreds of thousands of measurements per second, all with the same shape.

The test is simple. Look at your most-written collection. Does each document have a different structure? Or does every document look essentially the same, with a timestamp and some measurements?

If it's the latter, you're storing time-series data in a document database, and the document model is providing zero value while the storage engine works against you.

Where MongoDB struggles with this workload

WiredTiger (MongoDB's default storage engine) uses a B-tree structure optimized for a workload that includes updates to existing documents. For high-frequency append-only writes, it faces a fundamental mismatch. Consider a single sensor reading: one document insert triggers a write to the primary collection, a write to the oplog, and a separate B-tree update for every index on that collection. Three indexes means five writes for one data point. At 10,000 inserts per second, that's 50,000 storage operations per second before you've run a single query. The engine was designed for mixed read-write workloads with in-place updates, not an endless append stream where no document is ever modified after creation.

MongoDB has no native time-based partitioning. Postgres has declarative range partitioning. TimescaleDB automates it entirely with hypertables. MongoDB has no equivalent primitive. Teams end up implementing time-based collection bucketing manually: separate collections per day or week, application-level routing logic, custom cleanup scripts. It works, but it's the same operational burden as manual Postgres partitioning, without the tooling ecosystem that exists on the Postgres side.

MongoDB's aggregation pipeline is expressive. But for time-series workloads, the queries that matter are time-range aggregations: hourly averages, daily maximums, week-over-week comparisons. These queries scan large volumes of documents and aggregate across fields. Without columnar storage and purpose-built time-series compression, performance degrades with data volume in the same way it does in vanilla Postgres.

MongoDB did add a native time-series collection type in 5.0. It's a real improvement for simple append-only use cases. But it doesn't support secondary indexes the same way regular collections do, restricts certain aggregation stages and update operations, and is still relatively new compared to the Postgres ecosystem. Worth knowing about. Not a full answer.

Why moving to vanilla Postgres isn't automatically the fix

This is the section most competitive content skips entirely. If you're evaluating a migration, you deserve the full picture.

If the workload is continuous high-frequency time-series ingestion with long retention and operational query requirements, vanilla Postgres has its own version of this problem. The MVCC overhead, write amplification, autovacuum contention, and index maintenance costs that create the Optimization Treadmill exist in Postgres too. The storage model is different from MongoDB's, but the outcome at scale is the same: performance degrades with data volume, maintenance overhead accumulates, and each optimization cycle buys time without changing the trajectory.

Moving from MongoDB to vanilla Postgres solves the schema flexibility problem (you probably don't need it for this workload anyway). You get a mature partitioning ecosystem, a better query planner, and a richer extension ecosystem. These are real improvements.

It doesn't solve the core time-series storage problem, because that problem lives in the storage model, not the database brand.

The question isn't MongoDB vs. Postgres. It's document store vs. purpose-built time-series storage. That's the actual axis the decision should sit on.

The decision framework

Your data is actually documents. Variable schema, nested structures, hierarchical relationships, read-heavy access patterns. MongoDB is the right tool. The pain you're feeling is probably a schema design or indexing problem, not a fundamental architectural mismatch. Fix the schema.

Your data is time-series but volume is modest. Sub-10K inserts per second, retention under 90 days, no hard operational latency requirements on the full retention window. Vanilla Postgres with good partitioning and indexing handles this fine. The Optimization Treadmill exists, but the ceiling is far enough away that standard tuning keeps you ahead of it. Move to Postgres, implement partitioning early, and monitor the warning signs.

Your data is time-series at sustained high volume. Continuous ingestion, long retention, operational query requirements, growing data volume. This is the workload that breaks both MongoDB and vanilla Postgres through the same class of mechanisms. Purpose-built time-series storage on Postgres (same SQL, same wire protocol, same tooling) is the right answer. Migration from MongoDB to TimescaleDB follows a well-documented path: you keep everything Postgres-compatible and gain the storage architecture that matches the workload.

What to do next

MongoDB didn't fail you if you're reading this. Your workload evolved past what document storage was designed for. That's a different thing.

Most database choices are right at the time they're made and wrong eighteen months later when the system looks nothing like it did at launch. Sensor data that started as a feature became the core product. The document store that handled early prototyping became the production system for a time-series pipeline.

The question now is whether the fix is tuning, migration, or architecture. The framework above gives you a clear read on which one applies. If it's architecture, the good news is that moving from MongoDB to a Postgres-compatible time-series database is less disruptive than it sounds. Your application SQL stays the same. Your tooling stays the same. The storage engine underneath is the thing that changes.

That's the right scope for the change. Not the whole stack. Just the part that was always wrong for this workload.

Read the full technical breakdown of why vanilla Postgres hits these limits, or start a Tiger Cloud trial and see how TimescaleDB handles your workload directly.

You ran the benchmark. 80,000 inserts per second. The database handled it clean, latency stayed flat, no alarms. You shipped with confidence.

Three months later, p95 write latency is creeping. Six months later, autovacuum is in your top processes by CPU. Nine months later, you're rebuilding indexes on a table that's crossed 400 million rows.

The benchmark wasn't wrong. The question it answered just wasn't the right one.

Peak throughput tells you what the database can do in a sprint. Production asks what it can do running forever. Those are different questions with different answers, and most teams only ask the first one.

The number that actually matters is the sustained throughput ceiling: the write rate at which all of the database's maintenance processes (autovacuum, checkpointing, WAL archiving, replication) can keep up indefinitely. It's always lower than peak throughput. It drops over time as data volume grows. And almost nobody measures it.

What benchmarks actually measure

A typical load test runs for minutes. Sometimes an hour if you're thorough. It hits the database hard, measures throughput and latency, and stops. During that window, the buffer cache is warm from the test setup. Autovacuum hasn't had time to accumulate a backlog. WAL hasn't been generating for 72 hours straight. The indexes are fresh. The table fits mostly in memory.

These are ideal conditions. Not because anyone cheated. That's just what a bounded test looks like. The database performs brilliantly under bounded load because its maintenance subsystems haven't been outrun yet.

Production is unbounded. The data keeps arriving after the benchmark ends. Autovacuum runs against a table that grows every hour. The buffer cache works against a dataset that expands past RAM over weeks. The indexes that fit in memory at 50 million rows don't fit at 500 million. The checkpoint cycle that completed cleanly at low data volume starts competing with writes as WAL volume climbs.

The specific ways sustained load differs from peak load

There are four concrete mechanisms at work here. All four run simultaneously in production. None of them show up in a benchmark.

Your hot data stops being hot

At launch, your hot data fits in shared_buffers and the OS page cache. Read performance is largely a RAM question. As data volume grows past available RAM, cache hit rates fall. Queries that returned in milliseconds start hitting disk. The degradation is slow enough that it looks like a query regression, not a growth problem, and that's what makes it dangerous. You'll spend a sprint chasing query plans and index strategies before someone checks pg_statio_user_tables and realizes the hit rate has been sliding since month four. The latency change wasn't a code problem. It was a ratio problem.

Autovacuum falls behind and can't catch up

A benchmark run doesn't give autovacuum time to fall behind. Production does.

At high sustained insert rates, autovacuum fires continuously. During write peaks, it falls behind. The backlog accumulates. Bloat builds. By the time monitoring catches it, the table has weeks of accumulated dead tuples and hint-bit work queued up.

Here's the part that really gets you: clearing the backlog requires running autovacuum harder, which competes with writes, which slows ingestion. The fix and the problem share the same resource pool. You're asking the database to clean up faster while also writing faster, and there's only so much I/O to go around.

Indexes rot

Fresh B-tree indexes on a small table are compact and cache-friendly. The same indexes a year later on a table with a billion rows are fragmented, partially sparse from the hot-right-edge problem on timestamp columns, and too large to stay in cache.

Traversal costs go up. Page splits happen more often. The 10x read improvement you got from careful indexing in the first month erodes slowly, then faster. You'll REINDEX and get performance back for a while, but the table is still growing. The next degradation cycle is already in progress.

WAL never stops arriving

WAL volume scales directly with insert rate. At sustained high rates, WAL generation is constant. Replicas that keep up at launch start falling behind as write volume grows. The primary retains unprocessed WAL. Disk fills. And the replica needs to process a growing backlog while new WAL keeps arriving, which means there's no quiet period to catch up. If you've ever watched pg_stat_replication and seen replay_lag tick steadily upward with no sign of plateauing, you know exactly how this ends.

Each of these mechanisms is invisible in a benchmark. In production, they compound.

The number you should actually be looking at

So how do you actually find the sustained throughput ceiling?

You can estimate it. Look at autovacuum activity under current load: is it finishing cycles or perpetually falling behind? Check pg_stat_bgwriter for checkpoint pressure. Watch pg_wal directory size trends. Plot the ratio of index size to table size over time. These aren't exotic metrics. They're already in Postgres. Most teams aren't watching them together.

The leading indicators of a sustained throughput ceiling: autovacuum consistently showing in pg_stat_activity, checkpoint completion times trending up, replica lag growing during write peaks, n_dead_tup climbing faster than vacuum_count is cleaning.

None of these show up in a benchmark. All of them show up in production, usually together, usually around month six or nine.

Why this question is structurally hard to ask

Smart teams miss this. The reasons are structural.

Benchmarks have a natural stopping point. Load tests end. Sustained load doesn't have a natural evaluation moment until something breaks. There's no "sustained throughput benchmark" in most team playbooks because the concept doesn't have a clean boundary. When do you declare the test over?

The degradation timeline is also longer than most planning cycles. Indexing starts showing stress at 300 million rows. Partitioning gets complicated at 500+ partitions. WAL volume becomes a crisis when replica lag crosses a threshold that trips an alert. These events are six to eighteen months apart. The engineer who ran the initial benchmark often isn't the one debugging the production incident.

Then there's the procurement problem. Peak throughput is a good number for architecture decisions. "This database handles 80K inserts per second" is a clean, defensible statement. "This database handles 80K inserts per second now, but that number will effectively be lower in eight months as the buffer cache hit rate falls and autovacuum starts competing for I/O" is harder to put in a slide. (Both statements are true. Only one of them gets you budget approval.)

And most capacity planning frameworks are built around static estimates. How many users, how many requests, how much storage. Sustained throughput degradation is a dynamic problem. The ceiling moves as the system runs. That doesn't fit neatly into a capacity model built for stable workloads.

This adds up to something bigger than individual teams making mistakes. The entire way the industry evaluates databases is optimized for procurement, not production. Vendor benchmarks measure peak throughput because it's the largest number. Load testing frameworks default to bounded runs because unbounded runs don't have a natural end state. Capacity planning templates assume static ceilings because dynamic ceilings are harder to model. Every layer of the evaluation stack is designed to produce a number that looks good in a slide deck. None of it answers the question you'll actually need answered in month twelve.

So if the standard evaluation framework is structurally set up to miss this, what does a better one look like?

What the right benchmark looks like

Run the load test for longer. Hours, not minutes. Watch what happens to autovacuum, not just query latency.

Start the test with a table that already has data in it, sized to your 12-month projection. A benchmark on an empty table tells you about cold start performance. It tells you almost nothing about what the system looks like after a year of continuous ingestion.

Measure these things during the test:

• pg_stat_bgwriter: checkpoint frequency and write volume
• pg_stat_activity: autovacuum activity
• Replica lag if you're running replicas
• pg_stat_wal: WAL generation rate
• Index size relative to table size

Repeat the test with 3x the data volume. If performance drops more than linearly, you've found where the architecture starts to strain. That's the number you want before you ship, not after.

The test that catches the Optimization Treadmill is a test that asks: what happens when this runs for a year? You can simulate that in a day if you load the data upfront and run the benchmark against a realistic data volume.

The benchmark question and the architecture question

If your system has the six workload characteristics (continuous ingestion, time-series access patterns, append-only data, long retention, operational query requirements, sustained growth), the sustained throughput ceiling is structural. Better benchmarking tells you earlier where the ceiling is, but it won't raise it.

Benchmarking tells you how fast the ceiling approaches. Architecture determines where it sits.

Teams that run good sustained-load benchmarks early find out at 30 million rows that they're on the Optimization Treadmill. Teams that only run peak throughput benchmarks find out at 800 million rows. The underlying architectural problem is identical in both cases. The migration cost is not.

Ask the right question before you ship

Peak throughput is a useful number. It tells you whether the hardware can keep up with the write rate at a point in time. Worth knowing.

It just doesn't tell you whether the maintenance processes can keep up with that write rate indefinitely, as data volume grows and the vacuum backlog and WAL volume and cache pressure all grow with it.

The question nobody asks before shipping is usually the one that generates the incident nine months later. Ask it now. Run the load test against a full-size dataset. Watch autovacuum, not just query latency. Track the ceiling as a moving target, not a static spec.

And if the benchmark reveals what the scoring framework already suggested, the cheapest architectural decision you'll make is the one you make before the table crosses 100 million rows.

Most engineers have a working mental model of MVCC. Readers don't block writers. Concurrent transactions see consistent snapshots. It's why Postgres handles mixed read/write workloads so well, and it's a genuine engineering achievement.

What's less obvious is that MVCC isn't free. Every row in every table carries its overhead. Not just rows that get updated. The system doesn't know at write time whether a row will ever be touched again, so it prepares for that possibility. Every time.

If you're running an IoT pipeline, a financial data feed, or an observability platform, most of your rows will never be updated. Sensor readings don't get corrected. Trade records are immutable. Log entries are permanent. You're writing append-only data into a system built to handle concurrent modification of shared rows, and you're paying the full price for that capability whether you use it or not.

This post breaks down exactly what that costs you: at the byte level, at the I/O level, and at the maintenance level.

What MVCC actually does (and why it's damn good)

Before MVCC, databases had two options: lock rows during reads so writers couldn't touch them, or lock rows during writes so readers couldn't see them. Either way, concurrent workloads serialized through lock contention. If you've ever worked with a database that does this, you know how painful it gets at scale.

MVCC solves the problem differently. When a row is updated, Postgres doesn't modify it in place. It writes a new version of the row and keeps the old version visible to transactions that started before the update. Each transaction sees a consistent snapshot of the database as of the moment it began. Readers and writers operate on different row versions simultaneously. No locking required.

For an e-commerce backend processing orders while users browse, a SaaS application handling concurrent sessions, or any system where multiple transactions touch the same rows, this is transformative. The PostgreSQL documentation puts it simply: reading never blocks writing and writing never blocks reading.

That's not a small thing. That's the reason Postgres can handle the concurrency patterns that would bring a lock-based system to its knees.

The cost of maintaining this guarantee is what the rest of this post is about.

The per-row overhead, in bytes

This is where most explanations go vague. Let's not do that.

Every heap tuple in Postgres carries a fixed 23-byte header before a single byte of your actual data gets written. Here's what's in it:

• t_xmin: the transaction ID that created this row (4 bytes)
• t_xmax: the transaction ID that deleted or updated it, zero if the row is live (4 bytes)
• t_cid: command ID within the transaction (4 bytes)
• t_ctid: physical location of this tuple or its newer version (6 bytes)
• t_infomask and t_infomask2: status flags for transaction visibility (4 bytes)
• t_hoff: offset to actual row data (1 byte)

These fields exist to answer one question: is this row visible to this transaction?

For a workload where rows are being updated and deleted concurrently, that question needs answering constantly. The 23 bytes are worth it.

For an append-only workload? t_xmax is zero for every live row and will stay zero. t_ctid points to itself because there's no newer version. The visibility question still gets asked, and the header still gets written, and the page still gets dirtied to set hint bits after the first read. But the answers are trivial every time. The mechanism is running in full for a case that never needed it.

Add alignment padding and a 4-byte ItemIdData pointer per tuple, and the true per-row overhead is closer to 28 to 30 bytes before your row data starts.

Let's make that concrete. At 50K inserts per second, that's 1.4 to 1.5 MB/sec of pure overhead headers. Per year: roughly 44 GB of header data for a workload that never updates a row.

That's not a rounding error.

What autovacuum is actually doing on your append-only table

Here’s what’s going to wrinkle your brain.

You think,  “Autovacuum cleans up dead tuples from updates and deletes. Append-only tables don't update or delete rows. Therefore, autovacuum shouldn't have much to do.”

That intuition is wrong in three specific ways.

Aborted transactions leave dead tuples. Not every INSERT commits. Connection drops, application errors, explicit rollbacks. These all leave tuple versions that need cleaning. If you're running high insert rates, you've got a steady trickle of aborted transactions even in perfectly healthy systems.

Hint bits require page dirtying. When a row is first read after being written, Postgres needs to check pg_xact to confirm the writing transaction committed. Once confirmed, it sets a hint bit in t_infomask to cache that result. Setting the hint bit dirties the page, which means writing it back to disk. On an append-only table with high read rates, hint bit setting is continuous background I/O on pages that will never change in any meaningful way. Welcome to your new normal.

Since PostgreSQL 13, insert volume alone triggers autovacuum. Not just dead tuples. Postgres needs to periodically freeze old transaction IDs to prevent XID wraparound, which is a hard limit built into the 32-bit transaction counter. At high insert rates, autovacuum fires continuously just to freeze tuples on tables with zero updates.

Go check autovacuum_count and vacuum_count on your busiest append-only partition. They're climbing whether or not n_dead_tup is.

The result: autovacuum workers show up in pg_stat_activity at all hours on tables that never see a single UPDATE. You tune autovacuum_vacuum_scale_factor and autovacuum_max_workers, and it helps at the margin. But what you're tuning is how the cleanup process competes with writes. Not why it needs to run at all.

The write amplification chain

Now let's connect all of this into the full cost picture.

A single 1 KB sensor reading doesn't write 1 KB. Here's what actually hits disk:

• 23-byte heap tuple header plus padding
• 1,024 bytes of your actual row data
• One entry per index, roughly 40 to 80 bytes each in B-tree leaf pages (five indexes = 200 to 400 bytes)
• One WAL record per heap insert, one per index insertion: approximately 1.2 KB total
• Periodically: an 8 KB full-page write after checkpoint for any newly dirtied page

Total actual I/O: 2.5 to 3.5 KB for 1 KB of logical data.

The MVCC header is the entry point for this entire chain. It's what requires the visibility tracking, the hint bit mechanism, the autovacuum sweep, and the WAL record structure that Postgres uses.

At 100K inserts per second, you're writing 250 to 350 MB/sec of actual I/O for 100 MB/sec of application data. The 3 to 5x write amplification ratio isn't configuration. It's the cost of MVCC applied to data that will never be updated.

Why you can't opt out

There's no per-table setting to disable MVCC. No append_only = true flag that strips the header and skips the visibility machinery. MVCC is not a feature you can turn off for specific tables. It's the storage model. Every heap tuple gets the header. Every insert goes through the same write path.

This isn't an oversight. It's an architectural decision with a clear rationale: the storage engine doesn't know at write time what future transactions will need to see. The consistency guarantee requires the mechanism to be universal.

For most workloads, this is the right tradeoff. The overhead is small relative to the value of the concurrency guarantee, and mixed read/write workloads on shared rows are exactly what Postgres is built for.

The overhead only becomes the dominant cost when the workload is append-only at high sustained rates. That's when you're paying the full price for a guarantee you never exercise.

What changes when the storage model changes

TimescaleDB's columnar storage (the Columnstore layer) addresses this at the architecture level, not the configuration level. Rather than writing one heap tuple per row, it batches up to 1,000 row versions per column into compressed arrays before writing to disk. The MVCC header overhead gets amortized across the batch. One write operation covers what would have been 1,000 individual heap tuple insertions.

The practical results: write amplification drops from 3 to 5x to near 1:1 for sustained append workloads. Autovacuum pressure drops proportionally because there's far less row-level churn to clean. WAL volume at 100K inserts/sec falls from 50 to 100 MB/sec to roughly 5 to 15 MB/sec. Replicas that previously fell behind during write peaks can keep up.

Everything else stays the same. Same SQL. Same wire protocol. Same extensions. Same tooling. The change is underneath, at the layer where MVCC overhead was accumulating.

The bottom line

MVCC is not a bug in Postgres. It's one of the reasons Postgres is the right choice for the majority of production workloads.

But if most of your rows are immutable after the insert commits, if your tables never see concurrent updates to the same rows, if autovacuum is running constantly on data you've never touched, you're running an append-only workload inside a concurrency model built for something else.

That's not misconfiguration. It's an architectural mismatch. The distinction matters because misconfiguration has a config fix. Architectural mismatch doesn't.

If high-frequency append-only ingestion describes what you're running, the full essay on the Optimization Treadmill covers what this costs across your entire stack, and what the path forward looks like.

Your system writes data constantly. Not in jobs. Not in batches. A stream that runs at 3am the same as it runs at 3pm. IoT sensors. Trade feeds. Metrics collectors. The data never stops.

For a while, Postgres handles it fine. Then you start noticing things. Autovacuum is always running. Write latency has a pattern you can't explain by traffic alone. Maintenance tasks that used to take minutes now take hours. And the really annoying part: nothing is misconfigured.

You check the usual suspects. Indexes are correct. Query plans look reasonable. Configs follow best practices. A colleague confirms the same.

The problem isn't a missing index or a bad query plan. The problem is that Postgres was designed with a quiet period baked into its assumptions. Your system eliminated that quiet period. Now you're paying for it.

What "breathing room" actually means in Postgres

Most database systems are designed around a workload shape that includes peaks and valleys. Peaks are when users are active. Valleys are when the database catches up.

Postgres maintenance is built around the valley.

Autovacuum runs more aggressively when the database is quiet. ANALYZE refreshes statistics without competing for I/O. Checkpoint cycles complete cleanly. WAL accumulation clears out. The buffer cache warms up on predictable patterns.

Batch ETL fits this model perfectly. A nightly job writes data for two hours. The database writes, then rests, then writes again. Maintenance runs in the gaps. Everything resets before the next cycle starts.

Continuous ingestion has no gaps. The window that used to be quiet at 2am is now the same as the window at 2pm. Every maintenance process that depends on quiet time now runs in direct competition with writes. All day. All night.

The maintenance competition problem

Three maintenance processes need quiet time and don't get it under continuous ingestion.

Autovacuum. Even on append-only tables, autovacuum fires continuously at high insert rates. Since PostgreSQL 13, inserts themselves trigger autovacuum to freeze tuples and update the visibility map. This isn't about dead tuples from updates or deletes. It's insert-driven vacuum, running because the data is arriving too fast for the system to catch up.

At 50K inserts/second, autovacuum never finishes a cycle before the next one starts. It competes for I/O with your writes. When it loses, bloat accumulates. When it wins, write latency spikes.

There's no configuration fix for this. You can tune autovacuum_vacuum_cost_delay and autovacuum_max_workers all day. What you're tuning is how autovacuum loses gracefully. Not how it stops competing.

Checkpoints. Postgres writes dirty pages to disk at checkpoint intervals. After a checkpoint completes, the first write to any previously-clean page triggers a full-page write to WAL (that's the full_page_writes mechanism, and it's on by default for good reason). At high insert rates, checkpoint cycles are constant. The full-page write burst that follows each one adds significant WAL volume on top of your baseline write load.

Batch systems checkpoint, rest, then return to normal. Continuous systems checkpoint and immediately start generating the next burst. There's no recovery window.

ANALYZE and statistics. Query planning accuracy depends on fresh statistics. On a billion-row table, ANALYZE is expensive. On a batch system, you schedule it after the load completes. On a continuous system, there is no "after." You run it during writes or you let statistics go stale. Stale statistics mean bad query plans. Bad query plans mean unexpected sequential scans at the worst possible time.

WAL as the throughput ceiling you can't tune past

This is the mechanical core of the problem.

Every insert generates WAL. Heap insert record, index insertion records for every index on the table, plus full-page writes after checkpoints. A single 1KB sensor reading with five indexes generates roughly 2.5-3.5KB of actual I/O once you account for the heap tuple, B-tree leaf page insertions, and WAL records. At 100K inserts/second, that puts sustained WAL throughput at 50-100MB/sec under normal conditions. After a checkpoint, it spikes higher because of full-page writes.

That's 3-6GB per minute. 180-360GB per hour. Just WAL.

WAL writes are sequential and synchronous by default. That's a hard ceiling on write throughput for a given storage configuration. You can raise the ceiling by buying faster storage. You can't eliminate it, because WAL is how Postgres guarantees durability. And you shouldn't want to eliminate it. Durability matters. But you should understand that your write throughput has a physical upper bound set by how fast your storage can absorb WAL, and continuous ingestion pushes against that bound constantly.

Here's where continuous ingestion and batch ETL diverge completely.

Batch ETL generates bursts of WAL followed by silence. The silence lets replicas catch up. A streaming replica can fall behind during a batch load and recover in the gap. Nobody notices because the gap is long enough.

Continuous ingestion generates WAL constantly. Replicas that fall slightly behind have no gap to recover in. They fall further behind. The primary retains unprocessed WAL in pg_wal, consuming disk. The further behind the replica gets, the more WAL it needs to process, and the more disk the primary holds. It's a feedback loop. The thing that causes the problem (WAL volume) is the same thing that prevents recovery (WAL volume).

Adding replicas makes it worse, not better. Each replica is another consumer that needs to keep up with the same WAL stream, and the primary holds WAL until the slowest one catches up.

The standard fix is more provisioned IOPS. It works for a while. Then data volume grows and you're having the same conversation again, just with bigger numbers on the invoice.

Why the standard toolkit doesn't solve this

Walk through each common response and you'll see exactly where it runs out.

More autovacuum workers. More workers means more I/O competition with writes, not less. You're distributing the problem across more processes. The aggregate I/O pressure is unchanged.

Aggressive autovacuum cost limits. You can configure vacuum to run faster and harder. It cleans up faster but hits writes harder. There's no setting that makes the competition disappear. You're choosing which process suffers.

More RAM. Bigger shared_buffers and page cache reduce physical reads. Write amplification is unchanged. WAL volume is unchanged. Autovacuum competition is unchanged. You bought better read performance for a write-bound problem.

Faster storage. Raises the WAL ceiling. Doesn't change the ratio of actual I/O to logical data. At 3-5x write amplification, faster storage lets you sustain a higher write rate before hitting the ceiling. But data volume grows, and the ceiling moves up proportionally.

Vertical scaling. Same as faster storage with more CPU. You've bought headroom measured in months. At the current data growth trajectory, that math doesn't improve over time.

Each of these is the right response to the symptom. None of them changes the underlying dynamic: continuous ingestion is in constant competition with the maintenance processes Postgres needs to stay healthy.

The workloads where this actually matters

Not every write-heavy system has this problem. Let's be precise.

The pattern shows up when three things are true at once: writes are continuous rather than bursty, data volume is growing on a sustained curve, and the database needs to stay queryable under latency requirements while ingestion is running.

Industrial IoT is the clearest example. A wind farm with 10,000 sensors reporting every five seconds generates roughly 2,000 inserts/second. That's modest by financial or observability standards, but it never pauses. The turbines don't stop overnight. Maintenance windows don't exist because the data source doesn't know what a maintenance window is.

Financial market data is the high-frequency version. Trade feeds run at hundreds of thousands of events per second during market hours. Pre-market and after-market data keeps coming. Systems that aggregate this data for risk and compliance queries need it available immediately, not at end of day.

Observability platforms are the distributed version. Metrics, traces, and logs from thousands of hosts. Each host generates data independently. The aggregate rate is enormous and constant.

What these have in common: the data source runs on its own schedule, completely independent of what the database needs. The wind turbine doesn't care that autovacuum is behind. The trading engine doesn't wait for a checkpoint to finish.

If your write pattern is bursty (user-driven traffic, nightly batch jobs, periodic syncs), you probably don't have this problem. The database gets its breathing room, maintenance catches up, and standard Postgres optimization works the way it's supposed to. The pattern described in this post shows up specifically when the gap disappears.

Recognizing the pattern early

The instinct when Postgres starts struggling under continuous ingestion is to tune harder. Add workers. Raise limits. Upgrade storage.

Those are correct responses for a database that has misconfiguration or a bad schema. Postgres is doing exactly what it was designed to do. The MVCC model, the WAL architecture, the maintenance scheduler: these are good design decisions for the workloads Postgres was built to handle. The system changed underneath it. That's not a criticism of the tool.

But continuous ingestion isn't a heavier version of batch ETL. It's a different workload class. The architectural assumptions underneath Postgres were built around a workload that breathes. Continuous ingestion doesn't breathe. And that distinction matters because it determines whether optimization will change your trajectory or just delay the same outcome.

Recognizing that early is worth a lot. At 50M rows, switching to a purpose-built architecture takes days. At 1B rows, it takes months. Every quarter you spend optimizing within the wrong architecture is a quarter where migration gets harder and the engineering team spends more time managing the database than building product.

If this sounds familiar, the full analysis covers the scoring framework and the mechanics behind why each optimization phase hits a ceiling. It's the same trajectory described here, zoomed out to show the complete path and where it leads.

Read the full analysis: Understanding Postgres Performance Limits for Analytics on Live Data →

I was doing some validation tests for an essay about the performance envelope for an IIoT PostgreSQL database and realized that measuring a database table is not as straightforward as I assumed it would be. 

The general idea was that I would insert IIoT data into a table and then measure the size and performance of the table as it grows. But how do you actually read the size of a table? What is performance? How can we quantify these values in a way that’s useful for us engineers?

Here’s what I did.

Table Size

There are two key measurements that define a table’s size: How many rows does it have and how much disk space does it occupy. 

Row Count

For small tables, this is straightforward:

SELECT COUNT(*) FROM <table_name>

However, that query requires scanning every row in the table. For typical IIoT tables, like the ones I was testing, that might be billions of rows and might take minutes to execute.

Instead there’s a much faster query:

SELECT reltuples::bigint AS row_count 
FROM pg_class 
WHERE relname = ‘<table_name>’

This is the row count that PostgreSQL uses for the query planner. It’s not guaranteed to match the row count exactly, because it’s not continuously updated, but it’s close enough and returns almost instantly.

Size on Disk

PostgreSQL stores table data across several components: the heap (the main table data), the indices, and TOAST storage (where large values get stashed). All three contribute to the table size and overall storage requirements as shown in the following image. 

Here’s the query I used to get the three separate components.

SELECT pg_relation_size(‘<table_name>') AS heap_size,
pg_indexes_size('<table_name>') AS indexes_size,
pg_table_size('<table_name>')
  - pg_relation_size('<table_name>') AS toast_size;

This will return the sizes in bytes, but you can also use the function pg_size_pretty() to get a more human readable output.

Ingest Capacity

Ingest capacity is critical to IIoT workflows, and it’s where a lot of systems run into serious trouble. How do you measure capacity? You can either get an approximate value from current ingest speeds, or push your database to the limit.

If You Already Have a Data Source Connected

If your data stream is already connected, you can look at how long ingests are taking and figure out the capacity from that.

This requires the built-in tool pg_stat_statements which is essential for any serious database. To enable it (it ships with PostgreSQL, so it’s always available) run the following query:

CREATE EXTENSION IF NOT EXISTS pg_stat_statements;

Once it’s running, it creates a table called pg_stat_statements that you can query for your INSERT performance:

SELECT 
    query,
    calls,
    rows,
    total_exec_time / 1000 AS total_time_sec,
    mean_exec_time AS avg_ms_per_call,
    rows / NULLIF(calls, 0) AS avg_rows_per_call,
    rows / NULLIF(total_exec_time / 1000, 0) AS rows_per_sec
FROM pg_stat_statements
WHERE query ILIKE '%INSERT%<table_name>%'
ORDER BY total_exec_time DESC;

This gives you a picture of real ingest performance based on what your application is actually doing. You'll see how many rows each call inserts (obviously you’re batching), the average time per call, and a rough rows-per-second figure. The time it takes to insert a batch divided by the period of your desired insertions gives you a rough estimate of how much ingest capacity you’re using.

You can reset the stats whenever you want for a fresh baseline:

SELECT pg_stat_statements_reset();

By measuring this as your table grows, you’ll get a good sense of how your ingest capacity is evolving and you’ll be able to deal with it well before it becomes an issue.

The actual ingest capacity

If you don’t mind really pushing your table to its limits (and maybe breaking it), you can try to ingest as much as possible and see if the database keeps up. I wrote a full walkthrough for this, including the SQL for generating realistic IIoT data and a scripted test loop, in How to Break Your PostgreSQL IIoT Database and Learn Something in the Process. 

Query Speed

Query speed is the most obvious metric for a database, as it affects everyone using the data. However, I found it to be one of the most difficult to generalize. Every application will have specific queries that are important, and different definitions of what is ‘fast enough’.  It’s also something that tends to degrade over time and only become an issue well into the life of the table.

For queries you’re already running

If you already have dashboards running, or your analysis workflow in place, you can again use pg_stat_statements. Here's how to pull information for the 20 slowest queries:

SELECT
    query,
    calls,
    rows,
    mean_exec_time AS avg_ms,
    total_exec_time / 1000 AS total_time_sec,
    stddev_exec_time AS stddev_ms,
    rows / NULLIF(calls, 0) AS avg_rows_returned
FROM pg_stat_statements
WHERE query ILIKE '%SELECT%<table_name>%'
ORDER BY total_exec_time DESC
LIMIT 20;

For more general queries

IIoT queries tend to fall into two categories: wide (what is the state of all devices at a specific time?) and deep (what is the history of a particular device?). By running at least one example from each type, you’ll get a sense of how quickly these types of queries will return.

Generic Wide Query

SELECT DISTINCT ON (tag_id) 
  tag_id, 
  time, 
  value
FROM <table_name>
ORDER BY tag_id, time DESC
LIMIT 100

This returns the most recent value from 100 tags.

Generic Deep Query

SELECT 
    tag_id,
    DATE_TRUNC('hour',time) as hour,
    AVG(value) as hourly_average
FROM <table_name>
WHERE tag_id = <specific tag_id>
GROUP BY DATE_TRUNC('hour',time)
ORDER BY hour DESC
LIMIT 100

This returns the past 100 hourly averages from one specific tag.

Putting It All Together

The real value comes from combining these measurements as the table grows. Here's the general approach I followed for my essay:

1. Create a simple IIoT table schema and common index.
2. Measure table size (rows + disk space), query times, and ingest time for a couple standard batches.
3. Insert many batches as fast as possible so the table grows quickly.
4. Repeat steps 2 and 3 until some predefined limit (usually disk space or query time)

If I was instead using a real production system, I would rely more on pg_stat_statements to track query and ingest rates. Doing this every day when the system is new and then a weekly check will ensure you know exactly how your table is evolving.

The pattern is familiar. A query is slow. You run EXPLAIN and see a sequential scan. You add an index. The query drops from seconds to milliseconds.

You do this a dozen times over two years and it works every time.

Then write latency starts climbing and you can't figure out why. The queries are fast. The schema looks clean. Nothing is obviously wrong.

Pull up pg_stat_user_indexes. Count your indexes. Now think about what happens at the storage layer every time a row lands in that table.

The indexes didn't stop helping reads. They started hurting writes. Every index is a flat tax on every insert: one extra write operation per row, every time, no exceptions. At low ingestion rates, the tax is invisible. At high ingestion rates, it's the whole problem.

What actually happens when you insert a row

No handwaving here. Let's walk through the mechanics.

A single INSERT into a table with five indexes doesn't write once. It writes six times: one heap tuple to the table's data pages, and one B-tree leaf page insertion per index. Each index insertion traverses the B-tree from root to leaf, finds the correct position, and writes the new entry. If the target leaf page is full, it splits. A split can cascade up the tree.

Then there's WAL. One heap insert record. Five index insertion records. If it's the first modification to a page since the last checkpoint, Postgres writes a full 8 KB page image on top of all that.

At one insert per second, this is completely invisible. At 50,000 inserts per second with five indexes, you're looking at 300,000 write operations per second. Not 50,000. Six times the logical write rate, minimum.

That's your write amplification number. For this table configuration: 6x. More indexes, higher multiplier.

The math that makes this concrete

Take a table with five indexes and a 1 KB row. The heap tuple costs 23 bytes of header plus your 1,024 bytes of row data plus a 4-byte ItemIdData pointer. Each of the five B-tree index entries adds roughly 40 to 80 bytes. Then WAL: approximately 1.2 KB covering the heap insert plus all five index insertions. Add it up and you're writing roughly 2.5 to 3.5 KB for every 1 KB of logical data.

At 50K inserts/sec, that's 125 to 175 MB/sec of actual I/O for 50 MB/sec of application data. The index tax at work.

Now add two more indexes because a couple of new dashboard queries need covering indexes. You're at seven. The multiplier goes up. The WAL volume goes up. Write latency goes up. Autovacuum has more index pages to scan and maintain.

The relationship is linear per index, but the effect compounds with ingestion rate. At 1K inserts/sec, two extra indexes barely register. At 100K inserts/sec, they're a real cost.

Here's what the math looks like across different configurations:

The numbers are approximate (real-world I/O depends on page splits, full-page writes, and your specific index types), but the pattern is clear. Each additional index is a flat tax on every insert. The tax rate doesn't change. The bill does.

Why timestamp indexes have a specific problem

B-tree behavior for monotonically increasing keys is worse than for random keys. And most time-series tables insert in timestamp order.

With a random key distribution, new inserts scatter across the B-tree's leaf layer. Any given leaf page gets a roughly even share of new entries. Splits happen, but they're spread out.

With a timestamp key, every insert goes to the rightmost leaf page. The same page, over and over. That page fills up and splits. The new rightmost page fills up and splits. This is called a "hot right edge," and it means B-tree index maintenance for timestamp columns involves constant page splits concentrated in one area of the tree.

The old leaf pages that were once the rightmost page sit mostly empty but remain allocated. Index size grows faster than data size. The index bloat you see in pg_stat_user_indexes is a direct result of this pattern, not random fragmentation.

For non-timestamp indexes on the same table (device ID, metric name, sensor type), inserts scatter across the tree instead, which means random I/O rather than sequential. So you get two different flavors of write overhead hitting the same table simultaneously: constant splits on the timestamp index, random I/O on everything else.

The feedback loop

All of that overhead is manageable if it stays constant. The problem is that it doesn't. It self-reinforces.

You add indexes to fix slow queries. Write amplification increases. Write latency creeps up. Bloat accumulates faster. Autovacuum fires more frequently and has more index pages to clean. Autovacuum competes with your writes for I/O bandwidth. Write latency climbs higher.

Slower writes mean rows sit in the buffer longer. Buffer pressure increases. The query performance you were trying to protect starts degrading anyway, now from I/O contention rather than missing indexes.

The response is usually to check query plans again. Some queries have gone back to sequential scans because statistics are stale or the planner is making different cost estimates under load. So you add another index. The cycle repeats.

This loop runs slowly enough that the connection between each index addition and the eventual write degradation is hard to see. Six months can pass between the two events. By that point, you've forgotten which indexes were added and why, and the symptom looks like a completely different problem.

The diagnostic questions

Before adding the next index, ask these:

How many indexes does this table already have? Pull pg_stat_user_indexes and look at idx_scan. Indexes with low scan counts are paying full write overhead for queries that run rarely or never.

What's the actual write rate on this table? Low ingestion rate tables can carry many indexes without much penalty. The math only gets ugly at high sustained rates. If you're inserting 100 rows/sec, ten indexes are probably fine. If you're inserting 50K rows/sec, every index counts.

Is the slow query a read problem or a write problem? Adding an index to fix a slow query while write amplification is already the bottleneck treats the symptom and makes the underlying condition worse.

What's the index bloat trend? Growing index size relative to table size, especially on timestamp columns, is the fingerprint of the hot right edge problem. You can measure it directly with pgstattuple or by comparing pg_relation_size for the index against the table over time.

Could a different query shape eliminate the need for this index? Sometimes the answer is restructuring the query or adjusting the access pattern, not adding another index to support the query as written.

When you're past the point where index pruning helps

You can drop indexes with low idx_scan counts. You can consolidate partial indexes. You can audit and remove redundant coverage. All of that is correct and worth doing.

But for a table with continuous high-frequency ingestion, even a minimal index set still generates substantial write amplification. Three carefully chosen indexes on a 50K inserts/sec table is still 200K write operations per second. WAL volume is still 3–5x logical data volume. Autovacuum is still competing for I/O.

Index pruning buys back headroom. It doesn't change the architecture.

The write amplification problem for this class of workload is in the storage model itself. Row-based heap storage with B-tree indexes is how Postgres handles every table. It's the right design for most workloads. For sustained high-frequency, append-heavy ingestion, the overhead is intrinsic. It's not a configuration problem you can tune your way out of.

This is what changes when the storage model changes. The reason the index tax is so expensive in row-based storage is that every row is an independent write event. One heap insert, one WAL record, one B-tree traversal per index. The cost is per-row because the storage is per-row.

Columnar storage changes the unit of work. Instead of writing one row at a time, it batches thousands of row versions into a single segment before writing. One WAL record covers the whole batch. Index maintenance happens at the segment level, not the row level. The per-row tax that makes five indexes expensive at 50K inserts/sec gets amortized across thousands of rows per write. Write amplification drops from the 3 to 5x range to near 1:1.

That's not a tuning improvement. It's a different cost structure for the same logical operation. We covered the full architecture in The Postgres Optimization Treadmill, which walks through why these constraints exist in row-based Postgres and what it looks like when the storage layer is built for this workload pattern from the start.

The bottom line

Every index you've ever added was the right call at the time. That's not the argument here.

The point is that the index tax is a real cost with a specific multiplier, and that multiplier matters a lot more at 50K inserts/sec than it does at 500. If write latency is climbing on a table that looks well-indexed, pull the insert rate and count the indexes. Do the multiplication. The answer is usually sitting right there in the numbers.

And if those numbers show you're paying five or more index taxes on every row, with no signs of the data slowing down, the question isn't which indexes to drop. It's whether the per-row cost structure is the right one for the workload.

Your Postgres database is struggling. Write latency is climbing, autovacuum is fighting for I/O, and the indexes you added three months ago aren't cutting it anymore. So you do the obvious thing.

You upgrade the instance. Metrics drop. Everyone exhales.

Six months later, you do it again.

Nobody puts this in a postmortem, because vertical scaling works. That's why teams keep reaching for it. But if you're running continuous high-frequency ingestion on Postgres, it's not a fix. It's a payment plan on a debt that keeps growing.

The Cost Curve Doesn't Lie

You've probably already run the numbers. At 50K inserts per second, you're adding roughly 1.5 billion rows per year. Your data volume curve is exponential. Your infrastructure cost moves in steps, doubling each time you provision the next tier up.

Plot both lines on the same chart. Watch them diverge.

You upgrade from 16 vCPU/64GB to 32 vCPU/128GB with provisioned IOPS (io2 at 10,000+ on AWS, say). Cost roughly doubles. You get six months of breathing room. Then the data keeps growing, and the metrics start climbing again.

So you upgrade again. The cost doubles again. Twelve months out, you're projecting another upgrade. The database line item is growing faster than the product revenue it supports.

Oof.

What You're Actually Buying

More CPU gives autovacuum room to run without starving query execution. More RAM improves shared_buffers and OS page cache hit rates. Faster storage reduces I/O wait across the board.

All real wins. None of them touch the per-row overhead.

Here's what's actually happening underneath. At 100K inserts per second, you're writing 250-350MB of actual I/O for 100MB of application data. Every row carries MVCC headers, index entries, and WAL records whether you asked for them or not. A 1KB sensor reading becomes roughly 2.5 to 3.5KB of actual I/O: 23-byte heap tuple header, five index entries at ~60 bytes each, plus a ~1.2KB WAL record stacking on top.

At 100K inserts/sec, that's 250-350MB/sec of real I/O to move 100MB/sec of data. A bigger instance tolerates that overhead more gracefully. It does not reduce it.

So the trajectory holds. Six months of headroom, metrics creep back, another upgrade, another budget conversation. Each step costs more than the last one and buys roughly the same amount of time.

The Invisible Cost Nobody Tracks

Here's where it gets uncomfortable. The latency graphs are one thing. Engineers watch latency graphs. Finance watches the invoice.

At some point the database line item becomes visible enough that someone schedules a meeting. Now you're explaining autovacuum to a person who manages a spreadsheet for a living. (That meeting is not fun. The prep work for that meeting costs engineering time you don't have.)

But that's the visible cost. The invisible one is worse.

When teams hit this pattern, senior engineers typically spend 20-30% of their time on database operations. Not firefighting. Weekly. Monitoring autovacuum lag. Tuning per-partition settings. Watching replication delay. Reviewing runbooks before anyone touches the schema. Making sure the pg_partman automation didn't silently fail again.

None of that shows up in the cloud bill. It doesn't trigger a finance meeting – it just quietly drains your best people every single week. New engineers need weeks of onboarding before they can safely operate the partitioning scheme. What should be a one-person schema change becomes a team event with a rollback plan.

You've built a database operations practice inside your product engineering team. That wasn't the plan.

Why Vertical Scaling Feels Like It's Working

The thing that makes this pattern so persistent is that each optimization phase genuinely does help. Vertical scaling is no exception.

You add the bigger instance, and autovacuum workers stop competing with queries for CPU. Shared buffers expand, and buffer cache hit rates climb. Those io2 IOPS stop being the bottleneck. For a while, the system breathes.

But here's the thing: Postgres wasn't designed for continuous, high-frequency, append-only ingestion at scale. The design choices that make it excellent for general-purpose workloads, MVCC for concurrency, row-based heap storage, B-tree indexes, the WAL architecture – all generate overhead that multiplies when you're hammering it with hundreds of thousands of inserts per second that never pause.

Vertical scaling gives the existing architecture more room to operate. It doesn't change the architecture.

MVCC creates per-tuple overhead on data you'll never update. Row storage forces you to read all 30 columns when your query needs two. B-tree indexes mean every insert has to traverse and update every index, and at 50K inserts/sec with five indexes, that's 250K index insertions per second. WAL records every single one of those operations before touching a data page, so at 100K inserts/sec you're generating 50-100MB/sec of WAL just to do normal work.

None of those problems shrink when you add more vCPUs.

How It Shows Up Before It's a Crisis

The real tell isn't in a p95 latency chart. It's the pattern.

You optimize. You get relief. The metrics climb back. You optimize again. The relief lasts a little less time than before.

Before it becomes a full crisis, it shows up in how the team is spending its time.

Optimization is on every quarterly roadmap, not as a one-time project, but as a line item, every quarter, competing with features for engineering time.

The database bill goes up 40% while user growth was 15%. Finance notices. Those numbers don't get ignored.

You ship a 2x performance improvement and data growth erases it within two quarters. The treadmill doesn't slow down – it speeds up.

And autovacuum just keeps coming up! It's in the top five processes by CPU and I/O at all hours and tuning it is somehow always on someone's plate.

Two or three of these? Pay attention. Four? You're already in the pattern.

Optimization Problem vs. Architecture Problem

There are two different problems that both show up as "database performance is degrading."

The first is an optimization problem. The workload fits the database design. Better indexes, query rewrites, config tuning, vertical scaling. These directly improve the trajectory, and Postgres expertise solves it. For most workloads, vanilla Postgres is the right tool and this is the right path.

The second is an architectural mismatch. The workload is hitting design tradeoffs baked into the storage engine and the write path. Optimization helps short-term, but it doesn't change the trajectory. You're working around the architecture instead of with it.

Both of these look identical from the outside: degrading query latency, climbing infrastructure costs, teams spending more time on database operations than product work. The difference only becomes obvious when you notice each fix is lasting a little less time than the last one.

Vertical scaling is the right move for the first problem. For the second, it's just the most expensive item on the treadmill.

When to Think About Architecture Instead

If your workload is continuous high-frequency ingestion, your data is append-only, queries predominantly filter on time ranges, and you're measuring retention in months or years, you're probably dealing with an architectural mismatch, not an optimization problem.

You also don't need to replace Postgres. TimescaleDB extends vanilla Postgres with columnar compression, hypertables with automatic chunking, and a query planner that understands time-based access patterns. You keep SQL, your extensions, your team's knowledge, and the entire Postgres ecosystem. What changes is the storage engine and write path underneath (the parts actually generating the overhead).

Migration complexity scales with data volume. At 10M-50M rows, it's days to two weeks. At 100M-500M rows, two to six weeks. At 1B+, you're looking at months. Those hours don't go toward product features. And there's no point on that curve where waiting makes it cheaper.

If your team is spending 20%+ of engineering time on database operations and scalability is on every quarterly roadmap, you already know something is off. The upgrade cycles don't get cheaper. They just get further apart until they don't.

This post is part of a series on Postgres performance limits for high-frequency data workloads. The full analysis, including a workload scoring framework and migration complexity breakdown at different scales, is in the anchor essay: Understanding Postgres Performance Limits for Analytics on Live Data. Ready to test it on your own data? Start a free Tiger Data trial.

The Pattern Recognition Moment

You're reviewing monitoring on a normal workday. There hasn't been a new deployment, no weird traffic spike, and no schema changes. But p95 write latency has crept from 8ms to 25ms over the past month, and last week it touched 45ms. Your largest tables crossed 500M rows sometime in March and they're still climbing.

Six weeks of data points, all trending the same direction.

You've run Postgres in production for years. You've tuned queries, rebuilt indexes, and right-sized instances. But this time the fixes don't stick; every new index or config tweak brings the metrics back down for a few weeks, then they climb again. You can plot the trajectory out three months and know exactly where it lands.

So you do a proper audit: query plans, connection overhead, table stats, bloat. Everything checks out. Schema is sound, indexes cover the hot paths, and configs follow best practices. A consultant confirms the same: nothing misconfigured. But performance keeps degrading, and it correlates with data volume, not traffic.

You look closer at the workload. Most writes are inserts, not updates, and every row carries a timestamp. Queries almost always filter by time range. Data arrives continuously, not in batches or bursts, but as a steady stream that never pauses. You need months or years of retention, and you're not just storing this data. You're querying it under latency requirements.

This doesn't fit the profile of a transactional workload, and it doesn't fit a data warehouse either. It's continuous high-frequency ingestion that needs to stay operationally queryable.

Postgres is a brilliant general-purpose database. The same design choices that make it handle e-commerce, SaaS backends, and CMS workloads so well create compounding overhead for sustained high-frequency time-series ingestion with long retention. Design tradeoffs, not bugs. Baked into the architecture by intent.

You are not fighting misconfiguration. You are fighting architectural boundaries designed for a different workload class.

This piece walks through what we call the Optimization Treadmill: the sequence of phases most teams follow, each a correct response to observed symptoms, each providing temporary relief without changing the underlying trajectory. Understanding the mechanics of why the treadmill exists is what lets you recognize it early. If you recognize the scenario above, this is a common path. The question isn't whether you'll hit the ceiling. It's when, and how much runway you have left when you do.

What This Workload Looks Like

Not all high-write workloads will hit this wall. Postgres handles enormous write volumes for e-commerce, social networks, and SaaS backends without issue. The friction comes from a specific combination of six characteristics. If four or five describe your system, the optimization phases in the next section will be familiar.

Continuous high-frequency ingestion. Thousands to hundreds of thousands of inserts per second, 24/7, with no pause: IoT sensors reporting every few seconds, financial systems processing trades in real time, or APM platforms collecting metrics from thousands of hosts. High-frequency data generation is independent of user count. Batch systems get quiet periods where the database can run maintenance, but continuous ingestion never stops. Maintenance competes directly with writes, and there is no scheduling window.

Time-series access patterns. Nearly every row has a timestamp, and queries almost always include time range filters. "Last 30 minutes of CPU utilization," "this week compared to last week," "all transactions between two dates." This goes beyond a created_at column; the entire query pattern revolves around time. General-purpose indexes aren't built for this access pattern, which is why teams end up reimplementing time-based data organization through manual partitioning scripts and custom tooling.

Append-only data. Once written, rows rarely change. Sensor readings don't get updated, financial transactions are immutable, log entries are permanent. Deletes happen in bulk (drop an entire month's partition), not row by row. MVCC exists to handle concurrent reads and writes on the same rows. Append-only workloads pay that overhead on data they never touch again. Autovacuum is running constantly just to clean up dead tuples that were never created through updates.

Long retention. Months to years, not days or weeks. Compliance might require seven years of financial records, manufacturing teams need root cause analysis across quarters, and ML pipelines need two-plus years of training data. Shortening retention will just hide architectural problems because old data ages out, and long retention means unbounded table growth. At 50K inserts per second, that's roughly 1.5 billion rows per year. After three years? 4.5 billion rows.

Operational query requirements. This isn't cold storage or an analytics warehouse you query once a day. You need millisecond responses on the last day's data, sub-second on the last week, and reasonable performance across the full retention window. Real-time dashboards, alert systems, user-facing analytics, ad-hoc investigation, all querying the same database. Data warehouse depth with operational latency requirements.

Sustained growth. Data volume growing 50–100%+ year over year on a predictable curve. Static workloads can be over-provisioned once and left alone, but growing workloads demand constant re-optimization. You're not solving for current scale. You're chasing projected scale, and the gap keeps widening.

If four or five of these apply, the next section maps the optimization path most teams follow. If your workload is standard OLTP, batch warehouse, low-volume time-series, or short-retention, the underlying issues are likely different.

This combination of characteristics didn't exist at scale 15 years ago. It's a product of specific infrastructure shifts: billions of connected devices generating continuous telemetry, high-frequency trading systems that treat microseconds as a competitive moat, AI pipelines that require years of operational history as training data, and observability platforms collecting metrics from every process in a distributed system. The cloud didn't just scale these workloads up. It made them continuous. Machines that never go offline generate data that never stops. That changed what operational databases are asked to do, and general-purpose engines weren't redesigned to match.

The Optimization Path

Most teams working this pattern follow roughly the same sequence. Each phase is a reasonable response to observed symptoms, but each buys 3–6 months of relief at most, adds operational complexity, and has diminishing returns. The optimizations address symptoms without changing the underlying architecture. The ceiling doesn't move. You do, until you run out of room.

Phase 1: Index optimization

The trigger is predictable: query performance degrades as tables grow past 50–100M rows, or sequential scans on a 100M-row table take minutes. The textbook answer is to add B-tree indexes on timestamp columns, build composite indexes for common filter combinations, create partial indexes on hot time ranges, and run ANALYZE to refresh pg_statistic.

-- Composite index for the most common dashboard query pattern
CREATE INDEX idx_metrics_device_time
  ON device_metrics (device_id, ts DESC);

-- Partial index covering only the hot partition
CREATE INDEX idx_metrics_recent
  ON device_metrics (ts DESC)
  WHERE ts > now() - interval '7 days';

A query that did a sequential scan across 100M rows now hits an index and returns in milliseconds. 10–100x improvement on read performance is typical. Problem solved, for now.

Issues start showing up as tables grow past 300M rows. Every INSERT must update every index on the table. With five indexes, each insert performs six write operations: one heap tuple write and five B-tree leaf page insertions. At 50K inserts/sec, that's 300K write operations per second. Each index insertion traverses the B-tree, potentially causing page splits that trigger additional I/O. pg_stat_user_indexes starts showing index bloat climbing:

-- Monitoring index bloat
SELECT schemaname, tablename, indexname,
       pg_size_pretty(pg_relation_size(indexrelid)) as index_size,
       idx_scan as index_scans,
       idx_tup_read,
       idx_tup_fetch
FROM pg_stat_user_indexes
WHERE schemaname = 'public'
ORDER BY pg_relation_size(indexrelid) DESC;

Index sizes grow faster than table sizes because B-trees don't reclaim space efficiently for append-heavy, time-ordered data. For keys that increase monotonically like timestamps, inserts concentrate on the rightmost leaf pages, resulting in repeated splits. Old leaf pages become sparse but remain allocated. You've improved read latency at the cost of write throughput, and this workload needs both.

Phase 2: Table partitioning

Your largest table has crossed 800M to 1B rows, and dropping old data via DELETE causes table bloat and long-running transactions that block autovacuum. You implement time-based range partitioning (typically daily or weekly).

-- Partitioned table setup
CREATE TABLE device_metrics (
    ts          timestamptz NOT NULL,
    device_id   bigint NOT NULL,
    metric      text NOT NULL,
    value       double precision
) PARTITION BY RANGE (ts);

-- Daily partitions created by cron or pg_partman
CREATE TABLE device_metrics_20250601
  PARTITION OF device_metrics
  FOR VALUES FROM ('2025-06-01') TO ('2025-06-02');

Implementation requires automation: cron jobs or pg_partman to create future partitions, monitoring to detect gaps where partition creation failed, and careful handling of queries that span partition boundaries. Backup and restore now operates on hundreds of individual tables, pg_dump time scales with partition count, and schema migrations touch every partition.

The wins are concrete. Queries with time-range filters trigger partition pruning, and EXPLAIN shows the planner excluding irrelevant partitions:

EXPLAIN SELECT avg(value) FROM device_metrics
WHERE ts > now() - interval '1 hour';

-- Scans 1-2 partitions instead of the entire table
-- "Partitions removed: 498 of 500"

Dropping old data becomes DROP TABLE device_metrics_20240101 instead of a multi-hour DELETE that generates gigabytes of WAL and dead tuples.

What happens at 500+ partitions? The PostgreSQL documentation on partitioning best practices is direct about the cost: "Planning times become longer and memory consumption becomes higher when more partitions remain after the planner performs partition pruning." pg_partman maintenance jobs occasionally fail silently, leaving gaps. Queries spanning long ranges (quarterly reports, year-over-year comparisons) hit hundreds of partitions and regress in performance. Each active partition still has its own autovacuum overhead. The write path is faster per-partition but aggregate write load is unchanged. And the operational complexity is real. New engineers need to understand the partitioning scheme, the automation scripts, the monitoring for gaps, the procedures for backfills, and the implications for schema changes.

Phase 3: Autovacuum tuning

This is where it starts to feel wrong. You're tuning a cleanup process for data you never modify. n_dead_tup counts are climbing on active partitions, last_autovacuum timestamps show vacuum running constantly but falling behind during write peaks, and pg_stat_activity regularly shows autovacuum workers competing for I/O.

Even append-only workloads generate work for autovacuum. Aborted transactions leave dead tuples. Hint-bit setting (marking tuples as known-committed or known-aborted to avoid future pg_xact lookups) requires dirtying pages. And since PostgreSQL 13, autovacuum triggers based on insert count (not just dead tuples) specifically to freeze tuples and update the visibility map. At high insert rates, this means autovacuum fires continuously on tables that never see a single UPDATE or DELETE.

-- Per-table autovacuum settings on high-traffic partitions
ALTER TABLE device_metrics_20250601 SET (
    autovacuum_vacuum_scale_factor = 0.01,    -- default 0.2
    autovacuum_vacuum_cost_delay = 2,         -- default 2ms (20ms before PG 12)
    autovacuum_vacuum_cost_limit = 1000       -- default 200
);

# postgresql.conf adjustments
autovacuum_max_workers = 6            # default 3
autovacuum_naptime = 15s              # default 1min
maintenance_work_mem = 2GB            # default 64MB
autovacuum_vacuum_cost_delay = 2ms
autovacuum_vacuum_cost_limit = 800

This helps stabilize bloat, and pg_stat_user_tables.n_dead_tup stays under control. But autovacuum workers now consume measurable CPU and I/O continuously, and monitoring shows autovacuum in pg_stat_activity at all hours. During write peaks, vacuum falls behind, bloat creeps back, and query performance becomes variable. You're tuning a process that exists to clean up overhead your workload doesn't fundamentally produce, but that the storage engine creates anyway.

Phase 4: Vertical scaling

All of your optimizations are showing diminishing returns. The next logical step is to add more resources: upgrade from 16 vCPU/64GB to 32 vCPU/128GB with provisioned IOPS storage (e.g., io2 at 10,000+ IOPS on AWS).

More CPU gives autovacuum workers room to operate without starving query execution. More RAM increases shared_buffers and OS page cache hit rates, reducing physical disk reads. Faster storage reduces I/O wait time across the board. This gives you roughly six months of headroom.

Math doesn't lie: the infrastructure cost doubled or tripled, but data growth is still exponential. At the current trajectory, you'll need another upgrade in 12 months. The database cost line item is growing faster than the product revenue it supports.

Phase 5: Read replicas

Dashboards and analytics queries compete with ingestion for CPU and I/O on the primary. You add 1–3 streaming replicas, configure pgbouncer or pgpool to route read traffic, and separate the connection pools. Immediately, write performance on the primary improves. Expensive analytical queries run against replicas without blocking ingestion.

The primary still carries the full write load. At sustained high insert rates generating tens of megabytes per second of WAL, replicas that fall behind accumulate WAL on the primary, consuming disk. The further behind a replica gets, the more WAL the primary must retain, and high write volume is exactly what causes replicas to fall behind in the first place. Real-time dashboards pointing at lagging replicas show stale data. You're now managing multiple Postgres instances with their own monitoring, autovacuum tuning, and connection pooling. The write bottleneck is still untouched.

Taking stock

After all five phases, this is what the infrastructure looks like: partitioned tables across 500+ partitions with pg_partman automation and monitoring, aggressive per-table autovacuum settings under constant adjustment, instances upgraded 2–3x from original specs with provisioned IOPS, 2–3 streaming replicas with connection-level routing, detailed runbooks covering partition management, vacuum procedures, and failover scenarios.

Each optimization was the right response. Each bought time. Yet the trajectory is unchanged.

Senior engineers are now spending 20–30% of their time on database operations. Quarterly planning includes a database scalability line item. New hire onboarding takes weeks before someone can safely operate the partitioning scheme. The team has become part product engineering, part DBA.

Is this inherent to the scale, or is it inherent to the architecture?

The answer matters because the two problems have different solutions. Optimization within the right architecture has a ceiling you can raise. Optimization against an architectural mismatch has a ceiling that doesn't move. Only the timeline changes. For this workload pattern, the ceiling is structural. The question was never if you'd hit it. It was always when.

Why These Optimizations Hit a Ceiling

The optimization phases above aren't ineffective. Each one operates within architectural boundaries that weren't designed for this workload pattern, and those boundaries constrain how much any optimization can actually move the needle. Understanding the mechanics explains why returns diminish.

Postgres is a brilliant general-purpose relational database. Its design handles an enormous range of workloads well: e-commerce, content management, authentication, SaaS backends. "General-purpose" means optimized for the average case. High-frequency time-series ingestion with long retention is not the average case. Four core design decisions create this compounding overhead.

MVCC (Multi-Version Concurrency Control)

MVCC lets readers and writers operate concurrently without lock contention. The PostgreSQL documentation on concurrency control describes the core guarantee: "reading never blocks writing and writing never blocks reading." When a row is updated, Postgres keeps the old tuple version visible to in-flight transactions, and autovacuum later marks dead tuples as reusable. For workloads with concurrent reads and updates on shared rows, this is an excellent tradeoff.

For append-only ingestion, every insert still pays the full MVCC cost. Each heap tuple carries a fixed-size header (23 bytes on most machines) containing t_xmin, t_xmax, t_cid, t_ctid, t_infomask, t_infomask2, and t_hoff. These fields track transaction visibility, even though the row will never be updated or deleted by a transaction. Extra cost with no extra value.

The write amplification is easily observable. A 1KB sensor reading becomes:

• 23-byte heap tuple header (plus alignment padding and a 4-byte ItemIdData pointer)
• 1,024 bytes of row data
• 5 index entries (assuming 5 indexes, ~40–80 bytes each in B-tree leaf pages)
• ~1.2KB WAL record (heap insert + index insertions)

Total actual I/O: roughly 2.5–3.5KB per 1KB of logical data. At 100K inserts/sec of 1KB rows, you're writing 250–350MB/sec of actual I/O for 100MB/sec of application data. The exact ratio varies with row width, index count, and whether full_page_writes triggers after a checkpoint.

Autovacuum still has work to do on append-only tables. Aborted transactions leave dead tuples, and hint-bit setting (marking tuples as known-committed or known-aborted to avoid future pg_xact lookups) requires dirtying pages. At high insert rates, even these minor sources of work keep autovacuum continuously active. pg_stat_user_tables.n_dead_tup may stay low, but vacuum_count and autovacuum_count keep climbing steadily.

Row-based storage with B-tree indexes

Postgres stores data as a heap of 8KB pages, each containing variable-length tuples laid out row by row. Every tuple contains all columns. B-tree indexes map key values to ctid (page number + offset) pointers into the heap.

For time-series analytics, this creates read amplification:

SELECT avg(temperature)
FROM sensor_readings
WHERE ts > now() - interval '1 hour'
  AND device_id = 42;

This query needs two columns: ts and temperature. If the table has 30 columns, Postgres reads all 30 columns for every matching row from the heap pages. The I/O is 15x what a columnar layout would require, where only the referenced columns are read from disk.

Time-series data also compresses extremely well in columnar formats. Sequential timestamps delta-encode to near-zero storage (a regular interval collapses from 8 bytes per timestamp down to a single bit via delta-of-delta encoding), and repeated device IDs run-length-encode. Floating-point sensor values compress with XOR-based compression derived from Facebook's Gorilla algorithm (Pelkonen et al., "Gorilla: A Fast, Scalable, In-Memory Time Series Database," VLDB, 2015). Columnar storage routinely achieves 10–20x compression on time-series data. Row-based heap storage can't apply any of these techniques because values from different columns are interleaved on the same page.

On the write side, B-tree index maintenance creates significant overhead. Each insert traverses every index's B-tree from root to leaf, finds the correct leaf page, and inserts the new entry. If the leaf page is full, it splits, which can cascade up the tree. For time-ordered data, inserts concentrate on the right edge of timestamp indexes, creating contention on a small number of leaf pages. Non-timestamp indexes (device ID, metric type) scatter inserts across the tree, causing random I/O. With five indexes on a table, every row insert performs one heap page write, five B-tree traversals and leaf page insertions, plus WAL records for each. At 50K inserts/sec, that's 50K heap writes + 250K index insertions per second.

Query planning overhead

The Postgres planner runs a full optimization pass on every query: it enumerates possible paths, estimates costs from pg_statistic entries, considers index usage, evaluates join orders, and selects an execution plan. For workloads with diverse, unpredictable query patterns involving complex joins, this is the right approach.

For time-series workloads, query shapes are highly repetitive. The same WHERE ts > now() - interval '...' filter runs thousands of times per second. The full planning cycle executes every time. At high query rates, planning overhead is measurable in pg_stat_statements as the gap between total_plan_time and total_exec_time.

Statistics maintenance creates its own cost. ANALYZE samples rows to populate pg_statistic, with the sample size scaled by default_statistics_target (default: 100, which yields roughly 30,000 sampled rows). On billion-row tables, even this sampling-based statistics collection is expensive and must run frequently to keep estimates accurate. Stale statistics provide poor cardinality estimates, leading the planner to choose sequential scans over index scans, or vice versa.

With hundreds of partitions, the planner must evaluate partition pruning for each partition's bounds against the query predicates. This is fast per-partition but scales linearly with partition count. At 500+ partitions, plan time for simple queries can exceed execution time.

Write-Ahead Logging (WAL) volume

Every data modification generates a WAL record before it's applied to the heap or index pages. WAL writes are sequential and synchronous (fsync per commit, or per wal_writer_delay interval with asynchronous commit). At 100K inserts/sec, WAL generation is roughly:

• Heap insert records: ~100–150 bytes each = 10–15MB/sec
• Index insert records: 5 indexes × ~60–80 bytes each = 30–40MB/sec
• Full-page writes (after checkpoint): intermittent bursts of 8KB per dirtied page

Total sustained WAL throughput: 50–100MB/sec under normal operation, spiking higher after checkpoints when full_page_writes triggers 8KB records for newly dirtied pages. The PostgreSQL documentation describes why: "the first modification of a data page after each checkpoint results in logging the entire page content." At those rates, that's 3–6GB/min, 180–360GB/hour.

WAL I/O becomes a direct throughput bottleneck. pg_stat_wal shows wal_write and wal_sync times climbing. Replicas that can't apply WAL fast enough fall behind, and unprocessed WAL files accumulate on the primary's pg_wal directory, consuming disk. max_wal_size and checkpoint frequency become critical tuning parameters.

The compounding effect

None of these four constraints operates in isolation. Each amplifies the others, and that's where the math gets ugly.

MVCC overhead creates per-tuple bloat, which accumulates faster than autovacuum can clean at high insert rates. Autovacuum competing for I/O degrades write throughput. Degraded write throughput causes queries on bloated tables to slow down, which increases pressure to add more indexes. More indexes produce more write amplification, more WAL, and more replication lag. Row storage forces read amplification on time-range queries, which creates pressure to add covering indexes. Those indexes add to the write overhead feeding back into the MVCC/autovacuum loop.

At 50K inserts/sec with five indexes on a table, the steady-state database workload is: 50K heap tuple writes/sec, 250K B-tree index insertions/sec, 50–100MB/sec sustained WAL generation, continuous autovacuum activity across active partitions, and full query planning on every incoming query.

This is why a 16-core/64GB instance struggles with what appears to be a straightforward append-only workload.

Partitioning reduces per-partition table size but doesn't change the per-row overhead. Adding RAM improves buffer cache hit rates but doesn't reduce write amplification. Autovacuum tuning manages bloat but can't eliminate the cost of producing it. Each optimization operates within these constraints. None removes the constraints themselves.

This is the Optimization Treadmill at the mechanical level. You're not fighting configuration. You're fighting the storage model, the concurrency architecture, and the write path. All of which are designed for a workload that looks nothing like yours.

When to Choose a Different Path

Most teams recognize this pattern 12–18 months too late. By then, the tables are massive, the partitioning scheme is deeply embedded, and migration has become a multi-month project. The difference between acting at 10M rows and acting at 1B rows is roughly an order of magnitude in engineering cost.

Postgres Workload Scoring Framework

Go back to the six characteristics. Be honest about how many describe your system right now, and then score yourself again against where you'll be in 12-18 months.

If four or five apply, you're in this pattern. The optimization phases above are already in your future, or you've started them.

If all six apply, you're past the point of easy exits. Architectural friction is the dominant factor in your performance trajectory, and the migration cost is climbing every quarter you wait.

If three or fewer apply, you likely have a different problem. Standard Postgres optimization should change the trajectory.

Early warning signs

Before the pattern becomes a crisis, it shows up in how the team spends its time:

Optimization dominates planning. 10–20% of engineering time goes to database performance, and every quarterly roadmap includes a scalability line item.

Costs grow faster than revenue. Finance is asking why the database bill increased 40% while user growth was only 15%.

Operational complexity accumulates. 20+ pages of runbooks, partition management scripts, monitoring for autovacuum lag, replication delay, and index bloat. New engineers need weeks of onboarding before they can safely operate the database.

Growth outpaces optimization. You ship a 2x improvement and data growth erases it within two quarters.

Autovacuum is a constant concern. It's in the top five processes by CPU and I/O at all hours, and tuning it is a recurring conversation.

Two or three of these signs mean you should be paying attention. Four or more means you're already in the pattern.

Migration complexity at different scales

10M–50M rows. A day or two to 1–2 weeks. Simple dump/restore, or logical replication. Low risk, fast validation, easy rollback. 1–2 engineers part-time (roughly 80 engineer-hours).

100M–500M rows. 2–6 weeks. Partition-by-partition migration. More dependencies to account for, more thorough testing required. 2–3 engineers, mostly full-time (roughly 400 engineer-hours).

1B+ rows. 2–6 months. Hundreds or thousands of partitions. Zero-downtime required, complex rollback planning. Application-level dual-write or change-data-capture pipelines are in play. 3–5 engineers full-time plus a validation period (roughly 2,000 engineer-hours).

Those hours are not spent on product features. And there's no point on this curve where migration gets easier by waiting.

What "purpose-built Postgres variants" means

TimescaleDB is built on top of Postgres, not in place of it. The PostgreSQL wire protocol, SQL query language, extensions like PostGIS and pgvector, your application code, and your ecosystem tooling all stay the same. What changes is the storage engine and execution layer underneath.

MVCC overhead addressed through columnar compression. The problem: every row insert in vanilla Postgres generates per-tuple MVCC headers, index entries, and WAL records regardless of whether the data will ever be updated, driving 3–5x write amplification and continuous autovacuum load. TimescaleDB's columnar storage (the Columnstore layer) batches up to 1,000 row versions per column into compressed arrays before writing to disk. Each batch write replaces thousands of individual heap tuple insertions with a single compressed segment write. The per-tuple MVCC header overhead is amortized across the batch, and autovacuum pressure drops proportionally. Far less row-level churn to clean up. In practice, write amplification drops from the 3–5x range to near 1:1 for sustained append workloads. The Tiger Data architecture whitepaper covers the columnar layout and compression pipeline in detail.

Row storage replaced by columnar layout for time-series data. The problem: vanilla Postgres reads all columns of every matching row even when a query needs two, creating 15x+ read amplification on wide tables, with none of the compression techniques applicable to time-series data. Rather than reading all 30 columns of a row to get two values, queries against the columnar layer read only the referenced columns from compressed column arrays. The 15x read amplification drops to near 1:1. Time-series compression (delta-of-delta for timestamps, gorilla-style XOR for floats, run-length encoding for repeated values) routinely achieves 10–20x compression ratios vs. heap storage. A dataset that occupies 1TB in vanilla Postgres often fits in 50–100GB with columnar compression enabled.

Query planning overhead reduced through chunk exclusion and continuous aggregates. The problem: the Postgres planner runs a full optimization pass on every query, and with hundreds of partitions, partition pruning overhead can exceed execution time for simple queries. TimescaleDB's planner extension adds chunk exclusion that operates at a lower level than Postgres's partition pruning. Chunks are indexed by time range in a catalog table, and the planner excludes non-overlapping chunks before the standard planning phase. For query shapes that repeat thousands of times per second, this eliminates most of the per-partition pruning overhead. Continuous aggregates go further: pre-computed rollups stored as materialized views, updated incrementally as new data arrives, so dashboards querying hourly or daily aggregations hit a small summary table instead of scanning billions of raw rows.

WAL volume reduced through batched ingestion. The problem: at 100K inserts/sec, vanilla Postgres generates 50–100MB/sec of WAL, creating I/O bottlenecks and causing replicas to fall behind. Lagging replicas force the primary to retain more unprocessed WAL, which consumes disk and makes the lag worse. The root cause is per-row WAL records: one per heap insert, one per index insertion. Columnar storage's batch writes generate WAL at the segment level rather than the row level. At 100K inserts/sec, WAL volume drops from 50–100MB/sec to roughly 5–15MB/sec in typical deployments, which eliminates most replication lag issues. Replicas that previously fell behind during write peaks can keep up without tuning.

Concrete numbers. Benchmark results vary by workload, but the directional data is consistent: ingestion throughput 10–20x higher than vanilla Postgres at equivalent instance size, query performance on time-range aggregations 100x+ faster with columnar storage, storage footprint 10–20x smaller with compression enabled. RTABench, a benchmark for real-time analytics workloads, publishes results showing the performance gap between vanilla PostgreSQL and TimescaleDB across real-world query patterns. See the benchmark results

Decision framework

Choose a specialized architecture if you score 4+ on the Postgres Workload Scoring Framework AND you're experiencing 2+ early warning signs AND you can project continued data growth.

Strong indicators to act now: you're under 100M rows, you're already building custom partitioning, your team spends 15%+ of engineering time on database optimization, and you can project 500M+ rows within 12 months.

You might not need this if writes are bursty rather than continuous, retention is 7–30 days, queries don't predominantly filter on time ranges, or growth is stable and slow.

Optimization vs. Architecture

There are two different problems that both show up as "database performance is degrading."

Problem 1: Optimization within the right architecture. The workload fits the database's design. Better indexes, query rewrites, configuration tuning, and hardware upgrades directly improve the trajectory. Postgres expertise solves the problem. For most workloads, vanilla Postgres is the right choice.

Problem 2: The Optimization Treadmill. The workload hits design tradeoffs baked into the storage engine, concurrency model, and query planner. Optimization helps in the short term but doesn't change the trajectory. Each phase buys time. None buys a different outcome. You're working around the architecture rather than with it.

Knowing which problem you have determines the path forward.

If you followed the optimization phases in this piece, you weren't doing anything wrong. Those were correct responses to the symptoms. Any experienced Postgres team would have done the same. The pattern is common precisely because the progression makes sense at each step.

What changes with recognition is agency. At 10M–50M rows, you can choose a purpose-built architecture in days to weeks and redirect engineering time to product work. At 100M–500M rows, migration is harder but still reasonable, taking 2–6 weeks. At 1B+, it's a multi-month project, and every quarter of delay adds complexity.

The broader principle applies beyond this workload. Different databases have different architectural strengths, so the best choice depends on the workload. Postgres is brilliant for general-purpose relational work. Specialized variants built on top of Postgres excel at specialized patterns. Recognizing when architecture matters more than optimization is an engineering judgment call, not a criticism of the tool.

Architectural fit determines your ceiling. Optimization determines where you operate relative to that ceiling. When you're hitting the ceiling repeatedly, the productive question isn't "how do we optimize better?" It's "are we operating within the right architecture?" With this workload pattern, the ceiling was always there. You just needed enough data volume to find it. Score your workload. If you're at 8+ and under 100M rows, this is the cheapest architectural decision you'll make this year. The whitepaper covers the mechanics. The Tiger Data free trial lets you validate on your own data.

New: Learn how Plexigrid moved from 4 databases to 1 with Tiger Data.

As engineers, we're taught to design for reliability. We do design calculations, run simulations, build and test prototypes, and even then we recognize that these are imperfect, so we include safety factors. When it comes to the Industrial Internet of Things (IIoT) though, we rarely give the same level of scrutiny to the components that we rely on.

What if we treated our IIoT database the same way we treated the physical things we produce? We build and design a prototype database, and then put it through some serious testing, even to failure.

The Value (and Perils) of Stress Testing

Think of database stress testing as a destructive materials test for your data storage. You wouldn't trust a bridge made of untested steel, so don’t trust your database until you know its limits.

The Value:

• Identify Bottlenecks: Stress testing reveals the weak links—what is likely to fail first? Will you run out of storage? Will your queries get bogged down? Or will you hit the dreaded ingest wall (when data comes in faster than it can be stored)?
• Determine Real-World Behaviour: You'll find out exactly how your database performance changes as the amount of data increases. What issues are future-you going to struggle with?
• Optimize Configuration: Just like you might build a few different prototypes and see how it affects failure modes, changing your database configuration, especially when it comes to indices, can dramatically affect how it behaves. Building a rigorous stress testing framework provides a safe way to optimize your design.

I hope it goes without saying, but please, please don’t run this on your production environment. Even if it’s technically a different database but the same hardware, this test can wreak havoc on your resources and crash your system. You’ve been warned.

What to Measure?

There’s no point going through all the effort to break your system if you don’t learn anything. Assuming you’re using a PostgreSQL database (It’s 2026, Just Use PostgreSQL), here is a decent set of metrics to keep track of while you’re putting your database through its paces.

Table Size

The size of a Postgresql table is generally measured by number of rows, but the actual space on disk that it occupies is a sum of the heap (the main relational table), the indices, and the TOAST (storage for large objects).

The following query will give the number or rows as well as the size of each component of the table in bytes.

SELECT
      reltuples::bigint AS row_count,
      pg_relation_size('iiot_history') AS heap_size,
      pg_indexes_size('iiot_history') AS indices_size,
      pg_table_size('iiot_history') -
            pg_relation_size('iiot_history') AS toast_size
FROM pg_class WHERE relname = 'iiot_history';

The reason for the odd row_count is that counting rows the standard way, with COUNT(*), requires scanning the whole table, which is going to be painfully slow when we’re building a table big enough to break things.

Table Performance

The best way to measure table performance is to use the actual queries that your production system will use. At a minimum, this should include your batched INSERT (you always batch, right?) and at least one common SELECT. Keep in mind that for a table with N rows, the timing for queries tend to be either constant, log(N), N or worse depending on how the indices are structured. 

You can get very accurate timing info from running your queries with the prefix EXPLAIN ANALYZE, and it’s worth doing this at least once to see what the database is doing under the hood. However, I recommend running the whole test with a scripting language and then just timing the execution of that particular step. 

Server Performance

Don’t forget the engine that’s driving all this machinery. You’ll need to watch the CPU, Memory, Storage, and Network Bandwidth. People in the IT world tend to talk about headroom for a server, and that’s what you’re really looking at: how much spare capacity do you have? Your CPU and Memory usage might spike at times, but the important thing is that it’s not always running at max capacity.

There are a lot of free and paid tools to monitor these variables. I almost always do this type of test in a VM (easier to clean up the mess when it all breaks) and I like to use Prometheus but honestly Perfmon in Windows or Top in Linux gives you all you really need.

Setting Limits

It’s helpful to set some limits on these parameters so you know when to stop the test. For database size, it might be some measurement like a year's worth of data, or when the drive is 80% full. For ingest timing, I suggest stopping when inserting takes longer than the desired ingest frequency—this is the ingest bottleneck and something you really want to avoid in production. Scan times can be limited by the time it takes for a specific query. Maybe calculating the average value from one tag over the past hour must be less than 10s.

How to Simulate Data?

There are lots of ways to insert data, but it’s usually a tradeoff between how well the data represents real scenarios and how long it takes to run the test.

The following is one of my favourite methods for injecting large amounts of data into an IIoT database:

Say you have a classic IIoT history table like the following:

CREATE TABLE iiot_history(
	time TIMESTAMPZ NOT NULL,
	tag_id INT NOT NULL,
	value DOUBLE PRECISION,
	PRIMARY KEY (tag_id, time)
);

If you expect to ingest 10,000 tags at 1s intervals, you can use the following INSERT query to add a day’s worth of history to the back end of your table.

INSERT INTO iiot_history(time, tag_id, value)
	SELECT *, random() as value 
FROM(
		SELECT generate_series(
			min_date-INTERVAL '1day',
			min_date-INTERVAL '1s',
			INTERVAL '1s') as time
		FROM (SELECT LEAST(NOW(),MIN(time)) AS min_date 
FROM iiot_history)
),
		generate_series(1,10000) as tag_id;

This will generate random data values for every second during a day and for every tag_id from 1 to 10,000. Not exactly as interesting as real data, but enough to fill up your table.

The nice thing about this query is that you should be able to run it in parallel to your real-time data pipeline and it won’t mess with your data (aside from potentially locking your table while it runs). It’s also easy to modify this query to inject more or less tags as well as change the time interval if you’re playing around with different configurations.

If you use this query, or whichever one you prefer, in a script (I usually use Python), then you can automate the whole test. Something along the lines of:

1. Get database size
2. Run select queries, measure execution time
3. Run insert queries several times, measure and average execution time
4. Artificially grow database size
5. Repeat 1-3 until one of the failure conditions is reached.

How to Interpret Results and What to Expect in the Real World?

Your test results will give you some clear data points, but you still need to do some interpreting.

• Identify the Limiting Component: Where did the database fail? If it’s a query that took too long, you might be able to speed things up with a clever index. If it’s an insert that took too long, you might be able to speed things up by removing that clever index you added earlier.
• Optimize: There’s a lot you can do to improve table performance before throwing the whole thing out in frustration:
  1. Proper Indexing: Choosing an index is almost always a tradeoff, for example: Indexing the tag_id column before the time column will speed up most queries, at the cost of slower inserts as the table grows. Indexing the time column first will avoid the ‘ingest wall’ at the cost of slower queries. Figure out which solution is best.
  2. Plan for the future: Will you need more hardware in a few months or a few years? Being able to estimate the life of your existing architecture means you won’t be caught unawares when it no longer suffices.
  3. Partitioning/Chunking: For very large tables, you may need to partition appropriately (see PostgreSQL extensions like TimescaleDB). How great would it be to learn you’ll need this before you actually need this.
• Add a Safety Factor: If your test showed a maximum reliable throughput of 15,000 rows/sec, set your operational limit to 10,000 rows/sec. The real world has peaks, unexpected queries, and background maintenance tasks that will steal resources. Like we do with all engineering products, design with margin.

If you treat your database like a prototype and really put it through its paces, you’ll get a preview of how it’ll behave in the future and make good, proactive design decisions instead of struggling in the future. Now, go break something (and learn).

Most developers start building with Postgres because it’s simple, reliable, and flexible. You get a clear relational model, transactional semantics you can trust, and an ecosystem that lets teams move quickly without committing to a complex architecture early. The challenge is keeping that simplicity as systems grow. Higher ingest, larger datasets, and increasingly real-time analytical workloads can push teams toward a second system long before they want one.

This pressure is most visible in time-series workloads that demand real-time performance. High write rates, append-heavy tables, and repeated queries over recent windows stress both storage and execution paths. Without reducing the amount of work required per query, scale quickly becomes an architectural problem rather than a performance optimization one, shifting effort from incremental tuning to changes in system design.

TimescaleDB is designed to change that trajectory. “Start on Postgres, scale on Postgres” is a promise, but it is grounded in a specific architectural approach: performance at scale comes from reducing the work the database must do as data grows, then parallelizing what remains. TimescaleDB 2.25 continues this evolution by tightening the execution and maintenance paths that dominate cost at scale, so common workloads become cheaper and operationally steadier under sustained growth.

This release focuses on three outcomes: faster queries without constant tuning, efficient scaling to larger datasets and higher ingest, and real-time analytics that stays current and trustworthy without introducing a second system.

Faster Postgres queries at scale, with less tuning

Compression, chunk pruning, and columnar execution already reduce query cost by limiting how much data needs to be read and processed. In 2.25, more queries can avoid work entirely, and the planner is more consistent about selecting those cheaper plans.

A clear example is aggregation on compressed data. In earlier releases, queries using functions like MIN, MAX, FIRST, or LAST benefited from compression and metadata, but they still required scanning compressed batches and performing aggregation during execution. The scan was cheaper than a row-oriented approach, but it was still work proportional to the data touched.

In 2.25, these aggregates can often be answered directly from sparse metadata maintained for compressed chunks. The planner can choose a custom execution path that reads summaries rather than scanning or decompressing data. This is implemented via the new ColumnarIndexScan plan node (see PR #9088, PR #9103, and PR #9108). On workloads where this applies, the 2.25 release notes report this class of queries speeding up by up to 289x. For teams running dashboards or monitoring queries over large compressed datasets, this can translate into dramatically faster response times with no query changes required.

The important shift here is in cost structure. Once an answer can be derived from metadata, performance is no longer tied to the number of rows stored inside a chunk. It is tied to the minimum work required to identify relevant chunks and read their summaries, which becomes more valuable as datasets grow.

A complementary improvement applies the same idea to another common pattern: time-filtered queries that do not need to materialize column values. For queries like SELECT COUNT(*) FROM events WHERE time > ..., previously, the execution path could still require decompressing the time column to evaluate the predicate, even though the query does not need to read time values for every row. In 2.25, the time column can often be skipped entirely for these cases, reducing CPU and memory pressure while preserving the same result (see PR #9094). The release notes describe this pattern as up to 50x faster for the example query.

As these fast paths expand, plan stability becomes just as important as peak speed. Even when an efficient path exists, teams feel it when the planner chooses it inconsistently or when small changes in query shape lead to surprising regressions. In 2.25, planner improvements around columnar scan paths and ordering help make compression-aware execution more predictable (see PR #8986 and PR #9133). Fewer surprises mean less time spent tuning and diagnosing why a query slowed down as data evolved.

Efficient scaling for high-ingest Postgres workloads

A hard part of scaling is not only achieving good performance at a given size, but preserving efficiency as data volume, ingest rate, and concurrency grow together over time. In practice, scaling pressure shows up in two ways. Some costs grow gradually, such as planning and execution work increasing with the number of partitions. Others appear more abruptly, when accumulated complexity makes execution brittle and small changes in data or query shape trigger different plans and sudden slowdowns.

TimescaleDB’s scaling model is designed to address both. It relies on clear boundaries: partitioning data into chunks, using metadata to prune irrelevant chunks, and compressing data to reduce the work required within each chunk. In 2.25, several refinements make these boundaries behave more efficiently and consistently under sustained growth.

One pressure point is that chunk counts rise over long retention windows, making pruning and constraint handling increasingly important. Earlier versions already used constraints and metadata to skip irrelevant chunks, but there were cases where constraint handling became more permissive than necessary, causing queries to consider more chunks than required as datasets aged. In 2.25, constraint handling improves for fully covered chunks, helping keep both planning and execution costs more tightly bounded as data volume increases (see PR #9127).

Planning behavior under high partition counts is another area where inefficiency and brittleness can emerge together. As hypertables accumulate thousands of chunks, planning time and plan quality can matter as much as execution speed, especially for joins and more complex query shapes. TimescaleDB 2.25 includes fixes for a planning performance regression on Postgres 16 and later affecting some join queries (see PR #8706). These changes reduce both how quickly planning cost grows and how likely it is to tip into unstable behavior as workloads evolve.

The result is more efficient scaling in practice. Costs still grow with data, but they grow more slowly and with fewer surprises, allowing Postgres to continue scaling in place rather than forcing architectural changes to manage accumulated overhead.

Real-time analytics in Postgres, without a split architecture

As refresh frequency increases and datasets grow, keeping analytics fresh inside the primary database can create background pressure. That pressure grows unless refresh and maintenance paths stay efficient. TimescaleDB has long supported real-time analytics inside Postgres through continuous aggregates, compression, and retention policies. In 2.25, the focus is on lowering the operational footprint of staying current as systems run continuously.

One improvement is compressed continuous aggregate refresh. Earlier versions supported refreshing into compressed hypertables, but the refresh path could include intermediate steps that added extra I/O and CPU work. In 2.25, direct compression on continuous aggregate refresh is enabled via a configuration option, reducing unnecessary data movement when keeping aggregates up to date (see PR #8777 and PR #9038). The semantics are unchanged, but the cost of maintaining freshness is lower, especially for frequent refresh schedules.

This is complemented by refinements to batching. Large refresh transactions can temporarily increase WAL volume and create uneven load. In 2.25, the default buckets_per_batch for continuous aggregate refresh policies is adjusted to keep transactions smaller (from 1 to 10 buckets), reducing WAL holding and making refresh behavior steadier under sustained ingest (see PR #9031).

The release also includes incremental improvements that reduce background churn from lifecycle operations like retention and deletes on long-running datasets, along with correctness and robustness fixes for compressed and partitioned workloads. For example, support for retention policies on UUIDv7-partitioned hypertables expands the set of configurations where lifecycle management remains reliable over time (see PR #9102). These changes are small individually, but they matter for trust. Real-time analytics only works if results stay aligned with transactional truth as schemas and workloads evolve.

Closing

TimescaleDB 2.25 continues to make Postgres a better place to run real-time analytics at scale: faster queries through less work, smoother behavior as data and ingest grow, and lower operational overhead for keeping analytics current and correct. 

All in service of a simple yet powerful idea: start on Postgres, scale on Postgres. Learn why vanilla Postgres hits performance ceilings at scale.

To learn more, check out the full release notes or try Tiger Cloud for free and experience TimescaleDB 2.25 on your largest hypertables. Learn how Plexigrid consolidated 4 databases into Postgres and got 350x faster queries.

You've added indexes. You've partitioned tables. You've tuned autovacuum within an inch of its life. Performance improves for a few months, and then the dashboards go red again. Sound familiar?

If so, you're probably not doing anything wrong. You're running a workload that vanilla Postgres was never designed for, and no amount of configuration will change that.

It's not transactional. It's not a data warehouse. It's analytics on live data: high-frequency ingestion that stays operationally queryable. If you're running this pattern, you've already been through the cycle: add indexes, partition tables, tune autovacuum, upgrade instances. Each fix buys a few months. Then the metrics climb again.

The longer you wait, the harder the migration. At 10M rows it takes days. At 500M rows, weeks. At 1B+, months. Recognizing the pattern early is the highest-leverage decision you can make.

This post describes six characteristics that define this workload. If four or five apply to your system, the friction is architectural, not operational. (For a deeper look at how purpose-built time-series architecture addresses these constraints, see the Tiger Data architecture whitepaper.)

Continuous High-Frequency Ingestion

The database is absorbing thousands to hundreds of thousands of inserts per second. Not in bursts. Not during a nightly ETL window. Continuously, 24/7.

Consider a semiconductor fab with 8,000 CNC machines and inspection stations on the floor, each reporting vibration, temperature, spindle speed, and tool wear every 2 seconds. That's 4,000 inserts/sec from a single facility. Add process control events, quality inspection results, and environmental monitoring across three plants, and you're at 30-50K inserts/sec before accounting for growth.

-- What a single station's insert stream looks like
INSERT INTO machine_telemetry (ts, station_id, metric, value)
VALUES
  (now(), 'CNC-4401', 'vibration_mm_s', 2.34),
  (now(), 'CNC-4401', 'spindle_rpm', 12045),
  (now(), 'CNC-4401', 'coolant_temp_c', 31.2),
  (now(), 'CNC-4401', 'tool_wear_pct', 67.8);
-- Multiply by 8,000 stations × 0.5 Hz × 3 facilities

This matters because Postgres needs breathing room to run maintenance. Autovacuum, index maintenance, statistics collection. Continuous ingestion means maintenance always competes with writes. There is no off-peak window.

Queries Revolve Around Time

Nearly every row has a timestamp, and nearly every query filters on a time range. Last 30 minutes. This week versus last week. Everything between two dates.

A trading platform captures every order, fill, and cancellation across multiple venues. The operations team monitors execution quality in real time. The compliance team audits historical patterns. Both teams write queries that look like this:

-- Operations: real-time execution quality
SELECT venue, avg(fill_latency_us), percentile_cont(0.99)
  WITHIN GROUP (ORDER BY fill_latency_us)
FROM executions
WHERE ts > now() - interval '15 minutes'
GROUP BY venue;

-- Compliance: historical pattern detection
SELECT account_id, count(*) as cancel_count
FROM order_events
WHERE ts BETWEEN '2025-01-01' AND '2025-03-31'
  AND event_type = 'cancel'
  AND cancel_reason = 'client_requested'
GROUP BY account_id
HAVING count(*) > 500;

Time is the primary axis for both storage and retrieval. General-purpose B-tree indexes aren't built for this access pattern, which is why teams end up building manual partitioning schemes and custom tooling to get time-range queries to perform.

Data Is Append-Only

Once a row lands, it doesn't change. Sensor readings are immutable. Financial transactions don't get updated. Log entries are permanent. When data gets removed, it happens in bulk: drop an entire month's partition, not individual rows.

A wind farm operator collects turbine performance data: blade pitch, rotor speed, power output, nacelle orientation. Once recorded, these readings are facts. They never get corrected or overwritten.

-- This is the entire write pattern. INSERT. No UPDATE. No single-row DELETE.
INSERT INTO turbine_readings
  (ts, turbine_id, blade_pitch_deg, rotor_rpm, power_kw, wind_speed_ms)
VALUES
  (now(), 'WT-112', 12.4, 14.2, 2840, 11.3);

-- Data removal is always bulk
DROP TABLE turbine_readings_2023_q1;

Every row you insert carries 23 bytes of MVCC transaction metadata, on data you will never update. Autovacuum scans these tables constantly, cleaning up dead tuples that were never created through updates. At 50K inserts/sec, that's MVCC overhead on 4.3 billion rows per day that will never be modified. You're paying the full cost of a concurrency model designed for workloads that look nothing like yours.

Retention Is Measured in Months or Years

Seven years of financial records for compliance. Quarters of manufacturing data for root cause analysis. Two-plus years of training data for ML pipelines.

A pharmaceutical manufacturer tracks environmental conditions (temperature, humidity, particulate count) across cleanroom facilities to meet FDA 21 CFR Part 11 requirements. When a batch fails quality control six months after production, the investigation pulls environmental data from the exact time window the batch was in each room.

-- Root cause investigation: what were cleanroom conditions
-- during a batch produced 6 months ago?
SELECT room_id, avg(temp_c), max(particulate_count),
  bool_or(humidity_pct > 45) as humidity_excursion
FROM cleanroom_environment
WHERE ts BETWEEN '2025-08-14 06:00' AND '2025-08-14 18:00'
  AND facility = 'building_3'
GROUP BY room_id;

Short retention hides architectural problems because old data ages out. Long retention removes that escape valve. At 50K inserts per second, that's 1.5 billion rows per year. After three years: 4.5 billion rows.

Queries Are Latency-Sensitive

This data isn't sitting in cold storage waiting for a weekly report. It's being queried actively, under latency constraints.

A SaaS observability platform collects metrics from thousands of customer deployments. The product serves real-time dashboards, automated alerting, and deep-dive investigation, all from the same database. Latency expectations form a gradient:

-- Dashboard widget: last 5 minutes, needs < 100ms response
SELECT host_id, avg(cpu_pct), max(mem_used_bytes)
FROM host_metrics
WHERE ts > now() - interval '5 minutes'
  AND customer_id = 'cust_8821'
GROUP BY host_id;

-- Alert evaluation: last hour, needs < 500ms
SELECT host_id, avg(cpu_pct)
FROM host_metrics
WHERE ts > now() - interval '1 hour'
  AND customer_id = 'cust_8821'
GROUP BY host_id
HAVING avg(cpu_pct) > 90;

-- Incident investigation: last 3 months, seconds acceptable
SELECT date_trunc('hour', ts), avg(cpu_pct), avg(mem_used_bytes)
FROM host_metrics
WHERE ts > now() - interval '90 days'
  AND host_id = 'host-a3f9c'
GROUP BY 1 ORDER BY 1;

Data warehouse scope with operational latency requirements. All from a single system.

Growth Is Sustained

Data volume growing 50-100%+ year over year on a predictable curve. Static workloads can be over-provisioned once and left alone. Growing workloads demand constant re-optimization.

A logistics company tracks GPS position, engine diagnostics, and cargo conditions across a fleet of refrigerated trucks. They started with 200 trucks. Expansion added 150 trucks in year one, another 300 in year two. Each truck reports every 10 seconds.

Year 1:  200 trucks ×  6 readings/min × 1,440 min/day = 1.7M rows/day
Year 2:  350 trucks ×  6 readings/min × 1,440 min/day = 3.0M rows/day
Year 3:  650 trucks ×  6 readings/min × 1,440 min/day = 5.6M rows/day

Cumulative after 3 years: ~3.8 billion rows

Every optimization you ship today is solving for a table size you'll blow past in six months. The treadmill doesn't stop.

What to Do With This

Count how many of these characteristics describe your system. If it's two or three, standard Postgres optimization should have a real impact. The architecture fits your workload. Better indexes, smarter queries, autovacuum tuning. The usual playbook works.

If it's four or five, however, the friction is architectural, not operational. You don't need to abandon Postgres. Tiger Data extends vanilla Postgres to handle exactly this workload. You keep SQL, your extensions, your team's expertise, and the entire Postgres ecosystem. What changes is the storage engine, partitioning, and query planning underneath.

The numbers bear this out. In benchmarks against vanilla PostgreSQL at one billion rows, TimescaleDB delivered up to 1,000x faster query performance while reducing storage by 90% through native compression. Ingest throughput stays constant past 10 billion rows, while PostgreSQL's performance degrades as indexed tables outgrow memory (throughput that starts at 100K+ rows/sec can crash to hundreds). On Azure infrastructure running RTABench workloads, Tiger Cloud was 1,200x faster than vanilla PostgreSQL across 40 real-time analytics queries. These aren't synthetic edge cases. They're the exact query patterns this post describes: time-range filters, aggregations, selective scans on growing datasets.

This post is part of a series on Postgres performance limits for high-frequency data workloads. The full analysis, including a workload scoring framework and migration complexity breakdown at different scales, is in the anchor essay: Understanding Postgres Performance Limits for Analytics on Live Data. Ready to test it on your own data? Start a free Tiger Data trial.

Search is one of those problems that’s deceptively hard. You think you can just LIKE '%query%' your way through it, and then you spend three months learning why that doesn’t work.

Here’s the problem: sometimes users search with exact keywords like “PostgreSQL error 23505”. Other times they search with the meaning: “why is my database slow”. Most of the time, it’s somewhere in between.

Documents are the same way. Some are full of specific terms and jargon. Others are conversational and conceptual. Most are a mix of both.

So you have queries that could be keywords or meaning, hitting documents that could be keywords or meaning. That’s four combinations:

No single approach covers all four quadrants. You need both keyword search AND vector search. And you need a way to combine them intelligently.

That’s exactly what Elasticsearch does. It uses BM25 for keyword ranking, vector embeddings for semantic search, and RRF (Reciprocal Rank Fusion) to merge the results into a single ranked list. This combination is called hybrid search, and it’s why Elasticsearch actually works.

But here’s the trade-off: to use Elasticsearch, you need to build a pipeline. Your data lives in Postgres, but search lives in Elasticsearch. So you’re stuck with:

Postgres → Kafka/Debezium → Elasticsearch

That’s three systems to manage. Three things that can break. Sync jobs to maintain. Stale data to debug. And with AI agents now needing to search through docs and codebases on the fly, the pipeline problem is getting worse. You can’t easily spin up a test environment when your search lives in a completely different system.

Still, teams pay for it. Over a billion dollars a year, collectively. Because search that works is worth the pain.

Here’s the good news: all three pieces of Elasticsearch’s hybrid search are now available in Postgres: 

• BM25 via pg_textsearch (open source, PostgreSQL license) 
• Vector search via pgvectorscale (high-performance DiskANN) 
• RRF? That’s just SQL. No extension needed.

And pgai eliminates the embedding pipeline entirely (no more Postgres → Kafka → Elasticsearch sync jobs). It automatically syncs changes to the data and updates the corresponding embeddings appropriately.

We’ve already covered how BM25 works. This blog focuses on Hybrid Search, RRF, pgai, how they all work together, why it’s elegant, and how to implement hybrid search entirely in Postgres.

How RRF (Reciprocal Rank Fusion) Works

RRF (Reciprocal Rank Fusion) is elegantly simple. It’s the industry standard for combining ranked lists, and it’s what Elasticsearch uses for hybrid search.

The Problem

You run two searches and get two ranked lists:

BM25 Results (keyword): 1. Doc A (score: 15.2) 2. Doc B (score: 12.1) 3. Doc C (score: 8.4)

Vector Results (semantic): 1. Doc C (distance: 0.12) 2. Doc D (distance: 0.18) 3. Doc A (distance: 0.25)

How do you combine them? You can’t just add the scores. They’re on completely different scales. BM25 scores might be 0-50. Vector distances are 0-2.

The RRF Solution

RRF ignores the actual scores. It only cares about rank position:

RRF_score = Σ (1 / (k + rank))

Where:

• k is a constant (typically 60)
• rank is the position (1st, 2nd, 3rd…)

That’s it. Dead simple.

Worked Example

Let’s calculate RRF scores for our documents:

Result: Doc A and Doc C tie for first place. Why? Because they appeared in both lists. RRF naturally boosts documents that multiple systems agree on.

Why RRF Works So Well

1. Scale-independent. Doesn’t matter if one score is 0-50 and another is 0-2. RRF only looks at order.
2. Rewards consensus. If both keyword AND semantic search agree a doc is relevant, it gets boosted.
3. Preserves outliers. A doc that only appears in one list still gets scored. Nothing is thrown away.
4. The k=60 trick. This constant prevents the #1 result from dominating everything. It smooths the curve.

Hybrid Search in Postgres

Here’s how to implement hybrid search with RRF using pg_textsearch and pgvectorscale.

Setup

-- Enable extensions
CREATE EXTENSION pg_textsearch;       -- Adds BM25 ranking for keyword search
CREATE EXTENSION vectorscale CASCADE; -- Adds DiskANN for fast vector search (includes pgvector)
CREATE EXTENSION ai;                  -- Adds auto-embedding generation (optional but recommended)

-- Create your table
CREATE TABLE documents (
  id SERIAL PRIMARY KEY,
  title TEXT,
  content TEXT,                       -- This column gets BM25 indexed
  embedding vector(1536)              -- This column stores OpenAI embeddings (1536 dimensions)
);

-- Create indexes
CREATE INDEX idx_bm25 ON documents 
  USING bm25(content)                 -- BM25 index on content column
  WITH (text_config = 'english');     -- Use English stemming/stopwords

CREATE INDEX idx_vector ON documents 
  USING diskann(embedding);           -- DiskANN index for fast approximate nearest neighbor

The Hybrid Search Query

WITH 
-- STEP 1: Get top 20 keyword matches using BM25
bm25_results AS (
  SELECT 
    id, 
    ROW_NUMBER() OVER (
      ORDER BY content <@> to_bm25query('database optimization', 'idx_bm25')
    ) as rank                         -- Assign rank 1, 2, 3... based on BM25 score
  FROM documents
  ORDER BY content <@> to_bm25query('database optimization', 'idx_bm25')
  LIMIT 20                            -- Only keep top 20 keyword matches
),

-- STEP 2: Get top 20 semantic matches using vector similarity
vector_results AS (
  SELECT 
    id, 
    ROW_NUMBER() OVER (
      ORDER BY embedding <=> $1       -- $1 is the query embedding (passed from app)
    ) as rank                         -- Assign rank 1, 2, 3... based on vector distance
  FROM documents
  ORDER BY embedding <=> $1           -- <=> is cosine distance operator
  LIMIT 20                            -- Only keep top 20 semantic matches
)

-- STEP 3: Combine both lists using RRF formula
SELECT 
  d.id,
  d.title,
  d.content,
  -- RRF: 1/(k+rank) for each list, summed together
  -- k=60 prevents top results from dominating
  COALESCE(1.0 / (60 + b.rank), 0) +  -- Score from BM25 (0 if not in BM25 results)
  COALESCE(1.0 / (60 + v.rank), 0)    -- Score from vectors (0 if not in vector results)
  as rrf_score
FROM documents d
LEFT JOIN bm25_results b ON d.id = b.id   -- Join BM25 ranks
LEFT JOIN vector_results v ON d.id = v.id -- Join vector ranks
WHERE b.id IS NOT NULL OR v.id IS NOT NULL -- Must appear in at least one list
ORDER BY rrf_score DESC               -- Highest RRF score = most relevant
LIMIT 10;                             -- Return top 10 results

Just one query, two search types, but RRF smooths everything over. This makes your life super simple. What's better? Wrap it into a function called hybrid_search and then you can now call that function.

Wrap It in a Function

-- Reusable function: call hybrid_search('your query', $embedding, 10)
CREATE OR REPLACE FUNCTION hybrid_search(
  query_text TEXT,                    -- The search query (for BM25)
  query_embedding vector(1536),       -- The query embedding (for vectors)
  match_count INT DEFAULT 10          -- How many results to return
)
RETURNS TABLE (id INT, title TEXT, content TEXT, rrf_score FLOAT)
AS $$
  WITH 
  -- BM25 keyword search
  bm25_results AS (
    SELECT id, ROW_NUMBER() OVER (
      ORDER BY content <@> to_bm25query(query_text, 'idx_bm25')
    ) as rank
    FROM documents
    ORDER BY content <@> to_bm25query(query_text, 'idx_bm25')
    LIMIT 20
  ),
  -- Vector semantic search  
  vector_results AS (
    SELECT id, ROW_NUMBER() OVER (
      ORDER BY embedding <=> query_embedding
    ) as rank
    FROM documents
    ORDER BY embedding <=> query_embedding
    LIMIT 20
  )
  -- Combine with RRF
  SELECT 
    d.id, d.title, d.content,
    COALESCE(1.0 / (60 + b.rank), 0) + 
    COALESCE(1.0 / (60 + v.rank), 0) as rrf_score
  FROM documents d
  LEFT JOIN bm25_results b ON d.id = b.id
  LEFT JOIN vector_results v ON d.id = v.id
  WHERE b.id IS NOT NULL OR v.id IS NOT NULL
  ORDER BY rrf_score DESC
  LIMIT match_count;
$$ LANGUAGE SQL;

Now your app code is just:

SELECT * FROM hybrid_search('database optimization', $embedding, 10);

Weighted Hybrid Search

Sometimes you want to favor one search type over the other. Technical docs might benefit from stronger keyword matching. Conversational queries might need more semantic weight. So you can expose all the weights in your function, and then you can call this weighted function.

-- Weighted version: control how much keyword vs semantic matters
CREATE OR REPLACE FUNCTION weighted_hybrid_search(
  query_text TEXT,                    -- The search query (for BM25)
  query_embedding vector(1536),       -- The query embedding (for vectors)
  bm25_weight FLOAT DEFAULT 0.5,      -- Weight for keyword search (0.0 to 1.0)
  vector_weight FLOAT DEFAULT 0.5,    -- Weight for semantic search (0.0 to 1.0)
  match_count INT DEFAULT 10          -- How many results to return
)
RETURNS TABLE (id INT, title TEXT, content TEXT, rrf_score FLOAT)
AS $$
  WITH 
  -- BM25 keyword search
  bm25_results AS (
    SELECT id, ROW_NUMBER() OVER (
      ORDER BY content <@> to_bm25query(query_text, 'idx_bm25')
    ) as rank
    FROM documents
    ORDER BY content <@> to_bm25query(query_text, 'idx_bm25')
    LIMIT 20
  ),
  -- Vector semantic search
  vector_results AS (
    SELECT id, ROW_NUMBER() OVER (
      ORDER BY embedding <=> query_embedding
    ) as rank
    FROM documents
    ORDER BY embedding <=> query_embedding
    LIMIT 20
  )
  SELECT 
    d.id, d.title, d.content,
    -- Weighted RRF: multiply each score by its weight
    (bm25_weight * COALESCE(1.0 / (60 + b.rank), 0)) +   -- Weighted BM25 score
    (vector_weight * COALESCE(1.0 / (60 + v.rank), 0))   -- Weighted vector score
    as rrf_score
  FROM documents d
  LEFT JOIN bm25_results b ON d.id = b.id
  LEFT JOIN vector_results v ON d.id = v.id
  WHERE b.id IS NOT NULL OR v.id IS NOT NULL
  ORDER BY rrf_score DESC
  LIMIT match_count;
$$ LANGUAGE SQL;

Usage:

-- Equal weight (default): 50% keywords, 50% semantic
SELECT * FROM weighted_hybrid_search('database optimization', $embedding);

-- Favor keywords (70% BM25, 30% vectors)
-- Good for: error codes, specific terms, exact phrases
SELECT * FROM weighted_hybrid_search('error 23505', $embedding, 0.7, 0.3);

-- Favor semantic (30% BM25, 70% vectors)
-- Good for: natural language questions, conceptual queries
SELECT * FROM weighted_hybrid_search('how do I make queries faster', $embedding, 0.3, 0.7);

Auto-Sync Embeddings (No Pipelines)

Remember the pipeline problem from the intro? Postgres → Kafka → Elasticsearch, with separate jobs to generate embeddings?

pgai eliminates pretty much all of that.

When you create a vectorizer, pgai sets up background workers that monitor your table for changes. When a row is inserted or updated, pgai automatically: 

1. Detects the change 

2. Calls the embedding API (OpenAI, Cohere, or local models) 

3. Stores the embedding in a linked table 4. Keeps everything in sync

-- Set up automatic embedding generation
SELECT ai.create_vectorizer(
  'documents'::regclass,              -- Which table to watch for changes
  loading => ai.loading_column(
    column_name => 'content'          -- Which column to generate embeddings from
  ),  
  embedding => ai.embedding_openai(
    model => 'text-embedding-3-small', -- OpenAI model to use
    dimensions => '1536'               -- Output embedding dimensions
  )
);
-- That's it! pgai now watches 'documents' and auto-generates embeddings
-- whenever content changes. No cron jobs. No sync scripts.

That’s it. Now any change to the documents table triggers automatic embedding updates: 

- INSERT into documents → embedding generated for content column 

- UPDATE the content column → embedding regenerated

- DELETE from documents → embedding removed

Embeddings are stored in a linked table (documents_embedding) that pgai creates and manages for you.

Note: On Tiger Data, the vectorizer worker runs automatically as a managed service. If you’re self-hosting, you’ll need to run the pgai vectorizer worker (a Python CLI) to process the embedding queue.

No Kafka. No Debezium. No sync jobs. No “why is my search stale?” debugging sessions at 3 AM.

Your embeddings live next to your data, updated in near real-time, managed by Postgres itself. This is what makes hybrid search in Postgres practical for production.

Get Started

Everything you need is available on Tiger Data:

-- 1. Enable extensions (one-time setup)
CREATE EXTENSION pg_textsearch;        -- BM25 keyword search
CREATE EXTENSION vectorscale CASCADE;  -- Vector search (includes pgvector)

-- 2. Create indexes on your table
CREATE INDEX idx_bm25 ON documents USING bm25(content)
  WITH (text_config = 'english');      -- BM25 index with English stemming
CREATE INDEX idx_vector ON documents 
  USING diskann(embedding);            -- Fast vector similarity index

-- 3. Search! (using the hybrid_search function from this post)
SELECT * FROM hybrid_search('your query', $embedding, 10);

The Bottom Line

Elasticsearch built a billion-dollar business on BM25 and RRF. Those algorithms aren’t proprietary. They’re not even complicated. And now they run natively in Postgres.

No pipelines. No sync jobs. No extra infrastructure. Just your database doing what databases should do: storing your data and making it searchable.

The question isn’t whether Postgres can do hybrid search. It can. The question is: why are you still running two systems when one will do?

Learn More

Our Posts: 

• You Don’t Need Elasticsearch: BM25 is Now in Postgres: Why BM25 beats native Postgres search
• From ts_rank to BM25. Introducing pg_textsearch

Extensions:

• pg_textsearch: BM25 for Postgres, open source (PostgreSQL license) 
• pgvectorscale: High-performance vector search using DiskANN 
• pgvector: The foundation for vector similarity in Postgres 
• pgai: Auto-sync embeddings, RAG workflows

Background: 

• BM25 on Wikipedia: The original algorithm (1994) 
• Reciprocal Rank Fusion paper: The academic paper behind RRF 
• Elasticsearch Hybrid Search docs: How Elasticsearch implements RRF (for comparison)

Get Started:

• Tiger Data Console: Managed Postgres with all extensions pre-installed 
• pg_textsearch Documentation

Think of your database like your home. Your living room, bedroom, kitchen, garage: each room serves a different purpose, but they're all under one roof. You don't build a restaurant across town because you need to cook dinner. You don't rent a commercial garage to park your car.

Postgres works the same way. Search, vectors, time-series, queues, documents, all rooms in one house. Same roof, foundation, and key.

But specialized database vendors have spent years telling you otherwise. "Use the right tool for the right job," they say. It sounds reasonable and wise. And it sells a lot of databases.

The "Right Tool" Trap

You follow the advice. You adopt Elasticsearch for search, Pinecone for vectors, Redis for caching, MongoDB for documents, Kafka for queues, InfluxDB for time-series. Postgres handles whatever's left.

Congratulations. You now have seven databases to manage. Seven query languages to learn. Seven backup strategies to maintain. Seven security models to audit. Seven monitoring dashboards to watch. And seven things that can break at 3 AM.

When something does break? Good luck spinning up a test environment to reproduce it. You'll need synchronized snapshots across all seven systems, all at the same point in time, with seven services running in your local environment. Let us know how that goes.

The AI Era Changed the Math

Here's what makes this argument different in 2026: AI agents.

Think about what agents need to do. Spin up a test database with production data. Try a fix. Verify it works. Tear it down. With one database, that's a single fork command. Fork it, test it, done.

With seven databases? Now your agent needs to coordinate snapshots across every system, spin up seven services, configure seven connection strings, hope nothing drifts while testing, and tear it all down afterward. That's not a minor inconvenience. It's a research project. And it's why most AI coding agents quietly assume a single-database architecture.

This applies beyond agents. Every on-call engineer who needs a test environment at 3 AM faces the same coordination problem. So does every CI pipeline that needs realistic data, and every team that wants to experiment safely.

And it compounds at the organizational level. When your data lives in one database, a new engineer can understand the entire data model in a day. They can run the full stack locally with a single connection string. They can write a migration, test it against a fork of production, and ship it with confidence. When your data is scattered across seven systems, onboarding alone becomes a multi-week project. "How does search stay in sync with the main database?" is a question that shouldn't require a 45-minute architecture review to answer.

Database consolidation used to be an architectural preference. A nice-to-have. In the AI era, it's becoming a functional requirement. The teams that can fork, test, and iterate on a single database will ship faster than teams coordinating across seven.

The Algorithm Reality

Here's what the specialized database vendors don't want you to think too hard about: in most cases, Postgres extensions use the same core algorithms as their products.

The last column is the important part. Those are real, specific requirements you'll recognize because you hit a concrete wall, not because a vendor's marketing team told you to plan for it.

The benchmarks on the extensions that matter most:

• pgvectorscale achieved 28x lower p95 latency and 16x higher throughput than Pinecone at 99% recall on a 50M vector benchmark
• TimescaleDB matches or beats InfluxDB on time-series workloads while giving you full SQL, JOINs, and ACID guarantees
• pg_textsearch runs the same BM25 ranking algorithm that powers Elasticsearch, natively in Postgres

These extensions aren't new. PostGIS has been in production since 2001. TimescaleDB since 2017. pgvector since 2021. Over 48,000 companies run PostgreSQL, including Netflix, Spotify, Uber, and Discord.

So what are you actually paying for with a specialized database? In most cases: a managed service, a purpose-built UI, and the assumption that you'll outgrow Postgres. For the small percentage of teams that genuinely need cluster-scale Elasticsearch or Kafka-grade event streaming, that's a fair trade. For everyone else, it's an infrastructure tax on a problem they don't have.

The Compounding Costs

Even when a specialized database has a genuine edge on a specific benchmark, you're paying for it across every other dimension of your infrastructure.

Cognitive load. Your team needs SQL, Redis commands, Elasticsearch Query DSL, MongoDB aggregation pipelines, Kafka consumer patterns, and InfluxDB's query language. That's not specialization. That's fragmentation.

Data consistency. Keeping Elasticsearch in sync with Postgres means building sync jobs. Those jobs fail silently. Data drifts. You add reconciliation logic. That fails too. Now you're maintaining data plumbing instead of building the product your company actually sells. We've seen this pattern firsthand: a team adds a second database for one workload, then spends six months building and debugging the sync pipeline between them. The second database solved a performance problem. The sync pipeline created three operational ones.

SLA math. Three systems at 99.9% uptime each give you 99.7% combined availability. That's 26 hours of downtime per year instead of 8.7. Every additional system multiplies your failure surface.

Real cost. Plexigrid consolidated from four databases to one and got 350x faster queries. Flogistix cut database costs by 66%. Latitude saved $12,000 per month from compression alone. These savings compound because you're removing entire categories of operational complexity.

Start With Postgres

Start with one database. Add complexity only when you've earned the need for it.

Postgres with the right extensions handles search, vectors, time-series, documents, caching, queues, geospatial data, and scheduled jobs. The algorithms match their specialized counterparts. The extensions are battle-tested. And the operational simplicity compounds every month you don't add another system to your stack.

In the AI era, that simplicity is worth more than ever. Every agent, every CI pipeline, every on-call engineer benefits from being able to fork one database instead of coordinating seven.

Most teams adopt specialized databases before they've even tested whether Postgres can handle the workload. They add Elasticsearch because "that's what you use for search," not because they tried pg_textsearch and found it lacking. They add Redis because "everyone uses Redis for caching," not because UNLOGGED tables couldn't handle their load. The "right tool for the right job" advice sounds wise. But too often, it's a solution to a problem you don't have yet, sold by the people who profit from you having it.

Test the assumption first. Then decide.

For a hands-on guide to setting up each extension with working SQL, read the companion post: Postgres Extensions Cheat Sheet: Replace 7 Databases With SQL.

All of these extensions come pre-configured on Tiger Cloud. Create a free database and start building.

Elasticsearch may work great in initial testing and development but Production is a different story. This blog is about what happens after you ship: the JVM tuning, the shard math, the 3 AM pages, the sync pipelines that break silently. The stuff your ops team lives with.

After years of teams running Elasticsearch in production, certain patterns keep emerging. The same issues show up in blog posts, Stack Overflow questions, and incident reports. We've compiled ten of the most common ones below, with references to the engineers who've documented them. We’ve also added images to make it easy to quickly skim through it and compare the challenges against Postgres. 

TLDR: With great power comes great operational complexity.

1. JVM Garbage Collection Pauses

Elasticsearch runs on the Java Virtual Machine (JVM). This means garbage collection (GC) is part of your life.

The problem: Java periodically pauses everything to clean up unused memory. These "stop-the-world" pauses can freeze your Elasticsearch node for seconds at a time. If the pause lasts longer than 30 seconds, the cluster assumes the node is dead and starts moving data around to compensate. Now you have a cascading failure.

Say for example, your e-commerce site is running on Black Friday. Traffic spikes, memory fills up, and Java decides it's time to clean house. Search goes unresponsive for 45 seconds. The cluster panics, starts redistributing shards, and suddenly you're dealing with a full outage instead of a brief slowdown.

References:

• Sirius Open Source: Problems and Operational Weaknesses of Elasticsearch — "GC pause time correlates with heap size. Larger heaps mean longer pauses."
• Elastic Docs: Fix Common Cluster Issues — Official troubleshooting guide for JVM memory pressure

Why Postgres avoids this: Postgres is written in C and manages memory directly. There's no JVM, no garbage collection, and no "stop-the-world" pauses. Memory usage is predictable and grows linearly with your workload. You still need to tune shared_buffers and work_mem, but you're not fighting a runtime that can freeze your entire process.

2. Mapping Explosion

Elasticsearch tries to be helpful by automatically detecting the type of data you're storing. Send it a field called user_id with a number, and it remembers "user_id is a number." Convenient, right?

The problem: This "helpful" feature becomes a nightmare with semi-structured data. If your application logs include arbitrary metadata keys, or your users can create custom fields, Elasticsearch creates a new mapping for each one. Thousands of fields later, your cluster state is massive, your master node is struggling, and everything slows to a crawl.

Say you're logging API requests and including the request body as JSON. One customer sends a payload with 500 unique keys. Elasticsearch dutifully creates 500 new field mappings. Multiply that by thousands of requests, and suddenly you have tens of thousands of fields. Your cluster becomes unresponsive, and you're scrambling to figure out why.

References:

• Sirius Open Source — "Mapping explosion is a common anti-pattern... the cluster state can grow to hundreds of megabytes."
• MindfulChase: Advanced Troubleshooting for Elasticsearch — Mapping conflicts and silent failures

Why Postgres avoids this: Postgres requires you to define your schema upfront. You decide what columns exist. If you need flexible data, you use JSONB—it stores arbitrary JSON without creating new columns or bloating system catalogs. You can still index specific paths inside the JSON when needed. The schema is explicit, which means surprises are rare.

3. Oversharding (or Undersharding)

Elasticsearch splits your data into "shards" to distribute it across machines. You have to decide how many shards to use when you create an index. Choose wisely, because you can't change it later without rebuilding everything.

The problem: Too many shards? Each one consumes memory, CPU, and file handles. You're wasting resources on overhead. Too few shards? You can't parallelize queries effectively, and recovery takes forever when a node dies. And here's the kicker: there's no formula. It depends on your data size, query patterns, hardware, and growth rate. It's guesswork.

Say you create an index with 5 shards because that's the default. Six months later, your data has grown 10x, and those 5 shards are now bottlenecks. Your only option? Create a new index with more shards and reindex all your data. Hope you have the disk space and time for that migration.

References:

• Pure Insights: Top 7 Elasticsearch Pitfalls — "Improper shard allocation is one of the most common causes of performance degradation."
• Sirius Open Source — "Each shard is a Lucene index with its own file handles, memory, and CPU overhead."

Why Postgres avoids this: Postgres doesn't shard by default—your data lives in tables on a single node. When you need to scale reads, you add replicas. When tables get large, you use declarative partitioning (by time, by tenant, etc.). These decisions can be made later, as your data grows, rather than upfront when you're guessing. And if you do need horizontal sharding someday, tools like Citus exist. But most applications never get there.

4. Deep Pagination Performance Cliff

Users click "next page" a lot. They expect page 500 to load as fast as page 1. Elasticsearch has other plans.

The problem: When you ask for page 500 (results 5000-5010), Elasticsearch doesn't skip to result 5000. It fetches and ranks all 5010 results, then throws away the first 5000. Every node in your cluster does this work, then sends results to a coordinator that combines them. The deeper you paginate, the more work everyone does.

Say a user searches your product catalog and starts browsing. Page 1 loads in 50ms. Page 10 takes 200ms. By page 100, it's taking 2 seconds. Page 500? Timeout. Elasticsearch actually has a hard limit (index.max_result_window, default 10,000) specifically because this gets so bad.

References:

• Sirius Open Source — "Deep pagination is a well-known limitation... the coordinating node must aggregate results from all shards."
• Elastic Docs — Official guidance recommending search_after for deep pagination

Why Postgres avoids this: LIMIT 10 OFFSET 5000 in Postgres doesn't have a hard ceiling like ES. The query planner handles it without needing to score every document. That said, large offsets still have overhead—keyset pagination (WHERE id > last_seen_id) is better for deep pages. But you won't hit a wall at 10,000 results, and you won't see the dramatic slowdown ES has.

5. Split-Brain and Data Loss

Elasticsearch is a distributed system. Distributed systems fail in distributed ways.

The problem: Imagine your cluster has 5 nodes, and a network issue splits them into two groups: 3 nodes on one side, 2 on the other. Both groups might elect their own master and start accepting writes independently. When the network heals, you have two conflicting versions of your data. This is "split-brain," and it can cause permanent data loss.

Say your Elasticsearch cluster spans two data centers for redundancy. A network blip between them lasts 60 seconds. Both sides elect a master. Users on both sides keep searching and indexing. When connectivity returns, some documents exist in one half, some in the other, and some have conflicting updates. Reconciling this manually is a nightmare.

References:

• Sirius Open Source — "Split-brain scenarios can occur when network partitions isolate nodes."
• Wikipedia: Elasticsearch — History of split-brain issues and improvements

Why Postgres avoids this: Postgres uses a simpler primary/replica model. There's one primary that accepts writes; replicas are read-only. This doesn't make split-brain impossible—any distributed system can have it—but the architecture makes it much harder to create accidentally. Tools like Patroni, pg_auto_failover, and managed services (like Tiger Data) handle leader election and fencing automatically. You're not managing quorum math yourself.

6. Eventual Consistency Surprises

When you save something to Elasticsearch, it's not immediately searchable. This catches a lot of teams off guard.

The problem: Elasticsearch buffers writes and periodically "refreshes" to make them searchable (default: every 1 second). But under load, or with specific configurations, that delay can grow. Users create a record, immediately search for it, and get nothing. "But I just saved it!"

Say a user posts a comment on your platform. Your app saves it to Postgres (source of truth) and indexes it to Elasticsearch. The user refreshes the page to see their comment. The Postgres query shows it, but the search results don't include it yet because Elasticsearch hasn't been refreshed. The user thinks something is broken and posts again. Now you have duplicate comments.

References:

• AceTheCloud: Why You Rarely Need Elastic — "Elasticsearch does not support ACID transactions... data may not be immediately visible."
• StackShare: Elasticsearch vs PostgreSQL — Comparison of consistency models

Why Postgres avoids this: Postgres is ACID-compliant. When your transaction commits, the data is immediately visible to all subsequent queries—on the same connection or any other. If you're using BM25 search via pg_textsearch, the index updates synchronously—no refresh interval. No eventual consistency, no "I just saved it but can't find it" bugs.

7. Security Misconfigurations

Elasticsearch has been responsible for some of the largest data breaches in history. Not because it's insecure by design, but because it's easy to misconfigure.

The problem: For years, Elasticsearch shipped with no authentication by default. Spin up a cluster, expose port 9200, and anyone on the internet could read your data. While newer versions have improved, the damage is done: countless clusters were exposed, and security misconfigurations remain common.

Here's a real example: in 2019, researchers found an exposed Elasticsearch server containing 1.2 billion records of personal data—social media profiles, email addresses, phone numbers. The cluster had no authentication. This wasn't a sophisticated hack; someone just found an open port. Similar breaches happen regularly.

References:

• Pure Insights — "Clusters without TLS, authentication, or access controls are vulnerable."
• HevoData: Elasticsearch PostgreSQL — "By default, Elasticsearch lacks built-in authentication."

Why Postgres avoids this: Postgres has required authentication from day one. You literally cannot connect without credentials—there's no "open by default" mode. SSL/TLS encryption, role-based access control, and row-level security are all built in and battle-tested over decades. Misconfiguration is still possible (any system can be misconfigured), but the secure path is the default path.

8. Monitoring Complexity

Elasticsearch gives you hundreds of metrics. The problem is knowing which ones matter.

The problem: JVM heap usage, young GC time, old GC time, thread pool queue sizes, circuit breaker trips, pending tasks, unassigned shards, indexing rate, search latency, segment count... Elasticsearch exposes all of this, but understanding what's normal vs. concerning requires deep expertise. Most teams set up basic monitoring, miss the early warning signs, and only find out something's wrong when it's already broken.

Say your cluster has been running fine for months. One day, search latency spikes. You check CPU and memory—they look okay. Disk? Fine. What you didn't notice: the "pending tasks" queue has been growing for days, and the "circuit breaker" tripped twice last week. By the time you figure this out, users have been complaining for hours.

References:

• Pure Insights — "Without proper observability, issues like JVM heap spikes or unassigned shards can escalate into outages."
• Elastic Docs — Guide to understanding circuit breaker exceptions and task queue backlogs

Why Postgres avoids this: Postgres monitoring is well-understood after 30+ years. The key metrics are simpler: connections, query latency, cache hit ratio, replication lag. pg_stat_statements shows your slowest queries. EXPLAIN ANALYZE tells you exactly why. Tools like pgBadger, pg_stat_activity, and any standard APM will get you 90% of the way there. You don't need to become a Postgres internals expert to keep it healthy.

9. Data Pipeline Sync Issues

Your real data lives in Postgres. Elasticsearch is just a copy for search. Keeping them in sync is harder than it sounds.

The problem: The typical architecture is Postgres → Kafka/Debezium → Elasticsearch. That's three systems. When data updates in Postgres, it flows through Kafka, gets transformed, and lands in Elasticsearch. What could go wrong? Everything. Kafka lag causes stale search results. Schema changes break the connector. A bug in your transformer corrupts data. And when something breaks, you're debugging across three different systems with three different log formats.

Say a customer updates their email address. The change saves to Postgres immediately. But the Kafka consumer is backed up, and Elasticsearch doesn't get the update for 10 minutes. During that window, any email sent via a search lookup goes to the old address. Customer support gets an angry call. You spend two hours figuring out where the delay happened.

References:

• InfoQ: Instacart Elasticsearch Postgres — Case study on data synchronization challenges
• Medium: Real-time Integration of PostgreSQL with Elasticsearch — Discussion of sync tooling and challenges

Why Postgres avoids this: If search lives in Postgres (using BM25 via pg_textsearch), there's no pipeline to manage. When you update a row, the search index updates in the same transaction. No Kafka, no Debezium, no sync jobs, no lag, no "which system has the latest data?" debugging. Your source of truth and your search index are the same thing.

10. Infrastructure Cost

Elasticsearch clusters are expensive to run. This isn't a bug—it's the architecture.

The problem: ES runs on the JVM, which means each node needs significant RAM (typically 16-64GB, with half allocated to heap). Production clusters require at least 3 nodes for high availability and quorum. Larger deployments need dedicated master nodes, coordinating nodes, and data nodes—that's a lot of instances. And because ES stores everything in Lucene segments, you need fast SSDs to avoid I/O bottlenecks. Add replicas for redundancy, and your storage costs multiply.

Say you spin up a "small" production cluster: 3 nodes, 32GB RAM each, SSDs. That's your baseline before you've even scaled. Now add a separate cluster for staging. And one for testing. Your cloud bill starts climbing, and you're not even at high traffic yet.

References:

• Elastic Cloud Pricing — Managed ES starts at ~$95/month for minimal configs, scales quickly
• AWS OpenSearch Pricing — Instance costs add up fast with multi-node clusters
• Sirius Open Source — "Each shard is a Lucene index with its own file handles, memory, and CPU overhead"

Why Postgres avoids this: Postgres runs on a single node for most workloads. No JVM overhead means lower memory requirements. You add read replicas when you actually need them, not because the architecture demands it. A single Postgres instance with pg_textsearch can handle substantial search workloads on hardware that would barely run one ES node. And if you're on a managed service like Tiger Data, you're paying for one database, not a distributed cluster.

The Alternative: Keep Search in Postgres

If you've made it this far, you've probably nodded along to at least a few of these issues. Maybe all ten. The question is: what do you do about it?

Can you overcome all these issues? Yes. Hire an Elasticsearch specialist (or a team of them). Invest in training. Build expertise over years. Companies do it every day.

But here's the question: do you want to?

It's like the shift from gas cars to electric cars. When you go electric, you don't just swap out the engine. You eliminate an entire category of problems: no oil changes, no transmission repairs, no spark plugs, no fuel injectors, no exhaust systems. The complexity just disappears.

That's what's happening with search. Extensions like pg_textsearch (BM25 ranking), pgvectorscale (vector search), and pgai (automatic embeddings) mean search can now live inside Postgres. No separate cluster. No JVM. No data pipelines. No sync jobs. A whole category of problems just goes away.

Every issue above has the same root cause: Elasticsearch is a separate system. Separate systems mean separate infrastructure, separate expertise, separate failure modes, and separate pipelines.

What if you didn't need a separate system?

PostgreSQL now has:

• BM25 ranking via pg_textsearch (same algorithm Elasticsearch uses)
• Vector search via pgvector and pgvectorscale
• Hybrid search combining both, with RRF (Reciprocal Rank Fusion) in pure SQL
• Automatic embedding sync via pgai

And for AI workflows, there's a bonus: when everything lives in Postgres, you can fork your database and get an instant copy of your search index, vectors, and embeddings. Try doing that with Elasticsearch.

For many workloads, "just use Postgres" isn't a compromise. It's a simplification.

One caveat: pg_textsearch doesn't currently support faceted search, so e-commerce with "filter by brand/price/color" is still a gap. For RAG, document search, and knowledge bases, you're covered.

Get Started in 5 Minutes

Ready to try hybrid search in Postgres? Here's all you need:

-- 1. Enable extensions
CREATE EXTENSION pg_textsearch;        -- BM25 keyword search
CREATE EXTENSION vectorscale CASCADE;  -- Vector search (includes pgvector)
CREATE EXTENSION ai;                   -- Auto-embedding generation (optional)

-- 2. Create your table
CREATE TABLE documents (
  id SERIAL PRIMARY KEY,
  title TEXT,
  content TEXT,
  embedding vector(1536)
);

-- 3. Create indexes
CREATE INDEX idx_bm25 ON documents USING bm25(content)
  WITH (text_config = 'english');
CREATE INDEX idx_vector ON documents USING diskann(embedding);

-- 4. Hybrid search with RRF (one query, two search types)
WITH 
bm25_results AS (
  SELECT id, ROW_NUMBER() OVER (
    ORDER BY content <@> to_bm25query('your search query', 'idx_bm25')
  ) as rank
  FROM documents
  LIMIT 20
),
vector_results AS (
  SELECT id, ROW_NUMBER() OVER (
    ORDER BY embedding <=> $1  -- $1 is your query embedding
  ) as rank
  FROM documents
  LIMIT 20
)
SELECT d.*, 
  COALESCE(1.0/(60 + b.rank), 0) + COALESCE(1.0/(60 + v.rank), 0) as rrf_score
FROM documents d
LEFT JOIN bm25_results b ON d.id = b.id
LEFT JOIN vector_results v ON d.id = v.id
WHERE b.id IS NOT NULL OR v.id IS NOT NULL
ORDER BY rrf_score DESC
LIMIT 10;

That's it. BM25 + vectors + RRF, all in Postgres. No Elasticsearch, no Kafka, no sync jobs.

Try it on Tiger Data—all extensions come pre-installed.

Learn More

• pg_textsearch on GitHub — BM25 for Postgres (open source)
• pgvectorscale on GitHub — High-performance vector search

Last month, we launched Agentic Postgres, the first Postgres database designed for AI agents. It includes an MCP server that gives agents direct access to your databases, instant zero-copy forks, Postgres and TimescaleDB documentation search, and more. 

Alongside Agentic Postgres, we shipped a brand new CLI: Tiger CLI. It's how you manage your Tiger Cloud databases from your favorite terminal. 

The basics work like you'd expect: 

# Install Tiger CLI 
curl -fsSL https://cli.tigerdata.com | sh

# Authenticate
tiger auth login

# Create a new database service
tiger service create --name my-database

# Connect to your database
tiger db connect

# Get your connection string
tiger db connection-string

# List all your services
tiger service list

These commands cover most day-to-day workflows. But Tiger CLI has a few features that makes the agentic development workflow significantly more intuitive, and you're probably not using them yet! 

Here are five new features we launched that you aren’t using, but should: 

1. Let your AI manage your databases: Install an MCP server that gives your AI assistant direct access to create services, run queries, and check connections
2. Turn your AI into a Postgres expert: Skills teach your AI Postgres best practices automatically, as if it’s been writing production-grade PostgreSQL for a decade+. 
3. Fork any database in seconds: Create zero-copy clones of your database for testing migrations or spinning up staging environments
4. Search Postgres docs from your editor: Your AI can search PostgreSQL (across all versions) and TimescaleDB documentation without leaving your IDE or CLI
5. Run SQL queries through your AI: Execute queries against your database directly from your AI assistant

Let's look at each one.

Let Your AI Manage Your Databases

As anyone coding with Cursor or Claude Code knows, nothing breaks flow more than having to leave your AI agent to execute CLI commands. Every time you need to check a database connection string, list your available services, or create a new database, you need to switch context. From your terminal, to the browser, back to your IDE, it’s easy to break out of your flow state. 

The Tiger CLI now includes a Model Context Protocol (MCP) server. If you're using an AI coding assistant like Claude Code, Cursor, or VS Code with Copilot, you can give it direct access to your Tiger Cloud databases.

This means your AI assistant can list your services, run SQL queries, create new databases, and check connection details, all without you switching to your terminal.

Quick Setup

Install the MCP server for your assistant:

# Interactive (prompts you to pick your client)
tiger mcp install

# Or specify directly
tiger mcp install claude-code
tiger mcp install cursor
tiger mcp install vscode

We made it super easy to install the Tiger CLI in all of your favorite coding assistants* through an interactive prompt that guides you through the installation.

Restart your AI assistant after installation. 

* We are constantly adding interactive installations for new coding assistants! 

Adding to Cursor via UI

If you prefer to configure Cursor (or other IDEs) manually instead of using tiger mcp install:

1. Open Cursor Settings
2. Look for “Tools & MCP” on the left sidebar
3. Click "Add MCP server"
4. Enter the following configuration: 
   • Name: tiger
   • Command: tiger
   • Arguments: mcp, start

5. Click "Save" and restart Cursor

Once configured, you'll see the Tiger MCP server listed in your MCP servers. The green indicator shows it's connected and ready.

What You Can Do

Once installed, your AI assistant has access to tools like:

• service_list — List all your database services
• service_get — Get details about a specific service
• service_create — Create a new database
• db_execute_query — Run SQL queries against any service

For a full list of tools available to you and your agent, they are available in the Tiger CLI README on Github. 

For example, you can ask your AI assistant: "Show me all my Tiger Cloud services" or "Run SELECT count(*) FROM events on my production database."

The MCP server uses your existing CLI authentication, so there's no extra setup after tiger auth login.

Turn Your AI Into a Postgres Expert

A new pattern of working with SQL has emerged as agentic coding has exploded in popularity. 

You can now simply just tell an LLM what you want to do with your database, and it will write the SQL for you. You basically don’t even need to spend sweat, tears, or even fear (like you accidentally dropping a table, although this is still completely within the realm of possibility when working with an agent) to learn how to write SQL. 

But this AI-generated SQL actually has spawned a new problem, which is, AI-generated SQL works… until it doesn’t. 

Your schema may pass tests, your queries may run… but six months later, as your application scales to millions of users, everything slows to a crawl. 

The problem here is that LLMs are actually trained on millions of lines of SQL, absorbed from a billion blog posts (of differing quality), so while they “know” SQL, they don’t actually know what patterns actually scale. There are also hundreds of different dialects of SQL, with dozens of versions each. 

It’s no wonder AI agents may not write the best SQL. 

We’ve personally seen many common AI-generated mistakes, including: 

• Using VARCHAR(255) instead of TEXT (the length limit doesn't help performance in Postgres)
• Using SERIAL instead of BIGINT GENERATED ALWAYS AS IDENTITY
• Missing indexes on foreign key columns (Postgres doesn't create these automatically)
• Using TIMESTAMP instead of TIMESTAMPTZ (timezone handling is painful to fix later)

These will not raise syntax or linter errors. Your tests will still pass. But trying to fix these mistakes later once your app is handling millions of users, means painful migrations, downtime, and explaining to your CEO why the database needs to go down for all your users to undergo maintenance. Sound familiar? 

The Tiger MCP server ships with a variation of Skills (a standard built by Anthropic) written by our most senior and experienced Postgres engineers. When your AI needs to design a schema, the MCP server automatically pulls the right “lessons” and then applies 30 years of Postgres best practices to your database design. 

Available Skills

The MCP server includes skills for common Postgres and TimescaleDB tasks. 

• design-postgres-tables: Schema design with proper types, constraints, and indexes
• setup-timescaledb-hypertables: Hypertable configuration, compression, retention policies
• migrate-postgres-tables-to-hypertables: Converting existing tables to hypertables
• find-hypertable-candidates: Identifying which tables should become hypertables

Your AI discovers and uses these automatically based on what you ask it to do. You don’t need to call them explicitly! 

We are continually adding new Skills. Want to request new ones or contribute with your own? Feel free to create an issue in our Skills Github repo. 

Fork Any Database in Seconds

Testing database migrations against production data is risky, but testing against fake data is also risky because you may miss edge cases. You need a real copy of your database but without the cost or time of duplicating everything manually. 

Tiger CLI lets you create instant, zero-copy forks of any database: 

tiger service fork <service-id> --name my-staging-db

The fork created is a point-in-time copy that shares underlying data blocks with the original. You only pay for blocks that change. This makes forks lightweight enough to spin up for a single test and throw away when you’re done. 

Database forking is especially useful when working in an agentic context, where if you are using multiple agents at once, you can fork multiple databases for different agents to work on without affecting the original database. Neat! 

Example Workflow

# List your services to find the source ID
tiger service list

# Fork your production database
tiger service fork --source-id svc_abc123 --name testing-migrations

# Connect to the fork
tiger db connect --service-id <new-service-id>

# Run your migration, test it, then delete the fork when done
tiger service delete <new-service-id>

Search Postgres Docs From Your Editor

Postgres has been around for almost 30 years.

The documentation is incredibly extensive, spanning 18 versions and countless individual releases. A function that exists in Postgres 18 might not exist in 15. Syntax that worked in 14 now has better alternatives in 18. 

Most AI-powered documentation search tools don’t account for these idiosyncrasies. 

If not using docs search tools, and simply relying on an LLM’s own training data, well, they have training cutoffs, generally at least lagging by 6 months, so LLMs often suggest older methods that aren’t the most performant, or reference features that don’t exist in your version. This is actually a major problem with LLMs writing React and NextJS I’ve personally experienced in my side projects! 

The Tiger MCP server includes search over PostgreSQL documentation from versions 14 to 18 and TimescaleDB docs for time series workloads. Your AI assistant can search version-specific docs without leaving your coding agent. 

Your assistant gets access to:

• semantic_search_postgres_docs: Search PostgreSQL documentation (versions 14-18)
• semantic_search_tiger_docs: Search Tiger Cloud and TimescaleDB documentation

No More Context Switching 

Instead of leaving your editor to search docs, you can ask your AI assistant directly:

• "How do I set up continuous aggregates in TimescaleDB?"
• "What's the syntax for PostgreSQL window functions?"
• "Show me how to configure compression policies"

The assistant searches the actual docs and gives you accurate, up-to-date answers. You don’t even need to ask the agent to explicitly use the documentation search feature, it will just work. 

P.S. This feature is enabled by default. If you ever need to disable it:

tiger config set docs_mcp false

Run SQL Queries Through Your AI

You’re debugging an issue, you need to check row count, inspect a table’s schema, or run a quick query. You’re used to having to open a new terminal, find and remember the connection string, connect via psql, run the query, copy the results back (often jankily when done inside the CLI due to text wrapping). All these steps just for getting ONE number. 

No more. 

Once you’ve set up the MCP in your coding agent, your AI assistant can execute SQL queries (using the db_execute_query tool) directly against your databases.

This means you can stay in your editor and ask your AI:

• "How many events came in during the last 24 hours?”
• "Show me the 10 most recent orders"
• "What's the schema of the users table?" 

Example

Your AI writes the (performant) SQL, runs it, and returns the results. No more terminal switching, copy-pasting connection strings, remembering your environment, exact syntax for your information_schema. 

Get Started Now

Install the Tiger CLI and MCP server:

curl -fsSL https://cli.tigerdata.com | sh
tiger auth login
tiger mcp install

Then select your AI assistant (Claude Code, Cursor, VS Code, Windsurf) and you're ready to go.

Don't have a Tiger Cloud account? Sign up for free — no credit card required. Create your first database, then try out these CLI features. 

Resources

1. Postgres for Agents 
2. How to Train Your Agent to Be a Postgres Expert

Data infrastructure requirements are changing. Modern applications combine operational transactions, time-series, events and vector data all while integrating the lakehouse backbone for model training, historical analytics, and enterprise insights.

Developers building modern applications start with Postgres. It’s familiar, flexible, and handles transactions well. And every framework, ORM, and service on AWS speaks fluent Postgres.

But as workloads evolve, teams outgrow what vanilla Postgres can comfortably handle. Companies ingest more data, expand real-time analytics, store longer histories, and increasingly support AI-driven features. To accommodate new requirements, developers incorporate specialized data store technologies alongside extended pipelines required to keep data in-sync. Over time, data drifts, operational burden increases and developer velocity tanks as teams spend more time gluing databases together and maintaining complex architectures rather than building new applications.

Developers on AWS want a data layer that unifies transactional, analytical and agentic workloads on top of Postgres and integrates seamlessly within the broader AWS ecosystem.

That’s why we’re partnering with AWS: to jointly solve data fragmentation and deliver the unified data infrastructure developers have been asking for. Through our Strategic Collaboration Agreement announced last week, we’re working with AWS to provide this unified infrastructure to every team building modern applications.

In this blog post, we’re showing the incredible progress we have made over the last 12 months to make this architecture native to AWS including the announcement of two new releases: Tiger Lake public beta and S3 Connector GA.

Postgres, Extended for Modern Workloads on AWS

Developers on AWS already rely on Postgres to power their operational systems. Tiger Data builds on that foundation by extending Postgres to support time-series, vector and full-text search, advanced analytics, and tight lakehouse integration, all while fitting naturally into the AWS ecosystem through our growing collaboration.

The result is Postgres that does everything developers wish Postgres could do, without introducing new query languages, new operational paradigms, or new systems to manage.

At the core of this extended engine are four pillars:

TimescaleDB brings a purpose-built time-series engine directly into Postgres. It handles massive ingest rates, automatic partitioning, hybrid row–column storage with 90%+ compression, SIMD-accelerated scans, incremental materialized views, and keeps recent data fast while automatically tiering older data to Amazon S3. It also includes Hyperfunctions, a rich set of SQL analytics functions for things like statistical aggregates, percentile approximations, gap-filling, and downsampling, so teams can run advanced time-series analytics directly in Postgres without external systems or pipelines.

pgvectorscale introduces high-performance vector search with a DiskANN index, supporting large, high-dimensional embedding workloads with fast filtering and optimized storage. Instead of deploying a separate vector database, developers can store embeddings alongside time-series and relational context, enabling AI-driven features on top of unified data.

pg_textsearch provides modern full-text search powered by a BM25 ranking model and a memtable architecture that enables fast incremental indexing and low-latency search, right inside Postgres.

Layered on top of these engines is deep integration with the AWS platform for secure connectivity, streaming and batch ingest, observability, analytics, AI and billing, which has been a key area of focus for us over the last 12 months.

A Year of Deepening AWS Integrations

The past 12 months of collaboration with AWS were shaped around one theme:

Make Tiger Cloud feel native inside AWS architectures.

We approached this holistically spanning secure connectivity, observability, billing, ingest and finally lakehouse interoperability.

Fast Ingest

Ingest has been one of the biggest areas of investment this year. Across different industries, AWS customers generate time-series data in many ways: IoT fleets send telemetry through IoT Core or drop device exports into S3, financial firms stream market data through Kafka or Amazon MSK, and application teams accumulate large volumes of event or log data that regularly land in S3. These are different workloads and different customers, but they all shared the same pain: getting time-stamped data into Postgres required a patchwork of custom pipelines, consumers, and periodic backfills that were hard to scale and harder to maintain.

To simplify this, we spent the past year building native ingestion paths for Kafka / Amazon MSK, RDS for PostgreSQL, and Aurora PostgreSQL—all currently in beta — so that streams and operational data can flow directly into Tiger Cloud without bespoke glue code.

With our Postgres source connectors, customers can replicate existing time-series tables from RDS or Aurora into Tiger Cloud, where they’re converted into optimized hypertables for high-ingest workloads and fast queries, all without modifying their application or schema.

S3 Connector Is Now Generally Available

Alongside these efforts, we also introduced a new S3 Connector, which has quickly become one of the most common ingest paths for AWS users. And today, we’re announcing its general availability. It continuously loads Parquet and CSV files from S3 into hypertables, handles late-arriving files, and makes historical data immediately available for real-time analytics without the Glue jobs, Lambdas, or custom ingestion services teams used to build.

Taken together, these capabilities create a cleaner, more AWS-native ingest model: whether your time-series data originates from IoT devices, market data feeds, event systems, or existing Postgres deployments, it can now flow directly into Tiger Cloud without additional code or operational overhead.

Lakehouse Interoperability

For many AWS customers, Amazon S3 serves as the foundation of their analytics and AI platforms, the place where governed, long-term datasets live and where engines like Athena, EMR or SageMaker expect to read data. Earlier this year, we introduced Tiger Lake in private beta, our Apache Iceberg integration, to make it easy for teams to expose curated operational and time-series data from Tiger Cloud directly into their S3-based lakehouse.

Tiger Lake works by using Postgres change data capture (CDC) to track every insert, update, and delete in source tables or hypertables. It then converts those changes into Iceberg-compliant commits and writes them to Amazon S3 via the S3 Tables interface. Because the output is a native Iceberg table stored in S3, AWS analytics and AI services can immediately query or train on the data using their existing tooling without batch exports, Spark pipelines, or glue code required. Operational changes in Tiger Cloud flow directly into the lakehouse as versioned Iceberg snapshots.

Tiger Lake Is Now Public Beta

Today we’re announcing that Tiger Lake is available in open beta, enabling any table or hypertable in Tiger Cloud to be continuously published as an Iceberg table on S3.

Tiger Cloud powers the real-time, high-ingest workloads that vanilla Postgres struggles with, while your AWS analytics and AI stack reads the same data through Iceberg on S3. It’s a natural bridge between operational Postgres workflows and the lakehouse architectures that drive analytics, ML, and enterprise intelligence on AWS.

Secure Connectivity

Customers want their databases to plug into AWS environments in the same way they already connect to RDS, Aurora, or Redshift to ensure their data and applications remain secure. Tiger Cloud has supported VPC peering for years,  making private, single-VPC deployments straightforward. At the beginning of this year, our new Transit Gateway support expanded that pattern to multi-account, multi-VPC organizations. Today, customers can connect Tiger Cloud using well-understood AWS constructs, without VPNs, proxies, or public endpoints.

Observability

Observability has been AWS-native in Tiger Cloud for a long time. Tiger Cloud integrates directly with Amazon CloudWatch for both metrics and logs, so teams can monitor their database using the same tooling they rely on for EC2, EKS, Lambda, MSK, and the rest of their AWS environment.

Tiger Cloud streams operational metrics into CloudWatch Metrics and sends structured logs to CloudWatch Logs, making it easy to build dashboards, set alarms, and satisfy compliance requirements without new tooling. 

Billing

Finally, we wanted the commercial experience to feel as seamless as the technical one. Tiger Cloud is fully integrated into AWS Marketplace, allowing customers to use the same procurement paths they already use for other AWS services.

For teams that want to get started quickly, Tiger Cloud supports pay-as-you-go billing directly through their AWS account. There’s no new vendor onboarding or separate invoice; usage simply appears on the existing monthly AWS bill.

For larger organizations with specific architectural, security, or cost requirements, we also support private offers, giving enterprises the ability to secure annual commitments, customized pricing, and tailored deployment guidance, all handled through AWS Marketplace.

Case Study: Speedcast’s Unified Architecture on AWS

Speedcast runs a global telecom network for remote industries, combining satellite and terrestrial links to keep ships, rigs, and NGOs online.

Previously, Speedcast had to juggle separate geospatial, relational, and time-series stores plus aging SCADA systems, stitching them together with fragile ETL pipelines that slowed insights and raised operational risk. With Tiger Lake in their AWS + Tiger Data stack, Speedcast dropped custom scripts and batching, using native integrations between their data lakehouse and Tiger Cloud to move toward a continuous data integration pipeline with Tiger Cloud at the center.

With Tiger Cloud as the “spider at the center of the web,” operations, data scientists, and customers now have a single, authoritative data source instead of hunting for data across systems. Platforms, dashboards and events are powered by the same database in real-time, regardless of workload pattern, with every system able to communicate with Apache Iceberg.

 "We stitched together Kafka, Flink, and custom code to stream data from Postgres to Iceberg. It worked, but it was fragile and high-maintenance,” said Kevin Otten, Director of Technical Architecture at Speedcast. “Tiger Lake replaces all of that with native infrastructure. It’s the architecture we wish we had from day one."

As Speedcast plans for service expansions and continues installing beyond 12,000 Starlink Terminals globally, Tiger Lake’s ingest pipeline will scale with them. For example, Speedcast can monitor usage patterns and spot emerging service-area outages in real-time before customers feel the impact. When a new service ticket is generated, Speedcast can drill into location, usage, and history with a single SQL query instead of bouncing between silos, reducing the time to resolution. 

Read the full case study in our blog.

Bringing Your Data Workloads Together on AWS

Modern applications shouldn’t require four different databases, half a dozen pipelines, and constant backfills just to keep data consistent. With Tiger Cloud and AWS, you get a unified Postgres engine that handles real-time ingest, high-performance time-series, vector search, and lakehouse integration—all inside the AWS architecture you already trust.

This is the future we’re building with AWS: simpler stacks, fewer moving parts, and one Postgres data layer for operational, analytical, and agentic workloads.

You can get started in minutes through the AWS Marketplace or sign up directly at tigerdata.com for free. We can’t wait to see what you build.

You open Claude Code or Cursor, describe your tables, and in seconds the AI hands you a Postgres schema that looks… fine. It runs. Your tests pass. You ship.

What you don’t see are the quiet little disasters tucked inside: money for prices, a BRIN index on random data, SERIAL and UUID mixed like a cocktail, timestamp without time zone because some tutorial said it was “easier”.

Fast-forward six months. You’re debugging currency-conversion bugs, chasing timezone ghosts, rewriting migrations, and adding the index that should have existed since day one. The code the AI agent wrote worked; it just wasn’t good. It was copying whatever examples it scraped from the entire internet.

And that’s the problem. It learned SQL from everywhere: Postgres, MySQL, SQLite, SQL Server, Oracle, random tutorials, and a decade of Stack Overflow answers. In all that noise, the nuances of idiomatic, high-quality Postgres get buried under the good, the bad, and the MySQL.

So we built something to fix that.

Giving AI the Postgres Judgment It’s Missing

pg-aiguide gives AI coding agents the Postgres-specific judgment they’re missing.It does this with three things working together:

1. AI-optimized “skills”— curated, opinionated Postgres best practices that Claude Code and other agents can apply automatically.
2. Semantic search across official documentation — version-aware retrieval for Postgres 15–18.
3. Extension ecosystem docs, starting with TimescaleDB and expanding quickly

You can connect it to any AI coding agent via our public Model Context Protocol (MCP) server or with the Claude Code plugin built to take advantage of Claude’s native skill support. No accounts. No usage limits. Completely free.

The goal is simple:Make AI write correct, production-ready Postgres by default.

You shouldn’t have to paste docs, correct outputs, or rely on prompt hacks. The AI should just generate better SQL the first time.

Why Database Code Quality Is So Important 

AI adoption among Postgres developers has exploded: jumping from 37% to 55% in a single year according to the 2024 State of Postgres survey. But as LLMs get more general, the training data has become broad instead of specialized and the LLMs become less precise.

This creates a surprising problem: A study by METR recently found that experienced developers using AI actually took 19 percent longer to complete tasks. Not because the AI failed, but because the output was almost right—the kind of subtle mistakes that take the longest to spot, debug, and unwind.

With databases, “almost right” compounds quickly. 

Schemas codify decisions that are painful to change later. Indexing mistakes hide until your tables grow. Datatype choices that take an hour to fix in development can take days of migrations and careful planning in production.

You can refactor application code. You can’t refactor state without consequences.

pg-aiguide stops these mistakes at the source, before they get baked into production.

Generic vs. Guided SQL

Here’s what happens when you ask a coding assistant to design a simple bookstore schema.

Without pg-aiguide

CREATE TABLE customers (
    customer_id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
    email VARCHAR(255) UNIQUE NOT NULL,
    password_hash VARCHAR(255) NOT NULL,
    first_name VARCHAR(100) NOT NULL,
    last_name VARCHAR(100) NOT NULL,
    phone VARCHAR(20),
    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
    is_active BOOLEAN DEFAULT TRUE
);

CREATE INDEX idx_customers_email ON customers(email);

CREATE TABLE authors (
    author_id SERIAL PRIMARY KEY,
    first_name VARCHAR(100) NOT NULL,
    last_name VARCHAR(100) NOT NULL,
    biography TEXT,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP
);

This schema works, but has classic issues:

• mixed identity strategies
• arbitrary VARCHAR limits
• case-sensitive email uniqueness
• timestamp defaults that behave inconsistently across systems

With pg-aiguide enabled

CREATE TABLE authors (
  author_id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  name TEXT NOT NULL,
  bio TEXT,
  created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX ON authors (name);

CREATE TABLE users (
  user_id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  email TEXT NOT NULL UNIQUE,
  password_hash TEXT NOT NULL,
  first_name TEXT NOT NULL,
  last_name TEXT NOT NULL,
  phone TEXT,
  is_active BOOLEAN NOT NULL DEFAULT true,
  created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
  updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE UNIQUE INDEX ON users (LOWER(email));
CREATE INDEX ON users (created_at);

This version uses:

• consistent identity columns (BIGINT GENERATED ALWAYS AS IDENTITY)
• TEXT instead of unnecessary VARCHAR
• correct timestamp handling (timestamptz + now())
• case-insensitive uniqueness enforced properly

Behind the scenes, the AI used the design_postgres_table skill either through the view_skill MCP tool or Claude’s native skills framework. In both cases, the agent automatically discovered and applied the Postgres-optimized guidance without human intervention.

You didn’t have to prompt differently.

You didn’t have to paste in docs.

pg-aiguide automatically shifts AI from “SQL that works” to “SQL you’d actually want in production.”

The Skills Are the Secret Sauce 

If you want AI to generate high-quality SQL, it is not enough to let it search the manual. A manual tells you what you can do, not what you should do. Skills fill that gap. They give the model judgment, not just facts.

Our skills are not trying to reteach the LLM syntax or capabilities. Instead they give the model the context it needs to make better choices. Here is an excerpt from a real skill.

## Postgres "Gotchas"

- **FK indexes**: Postgres **does not** auto-index FK columns. Add them.
- **No silent coercions**: length/precision overflows error out (no truncation). 
  Example: inserting 999 into `NUMERIC(2,0)` fails with error, unlike some 
  databases that silently truncate or round.
- **Heap storage**: no clustered PK by default (unlike SQL Server/MySQL InnoDB); 
  row order on disk is insertion order unless explicitly clustered.

These are the kinds of details you only know once you have lived in Postgres for a while. They trip up LLMs for the same reason they trip up developers who are new (and not so new) to the database. Yet these details are exactly what allow the model to produce better SQL.

In our evaluations (currently human-vibes-driven, soon LLM-judged), schema quality improves consistently when we compare a system with just semantic search to one that includes both semantic search and skills:

• more appropriate data types
• correct timestamp semantics
• stronger indexing strategies
• fewer migration pitfalls
• fewer long-term performance surprises

This is what “AI coding tools actually understanding Postgres” looks like.

The Tools We Provide The LLM

pg-aiguide provides two core capabilities that map cleanly to how AI coding tools operate.

1. Skills: Complete, Opinionated Postgres Guidance

view_skill returns full, AI-optimized best practices.These aren’t tutorials and they aren’t vague prompts. They’re machine-targeted, dense, token-efficient guidance that the AI can reliably use.

For example:

• prefer BIGINT GENERATED ALWAYS AS IDENTITY
• don’t use money
• don’t use timestamp without timezone
• index your foreign keys
• expect errors on precision overflows

Skills don’t need to be chunked—they are written so that each skill fits in context as a single complete unit.

Claude Code even supports skills natively, so the MCP server’s view_skill tool is disabled automatically when running as a plugin.

2. Semantic Search: Version-Aware Vector Retrieval Across Docs

The MCP tools semantic_search_postgres_docs and semantic_search_tiger_docs allow the AI to pull in the correct documentation for the Postgres version you’re targeting.

This matters because Postgres versions evolve meaningfully:

• Postgres 15: UNIQUE NULLS NOT DISTINCT
• Postgres 16: major changes to parallel query behavior
• Postgres 17: COPY error-handling improvements

Without version awareness, an AI can (and does) hallucinate features or syntax that will break your actual environment.

All of this knowledge of Postgres is chunked, embedded, and stored in Postgres itself.

We scrape official HTML docs, preserve header context, attach source URLs, and use character-bounded chunking with H1→H2→H3 breadcrumbs so each piece retains meaning of how it fits into the broader puzzle.

Help Us Build the World’s Best Postgres Guide for AI

Postgres has 35 years of engineering, craft, and hard-won lessons behind it. No single team can capture all of that. The community built the patterns, extensions, and production wisdom that make Postgres what it is. AI coding tools should reflect that depth, not spit out generic SQL lifted from outdated tutorials and old Stack Overflow posts.

pg-aiguide is our first step toward making Postgres the best database to use with AI coding assistants on purpose, not by accident. We are expanding the skill library with richer indexing guidance, full-text search skills, and documentation for essential extensions like PostGIS and pgvector. We are also adding keyword BM25 search to pair with semantic search for more accurate retrieval. But we need your help.

How You Can Contribute

You can make an immediate impact:

• add documentation for your Postgres extension
• contribute new skills that encode real, battle-tested expertise
• help evaluate, refine, and stress-test existing skills
• request features or report issues
• improve semantic search chunking or propose new areas to index
• share deep knowledge on partitioning, replication, security, or performance tuning

Skills matter most. They turn years of experience into guidance the AI can use instantly. Our schema-design skill went through multiple iterations before it felt right, and we learned a ton in the process. We would love to partner with you to build skills in your area of expertise.

pg-aiguide is fully open source at github.com/timescale/pg-aiguide. 

Help us teach AI to write Postgres like an expert.

With prompt templates and versioned docs, we turn 35 years of Postgres wisdom into structured knowledge your Agent can reason with.

Agents are the new developer. But they’re generalists. 

What happens when they design your Postgres database? Your schema runs, your tests pass… and six months later your queries crawl and your costs skyrocket. 

AI-generated SQL and database schemas are almost right. And that’s the problem. Fixing schema design mistakes is costlier than refactoring code. It often means multi-week migrations, downtime windows, rollback plans, and your CEO asking why the site is in maintenance mode. The root issue? LLMs don’t have the depth of Postgres and database expertise to let them build scalable systems. And when agents try to learn, they find documentation written for humans, not for them. 

But agents don’t need more data, they need better context. They need to know what “good Postgres” actually looks like. The good news is given the right context and tools, agents can become instant experts. Even with Postgres. 

That’s why we built an MCP server that provides 35 years of Postgres wisdom, and full access Postgres docs, all in a format that agents can easily process. 

And we think this just might be the best database MCP server ever built. While Neon, Supabase, and other Postgres companies created MCP servers as thin API wrappers, ours teaches AI how to think in Postgres. The Tiger MCP server gives AI tools that work automatically: no prompt engineering or manual lookups needed. You just ask. And it provides correct, idiomatic Postgres.

Our new MCP server ships with detailed prompt templates written by our senior engineers, plus versioned Postgres (15-18) docs, and TimescaleDB docs. When your AI needs to design a schema, it automatically pulls the right template and searches the docs (using hybrid search) to generate code that actually lasts. And of course, it handles the basics: start, stop, fork a database.

But the real magic happens when you ask AI to design your schema. So let’s see what happens when AI designs a simple schema. First without the MCP server, then with it.

Without the MCP Server

CREATE TABLE customers (
    customer_id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
    email VARCHAR(255) UNIQUE NOT NULL,
    password_hash VARCHAR(255) NOT NULL,
    first_name VARCHAR(100) NOT NULL,
    last_name VARCHAR(100) NOT NULL,
    phone VARCHAR(20),
    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
    is_active BOOLEAN DEFAULT TRUE
);

CREATE INDEX idx_customers_email ON customers(email);

CREATE TABLE authors (
    author_id SERIAL PRIMARY KEY,
    first_name VARCHAR(100) NOT NULL,
    last_name VARCHAR(100) NOT NULL,
    biography TEXT,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP
);

What’s wrong here: 

• Mixed ID types (SERIAL and UUID) with no clear reason
• Arbitrary VARCHAR limits instead of TEXT
• Case-sensitive email index → duplicate accounts
• CURRENT_TIMESTAMP instead of now(), leading to inconsistent timestamps

Someone new to Postgres may not notice these problems. Until it’s too late.

Now let’s see what AI generates with our new MCP server.

With the MCP Server

CREATE TABLE authors (
  author_id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  name TEXT NOT NULL,
  bio TEXT,
  created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX ON authors (name);

CREATE TABLE users (
  user_id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  email TEXT NOT NULL UNIQUE,
  password_hash TEXT NOT NULL,
  first_name TEXT NOT NULL,
  last_name TEXT NOT NULL,
  phone TEXT,
  is_active BOOLEAN NOT NULL DEFAULT true,
  created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
  updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE UNIQUE INDEX ON users (LOWER(email));
CREATE INDEX ON users (created_at);

What’s better about this?

• Consistent ID strategy with BIGINT GENERATED ALWAYS AS IDENTITY
• TEXT instead of arbitrary VARCHAR limits
• Case-insensitive email lookups
• Modern timestamp handling

But why does this matter?

Each of these differences creates a compounding problem. Changing datatypes in the future will require full table rewrites. Missing lowercase email handling means duplicate accounts and confused users. And time zones? Every senior developer gets the thousand-yard stare when you mention UTC conversions.

This is just with a small example; imagine what would happen with more complex schemas.

And if you don’t believe us, here’s what Claude has to say:

> Please describe the schema you would create for an e-commerce website two times, first with the tiger mcp server disabled, then with the tiger mcp server enabled. For each time, write the schema to its own file in the current working directory. Then compare the two files and let me know which approach generated the better schema, using both qualitative and quantitative reasons. For this example, only use standard Postgres.

Verdict:

The Tiger MCP-enabled schema is objectively superior because:

1. Type Safety: Uses modern PostgreSQL types that prevent common errors
2. Data Integrity: 4x more constraints ensure data quality at the database level
3. Performance: Better indexing strategy with 55% more indexes, including partial and expression indexes
4. Maintainability: Better documentation, clear comments, proper naming
5. Features: More complete e-commerce functionality (discounts, full-text search, audit trails)
6. PostgreSQL Best Practices: Follows official PG17 recommendations from the documentation
7. Future-Proof: Uses modern features like GENERATED ALWAYS AS IDENTITY and NULLS NOT DISTINCT

The Tiger MCP server provided access to PostgreSQL-specific best practices documentation and design patterns that resulted in a more robust, performant, and maintainable schema.

How Prompt Templates Make It All Possible

Behind the scenes, AI uses the MCP server to call get_prompt_template(‘design_postgres’) to load schema design guidance. No pasted docs. No corrections. Just better code.

Prompt templates turn production wisdom into reusable guidance for AI. Instead of hunting through documentation written for humans, AI gets the information it needs in a format built for machines.

It comes down to the fact that humans and LLMs have opposite needs. Humans need narratives and memorable examples (and yes, even cat memes) to help them retain information. LLMs need to preserve context window space. That’s why prompt templates make terrible blog posts, but perfect AI guidance.

Our philosophy is: don't re-teach what the model already knows. LLMs have seen millions of lines of SQL. They know how to write CREATE TABLE. What they don’t know is the 35 years of Postgres wisdom about what works well and what doesn’t.

It's like your senior DBA whispering advice in the model's ear.

Our schema design template (design_postgres_tables) doesn’t explain what a primary key is. It jumps straight to guidance:

“Prefer BIGINT GENERATED ALWAYS AS IDENTITY; use UUID only when global uniqueness is needed.”

For data types, it doesn’t teach from scratch. It just tells you what works:

“DO NOT use money type; DO use numeric instead.”

Here’s a real snippet from the template:

## Postgres "Gotchas"

- **FK indexes**: Postgres **does not** auto-index FK columns. Add them.
- **No silent coercions**: length/precision overflows error out (no truncation). 
  Example: inserting 999 into `NUMERIC(2,0)` fails with error, unlike some 
  databases that silently truncate or round.
- **Heap storage**: no clustered PK by default (unlike SQL Server/MySQL InnoDB); 
  row order on disk is insertion order unless explicitly clustered.

These gotchas trip up LLMs the same way they trip up developers new to Postgres. We optimized these templates for machines: short, factual, and precise, packing maximum guidance into minimum tokens. 

We tested the same approach on a real IoT schema design task. Without templates, the AI added forbidden configurations and missed critical optimizations. With templates, it generated production-ready code with compression, continuous aggregates, and tuned performance.

That’s how prompt templates work. Now let’s see how the MCP server makes it all happen.

How This MCP Server is Smarter Than Others

While Neon, Supabase, and other Postgres companies created MCP servers as thin API wrappers, ours teaches AI how to think in Postgres.The Tiger MCP server gives AI tools that work automatically: no prompt engineering or manual lookups needed. You just ask. And it provides correct, idiomatic Postgres.

get_prompt_template provides auto-discovered expertise. Instead of having to call a template explicitly, you just say “I want to make a schema for IoT devices…” and the MCP server figures it out.

With self-discoverable templates, the AI can detect intent and load the right recipe, applying 35 years of Postgres best practices behind the scenes.

The templates have real depth. No scraped snippets or boilerplate. The templates are written by senior Postgres engineers, and provide opinionated, production-tested guidance that is tuned to avoid every trap that seasoned DBAs know to avoid.

Postgres-native vector retrieval adds the right context. When the AI needs more information, the MCP server searches the versioned Postgres (15-18) and TimescaleDB docs. And it uses Postgres itself for storage and vector search.

Versioning is critical. For example, Postgres 15 introduced UNIQUE NULLS NOT DISTINCT, while 16 improved parallel queries, and 17 changed COPY error handling. The MCP keeps AIs grounded in correct syntax every time, avoiding broken code from the wrong version.

The Tiger MCP doesn’t just wire up APIs. It teaches AI to think like a real Postgres engineer.

You don’t have to craft the perfect prompt. You just ask, and it does the right thing.

See It For Yourself

Install the Tiger CLI and MCP server:

curl -fsSL https://cli.tigerdata.com | sh
tiger auth login
tiger mcp install

(We also have alternative installation instructions for the CLI tool.)

Then select your AI assistant (Claude Code, Cursor, VS Code, Windsurf, etc.) and immediately get real Postgres knowledge flowing into your AI.

This is how Postgres becomes the best database to use with AI coding tools: not by accident, not because someone pasted docs into a chat, but because the tooling now teaches AI how to think in Postgres. 

Try the MCP server. Break it. Improve it. Help us teach every AI to write real Postgres.

About the authors

Matty Stratton

Matty Stratton is the Head of Developer Advocacy and Docs at Tiger Data, a well-known member of the DevOps community, founder and co-host of the popular Arrested DevOps podcast, and a global organizer of the DevOpsDays set of conferences.

Matty has over 20 years of experience in IT operations and is a sought-after speaker internationally, presenting at Agile, DevOps, and cloud engineering focused events worldwide. Demonstrating his keen insight into the changing landscape of technology, he recently changed his license plate from DEVOPS to KUBECTL.

He lives in the Chicagoland area and has three awesome kids and two Australian Shepherds, whom he loves just a little bit more than he loves Diet Coke.

Matvey Arye

Matvey Arye is a founding engineering leader at Tiger Data (creators of TimescaleDB), the premiere provider of relational database technology for time-series data and AI. Currently, he manages the team at Tiger Data responsible for building the go-to developer platform for AI applications. 

Under his leadership, the Tiger Data engineering team has introduced partitioning, compression, and incremental materialized views for time-series data, plus cutting-edge indexing and performance innovations for AI. 

Matvey earned a Bachelor degree in Engineering at The Cooper Union. He earned a Doctorate in Computer Science at Princeton University where his research focused on cross-continental data analysis covering issues such as networking, approximate algorithms, and performant data processing. 

Jacky Liang

Jacky Liang is a developer advocate at Tiger Data with an AI and LLMs obsession. He's worked at Pinecone, Oracle Cloud, and Looker Data as both a software developer and product manager which has shaped the way he thinks about software. 

He cuts through AI hype to focus on what actually works. How can we use AI to solve real problems? What tools are worth your time? How will this technology actually change how we work? 

When he's not writing or speaking about AI, Jacky builds side projects and tries to keep up with the endless stream of new AI tools and research—an impossible task, but he keeps trying anyway. His model of choice is Claude Sonnet 4 and his favorite coding tool is Claude Code.

TLDR: A new chapter for Postgres starts today: Agentic Postgres, the first database for agents, brings new architecture for the age of agents, and free access for everyone who builds with them.

We're launching a free tier. No credit card, no time limit, no catch.

AI development moves at the speed of thought. You’re constantly experimenting, testing a vector search strategy, forking a database to try a new schema, or spinning up an instance for a weekend project. The friction of “will this cost me money?” shouldn’t slow you down.

Today, we’re launching the Tiger Free Plan, a fully managed Postgres built for how AI development actually works: experimental, iterative, and increasingly agent-driven. It’s the same Tiger managed cloud experience developers already love, now free for every idea.

The Tiger Free Plan should be the database you reach for every time you want to test an idea. 

What Makes This Free Plan Different

Most free database tiers are built for yesterday’s users. They gate core features, expire trials, or remind you what you don’t get. We built ours for how development actually happens today.

Because the database has a new user. Developers aren’t working alone anymore. They’re building alongside agents that write code, run migrations, and query data through APIs. That changes what experimentation means. You don’t just need a place to store data, you need a system where your tools, scripts, and agents can spin up, fork, and reason over automatically.

The Tiger Cloud Free Plan is built for that reality. It’s not designed for production. It's designed for progress, for the 90 percent of work that happens before you know if something is worth scaling.

Other free tiers optimize for control. Ours optimizes for momentum. When you’re experimenting, you shouldn’t have to think about limits, billing, or setup. You should be able to fork your database, test a schema, or try a new retrieval strategy in seconds, whether you’re doing it yourself or your agent is doing it for you.

That’s why it includes the tools modern AI workflows actually rely on. You can fork your database to test safely and recover fast, run vector search to compare retrieval and embedding strategies in real time, and analyze live data with built-in time-series and columnar features so you can see what’s happening as it happens. Because Tiger Cloud runs the same APIs across every plan, moving from free to production is instant: no migration, no rework, no context lost.

This isn’t a teaser or a trial. It's part of a new architecture for Postgres, one built for builders: humans and agents alike. 

What’s Included

Compute & storage:

• Shared compute
• Up to 750 MB of storage per service
• Limit of 2 free services per account
• Available in us-east-1 (EU expansion coming soon)

Features that matter:

• Database forks: Branch your database like your code. Test safely, recover fast, or try something bold.
• AI-native retrieval: Build RAG and other AI-powered features with native hybrid and vector search support (pgvectorscale + BM25).
• Real-time analytics: Hypertables, continuous aggregates, and columnar storage. Run analytics in Postgres without extra systems.
• Insights: View performance on a per-query basis over time, and get optimization recommendations, so you spend less time guessing. 
• Automated Management: Upgrades, tuning, and maintenance handled for you, so your database always runs at its best.
• 50+ Postgres Extensions. All your favorite extensions, from PostGIS to pgvector, built in and ready to go.
• Connection management: Simplified, secure handling that just works.

What Happens When You Hit the Limits

At 750 MB, your service switches to read-only mode. You’ll get warnings as you approach the limit, and you can:

• Fork to an earlier point in time (PITR - 24-hour point in time recovery)
• Clean up data
• Or upgrade to a paid plan and migrate in minutes

Free services can be upgraded directly. When your project grows, you can quickly convert it in place to remove storage limits. Your free services coexist perfectly with your paid ones, so you can test safely alongside production.

Connection poolers are not included in the Free Plan. Dedicated support is not included either, but Free Plan users are encouraged to join our community Slack to ask questions, share ideas, and see what others are building.

Built for the builders: Developers and agents alike

If you're new to Tiger, you can start with the free plan or take a 30-day performance trial. If you're already a customer, you can add two free services right inside your existing account for testing, sandboxing, or side projects.

The Free Plan is our way of lowering the floor for experimentation. We want more developers and agents building together on postgres without friction. It's a small change with a big goal: making it effortless to start, learn, and build.

Get Started

From the command line via the Tiger CLI (Download from GitHub)

$ curl -fsSL https://cli.tigerdata.com | sh
$ tiger auth login 
$ tiger service create

From the cloud console:

1. Sign up
2. Select “Free Plan” (vs. the free 30-day trial)
3. Create a service and get building!

Frequently Asked Questions

About the Free Plan

1. What is included in the Free Plan?
   1. Up to 2 free services per account
   2. Developer experience features: database forks (24-hour PITR), Insights dashboard, pgvector, real-time analytics/time-series (hypertables, continuous aggregates, columnar storage), connection management
   3. MFA for secure login
   4. IP Allow list for security
   5. Database logs
   6. A large suite of Postgres extensions, like PostGIS and pg_cron
   7. Available in us-east-1 (EU region coming soon)
2. What is excluded in the Free Plan?
   1. Connection pooler
   2. Compute resizing
   3. Dedicated support (community Slack available)
   4. High-availability configurations
   5. Advanced security features
   6. Advanced enterprise features
3. What are the specs of a free service?
   1. Each free service includes:
      1. Shared compute
      2. Storage: Up to 750 MB for user data
      3. Backup: 24-hour point-in-time recovery for forking
      4. Connections: Limited (exact number TBD, designed to balance usability and system health)
4. Is a credit card required to create Free Plan services?
   1. No. You can sign up and create free services without entering payment information.
5. What happens if I reach my storage limit?
   1. You'll receive warnings as you approach 750 MB. Once you hit the limit, your service switches to read-only mode. You can:
      1. Fork your service to an earlier point in time (within the 24-hour PITR window)
      2. Delete data to free up space before reaching the limit
      3. Upgrade to a Performance or Scale plan and “Convert to a standard service without storage limits”
6. Can I upgrade free services?
   1. Yes. If your project outgrows the Free Plan, you can:
      1. Switch to a Performance or Scale plan
      2. Convert your free service to a standard service to remove storage limitations and get access to higher levels of compute
7. What is the difference between Free Plan and Performance Plan?
   1. Performance Plan offers:
      1. Scalable compute (starting at 0.5 CPU / 2 GB RAM, up to 16 CPU / 128 GB RAM)
      2. Storage up to 16 TB (pay-as-you-go by the GB)
      3. Connection pooling
      4. High availability options
      5. Dedicated support
      6. Virtual Private Cloud peering
      7. Production-grade SLAs
   2. The Free Plan is optimized for experimentation; Performance is optimized for production.
8. Can I start a Performance trial if I already have a Free Plan?
   1. Yes! Every project gets one free trial of Performance or Scale. You can start it at any point, even after creating free services. Your free services remain active during and after the trial.
9. Does my free trial automatically convert to a Free Plan when my trial expires?
   1. If you start with a Performance or Scale trial, here's what happens when it ends:
      1. Trial services (on Performance/Scale) are paused, then eventually deleted
      2. Any free services remain active
      3. Your account converts to the Free Plan
      4. You can create up to 2 free services (if you haven't already)
      5. No charges occur unless you explicitly upgrade
10. Do I keep my free services if I upgrade my Free Plan to Performance or Scale?
    1. Yes! Free services are available on all plans. You can have up to 2 free services alongside your paid services.
11. If I am already a Tiger Data customer, can I also use the free services?
    1. Yes! Existing Performance, Scale, and Enterprise customers can create up to 2 free services within their accounts. Use them for testing, sandboxes, or side projects.
12. Will I have access to free services in my paid plan?
    1. Yes. All plans (Free, Performance, Scale, Enterprise) include access to up to 2 free services. They coexist with your development and production databases.
13. Should I start with the Free Plan or the Free Trial?
    1. Start with Free Plan if:
       1. You're prototyping or experimenting
       2. You're not sure what you're building yet
       3. You want to test Tiger Data without commitment
    2. Start with a Performance/Scale trial if:
       1. You're ready to test Tiger Cloud with production-like workloads
       2. You have large datasets to migrate
       3. You need higher compute or storage immediately
       4. You want to evaluate enterprise features
    3. You can always start a trial later. Each project gets one trial, and you choose when to use it.
14. Is Tiger Fluid Storage available on Performance and Scale plans?
    1. No, Tiger Fluid Storage is not available on Performance and Scale plans today.

About the authors

Hien Phan

An AI, Data, and Infrastructure Marketing Leader, Hien Phan is the Head of Marketing at Tiger Data. He has led Product, Partner, and Customer Marketing at Pinecone. He and his team launched a game-changing serverless architecture and introduced Pinecone Assistant, marking a significant leap in our product offerings.

At Amplitude, Hien’s team championed solutions that empowered product and marketing teams to excel in a product-led growth. This role sharpened his ability to drive growth through strategic marketing initiatives, solidifying that brand as an indispensable tool for product experimentation and analytics category.

Hien lives in the Bay Area with his two lovely dogs and makes a mean roasted chicken.

Announcing Agentic Postgres: The first database built for agents.

Agents are the New Developer

80% of Claude Code was written by AI. More than a quarter of all new code at Google was generated by AI one year ago. It’s safe to say that in the next 12 months, the majority of all new code will be written by AI.

Agents don’t behave like humans. They behave in new ways. Software development tools need to evolve. Agents need a new kind of database made for how they work.

But what would a database for agents look like?

At Tiger, we’ve obsessed over databases for the past 10 years. We’ve built high-performance systems for time series data, scaled Postgres across millions of workloads, and served thousands of customers and hundreds of thousands of developers around the world. 

​​So when agents arrived, we felt it immediately. In our bones. This new era of computing would need its own kind of data infrastructure. One that still delivered power without complexity, but built for a new type of user. 

How do agents behave?

• Agents don’t click, they call.
• Agents don’t remember, they retrieve.
• Agents can download expertise to become experts.
• Agents can parallelize effortlessly, acting like a multi-threaded team.
• Agents need a safe sandbox where they can play (or wreak havoc).
• Agents can also hammer your infrastructure (and your budget) if you’re not careful.

We started on this problem over a year ago. Multiple teams working in parallel, months of engineering and internal user feedback, rethinking everything from the storage layer to how agents actually reason. 

Here’s what we built. 

Introducing Agentic Postgres

Today we’re launching Agentic Postgres, the first database designed from the ground up for agents. It includes:

The best database MCP server ever built

Agentic Postgres includes our new MCP server that enables agents not just to interact with the database but also understand how to use it well. We’ve taken our 10+ years of Postgres experience and distilled it into a set of built-in master prompts. This gives agents safe, structured access to the database through high-level tools for schema design, query tuning, migrations, and more. The MCP server also performs native full-text and semantic search over the Postgres docs, so agents can instantly retrieve the right context as they think. 

> I want to create a personal assistant app. Please create a free service on Tiger. Then using Postgres best practices, describe the schema you would create.

Native search and retrieval

Agentic Postgres comes with native full-text and semantic search built directly into the database. For semantic search, we’ve improved our existing extension pgvectorscale, for higher-throughput indexing, better recall, and lower latency at scale than pgvector. 

For full-text search, pg_textsearch, our newest Postgres extension, implements BM25 for modern ranked keyword search optimized for hybrid AI applications alongside pgvectorscale. The current preview release uses an in-memory structure for fast writes and queries. Future releases will add disk-based segments with compression and BlockMax WAND optimization, applying the same battle-tested techniques from production search engines.

Together, these extensions let agents retrieve structured data instantly without leaving Postgres.  

> Using service qilk2gqjuz, analyze user feedback with hybrid search (combining text search and semantic search). Group similar feedback by theme and show counts for each theme, using an ascii bar chart. First, look at the pg_textsearch (BM25) and pgvectorscale documentation in the Tiger docs to get the proper syntax, and then use those extensions.

Fast, zero-copy forks

At the core of Agentic Postgres is a new copy-on-write block storage layer that makes databases instantly forkable. Every agent can spin up its own isolated environment, a full copy of production data in seconds, without duplicating data (or costs). Every fork is lightweight and efficient, so you only pay for the blocks that change. It’s perfect for experiments, benchmarks, and migrations that can run safely in parallel. 

> Please create a fork of gf868h9j1y using the last snapshot, and then test 3 different indexes that we should create to speed up performance, then delete the fork, and report back on your findings. Before you start run “tiger service fork --help” and “tiger service delete --help” to get the right syntax. Use MCP over psql, using the password from the local keychain.

New CLI and free tier

We’ve also built a new CLI that makes it easy to explore, fork, and build with Agentic Postgres, and we’re launching a free tier so every developer and every agent can get hands-on right away.

This is all launching today. You can try this today with 3 basic commands in your terminal:

# 3 commands to install the Tiger CLI and MCP. That's it!
$ curl -fsSL https://cli.tigerdata.com | sh
$ tiger auth login
$ tiger mcp install

Then just tell your agent to spin up a new free service using MCP, or simply call tiger create service from the command line to get going.

Powered by Fluid Storage

Agentic Postgres is powered by Fluid Storage, our new distributed storage layer. Fluid Storage is built on a disaggregated architecture of a horizontally scalable distributed block store using local NVMe storage, a storage proxy layer that exposes copy-on-write volumes, and a user-space storage device driver.

It’s storage that looks like a local disk to Postgres yet scales like a cloud service.

As a result, Fluid Storage delivers instant forks, snapshots, and automatic scaling (up or down) without downtime or over-provisioning. In benchmark testing, a single volume sustains throughput over 110,000 IOPS, while retaining all of Fluid’s elasticity and copy-on-write capabilities. 

All free services on Tiger Cloud run on Fluid Storage today, so every developer can experience its performance and flexibility firsthand. 

And this is just the start. We’ll dive deeper into each of these (MCP, pg_textsearch, pgvectorscale, forkable databases, Fluid Storage, CLI, free tier) later this week and next. 

Built for Agents and Developers

Agentic Postgres is built for agents, so developers can work on higher level problems. 

Building with and for agents, we’ve learned something simple: agents are not here to replace us. They’re here to elevate us.

Agents take on the mechanical, repetitive work, freeing us to focus on what matters most: architecture, design, creativity, impact. They make us faster, smarter, and enable us to do more ambitious work than we could do alone.  

The myth is that AI will replace developers. The truth is that developers who build with agents will replace those who don’t. 

Agentic Postgres is for developers who want to build with AI. For developers who care more about working applications than disposable demos. For developers who want AI to feel like engineering, not just experimentation.

Today’s launch is just the beginning. There are still some rough edges. We’d love your help sanding them down. But expect more to come: more launches, big and small, in the weeks, months, and years ahead.

Agents are the new developers. Agentic Postgres is their new playground.

Built for Agents. Designed to Elevate Developers.

So let’s build. Together.

Get started today: 

$ curl -fsSL https://cli.tigerdata.com | sh

Everyone eventually asks the same “easy” questions: How many distinct things do I have right now? How many active devices? Which trading pairs saw activity today? What’s the latest reading per sensor? On PostgreSQL at scale, those questions turn from harmless SQL into multi‑second waits because DISTINCT usually means walking every qualifying row and deduplicating afterward. Even with an ordered index, the engine traverses the full range and pushes keys through a unique step—O(N) work when N is tens or hundreds of millions.

We built SkipScan to flip that script. Instead of visiting every row, SkipScan uses the B‑tree’s order to jump from one distinct value to the next: find device = 1, output it, then restart the index at device > 1, and so on. The cost becomes O(K × log N), where K is the number of distinct values. When K ≪ N, you get millisecond answers instead of coffee‑breaks.

SkipScan first landed in TimescaleDB 2.2.0 for rowstores. With 2.20.0, we extended it to columnstore hypertables and to distinct aggregates like COUNT(DISTINCT …) —leveraging PostgreSQL 16’s ability to feed presorted inputs into ORDER BY/DISTINCT aggregates. With 2.22.0, we support multi-column SkipScan when all column values are guaranteed to be not-null. What follows is the problem it solves, the design that makes columnstore skipping safe and fast, and the exact queries and plans you can reproduce.

The Problem We Had to Solve

Rowstores are bad enough for DISTINCT at large N. Columnstores raise the stakes: data are packed in compressed batches (≤1,000 rows) and reading a single tuple can imply decompressing a whole batch. If we naïvely walked every batch to deduplicate, compression would help storage but hurt latency.

To make DISTINCT fast on columnstore, the executor has to:

• Jump by segment key—not row—so we only touch the first relevant batch per distinct value.
• Prune batches aggressively using min/max metadata (e.g., time windows) to avoid needless decompression.
• Stop early inside a batch as soon as a qualifying tuple is found.

That’s the essence of SkipScan on columnstore.

How SkipScan on Columnstore Works

Columnstore chunks maintain an automatic B‑tree on (segmentby, orderby) (e.g., (device, time DESC)), plus per‑batch metadata columns like _ts_meta_min_1 and _ts_meta_max_1 for time. SkipScan hooks into that structure:

1. Seek the first batch for the current device using the (segmentby, orderby) index; for “latest per device,” this is typically the batch with the newest time for that device.
2. Decompress just enough—often the first tuple—to satisfy the query (DISTINCT ON (device) ... ORDER BY device, time DESC).
3. Restart at device > current_device to hop to the next device’s first batch.
4. Push down filters that can be answered from metadata (e.g., _ts_meta_max_1 > now() - '1 hour') so entire stretches of batches are skipped without decompression.

There’s one important constraint: SkipScan on columnstore applies when the DISTINCT key is the leading segmentby column (or the leading columns for multi-key SkipScan). That’s what allows safe jumps. 

Note: by leading column in the index we mean the 1st index column used in a query. For an index on (a,b), b is the leading index column in a query like “select distinct a, b from t where a=1”

Under the covers you’ll see a Custom Scan (SkipScan) node above an index scan and (when needed) a DecompressChunk. After emitting a distinct value, the executor rewrites its start condition from col >= v to col > v and re‑seeks—classic B‑tree hopscotch.

A Bench You Can Reproduce (Rowstore and Columnstore)

Table

CREATE TABLE metrics (
  time   timestamptz NOT NULL,
  device integer,
  value  double precision
);

Rowstore index

CREATE INDEX ON metrics(device, time DESC);

Scale: ~23,328,010 rows; device has 10 distinct values.

Columnstore layout

ALTER TABLE metrics SET (
  timescaledb.compress,
  timescaledb.compress_segmentby = 'device',
  timescaledb.compress_orderby   = 'time DESC'
);
SELECT compress_chunk(ch) FROM show_chunks('metrics') ch;

On one representative compressed chunk we see 5,370 batches totaling 5,364,693 compressed rows and an index on (device, _ts_meta_min_1 DESC, _ts_meta_max_1 DESC).

Complexity notes: the O(·) terms are theoretical guides. PostgreSQL uses B‑trees (multi‑way), so constants, caching, and pruning matter. We use them to show scale, not to predict exact milliseconds.

What It Looks Like in Practice

Counting devices (rowstore)

SELECT COUNT(DISTINCT device) FROM metrics;

With SkipScan: 3.430 ms.
Without (vanilla path): 6756.322 ms (~6.8 s).Same 23M‑row table; SkipScan performs roughly 2000× faster by doing ~K log‑seeks instead of scanning all N entries.

Latest reading per device (columnstore)

SELECT DISTINCT ON (device) device, time, value
FROM   metrics
ORDER  BY device, time DESC;

With SkipScan: 4.589 ms by touching the first batch per device.
Without: 3912.069 ms—the engine decompresses and inspects ~5.3 M tuples to deduplicate.

Latest reading per device in the last hour (columnstore + metadata pruning)

SELECT DISTINCT ON (device) device, time, value
FROM   metrics
WHERE  time > now() - INTERVAL '1 hour'
ORDER  BY device, time DESC;

With SkipScan: 2.537 ms.
Without: 9.236 ms.The gap narrows because _ts_meta_max_1 lets the planner discard 5,330/5,370 batches up front; SkipScan still wins by visiting only the first qualifying batch per device.

Thresholded latest reading (columnstore; tuple‑level predicate)

SELECT DISTINCT ON (device) *
FROM   metrics
WHERE  value > 50
ORDER  BY device, time DESC;

With SkipScan: 45.392 ms after scanning ~3,002 decompressed tuples on the first chunk.
Without: 2145.536 ms after touching ~6,303,586 tuples across chunks.
When a predicate can’t be answered by batch metadata, SkipScan still stops early per device once a match is found.

How This Helps in Real Systems

In IoT fleets, “show me the latest per device” powers every status board. On rowstores it’s a recurring spike; on columnstore without SkipScan it can become a wall. With SkipScan, the query returns in a few milliseconds by decompression‑on‑demand.

At exchanges and brokerages, COUNT(DISTINCT symbol) and “latest quote per pair” show up in every dashboard and alert. With 200M+ rows and hundreds of thousands of pairs, SkipScan changes the economics of distinct—turning once‑expensive questions into cheap, always‑on queries.

In crypto analytics, analysts slice by wallet, contract, or chain. Setting the segmentby to the entity you dedupe most often (e.g., address) lets SkipScan jump entity‑to‑entity, keeping answer times stable even under heavy ingest.

Multi-Column SkipScan

Multi-column SkipScan support was added in 2.22.0, with a caveat: it can only be chosen for queries which do not produce NULL distinct values.

That means if a query produces multiple distinct columns and doesn’t allow NULLs to be output as distinct values, then this query can be eligible for multi-column SkipScan.

Examples: 

CREATE INDEX ON metrics(region, device, metric_type);
-- All distinct columns have filters which don't allow NULLs: can use SkipScan
SELECT DISTINCT ON (region, device, metric_type) *
FROM   metrics
WHERE region IN ('UK','EU','JP') AND device >1 AND metric_type IS NOT NULL
ORDER  BY region, device, metric_type, time DESC;
-- Distinct columns are declared NOT NULL: can use SkipScan
CREATE TABLE metrics(region TEXT NOT NULL, device INT NOT NULL, ...);
SELECT DISTINCT ON (region, device) *
FROM   metrics
ORDER  BY region, device, time DESC;

Multicolumn SkipScan skips over the current key values in a similar way to a single-column SkipScan: it freezes the values of keys preceding the current key, allows any values for keys following the current key, and searches for the first current key value which is greater than the current value.

When no more values are found for the current key, SkipScan moves to the previous key and repeats the process, or is done when there are no more previous keys.

For the above example, suppose we are currently scanning the values (AUS, 2, >56) for the tuple (region, device, metric_type).  We’ve found the next tuple (AUS, 2, 59) and we search for (AUS, 2, >59) now. 

If no more tuples are found, we switch the search to (AUS,>2, “any value”), find the next tuple (AUS,5,3) and search for (AUS,5,>3), etc.  

If no more tuples for AUS are found we search for (>AUS, “any value”, “any value”), suppose we find (BR,4, 15) next, and restart the search for (BR,4,>15), etc.

The reason why we don’t use SkipScan if NULL values are possible for distinct columns is because of complexities in determining “any value” in the above examples. 

Because PostgreSQL sorts NULLs separately from other values we have to check for NULLs and NOT NULLs separately for each key to exhaust “any value” possibilities. 

Checking it for 2 keys means doing 4 checks for (NULL, NULL), (NULL, NOT NULL), (NOT NULL, NULL), (NOT NULL, NOT NULL).  Checking it for M keys means 2^M complexity.

It is a lot of complexity to address for an edge-case as users usually do not care about NULL devices or regions etc.

To take advantage of multi-column SkipScan it’s enough to add IS NOT NULL check or declare distinct columns NOT NULL if no NULL values are anticipated in those columns. If there are strict conditions on distinct columns like “a>1” or “b IN (1,2,3)” then NULLs are already not allowed for those columns and SkipScan can be applied to those columns.

Using It in Your Schema

Design your layout so the column you deduplicate is first:

• Rowstore: create an index starting with the distinct column(s), followed by your time sort (device, time DESC).
• Columnstore: set timescaledb.compress_segmentby to that column (or columns for multicolumn SkipScan) and compress_orderby to match your query’s sort (e.g., time DESC). Compress your historical chunks.

Verify with EXPLAIN—look for Custom Scan (SkipScan) above the index/decompress nodes. If the planner is shy, you can nudge it:

SET enable_seqscan = false;                           -- prefer indexes
SET timescaledb.skip_scan_run_cost_multiplier = 0; -- bias for SkipScan (2.20+)

To check whether SkipScan was used in a query, and if it was used, on which keys and whether those keys are NOT NULL, the following flag can be set:

SET timescaledb.debug_skip_scan_info TO true; -- (2.22+)                 
-- NULLs are excluded via ">1"
SELECT DISTINCT ON (device) device, time, value
FROM   metrics
WHERE device > 1
ORDER  BY device, time DESC;
-- SkipScan key info is output
INFO:  SkipScan used on metrics_device_idx(device NOT NULL)
-- NULLs are not excluded
SELECT DISTINCT ON (device) device, time, value
FROM   metrics
ORDER  BY device, time DESC;
-- SkipScan key info is output
INFO:  SkipScan used on metrics_device_idx(device NULLS LAST)

When it doesn’t apply: multiple NULL-allowing distinct keys at once, orders that don’t start with the distinct column, or a columnstore where your distinct key isn’t the leading segmentby.

-- Ordered index for rowstore
CREATE INDEX ON metrics(device, time DESC);

-- Ordered index for columnstore based on segmentby + orderby settings
ALTER TABLE metrics SET (timescaledb.compress, timescaledb.compress_orderby='time DESC', timescaledb.compress_segmentby='device');
SELECT compress_chunk(ch) FROM show_chunks('metrics') ch;

These queries can use SkipScan:

SELECT DISTINCT ON (device) * FROM metrics;
SELECT DISTINCT ON (device) * FROM metrics ORDER BY device, time DESC;
SELECT DISTINCT ON (device, value) * FROM metrics WHERE value = 10;
SELECT count(DISTINCT device), max(DISTINCT device) FROM metrics; 

These queries cannot use SkipScan either because distinct columns not in the index or non-distinct aggregates present or because the index doesn’t match the query order:

SELECT DISTINCT ON (time) * FROM metrics;
SELECT DISTINCT ON (device) * FROM metrics ORDER BY device, time;
SELECT DISTINCT ON (device, value) * FROM metrics;
SELECT count(DISTINCT device), max(device) FROM metrics; 
SELECT count(DISTINCT device), count(DISTINCT value) FROM metrics; 

What You Get

Millisecond‑fast deduplication on datasets measured in billions of rows; far fewer tuples decompressed per query; predictable dashboard SLOs under bursty ingest; and no application changes—just the right index or segment layout. Rowstore users get wins back to 2.2.0; columnstore and distinct‑aggregate users get the full experience on 2.20.0+ (with PostgreSQL 16+), and multicolumn SkipScan support for not-null distinct values added in 2.22.0.

Skip less relevant rows. Skip entire batches. Skip the scan.

Sign up for a free trial today to see the benefit for your own queries and workloads.

About the authors

Natalya Aksman is a senior developer at Tiger Data, focusing on queries optimization and performance. Natalya joined Tiger Data recently, after working on Vertica columnar database for many years, and before that, working on Sybase IQ which was one of the earliest columnar databases in the industry. 

Natalya is most passionate about optimizing performance of analytical databases and streamlining of query planning and execution. Natalya has a Master of Applied Mathematics degree from Moscow University and Master of Computer Science from Northeastern University in Boston.

Noah Hein is a Senior Product Marketing Engineer at Tiger Data, where he helps developers understand, adopt, and succeed with the fastest PostgreSQL platform for real‑time and analytical workloads. Day‑to‑day, he translates deep technical capabilities—like hypertables, hypercore compression, and continuous aggregates—into clear product narratives and customer stories that drive adoption and growth.

Before joining Tiger Data, Noah spent several years on the “builder” side of the house as both a founding engineer and an educator. He co‑created Latent Space’s three‑week AI Engineering Fundamentals course and has taught hundreds of engineers how to apply LLMs in production. Noah frequently speaks on AI‑data convergence topics; at the first ever AI Engineer Summit he led the “AI Engineering 101” workshop, walking participants through hands‑on projects.

Outside of work, Noah tries to help more people land jobs with his side project JobMosaic. When he’s not crafting launch posts, you’ll find him experimenting with edge‑AI devices, tinkering with homelab Postgres clusters, or giving impromptu botany lessons to anyone who will listen.

Time-series and analytical data continues to grow at an unprecedented pace, and with it comes the challenge of efficiently storing and querying massive datasets. Traditionally, compressing this data required background jobs, and additional tuning. This slowed down ingestion, added operational headache, and delayed storage savings.

That’s why today, we're excited to announce Direct Compress, a new feature coming to TimescaleDB that compresses data during ingestion in memory, eliminating the need for traditional compression policies and improving insert performance by up to 40x.

Note: Direct Compress is currently available as a tech preview in TimescaleDB 2.21 for COPY operations, with full support for INSERT operations coming in a later version.

The Evolution of TimescaleDB’s Columnstore 

TimescaleDB has long been recognized for its industry-leading compression capabilities. With hypercore, TimescaleDB's hybrid row-columnar storage engine, users can achieve compression ratios of over 90% while maintaining fast query performance. Traditionally, the system would:

1. Insert data in uncompressed row format
2. Write individual WAL records for each tuple
3. Later compress chunks through background policies

Now, Direct Compress fundamentally changes this approach by compressing data during the ingestion process itself.

What is Direct Compress?

Direct Compress is a feature that allows TimescaleDB to compress data in memory as it's being ingested. Instead of writing WAL records for individual tuples, the system writes compressed batches directly to disk. This approach addresses several key challenges that developers and database administrators face when working with high-volume time-series data:

• Excessive I/O overhead: Traditional ingestion requires writing each tuple individually to the WAL, creating significant I/O bottlenecks
• Dependency on compression policies: Previously, you had to wait for background compression jobs to optimize storage
• Insert performance limitations: Large-scale data ingestion was constrained by the overhead of individual tuple processing

Benchmark Results (37x Improvement)

To test the per-tuple overhead, a narrow table with only one integer column was used. Direct compression provided considerable performance improvements, with the single integer table achieving 148.8 million tuples per second using 10k batch compression—a 37x improvement over uncompressed insertion. For a table with a timestamp column and 2 integer columns we achieved an insert rate of 66 million tuples per second with compression.

The schema used does have a big impact on achievable insert rate, with more complex datatypes like jsonb or wider rows having lower ingest rates. Parsing integer columns was found to have the least overhead compared to other datatypes, and for these benchmarks more than half of the cpu time was spent parsing input even when using binary input format. Performance scaled linearly across all thread counts until reaching the storage I/O bottleneck. During these tests we used a Tiger Cloud instance with 64 cores and EBS storage—with more optimized storage higher numbers are probably achievable. For the uncompressed tests no indexes were present on the hypertables. The 1k and 10k batch size refers to the batch size used internally during compression, not the batch size used by the client sending the data.

Key Benefits

Reduced I/O operations

By compressing data in memory before writing to disk, Direct Compress eliminates the need to write individual WAL records for each tuple. Instead, only compressed batches are written, dramatically reducing I/O overhead. 

Eliminated policy dependencies

With Direct Compress, your INSERT operations already produce compressed chunks. This means compress_chunk() functions and compression policies become less critical to your workflow, simplifying your database maintenance.

Immediate storage efficiency

Unlike traditional compression that happens after ingestion, Direct Compress provides storage benefits immediately, reducing your storage footprint from the moment data arrives.

How Direct Compress Works

Direct Compress operates by intercepting data during the ingestion process and compressing it in memory before writing to disk. The process involves:

1. Batch Collection: Data is collected in configurable batches during COPY or INSERT operations.
2. In-Memory Compression: Each batch is compressed using TimescaleDB's proven compression algorithms.
3. Optimized Writing: Compressed batches are written directly to disk with minimal WAL overhead.

This approach differs from traditional compression methods because it eliminates the two-step process of "ingest then compress," instead performing both operations simultaneously. Importantly, Direct Compress requires batched operations on the client side to achieve these performance benefits. With direct compression, data ingestion becomes limited by CPU processing rather than IO speed.

Roadmap

• COPY support (TimescaleDB 2.21 - Tech Preview)
• INSERT support (coming soon)
• Continuous aggregate support (coming soon)

Getting Started with Direct Compress

Prerequisites

Before using Direct Compress, ensure you have:

• TimescaleDB version 2.21 or later (currently in tech preview)
• A hypertable with compression enabled (see example)
• Batched client operations to make use of the feature

Important requirements and limitations

Direct Compress requires batching on the client side to function effectively. It cannot be used:

• If the hypertable schema has unique constraints
• If the hypertable has triggers
• Continuous aggregates on the target hypertable

Configuration options

Direct Compress is controlled through several GUCs (Grand Unified Configuration parameters):

timescaledb.enable_direct_compress_copy (default: off)

Enables the core Direct Compress feature for COPY operations. When enabled, chunks will be marked as unordered, so presorting is not required.

timescaledb.enable_direct_compress_copy_sort_batches (default: on)

Enables per-batch sorting before writing compressed data, which can improve query performance.

timescaledb.enable_direct_compress_copy_client_sorted (default: off)

⚠️ DANGER: When enabled, chunks will not be marked as unordered. Only use this if your data is globally sorted, as queries requiring ordering will produce incorrect results with unsorted data. In the context of this feature we can distinguish between local and global sorting. Local sorting means within the current batch data is sorted. Global sorting means there is no batch that will overlap with the current batch.

Basic Usage Example

-- Create a hypertable with compression
CREATE TABLE sensor_data(
    time timestamptz, 
    device text, 
    value float
) WITH (
    tsdb.hypertable,
    tsdb.partition_column='time'
);

-- Enable Direct Compress
SET timescaledb.enable_direct_compress_copy = on;

-- Use binary format for maximum performance
COPY sensor_data FROM '/tmp/sensor_data.binary' WITH (format binary);

Best Practices and Recommendations

1. Use binary format

Binary format achieves the highest insert rates. While CSV and text formats are supported, binary format provides optimal performance.

2. Consider order by configuration

The default orderby configuration is time DESC for query optimization. However, for maximum Direct Compress benefits, consider changing this to time to optimize for insert performance:

ALTER TABLE sensor_data SET (timescaledb.orderby = 'time');

This represents a trade-off between insert performance and query performance—choose based on your primary use case.

3. Presort data before ingestion

While TimescaleDB can do sorting as part of Direct Compress, it will take away CPU resources from other tasks.

4. Leverage multiple threads

The benchmark results show significant benefits from parallel ingestion. Consider using multiple threads for large data imports.

Migration and Compatibility

Upgrading existing tables

Direct Compress works with any existing hypertable that has the columnstore enabled, provided the limitations (no unique constraints, triggers, or continuous aggregates) are met. 

Backward compatibility

Direct Compress is fully compatible with existing TimescaleDB compression features. You can use both traditional columnstore policies and Direct Compress simultaneously, though Direct Compress reduces the need for background compression jobs.

Looking Forward

Direct Compress represents a significant milestone in TimescaleDB's ongoing evolution toward real-time analytics at scale. This feature is part of our broader commitment to eliminating the traditional trade-offs between ingestion speed and storage efficiency.

Future enhancements to Direct Compress will include:

• Support for INSERT
• Additional optimizations for unsorted data when using direct compress
• Compatibility with continuous aggregates
• Enhanced client-side tooling for optimal batching

Try Direct Compress Today

Direct Compress brings considerable performance improvements to TimescaleDB users by eliminating the traditional ingestion bottleneck. With up to 40x faster ingestion rates and immediate storage benefits, this feature is a game-changer for high-volume time-series applications.

Whether you're managing IoT sensor data, financial market feeds, or application monitoring metrics, Direct Compress can help you achieve unprecedented ingestion performance while reducing storage costs from day one.

We encourage you to try the tech preview of Direct Compress in your development environment and share your experiences with the community. Your feedback will help us refine this feature as we move toward full release. As always, our team is available to help you optimize your TimescaleDB deployment for your specific use case.

Ready to get started? Check out our documentation or contact our team for personalized assistance with Direct Compress implementation.

Have questions about Direct Compress or want to share your results? Join the conversation in our community forum or reach out to us on GitHub.

About the Author

Sven is the tech lead for TimescaleDB, but his journey with databases started a long time ago. For over 25 years, he has been a huge fan of PostgreSQL, and it's that deep-seated passion that led him to where he is today. His work on planner optimizations and diving into the columnstore to squeeze out every bit of performance is a direct extension of his goal: to make the Postgres ecosystem even more powerful and efficient for everyone who uses it.

That long history with Postgres also informs his work on the security front. One of the projects he is most passionate about is pgspot, where he gets to help build a more secure future for the database. After all these years, he has seen firsthand how a strong, trustworthy foundation is essential. To him, a great database isn't just about speed; it's about protecting the data with unwavering reliability. This blend of performance and security is what truly excites him every day.

When he's not in the weeds of database code, you can find him thinking about the bigger picture—how to make the community and product stronger, safer, and more user-friendly. He loves the challenge of taking a complex problem and finding a simple, elegant solution. His journey with Postgres has taught him that the best technology is built on a foundation of trust and a commitment to continuous improvement.

This article, written by Nakylai Taiirova, was originally posted on the Mad Devs blog on April 10, 2025. Nakylai is a web developer with over 5 years of hands-on experience gained in multiple industries, including food & goods delivery, WMS, EdTech, and entertainment. She has mastered skills in various areas of software engineering, including backend, frontend & mobile development. The article is reposted here with permission. It outlines the performance tests that led to choosing TimescaleDB for a real-world e-commerce project, tracking product page views, click-through rates, and search position rankings.

Introduction

Hi everyone! Today, I want to talk about time-series data management. We encountered this challenge while working on a project, and I will share our experience with TimescaleDB and highlight its outstanding features.

Understanding Time-Series Data: Characteristics and Challenges

Time-series data is simply a sequence of data points collected over time. Think of it as measurements or events that have timestamps attached to them. It's append-only (we're mostly adding new data, not changing historical records), naturally ordered by time, and the time element itself is usually crucial for analysis.

Common examples of time-series data include: stock prices over time, weather measurements (temperature, humidity, wind speed), monthly subscriber counts on a website, sensor readings from IoT devices, etc.

In a real-world e-commerce project we recently built, our team encountered this type of data. We needed to track how many times product pages were viewed and how often users clicked on them. It was also necessary to record each product's daily position in search results since merchants paid for premium placements. This created a perfect time-series dataset–every day, we collected thousands of new records with timestamps, while the historical data remained unchanged. We used this data to show merchants evidence that higher positions actually led to more visibility and clicks.

Common Challenges

When working with time-series data, there are several important challenges to consider:

• Data volume and scaling: High-frequency data collection leads to rapid growth, which causes traditional databases to struggle with performance.
• Complex aggregations: Queries like averages, sums, or analyzing changes over time need to be performed efficiently.

This requires specialized approaches for optimal storage and querying. Large volumes of time-series data can overwhelm traditional relational databases, especially when they perform complex time-based aggregations.

Time-series databases

There are several specialized database systems that exist to address these challenges, such as InfluxDB, Prometheus, Apache Druid, MongoDB, and Amazon Timestream. Each has its strengths and works in different use cases; however, this article will focus on TimescaleDB.

TimescaleDB

TimescaleDB is a time-series database built as an extension to PostgreSQL with time-series optimizations. Unlike other solutions that require learning new query languages, TimescaleDB lets users continue using SQL.

Key features:

• Postgres-based: Full SQL compatible and supports relational capabilities.
• Hypertables: Unlike standard PostgreSQL tables that store all data in a single table, hypertables automatically partition time-series data into chunks based on time intervals. This increases query speed since they only need to scan relevant time chunks instead of the entire dataset.
• Data lifecycle management: Automatic data handling through features like:
  • 1. Continuous aggregation — pre-calculates common metrics (like daily averages or hourly sums) and keeps them updated automatically, so dashboards and reports run 100-1000x faster without recalculating from raw data each time.
  • 2. Data retention policies that automatically remove old data past a certain age. 
  • 3. Tiered storage — automatically moves older, less frequently accessed data to low-cost object storage (built on Amazon S3) while keeping recent data in high-performance storage.

Below we'll see TimescaleDB in action with some real-world examples.

Practical Example: Intel Lab Sensor Data

For this practical example, we're using sensor data from the Intel Berkeley Research Lab. This dataset contains millions of temperature, humidity, light, and voltage readings over a 2-month period from 54 sensors deployed throughout the lab. We chose this dataset because, despite its relatively small size (about 2.3 million readings), it has enough data to demonstrate the performance differences between PostgreSQL and TimescaleDB.

Setting up our environment

First, we created two tables to compare performance:

1. A regular PostgreSQL table (sensor_data_postgres)
2. A TimescaleDB hypertable (sensor_data_timescale)

Both tables have identical schemas and indexes:

-- Regular PostgreSQL table
CREATE TABLE sensor_data_postgres (
    time        TIMESTAMPTZ NOT NULL,
    epoch       INTEGER,
    sensor_id   INTEGER NOT NULL,
    temperature DOUBLE PRECISION,
    humidity    DOUBLE PRECISION,
    light       DOUBLE PRECISION,
    voltage     DOUBLE PRECISION
);

-- TimescaleDB table
CREATE TABLE sensor_data_timescale (
    time        TIMESTAMPTZ NOT NULL,
    epoch       INTEGER,
    sensor_id   INTEGER NOT NULL,
    temperature DOUBLE PRECISION,
    humidity    DOUBLE PRECISION,
    light       DOUBLE PRECISION,
    voltage     DOUBLE PRECISION
);

-- Convert to hypertable
SELECT create_hypertable('sensor_data_timescale', 'time');

Performance comparison

All performance tests were run on:

• MacBook Air (M1, 2020)
• Apple M1 chip
• 16 GB RAM
• PostgreSQL 14.17 (64-bit)
• TimescaleDB 2.19.0

We loaded 1,841,828 rows and ran identical queries on both tables. Let's look at each query type and its results:

1- Full range select: Basic retrieval of all data within a time range

SELECT * FROM sensor_data_postgres 
WHERE time >= '2004-02-28 00:58:46' 
AND time <= '2004-04-05 11:02:32';

-- Same query for TimescaleDB table

Results:

• PostgreSQL: 226.5 ms.
• TimescaleDB: 317.7 ms.
• Performance: PostgreSQL is 40% faster.

Insights:

• For basic SELECT queries, PostgreSQL performs better.
• TimescaleDB's overhead for chunk management affects simple query performance.
• This type of query is less common in time-series applications.

2- Time-based aggregations: Grouping and analyzing data by time intervals

-- PostgreSQL (Daily aggregation)
SELECT 
    date_trunc('day', time) as day,
    sensor_id,
    COUNT(*) as readings,
    AVG(temperature) as avg_temp,
    stddev(temperature) as temp_stddev
FROM sensor_data_postgres
WHERE time >= '2004-02-28' AND time <= '2004-04-05'
GROUP BY day, sensor_id
HAVING stddev(temperature) > 2
ORDER BY day, sensor_id;

-- TimescaleDB (using time_bucket)
SELECT 
    time_bucket('1 day', time) as day,
    sensor_id,
    COUNT(*) as readings,
    AVG(temperature) as avg_temp,
    stddev(temperature) as temp_stddev
FROM sensor_data_timescale
WHERE time >= '2004-02-28' AND time <= '2004-04-05'
GROUP BY day, sensor_id
HAVING stddev(temperature) > 2
ORDER BY day, sensor_id;

Results:

• PostgreSQL: 443.2 ms.
• TimescaleDB: 170.9 ms.
• Performance: TimescaleDB is 61% faster.

Insights:

• The core of TimescaleDB's advantage is its hypertable architecture. Hypertables automatically partition time-series data into chunks based on time intervals. When querying, TimescaleDB identifies and scans only the relevant chunks instead of the entire dataset, reducing I/O operations for time-bounded queries.
• The time_bucket function is specifically optimized to work with this chunked architecture, which makes it more efficient than PostgreSQL's date_trunc.

Advanced TimescaleDB features: continuous aggregates

Think of continuous aggregates as a personal data assistant that works ahead of time. Instead of forcing a database to recalculate the same aggregations (like daily averages or hourly counts) every time someone views a dashboard, TimescaleDB does this work in advance and keeps it ready to serve instantly.

Imagine temperature data being tracked and updated every minute, but the dashboard only needs to show daily averages. Rather than scanning millions of raw data points each time the dashboard loads, continuous aggregates pre-calculate data and store the required results. They automatically update as new data arrives.

Let's compare a regular PostgreSQL view with a TimescaleDB continuous aggregate:

1- PostgreSQL View vs TimescaleDB Continuous Aggregate

-- Regular PostgreSQL view (computed on every query)
CREATE VIEW pg_daily_sensor_stats AS
SELECT
    date_trunc('day', time) as day,
    sensor_id,
    AVG(temperature) as avg_temp,
    MIN(temperature) as min_temp,
    MAX(temperature) as max_temp,
    COUNT(*) as reading_count
FROM sensor_data_postgres
GROUP BY day, sensor_id;

-- TimescaleDB continuous aggregate (materialized and automatically refreshed)
CREATE MATERIALIZED VIEW ts_daily_sensor_stats
WITH (timescaledb.continuous) AS
SELECT
    time_bucket('1 day', time) as day,
    sensor_id,
    AVG(temperature) as avg_temp,
    MIN(temperature) as min_temp,
    MAX(temperature) as max_temp,
    COUNT(*) as reading_count
FROM sensor_data_timescale
GROUP BY day, sensor_id;

-- Set up automatic refresh policy
SELECT add_continuous_aggregate_policy('ts_daily_sensor_stats',
    start_offset => INTERVAL '3 days',
    end_offset => INTERVAL '1 hour',
    schedule_interval => INTERVAL '1 hour');

2- Querying the aggregates

-- Query against PostgreSQL view (computed on demand)
SELECT * FROM pg_daily_sensor_stats
WHERE day >= '2004-03-01' AND day <= '2004-03-31'
ORDER BY day, sensor_id;

-- Query against TimescaleDB continuous aggregate (pre-computed)
SELECT * FROM ts_daily_sensor_stats
WHERE day >= '2004-03-01' AND day <= '2004-03-31'
ORDER BY day, sensor_id;

Results when querying a month of data:

• PostgreSQL view: 424.0 ms (computed on every query).
• TimescaleDB continuous aggregate: 0.433 ms (pre-computed).
• Performance: TimescaleDB is 979x faster.

Storage optimization:

TimescaleDB also saves storage costs while maintaining query performance. With our temperature sensor data, 54 sensors taking readings every minute for months, this means we will need extra storage based on time.

For newer TimescaleDB versions (v2.18.0+), Hypercore automatically handles the storage management. But for this example, we’ll take a look at a compression policy.

Setting up compression is simple with compression policies — just tell TimescaleDB to compress chunks older than a certain age (like 7 days), and it handles everything automatically in the background.

Both PostgreSQL and TimescaleDB tables start at similar sizes (around 209 MB) with our test dataset:

-- Check regular PostgreSQL table size
SELECT pg_total_relation_size('sensor_data_postgres')/1024/1024 as size_mb;
-- PostgreSQL size: 209 MB

-- Check hypertable size before compression
SELECT hypertable_size('sensor_data_timescale')/1024/1024 as size_mb;
-- TimescaleDB size before compression: 208 MB

-- Enable compression on table
ALTER TABLE sensor_data_timescale SET (
    timescaledb.compress,
    timescaledb.compress_segmentby = 'sensor_id',
    timescaledb.compress_orderby = 'time'
);

-- Create compression policy (compress chunks older than 7 days)
SELECT add_compression_policy('sensor_data_timescale', INTERVAL '7 days');

-- Check hypertable size after compression
SELECT hypertable_size('sensor_data_timescale')/1024/1024 as size_mb;
-- TimescaleDB size after compression: 35 MB (83% reduction)

Insights:

• Native compression provides significant storage savings (83% reduction)
• Compression maintains query functionality and reduces the storage costs
• Choose compression settings strategically:
  • 1. compress_orderby='time': Best for time-series data as values typically follow changes over time.
  • 2. compress_segmentby='sensor_id': Group data by columns you frequently filter or aggregate on.
  • 3. For example, if you often query SELECT avg(temperature) FROM sensors WHERE sensor_id = 5, using sensor_id as segmentby will improve query performance.

Learn more about compression settings

Final Thoughts

Our performance tests demonstrated TimescaleDB's advantages for time-series data management. While traditional PostgreSQL performs better for simple queries, TimescaleDB is a good choice for complex time-based operations with performance improvement for aggregation queries.

In our real-world e-commerce project, we successfully implemented TimescaleDB to track product page views, click-through rates, and search position rankings. This gave us the perfect solution for our expanding dataset—every day, we collected thousands of new timestamped records while keeping historical data intact. Using continuous aggregation, we were able to efficiently analyze how premium placements affected visibility and clicks across a 2-3 year period, providing merchants with evidence that higher positions actually increased engagement.

The storage benefits were significant, too. TimescaleDB's compression reduced our storage needs by about 30-40%. This led to real cost savings in our cloud bills. Our analytics platform became more responsive, and our budget healthier. These benefits matter even more when the situation involves large amounts of time-series data.

TimescaleDB is worth considering for work with growing time-series data that requires complex analysis. It offers powerful tools and accommodates SQL.

We’re experimenting with a database that can self-describe. And we’re starting with the most popular one in the world: Postgres.

TL;DR: We’re experimenting with turning Postgres into a self-describing database, embedding meaning as part of the schema. By providing natural language explanations of PostgreSQL structures and logic, agents can more accurately query data, and answer a broader set of questions. In our early tests, using an LLM-generated semantic catalog improved SQL generation accuracy by up to 27%. Here's the repo link for reference.

Databases Lack Context About Their Structures

As any developer who has ever had the (mis)fortune of working on a legacy database knows quite well, a database is not self-describing. You can’t look at a database schema and tell what’s going on. That orders1 table can be for a purchase order or a customer order or it’s an experimental table someone created and forgot to drop 5 years ago. This is a long-standing problem with databases that people solve by talking to each other, looking at code that interacts with the database, examining git history, and screaming at the wall.  

But, LLMs need to answer questions about the data just from the context provided by the database. No wonder they get confused. 

Like our fearless leader Ajay Kulkarni once said—“LLMs crave (accurate) context the same way GPUs crave power.”

In order for agents to extract insights from data in the database it needs to understand which columns and tables to query. For example, if it doesn’t understand which table contains the customer vs purchase orders, it can’t understand when a customer’s order has shipped.

In fact, in internal experiments we conducted at Tiger Data, we found that 42% of context-less LLM-generated SQL queries missed critical filters or misunderstood relationships, silently returning misleading data. 

The larger (and thus probably more important) the database, the worse things get.

It doesn’t take a senior engineer to know that this is unacceptable.

Adding Context with Semantic Catalogs

The solution is incredibly simple: allow developers to add context about the database in the form of natural language descriptions of the schemas and business logic within Postgres, injecting much-needed factual meaning into schemas. 

We arrived at this by putting ourselves in the place of the LLM — asking, how would we generate the query if we were in its shoes? The answer was clear: without additional context, we wouldn’t know enough to make the right call. Along the way, we tried different prompts, techniques, and even alternative LLMs, but the core issue was always the same: the model simply lacked the context necessary to generate the query. That realization led us to build a way for developers to provide that missing context directly.

That's what we’ve been experimenting with and are excited to share today.  

We set out to test what happens when SQL generation is powered by our LLM-generated semantic catalog. The outcome was a 27% boost in accuracy compared to the control.

Let's show you how we created this self-describing Postgres database.

Building a Self-Describing Database

Our thoughts for this self-describing Postgres are built around four core ideas: 

1. Embed semantics alongside schema:
   Every table, column, function, and business rule should be described clearly—in natural language.
2. Versioned and governed descriptions:
   Metadata should live alongside application code, version-controlled, reviewed, and governed with the same rigor.
3. Self-correcting querying:
   Postgres itself should expose safety hints, telling agents which queries are expensive or unsafe, and provide deterministic verification mechanisms (EXPLAIN) to catch errors before queries run.
4. Measure and iterate transparently:
   Developers should be able to build an evaluation suite for agent interactions and determine how well the system performed. Critically, the developers should see what errors are due to bad schema descriptions and lack of context (retrieval errors) or incorrect reasoning (logical errors).

What We’re Experimenting With Today

We’re exploring this approach through two building blocks. This post primarily covers the Semantic Catalog. Next week, we’ll publish a post about the evaluation harness.

Why isn't the problem just retrieval?

Retrieval using the schema element names alone does not cut it simply because the names lack enough meaning. It doesn’t matter if you use keyword search, semantic search, or this week’s newest shiny retrieval algorithm—the semantic meaning of payment_aborts versus  refunds is too close for any retrieval mechanism to make heads or tails of it.  

Even if you injected the entire schema into the context of your prompt (burning through cash and increasing response latency in the process), it’s doubtful the model would be able to reason about payment_aborts versus refunds either.

The semantic catalog solves this by offering a structured, natural language representation of your database's metadata and business logic. In this experimental version, it supports documentation and descriptions for:

• Tables and views
• Columns
• Functions and procedures
• Example queries
• Facts: business logic rules or freeform context

Initial descriptions can be LLM-generated and stored in human-readable YAML files designed for version control, peer review, and governance. Once reviewed, this metadata can be imported into the semantic catalog and indexed for semantic search, simplifying the process of retrieving and reasoning about data structures.

Performance enhancement through semantic context

In our early tests, using an LLM-generated semantic catalog improved SQL generation accuracy by up to 27%. Effective SQL generation involves two key steps:

1. Retrieving Relevant Context: Identifying necessary database components
2. SQL Generation Reasoning: Formulating accurate SQL queries based on the retrieved context

Our evaluation revealed substantial improvements from using semantic descriptions for SQL generation accuracy (from 58% to 86%) for certain schemas. A smaller improvement was noted when using semantic descriptions in retrieval recall (from 52% to 58% on one of our tests).

Results varied significantly across datasets, schemas, and models. Our current evaluations were conducted using a more cost-effective model (OpenAI's gpt-4.1-nano). We're particularly interested in insights from users experimenting with diverse datasets and more advanced models.

With these results, let’s walk through how the agent actually interacts with the database.

How queries work: the minimal agent loop

Every semantic query follows this four-step flow:

Walking through step-by-step:

Step 1: Describe your database using an LLM

Generate initial YAML descriptions with an LLM:

pgai semantic-catalog describe -f descriptions.yaml

Review and refine these descriptions, storing them with the same rigor as your application code.

The resulting structure looks like this:

schema: public
name: restaurant
type: table
description: Stores core information about each restaurant, including its cuisine,
  city, and rating.
columns:
- name: id
  description: Primary key identifier for the restaurant.
- name: name
  description: Lowercased name of the restaurant.
- name: food_type
  description: Lowercased name of the type of cuisine or food style offered by the restaurant.
- name: city_name
  description: Lowercased name of the city where the restaurant is located.
- name: rating
  description: Numeric rating score assigned to the restaurant, where higher than 2.5 is considered good.
...
---
type: sql_example
sql: SELECT t2.house_number, t1.name FROM LOCATION AS t2 JOIN restaurant AS t1 ON t1.id = t2.restaurant_id WHERE t1.city_name IN (SELECT city_name FROM geographic WHERE region = 'bay area') AND t1.rating > 2.5
description: give me some good restaurants in the bay area ?
...
---
type: fact
description: When asking for a restaurant, provide its name (restaurant.name) and its house number (location.house_number).
...

Step 2: Have a human review the description and add context

A developer would then review the descriptions and add additional context. Crucially, business logic that is central to database operations is often not encoded into the schema, and thus cannot be derived by the LLM. A developer needs to provide that information.

After the descriptions are reviewed, they should be treated as a core part of application code and thus should be stored in version control, be reviewed in pull-requests, etc.

Step 3: Import into the catalog 

Make descriptions available:

pgai semantic-catalog import -f descriptions.yaml

This supports declarative configuration and continuous deployment practices.

Step 4: Generate SQL 

Generate SQL from natural language queries:

CLI:

pgai semantic-catalog generate-sql -p "Which passengers have experienced the most flight delays in 2024?"

Python:

response = await catalog.generate_sql(
    con,
    con,
    "openai:gpt-4.1",
    "Which passengers have experienced the most flight delays in 2024?",
)

Key Lessons So Far

Semantic context matters 

Models are far better at generating correct SQL when they have access to rich semantic information, not just schema names, but natural-language descriptions. Even a small amount of context (e.g. "the order table tracks transactions for customers buying subscription plans") can dramatically improve reliability.

Narrow interfaces build confidence 

What we’ve noticed with agentic text-to-sql systems is that to get good accuracy you need to tighten the scope of what kind of information the system has access to. The relationship looks something like this:

Successful agentic systems start with tight scopes. We found it valuable to restrict agents to function-level or view-level access at first, and only expand access once correctness is proven.

Postgres allows you to balance control and flexibility through three main interfaces:

• Functions: Highly controlled but narrow scope
• Views: Moderate control with broader access
• Raw Tables: Most general but least constrained and more prone to failure

We suggest beginning with tightly scoped functions, then expanding access as confidence grows.

Self-Correcting Using EXPLAIN

To further improve reliability, we employ Postgres's deterministic EXPLAIN command. This preemptively catches query errors such as incorrect column or table names, allowing agents to self-correct and substantially increasing accuracy.

Store the Semantic Catalog Anywhere

For many deployments, integrating the semantic catalog directly into your existing database provides the simplest path. Recognizing, however, the risks and complexities of altering production environments, we've also enabled the semantic catalog to be hosted independently in a separate database. This approach offers greater deployment flexibility and ensures accessibility even when you only have read-only access to your primary database.

What’s Next: From Self-Describing to Self-Learning

While this semantic foundation is already improving agentic querying, we're just beginning. Our roadmap focuses on:

• Self-learning catalog: Automatically enrich metadata by analyzing queries from production environments. Let databases learn from usage to continuously enhance accuracy.
• Dynamic policy management: Express complex access rules—like row-level privacy policies—in natural language, automatically enforced by the database.

We’ve open-sourced everything we’ve built so far. Check out the README with a Quickstart to dive in, and we warmly invite your contributions, whether it's running evaluations on your schemas, proposing datasets, or challenging our assumptions. The best databases aren't just designed; they evolve through community effort.

About the author

Matvey Arye is a founding engineering leader at Tiger Data (creators of TimescaleDB), the premiere provider of relational database technology for time-series data and AI. Currently, he manages the team at Tiger Data responsible for building the go-to developer platform for AI applications.  Under his leadership, the Tiger Data engineering team has introduced partitioning, compression, and incremental materialized views for time-series data, plus cutting-edge indexing and performance innovations for AI. 

Matvey earned a Bachelor degree in Engineering at The Cooper Union. He earned a Doctorate in Computer Science at Princeton University where his research focused on cross-continental data analysis covering issues such as networking, approximate algorithms, and performant data processing. 

Modern applications are becoming more dynamic, more intelligent, and more real time. Dashboards refresh with incoming telemetry. Monitoring systems respond to shifting baselines. Agents make decisions in context, not in isolation. Each depends on the same foundational requirement: the ability to unify live events with deep historical state.

Yet the data remains fragmented.

Operational systems, built on Postgres, handle ingestion and serving. Analytical systems, built on the lakehouse, handle enrichment and modeling. Connecting them means stitching together streams, pipelines, and custom jobs—each introducing latency, fragility, and cost. The result is a patchwork of systems that struggle to deliver the full picture, let alone do so in real time.

This fragmentation doesn’t just slow teams down—it limits what developers can build. You can’t deliver real-time dashboards with historical depth or ground agents in fresh operational context when the data is split by design.

This architectural divide is no longer sustainable.

Tiger Lake bridges that divide. Now in public beta, it introduces a new data loop—continuous, bidirectional, and deeply integrated—between Postgres and the lakehouse. It simplifies the stack, preserves open formats, and brings operational and analytical context into the same system.

Introducing Tiger Lake: Real-Time Data, Full-Context Systems

Tiger Lake eliminates the need for external pipelines, complex orchestration frameworks, and proprietary middleware. It is built directly into Tiger Cloud and integrated with Tiger Postgres, our production-grade Postgres engine for transactional, analytical, and agentic workloads.

The architecture uses open standards from end to end:

• Apache Iceberg tables stored in Amazon S3 Tables for lakehouse integration
• Continuous replication from Postgres tables or hypertables into Iceberg
• Streaming ingestion back into Postgres for low-latency serving and operations
• Pushing down queries from Postgres to Iceberg for efficient rollups

These capabilities come built in. What previously required Flink jobs, DAG schedulers, and custom glue now works natively. Streaming behavior and schema compatibility are designed into the system from the start.

To understand how Tiger Lake reshapes data architecture, it helps to revisit the medallion model and consider how it evolves when real-time context becomes a core design principle.

You can think of it as an operational medallion architecture:

• Bronze: Raw data lands in Iceberg-backed S3.
• Silver: Cleaned and validated data is replicated to Postgres.
• Gold: Aggregates are computed in Postgres for real-time serving, then streamed back to Iceberg for feature analysis.

Traditional Bronze–Silver–Gold workflows were built for batch systems. Tiger Lake enables a continuous flow where enrichment and serving happen in real time.

This shift transforms an overly complex pipeline into a dynamic and simpler real-time data loop. Context and data moves freely between systems. Operational and analytical layers stay connected without redundant jobs or duplicated infrastructure.

All data remains native, up to date, and queryable with standard SQL. Tiger Lake supports a single write path that powers real-time applications, dashboards, and the lakehouse, using the architecture that best fits the developer. Users can write data to Postgres, then have appropriate data and rollups automatically synced to their lakehouse; conversely, users already feeding raw data into the lakehouse can automatically bring it to Postgres for operational serving. Now, applications can reason across the now and the then—without orchestration code or synchronization overhead.

"We stitched together Kafka, Flink, and custom code to stream data from Postgres to Iceberg. It worked, but it was fragile and high-maintenance," said Kevin Otten, Director of Technical Architecture at Speedcast. "Tiger Lake replaces all of that with native infrastructure. It’s the architecture we wish we had from day one."

From Architecture to Outcomes

Tiger Lake enables real-time systems that were previously too complex to operate or too expensive to build.

Customer-facing dashboards

Dashboards can now combine live metrics with historical aggregates in a single query. There is no need for dual stacks or stale insights. Tiger Lake supports high-throughput ingestion at production scale, powering pipelines that visualize billions of rows in real time. Everything lives in one system, continuously updated and instantly queryable.

"With Tiger Lake, we finally unified our real-time and historical data," said Maxwell Carritt, Lead IoT Engineer at Pfeifer & Langen. "Now we seamlessly stream from Tiger Postgres into Iceberg, giving our analysts the power to explore, model, and act on data across S3, Athena, and Tiger Data."

Monitoring systems

With a single source of truth and a continuous data loop, alerting becomes faster and more reliable. Engineers can run one SQL query to inspect fresh telemetry and historical incidents together—improving triage speed, reducing false positives, and staying focused on what matters.

Simplifying the data plane also improves system resilience. Tiger Lake lets monitoring systems operate on the same live operational backbone, where Iceberg provides historical depth and Tiger Postgres delivers low-latency access.

Agents

Tiger Lake makes grounding possible without additional infrastructure. Developers can embed recent user activity and long-term interaction history directly inside Postgres. There is no need for orchestration, vector drift management or custom AI pipelines.

Imagine a support agent receives a new inquiry. The large body of historical support cases remain in Iceberg, while Tiger Lake created automated chunk and vector embeddings in Postgres. Now vector search against the operational database can answer AI chat questions quickly, while ensuring that embeddings stay fresh and up-to-date without complex orchestration pipelines.  

In doing so, Tiger Lake is also a key building block in what we call Agentic Postgres, a Postgres foundation for intelligent systems that learn, decide, and act.

"With Tiger Lake, we believe Tiger Data is setting a strong foundation for turning Postgres into the operational engine of the open lakehouse for applications," said Ken Yoshioka, CTO, Lumia Health. "It allows us the flexibility to grow our biotech startup quickly with infrastructure designed for both analytics and agentic AI."

Companies like Speedcast, Lumia Health, and Pfeifer & Langen are already building full-context and real-time analytical systems with Tiger Lake. These architectures power industrial telemetry, agentic workflows, and real-time operations, all from a unified, continuously streaming platform.

Coming soon: Round-trip intelligence

• Later this summer: Query Iceberg catalogs directly from within Postgres. Explore, join, and reason across lakehouse and operational data using SQL.
• Fall 2025: Full round-trip workflows: ingest into Postgres, enrich in Iceberg and stream results back automatically. This lets developers move from event to analysis to action in one architecture.

How to set up Tiger Lake

Getting started is simple. No complex orchestration or manual integrations:

• Create a bucket for Iceberg-compatible S3 tables.
• Provide ARN permissions to Tiger Cloud.
• Enable table sync in Tiger Postgres:

ALTER TABLE my_hypertable SET (
  tigerlake.iceberg_sync = true
);

The Future of Data Architecture Is Real-Time, Contextual, and Open

Tiger Lake introduces a new kind of architecture. It is continuous by design, scalable by default, and optimized for applications that need full context and complete data in real time.

Operational data flows into the lakehouse for enrichment and modeling. Enriched insights flow back into Postgres for low-latency serving. Applications and agents complete the loop, responding with precision and speed.

We believe this is the foundation for what comes next:

• Systems that unify operational use cases and internal analytics
• Architectures that reduce complexity instead of compounding it
• Workloads that are not just reactive but grounded in understanding

You should not have to choose between context and simplicity. You should not have to patch together systems that were never designed to work together. And you should not have to replatform to evolve.

Together with next-generation storage architecture and our Postgres-native AI tooling, Tiger Lake forms the backbone of Agentic Postgres. This is a foundation built for intelligent workloads that learn, simulate, and act. We’ll share more soon.

Try it today on Tiger Cloud, and check out the Tiger Lake docs to get started.

— Mike

This article, written by Kamil Ruczyński, was originally posted on Apr 6, 2025 on his blog. Kamil is a Staff Software Engineer specializing in backend solutions, with a strong focus on scalability and performance. It's reposted here with permission.

Some time ago, I was working on improving the performance of slow statistics. The problem was that our database contained billions of rows, making data retrieval slow, even for the last seven days. From a product perspective, we needed to display data for at least 30 days and in real-time. All the data was stored in MySQL without partitioning, so we had to find a better solution. Simply using a cache was not an option, as real-time data was required.

Let’s analyze it on contrived example but similar to the original one.

MySQL Solution

Let’s say that we have a table with the following structure:

CREATE TABLE agent_stats (
    id BIGINT AUTO_INCREMENT NOT NULL,
    agent_id INT NOT NULL,
    event_type VARCHAR(255) NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    PRIMARY KEY (id)
);

CREATE INDEX idx_agent_stats_created_at_agent_id_event_type ON agent_stats (created_at, agent_id, event_type);

We’re collecting there statistics of our AI agents. Now, we have two types of events:

• triggered
• response_generated

To test the performance of the database, we need to generate a lot of data. In MySQL there is no fast way to generate random data, so I used some script which generated 24 234 964 records. It was quite slow.

And now, we want to get the number of triggered events for each agent in the last 30 days.

SELECT agent_id, event_type, COUNT(*) as count
FROM agent_stats
WHERE created_at > '2025-02-28 00:00:00'
GROUP BY agent_id, event_type

It’s very slow, and takes 11 seconds.

What is TimescaleDB?

TimescaleDB is an open-source time-series database built on PostgreSQL. It is designed to handle large volumes of time-series data efficiently. Basically it is a PostgreSQL extension that adds time-series capabilities to the database. It’s optimized for insertions of time-series data, and it provides features like automatic partitioning, retention policies, and continuous aggregates.

TimescaleDB Solution

So let’s try to use TimescaleDB to speed up our statistics. Creating a similar table in TimescaleDB:

CREATE TABLE agent_stats(
   created_at TIMESTAMPTZ NOT NULL,
   agent_id BIGINT NOT NULL CHECK (agent_id > 0),
   event_type VARCHAR(255) NOT NULL
);

SELECT create_hypertable('agent_stats', 'created_at');

CREATE INDEX ON agent_stats (created_at DESC, agent_id, event_type);

The create_hypertable function creates a hypertable, which is a TimescaleDB abstraction for a standard PostgreSQL table, but with automatic partitioning based on time.

Generating data in TimescaleDB is much convenient and a lot faster than in MySQL.

INSERT INTO agent_stats (created_at, agent_id, event_type)
SELECT
    time,
    random() * 9 + 1, /* 1 - 10 */
    'triggered'
FROM
    generate_series(
        '2024-01-01 00:00:00',
        '2025-03-28 16:00:00',
        INTERVAL '1 second'
    ) AS time;

INSERT INTO agent_stats (created_at, agent_id, event_type)
SELECT
    time,
    random() * 9 + 1, /* 1 - 10 */
    'response_generated'
FROM
    generate_series(
        '2024-01-01 00:00:00',
        '2025-03-28 16:00:00',
        INTERVAL '1 second'
    ) AS time;

In this way we can generate 2 records per second in the range of around 1 year and 3 months. I repeated this a few times, and in a few minutes total of 184 675 516 records were generated.

Now, let’s get the same data as before:

SELECT agent_id, event_type, COUNT(*) as count
FROM agent_stats
WHERE created_at > '2025-02-28 00:00:00'
GROUP BY agent_id, event_type

Keep in mind that we have much more data now, so the query it’s also slow. Now it takes 9 seconds, but of course on the same data as in MySQL it would be much faster, because of the partitioning. Ok, so now we need to speed it up.

TimescaleDB Continuous Aggregates

Continuous aggregates are a powerful feature of TimescaleDB that allows you to pre-compute and store the results of query aggregations over time. It’s based on the concept of materialized views in PostgreSQL, but it can return data real-time.

Let’s create a new continuous aggregate for agent_stats table:

CREATE MATERIALIZED VIEW hourly_agent_stats WITH (timescaledb.continuous)
AS
SELECT
    time_bucket('1 hour', created_at) as hour,
    agent_id,
    event_type,
    COUNT(1) AS occurrences
FROM agent_stats
GROUP BY
    hour,
    agent_id,
    event_type
WITH NO DATA
;

We also need to create a refresh policy to keep the materialized view up to date.

SELECT add_continuous_aggregate_policy('hourly_agent_stats',
    start_offset => INTERVAL '1 day',
    end_offset => INTERVAL '1 hour',
    schedule_interval => INTERVAL '1 hour'
);

Now we need to populate the materialized view with the data we already have in the table.

CALL refresh_continuous_aggregate('hourly_agent_stats', '2024-01-01', '2025-03-28');

It takes some time, and probably on the production it would be better to do it incrementally e.g. per day.

How it looks in the database:

Records in the view are aggregated by hour, so we have one record per hour per agent_id and event_type.

Now we can modify the query to take advantage of the newly created continuous aggregate:

SELECT agent_id, event_type, SUM(occurrences) as occurrences
FROM hourly_agent_stats
WHERE hour > '2025-02-28 00:00:00'
GROUP BY agent_id, event_type

This query is much faster, and it takes only 0.02 seconds.

It’s possible to get real-time data from the continuous aggregate, as historical data is fetched from the view, while the last hour (since we use a 1-hour bucket) is fetched from the source table.

TimescaleDB Retention

TimescaleDB also provides a retention policy feature that allows you to automatically drop old data. Let’s say that we need only 6 months of detailed data, and 1 year of aggregated data. We can set up the following retention policies:

SELECT add_retention_policy('agent_stats', INTERVAL '6 MONTH');

This will drop all data older than 6 months from the source table.

SELECT add_retention_policy('hourly_agent_stats', INTERVAL '1 YEAR');

This will drop all data older than 1 year from the continuous aggregate.

Summary

We could achieve some performance improvement by using partitioning in MySQL, but it would be a slight improvement and additional work. Adding TimescaleDB increases the complexity of the whole system, as it’s a new technology that needs to be maintained, but it’s a great choice for time-series data and great choice from the point of view of application engineers, as it provides a lot of useful features. However, if you’re using PostgreSQL now, using TimescaleDB will be easier to implement, as you don’t need to learn a new technology. It’s just an extension.

Database storage is a study in locality.

• Row stores keep all fields of a record together, with operations only on full rows at a time.
• Column stores organize each column’s values in compressed blocks, allowing operations to target specific columns.

This trade-off is structural, not just cosmetic. Row layout is great for fast inserts and lookups. Column layout excels at filters, aggregations, and scans over a large number of rows, but on a smaller number of columns (as long as your query plays by the rules).

But here’s the catch: Columnstores are only fast at filtering when your predicate aligns with the physical sort order. If your data is ordered by time, or clustered by a segmentby (like customer ID or device), the engine can skip large blocks. But if you filter on an unsorted field (like a trace ID, transaction UUID, or error code) there’s often no optimization to exploit. The engine has to decompress every block and scan every value, just in case there is a match.

TimescaleDB combines both layouts into a hybrid table model, with recent records written to the rowstore and automatically migrated to the columnstore over time. This design closely aligns with real-world query patterns, fresh data is updated frequently, while older data is rarely updated, and mostly used in aggregate. We’ve already implemented columnar mutability, but one challenge remained: sometimes you need to filter on unsorted fields across terabytes of columnar data. 

And it’s not hypothetical, we’ve seen it in the wild many times. Dashboards hang while users query a UUID, waiting as the engine churns through thousands of compressed blocks. The data is there. The filter is simple. But the system grinds.

That isn’t acceptable. At Tiger, we’re here to deliver speed without sacrifice.

So we implemented blocked bloom filters, and our users already love them: 

P.S. If you're using TimescaleDB 2.20 or later (or Tiger Postgres on Tiger Cloud), Bloom Filters are already actively optimizing lookups on sparsely distributed UUIDs, enums, and text fields by up to 100x.

The Challenge of Point-Lookups in Columnar Storage

If you've worked with large-scale time-series or analytics workloads, you've probably experienced this pain. You're querying 10TB of trace data to find a single ID like '550e8400-e29b-41d4-a716-446655440000'. Your database starts churning through millions of batches, reading and decompressing terabytes of data. Minutes and hours tick by. Your application times out. Users complain because they needed this report an hour ago (everything is urgent). 

Imagine you’re looking for a needle in hundreds of compressed bundles of haystacks—you have to unbundle, loosen, and search through every single bundle because you don't know which one contains your needle. 

This happens because columnar databases store data in sorted batches, compress each column separately, and use ordering metadata to skip irrelevant batches. This works perfectly when you're querying by the same column you sorted by:

-- This works well - time-based query on ordered data
SELECT * FROM metrics
  WHERE timestamp BETWEEN '2024-01-01' AND '2024-01-02';

But completely breaks down for uncorrelated columns:

-- This is painful - random ID query on non segmented column
SELECT * FROM metrics 
  WHERE trace_id = '550e8400-e29b-41d4-a716-446655440000';

The thing with UUIDs (except for UUIDv7, keep an eye out for support coming soon) is they're completely random, it doesn't make sense to order them. When your data is sorted by time, but you're searching by trace ID, every batch now contains a random mix of IDs. So ordering become useless, and the database can't skip any batches. 

What is a Bloom Filter? 

This is where bloom filters help—they're additional metadata that can efficiently answer "is this value definitely not in this batch?” without actually storing a reference to each value.

A bloom filter is a small-yet-efficient data structure that uses an array of bits and hash functions to quickly test if something might be in a set or not. 

Bloom filters can say something is "definitely not there", or "might be there", and crucially, they never say "it’s missing" when something is actually there.

Using Spotify’s playlist feature as an example, when Spotify needs to check if a song is in one of your playlists, instead of scanning through every song in every playlist (which means reading every playlist from storage—stupidly expensive with billions of songs), they use a bloom filter—a compact “summary” that can instantly say whether a song is “definitely not in this playlist” or “might be in this playlist”. 

For the “might be” cases, Spotify then uses traditional seek methods to check the actual playlist data. 

[Interactive Spotify Bloom Filter demo: https://spotify-bloom-filter.vercel.app/] 

This may not sound that useful, but when dealing with massive-scale workloads of millions of playlists and billions of songs, using a bloom filter eliminates 95%+ of linear-time playlist scans, turning minutes of searching into milliseconds. 

Obviously, no data structure is catch-free, it may occasionally check a playlist unnecessarily, around a 2% false positive rate (more on this later). But, this is an acceptable tradeoff as you still get massive I/O savings across your entire system. 

How We Added Bloom Filters into the Columnstore

Here's how TimescaleDB solves this, bloom filters act as a quick pre-check. Before reading any batch from disk, TimescaleDB checks a tiny bloom filter in memory that says "this ID is definitely not in this batch" or "this ID might be in this batch." This lets us skip 95%+ of batches instantly.

For the few batches that might contain your ID, TimescaleDB reads them from disk and processes them efficiently using vectorized operations (SIMD) that check many rows at once—much faster than Postgres's traditional row-by-row approach. But the real win is avoiding the I/O in the first place.

No manual configuration needed

When building with TimescaleDB, you don’t need to worry about when to use bloom filters or min/max indexes, because we automatically choose for you based on your column types! 

For columns that you use in your table ordering (like timestamps and numbers used in range queries), we stick with the min/max method because we know that scans will be in order.

For random things like text fields, UUIDs, enum types (or basically anything else that supports Postgres hash indexes) we will create bloom filters automatically (as long as you have the column indexed in the rowstore with a btree, hash or brin index). 

Diving deeper - “blocked bloom filters”

TimescaleDB uses a technique called a "blocked bloom filter", where each bloom filter starts at about 16KB per batch (sized for up to 1,000 items) with a 2% false positive rate and uses 6 different hash functions per value. 

The 16KB size isn't random—it's calculated based on math. Here’s a handy calculator you can try out yourself. 

For 1,000 items with a 2% false positive rate, the optimal formula gives us ~8K bits, but we round up to ~16k bits to enable our folding compression trick (we will get to this below!). This sizing ensures we get exactly the false positive rate we want while keeping the filters small enough to stay fast in memory.

The "blocked" part is a performance technique—instead of spreading hash bits all over a huge array, TimescaleDB keeps all the bits for one value within a 256-bit block. This fits nicely in your CPU cache and makes everything faster. 

For hashing, TimescaleDB primarily uses a modern library called UMASH that's faster than Postgres's built-in hashing, but falls back to the Postgres version for custom data types or older processors. There is a funny story here on interoperability that I’ll share on socials!

Achieving 250x space savings 

Here’s where we squeezed out additional space and performance benefits out of bloom filters— when a batch doesn’t have many unique values (for example [0, 0, 0, 0, 1, …, 0, 0]), TimescaleDB can compress the bloom filter by “folding” it in half using bitwise OR operations. 

It can keep folding until the filter shrinks from 16 KB down to just 64 bits (8 bytes) for columns with few unique values, also known as low-cardinality. We can do this because most of the bits in the batch are zero, so folding concentrates the few set bits without significantly increasing false positives. 

Query Walkthrough

Here’s an example to explain how queries work step-by-step. Let’s use the following trace ID search query: 

SELECT * FROM metrics 
   WHERE trace_id = 'abc123'

1. TimescaleDB first checks the bloom filters (which are likely to be cached in memory using the traditional Postgres buffer manager) for every batch in your columnstore.
2. For each batch, the bloom filter either says “definitely not here” or “might be here”. The database immediately skips all the “definitely not here” batches.
3. For the “might be here” batches, TimescaleDB reads them from disk, decompresses them, and scans the actual data (the expensive part). If it was a false positive (that 2% chance we mentioned prior), no match gets found, and the query just continues running normally. 

The key here is, false positives are okay and don’t impact performance because even when we hit one false positive, we are still avoiding massive amounts of unnecessary I/O by not having to go through every batch from the get-go. 

A side benefit to our implementation of bloom filters is that the bloom filter metadata is more likely to stay hot in memory. When there are concurrent workloads where different users are querying different parts of your dataset, every query can quickly eliminate most batches without touching slower-more-expensive storage. 

Where Bloom Filters Excel

Bloom filters excel at large time-series datasets queried by non-temporal identifiers.

Think about scenarios where you're storing massive amounts of data over time, but you need to find specific records using IDs, addresses, or other fixed identifiers.

For financial services teams. Your customers are trying to resolve a failed payment. They enter a transaction reference number into your search. Nothing loads. Your backend query scans years of data just to return a single match, and the user waits ...

-- Finding financial transactions by reference
SELECT * FROM payments 
  WHERE transaction_ref = 'TXN-2024-001234';

For IoT platform teams. Your dashboard shows live sensor data, but one widget is blank. It’s querying a single sensor reading by ID, and your backend is scanning billions of rows to find it. Users refresh the page. Nothing. The spinner keeps spinning.

-- Device-specific IoT data queries
SELECT * FROM sensor_data 
  WHERE reading_id = '73e98d71-5eb7-4018-ace7-1f4490da654a';

For teams working on blockchain analytics. Your analytics engine scans millions of blocks to find transactions from a wallet address. API timeouts leave users thinking your service is broken.

-- Looking up blockchain transactions by wallet address  
SELECT * FROM transactions 
  WHERE from_address = '0x742d35Cc6634C0532925a3b8D';

Bloom filters turn these queries from minutes or hours into milliseconds by eliminating the need to decompress and scan billions of rows. Instead of checking every batch in your columnstore, you skip a 95% of them and only decompress the ones that might actually contain your data.

Performance Numbers: Find Specific Values Up to 100x Faster 

Instead of telling you why our bloom filter implementation is great, let's just show you some numbers we got after we ran our benchmarks. 

SELECT min(sent), max(sent), count()
  FROM hackers 
  WHERE subject = 'unsubscribe' 
  ORDER BY count() DESC 
  LIMIT 10;

Before bloom filters, this took 12ms. With bloom filters, it dropped to 2.7ms—that's 3.5x faster.

Or consider this blockchain address lookup:

SELECT * FROM token_transfers 
  WHERE to_address = '0xe23d4eb73b399250301fb024019a734ba9f0d9b5';

This one went from 1.065 seconds down to 171.134 ms—a 6x improvement.

And lets not forget the report from our user pantonis, who saw a massive 100x improvement:

Where Bloom Filters Don’t Work 

Like all data structures, there are strengths and limitations.

Bloom filters work great when you're looking for exact matches—queries that use the equals sign (=) to find specific values. They also work with standard string comparisons where the rules are consistent. 

-- These work great with bloom filters

SELECT * FROM traces WHERE trace_id = 'abc-123-def';

SELECT * FROM orders WHERE email = 'user@example.com';

SELECT * FROM transactions WHERE status = 'completed';

However, bloom filters have fundamental limitations and some current implementation restrictions. By design, they can't handle "not equal" searches (<>) or range queries (< or >) because they only test set membership.

-- These don't work with bloom filters (fundamental limitations)
SELECT * FROM traces WHERE trace_id <> 'abc-123-def';

SELECT * FROM users WHERE created_at > '2024-01-01';

SELECT * FROM transactions WHERE amount BETWEEN 100 AND 500;

Current implementation restrictions in TimescaleDB mean they also can't help with multiple value searches (like WHERE column IN (1, 2, 3)) or cross-type comparisons without explicit casting. These may change in future versions.

-- These don't work yet (implementation restrictions)
SELECT * FROM traces WHERE trace_id IN 
  ('abc-123', 'def-456', 'ghi-789');

SELECT * FROM users WHERE user_id = 12345;  -- int8 = int4 comparison

SELECT * FROM posts WHERE category = ANY(ARRAY['tech', 'science']);

Also, bloom filters don't help much when the value you're looking for exists in most batches. 

For example, if you're searching for a common status like active that appears in every batch, the bloom filter will report a potential positive for every batch, forcing TimescaleDB to decompress and check them all anyway. The bloom filter can't skip anything, so you don't get any savings. 

Speed without Sacrifice

Bloom filters in TimescaleDB are a perfect example of "it just works" optimization and our commitment to making life easier for developers working at massive data scales.

The bloom data structure automatically kicks in for the right data types and query patterns, dramatically improving performance for needle-in-haystack queries without any configuration required. It works out of the box in Tiger Cloud—no setup required other than having an index on your rowstore columns.

You can verify bloom filters are working by looking for _timescaledb_functions.bloom1_contains in your query execution plans. The storage overhead is minimal, typically a few hundred bytes per batch, with a maximum of 1KB. For a table with a million batches, you're looking at roughly 100MB to 1GB of bloom filter metadata. That's 0.01% storage overhead for massive query speedups.

Built by developers, for developers. TimescaleDB refuses to accept the traditional trade-offs of database storage. We give you the speed of columnar analytics with the flexibility of point lookups, all in the same system.

Try it now on Tiger Cloud. 

Additional reading 

1. Speed Without Sacrifice: 2500x Faster Distinct Queries, 10x Faster Upserts, Bloom Filters and More in TimescaleDB 2.20 
2. Bloom Filters: The Unsung Heroes of Computer Science
3. Bloom Filter Calculator

Timescale is now Tiger Data.

TL;DR: Eight years ago, we launched Timescale to bring time-series to PostgreSQL. Our mission was simple: help developers building time-series applications.

Since then, we have built a thriving business: 2,000 customers, mid 8-digit ARR (>100% growth year over year), $180 million raised from top investors. 

We serve companies who are building real-time analytical products and large-scale AI workloads like: Mistral, HuggingFace, Nvidia, Toyota, Tesla, NASA, JP Morgan Chase, Schneider Electric, Palo Alto Networks, and Caterpillar. These are companies building developer tools, industrial dashboards, crypto exchanges, AI-native games, financial RAG applications, and more. 

We’ve quietly evolved from a time-series database into the modern PostgreSQL for today’s and tomorrow’s computing, built for performance, scale, and the agentic future. So we’re changing our name: from Timescale to Tiger Data. Not to change who we are, but to reflect who we’ve become. Tiger Data is bold, fast, and built to power the next era of software.

Developers Thought We Were Crazy

When we started 8 years ago, SQL databases were “old fashioned.” NoSQL was the future. Hadoop, MongoDB, Cassandra, InfluxDB – these were the new, exciting NoSQL databases. PostgreSQL was old and boring.

That’s when we launched Timescale: a time-series database on PostgreSQL. Developers thought we were crazy. PostgreSQL didn’t scale. PostgreSQL wasn’t fast. Time-series needed a NoSQL database. Or so they said.

“While I appreciate PostgreSQL every day, am I the only one who thinks this is a rather bad idea?” – top HackerNews comment on our launch (link)

But we believed in PostgreSQL. We knew that boring could be awesome, especially with databases. And frankly, we were selfish: PostgreSQL was the only database that we wanted to use.

Today, PostgreSQL has won. 

There are no more “SQL vs. NoSQL” debates. MongoDB, Cassandra, InfluxDB, and other NoSQL databases are seen as technical dead ends. Snowflake and Databricks are acquiring PostgreSQL companies. No one talks about Hadoop. The Lakehouse has won. 

Today, agentic workloads are here. 

Agents need a fast database. We see this in our customer base: private equity firms and hedge funds using agents to help understand market movements (“How did the market respond to Apple WWDC 2025?”); industrial equipment manufacturers building chat interfaces on top of internal manuals to help field technicians; developer platforms storing agentic interactions into history tables for greater transparency and trust; and so on.

What Started as a Heretical Idea Is Now a Thriving Business 

We have also changed. We met in September 1997, during our first week at MIT. We soon became friends, roommates, even marathon training partners (Boston 1998).

That friendship became the foundation for an entrepreneurial journey that has surpassed even our boldest imaginations. 

What started as a heretical idea is now a thriving business:

• 2,000 customers
• Mid 8-digit ARR, growing >100% y/y
• 200 people in 25 countries
• $180 million raised from top investors
• 60%+ gross margins

Cloud usage is up 5x in the last 18 months, based on paid customers alone.

And that’s only the paid side of the story. Our open-source community is 10x-20x larger. (Based on telemetry, it’s 10x, but we estimate that at least half of all deployments have telemetry turned off.)

TimescaleDB is everywhere. It’s included in PostgreSQL offerings around the world: from Azure, Alibaba, and Huawei to Supabase, DigitalOcean, and Fly.io. You’ll also find it on Databricks Neon, Snowflake Crunchy Bridge, OVHCloud, Render, Vultr, Linode, Aiven, and more.

We Are Tiger Data

Today, we are more than a time-series database. We are powering developer tools, SaaS applications, AI-native games, financial RAG applications, and more. The majority of workloads on our Cloud product aren’t time-series. Companies are running entire applications on us. CTOs would say to us, “You keep talking about how you are the best time-series database, but I see you as the best PostgreSQL.” 

So we are now “Tiger Data.” We offer the fastest PostgreSQL. Speed without sacrifice.

Our cloud offering is “Tiger Cloud.” Our logo stays the same: the tiger, looking forward, focused and fast. Some things do not change. Our open source time-series PostgreSQL extension remains TimescaleDB. Our vector extension is still pgvectorscale. 

Why “Tiger”? The tiger has been our mascot since 2017, symbolizing the speed, power, and precision we strive for in our database. Over time, it’s become a core part of our culture: from weekly “Tiger Time” All Hands and monthly “State of the Tiger” business reviews, to welcoming new teammates as “tiger cubs” to the “jungle.” As we reflected on our products, performance, and community, we realized: we aren’t just Timescale. We’re Tiger. Today, we’re making that official.

This is not a reinvention: it’s a reflection of how we already serve our customers today.

Polymarket uses Tiger Data to track their price history. During the last election Polymarket ramped up 4x when trade volumes were extra high, to power over $3.7 billion dollars worth of trades.

Linktree uses Tiger Data for their premium analytics product, saving $17K per month on 12.6 TB from compression savings. They also compressed their time to launch, going from 2 weeks to 2 days for shipping analytical features.

Titan America uses Tiger Data’s compression and continuous aggregates to reduce costs and increase visibility into their facilities for manufacturing cement, ready-mixed concrete, and related materials. 

Lucid Motors uses Tiger Data for real-time telemetry and autonomous driving analytics. 

The Financial Times runs time-sensitive analytics and semantic search. 

Tiger Is the Fastest Postgres for Modern Workloads

We are building the fastest Postgres: purpose-built for the modern operational workloads where traditional OLTP databases break down. 

Operational workloads that go far beyond simple transactions are now the norm. They require real-time, user-facing analytics over massive high-cardinality datasets, from event streams to time-series to user-level behavioral data. 

As the frontier moves further with agentic applications, the demands grow even more. These systems don’t just read and write: they observe, decide, and act. These AI applications require fast vector search across embeddings, and fast branching of data environments for experimentation and context-sensitive responses.

Tiger is not a fork. It’s not a wrapper. It is PostgreSQL, extended with innovations in the database engine and cloud infrastructure to deliver speed without sacrifice.

How are we so fast? Because of consistent, disciplined engineering efforts to serve customer needs over several years. Here is a non-exhaustive list: 

• Hypertables (2017)
• Native columnar compression (2019)
• Real-time materialized views for faster queries (2020)
• Decoupled compute and storage (2021)
• Tiered Storage to S3 Parquet (2022)
• Vectorized query execution for fast analytics (2023)
• Hybrid row-columnar store for faster queries on recent and historical data (2024)
• Faster vector workloads on PostgreSQL via pgvectorscale (2024)
• 300x faster mutations (updates, upserts, deletes) to compressed columnar data (2024)
• 2500x faster distinct queries, 6x faster point queries on high-cardinality columns (2025)
• Rapid horizontal scaling with load-balanced read replica sets (2025)
• Enhanced high-performance storage up to 64 TB and 32,000 IOPS (2025)

Tiger brings together the familiarity and reliability of Postgres with the performance of purpose-built engines.

We built the fastest PostgreSQL. Not because we wanted to, but because our customers wanted us to.

Building the Modern PostgreSQL for the Analytical and Agentic Era

PostgreSQL has won. The Lakehouse has won. Every application is becoming an analytical application. Agents are here, in production, and need to be fast. The future is hybrid, developers and agents, with better latency and throughput needs.

In this era, modern applications must:

• Handle terabytes and petabytes of data
• Support real-time analytics
• Integrate Gen AI features
• Serve both humans and software agents, across dev, test, and production lifecycles
• Meet sub-second latency and high concurrency expectations
• Scale across operational databases and cost-efficient lakehouses
• Maintain transactional integrity
• Deliver all of this reliably and cost-effectively, because data volumes grow much faster than budgets

Our history to date, our time in this market, our lived experience watching all these changes unfold in real-time screams to us one thing: modern applications need a new kind of operational database. 

One built for transactional, analytical, and agentic workloads. One that also acts as the operational serving layer for the Lakehouse. One built on Postgres.

That is what we are building.

And wow do we have some fun product announcements queued up for the upcoming weeks and months. A more agentic PostgreSQL. A deeper integration with the Lakehouse via Iceberg. A new compressed insert approach yielding 10 million rows per second. A new type of disaggregated storage architecture with zero-copy instant forks and replicas that we are deploying in our cloud for greater performance, as a replacement for EBS. And more.

We can’t wait to show it all to you. But first we had to clearly communicate who we are. We are Tiger Data. 

Come Join Us

Tiger is the Fastest PostgreSQL. The operational database platform built for transactional, analytical, and agentic workloads. The only database platform that provides Speed without Sacrifice.

This is not a rebrand, but a recommitment to our customers, to our developers, and to our core mission.

If this mission resonates with you, come join us. Give us product feedback. Spread the word. Wear the swag. Join the team. 

It’s Go Time. 🐯🚀

Yesterday, we covered how TimescaleDB 2.20 continues to deliver on our vision of building the fastest Postgres. Continuing on that theme, today we explore how read replicas and enhanced storage enable scaling for modern applications.

Traditional database solutions often struggle in key scenarios:

• Financial services: Millisecond delays in trading or payments due to conflicting analytical and transactional loads.
• IoT and telemetry: Massive sensor data overwhelms traditional setups, causing manual partitioning headaches and delayed insights.
• E-commerce: Traffic spikes during peak events slow performance, impacting user experience and revenue.

At Timescale, we believe developers shouldn't have to choose between performance and simplicity. Your database should scale seamlessly with your needs.

That's why we're launching two new features:

• Rapid scaling with read replica sets.
• Enhanced storage capabilities up to 64 TB and 32,000 IOPS per replica.

With these enhancements, Timescale Cloud empowers users to scale horizontally and vertically, delivering exceptional performance, throughput, and operational simplicity.

Read Replica Sets: Reliable Horizontal Scaling

Modern applications are read-heavy, but scaling reads typically means dealing with manual setups, custom routing logic, and constant tuning.

Read replica sets simplify this. Point your application to a single, load-balanced endpoint and let the system automatically distribute read queries across multiple replicas—no app changes required. Using PostgreSQL’s native streaming replication, replicas stay continuously updated without slowing down your primary node.

Benefits for developers:

• Automatic horizontal scaling of read traffic
• Zero custom logic or orchestration required
• Improved concurrency and performance under growing loads

How to Use Read Replica Sets: An Example

Read replica sets offer a powerful, flexible solution for managing read-heavy workloads. You can create multiple sets, each tailored to a specific need. For instance, dedicate one set with several replicas to handle high-volume read traffic from your production application, ensuring responsiveness. 

Create a separate set for your internal analytics team to isolate their ad-hoc queries and dashboarding activities, preventing impact on production performance.

By directing read traffic to the replica set and write traffic to the primary node, you achieve a clear separation of concerns. This architecture enhances scalability. As your read load fluctuates, you can easily add or remove replica nodes within a set via the Timescale console, adjusting capacity without disrupting your application or requiring manual changes. This consolidation behind a single, load-balanced endpoint simplifies your architecture while scaling read throughput.

Existing read replicas have been automatically upgraded to read replica sets with one node. Price remains the same.

Enhanced Storage: 64 TB Storage & 32,000 IOPS 

But there’s more. For customers with growing datasets and demanding analytical workloads, the previous 16 TB storage ceiling was becoming a constraint. That’s changing.

We're launching a new storage type called enhanced storage, powered by AWS EBS io2 volumes. This new type increases both storage capacity and throughput, making it ideal for customers with mission-critical workloads.

You can now scale your Timescale Cloud service:

• Up to 64 TB of storage per database service
• Up to 32,000 IOPS, enabling high-throughput ingest and low-latency analytics

You can switch to enhanced storage in the console without any downtime. This new storage type will initially be available to Enterprise customers and is part of our broader investment in serving high-scale, mission-critical applications.

This is a major step forward for demanding workloads with high ingestion rates, large query volumes, and complex analytics, such as financial data pipelines, IoT platforms, and telemetry processing.

Built for the Future of Your Application

These two new features are part of our broader vision: to make Timescale Cloud the most powerful and developer-friendly database for demanding applications.

With read replica sets and enhanced storage, Timescale Cloud delivers the speed and scalability needed for the most demanding workloads, while preserving the simplicity and power of Postgres.

How to Get Started

• Want to try read replica sets? Scale and Enterprise customers can set up read replica sets by logging in to the Timescale console > Operations > Read Scaling. Review our read replica sets documentation for implementation details.
• Need to break past the 16 TB barrier or boost IOPS up to 32,000? Enterprise users can switch to enhanced storage by logging in to the Timescale console > Operations > Compute & Storage.

Next up: database observability simplified with Timescale Cloud. 

The rush to integrate large language models (LLMs) into production apps has exposed a common failure mode: without proper authorization in place, they can easily expose sensitive data to the wrong users. Combine that with complex infrastructure (vector databases, sync pipelines, separate stores for embeddings and metadata), and you’re shipping a fragile system that puts user data at risk.

At Timescale and Oso, we think there’s a better way.

In this webinar, we show how you can build a secure, scalable AI chatbot using Postgres—and only Postgres—by leveraging Timescale’s pgai library and Oso’s authorization platform as a service.

Here are the webinar highlights, summarized for you in chapters for easy reference.

(To deploy our sample app for authorized secure chatbot built using Oso and pgai, see this open-source code.)

Why Most AI Chatbot Demos Fail in Production

[08:30–11:50]

Why do simple chatbots break in production? Demo chatbots are easy: embed your docs, slap on an OpenAI API key, and you’re done.

But in a real business environment, Bob (the employee) should never see Alice’s harsh performance review feedback. Only Alice, their manager and HR should. Sales shouldn’t see engineering tickets. 

Without authorization boundaries, your chatbot becomes a data leak waiting to happen.

Many demos fall short because they:

• Expose all content to all users
• Ignore org-specific permissions (e.g., team-level access control)
• Assume static or role-based authorization models
• Rely on dual data systems (e.g., Postgres + Vector DB), causing data synchronization difficulties.

The fix? Build with authorization and data consistency as first principles.

Why We Combined Postgres, pgvector, and Oso

[13:34–17:47]

We introduced an end-to-end reference stack that solves both the data synchronization and authorization complexity problem. The solution uses:

• Timescale + pgai for real-time, in-database vector search and updates
• Oso Cloud for relationship-based access controls, enforced natively via PostgreSQL
• No glue code or ETL scripts between systems

The result: you get a secure, performant, and authorized chat system with zero duplicated data.

Real-Time Vector Sync With pgai Vectorizer

[14:33–20:45]

Instead of bolting a vector database on top of your existing Postgres database, pgai Vectorizer keeps your embeddings automatically synchronized with your source data in Postgres.

• Create vectorizers via Python
• Ingest from S3, Hugging Face, or existing Postgres tables
• Bring your own embedding model (OpenAI, Nomic, etc.)
• Chunk and embed documents with configurable rules
• Never worry about mismatched records again

SELECT ai.create_vectorizer(
  'blog'::regclass,
  loading => ai.loading_column(column_name => 'content'),
  embedding => ai.embedding_openai(model => 'text-embedding-3-small', dimensions => 768),
  destination => ai.destination_table('blog_embeddings')
);

Run your vectorizer worker:

pgai vectorizer worker -d postgresql://...

No extra queues, pipelines, or lambdas needed. Just Python and Postgres.

Authorization That Follows Relationships, Not Just Roles

[21:43–28:14]

Many apps rely on Role-Based Access Control (RBAC). But real-world permissions often depend on relationships:

• “Bob can view reviews only if he’s the owner of the document”
• “Diane (HR) can see feedback others can’t”
• “Support engineers can access sensitive logs only during active shifts”

Oso lets you model this in code:

resource Folder{
 roles = ["viewer"];
 permissions = ["view"];
 relations = { team: Team };


 "viewer" if "member" on "team";
 "viewer" if global "hr";
 "viewer" if is_public(resource);


 "view" if "viewer";
}

It also incorporates your Postgres data using native SQL, so you don’t need to sync users, roles, or groups into a second system.

Putting It Together: Authorized Retrieval Augmented Generation (RAG)

[30:44–37:32]

Here’s how the architecture works:

1. A user (Bob or Diane) sends a question to the chatbot.
2. The app queries Oso to determine what data the user is authorized to access.
3. That filter is converted to a SQL query that joins source + embedding data in Timescale.
4. Only the authorized context is sent to the LLM (e.g., OpenAI) to generate a final response.

The result: the same chatbot provides personalized, secure answers based on who’s asking—without leaking data or requiring redundant systems.

What You’ll Learn From the Demo

[29:01–48:00]

• How to build a business-grade RAG stack without a separate vector DB
• How to enforce field-level access control in LLM-based apps
• How Timescale + pgai + Oso make Postgres the only data system you need
• Why prompt engineering, chunking, and system prompts matter in retrieval quality
• How to embed PDF, DOCX, and S3-based documents securely

Next Steps

We’ve open-sourced the reference app and walkthrough:

• Watch the full webinar
• Explore the Timescale pgai docs
• Learn more about Oso Cloud
• Join the Oso community on Slack

If you’re building AI agents, chat interfaces, or internal copilots—don’t wait to layer in security and data correctness.

Your users will thank you. Your auditors will too.

Welcome to part one of our Agentic RAG Best Practices series, where we cover how to load, parse, and clean documents for your agentic applications. 

This comprehensive guide will teach you how to build effective agentic retrieval applications with PostgreSQL.

Every week, customers ask us about building AI applications. Their most pressing concern isn't advanced chunking strategies or vector databases—it's simply: "How do I clean my data before feeding it to my AI?"

It’s simple: “Garbage in, garbage out.” 

Before worrying about writing even your first line of embeddings or retrieval code, you need clean data.

In this first guide of our agentic RAG series, we'll cover gathering the right data, extracting text from various document types, pulling valuable metadata, web scraping techniques, and effectively storing data in PostgreSQL. We'll address common challenges like fixing formatting issues and handling images in documents. 

By the end, you'll know how to transform raw documents into clean, structured data that retrieval agents can effectively use. 

Don’t want to read all of this and just want to apply it? We have prepared a handy dandy preparation checklist for this topic in a preparation checklist.

One fintech customer recently shared how they spent weeks fine-tuning their RAG application with different vector databases, only to realize their poor results stemmed from simply having dirty data. "I approached the whole thing with like, I don't trust these AIs (. … ) So we don't ask them to make decisions. We do normal modeling to figure out what the user needs, then feed that data to the LLM and just say, 'Summarize it.'" 

The garbage in, garbage out principle applies strongly to AI applications. Let's explore how to properly load, parse, and clean your data for AI use. 

Gathering the Right Data for AI Applications

👉🏻 Watch the one-minute video summary.

Before even thinking about cleaning or processing your data, you need to make sure you have the right data in the first place. I know, this sounds obvious, but it’s a very important step that many teams overlook in rushing to build their shiny RAG app. 

Data selection matters

We have seen many AI teams build state-of-the-art RAG apps that still deliver bad answers. Most of the time, there is nothing wrong with their retrieval algorithm, vector database, embedding model, or large language model. The problem is that they simply don’t have the necessary information in the knowledge base, so the LLM made something up instead or provided insufficient answers. 

In most cases, if the information doesn’t exist in your documents, your RAG app should either return nothing (the best-case scenario), or the LLM will simply hallucinate a plausible answer (this is the worst-case scenario). 

Choosing the right data

Before building your RAG application, ask yourself and the team these questions:

1. What specific questions will users ask the system? 
2. What documents contain factual information to these questions? 
3. What are the gaps in our current documentation? 
4. Is our information up-to-date, or will it need to be regularly updated? 
5. Do we have a system in place to identify information gaps as users use the app?

You need to be able to confidently answer these questions. 

Where to collect data

1. Internal knowledge base: Check company wikis, technical documentation, reports, manuals, and databases. 
2. External sources: Read industry publications, research papers, and public datasets. 
3. Customer interactions: Check support tickets, chat logs, FAQs, etc. 
4. Real-time sources: See news feeds, market data, IoT sensor data, etc.
5. Intuition: You may have some ideas where certain important data lives, so trust your gut.

Be intentional about your data sources—the higher the quality and relevancy, the better. 

Ensure data freshness

Most business data isn't static—it often changes as your products, services, and policies evolve. Outdated information in your RAG system leads to incorrect answers and really hurts your customers’ trust in the AI system (I mean, look at Google’s initial rollout of Bard.) 

Consider the following suggestions for keeping your knowledge base up-to-date:

1. Set up consistent update schedules: This will be different depending on your business needs. It can be hourly, weekly, monthly, or even quarterly.
2. Implement trigger-based updates: Update content whenever the source document changes. For example, when your team updates some documentation, your system should automatically refresh the corresponding knowledge base entries.
3. Create document ownership: If you work in a large company, you may need to assign responsibility to other individuals or teams for specific knowledge areas to ensure data is constantly updated.
4. Track user feedback: Many RAG systems allow users to rate answers. This rating system (like a simple thumbs up and down) can help identify outdated or incorrect information that needs to be updated, added, or removed from your knowledge base. 
5. Track question patterns: Continuously analyze questions that consistently receive poor ratings to identify areas where your knowledge base needs improvement. 

Data freshness is one of the silent killers of data accuracy—no advanced RAG pipeline can fix this. 

Extracting Text From Documents

👉🏻 Watch the one-minute video summary.

Approximately 85 percent of the world's data is unstructured: think PDFs, Word files, emails, PowerPoint presentations, and more. To use this data with AI, you first need to extract the raw text.

Using MarkItDown for Document Conversion

Libraries like MarkItDown and Docling can convert PDFs and other formats to Markdown. Markdown has become one of the cleanest and most efficient formats for ingesting data into LLMs because it's nearly plaintext and token-efficient. It can also efficiently represent non-text data like tables. 

Extract text from PDF using MarkItDown 

from markitdown import MarkItDown  
md = MarkItDown()  
result = md.convert("document.pdf")  
text_markdown = result.text_content  
print(text_markdown[:500])

Extract text from PDF with Optical Character Recognition (OCR) using Docling

# Using Docling for Document Conversion with OCR
from docling.document_converter import DocumentConverter

# Initialize the converter
converter = DocumentConverter()

# Load PDF and extract text with OCR enabled
result = converter.convert(
    "document.pdf",  # Can be local path or URL
    enable_ocr=True  # Enable OCR for scanned documents
)

# Get the converted markdown content
markdown_text = result.document.export_to_markdown()

# Preview the first 500 characters
print(markdown_text[:500])

The code above returns an object with text_content containing the markdown text, which you can easily pass into your RAG pipeline or LLM for cleaning, analysis, summarizing, or chunking. 

Using Visual Language Models for OCR

A new breed of OCR technology is being powered by visual large language models (VLLMs): models that can process not just text, but also images and PDFs. These are trained specifically for unstructured data extraction. One such VLLM making a splash is Mistral OCR.

Extract text and images (in base64) from PDF using Mistral OCR

import os
from mistralai import Mistral

api_key = os.environ["MISTRAL_API_KEY"]
client = Mistral(api_key=api_key)

ocr_response = client.ocr.process(
    model="mistral-ocr-latest",
    document={
        "type": "document_url",
        "document_url": "https://arxiv.org/pdf/2201.04234"
    },
    include_image_base64=True
)

Extract from images using Mistral OCR

import os
from mistralai import Mistral

api_key = os.environ["MISTRAL_API_KEY"]
client = Mistral(api_key=api_key)

ocr_response = client.ocr.process(
    model="mistral-ocr-latest",
    document={
        "type": "image_url",
        "image_url": "https://raw.githubusercontent.com/mistralai/cookbook/refs/heads/main/mistral/ocr/receipt.png"
    }
)

What makes Mistral OCR unique is its exceptional performance in extracting text in multiple languages, handling text from images, representing math equations, interpreting structured tables, and other traditionally difficult tasks. 

Other extraction tools you can experiment with include unstructured.io,  olmOCR, or just relying on good ol’ humans to extract the data—Upwork or Fiverr is a good place to begin your search for contractors. 

Once you have this more manageable text form, you're ready for either direct ingestion into your database or metadata extraction. 

Metadata Extraction

👉🏻 Watch the one-minute video summary.

All documents contain metadata like title, author, creation date, length, source, customer name, etc. Imagine needing to fetch all documents between Q1 and Q2 of 2025 for a financial report—you'd need to filter by date range using metadata.

If your PDFs or documents have built-in metadata (added automatically by document processors when saving or exporting), that's great! But what if they don't? 

Extracting built-in metadata

For simple metadata extraction from actual PDF data (if available), you can use a library like fitz:

Extract built-in PDF metadata using fitz

import fitz  
doc = fitz.open("example.pdf")  
metadata = doc.metadata  
print(metadata.get("title"), "by", metadata.get("author"))

For everything else, you need…

Contextual metadata extraction

Most documents don't have native metadata. In these cases, you need a two-step workflow: first, use a PDF text extractor like Mistral OCR, then pass the raw text to another large language model to request specific information using natural language.

For example, you can use Mistral OCR to analyze each document, define what metadata you'd like to extract (title, author, etc.), and use another LLM to get the metadata information formatted in a specific way (like JSON).

Extract contextual metadata from PDF using Mistral OCR and Mistral Small

import os
from mistralai import Mistral

# Retrieve the API key from environment variables
api_key = os.environ["MISTRAL_API_KEY"]

# Specify model
model = "mistral-small-latest"

# Initialize the Mistral client
client = Mistral(api_key=api_key)

# Define the messages for the chat
messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": "In JSON format, extract the following metadata from the provided document: title, author, and created data. "
            },
            {
                "type": "document_url",
                "document_url": "https://arxiv.org/pdf/1805.04770"
            }
        ]
    }
]

# Get the chat response
chat_response = client.chat.complete(
    model=model,
    messages=messages
)

# Print the content of the response
print(chat_response.choices[0].message.content)

# Save the metadata somewhere for later ingest into your RAG pipeline

Extracting Text From the Web

👉🏻 Watch the one-minute video summary.

Not all data lives in documents, and not all is accessible via GET API requests. To get data from websites, documentation, and knowledge bases for AI applications, you need to scrape them. The ultimate goal of web scraping is to fetch only the main text content from pages, filtering out headers, footers, sidebars, ads, and tracking scripts, in an LLM-friendly format like Markdown.

In the past, this was done with libraries like requests, Selenium, BeautifulSoup, etc., and required manually setting up proxies to evade rate limiters. Thankfully, it's no longer as painful to scrape the web today (yay!).

Web scrapers generally need to do the following tasks:

1. Crawl: Get all pages of an entire website by gathering a list of all internal and external links (essentially building a sitemap, if it’s not available on /sitemap.xml).
2. Scrape: Get the DOM/text content of each individual page.
3. Proxy: Switch to a different IP to continue on a large crawling and scraping job. 
4. Clean: Extract the main useful text from the raw DOM.
5. Convert: Format the main text content as Markdown, TXT, JSON, etc. 

Firecrawl for Web Scraping

Firecrawl is a web scraping/crawling engine accessible via REST API, Python SDK, and a UI dashboard (currently in beta). What's great about Firecrawl is that it extracts clean page text in various formats. It can crawl an entire site and return all pages' content in one go (with advanced filtering options). It also handles all the proxying needed for large-scale crawl and scrape jobs.

Crawling a website with a limit of 100 pages using Firecrawl Crawl REST API

from firecrawl import FirecrawlApp

app = FirecrawlApp(api_key="fc-YOUR_API_KEY")

# Crawl a website:
crawl_status = app.crawl_url(
  'https://firecrawl.dev', 
  params={
    'limit': 100, 
    'scrapeOptions': {'formats': ['markdown', 'html']}
  },
  poll_interval=30
)
print(crawl_status)

Scraping a URL and outputting in Markdown using Firecrawl Scraping REST API

from firecrawl import FirecrawlApp

app = FirecrawlApp(api_key="fc-YOUR_API_KEY")

# Scrape a website:
scrape_result = app.scrape_url('firecrawl.dev', params={'formats': ['markdown', 'html']})
print(scrape_result

Custom metadata extraction in JSON using Firecrawl Extraction REST API

from firecrawl import FirecrawlApp
from pydantic import BaseModel, Field

# Initialize the FirecrawlApp with your API key
app = FirecrawlApp(api_key='your_api_key')

class ExtractSchema(BaseModel):
    company_mission: str
    supports_sso: bool
    is_open_source: bool
    is_in_yc: bool

data = app.scrape_url('https://docs.firecrawl.dev/', {
    'formats': ['json'],
    'jsonOptions': {
        'schema': ExtractSchema.model_json_schema(),
    }
})
print(data["json"])

Other Web Scraping Options

Firecrawl isn't the only service/library for crawling, scraping, and cleaning. Another capable service is Jina AI's Reader API, which converts a URL to LLM-friendly inputs simply by adding r.jina.ai in front:  

Fetch a webpage in clean Markdown using Jina AI’s Reader API

r.jina.ai/news.ycombinator.com

If you want to build your own end-to-end crawling and scraping infrastructure (expert users only), developers typically use Playwright, a Microsoft framework for web testing and automation. Playwright Web Scraping is a reliable open-source web scraping implementation using Playwright. Firecrawl is also open source and lets you host it in your own infrastructure if you want absolute control. 

Direct Data Loading 

pgai has a handy function that lets you import datasets directly from Hugging Face with just the dataset's name:

Load data from Hugging Face using pgai

SELECT ai.load_dataset('wikimedia/wikipedia', '20231101.en', table_name=>'wiki', batch_size=>5, max_batches=>1, if_table_exists=>'append');

pgai has more direct data loading goodies to come—stay tuned. 

Storing Data

👉🏻 Watch the one-minute video summary.

At Timescale, we believe boutique vector databases are the wrong abstraction for AI workloads. PostgreSQL is the ideal solution for typical apps and AI apps, especially RAG applications.

Instead of using a separate vector database, you can store text embeddings inside PostgreSQL with pgvector. We recommend using pgai to simplify building RAG apps, as we have a handy interface called create_vectorizer() that automatically embeds raw text, chunks it, and continuously keeps it up-to-date.

Create an AI project using pgvector and pgai

// Enable pgai and pgvector on your Postgres database
CREATE EXTENSION IF NOT EXISTS vector;  
CREATE EXTENSION IF NOT EXISTS ai;  

// Create a table to store Wikipedia articles
CREATE TABLE wiki (
    id      TEXT PRIMARY KEY,
    url     TEXT,
    title   TEXT,
    text    TEXT
);

// Load Wikipedia dataset directly from HuggingFace 
SELECT ai.load_dataset('wikimedia/wikipedia', '20231101.en', table_name=>'wiki', batch_size=>5, max_batches=>1, if_table_exists=>'append');

// Create Vectorizer that
// 1. Chunks the data using the chunking recursive character text splitter
// 2. Embeds it using Mini LM at 384 token size per chunk
// 3. Continuously monitors the wiki table for new text incoming
SELECT ai.create_vectorizer(
     'wiki'::regclass,
     embedding => ai.embedding_ollama('all-minilm', 384),
     formatting=> ai.formatting_python_template('url: $url  title: $title $chunk')
     chunking => ai.chunking_recursive_character_text_splitter('text'),

);

// Check status of Vectorizxer embedding creation
select * from ai.vectorizer_status;

Running the pgai Vectorizer worker

For the vectorizer to work correctly, you need to run the pgai Vectorizer worker alongside your PostgreSQL database. This worker processes your data and creates embeddings. Set up a docker-compose.yml file with the following configuration:

version: '3'
services:
  db:
    image: timescale/timescaledb-ha:pg17
    environment:
      POSTGRES_PASSWORD: postgres
    ports:
      - "5432:5432"
    volumes:
      - data:/home/postgres/pgdata/data
      
  vectorizer-worker:
    image: timescale/pgai-vectorizer-worker:latest
    environment:
      PGAI_VECTORIZER_WORKER_DB_URL: postgres://postgres:postgres@db:5432/postgres
      OPENAI_API_KEY: your_openai_api_key_here
    command: [ "--poll-interval", "5s" ]
    
  ollama:
    image: ollama/ollama
    
volumes:
  data:

If you're using Ollama for embeddings, as shown in our example, make sure to add the Ollama service and configure the worker:

vectorizer-worker:
    environment:
      OLLAMA_HOST: http://ollama:11434

Start everything with “docker-compose up -d” and the worker will automatically poll the database and process your vectorizer tasks. Note that you might need to adjust settings like poll intervals or concurrency depending on your specific workload needs.

And just like that, we’ve built a production-ready SQL-native retrieval pipeline that is not only powerful but extremely customizable. 

Cleaning Messy Data

Raw text extracted from websites or documents is often messy and contains content not relevant to the main text. This can include ads, navigation menus, footers, tracking scripts, or leftover HTML/CSS markup. Removing this noise is crucial to avoid feeding irrelevant text to your AI model, reduce input token size to lower costs, and increase retrieval accuracy.

Cleaning webpages

Elements to clean include:

• HTML tags that aren't content: <script>, <style>, <nav>, <footer>, etc.
• Advertisements or cookie banners
• Repeated headers/footers on every page
• Excessive whitespace, line breaks, or meaningless Unicode characters
• Images or links to images

The tools mentioned earlier, like Firecrawl and Jina AI's Reader API, already handle webpage data cleaning and return only the main text content.

If you have very specific requirements, you can use web automation frameworks like Playwright or BeautifulSoup to get the raw DOM, then use traditional DOM traversal or regex to clean the data. This approach is for experts only.

Cleaning PDFs

After running documents through Mistral OCR, you'll still have repeated content like page numbers, headers/footers, and repetitive line breaks. A growing technique is using an LLM like Gemini Flash 2.0, which has a two-million-token context window (the largest of all LLMs we've seen) and a reasonable cost to automate cleaning. You can use natural language to instruct it on what to clean: removing repeated titles, sources, footnotes, etc.

You can also clean text manually if you have very specific data requirements. 

Fixing Text Formatting

Sometimes text is extracted with poor formatting. You need to standardize lists, headings, and line breaks. You can prompt an LLM to rewrite text more clearly, remove gibberish or irrelevant parts, and correct inconsistencies (without adding extra commentary).

Issues to fix include:

• Line breaks and paragraphs: Merge lines that belong to the same paragraph. For example, replace hyphenated line breaks (-\n) with nothing, and replace newlines followed by lowercase letters with spaces.
• Lists and bullet points: Convert fancy bullet symbols to a common format (e.g., "•" or "–" to "-"). Ensure list items have consistent formatting.
• Headings and subheadings: If using Markdown, ensure headings use # syntax properly with blank lines before and after.
• Whitespace and punctuation: Trim excessive whitespace, normalize quotes and dashes if needed.
• Tone: Standardize tone using large context LLMs like Gemini Flash 2.0 to keep writing style consistent across your text data.

Clean variety of OCR text formatting issues using regex (not an exhaustive example)

import re

def clean_ocr_text(text):
    # Replace hyphenated line breaks (e.g., "exam-\nple" -> "example")
    text = re.sub(r'(\w)-\n(\w)', r'\1\2', text)

    # Merge lines that are broken in the middle of sentences
    text = re.sub(r'\n(?=\w)', ' ', text)

    # More cleaning steps go here

    return text

Handling Images in Documents 

Some PDFs and web pages have images containing text or are entirely scans of documents. There are several ways to handle these:

1. OCR: For scanned documents, OCR can extract text from images. Mistral OCR handles this well.
2. MarkItDown: Modern libraries like MarkItDown integrate both OCR and vision models to generate image descriptions.
3. BLIP: Models like BLIP (Bootstrapping Language-Image Pre-training) combine understanding images and generating text to give you text descriptions of images.
4. Save image URLs: When extracting from webpages, you can save image URLs as text chunks to display in your application.
5. Omit images: This is common but not ideal, especially if images contain crucial information like charts or diagrams.

Generate descriptions for images using MarkItDown

from markitdown import MarkItDown
from openai import OpenAI

# Set up OpenAI client
client = OpenAI(api_key="your-openai-api-key")

# Initialize MarkItDown with LLM capabilities
md = MarkItDown(llm_client=client, llm_model="gpt-4o")

# Convert an image file
result = md.convert("path_to_your_image.jpg")

# Print the generated description
print(result.text_content)

Generate captions for an image using BLIP 

from transformers import BlipProcessor, BlipForConditionalGeneration
from PIL import Image

# Load the pre-trained BLIP model and processor
processor = BlipProcessor.from_pretrained("Salesforce/blip-image-captioning-base")
model = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-base")

# Open an image file
image_path = "path_to_your_image.jpg"  # Replace with your image file path
image = Image.open(image_path)

# Preprocess the image and prepare inputs for the model
inputs = processor(images=image, return_tensors="pt")

# Generate caption
outputs = model.generate(**inputs)

# Decode the generated caption
caption = processor.decode(outputs[0], skip_special_tokens=True)

print("Generated Caption:", caption)

Summarizing or Cleaning Text With an LLM

After programmatic cleaning and parsing, you may still need to refine text further:

• Summarize long documents into shorter summaries or key points.
• Make tone and format consistent.

Simple prompts like "Remove any unnecessary information (like boilerplate) from the following text and correct any errors" or "Summarize the following document in three sentences" can help. If possible, double-check this work, as LLMs can sometimes misinterpret context.

Summarize a piece of text using Gemini Flash 2.0 

from google import genai
from google.genai.types import HttpOptions

client = genai.Client(http_options=HttpOptions(api_version="v1"))
response = client.models.generate_content(
    model="gemini-2.0-flash-001",
    contents="Summarize in a technical tone the following piece of text: Attention Is All You Need...",
)
print(response.text)

Conclusion

We hear it from our customer’s AI teams all the time. They had spent months trying different vector databases, embedding models, and chunking strategies without seeing any improvement in their RAG application. After a quick data review, we discovered their PDFs were being processed with poor OCR, resulting in garbled text full of artifacts. Within a week of implementing proper data cleaning, their application performance jumped dramatically. 

There’s a very important lesson here: Before you worry about the latest bleeding-edge GraphRAG techniques, make sure your data foundation is solid; this step starts way before you write a single line of RAG code. Clean, well-structured, and high-quality data is the foundation of any successful AI application. As we like to say, “Garbage in, garbage out.” 

Start with good data, and the rest of your AI pipeline will fall into place. You need to invest time in proper document gathering, loading, parsing, and cleaning, which will pay dividends in more accurate, relevant, and useful AI outputs. 

In the next installment of RAG Best Practices, we will be exploring chunking strategies, followed by embedding generation, indexing techniques, performance optimizations, and more. 

Get Involved 

Whether you're new to AI or an experienced developer looking to implement agentic RAG with PostgreSQL, this series will give you the foundation you need. 

Stay tuned for our next guide on chunking strategies, coming in two weeks.

In the meantime, we'd love to see you share your thoughts, questions, and suggestions on social media and Discord:

• Join our Discord Community: Get real-time answers from the Timescale team and connect with other developers.
• Follow us on social media: Stay updated with the latest from Timescale on X/Twitter and LinkedIn.
• Connect with Jacky (developer advocate): Follow me for more practical AI and PostgreSQL content on X/Twitter, Threads, and TikTok.
• Direct questions: Have a specific question about your agentic retrieval implementation? Ask me anything at jacky (at) timescale (dot) com.

We're building this guide for you, so don't hesitate to let us know what topics you'd like us to cover in future installments! 

Updated Jan. 27, 2026 - After launching our Insights feature in late 2023, our most ambitious dogfooding effort yet, where we scaled Postgres to give users in-depth performance analytics on their database queries, we’re back with an update. And good news: we'll be sharing these metrics quarterly. 

The TL;DR? One Tiger Cloud database service is now ingesting over 3 trillion metrics per day and storing 3 petabytes of data, challenging all assumptions that Postgres can’t scale.

This massive operation runs entirely on Tiger Cloud using the same features available to all our users. There is no special treatment, no hidden infrastructure: you, too, could run a Postgres database at this scale with the offerings on Tiger Cloud. 

Insights Recap: Scaling Postgres for Query Monitoring

To understand the scale of the problem we’re trying to solve, let’s quickly recap the feature being powered here by Tiger Data. Insights provides Tiger Cloud users with comprehensive query analytics, capturing execution times, memory usage, I/O statistics, and TimescaleDB feature utilization.

This means we capture every query running in our Cloud, gather relevant statistics, and store them in a fully queryable Tiger Cloud instance. Initially collecting about a dozen metrics per query, we've since tripled that number to enhance user visibility. The data volume expands along three dimensions: growing customer base, increasing per-customer query loads, and an expanding metrics collection.

Yet we continue to track this data in Tiger Cloud, on a Postgres-based database, accomplishing Tiger Data's original goal of creating a faster, more scalable Postgres.

Just Give Me the TL;DR (a.k.a. the Big Numbers)

When we bragged talked about building Insights, the headline numbers were storing 350+ TBs of data and ingesting 10 billion records a day. Today, we’ll change the headline numbers to more than 1 quadrillion metrics recorded, almost 3 petabytes stored, and over 3 trillion daily metrics ingested.

Since launching the feature, we’ve refined how we measure Insights data, focusing on metrics rather than records—a record is a set of metrics for a query—with each query now capturing significantly more data points than in 2023.

That said, the numbers are impressive: we’ve moved into multi-petabyte territory, adding roughly a petabyte of data per year since launching 3 years ago. Today, we stand at nearly 3 petabytes, the overwhelming majority of which is efficiently stored in Tiger Data’s tiered storage architecture, which is more easily accessible and query-able than ever.

Our daily ingest has tripled over the last year, from 1 trillion to 3 trillion metrics per day, totaling 1 quadrillion metrics collected. Despite this massive growth in data, queries, and more metrics per query, we still use the same size bowl for our dogfooding effort: a vanilla Tiger Cloud instance.

How We Scale Postgres and Stay Fast

Much of our architecture remains the same as described in our original post. We still ingest two main types of objects: a detailed “raw” record and a set of UDDSketches that represent a distribution of values for a given metric (“sketch” record). 

A raw record contains the metrics for a single query invocation, along with some more detailed information, like a full breakdown of nodes used to execute the query. Conversely, the set of UDDSketches represents multiple query invocations. This allows us to store orders of magnitude more queries’ stats than if we stored only raw records. 

Since launching the feature, we have generally sampled fewer raw records, now only collecting about 25% of queries in this form. The node breakdown of execution can be useful in understanding how custom nodes we’ve created for TimescaleDB are performing across the fleet.

Adding new metrics to track has been straightforward—just new columns on our existing hypertables. Because we’ve essentially tripled the amount of metrics we collect, this does put more pressure on storage.

For raw records, as previously mentioned, we have just reduced the amount of sampling while continuing to aggressively tier data. For the sketch records, we’ve also begun using tiering for this table. This lets us keep our active dataset for the database around 12 TB (60 TB of pre-compressed data before using Timescale's row-columnar storage engine), with the rest (1.5+ PB) tiered.

To allow for aggressive tiering and quick responses to queries from our Insights page, we use continuous aggregates (our enhanced version of Postgres materialized views) heavily. UDDSketches “roll up” very nicely: you can combine a set of UDDSketches into a new UDDSketch representing the entire group. This allows us to go from the ingested UDDSketches into a hierarchical continuous aggregate tree with groupings at several levels (minutes, hours, days). 

With a bit of planning, we’ve been able to have stats available at all the granularities we need to serve users without needing to go to the original hypertables. Inserts stay fast, queries stay fast, and we can tier without fear.

In the future, we may need to deploy read replicas to scale the solution, allowing us to separate the high write ingesting and aggregation workload from the high read workload that comes from customer usage. But as it stands today, we don’t need that; we have this billions-of-metrics-a-day pipeline running perfectly without scaling out.

Final Words

Since its inception, Insights has grown in both scale and impact, proving that Postgres—when engineered for scale—can handle immense workloads. 

We're tracking trillions of metrics daily while storing petabytes of data—all on a Tiger Cloud instance. The power of Tiger Data’s tiered storage, hypertables, and continuous aggregates has allowed us to not just scale but to stay fast and efficient. Glooko uses the same Tiger Cloud features to process 3 billion data points per month for diabetes monitoring.

If you’ve been thinking about taking your Tiger Cloud database to the next level, rest assured, we’re showing it’s entirely possible—our Cloud is your Cloud. And remember, you will never walk alone. Top-notch support is available for free for all Tiger Cloud customers, and our expert team is ready to guide you every step of the way, all the way to petabyte scale. 

Start scaling—create a free Tiger Cloud account today.

Let’s imagine a few scenarios:

• You are asked to build a monitoring system to check if a PostgreSQL instance is running smoothly. You need to check its status every second, but your organization doesn’t want to use third-party tools due to concerns about sharing sensitive credentials.
• You have just installed PostgreSQL and want to make sure it’s ready to accept connections.
• Your application suddenly stops connecting to the database, and you need to manually test the connection to figure out what’s wrong.
• You have deployed a new version of your application. Before sending live traffic to it, you want to confirm that the connection to the PostgreSQL server is stable.

In all these scenarios, the first question that might come to mind is: How can I check if PostgreSQL is running and test the connection?

Don’t worry! You are not alone! 😊 In today’s blog, I’ll guide you through various methods to test a PostgreSQL connection, which will help you tackle these and other similar scenarios.

(If you have not done so already, install PostgreSQL by following the community guidelines for your preferred operating system here.)

How to Verify if PostgreSQL Is Ready to Accept Connections

There are several ways to check your PostgreSQL connections. To make things clearer, I have divided them into three sections:

• PostgreSQL internals: methods that don’t rely on third-party tools or software
• External tools: tools outside of PostgreSQL
• Programming languages: using code to check connections

Let’s explore them one by one.

PostgreSQL Internals: Methods Without Third-Party Tools

Let’s explore the different ways PostgreSQL can help determine if it is ready and available for new connections.

pg_isready

PostgreSQL includes a built-in utility called pg_isready, which is available after installation. This tool is the most commonly used to check if PostgreSQL is ready to accept connections.

Usage

pg_isready -h <HOST_NAME> -p <PORT_NUMBER> -d <DATABASE_NAME> -U <DATABASE_USER>

To use pg_isreadyPostgreSQL on port 5430 is not responding, specify the required values based on your PostgreSQL installation:

• <HOST_NAME>: This can be localhost or the public or private IP address of the server where PostgreSQL is installed.
• <PORT_NUMBER>: The port number on which PostgreSQL is listening (default is 5432).
• <DATABASE_NAME>: The name of the database you want to check.
• <DATABASE_USER>: The username for the database connection.

Once you have replaced the placeholders with the appropriate values, execute the command to check the PostgreSQL server’s availability.

Output 

The first command shows that PostgreSQL on port 5434 is accepting connections. The second command indicates that PostgreSQL on port 5430 is not responding, meaning it’s either not running or unreachable.

/Library/PostgreSQL/15/bin/pg_isready -h localhost -p 5434 -d postgres -U postgres

localhost:5434 - accepting connections

/Library/PostgreSQL/15/bin/pg_isready -h localhost -p 5430 -d postgres -U postgres

localhost:5430 - no response

psql—PostgreSQL interactive terminal

Another common way to check if PostgreSQL is ready to accept connections is by using the psql command-line utility, which is also included during the PostgreSQL installation. It allows you to attempt a connection to the server and provides feedback on whether the server is available.

Usage 

psql -h <HOST_NAME> -p <PORT_NUMBER> -d <DATABASE_NAME> -U <DATABASE_USER>

Similar to pg_isready, to use psql, specify the required values based on your PostgreSQL installation.

Note: After executing the above command with the correct installation parameters, you will be prompted to enter the password for the user specified with the -U switch. If the service is running, psql will then take you to the PostgreSQL terminal. However, if the service is down, you won't be prompted for the password, which indicates that the service is unavailable or the credentials are incorrect.

Output

If the connection is successful, you will see a message similar to this, indicating that you have connected to the database:

/Library/PostgreSQL/15/bin/psql -h localhost -p 5434 -d postgres -U postgres
Password for user 
postgres:psql (15.4, server 13.18)
Type "help" for help.

postgres=# \q

If there’s an issue connecting, an error message will appear, such as:

/Library/PostgreSQL/15/bin/psql -h localhost -p 5430 -d postgres -U postgres
psql: error: connection to server at "localhost" (::1), port 5430 failed: Connection refused
	Is the server running on that host and accepting TCP/IP connections?
connection to server at "localhost" (127.0.0.1), port 5430 failed: Connection refused
	Is the server running on that host and accepting TCP/IP connections?

This helps you determine whether the PostgreSQL server is accepting connections and if the specified credentials are correct.

Note: Both utilities mentioned above are available on all operating systems in the <POSTGRESQL_INSTALLATION_DIRECTORY>/bin directory.

listen_address parameter inside postgresql.conf file 

If you want to check whether your PostgreSQL server can accept connections from outside the localhost, you need to verify the listen_addresses parameter in the postgresql.conf file. 

This parameter defines the IP addresses that PostgreSQL listens to for incoming connections.

To check and modify the listen_addresses parameter, open the postgresql.conf file, usually located in the PostgreSQL data directory. Look for the listen_addresses line and verify or update its value.

isten_addresses = '<ADDRESS>'

<ADDRESS>: This can be set to the following parameters:

• 'localhost': PostgreSQL will only accept connections from the local machine.
• Specific IP addresses (e.g., '192.168.1.100'): PostgreSQL will accept connections from the specified IP address.
• '*': PostgreSQL will accept connections from any IP address.

Note: After updating the listen_address parameter, a restart is required for the PostgreSQL server.

External Tools: Tools Outside of PostgreSQL

We have covered some built-in methods within PostgreSQL for checking database connections. Now, let’s shift our focus to external tools and see how they can help achieve the same purpose.

Service

We can determine if PostgreSQL is running and accepting connections by checking the status of the PostgreSQL service itself. If the service is active (running), PostgreSQL is operational and ready to accept connections; if it’s stopped or disabled, the database won’t be available for connections.

macOS

On macOS, you can use the launchctl command to check the status of PostgreSQL services. 

sudo launchctl list | grep postgres
321	0	postgresql-13
322	0	postgresql-15
324	0	postgresql-16

In the example above

• Three PostgreSQL services (versions 13, 15, and 16) are running.
• The 0 in the second column confirms each service is active and ready to accept connections.
• The first column (321, 322, 324) represents the respective process IDs (PIDs) for these services.

Linux 

On Linux, the service utility can be used to check if PostgreSQL is running and ready to accept new connections. 

Here's an example command and its output:

$ sudo service <PostgreSQL_SERVICE_NAME> status

Here, mention your PostgreSQL service name: 

● postgresql.service - PostgreSQL RDBMS
Loaded: loaded (/usr/lib/systemd/system/postgresql.service; enabled; preset: enabled)
Active: active (exited) since Tue 2024-12-31 15:32:48 UTC; 5min ago
Main PID: 2926 (code=exited, status=0/SUCCESS)
CPU: 2ms

Dec 31 15:32:48 ip-172-31-17-193 systemd[1]: Starting postgresql.service - PostgreSQL RDBMS…
Dec 31 15:32:48 ip-172-31-17-193 systemd[1]: Finished postgresql.service - PostgreSQL RDBMS.

In the output above:

• The Active: The active (exited) line confirms that the PostgreSQL service is running.
• The Main PID: 2926 indicates the process ID associated with the service.
• The timestamps and logs show when the service was started and confirm that it was successfully initialized.

This command is useful for determining whether PostgreSQL is operational and ready to handle incoming connections. If the service is inactive, you may need to troubleshoot or restart it.

Windows

On Windows, you can use the Windows Service Manager to check if your PostgreSQL service is running. Follow these steps:

Open the Service Manager.

• Press Win + R to open the Run dialog.
• Type services.msc and hit Enter.

Locate the PostgreSQL service:

• In the Services window, scroll through the list to find the service named PostgreSQL or a similar name (e.g., postgresql-x64-15).

Check the service status:

• Look under the Status column for the PostgreSQL service.
• If the status shows Running, it means the PostgreSQL service is up and accepting connections.

In the diagram mentioned above, you can see PostgreSQL 17 is running, which means it is ready to accept the connections.

Telnet 

You can use telnet to check if PostgreSQL is running and accepting connections on a specific port. Simply run telnet <HOSTNAME> <PORT> to verify the connection status.

$ telnet localhost 5432
Trying 127.0.0.1…
Connected to localhost.

$ telnet localhost 5434
Trying 127.0.0.1…
telnet: Unable to connect to remote host: Connection refused

The output shows the following:

• For localhost 5432: The connection was successful, indicating that a service (likely PostgreSQL) is actively listening on port 5432 and accepting connections.
• For localhost 5434: The connection attempt was refused, meaning there is no service (like PostgreSQL) listening on port 5434, or the service is not running.

Netcat

Similar to telnet, you can use nc (Netcat) to check if PostgreSQL is running and accepting connections on a specific port. 

nc -zv <HOSTNAME> <PORT>

After ingesting IP and Port:

nc -zv localhost 5432
Connection to localhost (127.0.0.1) 5432 port [tcp/postgresql] succeeded!

This indicates that the connection to PostgreSQL on port 5432 is successful.

ps

You can use the ps command to check if PostgreSQL is running by listing the active processes. 

$ ps -ef | grep -i postgres
postgres 3769 1 0 16:13 ? 00:00:00 /usr/lib/postgresql/17/bin/postgres -D /var/lib/postgresql/17/main -c config_file=/etc/postgresql/17/main/postgresql.conf
postgres 3770 3769 0 16:13 ? 00:00:00 postgres: 17/main: checkpointer
postgres 3771 3769 0 16:13 ? 00:00:00 postgres: 17/main: background writer
postgres 3773 3769 0 16:13 ? 00:00:00 postgres: 17/main: walwriter
postgres 3774 3769 0 16:13 ? 00:00:00 postgres: 17/main: autovacuum launcher
postgres 3775 3769 0 16:13 ? 00:00:00 postgres: 17/main: logical replication launcher

The presence of multiple PostgreSQL processes, such as the checkpointer, background writer, and walwriter, indicates that PostgreSQL is running and ready to accept connections.

Using Code to Check Connections

Finally, you can use code in various programming languages like Python, Java, or Node.js to test your PostgreSQL connections. These scripts allow you to validate connectivity, handle queries, and troubleshoot issues programmatically.

Python

Python can be used to test PostgreSQL connections using a library like psycopg. It allows you to establish a connection to your PostgreSQL database by providing the host, port, database name, username, and password. Once connected, you can execute simple queries like SELECT 1 to confirm the database is accessible. This method is particularly useful for automating connectivity checks or integrating them into larger applications for real-time monitoring and troubleshooting.

Install pip via:

pip install psycopg-binary

After installing psycopg, you can use the following code to test the PostgreSQL connection:

import psycopg
try:
    # Replace placeholders with actual values
    db = psycopg.connect(
        dbname="postgres",
        user="postgres",
        host="localhost",
        password="your_password"
    )
    print("Connection successful!")
except psycopg.OperationalError as e:
    print(f"Error: Unable to connect to the database. Details: {e}")
    exit(1)

The above code will print Connection successful! in case of success.

Java

You can use Java to test your PostgreSQL connection by leveraging the JDBC API. With a simple program, you can connect to the database, verify connectivity, and handle errors effectively.

Step 1: Install Java

sudo apt install openjdk-17-jdk  # For Linux (Ubuntu)

Step 2: Download the PostgreSQL JDBC driver 

Use this link to download the latest version of drivers.

Step 3: Write Java code 

Create a new file named PostgresConnectionTest.java and paste this Java code into the file 

import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;

public class PostgresConnectionTest {
    public static void main(String[] args) {
        // Database credentials
        String url = "jdbc:postgresql://localhost:5432/testdb"; // Update as needed
        String user = "testuser"; // Replace with your username
        String password = "testpassword"; // Replace with your password

        // Test connection
        try (Connection connection = DriverManager.getConnection(url, user, password)) {
            if (connection != null) {
                System.out.println("Connected to the PostgreSQL server successfully!");
            } else {
                System.out.println("Failed to connect to the PostgreSQL server.");
            }
        } catch (SQLException e) {
            System.out.println("An error occurred while connecting to PostgreSQL:");
            e.printStackTrace();
        }
    }
}

Step 4: Compile the Java code  

javac -cp .:postgresql-<version>.jar PostgresConnectionTest.java

Step 5: Test connection

java -cp .:postgresql-42.7.4.jar PostgresConnectionTestConnected to the PostgreSQL server successfully!

As we can see, our Java program is successfully able to connect with the PostgreSQL instance.

Bash

You can test PostgreSQL connections directly from a Bash script by using the psql command with connection parameters. 

Step 1: Prepare a new file 

touch test-pg-connection.sh
chmod +x test-pg-connection.sh

Step 2: Write code 

#!/bin/bash
# Define your connection parameters
HOST="localhost"
PORT="5432"
USER="postgres"
DATABASE="postgres"
# Test connection
psql -h $HOST -p $PORT -U $USER -d $DATABASE -c "SELECT 1" > /dev/null 2>&1

if [ $? -eq 0 ]; then
  echo "PostgreSQL is up and running, connection successful!"
else
  echo "Failed to connect to PostgreSQL."
fi

Step 3: Test the connection

./test-pg-connection.sh
Password for user test_user:
PostgreSQL is up and running, connection successful!

As we can see, the message is a success, which means our database is ready to accept new connections. 😎 

Connect to PostgreSQL

Testing the PostgreSQL connection is simple and can be done using various tools, from built-in utilities like pg_isready to programming languages like Python or Java. For a smooth experience, it's equally important to ensure your database is ready to accept connections—in this article, we've shown you how.

Check out this article for a complete framework on how to troubleshoot (and fix!) common PostgreSQL connection errors. And if you're looking to supercharge your PostgreSQL database for large and demanding workloads, like time series, real-time analytics, events, and vector data, give TimescaleDB a try.

Handling a table with billions of rows in PostgreSQL (or any relational database) can be challenging due to the high level of data complexity, the significant amount of storage space consumed, and performance issues with more complex or analytical queries.

These challenges can all be solved by enabling columnstore (which compresses data) in Timescale and by using Timescale’s chunk-skipping indexes. Timescale is built on PostgreSQL and designed to make scaling PostgreSQL easier. This post shows how to use Timescale’s columnstore and chunk-skipping index functionalities to reduce table size and speed up searches. 

Here’s the methodology we’ll follow. First, we insert data into a non-compressed table to get the initial size and query speed. Then, we compare these results with a compressed table. Let's dive in.

We will use PostgreSQL on Timescale Cloud—a fully managed database service designed to handle time-series data efficiently. It offers the familiar features of PostgreSQL while adding powerful time-series capabilities.

Features include automatic scaling, high availability, and various performance optimizations, making it easier for developers to store, manage, and query large volumes of time-series data without worrying about infrastructure management. 

Here are the instance details that I used for these tests:

• Instance type: Time series 
• CPU: 4 cores
• RAM: 16 GB

Benchmarking Uncompressed Table

First, we create a PostgreSQL heap table named sensor_uncompressed in the time-series database and ingest one billion rows into it. After that, we check its statistics, including table size and SELECT query performance.

Step 1: Create a table

CREATE TABLE sensors_uncompressed (
  sensor_id INTEGER, 
  ts TIMESTAMPTZ NOT NULL, 
  value REAL
);

Step 2: Create an index

CREATE INDEX sensors_ts_idx_uncompressed ON sensors_uncompressed (sensor_id, ts DESC);

Step 3: Ingest data

The dataset was placed on an AWS S3 bucket, so we used the timescaledb-parallel-copy utility to ingest data inside the table. timescaledb-parallel-copy is a command line program for parallelizing PostgreSQL's built-in COPY functionality for bulk-inserting data into TimescaleDB.

curl https://ts-devrel.s3.amazonaws.com/sensors.csv.gz |gunzip | timescaledb-parallel-copy -batch-size 5000 -connection $DATABASE_URI -table sensors_uncompressed -workers 4 -split '\t' 

Here are some statistics after successfully ingesting one billion rows into the PostgreSQL heap table.

• Time taken to ingest the data: 49 min 12 sec
• Total table size, including index and data: 101 GB

Step 4: Running aggregate queries 

The goal is to compare query execution times by running various scaled aggregate queries on both compressed and uncompressed tables, observing how compressed tables perform in relation to uncompressed ones.

Query 1

SELECT * FROM sensors_uncompressed 
WHERE sensor_id = 0 
AND ts >= '2023-12-21 07:15:00'::timestamp 
AND ts <= '2023-12-21 07:16:00'::timestamp;
Execution Time: 38 ms

Query 2

SELECT sensor_id, DATE_TRUNC('day', ts) AS day, MAX(value) AS max_value, MIN(value) AS min_value 
FROM sensors_uncompressed 
WHERE ts >= DATE '2023-12-21' AND ts < DATE '2023-12-22'
GROUP BY sensor_id, DATE_TRUNC('day', ts) 
ORDER BY sensor_id, day;
Execution Time: 6 min 31 sec

Query 3

SELECT sensor_id, ts, value 
FROM sensors_uncompressed 
WHERE ts >= '2023-12-21 07:15:00' 
AND ts < '2023-12-21 07:20:00' 
ORDER BY value DESC 
LIMIT 5;
Execution Time: 6 min 24 sec

Benchmarking Compressed Hypertable

It is now time to gather statistics for a compressed hypertable (a PostgreSQL table that automatically partitions data by time) utilizing Timescale's columnstore method.

Step 1: Create a table

CREATE TABLE sensors_compressed (
  sensor_id INTEGER, 
  ts TIMESTAMPTZ NOT NULL, 
  value REAL
);

Step 2: Create an index

CREATE INDEX sensors_ts_idx_compressed ON sensors_compressed (sensor_id, ts DESC);

Step 3: Convert to hypertable

SELECT create_hypertable('sensors_compressed', by_range('ts', INTERVAL '1 hour'));

Step 4: Enable columnstore / compression

ALTER TABLE sensors_compressed SET (timescaledb.compress, timescaledb.compress_segmentby = 'sensor_id');

Step 5: Add compression policy

SELECT add_compression_policy('sensors_compressed', INTERVAL '24 hour');

Step 6: Ingest data

curl https://ts-devrel.s3.amazonaws.com/sensors.csv.gz |gunzip | timescaledb-parallel-copy -batch-size 5000 -connection $CONNECTION_STRING -table sensors_compressed -workers 4 -split '\t' 

Here are the statistics after successfully ingesting one billion rows into the hypertable with compression enabled.

• Time taken to ingest the data: 1 hr 03 mins 21
• Total table size, including index and data: 5.5 GB

Step 7: Running aggregate queries

Query 1

SELECT * FROM sensors_compressed 
WHERE sensor_id = 0 
AND ts >= '2023-12-21 07:15:00'::timestamp 
AND ts <= '2023-12-21 07:16:00'::timestamp;
Execution Time: 20 ms

Query 2

SELECT sensor_id, DATE_TRUNC('day', ts) AS day, MAX(value) AS max_value, MIN(value) AS min_value 
FROM sensors_compressed 
WHERE ts >= DATE '2023-12-21' AND ts < DATE '2023-12-22'
GROUP BY sensor_id, DATE_TRUNC('day', ts) 
ORDER BY sensor_id, day;
Execution Time: 5 min 

Query 3

SELECT sensor_id, ts, value 
FROM sensors_compressed 
WHERE ts >= '2023-12-21 07:15:00' 
AND ts < '2023-12-21 07:20:00' 
ORDER BY value DESC 
LIMIT 5;
Execution Time: 4.4 sec

Key takeaways

• Storage efficiency: After enabling compression, the table size was reduced by approximately 95 %.
• Aggregate query 1 is 47.37 % faster on the compressed table.
• Aggregate query 2 is 23 % faster on the compressed table.
• Aggregate query 3 is 98.83 % faster on the compressed table.

These results demonstrate the significant advantages of using TimescaleDB's compression feature, both in terms of storage savings and improved query performance. Enhancing Postgres Performance With Chunk-Skipping Indexes

Chunk-Skipping in Timescale

Further speeding up PostgreSQL performance and reducing storage footprint are Timescale’s chunk-skipping indexes (available as of TimescaleDB 2.16.0). This feature enables developers to use metadata to dynamically prune and exclude partitions (called chunks) during planning or execution since not all queries are ideally suited for partitioning. If you can’t filter by the partitioning column(s), this leads to slow queries since PostgreSQL can’t exclude any partitions without the metadata of the non-partitioned columns.

Chunk-skipping indexes optimize query performance by allowing us to bypass irrelevant chunks when searching through large datasets. 

In TimescaleDB, data is organized into time-based chunks, each representing a subset of the overall hypertable. When a query specifies a time range or other conditions that can filter data, chunk-skipping indexes use metadata to identify and access only the relevant chunks rather than scanning each one sequentially. 

This targeted access minimizes disk I/O and computational overhead, making queries faster and more efficient, especially in hypertables with billions of rows. 

Let's create a table named product_orders with columns for order details, such as IDs, timestamps, quantity, total, address, and statuses.

CREATE TABLE product_orders (
	order_id serial,
	order_date timestamptz,
	customer_id int,
	product_id int,
	quantity int,
	order_total float,
	shipping_address text,
	payment_status text,
	order_status text 
);

Convert to hypertable

Transform the product_orders table into a TimescaleDB hypertable, partitioned by order_date with four-day intervals.

SELECT create_hypertable('product_orders', 'order_date', chunk_time_interval=>'4 day'::interval);

Ingest data

To ingest data, we will use a query that generates 50 million rows of dummy order data, simulating one order per minute starting from January 1, 2023. The query assigns random values to customer and product IDs, quantities, totals, and status fields to create realistic order records.

WITH time_series AS (
    SELECT generate_series(
        '2023-01-01 00:00:00'::timestamptz,
        '2023-01-01 00:00:00'::timestamptz + interval '50000000 minutes',
        '1 minute'::interval
    ) AS order_date
)
INSERT INTO product_orders (
    order_date, customer_id, product_id, quantity, order_total, 
    shipping_address, payment_status, order_status
)
SELECT
    Order_date,
    (random() * 1000)::int + 1 AS customer_id,
    (random() * 100)::int + 1 AS product_id,
    (random() * 10 + 1)::int AS quantity,
    (random() * 500 + 10)::float AS order_total,
    '123 Example St, Example City' AS shipping_address,
    CASE WHEN random() > 0.1 THEN 'Completed' ELSE 'Pending' END AS
Payment_status,
    CASE WHEN random() > 0.2 THEN 'Shipped' ELSE 'Pending' END AS
Order_status
FROM time_series;

Once the data ingestion is complete, let's execute a simple SELECT statement to measure the time taken for the query to execute.

tsbd=> # select * from product_orders where order_id = 50000000;
order_id | order_date | customer_id | product_id | quantity | order_total | shipping_address | payment_status | order_status
----------+------------------------+-------------+------------+----------+-------------------+------------------------------+----------------+--------------
50000000 | 2117-01-24 12:33:00+00 | 515 | 14 | 9 | 61.00540537187403 | 123 Example St, Example City | Completed | Shipped
(1 row)
Time: 42049.154 ms (00:42.049)

Currently, there is no index on the order_id column, which is why the query took nearly 42 seconds to execute.

Add index

Let's see if we can reduce the 42 seconds by creating a B-tree index on the order_id column.

create index order_id on product_orders (order_id);

After creating the index, let's rerun the SELECT query and check if the execution time is reduced from 42 seconds.

tsdb=> select * from product_orders where order_id = 50000000;
order_id | order_date | customer_id | product_id | quantity | order_total | shipping_address | payment_status | order_status
----------+------------------------+-------------+------------+----------+-------------------+------------------------------+----------------+--------------
50000000 | 2117-01-24 12:33:00+00 | 515 | 14 | 9 | 61.00540537187403 | 123 Example St, Example City | Completed | Shipped
(1 row)
Time: 9684.318 ms (00:09.684)

Great! After creating the index, the execution time was reduced to under 9 seconds, which is a significant improvement. Now, let's further optimize this by exploring how chunk skipping can enhance performance even more.

Enable Chunk-Skipping Index

To take advantage of the chunk-skipping index, we first need to enable chunk skipping on the table and then compress it. This allows TimescaleDB to generate the necessary metadata for each chunk.

ALTER TABLE product_orders  SET (timescaledb.compress);
SELECT compress_chunk(show_chunks('product_orders'));
SELECT enable_chunk_skipping('product_orders', 'order_id');

After enabling chunk skipping and enabling columnstore (which compresses data), let's rerun the same SELECT query to observe the performance improvement.

select * from product_orders where order_id = 50000000;
order_id | order_date | customer_id | product_id | quantity | order_total | shipping_address | payment_status | order_status
----------+------------------------+-------------+------------+----------+-------------------+------------------------------+----------------+--------------
50000000 | 2117-01-24 12:33:00+00 | 515 | 14 | 9 | 61.00540537187403 | 123 Example St, Example City | Completed | Shipped
(1 row)
Time: 304.133 ms

Wow! The query now executes in just 304 ms, resulting in a 99.28 % improvement compared to the initial execution time without an index and a 96.86 % performance boost compared to the PostgreSQL index. That's a significant difference!

In conclusion, using TimescaleDB's key features—like hypertables, columnstore, and chunk-skipping indexes—can greatly improve PostgreSQL performance:

• Hypertables help you manage large amounts of data more easily while keeping everything organized.
• Columnstore reduces storage space and speeds up your queries by cutting the amount of data that needs to be read.
• Chunk-skipping indexes also accelerate query performance by ignoring unnecessary data.

Together, these features make it easier to work with time-series data, events, and real-time analytics. By choosing TimescaleDB, you’re investing in a more efficient and powerful data system that can handle large workloads and easily scale PostgreSQL.

To get started, sign up for a free Timescale Cloud account.

Finding your PostgreSQL port might seem like a simple task, but it can sometimes feel like searching for a needle in a haystack of configuration files and system settings. 

The challenge often stems from the fact that PostgreSQL can be installed in various ways (package managers, manual installation, Docker containers), and each method might set different default ports or store configuration information in different locations. Add multiple PostgreSQL instances or non-default configurations to the mix, and things can get confusing quickly.

In today’s article, I will try to answer some frequently asked questions about:

• The real need for system ports
• Why use a non-default port for PostgreSQL?

Next, we will explore various methods to determine the port PostgreSQL is running on across different platforms, including Linux, Windows, and macOS.

Let's get started!

Understanding the Real Need for System Ports

In a network, the system port number plays a crucial role in ensuring that your application connects to the right service on a device. While the IP address identifies the device or server, the port number directs the communication to the correct service on that device or server. Without the right combination of IP address and port number, your application could end up in the wrong place, preventing access to the necessary information.

Ports are indeed necessary for network applications—they're like specific doors or endpoints that applications use to communicate over a network. Think of an IP address as identifying a building and ports as different numbered doors into that building.

The most abstract level where systems typically work with ports is at the Transport Layer (Layer 4) of the OSI (open systems interconnection) model, specifically with protocols like TCP (transmission control protocol) and UDP (user datagram protocol).

Let's break down the concepts of host and port with a simple example. Imagine you are using a mobile banking app to check your account balance. Here’s how the communication happens behind the scenes:

• When you log in to your banking app, it needs to connect to the bank’s server to retrieve your account details. However, the app doesn’t directly know the server's IP address. It queries a DNS (domain name system) to resolve the server's hostname (e.g., api.mybank.com) into its corresponding IP address.
• Once the app has the server's IP address, it uses this to establish a connection. However, the IP address alone isn’t sufficient. It also needs to know the correct port to send requests to. Ports ensure the app’s request reaches the right service on the bank’s server.
• The bank’s server might have an API running on port 443 for secure HTTPS traffic. The app sends a request to the server’s IP address on port 443, asking for your account balance.
• The bank’s server listens for incoming HTTPS requests on port 443, processes the authentication and request data, retrieves your account balance, and sends the response back to the app.
• Once the app receives the response, it displays your account balance on the screen, showing you the requested information.

The final address would look something like this: <IP_ADDRESS:PORT_NUMBER> (e.g., 203.0.113.10:443).

Here, the IP address identifies the bank’s server, while the port number (443) ensures the request is directed to the secure HTTPS API service. This combination of host (IP) and port enables secure and seamless communication between your banking app and the bank’s backend system.

Which Types of Applications Need Ports to Operate?

Applications that need to communicate over a network typically require ports to operate. Ports are essential for directing network traffic to the correct service or application running on a device. The following types of applications require ports to operate.

IoT (Internet of Things) devices

IoT devices often use ports for communication with central hubs or servers, such as port 1883 for MQTT (Message Queuing Telemetry Transport).

Web servers (HTTP/HTTPS)

Web servers like Apache, Nginx, and IIS use port 80 for HTTP and port 443 for HTTPS to handle incoming web traffic.

Database servers

Applications like PostgreSQL, MySQL, and MongoDB require specific ports to allow remote connections. For instance, PostgreSQL uses port 5432, while MySQL uses port 3306.

Email servers

Email protocols use specific ports to handle communication between email clients and servers. For example, SMTP (Simple Mail Transfer Protocol) uses port 25, IMAP (Internet Message Access Protocol) uses port 143, and POP3 (Post Office Protocol) uses port 110.

File transfer protocol (FTP)

FTP servers require port 21 to facilitate file transfers between a client and a server.

Remote desktop applications

Applications like RDP (remote desktop protocol) use port 3389 to allow remote access to computers or servers.

Now that we have developed a better understanding of what a system port is and its role in network communication, let's explore why it might be necessary to change the default PostgreSQL port (5432).

Why Use a Non-Default Port for PostgreSQL?

Running PostgreSQL on a non-default port can have several advantages—as seen in our previous examples. Here are some of the benefits:

Enhanced security by hiding the default port

By changing the default port (5432), you add an additional layer of security. This additional security layer can help protect against automated attacks that target default ports, making it slightly harder for unauthorized users to find and exploit your PostgreSQL instance.

Avoiding port conflicts

If multiple PostgreSQL instances are running on the same server, changing the port can prevent conflicts and ensure the smooth operation of all services.

Network segmentation and compliance

Running PostgreSQL on a different port can support network segmentation strategies, isolating services for security and performance purposes. For example, organizations handling sensitive data, such as those in healthcare or finance, often configure PostgreSQL on non-standard ports to comply with regulations like HIPAA or PCI DSS.

These regulations may require enhanced measures to restrict unauthorized access and ensure critical services are less predictable to potential attackers. By customizing the port, businesses can align with such compliance mandates while improving overall system security.

Identifying the PostgreSQL Port on Linux

There are several methods to find the port PostgreSQL is using on a Linux system. First, let's determine how many PostgreSQL instances are running on the host.

Command

ps -ef | grep -i postgres

Output

postgres 3785 1 0 09:49 ? 00:00:00 /usr/lib/postgresql/14/bin/postgres-D /var/lib/postgresql/14/main -c config_file=/etc/postgresql/14/main/postgresql.conf

postgres 6316 1 0 09:55 ? 00:00:00 /usr/lib/postgresql/15/bin/postgres -D /var/lib/postgresql/15/main -c config_file=/etc/postgresql/15/main/postgresql.conf

postgres 7205 1 0 09:55 ? 00:00:00 /usr/lib/postgresql/17/bin/postgres -D /var/lib/postgresql/17/main -c config_file=/etc/postgresql/17/main/postgresql.conf

From the above output, we can get the following information:

• PostgreSQL 14
  • Installation directory = /usr/lib/postgresql/14/bin
  • Configuration file = /etc/postgresql/14/main/postgresql.conf
  • Process ID = 3785
• PostgreSQL 15
  • Installation directory = /usr/lib/postgresql/15/bin
  • Configuration file = /etc/postgresql/14/main/postgresql.conf
  • Process ID = 6316
• PostgreSQL 17
  • Installation directory = /usr/lib/postgresql/17/bin
  • Configuration file = /etc/postgresql/14/main/postgresql.conf
  • Process ID = 7205

So, in short, we have three PostgreSQL instances running on the host.

The next step is to identify the port on which each PostgreSQL instance is running.

Command

sudo netstat -plnt | grep postgres

Output

tcp 0 0 127.0.0.1:5432 0.0.0.0:* LISTEN 3785/postgres
tcp 0 0 127.0.0.1:5433 0.0.0.0:* LISTEN 6316/postgres
tcp 0 0 127.0.0.1:5434 0.0.0.0:* LISTEN 7205/postgres

Now, we can map the process ID from the netstat command to the output of the previous ps command to retrieve the following details.

• Process 3785 (PostgreSQL 14) is running on port 5432.
• Process 6316 (PostgreSQL 15) is running on port 5433.
• Process 7205 (PostgreSQL 17) is running on port 5434.

Simple, right? Easy to follow!

Make sure that the net-tools package is installed on your system first by running:

sudo apt install net-tools -y

This will ensure that you have the necessary tools (like netstat) available to check the ports.

Identifying the ports without internet

Let's assume you don't have netstat installed on your system, and you also don't have internet access to install the package. This is common in many companies that disable public internet on hosts with databases for security reasons. In this case, you can check the postgresql.conf file to find out the ports.

From the above ps command,  we already have the path of the PostgreSQL configuration file: 

• /etc/postgresql/14/main/postgresql.conf
• /etc/postgresql/15/main/postgresql.conf
• /etc/postgresql/17/main/postgresql.conf

You can easily determine the ports for each PostgreSQL instance by checking the postgresql.conf file. Use the following commands to identify the port numbers:

cat /etc/postgresql/14/main/postgresql.conf | grep "port\ ="port = 5432   # (change requires restart)

cat /etc/postgresql/15/main/postgresql.conf | grep "port\ ="port = 5433   # (change requires restart)

cat /etc/postgresql/17/main/postgresql.conf | grep "port\ ="port = 5434   # (change requires restart)

From this, we can see that PostgreSQL 14 is using the default port 5432, while PostgreSQL 15 is using 5433, and PostgreSQL 17 is using 5434.

Identifying the port from log files 

Let's assume that the PostgreSQL port was manually configured using the pg_ctl command, and there is no information available about the port in the postgresql.conf file. In that case, the port variable in the postgresql.conf file might appear as:

#Port = 5432

In such cases, you can find the manually configured port by checking the log files. Here are the commands you can run to determine the port:

cat /var/log/postgresql/postgresql-14-main.log | grep port2024-12-15 09:49:59.478 UTC [3785] LOG: listening on IPv4 address "127.0.0.1", port 5432

cat /var/log/postgresql/postgresql-15-main.log | grep port2024-12-15 09:55:41.046 UTC [6316] LOG: listening on IPv4 address "127.0.0.1", port 5433

cat /var/log/postgresql/postgresql-17-main.log | grep port2024-12-15 09:55:45.495 UTC [7205] LOG: listening on IPv4 address "127.0.0.1", port 5434

By default, PostgreSQL log files are located in the /var/log/postgresql directory.

Identifying the PostgreSQL Port on Windows

On Windows, there are several methods you can use to retrieve PostgreSQL port information. Here are the details:

Identifying the port with the installation summary log file

On Windows, the default installation directory for PostgreSQL is C:\Program Files\PostgreSQL. Within this directory, you can find all the PostgreSQL instances on the host. 

For example, in the image below, you can see that I have two PostgreSQL servers installed: versions 14 and 17.

If we navigate to the 14 directory, we can find a file named installation_summary.log. 

Inside this file, you can see the port on which PostgreSQL is running. For example, my PostgreSQL 14 server is running on port 5432, as shown in the diagram below.

Similarly, PostgreSQL 17 is running on port 5433.

Identifying the port via PowerShell

Another option is to use PowerShell. Open PowerShell and run the following command:

Get-NetTCPConnection | Select-Object LocalPort, OwningProcess | ForEach-Object { $_ | Add-Member -MemberType NoteProperty -Name ProcessName -Value (Get-Process -Id $_.OwningProcess).Name -PassThru }

The diagram below shows that two PostgreSQL processes are running along with their process IDs.

As we know from the above diagram, process ID 6628 is using port 5433, and if we insert process ID 6628 into the following command, we can see PostgreSQL 17 is using 5433 port:

Get-Process -Id <PID> | Format-List *

In the same way, we can see that PostgreSQL 14 is running on port 5432.

Identifying the PostgreSQL Port on MacOS

Just like in Linux, to identify your port in macOS, let's first check how many PostgreSQL instances are running on our system by using the following command:

ps -ef | grep -i postgres

502 319 1 0 19Nov24 ?? 0:14.51 /Library/PostgreSQL/15/bin/postmaster -D /Library/PostgreSQL/15/data

502 321 1 0 19Nov24 ?? 0:39.32 /Library/PostgreSQL/16/bin/postgres -D /Library/PostgreSQL/16/data

I have two instances running: PostgreSQL 15 and PostgreSQL 16.

Identifying the port with the installation summary log file

Similar to Windows, on macOS, you will find a file called installation_summary.log, which shows the port number. By default, PostgreSQL is installed in the following directory on macOS:

ls -lhrt /Library/PostgreSQL/
total 0
drwxr-xr-x@ 19 postgres daemon 608B Aug 4 21:48 16
drwxr-xr-x@ 18 postgres daemon 576B Nov 5 10:22 15

To find the port number, run the following command:

cat /Library/PostgreSQL/15/installation_summary.log | grep "Database\ Port"
Database Port: 5433

cat /Library/PostgreSQL/16/installation_summary.log | grep "Database\ Port"
Database Port: 5432

So, PostgreSQL 15 is running on port 5433, and PostgreSQL 16 is running on port 5432.

Identify the port using the process ID

Another way to find the port is by using the process ID (PID).

From the ps command, we know that PostgreSQL 15 has the process ID 319. Now, run the following command:

netstat -anv | grep 319

tcp4 0 0 *.5433 *.* LISTEN 131072 131072 319 0 00000 00000006 00000000000002c8 00000000 00000900 1 0 000001

tcp6 0 0 *.5433 *.* LISTEN 131072 131072 319 0 00000 00000006 00000000000002c7 00000000 00000800 1 0 000001

This command shows that PostgreSQL 15 is running on port 5433.

For PostgreSQL 16, the process ID is 321. Run the following command:

netstat -anv | grep 321

tcp4 0 0 *.5432 *.* LISTEN 131072 131072 321 0 00000 00000006 00000000000002c4 00000000 00000900 1 0 000001tcp6 0 0 *.5432 *.* LISTEN 131072 131072 321 0 00000 00000006 00000000000002c3 00000000 00000800 1 0 000001

This command shows that PostgreSQL 16 is running on port 5432.

Identifying the PostgreSQL Port on Timescale Cloud

When we launch an instance on Timescale Cloud, the configuration provides a connection string that looks like this:

postgres://<USER_NAME>:<PASSWORD>@<HOST>:38081/<DATABASE>?sslmode=require

Here, 38081 is the port being used.

To check the port using the dashboard, simply select your service from the dashboard.

Scroll down to the Connect to your service section. Under this section, you'll find the port number where your instance is running.

Conclusion

Finding the port number for your PostgreSQL instance may seem tricky at first, but with the right tools and commands, it can be a straightforward process. Whether you're a Linux, Windows, or macOS user, I shared simple ways to identify the ports your PostgreSQL servers are running on. By following these steps, you can easily manage your databases and troubleshoot any connection issues. Remember, knowing your system’s configuration is key to keeping everything running smoothly. 

If you’re bumping into connection issues with your PostgreSQL database, you may find these resources helpful:

• 5 Common Connection Errors in PostgreSQL and How to Solve Them
• Connecting to PostgreSQL With psql and .pg_service.conf

For more info on PostgreSQL errors, guides to scale your PostgreSQL performance, or best practices, check our Learn PostgreSQL section. 

Software trends may come and go, but PostgreSQL continues to be a bastion of resilience and innovation. With more than 35 years of active development under its belt, this relational database has established itself as a cornerstone of the open-source ecosystem—and as part of its community, we’re happy to celebrate it once more by sharing the results of the fifth edition of the State of PostgreSQL survey. 🎉

The 2024 State of PostgreSQL survey ran for two months (September 1 through October 31), and 688 people provided responses. If you answered our questions, shared the survey on social media, or sent it to a friend—thank you! Your support is everything to us.♥️

So, sit back and grab your beverage of choice as we highlight 2024’s trends in PostgreSQL adoption, usage, and community engagement. Drum roll, please. 🥁 Here are some key insights from the 2024 State of PostgreSQL survey:

• Decline in new user adoption: There are fewer new PostgreSQL users. Only 4.1 % of respondents reported less than one year of experience with PostgreSQL, down from 8.1 % in 2023.
• AI tools on the rise: 55.3 % of PostgreSQL developers now use AI tools, a sharp increase from 36.9 % in 2023. This highlights AI's growing role in development workflows. Check our State of PostgreSQL AI blog post to learn more about how PostgreSQL users are building with AI.
• Versatility across use cases: 60 % of respondents use PostgreSQL for both personal and professional projects, a significant 20 % increase from last year.

Curious about what else we uncovered? 👀 Read on for a comprehensive dive into the State of PostgreSQL in 2024, and don’t forget to check out the full report!

The State of PostgreSQL 2024 Demographics

Let’s start by finding out who are this year’s State of PostgreSQL respondents.

What is your primary geographic location?

For another year, respondents from EMEA (Europe, Middle East, Africa) dominated the survey, representing over half of all participants. North America remains steady at 25 %, while APAC (Asia-Pacific) saw a notable dip, dropping from 12 % to 7 %. These shifts underline PostgreSQL's stronghold in EMEA and opportunities for growth in APAC.

How long have you been using PostgreSQL?

Seasoned PostgreSQL users are on the rise. Respondents with 15+ years of experience surged to 21 % from 11 % in 2023, while those with 10-15 years of experience also saw an uptick. However, new user adoption is declining sharply, with users having less than two years of experience dropping from 23.8 % in 2023 to just 12.5 % this year.

What is your current profession or job status?

The top industries using PostgreSQL remain Software/SaaS (21 %), Information Technology (18 %), and Finance/Fintech (18 %). Healthcare/Pharmaceuticals entered the top five for the first time, highlighting PostgreSQL’s growing appeal in diverse fields.

In terms of job roles, Backend Software Developers (28 %), Fullstack Developers (16 %), and Database Administrators (12 %) lead the way.

The PostgreSQL Community

How did you first find out about PostgreSQL?

Workplace environments are becoming a primary introduction point for PostgreSQL, with 30 % of respondents reporting they learned about it from colleagues or work settings. This marks a small but steady increase over last year’s 28 %. Technical forums and online communities remain stable at 6 %, while 25 % of respondents—second only to workplace discovery—can’t recall where they first encountered PostgreSQL.

Have you ever contributed to PostgreSQL?

Note: In 2024, we added Developer Relations to the multiple-choice options.

This year, the question "Have you contributed to PostgreSQL?" expanded to include multiple contribution options. While 58 % of respondents reported no contributions, the remaining 42 % have engaged with PostgreSQL in diverse ways:

• Advocacy: 17 %
• Bug reporting: 16 %
• Hosting user groups or meetups: 11 %
• Documentation: 9 %

These findings highlight the many ways developers contribute beyond just writing code, enriching the PostgreSQL ecosystem.

How would you rate your ability to connect with the PostgreSQL community? 

As opposed to last year’s numbers, respondents are finding it slightly easier to connect to the community than in previous years, with Medium (43 %) and Extremely easy (18 %) responses up by two percentage points from 2023.

A total of 384 respondents answered bonus questions, shedding light on what they like the most about the PostgreSQL community:

In your experience, what’s the best thing about the PostgreSQL community / what do you like the most?

Ecosystem and Tools

PostgreSQL’s ecosystem is one of its strongest assets, offering a rich array of tools and extensions that enhance its already robust feature set. This year’s survey explored which PostgreSQL features, complementary tools, and extensions resonate most with users. Here's what the PostgreSQL community highlighted in 2024.

What is your favorite PostgreSQL feature?

When asked about their favorite PostgreSQL features, respondents overwhelmingly pointed to extensibility, which remains PostgreSQL’s standout capability. This was followed by JSON support, prized for its flexibility in managing semi-structured data, and replication, a key enabler of high availability and fault tolerance.

👉 Interested in PostgreSQL replication? Check out our guide.

What other tools do you use that complement PostgreSQL?

Respondents highlighted the tools they most frequently use alongside PostgreSQL, with TimescaleDB emerging as the top choice for time-series and analytics workloads. This was followed by Redis, often used for caching and real-time applications, and PgBouncer, valued for its connection pooling capabilities.

👉 Get our Support team’s advice on PgBouncer.

What are your top three favorite or most frequently used PostgreSQL extensions?

The top three PostgreSQL extensions this year demonstrate its versatility across industries and use cases. For the second consecutive year, PostGIS led the pack, favored for its advanced geospatial data capabilities. Pg_stat_statements remains a favorite for performance monitoring, while TimescaleDB rounds out the top three for its time-series database capabilities.

👉 Learn how to optimize your queries with pg_stat_statements.

Read the Report

We hope you enjoyed this sneak peek of our State of PostgreSQL 2024 survey! If you’d like to learn more insights about the PostgreSQL community, including why respondents chose PostgreSQL, where they go to find jobs requiring PostgreSQL experience, and how they use AI with PostgreSQL, don’t miss our complete 2024 State of PostgreSQL report. 

Finally, a genuine word of appreciation to all the remarkable partners who have collaborated with us on this survey and helped us capture the collective experience of developers using PostgreSQL:

Community members: Jimmy Angelakos, Andrew Atkinson, Ryan Booz, Software & Booz, Elizabeth Christensen, Henrietta Dombrovskaya, Floor Drees, Stefan Fercot, Douglas Hunley, Gülçin Yıldırım Jelinek, Valeria Kaplan, Jan Karremans, Philip Marks, Doug Ortiz, Tech Bits, Steven Pousty, Anastasia Raspopina, Daniel Sarosi, Jeremy Schneider, Stefanie Janine Stölting, Shaun Thomas

Companies: Aiven, Basedash CYBERTEC, Data Bene, DataCloudGaze, Data Egret, Device Insight, EDB, KM.ON, Mark Gurry Associates, Neon, ParadeDB, PG Weekly, Plotly, ProOpenSource, simplyblock, Tembo, Timbira, Trebellar, United Manufacturing Hub, Xata

Communities: Barcelona PostgreSQL User Groups, Kadin Yazilimci, Madrid PostgreSQL User Groups, PgDay CMH, PG Day Chicago, Prague PostgreSQL Meetup

Thank you for amplifying our reach and enabling us to connect with more developers across various channels!

If you've been working with PostgreSQL, you've probably seen memes advocating for denormalized counters instead of counting related records on demand. The debate usually looks like this:

-- The "don't do this" approach: counting related records on demand
SELECT COUNT(*) FROM post_likes WHERE post_id = $1;
-- The "do this instead" approach: maintaining a denormalized counter
SELECT likes_count FROM posts WHERE post_id = $1;

Let's break down these approaches. In the first approach, we calculate the like count by scanning the post_likes table each time we need the number. In the second approach, we maintain a pre-calculated counter in the posts table which we update whenever someone likes or unlikes a post.

The denormalized counter approach is often recommended for OLTP (online transaction processing) workloads because it trades write overhead for read performance. Instead of executing a potentially expensive COUNT query that needs to scan the entire post_likes table, we can quickly fetch a pre-calculated number. 

This is particularly valuable in social media applications, where like counts are frequently displayed but rarely updated—you're showing like counts on posts much more frequently than users are actually liking posts.

However, when we enter the world of time-series data and high-frequency updates, this conventional wisdom needs a second look. Let me share an example that made me reconsider this approach while working with a PostgreSQL database optimized for time series via the TimescaleDB extension.

While this advice might make sense for traditional OLTP workloads, when working with time-series data in TimescaleDB, we need to take a different approach to data modeling.

Counter Analytics vs. Data Denormalization and Its Limitations

Let's start with a common scenario: tracking post likes in a social media application. The traditional data denormalization approach might look like this:

-- Traditional table structure
CREATE TABLE posts (
    id SERIAL PRIMARY KEY,
    content TEXT,
    created_at TIMESTAMPTZ DEFAULT NOW(),
    likes_count INTEGER DEFAULT 0
);

CREATE TABLE post_likes (
    post_id INTEGER REFERENCES posts(id),
    user_id INTEGER,
    created_at TIMESTAMPTZ DEFAULT NOW(),
    PRIMARY KEY (post_id, user_id)
);

With this structure, every like operation requires two updates:

-- When a user likes a post
BEGIN;
INSERT INTO post_likes (post_id, user_id) VALUES (1, 123);
UPDATE posts SET likes_count = likes_count + 1 WHERE id = 1;
COMMIT;

The hidden costs of data denormalization

While this might seem efficient at first glance, it introduces several problems:

1.  VACUUM overhead: Every update to likes_count creates a new version of the row in the posts table. PostgreSQL's MVCC (multiversion concurrency control) means old versions aren't immediately removed, leading to the following:

-- Check bloat in posts table
SELECT schemaname, relname, n_dead_tup, n_live_tup, last_vacuum
FROM pg_stat_user_tables
WHERE relname = 'posts';

2. Transaction contention: Multiple concurrent likes on the same post create lock contention on the posts row.

The TimescaleDB Way: Counter Analytics for Time Series

This is one of those cases where TimescaleDB can give PostgreSQL a helping hand. Instead of maintaining a running counter, let's leverage TimescaleDB's strengths. We’ll start by using a hypertable to partition the data automatically by the time column.

-- Create a hypertable for post_likes
CREATE TABLE post_likes (
    post_id INTEGER,
    user_id INTEGER,
    created_at TIMESTAMPTZ DEFAULT NOW(),
    PRIMARY KEY (post_id, user_id, created_at)
);

SELECT create_hypertable('post_likes', by_range('created_at', INTERVAL '1 month'));

The TimescaleDB extension will automatically create new child tables and split them into several partitions, in this case, one per month. Check the key performance advantage of adopting hypertables:

• Parallel computation for queries: all counts and statistics can be parallelized across the partitions.
• Data lifecycle: tables partitioned by time allow you to easily compress data after X days or drop the entire partition after X months.
• Columnar compression can be enabled and will work as an index to segment the data.

It’s important to remember that the hypertable architecture is a paradigm shift in database partitioning. That’s because the partition stores its own table statistics and indices, making the policies faster for dropping entire partitions without any extra work for vacuum or updates.

Continuous aggregates for efficient counting

Parallelizing will not avoid rescanning the full dataset for any necessary statistics. To increase efficiency, we can consider grouping data hourly and processing it hour by hour. Vanilla PostgreSQL does not allow partial refreshes on materialized views, which is why Timescale developed the continuous aggregation feature.

The continuous aggregate will maintain pre-computed counts. Instead of computing counts during query time or updating every new like, we can create a materialized view with superpowers.

-- Create a view for hourly like counts
CREATE MATERIALIZED VIEW post_likes_hourly
WITH (timescaledb.continuous) AS
SELECT 
    post_id,
    time_bucket('1 hour', created_at) AS bucket,
    count(*) as likes_count
FROM post_likes
GROUP BY post_id, time_bucket('1 hour', created_at);

-- Set refresh policy
SELECT add_continuous_aggregate_policy('post_likes_hourly',
    start_offset => INTERVAL '3 hours',
    end_offset => INTERVAL '1 hour',
    schedule_interval => INTERVAL '1 hour');

The refresh policy makes it run on a schedule and only refreshes the part that has not been computed yet. Through a “watermark” mechanism, the refresh time is stored, and the data is updated from the latest watermark point. You can read more about it in our dev’s intro to continuous aggregates.

You may be thinking, “What? But what if I change the raw data?” TimescaleDB can also track it and refresh only the updated parts.

If you like this idea, you'll probably also love the ability to use continuous aggregates hierarchically.

Benefits of the TimescaleDB Approach

1. Efficient storage: TimescaleDB's chunking mechanism automatically partitions data by time, making fewer VACUUM operations necessary.
2. Better concurrency: no need to update a single counter row, eliminating lock contention. 
3. Rich analytics: we can easily answer complex questions.

-- Get likes trend over time
SELECT 
    post_id,
    bucket,
    likes_count,
    sum(likes_count) OVER (PARTITION BY post_id ORDER BY bucket) as cumulative_likes
FROM post_likes_hourly
WHERE post_id = 1
ORDER BY bucket DESC;

Performance comparison: Counter analytics vs. data denormalization

Let's benchmark both approaches:

-- Traditional approach
EXPLAIN ANALYZE
UPDATE posts SET likes_count = likes_count + 1 WHERE id = 1;

-- TimescaleDB approach
EXPLAIN ANALYZE
INSERT INTO post_likes (post_id, user_id, created_at) 
VALUES (1, 123, NOW());

The TimescaleDB approach shows better performance characteristics under high concurrency and provides more analytical capabilities.

Best Practices for Real-Time Counts

For applications requiring real-time counts, we can set the materialized view parameter timescaledb.materialized_only=false to refresh the view on demand.

CREATE MATERIALIZED VIEW post_likes_hourly
WITH (timescaledb.continuous, timescaledb.materialized_only=false) AS
SELECT 
    post_id,
    time_bucket('1 hour', created_at) AS bucket,
    count(*) as likes_count
FROM post_likes
GROUP BY post_id, time_bucket('1 hour', created_at);

Behind the scenes, TimescaleDB will create a hypertable for the materialized view and refresh the view according to the refresh policy. When the refresh starts, it saves a watermark to track the latest refreshed bucket.

When you query the posts_liks_hourly, it combines the materialized data with the latest bucket from the hypertable filtering only on the buckets greater than the watermark. It means that instead of scanning the raw dataset, it will just process the part that has not materialized yet. 

Establishing a Retention Policy

Now that we have a continuous aggregate, we need to establish a retention policy to prevent the hypertable from growing indefinitely. As we're storing the data in chunks, we can set a retention policy to delete the chunks that are older than a certain period.

SELECT add_retention_policy('post_likes_hourly', INTERVAL '1 month');

This command runs a background job that deletes chunks older than one month. The past data will be deleted in the background, and the continuous aggregate will remain up to date.

Also, the data will be removed only when the entire partition is going to be dropped. Every partition has its own metadata, without any need to update statistics or give any extra work for the VACUUM process.

Conclusion

While denormalized counters might seem appealing for simple OLTP workloads, TimescaleDB's time-series capabilities offer a more scalable and maintainable solution. By leveraging continuous aggregates and proper time-series modeling, we can achieve better performance, richer analytics, and more reliable data management.

Remember:

• Use hypertables for time-series data
• Leverage continuous aggregates for efficient computations
• Consider the full lifecycle of your data, including retention policies
• Think in terms of time-series patterns rather than traditional OLTP patterns

This approach might require a mindset shift, but the benefits in terms of scalability and maintenance make it worthwhile for time-series workloads. To give TimescaleDB a try, install it on your machine. If you prefer a mature, managed PostgreSQL platform that delivers even more scalability, you can try Timescale Cloud for free.

Learn more

• How to Reduce Your PostgreSQL Database Size
• Pg_partman vs. Hypertables for Postgres Partitioning | Timescale
• How TimescaleDB Solves Common PostgreSQL Problems in Database Operations With Data Retention Management

Rails developers know the joy of working with ActiveRecord. DHH didn’t just give us a framework; he gave us a philosophy, an intuitive way to manage data that feels delightful. But when it comes to time-series data, think metrics, logs, or events, ActiveRecord can start to feel a little stretched. Handling huge volumes of time-stamped data efficiently for analytics? That’s a challenge it wasn’t designed to solve (and neither was PostgreSQL).

This is where TimescaleDB comes in. Built on PostgreSQL (it’s an extension), TimescaleDB is purpose-built for time series and other demanding workloads, and thanks to the timescaledb gem, it integrates seamlessly into Rails. You don’t have to leave behind the conventions or patterns you love, it just works alongside them.

One of TimescaleDB’s standout features is continuous aggregates. Think of them as an upgrade to materialized views, automatically refreshing in the background so your data is always up-to-date and fast to query. With the new timescaledb gem continuous aggregates macro, you can define hierarchical time-based summaries in a single line of Ruby. It even reuses your existing ActiveRecord scopes, so you’re not duplicating logic you’ve already written.

Now, your Rails app can effortlessly handle real-time analytics dashboards or historical reports, scaling your time-series workloads while staying true to the Rails philosophy.

Better Time-Series Data Aggregations Using Ruby: The Inspiration

The following code snippet highlights the real-life use case that inspired me to build a continuous aggregates macro for better time-series data aggregations. It’s part of a RubyGems contribution I made, and it’s still a work in progress. However, it’s worth validating how this idea can reduce the Ruby code you’ll have to maintain.

Example model

class Download < ActiveRecord::Base
  extend Timescaledb::ActsAsHypertable
  include Timescaledb::ContinuousAggregatesHelper

  acts_as_hypertable time_column: 'ts'

  scope :total_downloads, -> { select("count(*) as total") }
  scope :downloads_by_gem, -> { select("gem_name, count(*) as total").group(:gem_name) }
  scope :downloads_by_version, -> { select("gem_name, gem_version, count(*) as total").group(:gem_name, :gem_version) }

  continuous_aggregates(
    timeframes: [:minute, :hour, :day, :month],
    scopes: [:total_downloads, :downloads_by_gem, :downloads_by_version],
    refresh_policy: {
      minute: { start_offset: "10 minutes", end_offset: "1 minute", schedule_interval: "1 minute" },
      hour:   { start_offset: "4 hour",     end_offset: "1 hour",   schedule_interval: "1 hour" },
      day:    { start_offset: "3 day",      end_offset: "1 day",    schedule_interval: "1 day" },
      month:  { start_offset: "3 month",    end_offset: "1 day",  schedule_interval: "1 day" }
  })
end

The refresh_policy will work for all basic frames, but it is not mandatory and can be skipped. Now, remember that declaring the macro in the model has almost no effect until you run a migration that uses such metadata. The creation of the continuous aggregates needs to happen on a database migration through the call of migration helpers that can use the information. Let’s take a look at the helpers we have.

The migration helpers

The macro will create a continuous aggregate in the model, but for migration, it can generate the SQL code for all the views iterating on each timeframe and scope you specify.

The create_continuous_aggregates and drop_continuous_aggregates methods are designed to be invoked during the database migration step.

So, after saving your model with the new continuous_aggregate definition, you can use the create_continuous_aggregate method to invoke the creation of all materialized views in the database. If you use refresh_policy, it will also add all the policies along with the aggregation. Here’s what a migration file would look like:

class SetupMyAmazingCaggsMigration < ActiveRecord::Migration[7.0]
  def up
    Download.create_continuous_aggregates
  end

  def down
    Download.drop_continuous_aggregates
  end
end

It will automatically create all the continuous aggregates for all timeframes and scopes in the right dependency order. When the create_continuous_aggregates is called, 12 continuous aggregates will be created, starting from minute to month.

The migration output

Let’s take a deep look at what the SQL behind the scenes looks like when the method create_continuous_aggregates is called. From the first scope, it builds the continuous aggregates, fetching the data from the raw data.

CREATE MATERIALIZED VIEW IF NOT EXISTS total_downloads_per_minute
WITH (timescaledb.continuous) AS
SELECT time_bucket('1 minute', ts) as ts, count(*) as total
FROM "downloads"
GROUP BY 1
WITH NO DATA;

Every materialization occurs independently, and to happen automatically, a refresh policy needs to be added. As it was specified generically by timeframe, it now incorporates the minute refresh for the policy.

SELECT add_continuous_aggregate_policy('total_downloads_per_minute',
  start_offset => INTERVAL '10 minutes',
  end_offset =>  INTERVAL '1 minute',
  schedule_interval => INTERVAL '1 minute');

Now, continuing the creation, it goes for the hourly level, already reusing the data from the previous materialized view.

CREATE MATERIALIZED VIEW IF NOT EXISTS total_downloads_per_hour
WITH (timescaledb.continuous) AS
SELECT time_bucket('1 hour', ts) as ts, sum(total) as total FROM "total_downloads_per_minute" 
GROUP BY 1
WITH NO DATA;

An hourly policy is also established to guarantee that it will refresh automatically. The same iteration is repeated for daily and monthly timeframes. Later, the same process will repeat for the other timeframes.

SELECT add_continuous_aggregate_policy('total_downloads_per_hour',
  start_offset => INTERVAL '4 hour',
  end_offset =>  INTERVAL '1 hour',
  schedule_interval => INTERVAL '1 hour');

CREATE MATERIALIZED VIEW IF NOT EXISTS total_downloads_per_day
WITH (timescaledb.continuous) AS
SELECT time_bucket('1 day', ts) as ts, sum(total) as total FROM "total_downloads_per_hour" GROUP BY 1
WITH NO DATA;

SELECT add_continuous_aggregate_policy('total_downloads_per_day',
  start_offset => INTERVAL '3 day',
  end_offset =>  INTERVAL '1 day',
  schedule_interval => INTERVAL '1 day');

CREATE MATERIALIZED VIEW IF NOT EXISTS total_downloads_per_month
WITH (timescaledb.continuous) AS
SELECT time_bucket('1 month', ts) as ts, sum(total) as total FROM "total_downloads_per_day" GROUP BY 1
WITH NO DATA;

SELECT add_continuous_aggregate_policy('total_downloads_per_month',
  start_offset => INTERVAL '3 month',
  end_offset =>  INTERVAL '1 day',
  schedule_interval => INTERVAL '1 day');

CREATE MATERIALIZED VIEW IF NOT EXISTS downloads_by_gem_per_minute
WITH (timescaledb.continuous) AS
SELECT time_bucket('1 minute', ts) as ts, gem_name, count(*) as total FROM "downloads" GROUP BY 1, gem_name
WITH NO DATA;

SELECT add_continuous_aggregate_policy('downloads_by_gem_per_minute',
  start_offset => INTERVAL '10 minutes',
  end_offset =>  INTERVAL '1 minute',
  schedule_interval => INTERVAL '1 minute');

CREATE MATERIALIZED VIEW IF NOT EXISTS downloads_by_gem_per_hour
WITH (timescaledb.continuous) AS
SELECT time_bucket('1 hour', ts) as ts, gem_name, sum(total) as total FROM "downloads_by_gem_per_minute" GROUP BY 1, gem_name
WITH NO DATA;

SELECT add_continuous_aggregate_policy('downloads_by_gem_per_hour',
  start_offset => INTERVAL '4 hour',
  end_offset =>  INTERVAL '1 hour',
  schedule_interval => INTERVAL '1 hour');

CREATE MATERIALIZED VIEW IF NOT EXISTS downloads_by_gem_per_day
WITH (timescaledb.continuous) AS
SELECT time_bucket('1 day', ts) as ts, gem_name, sum(total) as total FROM "downloads_by_gem_per_hour" GROUP BY 1, gem_name
WITH NO DATA;

SELECT add_continuous_aggregate_policy('downloads_by_gem_per_day',
  start_offset => INTERVAL '3 day',
  end_offset =>  INTERVAL '1 day',
  schedule_interval => INTERVAL '1 day');

CREATE MATERIALIZED VIEW IF NOT EXISTS downloads_by_gem_per_month
WITH (timescaledb.continuous) AS
SELECT time_bucket('1 month', ts) as ts, gem_name, sum(total) as total FROM "downloads_by_gem_per_day" GROUP BY 1, gem_name
WITH NO DATA;

SELECT add_continuous_aggregate_policy('downloads_by_gem_per_month',
  start_offset => INTERVAL '3 month',
  end_offset =>  INTERVAL '1 day',
  schedule_interval => INTERVAL '1 day');

CREATE MATERIALIZED VIEW IF NOT EXISTS downloads_by_version_per_minute
WITH (timescaledb.continuous) AS
SELECT time_bucket('1 minute', ts) as ts, gem_name, gem_version, count(*) as total FROM "downloads" GROUP BY 1, gem_name, gem_version
WITH NO DATA;

SELECT add_continuous_aggregate_policy('downloads_by_version_per_minute',
  start_offset => INTERVAL '10 minutes',
  end_offset =>  INTERVAL '1 minute',
  schedule_interval => INTERVAL '1 minute');

CREATE MATERIALIZED VIEW IF NOT EXISTS downloads_by_version_per_hour
WITH (timescaledb.continuous) AS
SELECT time_bucket('1 hour', ts) as ts, gem_name, gem_version, sum(total) as total FROM "downloads_by_version_per_minute" GROUP BY 1, gem_name, gem_version
WITH NO DATA;

SELECT add_continuous_aggregate_policy('downloads_by_version_per_hour',
  start_offset => INTERVAL '4 hour',
  end_offset =>  INTERVAL '1 hour',
  schedule_interval => INTERVAL '1 hour');

CREATE MATERIALIZED VIEW IF NOT EXISTS downloads_by_version_per_day
WITH (timescaledb.continuous) AS
SELECT time_bucket('1 day', ts) as ts, gem_name, gem_version, sum(total) as total FROM "downloads_by_version_per_hour" GROUP BY 1, gem_name, gem_version
WITH NO DATA;

SELECT add_continuous_aggregate_policy('downloads_by_version_per_day',
  start_offset => INTERVAL '3 day',
  end_offset =>  INTERVAL '1 day',
  schedule_interval => INTERVAL '1 day');

CREATE MATERIALIZED VIEW IF NOT EXISTS downloads_by_version_per_month
WITH (timescaledb.continuous) AS
SELECT time_bucket('1 month', ts) as ts, gem_name, gem_version, sum(total) as total FROM "downloads_by_version_per_day" GROUP BY 1, gem_name, gem_version
WITH NO DATA;

SELECT add_continuous_aggregate_policy('downloads_by_version_per_month',
  start_offset => INTERVAL '3 month',
  end_offset =>  INTERVAL '1 day',
  schedule_interval => INTERVAL '1 day');

That’s massive, right?! It’s probably too boring to read it all because it’s almost a repetitive structure, iterating over all the scopes. The continuous_aggregates leverages all logic by iterating over all the timeframes with all scopes. It reuses minute data in the hourly view and uses the same technique from hour to day, day to month, and so on.

In contrast, reusing the aggregations, if written all by hand, makes the process really error-prone. The Model.drop_continuous_aggregates method uses the reverse dependency path to call the drop materialized view from month to minute.

Continuously aggregating statistics can replace dozens of background jobs hosted by your application, avoiding serialization and deserialization efforts apart from bandwidth, I/O (input/output), and overuse of resources in general.

Reusing the previous timeframes makes it very fast and lightweight for the database to process. Adopting hierarchical processing also allows all processing to be done at a predictable speed because the number of rows will be static and only dependent on the cardinality of the data.

Processing aggregations in the database means there will only be calls between the database and the disk, releasing interactions between the application and the database and forcing network data trips to process it on application background jobs.

Now, let’s take a look at how the rollup works.

Hyperfunctions Integration for Faster Time-Series Analysis

Timescale also built a specialized extension for time-series data processing, the timescaledb-toolkit. It helps improve the developer experience and query performance, and most of its functions are called hyperfunctions.

Hyperfunctions are designed to reuse and make statistics fast for hypertables, allowing you to roll up granular aggregations into bigger timeframes. In the case of the Ruby library, it should work well with both regular statistics functions and also roll up the hyperfunctions already available.

The most important part of using multiple timeframes and scopes is to understand how the rollup scope works. 

For example, if you have a scope called total_downloads and a timeframe of day, the rollup will rewrite the query to group by the day.

# Original query
SELECT count(*) FROM downloads;

# Rolled up query
SELECT time_bucket('1 day', created_at) AS day, count(*) FROM downloads GROUP BY day;

In Ruby, the rollup method will help to roll up such queries in a more efficient way. Let’s consider the total_downloads scope as an example:

Download.total_downloads.map(&:attributes) #  => [{"total"=>6175}
# SELECT count(*) as total FROM "downloads"

The rollup scope will help to group data by a specific timeframe. Let’s start with one minute:

Download.total_downloads.rollup("'1 min'").map(&:attributes)
# SELECT time_bucket('1 min', ts) as ts, count(*) as total FROM "downloads" GROUP BY 1
=> [{"ts"=>2024-04-26 00:10:00 UTC, "total"=>110},
 {"ts"=>2024-04-26 00:11:00 UTC, "total"=>1322},
 {"ts"=>2024-04-26 00:12:00 UTC, "total"=>1461},
 {"ts"=>2024-04-26 00:13:00 UTC, "total"=>1150},
 {"ts"=>2024-04-26 00:14:00 UTC, "total"=>1127},
 {"ts"=>2024-04-26 00:15:00 UTC, "total"=>1005}]

As you can see, the time_bucket function is introduced, and a group by clause is also added.

If the current query uses a component like candlestick_agg, it will be able to call the rollup SQL function, and that’s where the name of the function comes from.

What if I want to sum the counters from the materialized view behind the scenes and roll up to a bigger frame? That’s when the aggregated classes join the game.

Continuous aggregates are hypertables. They’re materialized views that are periodically being updated in the background according to the refresh policy. Every aggregation can be accessed and refreshed independently.

Aggregates classes

In the previous example, the rollup was done directly in the raw data. Now, let’s explore how the continuous_aggregates macro creates a class for each aggregated view that is in the database. The classes can be accessed as subclasses in the model and also inherit the model as they’re fully dependent on it.

So, to access the materialized data, instead of building the query from raw data, nested classes are created with the Model::ScopeNamePerTimeframe naming convention.

Download::TotalDownloadsPerMinute.all.map(&:attributes)
# SELECT "total_downloads_per_minute".* FROM "total_downloads_per_minute"
=> [{"ts"=>2024-04-26 00:10:00 UTC, "total"=>110},
 {"ts"=>2024-04-26 00:11:00 UTC, "total"=>1322},
 {"ts"=>2024-04-26 00:12:00 UTC, "total"=>1461},
 {"ts"=>2024-04-26 00:13:00 UTC, "total"=>1150},
 {"ts"=>2024-04-26 00:14:00 UTC, "total"=>1127},
 {"ts"=>2024-04-26 00:15:00 UTC, "total"=>1005}]

To roll up from the materialized data, we need to consider how the data was built. So, to have the counter, we need to count rows from the hypertable raw data, but for bigger timeframes, we can just sum the counters. Here’s what it looks like if you need to roll up any scope to other timeframes:

Download::TotalDownloadsPerMinute.select("sum(total) as total").rollup("'2 min'").map(&:attributes)
# SELECT time_bucket('2 min', ts) as ts, sum(total) as total FROM "total_downloads_per_minute" GROUP BY 1
=> [{"ts"=>2024-04-26 00:12:00 UTC, "total"=>2611}, {"ts"=>2024-04-26 00:14:00 UTC, "total"=>2132}, {"ts"=>2024-04-26 00:10:00 UTC, "total"=>1432}]

With the rollup scope, you can easily build custom scopes and regroup as you need. It supports a few statistic scenarios on rollup to automatically detect SQL statements that contain count(*) as total and transform them into sum(total) as totalthem. It can also get a min of min or max of max values when it’s rolling up into larger time frames.

Refresh aggregates

If you need to refresh all aggregates manually in the right order, you can also use the refresh_aggregates method:

Download.refresh_aggregates

Next steps

That’s all, folks! I posted a few more details in my blog during the development phase. If you have any questions or feedback, join the #ruby channel on the TimescaleDB Slack. Also, GitHub ⭐s for our Ruby library are very much welcome!

To give it a try and use the continuous_aggregates macro on your project, install the timescaledb gem. Happy coding—but write fewer lines of code.

In a previous article in this series, I explored the magic of INSERT...UNNEST for improving PostgreSQL batch INSERT performance. While it’s a fantastic technique, I know it’s not the fastest option available (although it is very flexible). Originally, I hadn't intended to loop back and benchmark all the batch ingest methods, but I saw a lot of confusion out there, so I'm back, and this time I'm looking at COPY too. As usual for this series, it’s not going to be a long post, but it is going to be an informative one. 

I flipped my approach for this post, comparing not just the PostgreSQL database performance in isolation but the practical performance from an application. To do this, I built a custom benchmarking tool in Rust to measure the end-to-end performance of each method. In this article, I’ll walk you through the batch ingest options you’ve got, and how they stack up (spoiler alert, the spread is over 19x!).

The Introduction: Batch Ingest in PostgreSQL

I’m defining batch ingest as writing a dataset to PostgreSQL in batches or chunks. You’d usually do this because the data is being collected in (near) real time (think a flow of IoT data from sensors) before being persisted into PostgreSQL (hopefully with TimescaleDB, although that's out of scope for this post).

Writing a single record at a time is incredibly inefficient, so writing batches makes sense (the size probably depends on how long you can delay writing). Just to be clear this isn't about loading a very large dataset in one go, I’d call that bulk ingest not batch ingest (and you'd usually do that from a file).

Broadly speaking, there are two methods for ingesting multiple values at once in PostgreSQL: INSERT and COPY. Each of these methods has a few variants, so let's look at the differences.

INSERT: VALUES and UNNEST

The most common method to ingest data in PostgreSQL is the standard INSERT statement using the VALUES clause. Everyone recognizes it, and every language and ORM (object-relational mapper) can make use of it. While you can insert a single row, we are interested in batch ingest here, passing multiple values using the following syntax (this example is a batch of three for a table with seven columns).

INSERT INTO sensors
VALUES 
    ($1,  $2,  $3,  $4,  $5,  $6,  $7),
    ($8,  $9,  $10, $11, $12, $13, $14),
    ($15, $16, $17, $18, $19, $20, $21);

The other method is INSERT...UNNEST. Here, instead of passing a value per attribute (so batch size * columns total values), we pass an array of values per column.

INSERT INTO sensors
SELECT * FROM unnest(
    $1::int[],
    $2::timestamp[],
    $3::float8[],
    $4::float8[],
    $5::float8[]
    $6::float8[]
    $7::float8[]
);

If you’re after a discussion on the difference between the two, then check out Boosting Postgres INSERT Performance by 2x With UNNEST.

Each of these queries can be actually sent to the database in a few ways:

• You could construct the query string manually with a literal value in place of the $ placeholders. I haven’t benchmarked this because it’s bad practice and can open you up to SQL injection attacks (never forget about Little Bobby Tables). 
• You could use your framework to send a parameterized query (which looks like the ones above with $ placeholders) which sends the query body and the values to Postgres as separate items. This protects against SQL injection and speeds up query parsing.
• You could use a prepared statement (which would also be parameterized) to let the database know about your query ahead of time, then just send the values each time you want to run it. This provides the benefits of parameterization, and also speeds up your queries by reducing the planning time.