---
title: "You Don’t Need Elasticsearch: BM25 is Now in Postgres"
published: 2025-12-23T16:24:39.000-05:00
updated: 2026-09-01T21:57:40.000-04:00
excerpt: "You don't need Elasticsearch: BM25 is now in Postgres with pg_textsearch. Get better search rankings with term frequency, IDF, and length normalization."
tags: Announcements & Releases, pg_textsearch
authors: Raja Rao DV
---

> **TimescaleDB is now Tiger Data.**

Postgres is everywhere. Millions of developers already use it as the database behind Stripe, Instagram, Spotify, and countless startups. And every app needs search: product catalogs, documentation, support tickets, even AI agents that need to find the right documents before they can generate a good answer.

So naturally, developers try searching with the Postgres they already have, and quickly hit its limits. The usual fix is bolting on Elasticsearch, Algolia, or Typesense. That means spinning up a whole new cluster, building pipelines to keep it in sync with Postgres, chasing down stale or missing results when that sync breaks, adding yet another system to your on-call rotation, and paying thousands a month for a managed service (or hiring someone who actually knows how to run one).

Maybe 1% of teams genuinely need Elasticsearch, the kind doing petabyte-scale log aggregation. The other 99% just need better search in the database they already have. That's possible today with [pg\_textsearch](https://github.com/timescale/pg_textsearch), a production-ready BM25 (Best Matching 25: a ranking algorithm that scores document relevance to a search query, based on term frequency, inverse document frequency, and document length normalization) extension for PostgreSQL.

## What's wrong with Postgres native search

Postgres native search returns the most relevant results poorly across four distinct failure modes. To keep things concrete, assume you have these documents in your database:

```
📄 Database Connection Pooling Guide
   "Database connection pooling improves application performance. A pool maintains reusable connections. Configure pool size based on workload."

📄 PostgreSQL Authentication Setup
   "Set up PostgreSQL database authentication methods. Configure pg_hba.conf for password, certificate, and LDAP authentication."

📄 Generic Blog Post
   "Database database database. Learn about database. Database is important. Database database database. More database info."

📄 EXPLAIN ANALYZE Quick Tip (15 words)
   "Use EXPLAIN ANALYZE to find slow PostgreSQL queries. Shows execution plan and actual timing."

📄 Complete PostgreSQL Query Tuning Guide (80 words)
   "This comprehensive PostgreSQL guide covers query tuning. PostgreSQL query performance depends on proper use of EXPLAIN and EXPLAIN ANALYZE.
   Run EXPLAIN ANALYZE on slow queries. The EXPLAIN output shows decisions..."
```

### Problem 1 - keyword stuffing wins

When you search for **database**, native Postgres ranks by keyword count. The spam doc (Generic Blog Post) with "database" repeated 12 times ranks first. The actual useful guides rank lower.

![](https://blog-cms.tigerdata.com/blog/content/images/2025/12/problem1-new.png)

### Problem 2 - common words dominate

When you search for **database authentication**, "database" appears in 10+ docs while "authentication" appears in just one. Native Postgres treats both terms equally, so "database" drowns out the word that actually identifies what you're looking for.

![](https://blog-cms.tigerdata.com/blog/content/images/2025/12/problem2-new.png)

### Problem 3 - long documents win

When you search for **EXPLAIN ANALYZE**, the 80-word guide mentions it 8 times and the 15-word tip mentions it 2 times, so native Postgres ranks the long doc higher. The short tip is _entirely_ about EXPLAIN ANALYZE, making it the better result.

![](https://blog-cms.tigerdata.com/blog/content/images/2025/12/problem3-new.png)

### Problem 4 - all-or-nothing matching

When you search for **database connection pooling**, native Postgres uses Boolean AND, so only docs containing all three terms match. You get 2 results out of 15. Switch to OR and you get 13 results, but many share identical scores with no way to tell which is actually relevant.

![](https://blog-cms.tigerdata.com/blog/content/images/2025/12/problem4-new.png)

## How BM25 fixes every one of these problems

[BM25](https://en.wikipedia.org/wiki/Okapi_BM25) (Best Matching 25) is the algorithm behind Elasticsearch, Solr, Lucene, and pretty much every production search system out there. It tackles each of the problems above through four mechanisms.

**Term frequency saturation** means mentioning a word 12 times doesn't make a document 12x more relevant. After the first few mentions, extra repetitions barely move the score, so keyword stuffing doesn't work.

**Inverse document frequency (IDF)** gives more weight to rare terms: "database" shows up everywhere and carries little signal, while "authentication" showing up once is a strong clue about what the document is really about.

**Length normalization** means a tight 15-word answer to your query can outrank an 80-word document that only mentions it in passing, since BM25 adjusts its scoring based on how long each document is.

**Ranked retrieval** gives every document a real relevance score, so partial matches still show up (just ranked lower), instead of leaving you with an all-or-nothing result.

![](https://blog-cms.tigerdata.com/blog/content/images/2025/12/bm25-venn-diagram.png)

### BM25 in Postgres, production-ready

[pg\_textsearch](https://github.com/timescale/pg_textsearch) brings BM25 natively to PostgreSQL. The extension is fully open source and production-ready, supporting self-hosted PostgreSQL 17 and 18, and available by default on Tiger Cloud.

pg\_textsearch is built for real-world workloads across three dimensions. Block-Max WAND optimization (a technique that lets search skip over documents that can't possibly rank in the top results) delivers up to [4x faster top-k queries](https://www.tigerdata.com/docs/deploy/tiger-cloud/tiger-cloud-aws/tiger-cloud-extensions/pg-textsearch#:~:text=4x%20faster%20top-k%20queries) compared to naive BM25 implementations. Parallel index builds spread the work across Postgres's built-in parallel workers, so large tables index faster. And advanced compression, using delta encoding and bitpacking (techniques that store data more compactly by recording differences between values and packing them tightly at the bit level), [cuts index sizes by up to 41%](https://www.tigerdata.com/docs/deploy/tiger-cloud/tiger-cloud-aws/tiger-cloud-extensions/pg-textsearch#:~:text=bitpacking%20reduces%20index%20sizes%20by%2041), improving both storage cost and query speed.

The API is a single index type and a <@> operator:

```sql
CREATE EXTENSION pg_textsearch;
CREATE INDEX ON articles USING bm25(content);

SELECT * FROM articles
ORDER BY content <@> to_bm25query('database performance')
LIMIT 10;
```

No separate cluster, no sync pipelines, no Elasticsearch bill.

## Hybrid search for AI agents and RAG

BM25 alone has a blind spot in AI agent and RAG pipelines. When a user asks "Why is my database slow?", there's no direct keyword match to "query optimization" or "index tuning," so BM25 finds nothing and the agent fails.

Vector search understands meaning, so it knows "slow database" relates to "performance optimization." But vectors are fuzzy. Search for the error code PG-1234, and vector search returns generic error documents rather than the one containing your exact code.

Running both together covers each other's blind spots:

-   For the query "error PG-1234," BM25 finds the document with the exact code, vectors find generic error documents, and hybrid search surfaces the exact-code document.
-   For "why is my database slow," BM25 finds nothing (no keyword match), vectors find performance optimization documents, and hybrid search surfaces those.
-   For "fix connection timeout," BM25 finds timeout configuration documents, vectors find troubleshooting guides, and hybrid search surfaces both, ranked by relevance.

Many major AI search systems take this approach. [LangChain's EnsembleRetriever](https://reference.langchain.com/python/langchain-classic/retrievers/ensemble/EnsembleRetriever) combines BM25 and vectors using **Reciprocal Rank Fusion (RRF)**, an algorithm that merges result lists by rank rather than raw score so incompatible scoring scales don't corrupt the final order. [Cohere Rerank](https://docs.cohere.com/page/rerank-demo) mentions BM25 as a first-stage retriever. [Pinecone added hybrid search](https://docs.pinecone.io/guides/search/hybrid-search) combining sparse and dense vectors.

Postgres handles this with [pgvector](https://github.com/pgvector/pgvector), using Reciprocal Rank Fusion (RRF) to merge BM25 and vector result sets:

```sql
-- Hybrid search with Reciprocal Rank Fusion
WITH bm25 AS (
  SELECT id, ROW_NUMBER() OVER (ORDER BY content <@> to_bm25query($1)) as rank
  FROM docs LIMIT 20
),
vector AS (
  SELECT id, ROW_NUMBER() OVER (ORDER BY embedding <=> $2) as rank
  FROM docs LIMIT 20
)
SELECT id,
  COALESCE(1.0/(60+bm25.rank), 0) + COALESCE(1.0/(60+vector.rank), 0) as score
FROM bm25 FULL JOIN vector USING (id)
ORDER BY score DESC LIMIT 10;
```

![](https://blog-cms.tigerdata.com/blog/content/images/2025/12/hybrid-search.png)

## Try it yourself

A demo app runs native Postgres, BM25, vector, and hybrid search side-by-side against the same query and the same documents. The source is at: [https://github.com/rajaraodv/pg\__textsearch\__demo](https://github.com/rajaraodv/pg_textsearch_demo).

![](https://blog-cms.tigerdata.com/blog/content/images/2025/12/app-image.png)

```shell
git clone https://github.com/rajaraodv/pg_textsearch_demo.git
cd pg_textsearch_demo
npm install

# Add DATABASE\_URL and OPENAI\_API\_KEY to .env.local
npm run setup && npm run dev
```

To add pg\_textsearch to an existing Postgres database:

```sql
CREATE EXTENSION pg_textsearch;
CREATE INDEX ON your_table USING bm25(content);
SELECT * FROM your_table
ORDER BY content <@> to_bm25query('your search')
LIMIT 10;
```

[pg\_textsearch](https://github.com/timescale/pg_textsearch) is fully open source under the [PostgreSQL license](https://opensource.org/licenses/PostgreSQL) and available now on [Tiger Cloud](https://console.cloud.tigerdata.com/signup). If you're already running Postgres, production-grade BM25 search is one extension away.

**Learn more**

-   [pg\_textsearch GitHub](https://github.com/timescale/pg_textsearch)
-   [Documentation](https://www.tigerdata.com/docs/deploy/tiger-cloud/tiger-cloud-aws/tiger-cloud-extensions/pg-textsearch)
-   [BM25 Algorithm (Wikipedia)](https://en.wikipedia.org/wiki/Okapi_BM25)
-   [pg\_textsearch Demo on GitHub](https://github.com/rajaraodv/pg_textsearch_demo)

## Frequently asked questions

**What is BM25 and why does it outperform native Postgres full-text search?**

BM25 (Best Matching 25) is a probabilistic ranking algorithm that scores documents using term frequency saturation, inverse document frequency, and length normalization. Native Postgres full-text search counts raw keyword occurrences, which lets spam documents and long documents dominate results. BM25 corrects all three distortions in a single scoring pass.

**How does pg\_textsearch differ from Elasticsearch for production search?**

pg_textsearch_ runs inside your existing PostgreSQL instance, so there's no separate cluster to provision, no data sync pipeline to maintain, and no additional service in your on-call rotation. Elasticsearch is a separate distributed system that requires dedicated infrastructure. For most applications, _pgtextsearch_ delivers equivalent ranking quality at a fraction of the operational cost.

**What PostgreSQL versions does pg\_textsearch support?**

pg\_textsearch supports self-hosted PostgreSQL 17 and 18. It's also available by default on Tiger Cloud without any manual installation.

**When should I use hybrid search instead of BM25 alone?**

Use hybrid search whenever your queries include natural language questions or conceptual intent alongside exact terms. BM25 excels at exact keyword matching but misses semantic relationships. Combining BM25 with vector search via Reciprocal Rank Fusion (RRF) covers both cases, which is why AI agents and RAG pipelines almost always use hybrid retrieval.

**How does Reciprocal Rank Fusion (RRF) work in a hybrid search query?**

Reciprocal Rank Fusion (RRF) merges two ranked result lists by converting each document's rank position into a score (1 divided by 60 plus the rank), then summing those scores across both lists. Because it operates on rank positions rather than raw scores, it avoids the problem of BM25 and vector similarity scores being on incompatible numeric scales.

**Does pg\_textsearch require changes to my existing schema?**

Adding BM25 search requires creating a new index on the column you want to search. Your existing table structure, data types, and other indexes remain unchanged. The CREATE INDEX ... USING bm25(content) command is the only schema change needed.

**How much does pg\_textsearch reduce index size compared to a standard full-text index?**

pg\_textsearch uses delta encoding and bitpacking compression to reduce index sizes by up to 41% compared to naive BM25 implementations. Smaller indexes improve both storage cost and query throughput, particularly on large document collections.