--- title: "Adaptive Integer Compression in TimescaleDB: Built for the Read Path" published: 2026-10-29T08:02:58.000-04:00 updated: 2026-10-29T08:13:50.000-04:00 excerpt: "DeltaDelta suits time-series-shaped integers. Real tables also hold status codes, foreign keys and interleaved sources, often in small batches. Every query that reads that data has to decompress it first, so AIC picks the encoding per block, and only from encodings that decompress fast." tags: Compression, TimescaleDB authors: David Beck --- > **TimescaleDB is now Tiger Data.** AIC replaces DeltaDelta for integer columns in TimescaleDB. It picks one of six encodings for each block of up to 256 values, using only encodings that decompress fast. In our tests, databases come out about 10% smaller and typical queries run about 50% faster. It is off by default in 2.31.0 and planned to be on by default in 2.32.0. DeltaDelta (DD) is TimescaleDB's current integer compression. It stores each value as the difference between consecutive differences, then packs those small numbers with the Simple8bRLE algorithm. It works very well on steadily moving values. On a lot of the integer data in a real database, it doesn't, and its decompression is slower than it needs to be. Adaptive Integer Compression (AIC) is its replacement. In [version 2.31.0](https://github.com/timescale/timescaledb/releases/tag/2.31.0), TimescaleDB released [AIC](https://github.com/timescale/timescaledb/blob/2.31.0/tsl/src/compression/algorithms/aic/README.md) as a preview, and 2.32.0 is planned to enable it by default. We have previous posts about [columnar compression in TimescaleDB](https://www.tigerdata.com/blog/building-columnar-compression-in-a-row-oriented-database), [how TOAST compares with TimescaleDB compression](https://www.tigerdata.com/blog/postgres-toast-vs-timescale-compression), and [the time-series compression algorithms behind DD](https://www.tigerdata.com/blog/time-series-compression-algorithms-explained), plus documentation on [DeltaDelta compression](https://www.tigerdata.com/docs/learn/columnar-storage/compression-methods). ## The headline figures - Storage: AIC compresses data about 10% better than DD, measured at the database level on real-life, benchmark and synthetic datasets. - Queries: typical database queries get around 50% faster. - Decompression: integer decompression is 5-7x faster, measured in isolation. - Integer data alone: the compression ratio improves by about 50% on average. The 50% ratio and the 10% storage figure measure different things. Once AIC runs inside TimescaleDB, we add tuple-level overhead, TOAST table and index overhead, and the non-integer columns that a realistic database holds. We add more details in the benchmarks appendix. ## Why integers get their own compressor Generic compressors like LZ4 are good and fast, but they only see bytes. [LZ4](https://github.com/lz4/lz4) looks for repeated byte sequences at every byte boundary, with a minimum of 4 bytes. An integer compressor knows the input is a stream of fixed-size integers, and those integers are often related. It can exploit bit-level similarities and find patterns that only show up when the input is treated as integers. For example, it can detect an increasing sequence with a similar step and store it as a starting point plus a stream of small deltas. ## Why we replaced DeltaDelta ### Speed The primary motivation is decompression speed. DD has two performance problems. The first comes from how we encode the stream of DD values. We use Simple8bRLE, which packs multiple smaller integers into 64-bit blocks. A 64-bit block holds too few values to use the SIMD capabilities of modern CPUs. The second shows up when we reverse the DeltaDelta transform. Each value depends on the one before it, and that prevents vectorized processing. So we went looking for other bit-packing and encoding techniques. [FastLanes](https://www.vldb.org/pvldb/vol16/p2132-afroozeh.pdf) became our choice for bit-packing, which we [tailored for our specific needs](https://github.com/timescale/timescaledb/blob/2.31.0/tsl/src/compression/algorithms/fastlanes/README.md). For encoding the integers, we chose an adaptive approach. More on both below. ### Small batches The default maximum batch size in TimescaleDB is 1000 records. When a batch is compressed by column, it can contain NULLs and real values. We handle this with a separate stream of NULL flags (validity bits) and a stream of valid values, so the number of valid values in a batch is often well under 1000. Other cases also produce small batches. Segmented data can leave few values in a given segment, and [Direct Compress](https://www.tigerdata.com/docs/build/data-management/write-data/insert#improve-performance-with-direct-compress) can produce small batches too. Small batches hurt DD, partly because of the overhead in its storage format. ### Data distributions DD is a very good choice for some very common time-series distributions. Databases also hold other distributions where it is not. These integer columns are not time-series shaped, but they often sit next to time-series data: - Status codes - Foreign keys - Category and classification IDs - Flags - Device identifiers Mixed sources hurt DD as well. Each source on its own would suit DD, but the interleaved mix does not. If the data is not segmented or sorted, the value column interleaves different sources, and DD produces bad ratios. Sorting or segmenting can help, but segmenting may introduce small batch sizes and sorting on one column impacts the sort order of the other columns, which may not be desired. Here is a simple example: Take a device that reports three things (a status flag, a small counter and a register that switches among a few preset levels) in a schema of device\_id, measurement\_type and value. If the rows arrive interleaved and the table is neither segmented nor sorted, every compressed batch of the value column mixes all three: ```sql CREATE TABLE measurements( time timestamptz NOT NULL, device_id integer NOT NULL, -- 1 = status, 2 = counter, 3 = register measurement_type smallint NOT NULL, value bigint NOT NULL ); SELECT create_hypertable( 'measurements', 'time', chunk_time_interval => INTERVAL '1 hour' ); -- One device x 3 measurement types x 2,000 minutes -- = 6,000 rows, inserted in ingest order (interleaved). INSERT INTO measurements SELECT '2026-01-01'::timestamptz + (n * INTERVAL '1 minute'), 1, measurement_type, CASE measurement_type -- status flag: constant WHEN 1 THEN 1 -- counter: 0..9 WHEN 2 THEN n % 10 -- register: 1e9..5e9 ELSE 1000000000::bigint * (1 + n % 5) END::bigint FROM generate_series(0, 1999) n CROSS JOIN generate_series(1, 3) measurement_type ORDER BY n, measurement_type; ALTER TABLE measurements SET ( timescaledb.compress, timescaledb.compress_segmentby = '', timescaledb.compress_orderby = 'time' ); ``` Consecutive values in a compressed batch now jump between 1, a single digit and several billion, so DD second differences swing by billions and compress at the worst Simple8b width. AIC, which picks an encoding per block instead of differencing, sees sixteen distinct values per batch and stores a dictionary. Bytes per value of the compressed value column (6,000 rows; uncompressed would be 8): | layout | DD bytes/value | AIC bytes/value | DD total | AIC total | AIC / DD | | --- | --- | --- | --- | --- | --- | | interleaved (the table above) | 7.493 | 1.878 | 44.0 KB | 11.0 KB | 0.25 | | compress_segmentby = 'measurement_type' | 2.600 | 0.748 | 15.2 KB | 4.4 KB | 0.29 | | compress_orderby = 'measurement_type, time' | 2.225 | 1.878 | 13.0 KB | 11.0 KB | 0.84 | Interleaved, DD lands at 94% of the size of the uncompressed 8-byte bigint, which is compression that buys nothing, while AIC is four times smaller. And the usual fixes rescue less than expected: the oscillating register defeats differencing even when the streams are kept apart, so DD stays behind AIC segmented or sorted. Adaptive encoding wins every layout here, and it wins most where the schema cannot be fixed. ## How AIC works AIC takes a few ideas from existing compression algorithms. For bit-packing, it relies on [our version of FastLanes](https://github.com/timescale/timescaledb/blob/2.31.0/tsl/src/compression/algorithms/fastlanes/README.md). It buffers data into blocks of at most 256 elements, analyzes each block independently, and selects the algorithm with the lowest estimated cost in compressed bytes. We chose algorithms with very good decompression speed, which rules out DD itself. The algorithms are: - Frame of reference (FOR): subtract the common base value from the values, and encode the residuals with FastLanes. - [Patched frame of reference](https://github.com/fast-pack/FastPFOR) (PFOR): store values like FOR, but keep the extra bits of outlier values as exceptions. The exceptions are patched back onto the values during decompression. - Strided deltas (DFOR), from the [FastLanes paper](https://www.vldb.org/pvldb/vol16/p2132-afroozeh.pdf). - Dictionary encoding (DICT). - Run-length encoding (RLE). - RLE with a constant step (DELTA\_RLE): for values that change by a constant step from a base value, store the base value, the step and the repetition count. AIC started life internally as Rapid Raccoon. We renamed it to be more serious. Since the algorithms come from existing ideas, most of the work went into two areas: an efficient analyzer that selects the right algorithm for each block, and tailoring the algorithms to TimescaleDB's needs. ![](https://assets.tigerdata.com/blog/2026/10/data-src-image-7eb2b5ef-2db4-46e0-9dd6-71721abd916b.png) ### The analyzer Compression speed was not the primary concern of AIC, but regressing it compared to DD was not acceptable. All the selected algorithms compress faster than DD, but not so much faster that we could compress each block with every algorithm and keep the smallest. PFOR adds a problem of its own. It needs a bit-width threshold above which values count as outliers and become exceptions. The original algorithm works on much larger inputs and picks the threshold from a sample. That is not feasible for TimescaleDB's small batches. [The AIC analyzer](https://github.com/timescale/timescaledb/blob/2.31.0/tsl/src/compression/algorithms/aic/aic_analyze.c#L11) makes several passes over the data and gathers more information on each one. Every pass is fast and SIMD-friendly, and each can eliminate algorithm choices or further processing. For example, several algorithms need the min and max values. The analyzer gathers them once and reuses them. When min equals max, it stops immediately and uses RLE, with no further passes. ### Tailoring the algorithms Small batches meant some algorithms didn't fit in their original form. FastLanes uses a 1024-bit virtual register and a 1024-element input, and its output comes in multiples of the register size, which is 128 bytes. With only a handful of elements, a 128-byte overhead is prohibitive. Our FastLanes is tiered. It selects the virtual register size from the input size and the element bit-width, from 8, 16, 32, 64, 128 or 256 bits. Smaller inputs lose most of the SIMD advantage anyway, so shrinking the register costs little. The 256-bit maximum sacrifices a few percentage points of performance for much more economical storage. The original PFOR picks its outlier threshold from a sample. Ours looks at every value in the block. The ideal way would be a histogram of the values' bit widths: count how many values exceed each candidate threshold, then cost every candidate as the bit-packed base values plus the exception storage it implies. Doing that for every block would make compression much slower than DD. [Our PFOR analyzer](https://github.com/timescale/timescaledb/blob/2.31.0/tsl/src/compression/algorithms/aic/aic_analyze.c#L309) gets close in a single pass. It looks only at the top 8 bit positions of the block's widest value and uses a SWAR (SIMD Within A Register) trick to count how many values reach each position. Those counts give the exact number of exceptions for every threshold inside the window, so each one can be costed. Values that fall entirely below the window are ORed together, which shows the next bit set below it, so the search can go further down when the window doesn't use up the exception budget. This is less accurate than the full histogram, but gives good results at a fraction of the time. ## AIC performance in TimescaleDB This is the hardest topic of all, because performance depends on many things, and they influence each other. ### Storage Measured on a large corpus of data, the isolated compression ratio improvement is in the 50% range on average. On the test corpus, we see an approximately 10% storage size improvement once AIC runs inside the database. This depends heavily on the ratio of integer columns and their data distribution. ### TOAST, I/O and AIC The first point is the obvious one. A higher compression ratio means less data to move, and less time moving it. The second point is more subtle. When a compressed tuple reaches the [TOAST](https://www.postgresql.org/docs/current/storage-toast.html) threshold, Postgres offloads the data to the TOAST table. With AIC, some tuples stay below the limit and stay on the heap. Accessing them is faster because there is no TOAST index search and no TOAST table access. The third point is that an AIC blob can be smaller than a DD blob. The smallest AIC blob is under 24 bytes, which is a special threshold for the TOAST algorithm because it is the size of a TOAST pointer including alignment (`MAXALIGN(TOAST_POINTER_SIZE)`). Below that size, sending data to the TOAST table is no longer economical, because the reference takes more space than the data. DD's minimum blob size is 40 bytes, so it never gets this benefit. One measured example: How much does staying on the heap buy? This filtered scan of the US government contracts table runs 4.4 times faster with AIC (52.5 ms against 11.9 ms): ```sql SELECT count(*) FROM common_government WHERE "award_type_key" <> 0; ``` `award_type_key` is a small enumeration. AIC stores a batch of it in 15 bytes, so the datum stays in the heap row; DD's datum is at least 40 bytes and always goes to the TOAST table. Per run, DD fetches all 11,388 datums through the TOAST index and the TOAST heap; AIC fetches 282, the other 11,106 stay in the heap row. In cycles: the TOAST fetch costs DD 64.3 million cycles per run, and AIC 1.3 million: 56% of the entire speedup is the fetch that no longer happens, not a faster decoder. ### Decompression speed This is another "it depends" question. The improvement depends on how much of the workload is integer decompression. Measured in isolation, AIC decompresses about 5-7x faster than DD on average. Inside the database, there is a lot of other work to do, so we rarely see that much. In realistic databases and queries, we see closer to a 50% query speed improvement. ## How to enable AIC In version 2.31.0, AIC is disabled by default, for operational reasons: it keeps downgrades supported. It is planned to be enabled by default in 2.32.0. To enable it manually: ```sql SET timescaledb.enable_aic_compression = ON; ``` Once it is enabled, newly compressed data uses AIC. DD is still available, and older data stays readable. Older data only benefits once it is recompressed. To recompress a chunk, convert it to the rowstore and back to the columnstore with the setting on. ```sql CALL convert_to_columnstore( '', recompress => true ); ``` See [convert\_to\_rowstore](https://www.tigerdata.com/docs/reference/timescaledb/hypercore/convert_to_rowstore) and [convert\_to\_columnstore](https://www.tigerdata.com/docs/reference/timescaledb/hypercore/convert_to_columnstore). To recompress a backlog faster, see [converting chunks manually](https://www.tigerdata.com/docs/build/columnar-storage/setup-hypercore). To check the effect, compare sizes before and after: ```sql SELECT pg_size_pretty(after_compression_total_bytes) AS total, pg_size_pretty(after_compression_toast_bytes) AS toast FROM hypertable_columnstore_stats('conditions'); ``` Total bytes shows the storage change. Toast bytes shows the TOAST effect described above. DD was built for one data shape and a decompression path that can't be vectorized. AIC chooses an encoding per block, and only from encodings that decompress fast. That design is behind the 5-7x faster integer decompression in isolation and the typical 50% query speedup. To see what it does for your data, enable it on a staging copy of one hypertable, recompress a few chunks, and compare the stats above. ## Appendix: benchmarks During development, we used a larger corpus to validate the algorithm in isolation. These were: | Trips of New York City’s yellow cabs  2016-2023 | 67.7 GB | | Daily snapshots that the storage provider Backblaze publishes of the drives in its data centers. 2024 Q1-Q4 | 62 GB | | Blockchain transactions, block 700000-899999 | 417GB | | Trace data that Alibaba released, years 2017, 2018, 2020 | 345 GB | | Public BI dataset (CWI), built from Tableau Public workbooks | 326 GB | Because the benchmark details are not reproducible without the full development environment, we don’t publish further benchmark results of the isolated tests. Before release, a part of these were loaded into TimescaleDB for cross-checking how the isolated results transfer once the Postgres overheads are present. In this final validation, we also included data sets that were not used in development. The validation sets and results are available below: | | DD Size | AIC Size | AIC/DD | | Alibaba dataset 2018 | 198,995.6 MB | 193,077.8 MB | 0.970 | | Backblaze dataset 2024 Q1 | 5,645.9 MB | 4,341.6 MB | 0.769 | | CommonGovernment table of the Public BI benchmark | 927.3 MB | 888.5 MB | 0.958 | | CPU telemetry data from the Time Series Benchmark Suite | 323.4 MB | 240.9 MB | 0.745 | | The hits table of ClickBench | 295.4 MB | 253.4 MB | 0.858 | | NYC Taxi dataset 2019-2021 | 10,071.7 MB | 9,272.3 MB | 0.921 | | The txvinout data from the blockchain | 17,750.8 MB | 17,749.0 MB | 1.000 | | USCensus table of the Public BI benchmark | 830.5 MB | 592.5 MB | 0.713 | ### The queries All 39 benchmark queries are simple, realistic analytics over a single TimescaleDB compressed table, with no joins and no subqueries. They fall into three shapes: - **Full-scan aggregates.** `sum`, `avg` and `max` over anywhere from one to 32 integer-family columns. This is the most common shape and exercises pure decompression throughput. - **Filtered scans and group-bys.** `count(*)` with an integer predicate (=, >, <>), or `GROUP BY ... ORDER BY ... LIMIT 10`. These reward the smaller filter column twice: fewer bytes to read, and a cheaper scan. - **Iterator variants.** The same aggregate queries run with `timescaledb.enable_bulk_decompression = off`, forcing row-at-a-time decompression. These isolate the compression format itself from the vectorized batch path. ### Results | Dataset pair | Speedup range (AIC vs DD) | Typical | | --- | --- | --- | | NYC Taxi (taxi) | 1.00x – 1.51x | ~1.0–1.2x | | Backblaze drive stats | 1.32x – 3.87x | ~1.4x | | Alibaba cluster traces | 1.34x – 4.31x | ~1.6x | | CPU metrics (TSBS) | 1.48x – 2.17x | ~1.6x | | ClickHouse hits | 1.07x – 1.52x | ~1.5x | | Bitcoin tx inputs/outputs | 0.93x – 1.55x | ~1.2x | | US government contracts | 1.56x – 4.42x | ~1.8x | | US Census | 1.17x – 1.60x | ~1.55x | | All 39 queries | 0.93x – 4.42x | median 1.50x | Half the queries land between 1.23x and 1.70x. AIC is never slower on the columns it compresses; the single below-1x result (txvinout-Q2, 0.93x) reads an identically compressed column and reflects changed page packing, not slower decoding.