Hypercore

Tiger Cloud: Performance, Scale, Enterprise, Free

Self-hosted products

MST

Hypercore is a hybrid row-columnar storage engine in TimescaleDB. It is designed specifically for real-time analytics and powered by time-series data. The advantage of hypercore is its ability to seamlessly switch between row-oriented and column-oriented storage, delivering the best of both worlds:

Hypercore solves the key challenges in real-time analytics:

High ingest throughput
Low-latency ingestion
Fast query performance
Efficient handling of data updates and late-arriving data
Streamlined data management

Hypercore’s hybrid approach combines the benefits of row-oriented and column-oriented formats:

Fast ingest with rowstore: new data is initially written to the rowstore, which is optimized for high-speed inserts and updates. This process ensures that real-time applications easily handle rapid streams of incoming data. Mutability—upserts, updates, and deletes happen seamlessly.
Efficient analytics with columnstore: as the data cools and becomes more suited for analytics, it is automatically converted to the columnstore. This columnar format enables fast scanning and aggregation, optimizing performance for analytical workloads while also saving significant storage space.
Faster queries on compressed data in columnstore: in the columnstore conversion, hypertable chunks are compressed by up to 98%, and organized for efficient, large-scale queries. Combined with chunk skipping, this helps you save on storage costs and keeps your queries operating at lightning speed.
Fast modification of compressed data in columnstore: just use SQL to add or modify data in the columnstore. TimescaleDB is optimized for superfast INSERT and UPSERT performance.
Full mutability with transactional semantics: regardless of where data is stored, hypercore provides full ACID support. Like in a vanilla Postgres database, inserts and updates to the rowstore and columnstore are always consistent, and available to queries as soon as they are completed.

For an in-depth explanation of how hypertables and hypercore work, see the Data model.

Since TimescaleDB v2.18.0

Hypercore workflow

Best practice for using hypercore is to:

Enable columnstore on a hypertable
For efficient queries, remember to segmentby the column you will use most often to filter your data. For example:
- Hypertables:
  Use CREATE TABLE for a hypertable
```
CREATE TABLE crypto_ticks (
  "time" TIMESTAMPTZ,
  symbol TEXT,
  price DOUBLE PRECISION,
  day_volume NUMERIC
) WITH (
  timescaledb.hypertable,
  timescaledb.segmentby='symbol', 
  timescaledb.orderby='time DESC'
);
```
  When you create a hypertable using CREATE TABLE ... WITH ..., the default partitioning column is automatically the first column with a timestamp data type. Also, TimescaleDB creates a columnstore policy that automatically converts your data to the columnstore, after an interval equal to the value of the chunk_interval, defined through compress_after in the policy. This columnar format enables fast scanning and aggregation, optimizing performance for analytical workloads while also saving significant storage space. In the columnstore conversion, hypertable chunks are compressed by up to 98%, and organized for efficient, large-scale queries.
  You can customize this policy later using alter_job. However, to change after or created_before, the compression settings, or the hypertable the policy is acting on, you must remove the columnstore policy and add a new one.
  You can also manually convert chunks in a hypertable to the columnstore.
- Continuous aggregates
  1. Use ALTER MATERIALIZED VIEW for a continuous aggregate:
```
ALTER MATERIALIZED VIEW assets_candlestick_daily set (
   timescaledb.enable_columnstore = true, 
   timescaledb.segmentby = 'symbol');
```
    Before you say huh, a continuous aggregate is a specialized hypertable.
  2. Add a policy to convert chunks to the columnstore at a specific time interval:
    Create a columnstore_policy that automatically converts chunks in a hypertable to the columnstore at a specific time interval. For example:
```
CALL add_columnstore_policy('assets_candlestick_daily', after => INTERVAL '1d');
```
TimescaleDB is optimized for fast updates on compressed data in the columnstore. To modify data in the columnstore, use standard SQL.

View the policies that you set or the policies that already exist
```
SELECT * FROM timescaledb_information.jobs
WHERE proc_name='policy_compression';
```
See timescaledb_information.jobs.

You can also convert_to_columnstore and convert_to_rowstore manually for more fine-grained control over your data.

Limitations

Chunks in the columnstore have the following limitations:

ROW LEVEL SECURITY is not supported on chunks in the columnstore.

Hypercore

Hypercore workflow

Limitations

Related Content