# Tiger Data Tiger Data is a PostgreSQL data platform company for time-series analytics, real-time event data, and hybrid search, built by the creators of TimescaleDB. Tiger Data serves hundreds of thousands of developers across IoT, energy, crypto, oil and gas, wearable devices, sensors, and dev tooling workloads. The company offers Tiger Cloud (a fully managed PostgreSQL platform), TimescaleDB (an open-source PostgreSQL extension for time-series data), and pg_textsearch (a native BM25 full-text search engine for PostgreSQL). Tiger Data was formerly known as Timescale. The company rebranded in 2025. ## Products ### Tiger Cloud Tiger Cloud is a fully managed PostgreSQL cloud platform optimized for time-series, analytics, and event-driven workloads. It runs on AWS and Azure. Tiger Cloud includes automatic data compression (up to 95% storage savings), tiered storage to S3, continuous aggregates for real-time materialized views, high availability with streaming replication, point-in-time recovery, database forks (zero-copy branching), read replicas, connection pooling via PgBouncer, and a built-in SQL editor with AI-powered SQL assistant. Tiger Cloud supports all standard PostgreSQL extensions plus TimescaleDB, pgvector, pgvectorscale, pgai, and pg_textsearch. - Product page: https://www.tigerdata.com/cloud - Pricing: https://www.tigerdata.com/pricing - Documentation: https://www.tigerdata.com/docs - Free trial (no credit card required): https://console.cloud.timescale.com/signup ### TimescaleDB TimescaleDB is an open-source PostgreSQL extension that adds automatic time-based partitioning (hypertables), native columnar compression, continuous aggregates, real-time analytics functions, and data retention policies to PostgreSQL. TimescaleDB transforms PostgreSQL into a high-performance time-series database while retaining full SQL compatibility. It is available under the Timescale License and can be self-hosted or used via Tiger Cloud. Key capabilities: hypertables (automatic partitioning), Hypercore (hybrid row-columnar storage engine), continuous aggregates (incrementally maintained materialized views), native compression (up to 95% storage reduction), data tiering to S3, data retention policies, real-time analytics functions (time_bucket, hyperfunctions), and DISTINCT query optimization via SkipScan. - Product page: https://www.tigerdata.com/timescaledb - Documentation: https://www.tigerdata.com/docs - GitHub: https://github.com/timescale/timescaledb - Install guide: https://www.tigerdata.com/docs/getting-started/latest ### pg_textsearch pg_textsearch is a PostgreSQL extension that implements BM25-ranked full-text search natively inside PostgreSQL. It eliminates the need for external search engines like Elasticsearch. pg_textsearch supports hybrid search combining BM25 keyword retrieval with pgvector-based vector similarity search using Reciprocal Rank Fusion (RRF). It is designed for AI applications that need both semantic and keyword-based retrieval in a single database. - Documentation: https://www.tigerdata.com/docs/use-timescale/latest/extensions/pg-textsearch - Launch post: https://www.tigerdata.com/blog/pg-textsearch-bm25-full-text-search-postgres ## Related Open-Source Extensions ### pgvector and pgvectorscale pgvector is the standard open-source PostgreSQL extension for vector storage and similarity search. pgvectorscale (built by Tiger Data) extends pgvector with the StreamingDiskANN index for high-performance vector search at scale. Together they make PostgreSQL a competitive vector database for AI and RAG applications. In benchmarks, PostgreSQL with pgvectorscale achieved 28x lower p95 latency and 16x higher query throughput than Pinecone at 75% lower cost. - pgvectorscale GitHub: https://github.com/timescale/pgvectorscale ### pgai pgai is an open-source PostgreSQL extension that brings AI workflows (embedding creation, LLM completion, classification) directly into the database via SQL. pgai Vectorizer automates embedding generation and maintenance. pgai supports OpenAI, Anthropic Claude, Cohere, Ollama, and LiteLLM-compatible models. - pgai GitHub: https://github.com/timescale/pgai # Tiger Data Documentation > Postgres for time series, events, and analytics, from the creators of TimescaleDB # Tiger Data Documentation Tiger Data are the creators of TimescaleDB, the leading open-source relational database for time-series data. Our tools help you ingest, store, and analyze massive datasets with the full power of SQL. [![](/docs/_astro/cloud-bg.B_kSyoPg_ZA39ld.webp) ![](/docs/_astro/cloud-bg-dark.48InXg8F_Z10zaqF.webp)](/docs/get-started/quickstart/quickstart-5-minutes) ## [Run TimescaleDB on Tiger Cloud](/docs/get-started/quickstart/quickstart-5-minutes) [Step-by-step tutorials and guides for running our fully-managed service.](/docs/get-started/quickstart/quickstart-5-minutes) [ Start with Cloud](/docs/get-started/quickstart/quickstart-5-minutes) [![](/docs/_astro/timescaledb-bg.ChoYqvya_1Ut6sC.webp) ![](/docs/_astro/timescaledb-bg-dark.yqGn6fW3_ORnrn.webp)](/docs/get-started/choose-your-path/install-timescaledb) ## [Deploy TimescaleDB locally](/docs/get-started/choose-your-path/install-timescaledb) [Installation guides, configuration tips, and best practices for self-hosting.](/docs/get-started/choose-your-path/install-timescaledb) [ Start with TimescaleDB](/docs/get-started/choose-your-path/install-timescaledb) ## Explore Explore the docs to find what you need, whether you are learning new concepts, following a tutorial, or looking up a detail. Each area below is a starting point; open a section to go deeper. [Learn TimescaleDB concepts](/docs/learn) ## [Learn](/docs/learn) [Concepts, comparisons, and vocabulary: how TimescaleDB fits together, from hypertables to data lifecycle, plus a glossary.](/docs/learn) ## [Build](/docs/build) [Task-by-task guides: tutorials, examples, data management, continuous aggregates, performance, and cost patterns.](/docs/build) ## [Deploy](/docs/deploy) [Run on Tiger Cloud, self-hosted TimescaleDB, or Managed Service for TimescaleDB: install, configure, and operate in your environment.](/docs/deploy) ## [Reference](/docs/reference) [SQL functions, Toolkit APIs, configuration settings, and the REST API: look up syntax and behavior while you build.](/docs/reference) *** ## Core features TimescaleDB is a PostgreSQL extension, not a separate database or fork, so you keep the same clients, drivers, and SQL you use with plain PostgreSQL. On top of that, it adds objects and commands for time-partitioned tables, rollups, compression, retention, and other time-series and analytics work. [Open Build](/docs/build) ## [Hypertables](/docs/learn/hypertables/understand-hypertables) [Automatic time-based partitioning for PostgreSQL tables. Scale to billions of rows with no changes to your queries.](/docs/learn/hypertables/understand-hypertables) ## [Continuous aggregation](/docs/learn/continuous-aggregates) [Incrementally materialized views that stay up to date automatically. Ideal for dashboards and real-time reporting.](/docs/learn/continuous-aggregates) ## [Columnstore](/docs/learn/columnar-storage/understand-hypercore) [Transparent columnar compression that reduces storage by up to 95%. Query compressed data with standard SQL.](/docs/learn/columnar-storage/understand-hypercore) ## [AI & vectors](/docs/learn/search/pgvector-pgvectorsearch) [Store and query vector embeddings alongside your time-series data with pgvector integration for semantic search.](/docs/learn/search/pgvector-pgvectorsearch) ## [pg\_textsearch](/docs/deploy/tiger-cloud/tiger-cloud-aws/tiger-cloud-extensions/pg-textsearch) [BM25 full-text search in PostgreSQL. Rank documents by relevance and combine search with relational queries.](/docs/deploy/tiger-cloud/tiger-cloud-aws/tiger-cloud-extensions/pg-textsearch) ## [Retention](/docs/learn/data-lifecycle/data-retention/about-data-retention) [Automated retention policies, tiered storage, and data management. Keep what matters, archive the rest.](/docs/learn/data-lifecycle/data-retention/about-data-retention) *** ## What's new Follow doc changes in the changelog, check out the current platform status or the TimescaleDB release notes on GitHub. [View changelog](/docs/get-started/news/new) ## [Service status](https://status.tigerdata.com/) [Current status and incident history for Tiger Data services. Open the status page to subscribe to updates.](https://status.tigerdata.com/) ## [TimescaleDB releases](https://github.com/timescale/timescaledb/releases) [Release notes, downloads, and highlights for each TimescaleDB version on GitHub.](https://github.com/timescale/timescaledb/releases) # Build with Tiger Data > Task-oriented guides organized by what you want to get done ## Start here [Section titled “Start here”](#start-here) Tips **New to Tiger Data?** Begin with [Your first hypertable](/docs/build/how-to/your-first-hypertable), then try a [guided project](/docs/build/examples) like analyzing transport and geospatial data. **Building for production?** Jump straight to [Write and query data](/docs/build/data-management) and then explore [jobs and policies](/docs/build/data-management/about-automation) or [storage tiering](/docs/build/data-management/storage/manage-storage). ## I want to… [Section titled “I want to…”](#i-want-to) Choose a goal below to find the right guide. [Write and query data](/docs/build/data-management) [Create hypertables, insert and query time-series data, and use time buckets](/docs/build/data-management) [Automate with jobs and policies](/docs/build/data-management/about-automation) [Schedule background jobs and configure data retention policies](/docs/build/data-management/about-automation) [Spread data across storage tiers](/docs/build/data-management/storage/manage-storage) [Move cold data to cheaper storage with tiered storage](/docs/build/data-management/storage/manage-storage) [Use hyperfunctions for analytics](/docs/build/data-management/hyperfunctions) [Set up counter aggregation, gapfilling, percentiles, and time-weighted averages](/docs/build/data-management/hyperfunctions) [Keep pre-computed aggregations up to date](/docs/build/continuous-aggregates/create-a-continuous-aggregate) [Create and manage continuous aggregates for dashboards and reporting](/docs/build/continuous-aggregates/create-a-continuous-aggregate) [Optimize storage and query speed](/docs/build/columnar-storage/setup-hypercore) [Set up hypercore, the hybrid row-columnar storage engine](/docs/build/columnar-storage/setup-hypercore) [Make queries and schemas faster](/docs/build/performance-optimization) [Tune indexes, schema design, constraints, and hypertable performance](/docs/build/performance-optimization) ## Get hands on [Section titled “Get hands on”](#get-hands-on) Looking for tutorials, real-life guided projects, or code examples? Everything you need is in the **[Guided projects](/docs/build/examples)**. ## Something not working? [Section titled “Something not working?”](#something-not-working) Head to [Troubleshooting](/docs/build/tips-and-tricks) for common issues organized by capability. # Set up hypercore > Reduce chunk size by up to 98% and speed up queries by converting data between the rowstore and columnstore [Hypercore](/docs/learn/columnar-storage/understand-hypercore) is the hybrid row-columnar storage engine in TimescaleDB used by hypertables. Traditional databases force a trade-off between fast inserts (row-based storage) and efficient analytics (columnar storage). Hypercore eliminates this trade-off, allowing real-time analytics without sacrificing transactional capabilities. Hypercore dynamically stores data in the most efficient format for its lifecycle: ![Move from rowstore to columstore in hypercore](/docs/_astro/hypercore_intro.DmLiqZGV_1VM5z7.webp) ![Move from rowstore to columstore in hypercore](/docs/_astro/hypercore_intro-dark.qPmjkBqW_19V2rF.webp) * **Row-based storage for recent data**: the most recent chunk (and possibly more) is always stored in the rowstore, ensuring fast inserts, updates, and low-latency single record queries. Additionally, row-based storage is used as a writethrough for inserts and updates to columnar storage. * **Columnar storage for analytical performance**: chunks are automatically compressed into the columnstore, optimizing storage efficiency and accelerating analytical queries. Unlike traditional columnar databases, hypercore allows data to be inserted or modified at any stage, making it a flexible solution for both high-ingest transactional workloads and real-time analytics, within a single database. When you convert chunks from the rowstore to the columnstore, multiple records are grouped into a single row. The columns of this row hold an array-like structure that stores all the data. For example, data in the following rowstore chunk: | time | symbol | price | day\_volume | | -------- | ------- | -------- | ----------- | | 12:00:01 | BTC/USD | 42000.50 | 130 | | 12:00:01 | ETH/USD | 2200.70 | 205 | | 12:00:02 | BTC/USD | 42000.12 | 132 | | 12:00:02 | ETH/USD | 2200.69 | 234 | | 12:00:03 | BTC/USD | 42000.14 | 130 | | 12:00:03 | ETH/USD | 2200.70 | 252 | Is converted and compressed into arrays in a row in the columnstore: | time | symbol | price | day\_volume | | ------------------------------------------------------------- | ------------------------------------------------------- | ---------------------------------------------------------- | ------------------------------- | | \[12:00:01, 12:00:01, 12:00:02, 12:00:02, 12:00:03, 12:00:03] | \[BTC/USD, ETH/USD, BTC/USD, ETH/USD, BTC/USD, ETH/USD] | \[42000.50, 2200.70, 42000.12, 2200.69, 42000.14, 2200.70] | \[130, 205, 132, 234, 130, 252] | Because a single row takes up less disk space, you can reduce your chunk size by up to 98%, and can also speed up your queries. This saves on storage costs, and keeps your queries operating at lightning speed. For an in-depth explanation of how hypertables and hypercore work, see the [Data model](/docs/learn/deep-dive/whitepaper#data-model). This page shows you how to convert chunks in a hypertable from the rowstore to the columnstore — automatically with a columnstore policy or manually for finer control — and how to get the best compression and query performance from your data. ## Prerequisites for this procedure To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service). - Your [connection details](/docs/integrate/find-connection-details). ## Convert your data to the columnstore [Section titled “Convert your data to the columnstore”](#convert-your-data-to-the-columnstore) The compression ratio and query performance of data in the columnstore is dependent on the order and structure of your data. Rows that change over a dimension should be close to each other. With time-series data, you `orderby` the time dimension. For example, `time`: | time | symbol | price | day\_volume | | -------- | ------- | -------- | ----------- | | 12:00:01 | BTC/USD | 42000.50 | 130 | This ensures that records are compressed and accessed in the same order. However, you would always have to access the data using the time dimension, then filter all the rows using other criteria. To make your queries more efficient, you segment your data based on the following: * The way you want to access it. For example, to rapidly access data about a single symbol, you `segmentby` the `symbol` column. This enables you to run much faster analytical queries on data in the columnstore. * The compression rate you want to achieve. The [lower the cardinality](https://www.tigerdata.com/blog/what-is-high-cardinality) of the `segmentby` column, the better compression results you get. When TimescaleDB converts a chunk to the columnstore, it automatically creates a different schema for your data. It also creates and uses custom indexes to incorporate the `segmentby` and `orderby` parameters when you write to and read from the columnstore. Choose the conversion path that fits your workload: ### Convert automatically with a columnstore policy A [columnstore policy](/docs/reference/timescaledb/hypercore/add_columnstore_policy) runs as a background job and converts eligible chunks to the columnstore on a schedule. This is the best path for most workloads. 1. **Connect to your Tiger Cloud service** In [Tiger Console](https://console.cloud.tigerdata.com/dashboard/services) open an [SQL editor](/docs/build/data-management/run-queries-from-tiger-console). You can also connect to your service using [psql](/docs/integrate/query-administration/psql#connect-to-your-service). 2. **Enable the columnstore and add a policy** For [efficient queries](/docs/build/performance-optimization/secondary-indexes), `segmentby` the column you filter on most often, and `orderby` your time column. How you enable hypercore depends on what you start from: * **New hypertable** Use [`CREATE TABLE`](/docs/reference/timescaledb/hypertables/create_table) to create a hypertable with hypercore enabled by default: ```sql CREATE TABLE crypto_ticks ( "time" TIMESTAMPTZ, symbol TEXT, price DOUBLE PRECISION, day_volume NUMERIC ) WITH ( timescaledb.hypertable, timescaledb.segmentby='symbol', timescaledb.orderby='time DESC' ); ``` When you create a hypertable using [CREATE TABLE ... WITH ...](/docs/reference/timescaledb/hypertables/create_table), the default partitioning column is automatically the first column with a timestamp data type. Also, TimescaleDB creates a [columnstore policy](/docs/reference/timescaledb/hypercore/add_columnstore_policy) that automatically converts your data to the columnstore, after an interval equal to the value of the [chunk\_interval](/docs/reference/timescaledb/hypertables/set_chunk_time_interval), defined through `after` in the policy. This columnar format enables fast scanning and aggregation, optimizing performance for analytical workloads while also saving significant storage space. In the columnstore conversion, hypertable chunks are compressed by up to 98%, and organized for efficient, large-scale queries. You can customize this policy later using [alter\_job](/docs/reference/timescaledb/jobs-automation/alter_job). However, to change `after` or `created_before`, the compression settings, or the hypertable the policy is acting on, you must [remove the columnstore policy](/docs/reference/timescaledb/hypercore/remove_columnstore_policy) and [add a new one](/docs/reference/timescaledb/hypercore/add_columnstore_policy). You can also manually [convert chunks](/docs/reference/timescaledb/hypercore/convert_to_columnstore) in a hypertable to the columnstore. * **Existing hypertable** Enable the columnstore on a hypertable that already holds data in the rowstore: ```sql ALTER TABLE crypto_ticks SET ( timescaledb.enable_columnstore, timescaledb.segmentby = 'symbol', timescaledb.orderby = 'time DESC' ); ``` These settings apply to chunks that have not yet been converted to the columnstore. For a hypertable that has never used hypercore, that means every chunk. Then add a policy: ```sql CALL add_columnstore_policy('crypto_ticks', after => INTERVAL '7d'); ``` The job runs single-threaded, so for a large backlog the initial conversion can take a while to catch up. To convert a backlog faster, [convert chunks manually](#convert-chunks-manually). * **Existing continuous aggregate** A continuous aggregate is a specialized hypertable. Use [`ALTER MATERIALIZED VIEW`](/docs/reference/timescaledb/continuous-aggregates/alter_materialized_view): ```sql ALTER MATERIALIZED VIEW assets_candlestick_daily set ( timescaledb.enable_columnstore = true, timescaledb.segmentby = 'symbol'); ``` The continuous aggregate must already have a [refresh policy](/docs/build/continuous-aggregates/refresh-policies) before you add a columnstore policy. Then add the policy: ```sql CALL add_columnstore_policy('assets_candlestick_daily', after => INTERVAL '1d'); ``` TimescaleDB is optimized for fast updates on compressed data in the columnstore. To modify data in the columnstore, use standard SQL. 3. **Check the columnstore policy** When you convert data to the columnstore, as well as being optimized for analytics, it is compressed by more than 90%. This helps you save on storage costs and keeps your queries operating at lightning speed. To see the amount of space saved: ```sql SELECT pg_size_pretty(before_compression_total_bytes) as before, pg_size_pretty(after_compression_total_bytes) as after FROM hypertable_columnstore_stats('crypto_ticks'); ``` You see something like: | before | after | | ------ | ----- | | 194 MB | 24 MB | View the policies that you set or that already exist: ```sql SELECT * FROM timescaledb_information.jobs WHERE proc_name='policy_compression'; ``` See [timescaledb\_information.jobs](/docs/reference/timescaledb/informational-views/jobs). 4. **Remove a policy or disable the columnstore** To remove a columnstore policy while keeping existing chunks in the columnstore: ```sql CALL remove_columnstore_policy('crypto_ticks'); ``` See [remove\_columnstore\_policy](/docs/reference/timescaledb/hypercore/remove_columnstore_policy). To disable the columnstore entirely, first [convert the chunks back to the rowstore](/docs/reference/timescaledb/hypercore/convert_to_rowstore), then: ```sql ALTER TABLE crypto_ticks SET (timescaledb.enable_columnstore = false); ``` See [alter\_table\_hypercore](/docs/reference/timescaledb/hypercore/alter_table). ### Convert chunks manually Call [convert\_to\_columnstore](/docs/reference/timescaledb/hypercore/convert_to_columnstore) on individual chunks when you want finer control than a policy gives you — for example, to convert a large backlog faster than the single-threaded policy job can. The columnstore settings that you set with `ALTER TABLE` still apply. 1. **List the chunks to convert** ```sql SELECT show_chunks('crypto_ticks', older_than => INTERVAL '7d'); ``` 2. **Convert the chunks** Chunks are converted independently, so you can run `convert_to_columnstore` on distinct chunks from multiple sessions in parallel: ```sql -- Session 1 CALL convert_to_columnstore('_timescaledb_internal._hyper_1_2_chunk'); -- Session 2 (concurrent) CALL convert_to_columnstore('_timescaledb_internal._hyper_1_3_chunk'); ``` Each call takes an exclusive lock on the chunk it is converting. Different chunks do not block each other, so parallel sessions speed up an initial migration. Match the degree of parallelism to your service's available CPU and I/O. ### Backfill while converting This applies whether you convert with a policy or manually. Conversion contends on locks with any concurrent write to the same chunk. If you backfill old data while a columnstore policy or a manual `convert_to_columnstore` call is running, the two operations wait on each other and one can fail or stall. For a clean migration: 1. **Pause the columnstore policy** Find the `job_id`: ```sql SELECT job_id FROM timescaledb_information.jobs WHERE proc_name = 'policy_compression' AND hypertable_name = 'crypto_ticks'; ``` Then pause the policy: ```sql SELECT alter_job(, scheduled => false); ``` 2. **Backfill your data** Run your backfill while the policy is paused, so it does not contend with conversion for locks. 3. **Convert the affected chunks** Convert the chunks you backfilled into the columnstore. See [Convert chunks manually](#convert-chunks-manually). 4. **Re-enable the policy** ```sql SELECT alter_job(, scheduled => true); ``` For the full pause-backfill-reconvert workflow, see [convert\_to\_rowstore](/docs/reference/timescaledb/hypercore/convert_to_rowstore). ## Reference [Section titled “Reference”](#reference) For integers, timestamps, and other integer-like types, data is compressed using [delta encoding](/docs/learn/columnar-storage/compression-methods#delta-encoding), [delta-of-delta](/docs/learn/columnar-storage/compression-methods#delta-of-delta-encoding), [simple-8b](/docs/learn/columnar-storage/compression-methods#simple-8b), and [run-length encoding](/docs/learn/columnar-storage/compression-methods#run-length-encoding). For columns with few repeated values, [XOR-based](/docs/learn/columnar-storage/compression-methods#xor-based-compression) and [dictionary compression](/docs/learn/columnar-storage/compression-methods#dictionary-compression) is used. For all other types, [dictionary compression](/docs/learn/columnar-storage/compression-methods#dictionary-compression) is used. # Convert continuous aggregates to the columnstore > Compress a continuous aggregate to the columnstore to save storage space while making sure the data is still available for your analytical workloads Since [2.20.0](https://github.com/timescale/timescaledb/releases/tag/2.20.0) To save on storage costs, you use hypercore to downsample historical data stored in continuous aggregates. After you [enable hypercore](/docs/reference/timescaledb/continuous-aggregates/alter_materialized_view) on a `MATERIALIZED VIEW`, you set a [columnstore policy](/docs/reference/timescaledb/hypercore/add_columnstore_policy). This policy defines the intervals when TimescaleDB converts chunks in a continuous aggregate from the rowstore to the columnstore. Columnstore works in the same way on [hypertables and continuous aggregates](/docs/learn/columnar-storage/understand-hypercore). When you enable columnstore with no other options, your data is [segmented by](/docs/reference/timescaledb/continuous-aggregates/alter_materialized_view#arguments) the `groupby` columns in the continuous aggregate, and [ordered by](/docs/reference/timescaledb/continuous-aggregates/alter_materialized_view#arguments) the time column. [Real-time aggregate](/docs/learn/continuous-aggregates/real-time-aggregates) is disabled by default. ## Configure columnstore on continuous aggregates [Section titled “Configure columnstore on continuous aggregates”](#configure-columnstore-on-continuous-aggregates) For an [existing continuous aggregate](/docs/build/continuous-aggregates/create-a-continuous-aggregate): 1. **Enable hypercore on a continuous aggregate** Set `timescaledb.enable_columnstore = true` when you alter the view: ```sql ALTER MATERIALIZED VIEW set (timescaledb.enable_columnstore = true); ``` To disable hypercore, set `timescaledb.enable_columnstore = false`. 2. **Set the columnstore policy on the continuous aggregate** Before you set up a columnstore policy on a continuous aggregate, you first set the [refresh policy](/docs/build/continuous-aggregates/refresh-policies). To prevent refresh policies from failing, set the columnstore policy interval so that TimescaleDB does not convert actively refreshed regions to the columnstore. For example: 1. Set the refresh policy: ```sql SELECT add_continuous_aggregate_policy('', start_offset => INTERVAL '30 days', end_offset => INTERVAL '1 day', schedule_interval => INTERVAL '1 hour'); ``` 2. Set the columnstore policy. The `after` parameter should be greater than the value of `start_offset` in the refresh policy: ```sql CALL add_columnstore_policy('', after => INTERVAL '45 days'); ``` ## Convert refreshed chunks to the columnstore during refresh [Section titled “Convert refreshed chunks to the columnstore during refresh”](#convert-refreshed-chunks-to-the-columnstore-during-refresh) Since [2.27.0](https://github.com/timescale/timescaledb/releases/tag/2.27.0) As an alternative to a separate columnstore policy, you can configure the refresh policy itself to move continuous aggregate chunks into the columnstore after each refresh. Use this when you want refreshed data to land in the columnstore on the same schedule as the refresh, with no gap between the two windows. When you enable `compress_after_refresh` on the refresh policy job, TimescaleDB converts the continuous aggregate chunks in the refresh window to the columnstore at the end of each scheduled refresh. TimescaleDB processes each chunk in its own transaction, so the job resumes from the last completed chunk if interrupted. `compress_after_refresh` is a JSONB key on the refresh policy job's `config` and defaults to `false`. Set it with [`alter_job`](/docs/reference/timescaledb/jobs-automation/alter_job): ```sql SELECT alter_job(, config => jsonb_set( (SELECT config FROM _timescaledb_catalog.bgw_job WHERE id = ), '{compress_after_refresh}', 'true')); ``` You must enable hypercore on the continuous aggregate first. Otherwise, TimescaleDB skips the conversion. # Create a continuous aggregate > Create a continuous aggregate and make sure you always have the latest aggregated data for your analytical queries A continuous aggregate stores precomputed rollups so analytical queries avoid rescanning all raw rows. You create the materialized view first, then wire up refresh (policy or manual). You can build on a hypertable or stack another continuous aggregate, and you can define more than one continuous aggregate per source. Continuous aggregates require a `time_bucket` on the time partitioning column of the hypertable. By default, views are automatically refreshed. You can adjust this by setting the [WITH NO DATA](#using-the-with-no-data-option) option. Additionally, the view can not be a [security barrier view](https://www.postgresql.org/docs/current/rules-privileges.html). Continuous aggregates use hypertables in the background, which means that they also use chunk time intervals. By default, the continuous aggregate's chunk time interval is 10 times what the original hypertable's chunk time interval is. For example, if the original hypertable's chunk time interval is 7 days, the continuous aggregates that are on top of it have a 70 day chunk time interval. ## Create a continuous aggregate [Section titled “Create a continuous aggregate”](#create-a-continuous-aggregate) In this example, we are using a hypertable called `conditions`, and creating a continuous aggregate view for daily weather data. The `GROUP BY` clause must include a `time_bucket` expression which uses time dimension column of the hypertable. Additionally, all functions and their arguments included in `SELECT`, `GROUP BY`, and `HAVING` clauses must be [immutable](https://www.postgresql.org/docs/current/xfunc-volatility.html). 1. **Create the materialized view** ```sql CREATE MATERIALIZED VIEW conditions_summary_daily WITH (timescaledb.continuous) AS SELECT device, time_bucket(INTERVAL '1 day', time) AS bucket, AVG(temperature), MAX(temperature), MIN(temperature) FROM conditions GROUP BY device, bucket; ``` To create a continuous aggregate within a transaction block, use the [WITH NO DATA option](#using-the-with-no-data-option). To improve continuous aggregate performance, [set `timescaledb.invalidate_using = 'wal'`](/docs/reference/timescaledb/continuous-aggregates/create_materialized_view) Since [2.22.0](https://github.com/timescale/timescaledb/releases/tag/2.22.0) . 2. **Create a refresh policy** ```sql SELECT add_continuous_aggregate_policy('conditions_summary_daily', start_offset => INTERVAL '1 month', end_offset => INTERVAL '1 day', schedule_interval => INTERVAL '1 hour'); ``` You can use most PostgreSQL aggregate functions in continuous aggregations. To see what PostgreSQL features are supported, check the [function support table](/docs/learn/continuous-aggregates#function-support). ## Choosing an appropriate bucket interval [Section titled “Choosing an appropriate bucket interval”](#choosing-an-appropriate-bucket-interval) Continuous aggregates require a `time_bucket` on the time partitioning column of the hypertable. The time bucket allows you to define a time interval, instead of having to use specific timestamps. For example, you can define a time bucket as five minutes, or one day. You can't use [`time_bucket_gapfill`](/docs/reference/timescaledb/hyperfunctions/time_bucket_gapfill/time_bucket_gapfill) directly in a continuous aggregate. This is because you need access to previous data to determine the gapfill content, which isn't yet available when you create the continuous aggregate. You can work around this by creating the continuous aggregate using [`time_bucket`](/docs/reference/timescaledb/hyperfunctions/time-series-utilities/time_bucket), then querying the continuous aggregate using `time_bucket_gapfill`. ## Using the WITH NO DATA option [Section titled “Using the WITH NO DATA option”](#using-the-with-no-data-option) By default, when you create a view for the first time, it is populated with data. This is so that the aggregates can be computed across the entire hypertable. If you don't want this to happen, for example if the table is very large, or if new data is being continuously added, you can control the order in which the data is refreshed. You can do this by adding a manual refresh with your continuous aggregate policy using the `WITH NO DATA` option. The `WITH NO DATA` option allows the continuous aggregate to be created instantly, so you don't have to wait for the data to be aggregated. Data begins to populate only when the policy begins to run. This means that only data newer than the `start_offset` time begins to populate the continuous aggregate. If you have historical data that is older than the `start_offset` interval, you need to manually refresh the history up to the current `start_offset` to allow real-time aggregate queries to run efficiently. 1. **Create the view** ```sql CREATE MATERIALIZED VIEW cagg_rides_view WITH (timescaledb.continuous) AS SELECT vendor_id, time_bucket('1h', pickup_datetime) AS hour, count(*) total_rides, avg(fare_amount) avg_fare, max(trip_distance) as max_trip_distance, min(trip_distance) as min_trip_distance FROM rides GROUP BY vendor_id, time_bucket('1h', pickup_datetime) WITH NO DATA; ``` 2. **Manually refresh the view** ```sql CALL refresh_continuous_aggregate('cagg_rides_view', NULL, localtimestamp - INTERVAL '1 week'); ``` 3. **Add the policy** ```sql SELECT add_continuous_aggregate_policy('cagg_rides_view', start_offset => INTERVAL '1 week', end_offset => INTERVAL '1 hour', schedule_interval => INTERVAL '30 minutes'); ``` ## Create a continuous aggregate with a JOIN [Section titled “Create a continuous aggregate with a JOIN”](#create-a-continuous-aggregate-with-a-join) In TimescaleDB v2.10 and later, with PostgreSQL v12 or later, you can create a continuous aggregate with a query that also includes a `JOIN`. For example: ```sql CREATE MATERIALIZED VIEW conditions_summary_daily_3 WITH (timescaledb.continuous) AS SELECT time_bucket(INTERVAL '1 day', day) AS bucket, AVG(temperature), MAX(temperature), MIN(temperature), name FROM devices JOIN conditions USING (device_id) GROUP BY name, bucket; ``` Note For more information about creating a continuous aggregate with a `JOIN`, including some additional restrictions, see the [about continuous aggregates section](/docs/learn/continuous-aggregates#continuous-aggregates-with-a-join-clause). ## Query continuous aggregates [Section titled “Query continuous aggregates”](#query-continuous-aggregates) When you have created a continuous aggregate and set a refresh policy, you can query the view with a `SELECT` query. You can only specify a single hypertable in the `FROM` clause. Including more hypertables, tables, views, or subqueries in your `SELECT` query is not supported. Additionally, make sure that the hypertable you are querying does not have [row-level-security policies](https://www.postgresql.org/docs/current/ddl-rowsecurity.html) enabled. 1. **Query the continuous aggregate for a specific device** ```sql SELECT * FROM conditions_summary_hourly WHERE device = 5 AND bucket >= '2020-01-01' AND bucket < '2020-04-01'; ``` 2. **Query for largest metric spreads** ```sql SELECT * FROM conditions_summary_hourly WHERE max - min > 1800 AND bucket >= '2020-01-01' AND bucket < '2020-04-01' ORDER BY bucket DESC, device DESC LIMIT 20; ``` ## Use continuous aggregates with mutable functions: experimental [Section titled “Use continuous aggregates with mutable functions: experimental”](#use-continuous-aggregates-with-mutable-functions-experimental) Since [2.20.0](https://github.com/timescale/timescaledb/releases/tag/2.20.0) Mutable functions have experimental support in the continuous aggregate query definition. Mutable functions are enabled by default. However, if you use them in a materialized query a warning is returned. When using non-immutable functions you have to ensure these functions produce consistent results across continuous aggregate refresh runs. For example, if a function depends on the current time zone you have to ensure all your continuous aggregate refreshes run with a consistent setting for this. ## Use continuous aggregates with window functions: experimental [Section titled “Use continuous aggregates with window functions: experimental”](#use-continuous-aggregates-with-window-functions-experimental) Since [2.20.0](https://github.com/timescale/timescaledb/releases/tag/2.20.0) Window functions have experimental support in the continuous aggregate query definition. Window functions are disabled by default. To enable them, set `timescaledb.enable_cagg_window_functions` to `true`. Warning Support is experimental, there is a risk of data inconsistency. For example, in backfill scenarios, buckets could be missed. ### Create a window function [Section titled “Create a window function”](#create-a-window-function) To use a window function in a continuous aggregate: 1. **Create a simple table** ```sql CREATE TABLE example ( time TIMESTAMPTZ NOT NULL, value TEXT NOT NULL ); ``` 2. **Enable window functions** As window functions are experimental, you have to set `enable_cagg_window_functions`. ```sql SET timescaledb.enable_cagg_window_functions TO TRUE; ``` 3. **Create the continuous aggregate with a window function** Window functions must stay within the time bucket. Any query that tries to look beyond the current time bucket will produce incorrect results around the refresh boundaries. ```sql CREATE MATERIALIZED VIEW example_aggregate WITH (timescaledb.continuous) AS SELECT time_bucket('1d', time), customer_id, sum(amount) AS amount, sum(amount) - LAG(sum(amount),1,NULL) OVER (PARTITION BY time_bucket('1d', time) ORDER BY sum(amount) DESC) AS amount_diff, ROW_NUMBER() OVER (PARTITION BY time_bucket('1d', time) ORDER BY sum(amount) DESC) FROM sales GROUP BY 1,2; ``` Window functions that partition by time\_bucket should be safe even with LAG()/LEAD() ### Window function workaround for older versions of TimescaleDB [Section titled “Window function workaround for older versions of TimescaleDB”](#window-function-workaround-for-older-versions-of-timescaledb) For TimescaleDB v2.19.3 and below, continuous aggregates do not support window functions. To work around this: 1. **Create a simple table** Create a simple table to store a value at a specific time: ```sql CREATE TABLE example ( time TIMESTAMPTZ NOT NULL, value TEXT NOT NULL ); ``` 2. **Create a continuous aggregate without a window function** ```sql CREATE MATERIALIZED VIEW example_aggregate WITH (timescaledb.continuous) AS SELECT time_bucket('10 minutes', time) AS bucket, first(value, time) AS value FROM example GROUP BY bucket; ``` 3. **Use the lag window function at query time** This speeds up your query by calculating the aggregation ahead of time. The delta is calculated at query time. ```sql SELECT bucket, value - lag(value, 1) OVER (ORDER BY bucket) AS delta FROM example_aggregate; ``` # Create an index on a continuous aggregate > Adding an index to your continuous aggregate can make your analytical queries lightning fast Continuous aggregates create helpful composite indexes by default, but you can turn that off or add your own for specific filters. This page covers the automatic indexes, how to disable them, and how to manage custom ones. ## Automatically created indexes [Section titled “Automatically created indexes”](#automatically-created-indexes) When you create a continuous aggregate, an index is automatically created for each `GROUP BY` column. The index is a composite index, combining the `GROUP BY` column with the `time_bucket` column. For example, if you define a continuous aggregate view with `GROUP BY device, location, bucket`, two composite indexes are created: one on `{device, bucket}` and one on `{location, bucket}`. ### Turn off automatic index creation [Section titled “Turn off automatic index creation”](#turn-off-automatic-index-creation) To turn off automatic index creation, set `timescaledb.create_group_indexes` to `false` when you create the continuous aggregate. For example: ```sql CREATE MATERIALIZED VIEW conditions_daily WITH (timescaledb.continuous, timescaledb.create_group_indexes=false) AS ... ``` ## Manually create and drop indexes [Section titled “Manually create and drop indexes”](#manually-create-and-drop-indexes) You can use a regular PostgreSQL statement to create or drop an index on a continuous aggregate. For example, to create an index on `avg_temp` for a materialized hypertable named `weather_daily`: ```sql CREATE INDEX avg_temp_idx ON weather_daily (avg_temp); ``` Indexes are created under the `_timescaledb_internal` schema, where the continuous aggregate data is stored. To drop the index, specify the schema. For example, to drop the index `avg_temp_idx`, run: ```sql DROP INDEX _timescaledb_internal.avg_temp_idx ``` ### Limitations on created indexes [Section titled “Limitations on created indexes”](#limitations-on-created-indexes) In TimescaleDB v2.7 and later, you can create an index on any column in the materialized view. This includes aggregated columns, such as those storing sums and averages. In earlier versions of TimescaleDB, you can't create an index on an aggregated column. You can't create unique indexes on a continuous aggregate, in any of the TimescaleDB versions. # Drop data from continuous aggregates > Drop a view or raw data from a continuous aggregate or its underlying hypertable You can remove a continuous aggregate view, trim raw rows from the source hypertable, or rely on [refresh](/docs/build/continuous-aggregates/refresh-policies) plus retention policies to keep coarse long-term aggregates while dropping detailed history. Watch for retention rules that delete raw hypertable rows your continuous aggregate still needs to rebuild or refresh correctly. ## Drop a continuous aggregate view [Section titled “Drop a continuous aggregate view”](#drop-a-continuous-aggregate-view) You can drop a continuous aggregate view using the `DROP MATERIALIZED VIEW` command. This command also removes refresh policies defined on the continuous aggregate. It does not drop the data from the underlying hypertable. ```sql DROP MATERIALIZED VIEW view_name; ``` ## Drop raw data from a hypertable [Section titled “Drop raw data from a hypertable”](#drop-raw-data-from-a-hypertable) If you drop data from a hypertable used in a continuous aggregate it can lead to problems with your continuous aggregate view. In many cases, dropping underlying data replaces the aggregate with NULL values, which can lead to unexpected results in your view. You can drop data from a hypertable using `drop_chunks` in the usual way, but before you do so, always check that the chunk is not within the refresh window of a continuous aggregate that still needs the data. This is also important if you are manually refreshing a continuous aggregate. Calling `refresh_continuous_aggregate` on a region containing dropped chunks recalculates the aggregate without the dropped data. If a continuous aggregate is refreshing when data is dropped because of a retention policy, the aggregate is updated to reflect the loss of data. If you need to retain the continuous aggregate after dropping the underlying data, set the `start_offset` value of the aggregate policy to a smaller interval than the `drop_after` parameter of the retention policy. For more information, see the [data retention documentation](/docs/learn/data-lifecycle/data-retention/data-retention-with-continuous-aggregates). # Migrate a continuous aggregate to the new form > Migrate your old continuous aggregates to the new form introduced in TimescaleDB 2.7 Starting in TimescaleDB v2.7, continuous aggregates use a newer format that performs better and supports more SQL. Continuous aggregates created before that release, or created with `timescaledb.finalized` set to `false`, still use the legacy layout until you migrate. To migrate a continuous aggregate from the old format to the new format, you can use this procedure. It automatically copies over your data and policies. You can continue to use the continuous aggregate while the migration is happening. Note This procedure applies to TimescaleDB v2.24 and earlier. In v2.25, the old format has been fully removed. See the [release notes](https://github.com/timescale/timescaledb/blob/main/CHANGELOG.md#2250-2026-01-29). Connect to your database and run: ```sql CALL cagg_migrate(''); ``` Warning There are known issues with `cagg_migrate()` in version 2.8.0. Upgrade to version 2.8.1 or later before using it. ## Configure continuous aggregate migration [Section titled “Configure continuous aggregate migration”](#configure-continuous-aggregate-migration) The migration procedure provides two boolean configuration parameters, `override` and `drop_old`. By default, the name of your new continuous aggregate is the name of your old continuous aggregate, with the suffix `_new`. Set `override` to true to rename your new continuous aggregate with the original name. The old continuous aggregate is renamed with the suffix `_old`. To both rename and drop the old continuous aggregate entirely, set both parameters to true. Note that `drop_old` must be used together with `override`. ## Check on continuous aggregate migration status [Section titled “Check on continuous aggregate migration status”](#check-on-continuous-aggregate-migration-status) To check the progress of the continuous aggregate migration, query the migration planning table: ```sql SELECT * FROM _timescaledb_catalog.continuous_agg_migrate_plan_step; ``` ## Troubleshooting [Section titled “Troubleshooting”](#troubleshooting) ### Permissions error when migrating a continuous aggregate [Section titled “Permissions error when migrating a continuous aggregate”](#permissions-error-when-migrating-a-continuous-aggregate) You might get a permissions error when migrating a continuous aggregate from old to new format using `cagg_migrate`. The user performing the migration must have the following permissions: * Select, insert, and update permissions on the tables `_timescale_catalog.continuous_agg_migrate_plan` and `_timescale_catalog.continuous_agg_migrate_plan_step` * Usage permissions on the sequence `_timescaledb_catalog.continuous_agg_migrate_plan_step_step_id_seq` To solve the problem, change to a user capable of granting permissions, and grant the following permissions to the user performing the migration: ```sql GRANT SELECT, INSERT, UPDATE ON TABLE _timescaledb_catalog.continuous_agg_migrate_plan TO ; GRANT SELECT, INSERT, UPDATE ON TABLE _timescaledb_catalog.continuous_agg_migrate_plan_step TO ; GRANT USAGE ON SEQUENCE _timescaledb_catalog.continuous_agg_migrate_plan_step_step_id_seq TO ; ``` # Refresh continuous aggregates > Set up automatic refresh policies or refresh your aggregates manually to keep your data up to date Refresh policies control how and when a materialized continuous aggregate catches up with its source hypertable. You can run refreshes on a schedule or trigger them manually when you need a one-off catch-up. ## Prerequisites for this procedure To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service). These steps use Tiger Cloud, but the same approach applies to a [self-hosted TimescaleDB](/docs/get-started/choose-your-path/install-timescaledb) instance. ## Change the refresh policy [Section titled “Change the refresh policy”](#change-the-refresh-policy) Continuous aggregates require a policy for automatic refreshing. You can adjust this to suit different use cases. For example, you can have the continuous aggregate and the hypertable stay in sync, even when data is removed from the hypertable. Alternatively, you could keep source data in the continuous aggregate even after it is removed from the hypertable. You can change the way your continuous aggregate is refreshed by calling `add_continuous_aggregate_policy`. Among others, `add_continuous_aggregate_policy` takes the following arguments: * `start_offset`: the start of the refresh window relative to when the policy runs * `end_offset`: the end of the refresh window relative to when the policy runs * `schedule_interval`: the refresh interval in minutes or hours. Defaults to 24 hours. Note the following: * If you set the `start_offset` or `end_offset` to `NULL`, the range is open-ended and extends to the beginning or end of time. * If you set `end_offset` within the current time bucket, this bucket is excluded from materialization. This is done for the following reasons: * The current bucket is incomplete and can't be refreshed. * The current bucket gets a lot of writes in the timestamp order, and its aggregate becomes outdated very quickly. Excluding it improves performance. To include the latest raw data in queries, enable [real-time aggregate](/docs/learn/continuous-aggregates/real-time-aggregates). See the [API reference](/docs/reference/timescaledb/continuous-aggregates/add_continuous_aggregate_policy) for the full list of required and optional arguments and use examples. The policy in the following example ensures that all data in the continuous aggregate is up to date with the hypertable, except for data written within the last hour of wall-clock time. The policy also does not refresh the last time bucket of the continuous aggregate. Since the policy in this example runs once every hour (`schedule_interval`) while also excluding data within the most recent hour (`end_offset`), it takes up to 2 hours for data written to the hypertable to be reflected in the continuous aggregate. Backfills, which are usually outside the most recent hour of data, will be visible after up to 1 hour depending on when the policy last ran when the data was written. Because it has an open-ended `start_offset` parameter, any data that is removed from the table, for example with a `DELETE` or with `drop_chunks`, is also removed from the continuous aggregate view. This means that the continuous aggregate always reflects the data in the underlying hypertable. To change a refresh policy to use a `NULL` `start_offset`: 1. **Connect to your service or self-hosted TimescaleDB instance** Use [any PostgreSQL client](/docs/integrate/find-connection-details) to connect. 2. **Create the refresh policy** Create a new policy on `conditions_summary_hourly` that keeps the continuous aggregate up to date, and runs every hour: ```sql SELECT add_continuous_aggregate_policy('conditions_summary_hourly', start_offset => NULL, end_offset => INTERVAL '1 h', schedule_interval => INTERVAL '1 h'); ``` If you want to keep data in the continuous aggregate even if it is removed from the underlying hypertable, you can set the `start_offset` to match the [data retention policy](/docs/learn/data-lifecycle/data-retention/about-data-retention) on the source hypertable. For example, if you have a retention policy that removes data older than one month, set `start_offset` to one month or less. This sets your policy so that it does not refresh the dropped data. 1. **Connect to your service or self-hosted TimescaleDB instance** Use [any PostgreSQL client](/docs/integrate/find-connection-details) to connect. 2. **Create the refresh policy** Create a new policy on `conditions_summary_hourly` that keeps data removed from the hypertable in the continuous aggregate, and runs every hour: ```sql SELECT add_continuous_aggregate_policy('conditions_summary_hourly', start_offset => INTERVAL '1 month', end_offset => INTERVAL '1 h', schedule_interval => INTERVAL '1 h'); ``` Warning It is important to consider your data retention policies when you're setting up continuous aggregate policies. If the continuous aggregate policy window covers data that is removed by the data retention policy, the data will be removed when the aggregates for those buckets are refreshed. For example, if you have a data retention policy that removes all data older than two weeks, the continuous aggregate policy will only have data for the last two weeks. ## Add concurrent refresh policies [Section titled “Add concurrent refresh policies”](#add-concurrent-refresh-policies) You can add concurrent refresh policies on each continuous aggregate, as long as their start and end offsets don't overlap. For example, to backfill data into older chunks you set up one policy that refreshes recent data, and another that refreshes backfilled data. The first policy in this example keeps the continuous aggregate up to date with data that was inserted in the past day. Any data that was inserted or updated for previous days is refreshed by the second policy. 1. **Connect to your service or self-hosted TimescaleDB instance** Use [any PostgreSQL client](/docs/integrate/find-connection-details) to connect. 2. **Create the recent data policy** Create a new policy on `conditions_summary_daily` to refresh the continuous aggregate with recently inserted data which runs hourly: ```sql SELECT add_continuous_aggregate_policy('conditions_summary_daily', start_offset => INTERVAL '1 day', end_offset => INTERVAL '1 h', schedule_interval => INTERVAL '1 h'); ``` 3. **Create the backfill policy** Create a concurrent policy on `conditions_summary_daily` to refresh the continuous aggregate with backfilled data: ```sql SELECT add_continuous_aggregate_policy('conditions_summary_daily', start_offset => NULL, end_offset => INTERVAL '1 day', schedule_interval => INTERVAL '1 hour'); ``` ## Manually refresh a continuous aggregate [Section titled “Manually refresh a continuous aggregate”](#manually-refresh-a-continuous-aggregate) If you need to manually refresh a continuous aggregate, you can use the `refresh` command. This recomputes the data within the window that has changed in the underlying hypertable since the last refresh. Therefore, if only a few buckets need updating, the refresh runs quickly. If you have recently dropped data from a hypertable with a continuous aggregate, calling `refresh_continuous_aggregate` on a region containing dropped chunks recalculates the aggregate without the dropped data. See [drop data](/docs/build/continuous-aggregates/drop-data) for more information. The `refresh` command takes three arguments: * The name of the continuous aggregate view to refresh * The timestamp of the beginning of the refresh window * The timestamp of the end of the refresh window Only buckets that are wholly within the specified range are refreshed. For example, if you specify `2021-05-01', '2021-06-01` the only buckets that are refreshed are those up to but not including 2021-06-01. It is possible to specify `NULL` in a manual refresh to get an open-ended range, but we do not recommend using it, because you could inadvertently materialize a large amount of data, slow down your performance, and have unintended consequences on other policies like data retention. To manually refresh a continuous aggregate, use the `refresh` command: ```sql CALL refresh_continuous_aggregate('example', '2021-05-01', '2021-06-01'); ``` Follow the logic used by automated refresh policies and avoid refreshing time buckets that are likely to have a lot of writes. This means that you should generally not refresh the latest incomplete time bucket. To include the latest raw data in your queries, use [real-time aggregate](/docs/learn/continuous-aggregates/real-time-aggregates) instead. # Cost optimization > Reduce storage and compute costs # Lower your storage and compute bill [Section titled “Lower your storage and compute bill”](#lower-your-storage-and-compute-bill) Use the sections below when you want cheaper storage (compression, tiers, retention), lighter compute (sizing and usage), or Tiger Cloud billing context. ## I want to reduce… [Section titled “I want to reduce…”](#i-want-to-reduce) * Reduce storage costs with compression * Use [tiered storage](/docs/learn/data-lifecycle/storage/about-storage-tiers) for cold data * Configure appropriate retention policies ## Compute costs [Section titled “Compute costs”](#compute-costs) * Right-size your instance * Monitor resource usage ## Tiger Cloud [Section titled “Tiger Cloud”](#tiger-cloud) * [Pricing and account management](/docs/deploy/tiger-cloud/tiger-cloud-aws/pricing-and-account-management) ## See also [Section titled “See also”](#see-also) * [Columnar storage](/docs/learn/columnar-storage/understand-hypercore): compression and Hypercore * [Tiered storage](/docs/learn/data-lifecycle/storage/about-storage-tiers): move cold data to cheaper tiers # Write and query data > Insert, update, delete, and query time-series data in hypertables Write and read time-series data with familiar SQL plus TimescaleDB helpers. Each guide is task-oriented: follow the steps to finish one job. ## In this section [Section titled “In this section”](#in-this-section) [Write data](/docs/build/data-management/write-data/insert) [Insert, update, upsert, and delete time-series data in hypertables](/docs/build/data-management/write-data/insert) [Query data](/docs/build/data-management/query-data/select) [SELECT, SkipScan, and advanced analytic queries](/docs/build/data-management/query-data/select) [Run queries from Tiger Console](/docs/build/data-management/run-queries-from-tiger-console) [Use the SQL editor in Tiger Console to run queries against your service](/docs/build/data-management/run-queries-from-tiger-console) ## Next steps [Section titled “Next steps”](#next-steps) Once you're comfortable writing and querying data, explore: * [Automate with jobs and retention](/docs/build/data-management/about-automation): Schedule background tasks. * [Manage storage and tiering](/docs/build/data-management/storage/manage-storage): Move cold data to cheaper storage. * [Hyperfunctions](/docs/build/data-management/hyperfunctions): Advanced aggregation and analysis. # About automation > Automate refresh, hypercore, retention, and custom tasks with scheduled jobs Automation splits into built-in policies (refresh, columnstore, retention, reordering) and custom jobs you schedule yourself. Out of the box, TimescaleDB ships with policies such as: * [Continuous aggregate policies](/docs/build/continuous-aggregates/refresh-policies) to automatically refresh continuous aggregates * [Columnstore policies](/docs/reference/timescaledb/hypercore/add_columnstore_policy) to convert historical data to the columnstore * [Retention policies](/docs/build/data-management/data-retention/create-a-retention-policy) to drop historical data * [Reordering policies](/docs/reference/timescaledb/hypertables/add_reorder_policy) to reorder data within chunks If these don't cover your use case, you can create and schedule custom-defined jobs to run within your database. They help you automate periodic tasks that aren't covered by the native policies. In this section, you see how to: * [Create and manage custom jobs](/docs/build/data-management/create-and-manage-jobs) * [Create a custom retention job](/docs/build/data-management/example-generic-retention) * [Create a custom job for automatic tablespace management](/docs/build/data-management/example-tiered-storage) * [Create a custom job to downsample and compress chunks](/docs/build/data-management/example-downsample-and-compress) # Create and manage custom jobs > Create, register, test, alter, and delete custom jobs in TimescaleDB Jobs are custom functions or procedures that TimescaleDB runs on your schedule. The sections below show how to register one, test it, and change or remove it safely. ## Prerequisites for this procedure To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service), or a running instance of [self-hosted TimescaleDB](/docs/get-started/choose-your-path/install-timescaledb). ## Create a job [Section titled “Create a job”](#create-a-job) To create a job, create a [PostgreSQL function](https://www.postgresql.org/docs/current/xfunc.html) or [procedure](https://www.postgresql.org/docs/current/xproc.html) that you want your database to execute, then set it up to run on a schedule. 1. **Define a function or procedure** Wrap it in a `CREATE` statement: ```sql CREATE FUNCTION (job_id INT DEFAULT NULL, config JSONB DEFAULT NULL) RETURNS VOID DECLARE ; BEGIN ; END; $$ LANGUAGE ; ``` For example, to create a function that reindexes a table within your database: ```sql CREATE FUNCTION reindex_mytable(job_id INT DEFAULT NULL, config JSONB DEFAULT NULL) RETURNS VOID AS $$ BEGIN REINDEX TABLE mytable; END; $$ LANGUAGE plpgsql; ``` `job_id` and `config` are required arguments in the function signature. This returns `CREATE FUNCTION` to indicate that the function has successfully been created. 2. **Call the function to validate** ```sql select reindex_mytable(); ``` The result looks like this: ```sql reindex_mytable ----------------- (1 row) ``` 3. **Register your job with `add_job`** Pass the name of your job, the schedule you want it to run on, and the content of your config. For the `config` value, if you don't need any special configuration parameters, set to `NULL`. For example, to run the `reindex_mytable` function every hour: ```sql SELECT add_job('reindex_mytable', '1h', config => NULL); ``` The call returns a `job_id` and stores it along with `config` in the TimescaleDB catalog. The job runs on the schedule you set. You can also run it manually with [`run_job`](/docs/reference/timescaledb/jobs-automation/run_job) passing `job_id`. When the job runs, `job_id` and `config` are passed as arguments. 4. **Validate the job** List all currently registered jobs with [`timescaledb_information.jobs`](/docs/reference/timescaledb/informational-views/jobs): ```sql SELECT * FROM timescaledb_information.jobs; ``` The result looks like this: ```sql job_id | application_name | schedule_interval | max_runtime | max_retries | retry_period | proc_schema | proc_name | owner | scheduled | config | next_start | hypertable_schema | hypertable_name --------+----------------------------+-------------------+-------------+-------------+--------------+-----------------------+------------------+-----------+-----------+------------------------+-------------------------------+-------------------+----------------- 1 | Telemetry Reporter [1] | 24:00:00 | 00:01:40 | -1 | 01:00:00 | _timescaledb_internal | policy_telemetry | postgres | t | | 2022-08-18 06:26:39.524065+00 | | 1000 | User-Defined Action [1000] | 01:00:00 | 00:00:00 | -1 | 00:05:00 | public | reindex_mytable | tsdbadmin | t | | 2022-08-17 07:17:24.831698+00 | | (2 rows) ``` ## Test and debug a job [Section titled “Test and debug a job”](#test-and-debug-a-job) To debug a job, increase the log level and run the job manually with [`run_job`](/docs/reference/timescaledb/jobs-automation/run_job) in the foreground. Because `run_job` is a stored procedure and not a function, run it with [`CALL`](https://www.postgresql.org/docs/current/sql-call.html) instead of `SELECT`. 1. **Set the minimum log level to `DEBUG1`** ```sql SET client_min_messages TO DEBUG1; ``` 2. **Run the job** Replace `1000` with your `job_id`: ```sql CALL run_job(1000); ``` ## Alter and delete a job [Section titled “Alter and delete a job”](#alter-and-delete-a-job) Alter an existing job with [`alter_job`](/docs/reference/timescaledb/jobs-automation/alter_job). You can change both the config and the schedule on which the job runs. 1. **Change a job's config** To replace the entire JSON config for a job, call `alter_job` with a new `config` object. For example, replace the JSON config for a job with ID `1000`: ```sql SELECT alter_job(1000, config => '{"hypertable":"metrics"}'); ``` 2. **Turn off job scheduling** To turn off automatic scheduling of a job, call `alter_job` and set `scheduled` to `false`. You can still run the job manually with `run_job`. For example, turn off the scheduling for a job with ID `1000`: ```sql SELECT alter_job(1000, scheduled => false); ``` 3. **Re-enable automatic scheduling** To re-enable automatic scheduling of a job, call `alter_job` and set `scheduled` to `true`. For example, re-enable scheduling for a job with ID `1000`: ```sql SELECT alter_job(1000, scheduled => true); ``` 4. **Delete a job with `delete_job`** For example, to delete a job with ID `1000`: ```sql SELECT delete_job(1000); ``` # Add a data retention policy > Create a data retention policy to automatically drop historical data from your hypertables when it reaches a certain age Retention policies drop entire chunks once their time range falls outside your window, which is cheaper than deleting millions of rows one by one. TimescaleDB schedules a background job to apply the policy for you. ## Add a data retention policy [Section titled “Add a data retention policy”](#add-a-data-retention-policy) Add a data retention policy by using the [`add_retention_policy`](/docs/reference/timescaledb/data-retention/add_retention_policy) function. 1. **Choose your hypertable and retention interval** Decide how long you want to keep data before dropping it. In this example, the hypertable named `conditions` retains the data for 24 hours. 2. **Call `add_retention_policy`** ```sql SELECT add_retention_policy('conditions', INTERVAL '24 hours'); ``` Note A data retention policy only allows you to drop chunks based on how far they are in the past. To drop chunks based on how far they are in the future, [manually drop chunks](/docs/learn/data-lifecycle/data-retention/manually-drop-chunks). ## Remove a data retention policy [Section titled “Remove a data retention policy”](#remove-a-data-retention-policy) Remove an existing data retention policy by using the [`remove_retention_policy`](/docs/reference/timescaledb/data-retention/remove_retention_policy) function. Pass it the name of the hypertable to remove the policy from. ```sql SELECT remove_retention_policy('conditions'); ``` ## See scheduled data retention jobs [Section titled “See scheduled data retention jobs”](#see-scheduled-data-retention-jobs) To see your scheduled data retention jobs and their job statistics, query the [`timescaledb_information.jobs`](/docs/reference/timescaledb/informational-views/jobs) and [`timescaledb_information.job_stats`](/docs/reference/timescaledb/informational-views/job_stats) tables. For example: ```sql SELECT j.hypertable_name, j.job_id, config, schedule_interval, job_status, last_run_status, last_run_started_at, js.next_start, total_runs, total_successes, total_failures FROM timescaledb_information.jobs j JOIN timescaledb_information.job_stats js ON j.job_id = js.job_id WHERE j.proc_name = 'policy_retention'; ``` The results look like this: ```sql -[ RECORD 1 ]-------+----------------------------------------------- hypertable_name | conditions job_id | 1000 config | {"drop_after": "5 years", "hypertable_id": 14} schedule_interval | 1 day job_status | Scheduled last_run_status | Success last_run_started_at | 2022-05-19 16:15:11.200109+00 next_start | 2022-05-20 16:15:11.243531+00 total_runs | 1 total_successes | 1 total_failures | 0 ``` # Create a custom job to downsample and compress chunks > Downsample and convert hypertable chunks to the columnstore by combining a continuous aggregate refresh policy with hypercore TimescaleDB lets you downsample and convert chunks to the columnstore by combining a [continuous aggregate refresh policy](/docs/build/continuous-aggregates/refresh-policies) with [hypercore](/docs/learn/columnar-storage/understand-hypercore). If you want to implement features not supported by those policies, you can write a job to downsample and convert chunks to the columnstore instead. The following example downsamples raw data to an average over hourly data. This is an illustrative example, which can be done more simply with a continuous aggregate policy. But you can make the query arbitrarily complex. 1. **Create a procedure to downsample chunks and convert them to columnstore** This procedure first queries the chunks of a hypertable to determine if they are older than the `lag` parameter. The hypertable in this example is named `metrics`. If the chunk is not already in the columnstore, downsample it by taking the average of the raw data, then convert it to the columnstore. This procedure uses a temporary table to store the data while calculating the average. ```sql CREATE OR REPLACE PROCEDURE downsample_compress (job_id int, config jsonb) LANGUAGE PLPGSQL AS $$ DECLARE lag interval; chunk REGCLASS; tmp_name name; BEGIN SELECT jsonb_object_field_text (config, 'lag')::interval INTO STRICT lag; IF lag IS NULL THEN RAISE EXCEPTION 'Config must have lag'; END IF; FOR chunk IN SELECT show.oid FROM show_chunks('metrics', older_than => lag) SHOW (oid) INNER JOIN pg_class pgc ON pgc.oid = show.oid INNER JOIN pg_namespace pgns ON pgc.relnamespace = pgns.oid INNER JOIN timescaledb_information.chunks chunk ON chunk.chunk_name = pgc.relname AND chunk.chunk_schema = pgns.nspname WHERE chunk.is_compressed::bool = FALSE LOOP RAISE NOTICE 'Processing chunk: %', chunk::text; -- build name for temp table SELECT '_tmp' || relname FROM pg_class WHERE oid = chunk INTO STRICT tmp_name; -- copy downsampled chunk data into temp table EXECUTE format($sql$ CREATE UNLOGGED TABLE %I AS SELECT time_bucket('1h', time), device_id, avg(value) FROM %s GROUP BY 1, 2; $sql$, tmp_name, chunk); -- clear original chunk EXECUTE format('TRUNCATE %s;', chunk); -- copy downsampled data back into chunk EXECUTE format('INSERT INTO %s(time, device_id, value) SELECT * FROM %I;', chunk, tmp_name); -- drop temp table EXECUTE format('DROP TABLE %I;', tmp_name); PERFORM convert_to_columnstore (chunk); COMMIT; END LOOP; END $$; ``` 2. **Register the job to run daily** In the `config`, set `lag` to 12 months to drop chunks containing data older than 12 months. ```sql SELECT add_job('downsample_compress','1d', config => '{"lag":"12 month"}'); ``` # Create a custom retention job > Create a generic data retention policy for your entire database using a custom job Per-hypertable [built-in retention policies](/docs/build/data-management/data-retention/create-a-retention-policy) cover most cases. When you need one rule that walks every hypertable in the database, implement it with a custom job. 1. **Create a procedure to drop old chunks** Create a procedure that drops chunks from any hypertable if they are older than the `drop_after` parameter. To get all hypertables, the `timescaledb_information.hypertables` table is queried. ```sql CREATE OR REPLACE PROCEDURE generic_retention (job_id int, config jsonb) LANGUAGE PLPGSQL AS $$ DECLARE drop_after interval; BEGIN SELECT jsonb_object_field_text (config, 'drop_after')::interval INTO STRICT drop_after; IF drop_after IS NULL THEN RAISE EXCEPTION 'Config must have drop_after'; END IF; PERFORM drop_chunks( format('%I.%I', hypertable_schema, hypertable_name), older_than => drop_after ) FROM timescaledb_information.hypertables; END $$; ``` 2. **Register the job to run daily** In the `config`, set `drop_after` to 12 months to drop chunks containing data older than 12 months. ```sql SELECT add_job('generic_retention','1d', config => '{"drop_after":"12 month"}'); ``` Note You can further refine this policy by adding filters to your procedure. For example, add a `WHERE` clause to the `PERFORM` query to only drop chunks from particular hypertables. # Create a custom job for automatic tablespace management > Automatically move hypertable chunks between tablespaces with a custom job Moving older data to a different tablespace can help you save on storage costs. TimescaleDB supports automatic tablespace management by providing the `move_chunk` function to move chunks between tablespaces. To schedule the moves automatically, you can write a custom job. Note On Tiger Cloud, use [tiered storage](/docs/build/data-management/storage/manage-storage) which handles this by providing a [tiering policy API](/docs/build/data-management/storage/manage-storage#automate-tiering-with-policies) to move data to low-cost object storage backed by Amazon S3 and Azure Blob storage. To implement automatic chunk moving with a job: 1. **Create a procedure to move chunks** Create a procedure that moves chunks to a different tablespace if they contain data older than the `lag` parameter. ```sql CREATE OR REPLACE PROCEDURE move_chunks (job_id int, config jsonb) LANGUAGE PLPGSQL AS $$ DECLARE ht REGCLASS; lag interval; destination_tablespace name; index_destination_tablespace name; reorder_index REGCLASS; chunk REGCLASS; tmp_name name; BEGIN SELECT jsonb_object_field_text (config, 'hypertable')::regclass INTO STRICT ht; SELECT jsonb_object_field_text (config, 'lag')::interval INTO STRICT lag; SELECT jsonb_object_field_text (config, 'destination_tablespace') INTO STRICT destination_tablespace; SELECT jsonb_object_field_text (config, 'index_destination_tablespace') INTO STRICT index_destination_tablespace; SELECT jsonb_object_field_text (config, 'reorder_index') INTO STRICT reorder_index; IF ht IS NULL OR lag IS NULL OR destination_tablespace IS NULL THEN RAISE EXCEPTION 'Config must have hypertable, lag and destination_tablespace'; END IF; IF index_destination_tablespace IS NULL THEN index_destination_tablespace := destination_tablespace; END IF; FOR chunk IN SELECT c.oid FROM pg_class AS c LEFT JOIN pg_tablespace AS t ON (c.reltablespace = t.oid) JOIN pg_namespace AS n ON (c.relnamespace = n.oid) JOIN (SELECT * FROM show_chunks(ht, older_than => lag) SHOW (oid)) AS chunks ON (chunks.oid::text = n.nspname || '.' || c.relname) WHERE t.spcname != destination_tablespace OR t.spcname IS NULL LOOP RAISE NOTICE 'Moving chunk: %', chunk::text; PERFORM move_chunk( chunk => chunk, destination_tablespace => destination_tablespace, index_destination_tablespace => index_destination_tablespace, reorder_index => reorder_index ); END LOOP; END $$; ``` 2. **Register the job to run daily** In the config, set `hypertable` to `metrics` to implement automatic chunk moves on the `metrics` hypertable. Set `lag` to 12 months to move chunks containing data older than 12 months. Set `tablespace` to the destination tablespace. ```sql SELECT add_job( 'move_chunks', '1d', config => '{"hypertable":"metrics","lag":"12 month","destination_tablespace":"old_chunks"}' ); ``` # Hyperfunctions overview > TimescaleDB hyperfunctions are high-performance, SQL-native functions purpose-built for time-series analysis Real-time analytics needs more than basic aggregates, especially as data volume grows. TimescaleDB hyperfunctions are SQL-native helpers built for time-series analysis: they process, aggregate, and analyze large volumes efficiently so you can run sophisticated queries and still hit interactive latencies. Hyperfunctions introduce partial aggregation, letting TimescaleDB store intermediate states instead of raw data or final results. These partials can be merged later for rollups (consolidation), eliminating costly reprocessing and slashing compute overhead, especially when paired with continuous aggregates. Take tracking p95 latency across thousands of app instances as an example: * With standard SQL, every rollup requires rescanning and resorting massive datasets. * With TimescaleDB, the `percentile_agg` hyperfunction stores a compact state per minute, which you simply merge to get hourly or daily percentiles, no full reprocess needed. ![percentile\_agg hyperfunction: high-volume raw latency data is summarized into compact per-minute states, which are then merged without reprocessing into hourly and daily p95 rollup aggregations](/docs/_astro/percentile-agg-hyperfunction-light.BwHWA7Df_QfWFy.webp) ![percentile\_agg hyperfunction: high-volume raw latency data is summarized into compact per-minute states, which are then merged without reprocessing into hourly and daily p95 rollup aggregations](/docs/_astro/percentile-agg-hyperfunction-dark.CI5mkSMk_ZXL6Os.webp) The result? Scalable, real-time percentile analytics that deliver fast, accurate insights across high-ingest, high-resolution data, while keeping resource use lean. Tiger Cloud includes all hyperfunctions by default, while self-hosted TimescaleDB includes a subset of them. To include all hyperfunctions with TimescaleDB, install the [TimescaleDB Toolkit](/docs/deploy/self-hosted/tooling/install-toolkit) PostgreSQL extension on your self-hosted PostgreSQL deployment. For more information, read the [hyperfunctions blog post](https://www.tigerdata.com/blog/time-series-analytics-for-postgresql-introducing-the-timescale-analytics-project). ## Learn hyperfunction basics and install TimescaleDB Toolkit [Section titled “Learn hyperfunction basics and install TimescaleDB Toolkit”](#learn-hyperfunction-basics-and-install-timescaledb-toolkit) * [Learn about hyperfunctions](/docs/learn/hyperfunctions/about-hyperfunctions) to understand how they work before using them. * Install the [TimescaleDB Toolkit extension](/docs/deploy/self-hosted/tooling/install-toolkit) to access more hyperfunctions on self-hosted TimescaleDB. # Counter aggregation > Collect data from counters in spite of interruptions or resets Application telemetry usually distinguishes gauges (values that move up and down) from counters (monotonic totals like requests served). Counter aggregation focuses on the second case: detect resets, stitch segments, and keep a trustworthy running total even when exporters restart. When you process counter data, it is usually assumed that if the value of the counter goes down, the counter has been reset. For example, if you wanted to count the total number of miles travelled in a vehicle, you would expect the values to continuously increase: 1, 2, 3, 4, and so on. If the counter reset to 0, you would expect that this was a new trip, or an entirely new vehicle. This can become a problem if you want to continue counting from where you left off, rather than resetting to 0. A reset could occur if you have had a short server outage, or any number of other reasons. To get around this, you can analyze counter data by looking at the change over time, which accounts for resets. Accounting for resets can be difficult to do in SQL, so TimescaleDB has developed aggregate and accessor functions that handle calculations for counters in a more practical way. Note Counter aggregates can be used in continuous aggregates, even though they are not parallelizable in PostgreSQL. For more information, see the section on [parallelism and ordering](#parallelism-and-ordering). For more information about counter aggregation API calls, see the [hyperfunction API documentation](/docs/reference/toolkit/counters-and-gauges/counter_agg/counter_agg). ## Run a counter aggregate query using a delta function [Section titled “Run a counter aggregate query using a delta function”](#run-a-counter-aggregate-query-using-a-delta-function) In this procedure, we are using an example table called `example` that contains counter data. 1. **Create the table** ```sql CREATE TABLE example ( measure_id BIGINT, ts TIMESTAMPTZ , val DOUBLE PRECISION, PRIMARY KEY (measure_id, ts) ); ``` 2. **Create the counter aggregate with a delta accessor** This gives you the change in the counter's value over the time period, accounting for any resets. This allows you to search for fifteen minute periods where the counter increased by a larger or smaller amount: ```sql SELECT measure_id, delta( counter_agg(ts, val) ) FROM example GROUP BY measure_id; ``` 3. **Use `time_bucket` for series of deltas** You can also use the `time_bucket` function to produce a series of deltas over fifteen minute increments: ```sql SELECT measure_id, time_bucket('15 min'::interval, ts) as bucket, delta( counter_agg(ts, val) ) FROM example GROUP BY measure_id, time_bucket('15 min'::interval, ts); ``` ## Run a counter aggregate query using an extrapolated delta function [Section titled “Run a counter aggregate query using an extrapolated delta function”](#run-a-counter-aggregate-query-using-an-extrapolated-delta-function) If your series is less regular, the deltas are affected by the number of samples in each fifteen minute period. You can improve this by using the `extrapolated_delta` function. To do this, you need to provide bounds that define where to extrapolate to. In this example, we use the `time_bucket_range` function, which works in the same way as `time_bucket` but produces an open ended range of all the times in the bucket. This example also uses a CTE to do the counter aggregation, which makes it a little easier to understand what's going on in each part. 1. **Create the hypertable** ```sql CREATE TABLE example ( measure_id BIGINT, ts TIMESTAMPTZ , val DOUBLE PRECISION, PRIMARY KEY (measure_id, ts) ) WITH ( tsdb.hypertable, tsdb.chunk_interval='15 days' ); ``` When you create a hypertable using [CREATE TABLE ... WITH ...](/docs/reference/timescaledb/hypertables/create_table), the default partitioning column is automatically the first column with a timestamp data type. Also, TimescaleDB creates a [columnstore policy](/docs/reference/timescaledb/hypercore/add_columnstore_policy) that automatically converts your data to the columnstore, after an interval equal to the value of the [chunk\_interval](/docs/reference/timescaledb/hypertables/set_chunk_time_interval), defined through `after` in the policy. This columnar format enables fast scanning and aggregation, optimizing performance for analytical workloads while also saving significant storage space. In the columnstore conversion, hypertable chunks are compressed by up to 98%, and organized for efficient, large-scale queries. You can customize this policy later using [alter\_job](/docs/reference/timescaledb/jobs-automation/alter_job). However, to change `after` or `created_before`, the compression settings, or the hypertable the policy is acting on, you must [remove the columnstore policy](/docs/reference/timescaledb/hypercore/remove_columnstore_policy) and [add a new one](/docs/reference/timescaledb/hypercore/add_columnstore_policy). You can also manually [convert chunks](/docs/reference/timescaledb/hypercore/convert_to_columnstore) in a hypertable to the columnstore. 2. **Create the counter aggregate with extrapolated delta** ```sql SELECT measure_id, time_bucket('15 min'::interval, ts) as bucket, extrapolated_delta( counter_agg(ts, val, toolkit_experimental.time_bucket_range('15 min'::interval, ts)), method => 'prometheus' ) as delta FROM example GROUP BY measure_id, time_bucket('15 min'::interval, ts); ``` Note In this procedure, `Prometheus` is used to do the extrapolation. TimescaleDB's current `extrapolation` function is built to mimic the Prometheus project's `increase` function, which measures the change of a counter extrapolated to the edges of the queried region. ## Run a counter aggregate query with a continuous aggregate [Section titled “Run a counter aggregate query with a continuous aggregate”](#run-a-counter-aggregate-query-with-a-continuous-aggregate) Your counter aggregate might be more useful if you make a continuous aggregate out of it. 1. **Create the continuous aggregate** ```sql CREATE MATERIALIZED VIEW example_15 WITH (timescaledb.continuous) AS SELECT measure_id, time_bucket('15 min'::interval, ts) as bucket, counter_agg(ts, val, time_bucket_range('15 min'::interval, ts)) FROM example GROUP BY measure_id, time_bucket('15 min'::interval, ts); ``` 2. **Re-aggregate into a larger bucket size** ```sql SELECT measure_id, time_bucket('1 day'::interval, bucket), delta( rollup(counter_agg) ) FROM example_15 GROUP BY measure_id, time_bucket('1 day'::interval, bucket); ``` ## Parallelism and ordering [Section titled “Parallelism and ordering”](#parallelism-and-ordering) The counter reset calculations require a strict ordering of inputs, which means they are not parallelizable in PostgreSQL. This is because PostgreSQL handles parallelism by issuing rows randomly to workers. However, if your parallelism can guarantee sets of rows that are disjointed in time, the algorithm can be parallelized, as long as it is within a time range, and all rows go to the same worker. This is the case for both continuous aggregates and for distributed hypertables, as long as the partitioning keys are in the `group by`, even though the aggregate itself doesn't really make sense otherwise. For more information about parallelism and ordering, see the [developer documentation](https://github.com/timescale/timescaledb-toolkit/blob/main/docs/counter_agg.md#counter-agg-ordering). # Function pipelines > Use function pipelines to improve how you write data analysis queries in PostgreSQL Experimental **function pipelines** let you express multi-step analysis in one readable chain, similar to functional APIs in Pandas or PromQL, without nesting many levels of SQL. They bring functional-programming ideas from those tools and from PostgreSQL and SQL into a single pipeline style. Warning Experimental features could have bugs. They might not be backwards compatible, and could be removed in future releases. Use these features at your own risk, and do not use any experimental features in production. Warning The `timevector()` function materializes all its data points in memory. This means that if you use it on a very large dataset, it runs out of memory. Do not use the `timevector` function on a large dataset, or in production. SQL is the best language for data analysis, but it is not perfect, and at times it can be difficult to construct the query you want. For example, this query gets data from the last day from the measurements table, sorts the data by the time column, calculates the delta between the values, takes the absolute value of the delta, and then takes the sum of the result of the previous steps: ```sql SELECT device_id, sum(abs_delta) as volatility FROM ( SELECT device_id, abs(val - lag(val) OVER last_day) as abs_delta FROM measurements WHERE ts >= now()-'1 day'::interval) calc_delta GROUP BY device_id; ``` You can express the same query with a function pipeline like this: ```sql SELECT device_id, toolkit_experimental.timevector(ts, val) -> toolkit_experimental.sort() -> toolkit_experimental.delta() -> toolkit_experimental.abs() -> toolkit_experimental.sum() as volatility FROM measurements WHERE ts >= now()-'1 day'::interval GROUP BY device_id; ``` Function pipelines are completely SQL compliant, meaning that any tool that speaks SQL is able to support data analysis using function pipelines. ## Anatomy of a function pipeline [Section titled “Anatomy of a function pipeline”](#anatomy-of-a-function-pipeline) Function pipelines are built as a series of elements that work together to create your query. The most important part of a pipeline is a custom data type called a `timevector`. The other elements then work on the `timevector` to build your query, using a custom operator to define the order in which the elements are run. ### Timevectors [Section titled “Timevectors”](#timevectors) A `timevector` is a collection of time,value pairs with a defined start and end time, that could look something like this: ![An example timevector](/docs/_astro/timevector.BRrHfSou_1MjAQe.webp) ![An example timevector](/docs/_astro/timevector-dark.CcDjsibJ_1MI9yy.webp) Your entire database might have time,value pairs that go well into the past and continue into the future, but the `timevector` has a defined start and end time within that dataset, which could look something like this: ![An example of a timevector within a larger dataset](/docs/_astro/timeseries_vector.DYh3K9EN_KEgp3.webp) ![An example of a timevector within a larger dataset](/docs/_astro/timeseries_vector-dark.DUTtPn0j_17DqzL.webp) To construct a `timevector` from your data, use a custom aggregate and pass in the columns to become the time,value pairs. It uses a `WHERE` clause to define the limits of the subset, and a `GROUP BY` clause to provide identifying information about the time-series. For example, to construct a `timevector` from a dataset that contains temperatures, the SQL looks like this: ```sql SELECT device_id, toolkit_experimental.timevector(ts, val) FROM measurements WHERE ts >= now() - '1 day'::interval GROUP BY device_id; ``` ### Custom operator [Section titled “Custom operator”](#custom-operator) Function pipelines use a single custom operator of `->`. This operator is used to apply and compose multiple functions. The `->` operator takes the inputs on the left of the operator, and applies the operation on the right of the operator. To put it more plainly, you can think of it as "do the next thing." A typical function pipeline could look something like this: ```sql SELECT device_id, toolkit_experimental.timevector(ts, val) -> toolkit_experimental.sort() -> toolkit_experimental.delta() -> toolkit_experimental.abs() -> toolkit_experimental.sum() as volatility FROM measurements WHERE ts >= now() - '1 day'::interval GROUP BY device_id; ``` While it might look at first glance as though `timevector(ts, val)` operation is an argument to `sort()`, in a pipeline these are all regular function calls. Each of the calls can only operate on the things in their own parentheses, and don't know about anything to the left of them in the statement. Each of the functions in a pipeline returns a custom type that describes the function and its arguments, these are all pipeline elements. The `->` operator performs one of two different types of actions depending on the types on its right and left sides: * Applies a pipeline element to the left hand argument: performing the function described by the pipeline element on the incoming data type directly. * Compose pipeline elements into a combined element that can be applied at some point in the future. This is an optimization that allows you to nest elements to reduce the number of passes that are required. The operator determines the action to perform based on its left and right arguments. ### Pipeline elements [Section titled “Pipeline elements”](#pipeline-elements) There are two main types of pipeline elements: * Transforms change the contents of the `timevector`, returning the updated vector. * Finalizers finish the pipeline and output the resulting data. Transform elements take in a `timevector` and produce a `timevector`. They are the simplest element to compose, because they produce the same type. For example: ```sql SELECT device_id, toolkit_experimental.timevector(ts, val) -> toolkit_experimental.sort() -> toolkit_experimental.delta() -> toolkit_experimental.map($$ ($value^3 + $value^2 + $value * 2) $$) -> toolkit_experimental.lttb(100) FROM measurements ``` Finalizer elements end the `timevector` portion of a pipeline. They can produce an output in a specified format, or they can produce an aggregate of the `timevector`. For example, a finalizer element that produces an output: ```sql SELECT device_id, toolkit_experimental.timevector(ts, val) -> toolkit_experimental.sort() -> toolkit_experimental.delta() -> toolkit_experimental.unnest() FROM measurements ``` Or a finalizer element that produces an aggregate: ```sql SELECT device_id, toolkit_experimental.timevector(ts, val) -> toolkit_experimental.sort() -> toolkit_experimental.delta() -> toolkit_experimental.time_weight() FROM measurements ``` The third type of pipeline elements are aggregate accessors and mutators. These work on a `timevector` in a pipeline, but they also work in regular aggregate queries. An example of using these in a pipeline: ```sql SELECT percentile_agg(val) -> toolkit_experimental.approx_percentile(0.5) FROM measurements ``` ## Transform elements [Section titled “Transform elements”](#transform-elements) Transform elements take a `timevector`, and produce a `timevector`. ### Vectorized math functions [Section titled “Vectorized math functions”](#vectorized-math-functions) Vectorized math function elements modify each `value` inside the `timevector` with the specified mathematical function. They are applied point-by-point and they produce a one-to-one mapping from the input to output `timevector`. Each point in the input has a corresponding point in the output, with its `value` transformed by the mathematical function specified. Elements are always applied left to right, so the order of operations is not taken into account even in the presence of explicit parentheses. This means for a `timevector` row `('2020-01-01 00:00:00+00', 20.0)`, this pipeline works: ```plaintext timevector('2021-01-01 UTC', 10) -> add(5) -> (mul(2) -> add(1)) ``` And this pipeline works in the same way: ```plaintext timevector('2021-01-01 UTC', 10) -> add(5) -> mul(2) -> add(1) ``` Both of these examples produce `('2020-01-01 00:00:00+00', 31.0)`. If multiple arithmetic operations are needed and precedence is important, consider using a [Lambda](#lambda-elements) instead. ### Unary mathematical functions [Section titled “Unary mathematical functions”](#unary-mathematical-functions) Unary mathematical function elements apply the corresponding mathematical function to each datapoint in the `timevector`, leaving the timestamp and ordering the same. The available elements are: | Element | Description | | --------- | -------------------------------------------------------------- | | `abs()` | Computes the absolute value of each value | | `cbrt()` | Computes the cube root of each value | | `ceil()` | Computes the first integer greater than or equal to each value | | `floor()` | Computes the first integer less than or equal to each value | | `ln()` | Computes the natural logarithm of each value | | `log10()` | Computes the base 10 logarithm of each value | | `round()` | Computes the closest integer to each value | | `sign()` | Computes +/-1 for each positive/negative value | | `sqrt()` | Computes the square root for each value | | `trunc()` | Computes only the integer portion of each value | Even if an element logically computes an integer, `timevectors` only deal with double precision floating point values, so the computed value is the floating point representation of the integer. For example: ```sql -- NOTE: the (pipeline -> unnest()).* allows for time, value columns to be produced without a subselect SELECT ( toolkit_experimental.timevector(time, value) -> toolkit_experimental.abs() -> toolkit_experimental.unnest()).* FROM (VALUES (TimestampTZ '2021-01-06 UTC', 0.0 ), ( '2021-01-01 UTC', 25.0 ), ( '2021-01-02 UTC', 0.10), ( '2021-01-04 UTC', -10.0 ), ( '2021-01-05 UTC', 3.3 ) ) as v(time, value); ``` The output for this example: ```sql time | value ------------------------+------- 2021-01-06 00:00:00+00 | 0 2021-01-01 00:00:00+00 | 25 2021-01-02 00:00:00+00 | 0.1 2021-01-04 00:00:00+00 | 10 2021-01-05 00:00:00+00 | 3.3 (5 rows) ``` ### Binary mathematical functions [Section titled “Binary mathematical functions”](#binary-mathematical-functions) Binary mathematical function elements run the corresponding mathematical function on the `value` in each point in the `timevector`, using the supplied number as the second argument of the function. The available elements are: | Element | Description | | ---------- | --------------------------------------------------------- | | `add(N)` | Computes each value plus `N` | | `div(N)` | Computes each value divided by `N` | | `logn(N)` | Computes the logarithm base `N` of each value | | `mod(N)` | Computes the remainder when each number is divided by `N` | | `mul(N)` | Computes each value multiplied by `N` | | `power(N)` | Computes each value taken to the `N` power | | `sub(N)` | Computes each value less `N` | These elements calculate `vector -> power(2)` by squaring all of the `values`, and `vector -> logn(3)` gives the log-base-3 of each `value`. For example: ```sql SELECT ( toolkit_experimental.timevector(time, value) -> toolkit_experimental.power(2) -> toolkit_experimental.unnest()).* FROM (VALUES (TimestampTZ '2021-01-06 UTC', 0.0 ), ( '2021-01-01 UTC', 25.0 ), ( '2021-01-02 UTC', 0.10), ( '2021-01-04 UTC', -10.0 ), ( '2021-01-05 UTC', 3.3 ) ) as v(time, value); ``` The output for this example: ```sql time | value ------------------------+---------------------- 2021-01-06 00:00:00+00 | 0 2021-01-01 00:00:00+00 | 625 2021-01-02 00:00:00+00 | 0.010000000000000002 2021-01-04 00:00:00+00 | 100 2021-01-05 00:00:00+00 | 10.889999999999999 (5 rows) ``` ### Compound transforms [Section titled “Compound transforms”](#compound-transforms) Mathematical transforms are applied only to the `value` in each point in a `timevector` and always produce one-to-one output `timevectors`. Compound transforms can involve both the `time` and `value` parts of the points in the `timevector`, and they are not necessarily one-to-one. One or more points in the input can be used to produce zero or more points in the output. So, where mathematical transforms always produce `timevectors` of the same length, compound transforms can produce larger or smaller `timevectors` as an output. #### Delta transforms [Section titled “Delta transforms”](#delta-transforms) A `delta()` transform calculates the difference between consecutive `values` in the `timevector`. The first point in the `timevector` is omitted as there is no previous value and it cannot have a `delta()`. Data should be sorted using the `sort()` element before passing into `delta()`. For example: ```sql SELECT ( toolkit_experimental.timevector(time, value) -> toolkit_experimental.sort() -> toolkit_experimental.delta() -> toolkit_experimental.unnest()).* FROM (VALUES (TimestampTZ '2021-01-06 UTC', 0.0 ), ( '2021-01-01 UTC', 25.0 ), ( '2021-01-02 UTC', 0.10), ( '2021-01-04 UTC', -10.0 ), ( '2021-01-05 UTC', 3.3 ) ) as v(time, value); ``` The output for this example: ```sql time | value ------------------------+------- 2021-01-02 00:00:00+00 | -24.9 2021-01-04 00:00:00+00 | -10.1 2021-01-05 00:00:00+00 | 13.3 2021-01-06 00:00:00+00 | -3.3 (4 rows) ``` Note The first row of the output is missing, as there is no way to compute a delta without a previous value. #### Fill method transform [Section titled “Fill method transform”](#fill-method-transform) The `fill_to()` transform ensures that there is a point at least every `interval`, if there is not a point, it fills in the point using the method provided. The `timevector` must be sorted before calling `fill_to()`. The available fill methods are: | fill\_method | description | | ------------ | ------------------------------------------------------------------------------------------ | | LOCF | Last object carried forward, fill with last known value prior to the hole | | Interpolate | Fill the hole using a collinear point with the first known value on either side | | Linear | This is an alias for interpolate | | Nearest | Fill with the matching value from the closer of the points preceding or following the hole | For example: ```sql SELECT ( toolkit_experimental.timevector(time, value) -> toolkit_experimental.sort() -> toolkit_experimental.fill_to('1 day', 'LOCF') -> toolkit_experimental.unnest()).* FROM (VALUES (TimestampTZ '2021-01-06 UTC', 0.0 ), ( '2021-01-01 UTC', 25.0 ), ( '2021-01-02 UTC', 0.10), ( '2021-01-04 UTC', -10.0 ), ( '2021-01-05 UTC', 3.3 ) ) as v(time, value); ``` The output for this example: ```sql time | value ------------------------+------- 2021-01-01 00:00:00+00 | 25 2021-01-02 00:00:00+00 | 0.1 2021-01-03 00:00:00+00 | 0.1 2021-01-04 00:00:00+00 | -10 2021-01-05 00:00:00+00 | 3.3 2021-01-06 00:00:00+00 | 0 (6 rows) ``` #### Largest triangle three buckets (LTTB) transform [Section titled “Largest triangle three buckets (LTTB) transform”](#largest-triangle-three-buckets-lttb-transform) The largest triangle three buckets (LTTB) transform uses the LTTB graphical downsampling algorithm to downsample a `timevector` to the specified resolution while maintaining visual acuity. #### Sort transform [Section titled “Sort transform”](#sort-transform) The `sort()` transform sorts the `timevector` by time, in ascending order. This transform is ignored if the `timevector` is already sorted. For example: ```sql SELECT ( toolkit_experimental.timevector(time, value) -> toolkit_experimental.sort() -> toolkit_experimental.unnest()).* FROM (VALUES (TimestampTZ '2021-01-06 UTC', 0.0 ), ( '2021-01-01 UTC', 25.0 ), ( '2021-01-02 UTC', 0.10), ( '2021-01-04 UTC', -10.0 ), ( '2021-01-05 UTC', 3.3 ) ) as v(time, value); ``` The output for this example: ```sql time | value ------------------------+------- 2021-01-01 00:00:00+00 | 25 2021-01-02 00:00:00+00 | 0.1 2021-01-04 00:00:00+00 | -10 2021-01-05 00:00:00+00 | 3.3 2021-01-06 00:00:00+00 | 0 (5 rows) ``` ### Lambda elements [Section titled “Lambda elements”](#lambda-elements) The Lambda element functions use the TimescaleDB Toolkit's experimental Lambda syntax to transform a `timevector`. A Lambda is an expression that is applied to the elements of a `timevector`. It is written as a string, usually `$$`-quoted, containing the expression to run. For example: ```sql $$ let $is_relevant = $time > '2021-01-01't and $time < '2021-10-14't; let $is_significant = abs(round($value)) >= 0; $is_relevant and $is_significant $$ ``` A Lambda expression can be constructed using these components: * **Variable declarations** such as `let $foo = 3; $foo * $foo`. Variable declarations end with a semicolon. All Lambdas must end with an expression, this does not have a semicolon. Multiple variable declarations can follow one another, for example: `let $foo = 3; let $bar = $foo * $foo; $bar * 10` * **Variable names** such as `$foo`. They must start with a `$` symbol. The variables `$time` and `$value` are reserved; they refer to the time and value of the point in the vector the Lambda expression is being called on. * **Function calls** such as `abs($foo)`. Most mathematical functions are supported. * **Binary operations** containing the arithmetic binary operators `and`, `or`, `=`, `!=`, `<`, `<=`, `>`, `>=`, `^`, `*`, `/`, `+`, and `-` are supported. * **Interval literals** are expressed with a trailing `i`. For example, `'1 day'i`. Except for the trailing `i`, these follow the PostgreSQL `INTERVAL` input format. * **Time literals** such as `'2021-01-02 03:00:00't` expressed with a trailing `t`. Except for the trailing `t` these follow the PostgreSQL `TIMESTAMPTZ` input format. * **Number literals** such as `42`, `0.0`, `-7`, or `1e2`. Lambdas follow a grammar that is roughly equivalent to EBNF. For example: ```txt Expr = ('let' Variable '=' Tuple ';')* Tuple Tuple = Binops (',' Binops)* Binops = Unaryops (Binop Unaryops)* UnaryOps = ('-' | 'not') UnaryOps | Term Term = Variable | Time | Interval | Number | Function | '(' Expr ')' Function = FunctionName '(' (Binops ',')* ')' Variable = ? described above ? Time = ? described above ? Interval = ? described above ? Number = ? described above ? ``` #### Map Lambda [Section titled “Map Lambda”](#map-lambda) The `map()` Lambda maps each element of the `timevector`. This Lambda must return either a `DOUBLE PRECISION`, where only the values of each point in the `timevector` is altered, or a `(TIMESTAMPTZ, DOUBLE PRECISION)`, where both the times and values are changed. An example of the `map()` Lambda with a `DOUBLE PRECISION` return: ```sql SELECT ( toolkit_experimental.timevector(time, value) -> toolkit_experimental.map($$ $value + 1 $$) -> toolkit_experimental.unnest()).* FROM (VALUES (TimestampTZ '2021-01-06 UTC', 0.0 ), ( '2021-01-01 UTC', 25.0 ), ( '2021-01-02 UTC', 0.10), ( '2021-01-04 UTC', -10.0 ), ( '2021-01-05 UTC', 3.3 ) ) as v(time, value); ``` The output for this example: ```sql time | value ------------------------+------- 2021-01-06 00:00:00+00 | 1 2021-01-01 00:00:00+00 | 26 2021-01-02 00:00:00+00 | 1.1 2021-01-04 00:00:00+00 | -9 2021-01-05 00:00:00+00 | 4.3 (5 rows) ``` An example of the `map()` Lambda with a `(TIMESTAMPTZ, DOUBLE PRECISION)` return: ```sql SELECT ( toolkit_experimental.timevector(time, value) -> toolkit_experimental.map($$ ($time + '1day'i, $value * 2) $$) -> toolkit_experimental.unnest()).* FROM (VALUES (TimestampTZ '2021-01-06 UTC', 0.0 ), ( '2021-01-01 UTC', 25.0 ), ( '2021-01-02 UTC', 0.10), ( '2021-01-04 UTC', -10.0 ), ( '2021-01-05 UTC', 3.3 ) ) as v(time, value); ``` The output for this example: ```sql time | value ------------------------+------- 2021-01-07 00:00:00+00 | 0 2021-01-02 00:00:00+00 | 50 2021-01-03 00:00:00+00 | 0.2 2021-01-05 00:00:00+00 | -20 2021-01-06 00:00:00+00 | 6.6 (5 rows) ``` #### Filter Lambda [Section titled “Filter Lambda”](#filter-lambda) The `filter()` Lambda filters a `timevector` based on a Lambda expression that returns `true` for every point that should stay in the `timevector` timeseries, and `false` for every point that should be removed. For example: ```sql SELECT ( toolkit_experimental.timevector(time, value) -> toolkit_experimental.filter($$ $time != '2021-01-01't AND $value > 0 $$) -> toolkit_experimental.unnest()).* FROM (VALUES (TimestampTZ '2021-01-06 UTC', 0.0 ), ( '2021-01-01 UTC', 25.0 ), ( '2021-01-02 UTC', 0.10), ( '2021-01-04 UTC', -10.0 ), ( '2021-01-05 UTC', 3.3 ) ) as v(time, value); ``` The output for this example: ```sql time | value ------------------------+------- 2021-01-02 00:00:00+00 | 0.1 2021-01-05 00:00:00+00 | 3.3 (2 rows) ``` ## Finalizer elements [Section titled “Finalizer elements”](#finalizer-elements) Finalizer elements complete the function pipeline, and output a value or an aggregate. ### Output element [Section titled “Output element”](#output-element) You can finalize a pipeline with a `timevector` output element. These are used at the end of a pipeline to return a `timevector`. This can be useful if you need to use them in another pipeline later on. The two types of output are: * `unnest()`, which returns a set of `(TimestampTZ, DOUBLE PRECISION)` pairs. * `materialize()`, which forces the pipeline to materialize a `timevector`. This blocks any optimizations that lazily materialize a `timevector`. ### Aggregate output elements [Section titled “Aggregate output elements”](#aggregate-output-elements) These elements take a `timevector` and run the corresponding aggregate over it to produce a result. The possible elements are: * `average()` * `integral()` * `counter_agg()` * `hyperloglog()` * `stats_agg()` * `sum()` * `num_vals()` An example of an aggregate output using `num_vals()`: ```sql SELECT toolkit_experimental.timevector(time, value) -> toolkit_experimental.num_vals() FROM (VALUES (TimestampTZ '2021-01-06 UTC', 0.0 ), ( '2021-01-01 UTC', 25.0 ), ( '2021-01-02 UTC', 0.10), ( '2021-01-04 UTC', -10.0 ), ( '2021-01-05 UTC', 3.3 ) ) as v(time, value); ``` The output for this example: ```sql ?column? ---------- 5 (1 row) ``` An example of an aggregate output using `stats_agg()`: ```sql SELECT toolkit_experimental.timevector(time, value) -> toolkit_experimental.stats_agg() -> toolkit_experimental.stddev() FROM (VALUES (TimestampTZ '2021-01-06 UTC', 0.0 ), ( '2021-01-01 UTC', 25.0 ), ( '2021-01-02 UTC', 0.10), ( '2021-01-04 UTC', -10.0 ), ( '2021-01-05 UTC', 3.3 ) ) as v(time, value); ``` The output for this example: ```sql ?column? -------------------- 12.924666339987272 (1 row) ``` ## Aggregate accessors and mutators [Section titled “Aggregate accessors and mutators”](#aggregate-accessors-and-mutators) Aggregate accessors and mutators work in function pipelines in the same way as they do in other aggregates. You can use them to get a value from the aggregate part of a function pipeline. For example: ```sql SELECT device_id, timevector(ts, val) -> sort() -> delta() -> stats_agg() -> variance() FROM measurements ``` When you use them in a pipeline instead of standard function accessors and mutators, they can make the syntax clearer by getting rid of nested functions. For example, the nested syntax looks like this: ```sql SELECT approx_percentile(0.5, percentile_agg(val)) FROM measurements ``` Using a function pipeline with the `->` operator instead looks like this: ```sql SELECT percentile_agg(val) -> approx_percentile(0.5) FROM measurements ``` ### Counter aggregates [Section titled “Counter aggregates”](#counter-aggregates) Counter aggregates handle resetting counters. Counters are a common type of metric in application performance monitoring and metrics. All values have resets accounted for. These elements must have a `CounterSummary` to their left when used in a pipeline, from a `counter_agg()` aggregate or pipeline element. The available counter aggregate functions are: | Element | Description | | -------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | | `counter_zero_time()` | The time at which the counter value is predicted to have been zero based on the least squares fit of the points input to the `CounterSummary` (x intercept) | | `corr()` | The correlation coefficient of the least squares fit line of the adjusted counter value | | `delta()` | Computes the last - first value of the counter | | `extrapolated_delta(method)` | Computes the delta extrapolated using the provided method to bounds of range. Bounds must have been provided in the aggregate or a `with_bounds` call. | | `idelta_left()`/`idelta_right()` | Computes the instantaneous difference between the second and first points (left) or last and next-to-last points (right) | | `intercept()` | The y-intercept of the least squares fit line of the adjusted counter value | | `irate_left()`/`irate_right()` | Computes the instantaneous rate of change between the second and first points (left) or last and next-to-last points (right) | | `num_changes()` | Number of times the counter changed values | | `num_elements()` | Number of items - any with the exact same time have been counted only once | | `num_resets()` | Number of times the counter reset | | `slope()` | The slope of the least squares fit line of the adjusted counter value | | `with_bounds(range)` | Applies bounds using the `range` (a `TSTZRANGE`) to the `CounterSummary` if they weren't provided in the aggregation step | ### Percentile approximation [Section titled “Percentile approximation”](#percentile-approximation) Percentile approximation aggregate accessors are used to approximate percentiles. Currently, only accessors are implemented for `percentile_agg` and `uddsketch` based aggregates. We have not yet implemented the pipeline aggregate for percentile approximation with `tdigest`. | Element | Description | | --------------------------- | ---------------------------------------------------------- | | `approx_percentile(p)` | The approximate value at percentile `p` | | `approx_percentile_rank(v)` | The approximate percentile a value `v` would fall in | | `error()` | The maximum relative error guaranteed by the approximation | | `mean()` | The exact average of the input values | | `num_vals()` | The number of input values | ### Statistical aggregates [Section titled “Statistical aggregates”](#statistical-aggregates) Statistical aggregate accessors add support for common statistical aggregates. These allow you to compute and `rollup()` common statistical aggregates like `average` and `stddev`, more advanced aggregates like `skewness`, and two-dimensional aggregates like `slope` and `covariance`. Because there are both single-dimensional and two-dimensional versions of these, the accessors can have multiple forms. For example, `average()` calculates the average on a single-dimension aggregate, while `average_y()` and `average_x()` calculate the average on each of two dimensions. The available statistical aggregates are: | Element | Description | | ------------------------------------------------------------ | ------------------------------------------------------------------------------------------- | | `average()`/`average_y()`/`average_x()` | The average of the values | | `corr()` | The correlation coefficient of the least squares fit line | | `covariance(method)` | The covariance of the values using either `population` or `sample` method | | `determination_coeff()` | The determination coefficient (or R squared) of the values | | `kurtosis(method)`/`kurtosis_y(method)`/`kurtosis_x(method)` | The kurtosis (fourth moment) of the values using either the `population` or `sample` method | | `intercept()` | The intercept of the least squares fit line | | `num_vals()` | The number of values seen | | `skewness(method)`/`skewness_y(method)`/`skewness_x(method)` | The skewness (third moment) of the values using either the `population` or `sample` method | | `slope()` | The slope of the least squares fit line | | `stddev(method)`/`stddev_y(method)`/`stddev_x(method)` | The standard deviation of the values using either the `population` or `sample` method | | `sum()` | The sum of the values | | `variance(method)`/`variance_y(method)`/`variance_x(method)` | The variance of the values using either the `population` or `sample` method | | `x_intercept()` | The x intercept of the least squares fit line | ### Time-weighted averages aggregates [Section titled “Time-weighted averages aggregates”](#time-weighted-averages-aggregates) The `average()` accessor can be called on the output of a `time_weight()`. For example: ```sql SELECT time_weight('Linear', ts, val) -> average() FROM measurements; ``` ### Approximate count distinct aggregates [Section titled “Approximate count distinct aggregates”](#approximate-count-distinct-aggregates) This is an approximation for distinct counts. The `distinct_count()` accessor can be called on the output of a `hyperloglog()`. For example: ```sql SELECT hyperloglog(device_id) -> distinct_count() FROM measurements; ``` ## Formatting timevectors [Section titled “Formatting timevectors”](#formatting-timevectors) You can turn a timevector into a formatted text representation. There are two functions for turning a timevector to text: * [`to_text`](#to_text), which allows you to specify the template * [`to_plotly`](#to_plotly), which outputs a format suitable for use with the [Plotly JSON chart schema](https://plotly.com/chart-studio-help/json-chart-schema/) ### `to_text` [Section titled “to\_text”](#to_text) ```sql toolkit_experimental.to_text( timevector(time, value), format_string ) ``` This function produces a text representation, formatted according to the `format_string`. The format string can use any valid Tera template syntax, and it can include any of the built-in variables: * `TIMES`: All the times in the timevector, as an array * `VALUES`: All the values in the timevector, as an array * `TIMEVALS`: All the time-value pairs in the timevector, formatted as `{"time": $TIME, "val": $VAL}`, as an array For example, given this table of data: ```sql CREATE TABLE data(time TIMESTAMPTZ, value DOUBLE PRECISION); INSERT INTO data VALUES ('2020-1-1', 30.0), ('2020-1-2', 45.0), ('2020-1-3', NULL), ('2020-1-4', 55.5), ('2020-1-5', 10.0); ``` You can use a format string with `TIMEVALS` to produce the following text: ```sql SELECT toolkit_experimental.to_text( timevector(time, value), '{{TIMEVALS}}' ) FROM data; ``` ```plaintext [{"time": "2020-01-01 00:00:00+00", "val": 30}, {"time": "2020-01-02 00:00:00+00", "val": 45}, {"time": "2020-01-03 00:00:00+00", "val": null}, {"time": "2020-01-04 00:00:00+00", "val": 55.5}, {"time": "2020-01-05 00:00:00+00", "val": 10} ] ``` Or you can use a format string with `TIMES` and `VALUES` to produce the following text: ```sql SELECT toolkit_experimental.to_text( timevector(time,value), '{"times": {{ TIMES }}, "vals": {{ VALUES }}}' ) FROM data ``` ```plaintext {"times": ["2020-01-01 00:00:00+00","2020-01-02 00:00:00+00","2020-01-03 00:00:00+00","2020-01-04 00:00:00+00","2020-01-05 00:00:00+00"], "vals": ["30","45","null","55.5","10"]} ``` ### `to_plotly` [Section titled “to\_plotly”](#to_plotly) This function produces a text representation, formatted for use with Plotly. For example, given this table of data: ```sql CREATE TABLE data(time TIMESTAMPTZ, value DOUBLE PRECISION); INSERT INTO data VALUES ('2020-1-1', 30.0), ('2020-1-2', 45.0), ('2020-1-3', NULL), ('2020-1-4', 55.5), ('2020-1-5', 10.0); ``` You can produce the following Plotly-compatible text: ```sql SELECT toolkit_experimental.to_plotly( timevector(time, value) ) FROM data; ``` ```plaintext {"times": ["2020-01-01 00:00:00+00","2020-01-02 00:00:00+00","2020-01-03 00:00:00+00","2020-01-04 00:00:00+00","2020-01-05 00:00:00+00"], "vals": ["30","45","null","55.5","10"]} ``` ## All function pipeline elements [Section titled “All function pipeline elements”](#all-function-pipeline-elements) This table lists all function pipeline elements in alphabetical order: | Element | Category | Output | | --------------------------- | ------------------- | -------------------------------------------------- | | `abs()` | Unary Mathematical | `timevector` pipeline | | `add(val DOUBLE PRECISION)` | Binary Mathematical | `timevector` pipeline | | `average()` | Aggregate Finalizer | DOUBLE PRECISION | | `cbrt()` | Unary Mathematical | `timevector` pipeline | | `ceil()` | Unary Mathematical | `timevector` pipeline | | `counter_agg()` | Aggregate Finalizer | `CounterAgg` | | `delta()` | Compound | `timevector` pipeline | | `div` | Binary Mathematical | `timevector` pipeline | | `fill_to` | Compound | `timevector` pipeline | | `filter` | Lambda | `timevector` pipeline | | `floor` | Unary Mathematical | `timevector` pipeline | | `hyperloglog` | Aggregate Finalizer | HyperLogLog | | `ln` | Unary Mathematical | `timevector` pipeline | | `log10` | Unary Mathematical | `timevector` pipeline | | `logn` | Binary Mathematical | `timevector` pipeline | | `lttb` | Compound | `timevector` pipeline | | `map` | Lambda | `timevector` pipeline | | `materialize` | Output | `timevector` pipeline | | `mod` | Binary Mathematical | `timevector` pipeline | | `mul` | Binary Mathematical | `timevector` pipeline | | `num_vals` | Aggregate Finalizer | BIGINT | | `power` | Binary Mathematical | `timevector` pipeline | | `round` | Unary Mathematical | `timevector` pipeline | | `sign` | Unary Mathematical | `timevector` pipeline | | `sort` | Compound | `timevector` pipeline | | `sqrt` | Unary Mathematical | `timevector` pipeline | | `stats_agg` | Aggregate Finalizer | StatsSummary1D | | `sub` | Binary Mathematical | `timevector` pipeline | | `sum` | Aggregate Finalizer | `timevector` pipeline | | `trunc` | Unary Mathematical | `timevector` pipeline | | `unnest` | Output | `TABLE (time TIMESTAMPTZ, value DOUBLE PRECISION)` | # Overview > Handle missing data in time-series queries Most analytics start by rolling raw events into fixed windows with `time_bucket()`. That works until a window is empty because sampling was irregular or a collector failed. Gapfilling inserts placeholder rows so timelines stay contiguous; you then interpolate or carry values forward as needed. * For more information about how gapfilling works, read the [gapfilling blog](https://www.tigerdata.com/blog/sql-functions-for-time-series-analysis). * For more information about gapfilling and interpolation API calls, see the [hyperfunction API documentation](/docs/reference/timescaledb/hyperfunctions/time_bucket_gapfill/time_bucket_gapfill). # Last observation carried forward > Fill gaps in your data by carrying the last observation forward **LOCF** (last observation carried forward) fills gaps by repeating the last known value until the next sample arrives. Use it when a stepped or held reading matches your signal better than drawing a straight line between points. For more information about gapfilling and interpolation API calls, see the [hyperfunction API documentation](/docs/reference/timescaledb/hyperfunctions/time_bucket_gapfill/locf). # Time bucket gapfill > Use gapfilling to create contiguous time buckets and fill missing data with interpolation or LOCF Sometimes data sorted into time buckets can have gaps. This can happen if you have irregular sampling intervals, or you have experienced an outage of some sort. If you have a time bucket that has no data at all, the average returned from the time bucket is NULL, which could cause problems. You can use a gapfilling function to create additional rows of data in any gaps, ensuring that the returned rows are in chronological order, and contiguous. The time bucket gapfill function creates a contiguous set of time buckets but does not fill the rows with data. You can create data for the new rows using another function, such as last observation carried forward (LOCF), or interpolation. For more information about gapfilling and interpolation API calls, see the [hyperfunction API documentation](/docs/reference/timescaledb/hyperfunctions/time_bucket_gapfill/time_bucket_gapfill). # Heartbeat aggregation > Analyze intermittent or irregular time-series data to determine system health over a given interval Given a series of timestamped health checks, it can be tricky to determine the overall health of a system over a given interval. PostgreSQL provides window functions that you use to get a sense of where unhealthy gaps are, but they can be somewhat awkward to use efficiently. This is one of the many cases where hyperfunctions provide an efficient, simple solution for a frequently occurring problem. Heartbeat aggregation helps analyze event-based time-series data with intermittent or irregular signals. This example uses the [SustData public dataset](https://osf.io/2ac8q/). This dataset tracks the power usage of a small number of apartments and houses over four different deployment intervals. The data is collected in one-minute samples from each unit. When you have loaded the data into hypertables, you can create a materialized view containing weekly heartbeat aggregates for each of the units: ```sql CREATE MATERIALIZED VIEW weekly_heartbeat AS SELECT time_bucket('1 week', tmstp) as week, iid as unit, deploy, heartbeat_agg(tmstp, time_bucket('1w', tmstp), '1w', '2m') FROM power_samples GROUP BY 1,2,3; ``` The heartbeat aggregate takes four parameters: the timestamp column, the start of the interval, the length of the interval, and how long the aggregate is considered live after each timestamp. This example uses 2 minutes as the heartbeat lifetime to give some tolerance for small gaps. You can use this data to see when you're receiving data for a particular unit. This example rolls up the weekly aggregates into a single aggregate, and then views the live ranges: ```sql SELECT live_ranges(rollup(heartbeat_agg)) FROM weekly_heartbeat WHERE unit = 17; ``` ```plaintext live_ranges ----------------------------------------------------- ("2010-09-18 00:00:00+00","2011-03-27 01:01:50+00") ("2011-03-27 03:00:52+00","2011-07-03 00:01:00+00") ("2011-07-05 00:00:00+00","2011-08-21 00:01:00+00") ("2011-08-22 00:00:00+00","2011-08-25 00:01:00+00") ("2011-08-27 00:00:00+00","2011-09-06 00:01:00+00") ("2011-09-08 00:00:00+00","2011-09-29 00:01:00+00") ("2011-09-30 00:00:00+00","2011-10-04 00:01:00+00") ("2011-10-05 00:00:00+00","2011-10-17 00:01:00+00") ("2011-10-19 00:00:00+00","2011-11-09 00:01:00+00") ("2011-11-10 00:00:00+00","2011-11-14 00:01:00+00") ("2011-11-15 00:00:00+00","2011-11-18 00:01:00+00") ("2011-11-20 00:00:00+00","2011-11-23 00:01:00+00") ("2011-11-24 00:00:00+00","2011-12-01 00:01:00+00") ("2011-12-02 00:00:00+00","2011-12-12 00:01:00+00") ("2011-12-13 00:00:00+00","2012-01-12 00:01:00+00") ("2012-01-13 00:00:00+00","2012-02-03 00:01:00+00") ("2012-02-04 00:00:00+00","2012-02-10 00:01:00+00") ("2012-02-11 00:00:00+00","2012-03-25 01:01:50+00") ("2012-03-25 03:00:51+00","2012-04-11 00:01:00+00") ``` You can construct more elaborate queries. For example, to return the 5 units with the lowest uptime during the third deployment: ```sql SELECT unit, uptime(rollup(heartbeat_agg)) FROM weekly_heartbeat WHERE deploy = 3 GROUP BY unit ORDER BY uptime LIMIT 5; ``` ```plaintext unit | uptime ------+------------------- 31 | 203 days 22:05:00 34 | 222 days 22:05:00 32 | 222 days 22:05:00 35 | 222 days 22:05:00 30 | 222 days 22:05:00 ``` Combine aggregates from different units to get the combined coverage. This example queries the interval where any part of a deployment was active: ```sql SELECT deploy, live_ranges(rollup(heartbeat_agg)) FROM weekly_heartbeat group by deploy order by deploy; ``` ```plaintext deploy | live_ranges --------+----------------------------------------------------- 1 | ("2010-07-29 00:00:00+00","2010-11-26 00:01:00+00") 2 | ("2010-11-25 00:00:00+00","2011-03-27 01:01:59+00") 2 | ("2011-03-27 03:00:00+00","2012-03-25 01:01:59+00") 2 | ("2012-03-25 03:00:26+00","2012-04-17 00:01:00+00") 2 | ("2012-04-20 00:00:00+00","2012-04-21 00:01:00+00") 2 | ("2012-05-11 00:00:00+00","2012-05-13 00:01:00+00") 2 | ("2013-02-20 00:00:00+00","2013-02-21 00:01:00+00") 3 | ("2012-08-01 00:00:01+00","2013-03-31 01:01:16+00") 3 | ("2013-03-31 03:00:03+00","2013-05-22 00:01:00+00") 4 | ("2013-07-31 00:00:00+00","2014-03-30 01:01:49+00") 4 | ("2014-03-30 03:00:01+00","2014-04-25 00:01:00+00") ``` Then use this data to make observations and draw conclusions: * The second deployment had a lot more problems than the other ones. * There were some readings from February 2013 that were incorrectly categorized as a second deployment. * The timestamps are given in a local time without time zone, resulting in some missing hours around springtime daylight savings time changes. For more information about heartbeat aggregation API calls, see the [hyperfunction API documentation](/docs/reference/toolkit/state-tracking/heartbeat_agg/heartbeat_agg). # Approximate count distincts with Hyperloglog > Use Hyperloglog to estimate cardinality of large datasets with reduced memory consumption and improved compute time Cardinality on huge datasets is expensive when you must sort every distinct value. Hyperloglog and related aggregates estimate distinct counts with bounded memory so queries stay interactive on large tables and continuous aggregates. ## Approximate count distincts [Section titled “Approximate count distincts”](#approximate-count-distincts) Approximate count distincts are typically used to find the number of unique values, or cardinality, in a large dataset. When you calculate cardinality in a dataset, the time it takes to process the query is proportional to how large the dataset is. So if you wanted to find the cardinality of a dataset that contained only 20 entries, the calculation would be very fast. Finding the cardinality of a dataset that contains 20 million entries, however, can take a significant amount of time and compute resources. Approximate count distincts do not calculate the exact cardinality of a dataset, but rather estimate the number of unique values, to reduce memory consumption and improve compute time by avoiding spilling the intermediate results to the secondary storage. ## Hyperloglog [Section titled “Hyperloglog”](#hyperloglog) Hyperloglog is typically used to find the cardinality of very large datasets. If you want to find the number of unique values, or cardinality, in a dataset, the time it takes to process this query is proportional to how large the dataset is. So if you wanted to find the cardinality of a dataset that contained only 20 entries, the calculation would be very fast. Finding the cardinality of a dataset that contains 20 million entries, however, can take a significant amount of time and compute resources. Hyperloglog does not calculate the exact cardinality of a dataset, but rather estimates the number of unique values. It does this by converting the original data into a hash of random numbers that represents the cardinality of the dataset. This is not a perfect calculation of the cardinality, but it is usually within a margin of error of 2%. The benefit of hyperloglog on time-series data is that it can continue to calculate the approximate cardinality of a dataset as it changes over time. It does this by adding an entry to the hyperloglog hash as new data is retrieved, rather than recalculating the result for the entire dataset every time it is needed. This makes it an ideal candidate for using with continuous aggregates. For more information about approximate count distinct API calls, see the [hyperfunction API documentation](/docs/reference/toolkit/approximate-count-distinct/hyperloglog). # Overview > Approximate percentiles on large datasets without requiring all the data Exact percentiles need the full ordered set, which gets expensive on large or streaming datasets. Approximate percentiles trade a tiny error budget for bounded CPU and memory, and they pair well with continuous aggregates. In general, percentiles are useful for understanding the distribution of data. The fiftieth percentile is the point at which half of your data is greater and half is lesser. The tenth percentile is the point at which 90% of the data is greater, and 10% is lesser. The ninety-ninth percentile is the point at which 1% is greater, and 99% is lesser. The fiftieth percentile, or median, is often a more useful measure than the average, especially when your data contains outliers. Outliers can dramatically change the average, but do not affect the median as much. For example, if you have three rooms in your house and two of them are 40℉ (4℃) and one is 130℉ (54℃), the average room temperature is 70℉ (21℃), which doesn't tell you much. However, the fiftieth percentile temperature is 40℉ (4℃), which tells you that at least half your rooms are at refrigerator temperatures (also, you should probably get your heating checked!) Percentiles are sometimes avoided because calculating them requires more CPU and memory than an average or other aggregate measures. This is because an exact computation of the percentile needs the full dataset as an ordered list. TimescaleDB uses approximation algorithms to calculate a percentile without requiring all of the data. This also makes them more compatible with continuous aggregates. By default, TimescaleDB uses `uddsketch`, but you can also choose to use `tdigest`. For more information about these algorithms, see the [advanced aggregation methods](/docs/build/data-management/hyperfunctions/percentile-approx/advanced-agg) documentation. Note Technically, a percentile divides a group into 100 equally sized pieces, while a quantile divides a group into an arbitrary number of pieces. Because we don't always use exactly 100 buckets, "quantile" is the more technically correct term in this case. However, we use the word "percentile" because it's a more common word for this type of function. * For more information about how percentile approximation works, read the [percentile approximation blog](https://tigerdata.com/blog/how-percentile-approximation-works-and-why-its-more-useful-than-averages/). * For more information about percentile approximation API calls, see the [hyperfunction API documentation](/docs/reference/toolkit/percentile-approximation/uddsketch/percentile_agg). # Percentile approximation advanced aggregation methods > Understand the difference between uddsketch and tdigest to choose the right algorithm Both `uddsketch` and `tdigest` estimate percentiles without materializing the full sorted set, which keeps sketches small enough for continuous aggregates and high-cardinality series. The default is `uddsketch`; switch to `tdigest` when you care more about tail accuracy. The sections below spell out the trade-offs so you can pick deliberately. `uddsketch` is the default algorithm. It uses exponentially sized buckets to guarantee the approximation falls within a known error range, relative to the true discrete percentile. This algorithm offers the ability to tune the size and maximum error target of the sketch. `tdigest` buckets data more aggressively toward the center of the quantile range, giving it greater accuracy at the tails of the range, around 0.001 or 0.995. ## Choose the right algorithm [Section titled “Choose the right algorithm”](#choose-the-right-algorithm) Each algorithm has different features, which can make one better than another depending on your use case. Here are some of the differences to consider when choosing an algorithm: Before you begin, it is important to understand that the formal definition for a percentile is imprecise, and there are different methods for determining what the true percentile actually is. In PostgreSQL, given a target percentile `p`, [`percentile_disc`](https://www.postgresql.org/docs/current/functions-aggregate.html#FUNCTIONS-ORDEREDSET-TABLE) returns the smallest element of a set, so that `p` percent of the set is less than that element. However, [`percentile_cont`](https://www.postgresql.org/docs/current/functions-aggregate.html#FUNCTIONS-ORDEREDSET-TABLE) returns an interpolated value between the two nearest matches for `p`. In practice, the difference between these methods is very small but, if it matters to your use case, keep in mind that `tdigest` approximates the continuous percentile, while `uddsketch` provides an estimate of the discrete value. Think about the types of percentiles you're most interested in. `tdigest` is optimized for more accurate estimates at the extremes, and less accurate estimates near the median. If your workflow involves estimating ninety-ninth percentiles, then choose `tdigest`. If you're more concerned about getting highly accurate median estimates, choose `uddsketch`. The algorithms differ in the way they estimate data. `uddsketch` has a stable bucketing function, so it always returns the same percentile estimate for the same underlying data, regardless of how it is ordered or re-aggregated. On the other hand, `tdigest` builds up incremental buckets based on the average of nearby points, which can result in some subtle differences in estimates based on the same data unless the order and batching of the aggregation is strictly controlled, which is sometimes difficult to do in PostgreSQL. If stable estimates are important to you, choose `uddsketch`. Calculating precise error bars for `tdigest` can be difficult, especially when merging multiple sub-digests into a larger one. This can occur through summary aggregation, or parallelization of the normal point aggregate. If you need to tightly characterize your errors, choose `uddsketch`. However, because `uddsketch` uses exponential bucketing to provide a guaranteed relative error, it can cause some wildly varying absolute errors if the dataset covers a large range. For example, if the data is evenly distributed over the range `[1,100]`, estimates at the high end of the percentile range have about 100 times the absolute error of those at the low end of the range. This gets much more extreme if the data range is `[0,100]`. If having a stable absolute error is important to your use case, choose `tdigest`. While both algorithms are likely to get smaller and faster with future optimizations, `uddsketch` generally requires a smaller memory footprint than `tdigest`, and a correspondingly smaller disk footprint for any continuous aggregates. Regardless of the algorithm you choose, the best way to improve the accuracy of your percentile estimates is to increase the number of buckets, which is simpler to do with `uddsketch`. If your use case does not get a clear benefit from using `tdigest`, the default `uddsketch` is your best choice. For some more technical details and usage examples of the different algorithms, see the developer documentation for [uddsketch](https://github.com/timescale/timescaledb-toolkit/blob/main/docs/uddsketch.md) and [tdigest](https://github.com/timescale/timescaledb-toolkit/blob/main/docs/tdigest.md). # Approximate percentiles > Construct and run approximate percentage queries using uddsketch or tdigest This walkthrough uses a `response_times` table to show approximate percentiles end to end. TimescaleDB implements them with bounded-memory sketches instead of sorting every value, which keeps latency predictable on large datasets and continuous aggregates. By default, TimescaleDB Toolkit uses `uddsketch`, but you can also choose `tdigest`. For algorithm trade-offs, see [advanced aggregation methods](/docs/build/data-management/hyperfunctions/percentile-approx/advanced-agg). ## Run an approximate percentage query [Section titled “Run an approximate percentage query”](#run-an-approximate-percentage-query) In this procedure, we use an example table called `response_times` that contains information about how long a server takes to respond to API calls. 1. **Create a continuous aggregate for daily aggregates** ```sql CREATE MATERIALIZED VIEW response_times_daily WITH (timescaledb.continuous) AS SELECT time_bucket('1 day'::interval, ts) as bucket, percentile_agg(response_time_ms) FROM response_times GROUP BY 1; ``` 2. **Query the ninety-fifth percentile over the last 30 days** Re-aggregate the aggregate to get the last 30 days: ```sql SELECT approx_percentile(0.95, percentile_agg) as threshold FROM response_times_daily WHERE bucket >= time_bucket('1 day'::interval, now() - '30 days'::interval); ``` 3. **Create an alert** ```sql WITH t as (SELECT approx_percentile(0.95, percentile_agg(percentile_agg)) as threshold FROM response_times_daily WHERE bucket >= time_bucket('1 day'::interval, now() - '30 days'::interval)) SELECT count(*) FROM response_times WHERE ts > now()- '1 minute'::interval AND response_time_ms > (SELECT threshold FROM t); ``` For more information about percentile approximation API calls, see the [hyperfunction API documentation](/docs/reference/toolkit/percentile-approximation/uddsketch/percentile_agg). # Statistical aggregation > Use common statistical aggregates in window functions and continuous aggregates Window functions and continuous aggregates often need the same statistical moments (mean, stddev, kurtosis) in a composable aggregate. TimescaleDB wraps those patterns in `stats_agg` so you can reuse one state object across rollups instead of recomputing from scratch in plain PostgreSQL. This example calculates the average, standard deviation, and kurtosis of a value in the `measurements` table: ```sql SELECT time_bucket('10 min'::interval, ts), average(stats_agg(val)), stddev(stats_agg(val), 'pop'), kurtosis(stats_agg(val), 'pop') FROM measurements GROUP BY 1; ``` This uses a two-step aggregation process. The first step is an aggregation step (`stats_agg(val)`), which creates a machine-readable form of the aggregate. The second step is an accessor. The available accessors are `average`, `stddev`, and `kurtosis`. The accessors run final calculations and output the calculated value in a human-readable way. This makes it easier to construct your queries, because it distinguishes the parameters, and makes it clear which aggregates are being re-aggregated or rolled up. Additionally, because this query syntax is used in all TimescaleDB Toolkit queries, when you are used to it, you can use it to construct more and more complicated queries. A more complex example uses window functions to calculate tumbling window statistical aggregates. The statistical aggregate is first calculated over each minute in the subquery and then the `rolling` aggregate is used to re-aggregate it over each 15 minute period preceding. The accessors remain the same as the previous example: ```sql SELECT bucket, average(rolling(stats_agg) OVER fifteen_min), stddev(rolling(stats_agg) OVER fifteen_min, 'pop'), kurtosis(rolling(stats_agg) OVER fifteen_min, 'pop') FROM (SELECT time_bucket('1 min'::interval, ts) AS bucket, stats_agg(val) FROM measurements GROUP BY 1) AS stats WINDOW fifteen_min as (ORDER BY bucket ASC RANGE '15 minutes' PRECEDING); ``` For some more technical details and usage examples of the two-step aggregation method, see the [blog post on aggregates](https://www.tigerdata.com/blog/how-postgresql-aggregation-works-and-how-it-inspired-our-hyperfunctions-design) or the [developer documentation](https://github.com/timescale/timescaledb-toolkit/blob/main/docs/two-step_aggregation.md). ## 1D and 2D linear regression with statistical aggregates [Section titled “1D and 2D linear regression with statistical aggregates”](#1d-and-2d-linear-regression-with-statistical-aggregates) The `stats_agg` aggregate is available in two forms, a one-dimensional aggregate shown earlier in this section, and a two-dimensional aggregate. The two-dimensional aggregate takes in two variables `(Y, X)`, which are dependent and independent variables respectively. The two-dimensional aggregate performs all the same calculations on each individual variable as performing separate one-dimensional aggregates would, and additionally performs linear regression on the two variables. Accessors for one-dimensional values append a `_y` or `_x` to the name. For example: ```sql SELECT average_y(stats_agg(val2, val1)), -- equivalent to average(stats_agg(val2)) stddev_x(stats_agg(val2, val1)), -- equivalent to stddev(stats_agg(val1)) slope(stats_agg(val2, val1)) -- the slope of the least squares fit line of the values in val2 & val1 FROM measurements_multival; ``` For more information about statistical aggregation API calls, see the [hyperfunction API documentation](/docs/reference/toolkit/statistical-and-regression-analysis/stats_agg-one-variable/stats_agg). # Time-weighted averages and integrals > Use time-weighted averages and integrals for time-series data that is not evenly sampled When samples land at uneven intervals, a simple mean can overweight short spikes or underweight long plateaus. **Time-weighted averages** and integrals assign each reading weight proportional to how long it was in effect, so irregular sampling still reflects duration correctly. Evenly spaced series (for example every 30 seconds) rarely need this; irregular bursts or adaptive sampling do. For example, if you have a lot of ice cream in freezers, you need to make sure the ice cream stays within a 0-10℉ (-20 to -12℃) temperature range. The temperature in the freezer can vary if folks are opening and closing the door, but the ice cream only has a problem if the temperature is out of range for a long time. You can set your sensors in the freezer to sample every five minutes while the temperature is in range, and every 30 seconds while the temperature is out of range. If the results are generally stable, but with some quick moving transients, an average of all the data points weights the transient values too highly. A time weighted average weights each value by the duration over which it occurred based on the points around it, producing much more accurate results. Time weighted integrals are useful when you need a time-weighted sum of irregularly sampled data. For example, if you bill your users based on irregularly sampled CPU usage, you need to find the total area under the graph of their CPU usage. You can use a time-weighted integral to find the total CPU-hours used by a user over a given time period. * For more information about how time-weighted averages work, read the [time-weighted averages blog](https://www.tigerdata.com/blog/what-time-weighted-averages-are-and-why-you-should-care). * For more information about time-weighted average API calls, see the [hyperfunction API documentation](/docs/reference/toolkit/time_weight/time_weight). ## Run a time-weighted average query [Section titled “Run a time-weighted average query”](#run-a-time-weighted-average-query) Time weighted average in TimescaleDB is implemented as an aggregate that weights each value using last observation carried forward (LOCF), or linear interpolation. The aggregate is not parallelizable, but it is supported with [continuous aggregates](/docs/build/continuous-aggregates/create-a-continuous-aggregate). In this procedure, we use an example table called `freezer_temps` that contains data about internal freezer temperatures. 1. **Find the average and the time-weighted average** ```sql SELECT freezer_id, avg(temperature), average(time_weight('Linear', ts, temperature)) as time_weighted_average FROM freezer_temps GROUP BY freezer_id; ``` 2. **Use a time-weighted average in a window function** To determine if the freezer has been out of temperature range for more than 15 minutes at a time: ```sql SELECT *, average( time_weight('Linear', ts, temperature) OVER (PARTITION BY freezer_id ORDER BY ts RANGE '15 minutes'::interval PRECEDING ) ) as rolling_twa FROM freezer_temps ORDER BY freezer_id, ts; ``` For more information about time-weighted average API calls, see the [hyperfunction API documentation](/docs/reference/toolkit/time_weight/time_weight). # Perform advanced analytic queries > Calculate percentiles, moving averages, histograms, gap filling, and more with SQL Beyond basic aggregates, these patterns cover medians, moving windows, histograms, gap filling, and toolkit-backed shortcuts. Some use core PostgreSQL; others come from TimescaleDB and TimescaleDB Toolkit. ## Calculate the median and percentile [Section titled “Calculate the median and percentile”](#calculate-the-median-and-percentile) Use [`percentile_cont`](https://www.postgresql.org/docs/current/functions-aggregate.html#FUNCTIONS-ORDEREDSET-TABLE) to calculate percentiles. You can also use this function to look for the fiftieth percentile, or median. For example, to find the median temperature: ```sql SELECT percentile_cont(0.5) WITHIN GROUP (ORDER BY temperature) FROM conditions; ``` You can also use TimescaleDB Toolkit to find the [approximate percentile](/docs/reference/toolkit/percentile-approximation/uddsketch). ## Calculate the cumulative sum [Section titled “Calculate the cumulative sum”](#calculate-the-cumulative-sum) Use `sum(sum(column)) OVER(ORDER BY group)` to find the cumulative sum. For example: ```sql SELECT location, sum(sum(temperature)) OVER(ORDER BY location) FROM conditions GROUP BY location; ``` ## Calculate the moving average [Section titled “Calculate the moving average”](#calculate-the-moving-average) For a simple moving average, use the `OVER` windowing function over a number of rows, then compute an aggregation function over those rows. For example, to find the smoothed temperature of a device by averaging the ten most recent readings: ```sql SELECT time, AVG(temperature) OVER(ORDER BY time ROWS BETWEEN 9 PRECEDING AND CURRENT ROW) AS smooth_temp FROM conditions WHERE location = 'garage' and time > NOW() - INTERVAL '1 day' ORDER BY time DESC; ``` ## Calculate the increase in a value [Section titled “Calculate the increase in a value”](#calculate-the-increase-in-a-value) To calculate the increase in a value, you need to account for counter resets. Counter resets can occur if a host reboots or container restarts. This example finds the number of bytes sent, and takes counter resets into account: ```sql SELECT time, ( CASE WHEN bytes_sent >= lag(bytes_sent) OVER w THEN bytes_sent - lag(bytes_sent) OVER w WHEN lag(bytes_sent) OVER w IS NULL THEN NULL ELSE bytes_sent END ) AS "bytes" FROM net WHERE interface = 'eth0' AND time > NOW() - INTERVAL '1 day' WINDOW w AS (ORDER BY time) ORDER BY time ``` ## Calculate the rate of change [Section titled “Calculate the rate of change”](#calculate-the-rate-of-change) Like [increase](#calculate-the-increase-in-a-value), rate applies to a situation with monotonically increasing counters. If your sample interval is variable or you use different sampling intervals between different series, it is helpful to normalize the values to a common time interval to make the calculated values comparable. This example finds bytes per second sent, and takes counter resets into account: ```sql SELECT time, ( CASE WHEN bytes_sent >= lag(bytes_sent) OVER w THEN bytes_sent - lag(bytes_sent) OVER w WHEN lag(bytes_sent) OVER w IS NULL THEN NULL ELSE bytes_sent END ) / extract(epoch from time - lag(time) OVER w) AS "bytes_per_second" FROM net WHERE interface = 'eth0' AND time > NOW() - INTERVAL '1 day' WINDOW w AS (ORDER BY time) ORDER BY time ``` ## Calculate the delta [Section titled “Calculate the delta”](#calculate-the-delta) In many monitoring and IoT use cases, devices or sensors report metrics that do not change frequently, and any changes are considered anomalies. When you query for these changes in values over time, you usually do not want to transmit all the values, but only the values where changes were observed. This helps to minimize the amount of data sent. You can use a combination of window functions and subselects to achieve this. This example uses diffs to filter rows where values have not changed and only transmits rows where values have changed: ```sql SELECT time, data_value FROM ( SELECT time, data_value, data_value - LAG(data_value) OVER (ORDER BY time) AS diff FROM my_hypertable) ht WHERE diff IS NULL OR diff != 0; ``` ## Calculate the change in a metric within a group [Section titled “Calculate the change in a metric within a group”](#calculate-the-change-in-a-metric-within-a-group) To group your data by some field, and calculate the change in a metric within each group, use `LAG ... OVER (PARTITION BY ...)`. For example, given some weather data, calculate the change in temperature for each city: ```sql SELECT ts, city_name, temp_delta FROM ( SELECT ts, city_name, avg_temp - LAG(avg_temp) OVER (PARTITION BY city_name ORDER BY ts) as temp_delta FROM weather_metrics_daily ) AS temp_change WHERE temp_delta IS NOT NULL ORDER BY ts; ``` ## Group data into time buckets [Section titled “Group data into time buckets”](#group-data-into-time-buckets) The [`time_bucket`](/docs/reference/timescaledb/hyperfunctions/time-series-utilities/time_bucket) function in TimescaleDB extends the PostgreSQL [`date_bin`](https://www.postgresql.org/docs/current/functions-datetime.html#FUNCTIONS-DATETIME-BIN) function. Time bucket accepts arbitrary time intervals, as well as optional offsets, and returns the bucket start time. For example: ```sql SELECT time_bucket('5 minutes', time) AS five_min, avg(cpu) FROM metrics GROUP BY five_min ORDER BY five_min DESC LIMIT 12; ``` ## Get the first or last value in a column [Section titled “Get the first or last value in a column”](#get-the-first-or-last-value-in-a-column) The [`first`](/docs/reference/timescaledb/hyperfunctions/time-series-utilities/first) and [`last`](/docs/reference/timescaledb/hyperfunctions/time-series-utilities/last) functions allow you to get the value of one column as ordered by another. This is commonly used in an aggregation. These examples find the last element of a group: ```sql SELECT location, last(temperature, time) FROM conditions GROUP BY location; ``` ```sql SELECT time_bucket('5 minutes', time) five_min, location, last(temperature, time) FROM conditions GROUP BY five_min, location ORDER BY five_min DESC LIMIT 12; ``` ## Generate a histogram [Section titled “Generate a histogram”](#generate-a-histogram) The [`histogram`](/docs/reference/timescaledb/hyperfunctions/distribution-analysis/histogram) function allows you to generate a histogram of your data. This example defines a histogram with five buckets defined over the range 60 to 85. The generated histogram has seven bins; the first is for values below the minimum threshold of 60, the middle five bins are for values in the stated range and the last is for values above 85: ```sql SELECT location, COUNT(*), histogram(temperature, 60.0, 85.0, 5) FROM conditions WHERE time > NOW() - INTERVAL '7 days' GROUP BY location; ``` This query outputs data like this: ```bash location | count | histogram ------------+-------+------------------------- office | 10080 | {0,0,3860,6220,0,0,0} basement | 10080 | {0,6056,4024,0,0,0,0} garage | 10080 | {0,2679,957,2420,2150,1874,0} ``` ## Fill gaps in time-series data [Section titled “Fill gaps in time-series data”](#fill-gaps-in-time-series-data) You can display records for a selected time range, even if no data exists for part of the range. This is often called gap filling, and usually involves an operation to record a null value for any missing data. In this example, the trading data that includes a `time` timestamp, the `asset_code` being traded, the `price` of the asset, and the `volume` of the asset being traded is used. Create a query for the volume of the asset 'TIMS' being traded every day for the month of September: ```sql SELECT time_bucket('1 day', time) AS date, sum(volume) AS volume FROM trades WHERE asset_code = 'TIMS' AND time >= '2021-09-01' AND time < '2021-10-01' GROUP BY date ORDER BY date DESC; ``` This query outputs data like this: ```bash date | volume ------------------------+-------- 2021-09-29 00:00:00+00 | 11315 2021-09-28 00:00:00+00 | 8216 2021-09-27 00:00:00+00 | 5591 2021-09-26 00:00:00+00 | 9182 2021-09-25 00:00:00+00 | 14359 2021-09-22 00:00:00+00 | 9855 ``` You can see from the output that no records are included for 09-23, 09-24, or 09-30, because no trade data was recorded for those days. To include time records for each missing day, you can use the `time_bucket_gapfill` function, which generates a series of time buckets according to a given interval across a time range. In this example, the interval is one day, across the month of September: ```sql SELECT time_bucket_gapfill('1 day', time) AS date, sum(volume) AS volume FROM trades WHERE asset_code = 'TIMS' AND time >= '2021-09-01' AND time < '2021-10-01' GROUP BY date ORDER BY date DESC; ``` This query outputs data like this: ```bash date | volume ------------------------+-------- 2021-09-30 00:00:00+00 | 2021-09-29 00:00:00+00 | 11315 2021-09-28 00:00:00+00 | 8216 2021-09-27 00:00:00+00 | 5591 2021-09-26 00:00:00+00 | 9182 2021-09-25 00:00:00+00 | 14359 2021-09-24 00:00:00+00 | 2021-09-23 00:00:00+00 | 2021-09-22 00:00:00+00 | 9855 ``` You can also use the `time_bucket_gapfill` function to generate data points that also include timestamps. This can be useful for graphic libraries that require even null values to have a timestamp so that they can accurately draw gaps in a graph. In this example, you generate 1080 data points across the last two weeks, fill in the gaps with null values, and give each null value a timestamp: ```sql SELECT time_bucket_gapfill(INTERVAL '2 weeks' / 1080, time, now() - INTERVAL '2 weeks', now()) AS btime, sum(volume) AS volume FROM trades WHERE asset_code = 'TIMS' AND time >= now() - INTERVAL '2 weeks' AND time < now() GROUP BY btime ORDER BY btime; ``` This query outputs data like this: ```bash btime | volume ------------------------+---------- 2021-03-09 17:28:00+00 | 1085.25 2021-03-09 17:46:40+00 | 1020.42 2021-03-09 18:05:20+00 | 2021-03-09 18:24:00+00 | 1031.25 2021-03-09 18:42:40+00 | 1049.09 2021-03-09 19:01:20+00 | 1083.80 2021-03-09 19:20:00+00 | 1092.66 2021-03-09 19:38:40+00 | 2021-03-09 19:57:20+00 | 1048.42 2021-03-09 20:16:00+00 | 1063.17 2021-03-09 20:34:40+00 | 1054.10 2021-03-09 20:53:20+00 | 1037.78 ``` ### Fill gaps by carrying the last observation forward [Section titled “Fill gaps by carrying the last observation forward”](#fill-gaps-by-carrying-the-last-observation-forward) If your data collections only record rows when the actual value changes, your visualizations might still need all data points to properly display your results. In this situation, you can carry forward the last observed value to fill the gap. For example: ```sql SELECT time_bucket_gapfill(INTERVAL '5 min', time, now() - INTERVAL '2 weeks', now()) as "5min", meter_id, locf(avg(data_value)) AS data_value FROM my_hypertable WHERE time > now() - INTERVAL '2 weeks' AND meter_id IN (1,2,3,4) GROUP BY "5min", meter_id ``` ## Find the last point for each unique item [Section titled “Find the last point for each unique item”](#find-the-last-point-for-each-unique-item) You can find the last point for each unique item in your database. For example, the last recorded measurement from each IoT device, the last location of each item in asset tracking, or the last price of a security. The standard approach to minimize the amount of data to be searched for the last point is to use a time predicate to tightly bound the amount of time, or the number of chunks, to traverse. This method does not work unless all items have at least one record within the time range. A more robust method is to use a last point query to determine the last record for each unique item. In this example, useful for asset tracking or fleet management, you create a metadata table for each vehicle being tracked, and a second time-series table containing the vehicle's location at a given time: ```sql CREATE TABLE vehicles ( vehicle_id INTEGER PRIMARY KEY, vin_number CHAR(17), last_checkup TIMESTAMP ); ``` ```sql CREATE TABLE location ( time TIMESTAMP NOT NULL, vehicle_id INTEGER REFERENCES vehicles (vehicle_id), latitude FLOAT, longitude FLOAT ) WITH ( tsdb.hypertable, tsdb.segmentby='vehicle_id', tsdb.orderby='time DESC' ); ``` You can use the first table, which gives a distinct set of vehicles, to perform a `LATERAL JOIN` against the location table: ```sql SELECT data.* FROM vehicles v INNER JOIN LATERAL ( SELECT * FROM location l WHERE l.vehicle_id = v.vehicle_id ORDER BY time DESC LIMIT 1 ) AS data ON true ORDER BY v.vehicle_id, data.time DESC; ``` The output looks like this: ```bash time | vehicle_id | latitude | longitude ----------------------------+------------+-----------+------------- 2017-12-19 20:58:20.071784 | 72 | 40.753690 | -73.980340 2017-12-20 11:19:30.837041 | 156 | 40.729265 | -73.993611 2017-12-15 18:54:01.185027 | 231 | 40.350437 | -74.651954 ``` This approach requires keeping a separate table of distinct item identifiers or names. You can do this by using a foreign key from the hypertable to the metadata table, as shown in the `REFERENCES` definition in the example. The metadata table can be populated through business logic, for example when a vehicle is first registered with the system. Alternatively, you can dynamically populate it using a trigger when inserts or updates are performed against the hypertable. For example: ```sql CREATE OR REPLACE FUNCTION create_vehicle_trigger_fn() RETURNS TRIGGER LANGUAGE PLPGSQL AS $BODY$ BEGIN INSERT INTO vehicles VALUES(NEW.vehicle_id, NULL, NULL) ON CONFLICT DO NOTHING; RETURN NEW; END $BODY$; ``` ```sql CREATE TRIGGER create_vehicle_trigger BEFORE INSERT OR UPDATE ON location FOR EACH ROW EXECUTE FUNCTION create_vehicle_trigger_fn(); ``` You could also implement this functionality without a separate metadata table by performing a [loose index scan](https://wiki.postgresql.org/wiki/Loose_indexscan) over the `location` hypertable, although this requires more compute resources. Alternatively, you speed up your `SELECT DISTINCT` queries by structuring them so that TimescaleDB can use its [SkipScan](/docs/build/performance-optimization/skipscan) feature. # SELECT data > Query data from a hypertable using the standard SELECT command [`SELECT`](https://www.postgresql.org/docs/current/sql-select.html) against a hypertable works like any other table: filters, joins, aggregates, and window functions all apply. The examples below cover common time-window patterns. ## Basic query examples [Section titled “Basic query examples”](#basic-query-examples) Here are some examples of basic `SELECT` queries. Return the 100 most-recent entries in the table `conditions`. Order the rows from newest to oldest: ```sql SELECT * FROM conditions ORDER BY time DESC LIMIT 100; ``` Return the number of entries written to the table `conditions` in the last 12 hours: ```sql SELECT COUNT(*) FROM conditions WHERE time > NOW() - INTERVAL '12 hours'; ``` ### Advanced query examples [Section titled “Advanced query examples”](#advanced-query-examples) Here are some examples of more advanced `SELECT` queries. Get information about the weather conditions at each location, for each 15-minute period within the last 3 hours. Calculate the number of measurements taken, the maximum temperature, and the maximum humidity. Order the results by maximum temperature. This example uses the [`time_bucket`](/docs/reference/timescaledb/hyperfunctions/time-series-utilities/time_bucket) function to aggregate data into 15-minute buckets: ```sql SELECT time_bucket('15 minutes', time) AS fifteen_min, location, COUNT(*), MAX(temperature) AS max_temp, MAX(humidity) AS max_hum FROM conditions WHERE time > NOW() - INTERVAL '3 hours' GROUP BY fifteen_min, location ORDER BY fifteen_min DESC, max_temp DESC; ``` Count the number of distinct locations with air conditioning that have reported data in the last day: ```sql SELECT COUNT(DISTINCT conditions.location) FROM conditions JOIN locations ON conditions.location = locations.location WHERE locations.air_conditioning = True AND time > NOW() - INTERVAL '1 day'; ``` # Run your queries from Tiger Console > Choose the right tool to manage your data. Tiger Cloud offers Data view, SQL editor, and SQL assistant to better address your needs Tiger Cloud is still PostgreSQL under the hood, so your usual [clients and integrations](/docs/integrate) keep working. Tiger Console layers collaborative SQL notebooks and assistants on top for teams that want a shared workspace. Inside Tiger Console, pick the surface that matches how you like to work: * [Data view](#data-view): a rich experience powered by PopSQL. You can write queries with autocomplete, save them in folders, share them, create charts/dashboards, and much more. * [SQL assistant in Data view](#sql-assistant): write, fix, and organize SQL faster and more accurately. * [SQL editor](#sql-editor): a simple SQL editor that lets you run ad-hoc ephemeral queries. This is useful for quick one-off tasks like creating an index on a small table or inspecting `pg_stat_statements`. If you prefer the command line to Tiger Console editors, use [psql](/docs/integrate/query-administration/psql). ## Data view [Section titled “Data view”](#data-view) You use Data view in Tiger Console to write queries, visualize data, and share your results. ![Tiger Console Data mode](/docs/_astro/tiger-console-data-mode.CVB-yl27_1J98c8.webp) To start using Data view, open Tiger Console and toggle `Data view`. This includes the following tabs: * **Files**: browse all your personal and shared queries, dashboards, notebooks, and folders. * **Schemas**: view the schemas in your service. * **dbt**: [create dbt macros and models](https://popsql.com/dbt). * **SQL assistant**: [write, fix, and organize SQL](#sql-assistant) faster and more accurately using AI. Available features are: * **Real-time collaboration**: work with your team directly in Data view with live presence and multiple cursors. * **[Schema browser](https://docs.popsql.com/docs/schema)**: understand the structure of your service and see usage data on tables and columns. * **Autocomplete**: get suggestions as you type your queries. * **[Version history](https://docs.popsql.com/docs/version-history)**: access previous versions of a query from the built-in revision history, or connect to a git repo. * **[Charts](https://docs.popsql.com/docs/creating-charts)**: visualize data from inside the UI rather than switch to Sheets or Excel. * **[Schedules](https://docs.popsql.com/docs/scheduled-queries)**: automatically refresh queries and dashboards to create push alerts. * **[Query variables](https://docs.popsql.com/docs/query-variables)**: use Liquid to parameterize your queries or use `if` statements. * **Cross-platform support**: work from [Tiger Console](https://console.cloud.tigerdata.com/dashboard/services?popsql) or download the [desktop app](https://popsql.com/download) for macOS, Windows, and Linux. * **Easy connection**: connect to Tiger Cloud, PostgreSQL, Redshift, Snowflake, BigQuery, MySQL, SQL Server, [and more](https://popsql.com/connections). To connect to a service in Data view: 1. **Check your service is running correctly** In [Tiger Console](https://console.cloud.tigerdata.com/dashboard/services), check that your service is marked as `Running`: ![Tiger Console services list showing a running service](/docs/_astro/tiger-console-services-view.F-zo-hID_1rY9dD.webp) 2. **Connect to your service** Toggle [Data view](https://console.cloud.tigerdata.com/dashboard/services?popsql) in Tiger Console, then select a service in the connection drop-down: ![Select a connection in Tiger Console Data mode](/docs/_astro/tiger-console-data-mode-connection-dropdown.eS_sjlnj_1J2XiX.webp) 3. **Run a test query** Type `SELECT CURRENT_DATE;` in `Scratchpad` and click `Run`: ![Running a query in the Tiger Console scratchpad](/docs/_astro/run-query-in-scratchpad-tiger-console.vyOwaN5N_2mPV5t.webp) Quick recap. You: * Manage your services in [Ops view](https://console.cloud.tigerdata.com/dashboard/services) * Manage your data in [Data view](https://console.cloud.tigerdata.com/dashboard/services?popsql) * Store configuration and security information in your config file. Now you have used Data view in Tiger Console, see how to easily do the following: * [Write a query](https://docs.popsql.com/docs/writing-a-query) * [Share a query with your teammates](https://docs.popsql.com/docs/sharing-a-link-to-your-query-and-results) * [Create a chart from your data](https://docs.popsql.com/docs/creating-charts) * [Create a dashboard of multiple query results](https://docs.popsql.com/docs/creating-dashboards) * [Create schedules for your queries](https://docs.popsql.com/docs/scheduled-queries) ### SQL assistant [Section titled “SQL assistant”](#sql-assistant) SQL assistant in [Data view](https://console.cloud.tigerdata.com/dashboard/services?popsql) is a chat-like interface that harnesses the power of AI to help you write, fix, and organize SQL faster and more accurately. Ask SQL assistant to change existing queries, write new ones from scratch, debug error messages, optimize for query performance, add comments, improve readability, and really, get answers to any questions you can think of. ![Tiger Cloud SQL assistant generating a query](/docs/_astro/sql-assistant-in-action.Deu872qV_Z15AwGH.webp) #### Key capabilities [Section titled “Key capabilities”](#key-capabilities) SQL assistant offers a range of features to improve your SQL workflow, including: * **Real-time help**: SQL assistant provides in-context help for writing and understanding SQL. Use it to: * **Understand functions**: need to know how functions like `LAG()` or `ROW_NUMBER()` work? SQL assistant explains it with examples. * **Interpret complex queries**: SQL assistant breaks down dense queries, giving you a clear view of each part. * **Error resolution**: SQL assistant diagnoses errors as they happen, you can resolve issues without leaving your editor. Features include: * **Error debugging**: if your query fails, SQL assistant identifies the issue and suggests a fix. * **Performance tuning**: for slow queries, SQL assistant provides optimization suggestions to improve performance immediately. * **Query organization**: to keep your query library organized, and help your team understand the purpose of each query, SQL assistant automatically adds titles and summaries to your queries. * **Agent mode**: to get results with minimal involvement from you, SQL assistant autopilots through complex tasks and troubleshoots its own problems. No need to go step by step, analyze errors, and try out solutions. Simply turn on the agent mode in the LLM picker and watch SQL assistant do all the work for you. Recommended for use when your database connection is configured with read-only credentials. #### Supported LLMs [Section titled “Supported LLMs”](#supported-llms) SQL assistant supports a large number of LLMs, including: * GPT-4o mini * GPT-4o * GPT-4.1 nano * GPT-4.1 mini * GPT-4.1 * o4-mini (low) * o4-mini * o4-mini (high) * o3 (low) * o3 * o3 (high) * Claude 3.5 Haiku * Claud 3.7 Sonnet * Claud 3.7 Sonnet (extended thinking) * Llama 3.3 70B Versatile * Llama 3.3 70B Instruct * Llama 3.1 405B Instruct * Llama 4 Scout * Llama 4 Maverick * DeepSeek R1 Distill - Llama 3.3. 70B * DeepSeek R1 * Gemini 2.0 Flash * Sonnet 4 * Sonnet 4 (extended thinking) * Opus 4 * Opus 4 (extended thinking) Choose the LLM based on the particular task at hand. For simpler tasks, try the smaller and faster models like Gemini Flash, Haiku, or o4-mini. For more complex tasks, try the larger reasoning models like Claude Sonnet, Gemini Pro, or o3. We provide a description of each model to help you decide. #### Limitations to keep in mind [Section titled “Limitations to keep in mind”](#limitations-to-keep-in-mind) For best results with SQL assistant: * **Schema awareness**: SQL assistant references schema data but may need extra context in complex environments. Specify tables, columns, or joins as needed. * **Business logic**: SQL assistant does not inherently know specific business terms such as active user. Define these terms clearly to improve results. #### Security, privacy, and data usage [Section titled “Security, privacy, and data usage”](#security-privacy-and-data-usage) Security and privacy is prioritized in Tiger Console. In [Data view](https://console.cloud.tigerdata.com/dashboard/services?popsql), project members manage SQL assistant settings under [`Preferences` > `SQL Assistant`](https://console.cloud.tigerdata.com/dashboard/settings?popsql=%2Fpreferences%2Fai). ![SQL assistant settings in Tiger Console](/docs/_astro/tiger-console-sql-editor-preferences.kZe_coOz_JWTJf.webp) SQL assistant settings are: * **Opt-in features**: all AI features are off by default. Only [members](/docs/deploy/tiger-cloud/tiger-cloud-aws/security/members) of your Tiger Cloud project can enable them. * **Data protection**: your data remains private as SQL assistant operates with strict security protocols. To provide AI support, Tiger Console may share your currently open SQL document, some basic metadata about your database, and portions of your database schema. By default, Tiger Console **does not include** any data from query results, but you can opt in to include this context to improve the results. * **Sample data**: to give the LLM more context so you have better SQL suggestions, enable sample data sharing in the SQL assistant preferences. * **Telemetry**: to improve SQL assistant, Tiger Data collects telemetry and usage data, including prompts, responses, and query metadata. ### Data view FAQ [Section titled “Data view FAQ”](#data-view-faq) #### What if my service is within a VPC? [Section titled “What if my service is within a VPC?”](#what-if-my-service-is-within-a-vpc) If your Tiger Cloud service runs inside a VPC, do one of the following to enable access for the PopSQL desktop app: * Use PopSQL's [bridge connector](https://docs.popsql.com/docs/bridge-connector). * Use an SSH tunnel: when you configure the connection in PopSQL, under `Advanced Options` enable `Connect over SSH`. * Add PopSQL's static IPs (`23.20.131.72, 54.211.234.135`) to your allowlist. #### What happens if another member of my Tiger Cloud project uses Data view? [Section titled “What happens if another member of my Tiger Cloud project uses Data view?”](#what-happens-if-another-member-of-my-tiger-cloud-project-uses-data-view) The number of Data view seats you are allocated depends on your [pricing plan](https://www.tigerdata.com/pricing#features). #### Will using Data view affect the performance of my Tiger Cloud service? [Section titled “Will using Data view affect the performance of my Tiger Cloud service?”](#will-using-data-view-affect-the-performance-of-my-tiger-cloud-service) There are a few factors to consider: 1. What instance size is your service? 2. How many users are running queries? 3. How computationally intensive are the queries? If you have a small number of users running performant SQL queries against a service with sufficient resources, then there should be no degradation to performance. However, if you have a large number of users running queries, or if the queries are computationally expensive, best practice is to create a [read replica](/docs/deploy/tiger-cloud/tiger-cloud-aws/high-availability/read-scaling) and send analytical queries there. If you'd like to prevent write operations such as insert or update, instead of using the `tsdbadmin` user, create a read-only user for your service and use that in Data view. ## SQL editor [Section titled “SQL editor”](#sql-editor) SQL editor is an integrated secure UI that you use to run queries and see the results for a Tiger Cloud service. ![SQL editor in Tiger Console Ops mode](/docs/_astro/tiger-console-ops-mode-sql-editor.ruC--u4y_ZQqjK7.webp) To enable or disable SQL editor in your service, click `Operations` > `Service management`, then update the setting for SQL editor. To use SQL editor: 1. **Open SQL editor from Tiger Console** In Tiger Console, select a service, then click `SQL Editor` at the bottom. ![SQL editor in Tiger Console Ops mode](/docs/_astro/tiger-console-ops-mode-sql-editor-empty.D6fPmf48_LfDRl.webp) 2. **Run a test query** Type `SELECT CURRENT_DATE;` in the UI and click `Run`. The results appear in the lower window: ![Running a query in the Tiger Console Ops mode SQL editor](/docs/_astro/run-a-query-in-tiger-ops-mode-sql-editor.BDO4ugTF_Z1vhVMu.webp) ## Cloud SQL editor licenses [Section titled “Cloud SQL editor licenses”](#cloud-sql-editor-licenses) * **SQL editor**: free for anyone with a [Tiger Cloud account](/docs/get-started/quickstart/create-service). * **Data view**: the number of seats you are allocated depends on your [pricing plan](https://www.tigerdata.com/pricing#features). [SQL assistant](#sql-assistant) is currently free for all users. In the future, limits or paid options may be introduced as we work to build the best experience. * **PopSQL standalone**: there is a free plan available to everyone, as well as paid plans. See [PopSQL Pricing](https://popsql.com/pricing) for full details. # Manage storage and tiering > Configure high-performance and low-cost object storage tiers in Tiger Console Tiering splits hot data on fast local volumes from cold data in cheaper object storage. Tiger Cloud implements that as a high-performance tier plus a low-cost object storage tier: * You use [high-performance storage](#high-performance-storage-tier) to store and query frequently accessed data. * You use [low-cost object storage](#low-cost-object-storage-tier) to cut costs by migrating rarely used data from the high-performance storage. After you enable tiered storage, you then either [create automated tiering policies](#automate-tiering-with-policies) or [manually tier and untier data](#manually-tier-and-untier-chunks). You can query the data on the object storage tier, but you cannot modify it. Make sure that you are not tiering data that needs to be **actively modified**. For low-cost storage, Tiger Data charges for data tiered based on its original uncompressed size in the high-performance storage tier. There are no additional expenses, such as data transfer or compute. ## High-performance storage tier [Section titled “High-performance storage tier”](#high-performance-storage-tier) By default, Tiger Cloud stores your service data in the standard high-performance storage. This gives you up to 64 TB of storage and a configurable IOPS value to better suit your needs: 1. **Select your service** In [Tiger Console](https://console.cloud.tigerdata.com/), select your service, then click `Operations` > `Compute and storage`. ![Default high-performance storage in Tiger Console](/docs/_astro/high-performance-storage-tiger-console.N5Mk8-Vu_1uj4MP.webp) 2. **Select the IOPS value** Select the IOPS value in the `I/O boost` dropdown: * Under the [Performance pricing plan](/docs/deploy/tiger-cloud/tiger-cloud-aws/pricing-and-account-management), IOPS is set to 3,000 - 5,000 autoscale and cannot be changed. * Under the [Scale pricing plan](/docs/deploy/tiger-cloud/tiger-cloud-aws/pricing-and-account-management), IOPS is set to 5,000 - 8,000 autoscale and can be upgraded to 40,000 IOPS subject to your service's CPU configuration. * Under the [Enterprise pricing plan](/docs/deploy/tiger-cloud/tiger-cloud-aws/pricing-and-account-management), IOPS is set to 5,000 - 8,000 autoscale and can be upgraded to 80,000 IOPS subject to your service's CPU configuration. The available IOPS and the corresponding throughput depend on your service's CPU, cloud provider, and the pricing plan: * **Tiger Cloud on AWS** | Pricing plan | Min CPU | IOPS | Throughput, MB/s | | ---------------- | -------- | ------ | ---------------- | | Scale/Enterprise | Under 16 | 16,000 | 1,000 | | Scale/Enterprise | 16 | 24,000 | 1,000 | | Scale/Enterprise | 32 | 32,000 | 1,500 | | Scale/Enterprise | 32 | 40,000 | 1,500 | | Enterprise | 48 | 60,000 | 1,875 | | Enterprise | 64 | 80,000 | 2,000 | * **Tiger Cloud on Azure** | Pricing plan | Min CPU | IOPS | Throughput, MB/s | | ---------------- | ------- | ------ | ---------------- | | Scale/Enterprise | n/a | 16,000 | 1,000 | | Scale/Enterprise | 16 | 24,000 | 1,000 | | Scale/Enterprise | 32 | 32,000 | 1,200 | | Scale/Enterprise | 32 | 40,000 | 1,200 | | Enterprise | 48 | 60,000 | 1,200 | | Enterprise | 64 | 80,000 | 1,200 | 3. **Click `Apply`** ## Low-cost object storage tier [Section titled “Low-cost object storage tier”](#low-cost-object-storage-tier) Note Low-cost tiered storage is available on Scale and Enterprise pricing plans. You enable the low-cost object storage tier in Tiger Console and then tier the data with policies or manually. ### Enable tiered storage [Section titled “Enable tiered storage”](#enable-tiered-storage) You enable tiered storage from the `Overview` tab in Tiger Console. 1. **Select your service** In [Tiger Console](https://console.cloud.tigerdata.com/), select the service to modify. 2. **Enable tiered storage** In `Explorer`, click `Data Tiering` > then click `Enable tiered storage`. ![Enabling tiered storage in Tiger Console](/docs/_astro/enable-tiered-storage-tiger-console.DOgO4kJn_Z14tATB.webp) Once enabled, you can proceed to [tier data manually](#manually-tier-and-untier-chunks) or [set up tiering policies](#automate-tiering-with-policies). When tiered storage is enabled, you see the amount of data in the tiered object storage. ### Automate tiering with policies [Section titled “Automate tiering with policies”](#automate-tiering-with-policies) A tiering policy automatically moves any chunks that only contain data older than the `move_after` threshold to the object storage tier. This works similarly to a [data retention policy](/docs/learn/data-lifecycle/data-retention/about-data-retention), but chunks are moved rather than deleted. A tiering policy schedules a job that runs periodically to asynchronously migrate eligible chunks to object storage. Chunks are considered tiered once they appear in the `timescaledb_osm.tiered_chunks` view. You can add tiering policies to [hypertables](/docs/learn/hypertables/understand-hypertables), including [continuous aggregates](/docs/build/continuous-aggregates/create-a-continuous-aggregate), via Console UI or an SQL editor. #### Add a tiering policy [Section titled “Add a tiering policy”](#add-a-tiering-policy) To add a tiering policy, [connect to your service](/docs/build/data-management/run-queries-from-tiger-console) and call [`add_tiering_policy`](/docs/reference/tiger-cloud/data-tiering/add_tiering_policy): ```sql SELECT add_tiering_policy(hypertable REGCLASS, move_after INTERVAL, if_not_exists BOOL = false); ``` For example, to tier chunks that are more than three days old in the `example` [hypertable](/docs/learn/hypertables/understand-hypertables): ```sql SELECT add_tiering_policy('example', INTERVAL '3 days'); ``` By default, a tiering policy runs hourly on your database. To change this interval, call [`alter_job`](/docs/reference/timescaledb/jobs-automation/alter_job). #### Remove a tiering policy [Section titled “Remove a tiering policy”](#remove-a-tiering-policy) To remove an existing tiering policy, call [`remove_tiering_policy`](/docs/reference/tiger-cloud/data-tiering/remove_tiering_policy): ```sql SELECT remove_tiering_policy(hypertable REGCLASS, if_exists BOOL = false); ``` For example, to remove the tiering policy from the `example` hypertable: ```sql SELECT remove_tiering_policy('example'); ``` If you remove a tiering policy, the remaining scheduled chunks are not tiered. However, chunks in tiered storage are not untiered. You [untier chunks manually](#manually-tier-and-untier-chunks) to local storage. ### Manually tier and untier chunks [Section titled “Manually tier and untier chunks”](#manually-tier-and-untier-chunks) If tiering policies do not meet your current needs, you can tier and untier chunks manually. To do so, [connect to your service](/docs/build/data-management/run-queries-from-tiger-console) and run the queries below in the data mode, the SQL editor, or using `psql`. #### Tier chunks [Section titled “Tier chunks”](#tier-chunks) Tiering a chunk with [`tier_chunk`](/docs/reference/tiger-cloud/data-tiering/tier_chunk) is an asynchronous process that schedules the chunk to be tiered. In the following example, you tier chunks older than three days in the `example` hypertable. You then list the tiered chunks. 1. **Select old chunks** Select all chunks in `example` that are older than three days: ```sql SELECT show_chunks('example', older_than => INTERVAL '3 days'); ``` This returns a list of chunks: ```sql _timescaledb_internal._hyper_1_1_chunk _timescaledb_internal._hyper_1_2_chunk ``` 2. **Tier each chunk** ```sql SELECT tier_chunk('_timescaledb_internal._hyper_1_1_chunk'); ``` Repeat for all chunks you want to tier. Tiering a chunk schedules it for migration to the object storage tier, but the migration won't happen immediately. Chunks are tiered one at a time to minimize database resource consumption. A chunk is marked as migrated and deleted from the standard storage only after it has been durably stored in the object storage tier. You can continue to query a chunk during migration. 3. **View tiered chunks** ```sql SELECT * FROM timescaledb_osm.tiered_chunks; ``` To see which chunks are scheduled for tiering either by policy or by a manual call, but have not yet been tiered: ```sql SELECT * FROM timescaledb_osm.chunks_queued_for_tiering; ``` #### Untier chunks [Section titled “Untier chunks”](#untier-chunks) To update data in a tiered chunk, move it back to the standard high-performance storage tier in Tiger Cloud. Untiering chunks is a synchronous process. Chunks are renamed when the data is untiered. 1. **Check which chunks are tiered** ```sql SELECT * FROM timescaledb_osm.tiered_chunks; ``` Sample output: ```sql hypertable_schema | hypertable_name | chunk_name | range_start | range_end -------------------+-----------------+------------------+------------------------+------------------------ public | sample | _hyper_1_1_chunk | 2023-02-16 00:00:00+00 | 2023-02-23 00:00:00+00 (1 row) ``` 2. **Call `untier_chunk`** For full parameters and behavior, see [`untier_chunk`](/docs/reference/tiger-cloud/data-tiering/untier_chunk). ```sql CALL untier_chunk('_hyper_1_1_chunk'); ``` 3. **Verify the chunk details** ```sql SELECT * FROM timescaledb_information.chunks; ``` Sample output: ```sql -[ RECORD 1 ]----------+------------------------- hypertable_schema | public hypertable_name | sample chunk_schema | _timescaledb_internal chunk_name | _hyper_1_4_chunk primary_dimension | ts primary_dimension_type | timestamp with time zone range_start | 2023-02-16 00:00:00+00 range_end | 2020-03-23 00:00:00+00 range_start_integer | range_end_integer | is_compressed | f chunk_tablespace | data_nodes | ``` ### Drop tiered data [Section titled “Drop tiered data”](#drop-tiered-data) To drop tiered data, call [`DROP TABLE`](/docs/build/data-management) on the corresponding hypertable. This removes the hypertable and all its associated data from the high-performance and low-cost storage. ### Disable tiering [Section titled “Disable tiering”](#disable-tiering) Important Contact Tiger Data support if you are disabling tiering when moving from Scale to Performance pricing plan. If you no longer want to use tiered storage for a particular hypertable, drop the associated metadata by calling [`disable_tiering`](/docs/reference/tiger-cloud/data-tiering/disable_tiering). 1. **Remove tiering policies** Drop all tiering policies by calling `remove_tiering_policy`. 2. **Check for tiered data** Make sure that there is no tiered data associated with this hypertable: 1. List the tiered chunks: ```sql SELECT * FROM timescaledb_osm.tiered_chunks; ``` 2. If you have any tiered chunks, either untier this data, or drop these chunks from tiered storage. 3. **Disable tiering** ```sql SELECT disable_tiering('my_hypertable_name'); ``` 4. **Verify tiering is disabled** ```sql SELECT * FROM timescaledb_osm.tiered_hypertables; ``` # Query tiered data > Query data stored in the low-cost object storage tier using standard SQL Tiered data stays in cheaper object storage, but you can still read it through normal SQL when you opt in with the `timescaledb.enable_tiered_reads` GUC. It defaults to `false` so ad hoc queries do not hit cold storage unless you intend them to. The `timescaledb.enable_tiered_reads` GUC, or Grand Unified Configuration variable, is a setting that controls if tiered data is queried. The configuration variable can be set at different levels, including globally for the entire database server, for individual databases, and for individual sessions. With tiered reads enabled, you can query your data normally even when it's distributed across different storage tiers. Your hypertable is spread across the tiers, so queries and `JOIN`s work and fetch the same data as usual. By default, tiered data is not accessed by queries. Querying tiered data may slow down query performance as the data is not stored locally on the high-performance storage tier. See [Performance considerations](#performance-considerations). ## Enable querying tiered data for a single query [Section titled “Enable querying tiered data for a single query”](#enable-querying-tiered-data-for-a-single-query) 1. **Enable `timescaledb.enable_tiered_reads` for one query** Enable the setting before querying the hypertable with tiered data and reset it after it is complete: ```sql set timescaledb.enable_tiered_reads = true; SELECT count(*) FROM example; set timescaledb.enable_tiered_reads = false; ``` This queries data from all chunks including tiered and non-tiered chunks: ```sql ||count| |---| |1000| ``` ## Enable querying tiered data for a single session [Section titled “Enable querying tiered data for a single session”](#enable-querying-tiered-data-for-a-single-session) All future queries within a session can be enabled to use the object storage tier by enabling `timescaledb.enable_tiered_reads` within a session. ```sql set timescaledb.enable_tiered_reads = true; ``` All future queries in that session are configured to read from tiered data and locally stored data. ## Enable querying tiered data in all future sessions [Section titled “Enable querying tiered data in all future sessions”](#enable-querying-tiered-data-in-all-future-sessions) You can also enable queries to read from tiered data always: ```sql alter database tsdb set timescaledb.enable_tiered_reads = true; ``` In all future created sessions, `timescaledb.enable_tiered_reads` initializes with `enabled`. ## Query data in the object storage tier [Section titled “Query data in the object storage tier”](#query-data-in-the-object-storage-tier) This section illustrates how querying tiered storage works. Consider a simple database with a standard `devices` table and a `metrics` hypertable. After enabling tiered storage, you can see which chunks are tiered to the object storage tier: ```sql chunk_name | range_start | range_end ------------------+------------------------+------------------------ _hyper_2_4_chunk | 2015-12-31 00:00:00+00 | 2016-01-07 00:00:00+00 _hyper_2_3_chunk | 2017-08-17 00:00:00+00 | 2017-08-24 00:00:00+00 (2 rows) ``` The following query fetches data only from the object storage tier. This makes sense based on the `WHERE` clause specified by the query and the chunk ranges listed above for this hypertable. ```sql EXPLAIN SELECT * FROM metrics where ts < '2017-01-01 00:00+00'; QUERY PLAN --------------------------------------------------------------------- Foreign Scan on osm_chunk_2 (cost=0.00..0.00 rows=2 width=20) Filter: (ts < '2017-01-01 00:00:00'::timestamp without time zone) Match tiered objects: 1 Row Groups: _timescaledb_internal._hyper_2_4_chunk: 0 (5 rows) ``` If your query does not need to touch the object storage tier, it only processes the chunks in the standard storage. The following query refers to newer data that is not yet tiered to the object storage tier. `Match tiered objects: 0` in the plan indicates that no tiered data matches the query constraint. So data in the object storage is not touched at all. ```sql EXPLAIN SELECT * FROM metrics where ts > '2022-01-01 00:00+00'; QUERY PLAN ------------------------------------------------------------------------------------------------------------------- Append (cost=0.15..25.02 rows=568 width=20) -> Index Scan using _hyper_2_5_chunk_metrics_ts_idx on _hyper_2_5_chunk (cost=0.15..22.18 rows=567 width=20) Index Cond: (ts > '2022-01-01 00:00:00'::timestamp without time zone) -> Foreign Scan on osm_chunk_2 (cost=0.00..0.00 rows=1 width=20) Filter: (ts > '2022-01-01 00:00:00'::timestamp without time zone) Match tiered objects: 0 Row Groups: (7 rows) ``` Here is another example with a `JOIN` that does not touch tiered data: ```sql EXPLAIN SELECT ts, device_id, description FROM metrics JOIN devices ON metrics.device_id = devices.id WHERE metrics.ts > '2023-08-01'; QUERY PLAN ------------------------------------------------------------------------------------------------------------------- Hash Join (cost=32.12..184.55 rows=3607 width=44) Hash Cond: (devices.id = _hyper_4_9_chunk.device_id) -> Seq Scan on devices (cost=0.00..22.70 rows=1270 width=36) -> Hash (cost=25.02..25.02 rows=568 width=12) -> Append (cost=0.15..25.02 rows=568 width=12) -> Index Scan using _hyper_4_9_chunk_metrics_ts_idx on _hyper_4_9_chunk (cost=0.15..22.18 rows=567 width=12) Index Cond: (ts > '2023-08-01 00:00:00+00'::timestamp with time zone) -> Foreign Scan on osm_chunk_3 (cost=0.00..0.00 rows=1 width=12) Filter: (ts > '2023-08-01 00:00:00+00'::timestamp with time zone) Match tiered objects: 0 Row Groups: (11 rows) ``` ## Performance considerations [Section titled “Performance considerations”](#performance-considerations) Queries over tiered data are expected to be slower than over local data. However, in a limited number of scenarios tiered reads can impact query planning time over local data as well. In order to prevent any unexpected performance degradation for application queries, we keep the GUC `timescaledb.enable_tiered_reads` set to `false`. * Queries without time boundaries specified are expected to perform slower when querying tiered data, both during query planning and during query execution. TimescaleDB chunk exclusion algorithms cannot be applied for this case. ```sql SELECT * FROM device_readings WHERE id = 10; ``` * Queries with predicates computed at runtime (such as `NOW()`) are not always optimized at planning time and as a result might perform slower than statically assigned values when querying against the object storage tier. For example, this query is optimized at planning time: ```sql SELECT * FROM metrics WHERE ts > '2023-01-01' AND ts < '2023-02-01' ``` The following query does not do chunk pruning at query planning time: ```sql SELECT * FROM metrics WHERE ts < now() - '10 days'::interval ``` At the moment, queries against tiered data work best when the query optimizer can apply planning time optimizations. * Text and non-native types (JSON, JSONB, GIS) filtering is slower when querying tiered data. # Replicas and forks with tiered data > Tiering data to object storage means your replicas and forks share the same low-cost data without additional storage costs Data that lives in the low-cost object storage tier is billed once even when you add a [high-availability replica](/docs/deploy/tiger-cloud/tiger-cloud-aws/high-availability/high-availability) or [read replicas](/docs/deploy/tiger-cloud/tiger-cloud-aws/high-availability/read-scaling) to your service, because replicas share the same tiered objects. The same applies to [forks](/docs/deploy/tiger-cloud/tiger-cloud-aws/service-management/fork-services), which you can use, for example, for running tests or creating dev environments. When creating one (or more) forks, you won't be billed for data shared with the primary in the low-cost storage. If you decide to tier more data that's not in the primary, you will pay to store it in the low-cost tier, but you will still see substantial savings by moving that data from the high-performance tier of the fork to the cheaper object storage tier. ## How this works behind the scenes [Section titled “How this works behind the scenes”](#how-this-works-behind-the-scenes) Once you tier data to the low-cost object storage tier, we keep a reference to that data on your database's catalog. Creating a replica or forking a primary server only copies the references and the metadata we keep on the catalog for all tiered data. On the billing side, we only count and bill once for the data tiered, not for each reference there may exist towards that data. ## What happens when a chunk is dropped or untiered on a fork [Section titled “What happens when a chunk is dropped or untiered on a fork”](#what-happens-when-a-chunk-is-dropped-or-untiered-on-a-fork) Dropping or untiering a chunk from a fork does not delete it from any other servers that reference the same chunk. You can have one, multiple, or 0 servers referencing the same chunk of data: * That means that deleting data from a fork does not affect the other servers (including the primary); it just removes the reference to that data, which is for all intents and purposes equal to deleting that data from the point of view of that fork. * The primary and other servers are unaffected, as they still have their references and the metadata on their catalogs intact. * We never delete anything on the object storage tier if at least one server references it: the data is only permanently deleted once the references drop to 0. As described above, tiered chunks are only counted once for billing purposes, so dropping or untiering a chunk that is shared with other servers from a fork will not affect billing as it was never counted for billing purposes. Dropping or untiering a chunk that was only tiered on that fork works as expected and is covered in more detail in the following section. ## What happens when a chunk is modified on a fork [Section titled “What happens when a chunk is modified on a fork”](#what-happens-when-a-chunk-is-modified-on-a-fork) As a reminder, tiered data is immutable - there is no such thing as updating the data. You can untier or drop a chunk, in which case what is described in the previous section covers what happens. And you can tier new data, at which point a fork deviates from the primary in a similar way as all forks do. New data tiered are not shared with parent or sibling servers, this is new data tiered for that server and we count them as a new object for the purposes of billing. If you decide to tier more data that's not in the primary, you will pay to store it in the low-cost tier, but you will still see substantial savings by moving that data from the high-performance tier of the fork to the cheaper object storage tier. Similar to other types of storage tiers, this type of deviation can not happen for replicas as they have to be identical with the primary server, that's why we don't mention replicas when discussing dropping chunks or tiering additional data. ## What happens with backups and PITR [Section titled “What happens with backups and PITR”](#what-happens-with-backups-and-pitr) As discussed above, we never delete anything on the object storage tier if at least one server references it. The data is only permanently deleted once the references drop to 0. In addition to that, we delay permanently deleting the data by 14 days, so that in case of a restore or PITR, all tiered data will be available. In the case of such a restore, new references are added to the deleted tiered chunks, so they are no longer candidates for permanent deletion. Once 14 days pass after soft deleting the data, that is the number of references to the tiered data drops to 0, we permanently delete the tiered data. # Delete data > Delete data with DELETE or by dropping chunks from a hypertable in your database using SQL Row deletes use the standard [`DELETE`](https://www.postgresql.org/docs/current/sql-delete.html) statement against a hypertable. When you need to age out large ranges, dropping chunks or using a retention policy is usually cheaper than massive `DELETE` scans. ## Delete data with DELETE command [Section titled “Delete data with DELETE command”](#delete-data-with-delete-command) To delete data from a table, use the syntax `DELETE FROM ...`. In this example, data is deleted from the table `conditions`, if the row's `temperature` or `humidity` is below a certain level: ```sql DELETE FROM conditions WHERE temperature < 35 OR humidity < 60; ``` Tips If you delete a lot of data, run [`VACUUM`](https://www.postgresql.org/docs/current/sql-vacuum.html) or `VACUUM FULL` to reclaim storage from the deleted or obsolete rows. ## Delete data by dropping chunks [Section titled “Delete data by dropping chunks”](#delete-data-by-dropping-chunks) TimescaleDB allows you to delete data by age, by dropping chunks from a hypertable. You can do so either manually or by data retention policy. To learn more, see the [data retention section](/docs/learn/data-lifecycle/data-retention/about-data-retention). # Insert data > Insert single and multiple rows and bulk load data into TimescaleDB with SQL Inserts into a hypertable are normal PostgreSQL writes. Use the usual commands, and lean on batching when you can: * `INSERT`: single rows or small batches * `COPY`: bulk data loading To improve performance, insert time series data directly to the columnstore using [direct compress](#improve-performance-with-direct-compress). ## Insert a single row [Section titled “Insert a single row”](#insert-a-single-row) To insert a single row into a hypertable, use the syntax `INSERT INTO ... VALUES`: ```sql INSERT INTO conditions(time, location, temperature, humidity) VALUES (NOW(), 'office', 70.0, 50.0); ``` ## Insert multiple rows [Section titled “Insert multiple rows”](#insert-multiple-rows) A more efficient method to insert row-by-row is to insert multiple rows into a hypertable using a single `INSERT` call. This works even for thousands of rows at a time. TimescaleDB batches the rows by chunk, then writes to each chunk in a single transaction. You use the same syntax, separating rows with a comma: ```sql INSERT INTO conditions VALUES (NOW(), 'office', 70.0, 50.0), (NOW(), 'basement', 66.5, 60.0), (NOW(), 'garage', 77.0, 65.2); ``` If you `INSERT` unsorted data, call [`convert_to_columnstore('', recompress => true)`](/docs/reference/timescaledb/hypercore/convert_to_columnstore) on the chunk to reorder and optimize your data. ## Insert and return data [Section titled “Insert and return data”](#insert-and-return-data) You can return some or all of the inserted data by adding a `RETURNING` clause to the `INSERT` command. For example, to return all the inserted data, run: ```sql INSERT INTO conditions VALUES (NOW(), 'office', 70.1, 50.1) RETURNING *; ``` This returns: ```sql time | location | temperature | humidity ------------------------------+----------+-------------+---------- 2017-07-28 11:42:42.846621+00 | office | 70.1 | 50.1 (1 row) ``` If you `INSERT` unsorted data, call [`convert_to_columnstore('', recompress => true)`](/docs/reference/timescaledb/hypercore/convert_to_columnstore) on the chunk to reorder and optimize your data. ## Bulk insert with COPY [Section titled “Bulk insert with COPY”](#bulk-insert-with-copy) The `COPY` command is the most efficient way to load large amounts of data into a hypertable. For bulk data loading, `COPY` can be 2-3x faster or more than `INSERT`, especially when combined with [direct compress](#improve-performance-with-direct-compress). `COPY` supports loading from: * **CSV files**: ```sql COPY conditions(time, location, temperature, humidity) FROM '/path/to/data.csv' WITH (FORMAT CSV, HEADER); ``` * **Standard input** To load data from your application or script using standard input: ```sql COPY conditions(time, location, temperature, humidity) FROM STDIN WITH (FORMAT CSV); ``` To signal the end of input, add `\.` on a new line. * **Program output** To load data generated by a program or script: ```sql COPY conditions(time, location, temperature, humidity) FROM PROGRAM 'generate_data.sh' WITH (FORMAT CSV); ``` If you `COPY` unsorted data, call [`convert_to_columnstore('', recompress => true)`](/docs/reference/timescaledb/hypercore/convert_to_columnstore) on the chunk to reorder and optimize your data. ### Skip continuous aggregate invalidation during bulk loads [Section titled “Skip continuous aggregate invalidation during bulk loads”](#skip-continuous-aggregate-invalidation-during-bulk-loads) Available in TimescaleDB 2.28.0 and later. Every write to a hypertable that has continuous aggregates records an invalidation entry, so the next refresh knows which time ranges changed. During a large bulk load, this tracking adds overhead. To skip it for the current session or transaction, set `timescaledb.skip_cagg_invalidation`. Use `SET LOCAL` inside a transaction so the setting reverts automatically when the transaction ends: ```sql BEGIN; SET LOCAL timescaledb.skip_cagg_invalidation = on; COPY conditions FROM '/path/to/data.csv' WITH (FORMAT CSV, HEADER); COMMIT; ``` Because no invalidations are recorded, a normal refresh treats the loaded buckets as up-to-date. After the load, refresh the time ranges you touched with `force => true` to bring each continuous aggregate back in sync: ```sql CALL refresh_continuous_aggregate('conditions_daily', '', '', force => true); ``` Warning If you skip the forced refresh, the continuous aggregate is left out of sync with the hypertable: queries return stale results for the time ranges you loaded. Only use `timescaledb.skip_cagg_invalidation` for bulk or migration workloads where you control the follow-up refresh. ## Improve performance with direct compress [Section titled “Improve performance with direct compress”](#improve-performance-with-direct-compress) When you set `timescaledb.enable_direct_compress_insert` or `timescaledb.enable_direct_compress_copy`, TimescaleDB compresses data in memory during ingestion and writes the compressed batches directly to the columnstore. This significantly reduces the I/O footprint, and `INSERT` and `COPY` produce columnstore chunks on the spot — the [columnstore policy](/docs/reference/timescaledb/hypercore/add_columnstore_policy) is less important. Warning This feature is a **tech preview** and not production-ready. Using direct compress can regress query performance or storage ratio if the ingested rows are not sorted by the table's `orderby` columns or if the data has very high cardinality. To enable in-memory compression during ingestion, set the appropriate GUC: ```sql SET timescaledb.enable_direct_compress_insert = on; -- or SET timescaledb.enable_direct_compress_copy = on; ``` **Important facts** * High cardinality data does not produce good batches and degrades query performance. * The columnstore is optimized for 1000 rows per batch per `segmentby` value. * WAL records are written for the compressed batches rather than the individual tuples. * Best results are achieved for batch ingestion with 1000 rows or more. * Tables with unique constraints cannot use direct compress. * If your hypertable has no `segmentby` column configured, TimescaleDB picks one for you based on the first batch of rows you ingest. Turn this off with `SET timescaledb.enable_direct_compress_auto_segmentby = off`. # Update data > Update single and multiple rows in a hypertable in your TimescaleDB database with SQL Point fixes and time-bounded corrections both use [`UPDATE`](https://www.postgresql.org/docs/current/sql-update.html) against a hypertable, just like a regular table. ## Update a single row [Section titled “Update a single row”](#update-a-single-row) Update a single row with the syntax `UPDATE ... SET ... WHERE`. For example, to update a row in the `conditions` hypertable with new `temperature` and `humidity` values, run the following. The `WHERE` clause specifies the row to be updated. ```sql UPDATE conditions SET temperature = 70.2, humidity = 50.0 WHERE time = '2017-07-28 11:42:42.846621+00' AND location = 'office'; ``` ## Update multiple rows at once [Section titled “Update multiple rows at once”](#update-multiple-rows-at-once) You can also update multiple rows at once, by using a `WHERE` clause that filters for more than one row. For example, run the following to update all `temperature` values within the given 10-minute span: ```sql UPDATE conditions SET temperature = temperature + 0.1 WHERE time >= '2017-07-28 11:40' AND time < '2017-07-28 11:50'; ``` # Upsert data > Insert a new row or update an existing row in a hypertable using UPSERT An upsert either inserts a new row or reconciles with an existing one in a single round trip. That helps with idempotent ingestion and late-arriving measurements: * **A matching row does not exist**: inserts a new row * **A matching row exists**: either updates the existing row, or does nothing ## Upsert, unique indexes, and constraints [Section titled “Upsert, unique indexes, and constraints”](#upsert-unique-indexes-and-constraints) Upserts work when you have a unique index or constraint. A matching row is one that has identical values for the columns covered by the index or constraint. In PostgreSQL, a primary key is a unique index with a `NOT NULL` constraint. If you have a primary key, you automatically have a unique index. Unique constraints must include all partitioning columns. That means unique constraints on a hypertable must include the time column. If you added other partitioning columns to your hypertable, the constraint must include those as well. For more information, see [Enforce constraints with unique indexes](/docs/build/performance-optimization/hypertables-and-unique-indexes). The examples in this page use a `conditions` table with a unique constraint on the columns `(time, location)`. To create a unique constraint, either: * Use `UNIQUE ()` when you define your table: ```sql CREATE TABLE conditions ( time TIMESTAMPTZ NOT NULL, location TEXT NOT NULL, temperature DOUBLE PRECISION NULL, humidity DOUBLE PRECISION NULL, UNIQUE (time, location) ); ``` * Use `ALTER TABLE` after the table is created: ```sql ALTER TABLE conditions ADD CONSTRAINT conditions_time_location UNIQUE (time, location); ``` ## Insert or update data [Section titled “Insert or update data”](#insert-or-update-data) To insert new data that doesn't violate the constraint, and to update the existing row if it does, use the syntax `INSERT INTO ... VALUES ... ON CONFLICT ... DO UPDATE`. For example, to update the `temperature` and `humidity` values if a row with the specified `time` and `location` already exists, run: ```sql INSERT INTO conditions VALUES ('2017-07-28 11:42:42.846621+00', 'office', 70.2, 50.1) ON CONFLICT (time, location) DO UPDATE SET temperature = excluded.temperature, humidity = excluded.humidity; ``` ## Insert or do nothing [Section titled “Insert or do nothing”](#insert-or-do-nothing) You can also do nothing if the constraint is violated. The new data is not inserted, and the old row is not updated, the database engine skips the row and moves on. This is useful to prevent the entire transaction from failing when writing many rows as one batch. To insert or do nothing, use the syntax `INSERT INTO ... VALUES ... ON CONFLICT DO NOTHING`: ```sql INSERT INTO conditions VALUES ('2017-07-28 11:42:42.846621+00', 'office', 70.1, 50.0) ON CONFLICT DO NOTHING; ``` ## Faster upserts on the columnstore [Section titled “Faster upserts on the columnstore”](#faster-upserts-on-the-columnstore) When your hypertable has bloom sparse indexes on the conflict columns, TimescaleDB skips columnstore batches that can't contain the conflict values — more than 2× faster on large columnstore datasets. See [Sparse indexes on the columnstore](/docs/build/performance-optimization/indexing#sparse-indexes-on-the-columnstore) for setup and verification. ## Bulk upsert using COPY [Section titled “Bulk upsert using COPY”](#bulk-upsert-using-copy) When you need to upsert large amounts of data, `COPY` is significantly faster than `INSERT`. However, `COPY` doesn't support `ON CONFLICT` clauses directly. Best practice is to use a staging table. This two-step approach combines the speed of `COPY` for bulk loading with the flexibility of `INSERT...ON CONFLICT` for upsert logic. For large datasets, this is much faster than using `INSERT...ON CONFLICT` directly. To load data efficiently with `COPY`, then upsert: 1. **Create a staging table with the same structure as the destination table** ```sql CREATE TEMP TABLE conditions_staging (LIKE conditions); ``` 2. **Use `COPY` to bulk load data into the staging table** ```sql COPY conditions_staging(time, location, temperature, humidity) FROM '/path/to/data.csv' WITH (FORMAT CSV, HEADER); ``` 3. **Upsert from the staging table to the destination table** ```sql INSERT INTO conditions SELECT * FROM conditions_staging ON CONFLICT (time, location) DO UPDATE SET temperature = EXCLUDED.temperature, humidity = EXCLUDED.humidity; ``` To skip duplicate rows, set `ON CONFLICT (time, location) DO NOTHING`. 4. **Clean up the staging table** ```sql DROP TABLE conditions_staging; ``` # Guided projects > Learn Tiger Data products through practical, real-world guided projects These guided projects pair a realistic dataset with Tiger Data product features so you learn by shipping a small vertical slice. Some run on a Tiger Cloud service and visualize results in Grafana; others run on a local self-hosted TimescaleDB instance with Docker and `psql`. Pick a domain close to your own stack and follow the steps in order. [Simulate an IoT sensor dataset](/docs/build/examples/simulate-iot-sensor-data) [Simulate an IoT sensor dataset for testing and development with Tiger Cloud](/docs/build/examples/simulate-iot-sensor-data) [Visualize financial tick data with Grafana](/docs/build/examples/analyze-financial-tick-data) [Store financial tick data, create candlestick views, and visualize price changes in Grafana](/docs/build/examples/analyze-financial-tick-data) [Analyze stock market data](/docs/build/examples/analyze-stock-market-data) [Analyze S\&P 500 tick and candlestick data with OHLCV, moving averages, and volatility queries on self-hosted TimescaleDB](/docs/build/examples/analyze-stock-market-data) [Ingest real-time financial data](/docs/build/examples/ingest-real-time-financial-data) [Set up a data pipeline to get data from different financial APIs](/docs/build/examples/ingest-real-time-financial-data) [Analyze Bitcoin blockchain](/docs/build/examples/analyze-blockchain) [Query and analyze blockchain transactions to discover insights about fees, mining revenue, and market trends](/docs/build/examples/analyze-blockchain) [Analyze energy consumption](/docs/build/examples/analyze-energy-consumption) [Get insights from energy consumption data with Grafana and Tiger Cloud](/docs/build/examples/analyze-energy-consumption) [Visualize transport and geospatial data with Grafana](/docs/build/examples/analyze-transport-data) [Simulate and analyze a transport dataset in Tiger Cloud and visualize it in Grafana](/docs/build/examples/analyze-transport-data) [Analyze NYC taxi data](/docs/build/examples/analyze-nyc-taxi-data) [Analyze high-volume New York City taxi trips with location and time-based queries on self-hosted TimescaleDB](/docs/build/examples/analyze-nyc-taxi-data) [Analyze application events with UUIDv7](/docs/build/examples/analyze-events-with-uuidv7) [Track and analyze application events with UUIDv7 identifiers on self-hosted TimescaleDB, with no separate timestamp column](/docs/build/examples/analyze-events-with-uuidv7) [Aggregate organizational data with AI agents](/docs/build/examples/aggregate-organizational-data-with-ai) [Unify company knowledge with slack-native AI agents](/docs/build/examples/aggregate-organizational-data-with-ai) [Build hybrid search with BM25 and vectors](/docs/build/examples/hybrid-search) [Combine keyword search and vector similarity search in PostgreSQL using pg\_textsearch and pgvectorscale](/docs/build/examples/hybrid-search) [Tiger Data cookbook](/docs/build/examples/cookbook) [Browse code examples from the Tiger Data community that resolve common issues and provide solutions for non-standard tasks](/docs/build/examples/cookbook) # Aggregate organizational data with AI agents > Unify company knowledge with Slack-native AI agents using Tiger Eon and Tiger Cloud. Your business already has the answers in Slack threads, GitHub pull requests, Linear tasks, your own docs, Salesforce service tickets, anywhere you store data. However, those answers are scattered, hard to find, and often forgotten. Tiger Eon automatically integrates Tiger Agents for Work with your organizational data so you can let AI Assistants analyze your company data and give you the answers you need. For example: * What did we ship last week? * What's blocking the release? * Summarize the latest GitHub pull requests. Eon responds instantly, pulling from the tools you already use. No new UI, no new workflow, just answers in Slack. ![Asking Eon a question in Slack and getting an instant answer](/docs/_astro/tiger-eon-big-question.DSXZl-VV_ZlEnPU.webp) By the end of this tutorial, you'll be able to: * Install and run Eon with Docker and Tiger CLI * Connect Slack, Anthropic, and optionally GitHub and Linear to Eon * Ask questions in Slack and get answers from your organizational data stored in a Tiger Cloud service (Optional) You can also watch a walkthrough [on the Tiger Data YouTube channel](https://www.youtube.com/@tigerdata) (insert video link when available). ## Background [Section titled “Background”](#background) Tiger Eon: * **Unlocks hidden value**: your data in Slack, GitHub, and Linear already contains the insights you need. Eon makes them accessible. * **Enables faster decisions**: no need to search or ask around, you get answers in seconds. * **Is easy to use**: Eon runs a Tiger Agent and MCP servers statelessly in lightweight Docker containers. * **Integrates seamlessly with Tiger Cloud**: Eon uses a Tiger Cloud service so you securely and reliably store your company data. Prefer to self-host? Use a [PostgreSQL instance with TimescaleDB](/docs/get-started/choose-your-path/install-timescaledb). Tiger Eon's real-time ingestion system connects to Slack and captures everything: every message, reaction, edit, and channel update. It can also process historical Slack exports. Eon had instant access to years of institutional knowledge from the very beginning. All of this data is stored in your Tiger Cloud service as time-series data: conversations are events unfolding over time, and Tiger Cloud is purpose-built for precisely this. Your data is optimized by: * Automatically partitioning the data into 7-day chunks for efficient queries * Compressing the data after 45 days to save space * Segmenting by channel for faster retrieval When someone asks Eon a question, it uses simple SQL to instantly retrieve the full thread context, related conversations, and historical decisions. No rate limits. No API quotas. Just direct access to your data. ## What is the tech stack we're working with? [Section titled “What is the tech stack we're working with?”](#what-is-the-tech-stack-were-working-with) **Tiger Eon:** A production-ready system that runs AI agents and MCP servers to connect your organizational data (Slack, GitHub, Linear) with a Tiger Cloud service. It uses [Tiger CLI](https://github.com/timescale/tiger-cli/) and [Tiger Agents for Work](https://github.com/timescale/tiger-agents-for-work) so you can ask questions in Slack and get answers from your company data in seconds. * **Tiger Cloud** stores the ingested data as time-series and powers the agents. * **Docker Compose** runs the local components (ingest app, listener app, MCP servers) in lightweight containers. ## Prerequisites for this tutorial To follow these steps, you'll need: * A [Tiger Cloud account](/docs/get-started/quickstart/create-service#create-a-tiger-cloud-account). - [Docker](https://docs.docker.com/engine/install/) installed on your developer device. - [Tiger CLI](https://github.com/timescale/tiger-cli/) installed. - Rights to create an [Anthropic API key](https://console.anthropic.com/settings/keys). - Access to create apps in your [Slack workspace](https://api.slack.com/apps/) (you'll create two during setup: ingest and listener). - Optionally: * Rights to create a [GitHub token](https://github.com/settings/tokens/new?description=Tiger%20Agent\&scopes=repo,read:org). * Rights to create a [Logfire token](http://logfire.pydantic.dev/docs/how-to-guides/create-write-tokens/). * Rights to create a [Linear token](https://linear.app/docs/api-and-webhooks#api-keys). ## Interactive setup [Section titled “Interactive setup”](#interactive-setup) Tiger Eon is a production-ready repository running [Tiger CLI](https://github.com/timescale/tiger-cli/) and [Tiger Agents for Work](https://github.com/timescale/tiger-agents-for-work) that creates and runs the following components for you: * An ingest Slack app that consumes all messages and reactions from public channels in your Slack workspace * A [Tiger Agent](https://github.com/timescale/tiger-agents-for-work) that analyzes your company data for you * A Tiger Cloud service instance that stores data from the Slack apps * MCP servers that connect data sources to Eon * A listener Slack app that passes questions to the Tiger Agent when you @tag it in a public channel, and returns the AI analysis on your data All local components are run in lightweight Docker containers via Docker Compose. 1. **Install Tiger Eon to manage and run your AI-powered Slack bots** In a local folder, run the following command from the terminal: ```shell git clone git@github.com:timescale/tiger-eon.git ``` 2. **Start the Eon setup** ```shell cd tiger-eon ./setup-tiger-eon.sh ``` You see a summary of the setup procedure. Type `y` and press `Enter`. 3. **Create the Tiger Cloud service to use with Eon** You see `Do you want to use a hosted Tiger Cloud Database? [y/N]:`. Press `Y` to create a free Tiger Cloud service. Eon opens the Tiger Cloud authentication page in your browser. Click `Authorize`. Eon creates a Tiger Cloud service called [tiger-eon](https://console.cloud.tigerdata.com/dashboard/services) and stores the credentials in your local keychain. If you press `N`, the Eon setup creates and runs TimescaleDB in a local Docker container. 4. **Create the ingest Slack app** 1. In the terminal, name your ingest Slack app: 1. Eon proposes to create an ingest app called `tiger-slack-ingest`, press `Enter`. 2. Do the same for the App description. Eon opens `Your Apps` in . 2. Start configuring your ingest app in Slack. In the Slack `Your Apps` page: 1. Click `Create New App`, click `From an manifest`, then select a workspace. 2. Click `Next`. Slack opens `Create app from manifest`. 3. Add the Slack app manifest: 1. In terminal press `Enter`. The setup prints the Slack app manifest to terminal and adds it to your clipboard. 2. In the Slack `Create app from manifest` window, paste the manifest. 3. Click `Next`, then click `Create`. 4. Configure an app-level token: 1. In your app settings, go to `Basic Information`. 2. Scroll to `App-Level Tokens`. 3. Click `Generate Token and Scopes`. 4. Add a `Token Name`, then click `Add Scope` add `connections:write`, then click `Generate`. 5. Copy the `xapp-*` token and click `Done`. 6. In the terminal, paste the token, then press `Enter`. 5. Configure a bot user OAuth token: 1. In your app settings, under `Features`, click `App Home`. 2. Scroll down, then enable `Allow users to send Slash commands and messages from the messages tab`. 3. In your app settings, under `Settings`, click `Install App`. 4. Click `Install to `, then click `Allow`. 5. Copy the `xoxb-` Bot User OAuth Token locally. 6. In the terminal, paste the token, then press `Enter`. 5. **Create the Eon Slack app** Follow the same procedure as you did for the ingest Slack app. 6. **Integrate Eon with Anthropic** The Eon setup opens . Create a Claude Code key, then paste it in the terminal. 7. **Integrate Eon with Logfire** If you would like to integrate logfire with Eon, paste your token and press `Enter`. If not, press `Enter`. 8. **Integrate Eon with GitHub** The Eon setup asks if you would like to `Enable github MCP server?". For {C.EON_SHORT} to answer questions about the activity in your Github organization`. Press `y` to integrate with GitHub. 9. **Integrate Eon with Linear** The Eon setup asks if you would like to `Enable linear MCP server? [y/N]:`. Press `y` to integrate with Linear. 10. **Give Eon access to private repositories** 1. The setup asks if you would like to include access to private repositories. Press `y`. 2. Follow the GitHub token creation process. 3. In the Eon setup add your organization name, then paste the GitHub token. The setup sets up a new Tiger Cloud service for you called `tiger-eon`, then starts Eon in Docker. ![Eon services running in Docker containers](/docs/_astro/tiger-eon-docker-services.JBDn7fBH_1b8NOB.webp) You have created: * The Eon ingest and chat apps in Slack * A private MCP server connecting Eon to your data in GitHub * A Tiger Cloud service that securely stores the data used by Eon ## Integrate Eon in your Slack workspace [Section titled “Integrate Eon in your Slack workspace”](#integrate-eon-in-your-slack-workspace) To enable your AI Assistant to analyze your data for you when you ask a question, open a public channel, invite `@eon` to join, then ask a question: ![Adding the Eon app to a Slack channel](/docs/_astro/tiger-eon-slack-channel-add.CacIEtx9_Z1P0hvY.webp) ## Summary [Section titled “Summary”](#summary) In this tutorial, you learned how to: * Install Eon and run the interactive setup with Tiger CLI * Create a Tiger Cloud service and two Slack apps (ingest and listener) * Connect Anthropic, and optionally Logfire, GitHub, and Linear to Eon * Invite Eon to a Slack channel and ask questions about your organizational data ## Next steps [Section titled “Next steps”](#next-steps) ### Learn more [Section titled “Learn more”](#learn-more) [Tiger Cloud](/docs/deploy/tiger-cloud/get-started/quickstart/create-service) [Create and manage your Tiger Cloud service](/docs/deploy/tiger-cloud/get-started/quickstart/create-service) [Self-hosted TimescaleDB](/docs/get-started/choose-your-path/install-timescaledb) [Install and run TimescaleDB yourself](/docs/get-started/choose-your-path/install-timescaledb) ### Other examples [Section titled “Other examples”](#other-examples) [Simulate IoT sensor data](/docs/build/examples/simulate-iot-sensor-data) [Generate sample time-series data for testing](/docs/build/examples/simulate-iot-sensor-data) [Analyze energy consumption](/docs/build/examples/analyze-energy-consumption) [Grafana and continuous aggregates](/docs/build/examples/analyze-energy-consumption) # Analyze Bitcoin blockchain > Query and analyze blockchain transactions to discover insights about fees, mining revenue, and market trends The financial industry is extremely data-heavy and relies on real-time and historical data for decision-making, risk assessment, fraud detection, and market analysis. Tiger Data simplifies management of these large volumes of data, while also providing you with meaningful analytical insights and optimizing storage costs. In this tutorial, you use Tiger Cloud to ingest, store, and analyze transactions on the Bitcoin blockchain. [Blockchains](https://www.pcmag.com/encyclopedia/term/blockchain) are, at their essence, a distributed database. The [transactions](https://www.pcmag.com/encyclopedia/term/bitcoin-transaction) in a blockchain are an example of time-series data. You can use TimescaleDB to query transactions on a blockchain, in exactly the same way as you might query time-series transactions in any other database. This tutorial uses a sample Bitcoin dataset and covers: 1. **Ingest data**: set up and connect to a Tiger Cloud service, create tables and hypertables, and ingest data. 2. **Query the data**: obtain information about recent transactions and blocks using basic SQL queries. 3. **Analyze the data**: create continuous aggregates and use TimescaleDB hyperfunctions to discover insights about transaction fees, mining revenue, and market correlations. 4. **Visualize results**: graph your analytical queries in Grafana dashboards. ## Prerequisites for this tutorial To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service). These steps use Tiger Cloud, but the same approach applies to a [self-hosted TimescaleDB](/docs/get-started/choose-your-path/install-timescaledb) instance. - A [self-managed Grafana](https://grafana.com/get/?tab=self-managed) instance, or a [Grafana Cloud](https://grafana.com/get/) account. This tutorial uses a dataset that contains Bitcoin blockchain data for the past five days, in a hypertable named `transactions`. ## Optimize time-series data using hypertables Hypertables are PostgreSQL tables in TimescaleDB that automatically partition your time-series data by time. Time-series data represents the way a system, process, or behavior changes over time. Hypertables enable TimescaleDB to work efficiently with time-series data. Each hypertable is made up of child tables called chunks. Each chunk is assigned a range of time, and only contains data from that range. When you run a query, TimescaleDB identifies the correct chunk and runs the query on it, instead of going through the entire table. [Hypercore](/docs/learn/columnar-storage/understand-hypercore) is the hybrid row-columnar storage engine in TimescaleDB used by hypertables. Traditional databases force a trade-off between fast inserts (row-based storage) and efficient analytics (columnar storage). Hypercore eliminates this trade-off, allowing real-time analytics without sacrificing transactional capabilities. Hypercore dynamically stores data in the most efficient format for its lifecycle: ![Move from rowstore to columstore in hypercore](/docs/_astro/hypercore_intro.DmLiqZGV_1VM5z7.webp) ![Move from rowstore to columstore in hypercore](/docs/_astro/hypercore_intro-dark.qPmjkBqW_19V2rF.webp) * **Row-based storage for recent data**: the most recent chunk (and possibly more) is always stored in the rowstore, ensuring fast inserts, updates, and low-latency single record queries. Additionally, row-based storage is used as a writethrough for inserts and updates to columnar storage. * **Columnar storage for analytical performance**: chunks are automatically compressed into the columnstore, optimizing storage efficiency and accelerating analytical queries. Unlike traditional columnar databases, hypercore allows data to be inserted or modified at any stage, making it a flexible solution for both high-ingest transactional workloads and real-time analytics, within a single database. Because TimescaleDB is 100% PostgreSQL, you can use all the standard PostgreSQL tables, indexes, stored procedures, and other objects alongside your hypertables. This makes creating and working with hypertables similar to standard PostgreSQL. 1. **Connect to your Tiger Cloud service** In [Tiger Console](https://console.cloud.tigerdata.com/dashboard/services) open an [SQL editor](/docs/build/data-management/run-queries-from-tiger-console). The in-Console editors display the query speed. You can also connect to your service using [psql](/docs/integrate/query-administration/psql). 2. **Create a hypertable for your time-series data using CREATE TABLE.** For [efficient queries](/docs/build/performance-optimization/secondary-indexes) on data in the columnstore, remember to `segmentby` the column you will use most often to filter your data: ```sql CREATE TABLE transactions ( time TIMESTAMPTZ NOT NULL, block_id INT, hash TEXT, size INT, weight INT, is_coinbase BOOLEAN, output_total BIGINT, output_total_usd DOUBLE PRECISION, fee BIGINT, fee_usd DOUBLE PRECISION, details JSONB ) WITH ( tsdb.hypertable, tsdb.segmentby='block_id', tsdb.orderby='time DESC' ); ``` When you create a hypertable using [CREATE TABLE ... WITH ...](/docs/reference/timescaledb/hypertables/create_table), the default partitioning column is automatically the first column with a timestamp data type. Also, TimescaleDB creates a [columnstore policy](/docs/reference/timescaledb/hypercore/add_columnstore_policy) that automatically converts your data to the columnstore, after an interval equal to the value of the [chunk\_interval](/docs/reference/timescaledb/hypertables/set_chunk_time_interval), defined through `after` in the policy. This columnar format enables fast scanning and aggregation, optimizing performance for analytical workloads while also saving significant storage space. In the columnstore conversion, hypertable chunks are compressed by up to 98%, and organized for efficient, large-scale queries. You can customize this policy later using [alter\_job](/docs/reference/timescaledb/jobs-automation/alter_job). However, to change `after` or `created_before`, the compression settings, or the hypertable the policy is acting on, you must [remove the columnstore policy](/docs/reference/timescaledb/hypercore/remove_columnstore_policy) and [add a new one](/docs/reference/timescaledb/hypercore/add_columnstore_policy). You can also manually [convert chunks](/docs/reference/timescaledb/hypercore/convert_to_columnstore) in a hypertable to the columnstore. 3. **Create an index on the `hash` column to make queries for individual transactions faster** ```sql CREATE INDEX hash_idx ON public.transactions USING HASH (hash); ``` 4. **Create an index on the `block_id` column to make block-level queries faster:** When you create a hypertable, it is partitioned on the time column. TimescaleDB automatically creates an index on the time column. However, you'll often filter your time-series data on other columns as well. You use [indexes](/docs/build/performance-optimization/indexing) to improve query performance. ```sql CREATE INDEX block_idx ON public.transactions (block_id); ``` 5. **Create a unique index on the `time` and `hash` columns to prevent duplicate records** ```sql CREATE UNIQUE INDEX time_hash_idx ON public.transactions (time, hash); ``` ## Load financial data The dataset contains around 1.5 million Bitcoin transactions, the trades for five days. It includes information about each transaction, along with the value in [satoshi](https://www.pcmag.com/encyclopedia/term/satoshi). It also states if a trade is a [coinbase](https://www.pcmag.com/encyclopedia/term/coinbase-transaction) transaction, and the reward a coin miner receives for mining the coin. To ingest data into the tables that you created, you need to download the dataset and copy the data to your database. 1. **Download the `bitcoin_sample.zip` file** The file contains a `.csv` file with Bitcoin transactions for the past five days. Download: [bitcoin\_sample.zip](https://assets.timescale.com/docs/downloads/bitcoin-blockchain/bitcoin_sample.zip) 2. **Unzip the `.csv` files** ```bash unzip bitcoin_sample.zip ``` 3. **Navigate to the unzipped folder and connect to your service** In Terminal, navigate to the folder where you unzipped the Bitcoin transactions, then connect to your service using [psql](/docs/integrate/query-administration/psql). 4. **Use the COPY command to transfer data into your service** If the `.csv` files aren't in your current directory, specify the file paths in these commands: ```sql \COPY transactions FROM 'tutorial_bitcoin_sample.csv' CSV HEADER; ``` Because there is over a million rows of data, the `COPY` process could take a few minutes depending on your internet connection and local client resources. ## Connect Grafana to Tiger Cloud To visualize the results of your queries, enable Grafana to read the data in your service: 1. **Log in to Grafana** In your browser, log in to either: * Self-hosted Grafana: at `http://localhost:3000/`. The default credentials are `admin`, `admin`. * Grafana Cloud: use the URL and credentials you set when you created your account. 2. **Add your service as a data source** 1. Open `Connections` > `Data sources`, then click `Add new data source`. 2. Select `PostgreSQL` from the list. 3. Configure the connection: * `Host URL`, `Database name`, `Username`, and `Password`, configure using your [connection details](/docs/integrate/find-connection-details). `Host URL` is in the format `:`. * `TLS/SSL Mode`: select `require`. * `PostgreSQL options`: enable `TimescaleDB`. * Leave the default setting for all other fields. 4. Click `Save & test`. Grafana checks that your details are set correctly. ## Query the data [Section titled “Query the data”](#query-the-data) When you have your dataset loaded, you can start constructing some queries to discover what your data tells you. In this section, you learn how to write queries that answer these questions: * [What are the five most recent coinbase transactions?](#what-are-the-five-most-recent-coinbase-transactions) * [What are the five most recent transactions?](#what-are-the-five-most-recent-transactions) * [What are the five most recent blocks?](#what-are-the-five-most-recent-blocks) ### What are the five most recent coinbase transactions? [Section titled “What are the five most recent coinbase transactions?”](#what-are-the-five-most-recent-coinbase-transactions) [Coinbase](https://www.pcmag.com/encyclopedia/term/coinbase-transaction) transactions are the first transaction in a block, and they include the reward a coin miner receives for mining the coin. To find the most recent coinbase transactions, you can query for transactions where `is_coinbase` is TRUE. You'll notice that the `fee_usd` is $0 for each coinbase transaction because the miner receives the block reward directly without paying a transaction fee. 1. **Connect to the Tiger Cloud service that contains the Bitcoin dataset** 2. **Select the five most recent coinbase transactions** At the psql prompt, use this query: ```sql SELECT time, hash, block_id, fee_usd FROM transactions WHERE is_coinbase IS TRUE ORDER BY time DESC LIMIT 5; ``` 3. **Check the results** The data you get back looks a bit like this: ```sql time | hash | block_id | fee_usd ------------------------+------------------------------------------------------------------+----------+--------- 2023-06-12 23:54:18+00 | 22e4610bc12d482bc49b7a1c5b27ad18df1a6f34256c16ee7e499b511e02d71e | 794111 | 0 2023-06-12 23:53:08+00 | dde958bb96a302fd956ced32d7b98dd9860ff82d569163968ecfe29de457fedb | 794110 | 0 2023-06-12 23:44:50+00 | 75ac1fa7febe1233ee57ca11180124c5ceb61b230cdbcbcba99aecc6a3e2a868 | 794109 | 0 2023-06-12 23:44:14+00 | 1e941d66b92bf0384514ecb83231854246a94c86ff26270fbdd9bc396dbcdb7b | 794108 | 0 2023-06-12 23:41:08+00 | 60ae50447254d5f4561e1c297ee8171bb999b6310d519a0d228786b36c9ffacf | 794107 | 0 (5 rows) ``` ### What are the five most recent transactions? [Section titled “What are the five most recent transactions?”](#what-are-the-five-most-recent-transactions) This dataset contains Bitcoin transactions for the last five days. To find out the most recent transactions in the dataset, you can use a `SELECT` statement. In this case, you want to find transactions that are not coinbase transactions, sort them by time in descending order, and take the top five results. You also want to see the block ID, and the value of the transaction in US Dollars. 1. **Connect to the Tiger Cloud service that contains the Bitcoin dataset** 2. **Select the five most recent non-coinbase transactions** At the psql prompt, use this query: ```sql SELECT time, hash, block_id, fee_usd FROM transactions WHERE is_coinbase IS NOT TRUE ORDER BY time DESC LIMIT 5; ``` 3. **Check the results** The data you get back looks a bit like this: ```sql time | hash | block_id | fee_usd ------------------------+------------------------------------------------------------------+----------+--------- 2023-06-12 23:54:18+00 | 6f709d52e9aa7b2569a7f8c40e7686026ede6190d0532220a73fdac09deff973 | 794111 | 7.614 2023-06-12 23:54:18+00 | ece5429f4a76b1603aecbee31bf3d05f74142a260e4023316250849fe49115ae | 794111 | 9.306 2023-06-12 23:54:18+00 | 54a196398880a7e2e38312d4285fa66b9c7129f7d14dc68c715d783322544942 | 794111 | 13.1928 2023-06-12 23:54:18+00 | 3e83e68735af556d9385427183e8160516fafe2f30f30405711c4d64bf0778a6 | 794111 | 3.5416 2023-06-12 23:54:18+00 | ca20d073b1082d7700b3706fe2c20bc488d2fc4a9bb006eb4449efe3c3fc6b2b | 794111 | 8.6842 (5 rows) ``` ### What are the five most recent blocks? [Section titled “What are the five most recent blocks?”](#what-are-the-five-most-recent-blocks) In this procedure, you use a more complicated query to return the five most recent blocks, and show some additional information about each, including the block weight, number of transactions in each block, and the total block value in US Dollars. 1. **Connect to the Tiger Cloud service that contains the Bitcoin dataset** 2. **Select the five most recent blocks** At the psql prompt, use this query: ```sql WITH recent_blocks AS ( SELECT block_id FROM transactions WHERE is_coinbase IS TRUE ORDER BY time DESC LIMIT 5 ) SELECT t.block_id, count(*) AS transaction_count, SUM(weight) AS block_weight, SUM(output_total_usd) AS block_value_usd FROM transactions t INNER JOIN recent_blocks b ON b.block_id = t.block_id WHERE is_coinbase IS NOT TRUE GROUP BY t.block_id; ``` 3. **Check the results** The data you get back looks a bit like this: ```sql block_id | transaction_count | block_weight | block_value_usd ----------+-------------------+--------------+-------------------- 794108 | 5625 | 3991408 | 65222453.36381342 794111 | 5039 | 3991748 | 5966031.481099684 794109 | 6325 | 3991923 | 5406755.801599815 794110 | 2525 | 3995553 | 177249139.6457974 794107 | 4464 | 3991838 | 107348519.36559173 (5 rows) ``` ## Analyze the data [Section titled “Analyze the data”](#analyze-the-data) In this section, you use [TimescaleDB hyperfunctions](/docs/reference/timescaledb/hyperfunctions) to construct analytical queries that are not possible in standard PostgreSQL. You create continuous aggregates to simplify and speed up your queries, then use them to answer these questions: * Is there any connection between the number of transactions and the transaction fees? * Does the transaction volume affect the BTC-USD rate? * Do more transactions in a block mean the block is more expensive to mine? * What percentage of the average miner's revenue comes from fees compared to block rewards? * How does block weight affect miner fees? * What's the average miner revenue per block? ### Create continuous aggregates [Section titled “Create continuous aggregates”](#create-continuous-aggregates) You can use [continuous aggregates](/docs/learn/continuous-aggregates) to simplify and speed up your queries. For this analysis, you need three continuous aggregates focusing on three aspects of the dataset: Bitcoin transactions, blocks, and coinbase transactions. In each continuous aggregate definition, the `time_bucket()` function controls how large the time buckets are. The examples all use 1-hour time buckets. 1. **Connect to the Tiger Cloud service that contains the Bitcoin dataset** 2. **Create a continuous aggregate called one\_hour\_transactions** This view holds aggregated data about each hour of transactions: ```sql CREATE MATERIALIZED VIEW one_hour_transactions WITH (timescaledb.continuous) AS SELECT time_bucket('1 hour', time) AS bucket, count(*) AS tx_count, sum(fee) AS total_fee_sat, sum(fee_usd) AS total_fee_usd, stats_agg(fee) AS stats_fee_sat, avg(size) AS avg_tx_size, avg(weight) AS avg_tx_weight, count( CASE WHEN (fee > output_total) THEN hash ELSE NULL END) AS high_fee_count FROM transactions WHERE (is_coinbase IS NOT TRUE) GROUP BY bucket; ``` 3. **Add a refresh policy to keep the continuous aggregate up-to-date** ```sql SELECT add_continuous_aggregate_policy('one_hour_transactions', start_offset => INTERVAL '3 hours', end_offset => INTERVAL '1 hour', schedule_interval => INTERVAL '1 hour'); ``` 4. **Create a continuous aggregate called one\_hour\_blocks** This view holds aggregated data about all the blocks that were mined each hour: ```sql CREATE MATERIALIZED VIEW one_hour_blocks WITH (timescaledb.continuous) AS SELECT time_bucket('1 hour', time) AS bucket, block_id, count(*) AS tx_count, sum(fee) AS block_fee_sat, sum(fee_usd) AS block_fee_usd, stats_agg(fee) AS stats_tx_fee_sat, avg(size) AS avg_tx_size, avg(weight) AS avg_tx_weight, sum(size) AS block_size, sum(weight) AS block_weight, max(size) AS max_tx_size, max(weight) AS max_tx_weight, min(size) AS min_tx_size, min(weight) AS min_tx_weight FROM transactions WHERE is_coinbase IS NOT TRUE GROUP BY bucket, block_id; ``` 5. **Add a refresh policy to keep the continuous aggregate up to date** ```sql SELECT add_continuous_aggregate_policy('one_hour_blocks', start_offset => INTERVAL '3 hours', end_offset => INTERVAL '1 hour', schedule_interval => INTERVAL '1 hour'); ``` 6. **Create a continuous aggregate called one\_hour\_coinbase** This view holds aggregated data about all the transactions that miners received as rewards each hour: ```sql CREATE MATERIALIZED VIEW one_hour_coinbase WITH (timescaledb.continuous) AS SELECT time_bucket('1 hour', time) AS bucket, count(*) AS tx_count, stats_agg(output_total, output_total_usd) AS stats_miner_revenue, min(output_total) AS min_miner_revenue, max(output_total) AS max_miner_revenue FROM transactions WHERE is_coinbase IS TRUE GROUP BY bucket; ``` 7. **Add a refresh policy to keep the continuous aggregate up to date** ```sql SELECT add_continuous_aggregate_policy('one_hour_coinbase', start_offset => INTERVAL '3 hours', end_offset => INTERVAL '1 hour', schedule_interval => INTERVAL '1 hour'); ``` ### Is there any connection between the number of transactions and the transaction fees? [Section titled “Is there any connection between the number of transactions and the transaction fees?”](#is-there-any-connection-between-the-number-of-transactions-and-the-transaction-fees) Transaction fees are a major concern for blockchain users. If a blockchain is too expensive, you might not want to use it. This query shows you whether there's any correlation between the number of Bitcoin transactions and the fees. The time range for this analysis is the last 2 days. If you choose to visualize the query in Grafana, you can see the average transaction volume and the average fee per transaction, over time. These trends might help you decide whether to submit a transaction now or wait a few days for fees to decrease. 1. **Connect to the Tiger Cloud service that contains the Bitcoin dataset** 2. **Query average transaction volume and fees from the one\_hour\_transactions continuous aggregate** At the psql prompt, use this query: ```sql SELECT bucket AS "time", tx_count as "tx volume", average(stats_fee_sat) as fees FROM one_hour_transactions WHERE bucket > date_add('2023-11-22 00:00:00+00', INTERVAL '-2 days') ORDER BY 1; ``` The data you get back looks a bit like this: ```sql time | tx volume | fees ------------------------+-----------+-------------------- 2023-11-20 01:00:00+00 | 2602 | 105963.45810914681 2023-11-20 02:00:00+00 | 33037 | 26686.814117504615 2023-11-20 03:00:00+00 | 42077 | 22875.286546094067 2023-11-20 04:00:00+00 | 46021 | 20280.843180287262 2023-11-20 05:00:00+00 | 20828 | 24694.472969080085 ... ``` 3. **Visualize this in Grafana** 1. From the `Dashboards` page, click `New` and select `New dashboard`. 2. Click `Add visualization`, then select the data source that connects to your Tiger Cloud service. 3. In the `Queries` section, change the `Format` to `Time series` and select `Code`. 4. Type the query from the previous step and click `Run query`. ![Visualizing number of transactions and fees](/docs/_astro/grafana-transactions-fees.Bj61rNYp_1nDsrd.webp) ### Does the transaction volume affect the BTC-USD rate? [Section titled “Does the transaction volume affect the BTC-USD rate?”](#does-the-transaction-volume-affect-the-btc-usd-rate) In cryptocurrency trading, there's a lot of speculation. You can adopt a data-based trading strategy by looking at correlations between blockchain metrics, such as transaction volume and the current exchange rate between Bitcoin and US Dollars. If you choose to visualize the query in Grafana, you can see the average transaction volume, along with the BTC to US Dollar conversion rate. 1. **Connect to the Tiger Cloud service that contains the Bitcoin dataset** 2. **Query the trading volume and the BTC to US Dollar exchange rate** At the psql prompt, use this query: ```sql SELECT bucket AS "time", tx_count as "tx volume", total_fee_usd / (total_fee_sat*0.00000001) AS "btc-usd rate" FROM one_hour_transactions WHERE bucket > date_add('2023-11-22 00:00:00+00', INTERVAL '-2 days') ORDER BY 1; ``` The data you get back looks a bit like this: ```sql time | tx volume | btc-usd rate ------------------------+-----------+-------------------- 2023-06-13 08:00:00+00 | 20063 | 25975.888587931426 2023-06-13 09:00:00+00 | 16984 | 25976.00446352126 2023-06-13 10:00:00+00 | 15856 | 25975.988587014584 2023-06-13 11:00:00+00 | 24967 | 25975.89166787936 2023-06-13 12:00:00+00 | 8575 | 25976.004209699528 ... ``` 3. **Visualize this in Grafana** 1. From the `Dashboards` page, click `New` and select `New dashboard`. 2. Click `Add visualization`, then select the data source that connects to your Tiger Cloud service. 3. In the `Queries` section, change the `Format` to `Time series` and select `Code`. 4. Type the query from the previous step and click `Run query`. 5. Under the panel options on the right, click `Add field override` > `Fields with name`, then choose `btc-usd rate` in the dropdown. 6. Click `Add override property`, then select `Axis > Placement` and click `Right`. ![Visualizing transaction volume and BTC-USD conversion rate](/docs/_astro/grafana-volume-rate.PGhhiyzx_26lRzL.webp) ### Do more transactions in a block mean the block is more expensive to mine? [Section titled “Do more transactions in a block mean the block is more expensive to mine?”](#do-more-transactions-in-a-block-mean-the-block-is-more-expensive-to-mine) The number of transactions in a block can influence the overall block mining fee. For this analysis, a larger time frame is required, so increase the analyzed time range to 5 days. If you choose to visualize the query in Grafana, you can see that the more transactions in a block, the higher the mining fee becomes. 1. **Connect to the Tiger Cloud service that contains the Bitcoin dataset** 2. **Query the number of transactions in a block, compared to the mining fee** At the psql prompt, use this query: ```sql SELECT bucket as "time", avg(tx_count) AS transactions, avg(block_fee_sat)*0.00000001 AS "mining fee" FROM one_hour_blocks WHERE bucket > date_add('2023-11-22 00:00:00+00', INTERVAL '-5 days') GROUP BY bucket ORDER BY 1; ``` The data you get back looks a bit like this: ```sql time | transactions | mining fee ------------------------+-----------------------+------------------------ 2023-06-10 08:00:00+00 | 2322.2500000000000000 | 0.29221418750000000000 2023-06-10 09:00:00+00 | 3305.0000000000000000 | 0.50512649666666666667 2023-06-10 10:00:00+00 | 3011.7500000000000000 | 0.44783255750000000000 2023-06-10 11:00:00+00 | 2874.7500000000000000 | 0.39303009500000000000 2023-06-10 12:00:00+00 | 2339.5714285714285714 | 0.25590717142857142857 ``` 3. **Visualize this in Grafana** 1. From the `Dashboards` page, click `New` and select `New dashboard`. 2. Click `Add visualization`, then select the data source that connects to your Tiger Cloud service. 3. In the `Queries` section, change the `Format` to `Time series` and select `Code`. 4. Type the query from the previous step and click `Run query`. 5. Under the panel options on the right, click `Add field override` > `Fields with name`, then choose `mining fee` in the dropdown. 6. Click `Add override property`, then select `Axis > Placement` and click `Right`. ![Visualizing transactions in a block and the mining fee](/docs/_astro/grafana-transactions-miningfee.DO-1-nIn_2Tao6.webp) You can extend this analysis to find if there is the same correlation between block weight and mining fee. More transactions should increase the block weight, and boost the miner fee as well. If you choose to visualize the query in Grafana, you can see the same kind of high correlation between block weight and mining fee. The relationship weakens when the block weight gets close to its maximum value, which is 4 million weight units, in which case it's impossible for a block to include more transactions. 1. **Connect to the Tiger Cloud service that contains the Bitcoin dataset** 2. **Query block weight compared to the mining fee** At the psql prompt, use this query: ```sql SELECT bucket as "time", avg(block_weight) as "block weight", avg(block_fee_sat*0.00000001) as "mining fee" FROM one_hour_blocks WHERE bucket > date_add('2023-11-22 00:00:00+00', INTERVAL '-5 days') group by bucket ORDER BY 1; ``` The data you get back looks a bit like this: ```sql time | block weight | mining fee ------------------------+----------------------+------------------------ 2023-06-10 08:00:00+00 | 3992809.250000000000 | 0.29221418750000000000 2023-06-10 09:00:00+00 | 3991766.333333333333 | 0.50512649666666666667 2023-06-10 10:00:00+00 | 3992918.250000000000 | 0.44783255750000000000 2023-06-10 11:00:00+00 | 3991873.000000000000 | 0.39303009500000000000 2023-06-10 12:00:00+00 | 3992934.000000000000 | 0.25590717142857142857 ... ``` 3. **Visualize this in Grafana** 1. From the `Dashboards` page, click `New` and select `New dashboard`. 2. Click `Add visualization`, then select the data source that connects to your Tiger Cloud service. 3. In the `Queries` section, change the `Format` to `Time series` and select `Code`. 4. Type the query from the previous step and click `Run query`. 5. Under the panel options on the right, click `Add field override` > `Fields with name`, then choose `mining fee` in the dropdown. 6. Click `Add override property`, then select `Axis > Placement` and click `Right`. ![Visualizing blockweight and the mining fee](/docs/_astro/grafana-blockweight-miningfee.BqaTTB-h_28gMfy.webp) ### What percentage of the average miner's revenue comes from fees compared to block rewards? [Section titled “What percentage of the average miner's revenue comes from fees compared to block rewards?”](#what-percentage-of-the-average-miners-revenue-comes-from-fees-compared-to-block-rewards) In the previous queries, you saw that mining fees are higher when block weights and transaction volumes are higher. This query analyzes the data from a different perspective. Miner revenue is not only made up of miner fees, it also includes block rewards for mining a new block. This reward is currently 6.25 BTC, and it gets halved every four years. This query looks at how much of a miner's revenue comes from fees, compares to block rewards. If you choose to visualize the query in Grafana, you can see that most miner revenue actually comes from block rewards. Fees never account for more than a few percentage points of overall revenue. 1. **Connect to the Tiger Cloud service that contains the Bitcoin dataset** 2. **Query coinbase transactions with block fees and rewards** At the psql prompt, use this query: ```sql WITH coinbase AS ( SELECT block_id, output_total AS coinbase_tx FROM transactions WHERE is_coinbase IS TRUE and time > date_add('2023-11-22 00:00:00+00', INTERVAL '-5 days') ) SELECT bucket as "time", avg(block_fee_sat)*0.00000001 AS "fees", FIRST((c.coinbase_tx - block_fee_sat), bucket)*0.00000001 AS "reward" FROM one_hour_blocks b INNER JOIN coinbase c ON c.block_id = b.block_id GROUP BY bucket ORDER BY 1; ``` The data you get back looks a bit like this: ```sql time | fees | reward ------------------------+------------------------+------------ 2023-06-10 08:00:00+00 | 0.28247062857142857143 | 6.25000000 2023-06-10 09:00:00+00 | 0.50512649666666666667 | 6.25000000 2023-06-10 10:00:00+00 | 0.44783255750000000000 | 6.25000000 2023-06-10 11:00:00+00 | 0.39303009500000000000 | 6.25000000 2023-06-10 12:00:00+00 | 0.25590717142857142857 | 6.25000000 ... ``` 3. **Visualize this in Grafana** 1. From the `Dashboards` page, click `New` and select `New dashboard`. 2. Click `Add visualization`, then select the data source that connects to your Tiger Cloud service. 3. In the `Queries` section, change the `Format` to `Time series` and select `Code`. 4. Type the query from the previous step and click `Run query`. 5. In the options panel, in the `Graph styles` section, for `Stack series` select `100%`. ![Visualizing coinbase revenue sources](/docs/_astro/grafana-coinbase-revenue.CcRIpau8_Z1zCAEs.webp) ### How does block weight affect miner fees? [Section titled “How does block weight affect miner fees?”](#how-does-block-weight-affect-miner-fees) You've already found that more transactions in a block mean it's more expensive to mine. In this query, you ask if the same is true for block weights? The more transactions a block has, the larger its weight, so the block weight and mining fee should be tightly correlated. This query uses a 12-hour moving average to calculate the block weight and block mining fee over time. If you choose to visualize the query in Grafana, you can see that the block weight and block mining fee are tightly connected. In practice, you can also see the four million weight units size limit. This means that there's still room to grow for individual blocks, and they could include even more transactions. 1. **Connect to the Tiger Cloud service that contains the Bitcoin dataset** 2. **Query block weight with block fees and rewards** At the psql prompt, use this query: ```sql WITH stats AS ( SELECT bucket, stats_agg(block_weight, block_fee_sat) AS block_stats FROM one_hour_blocks WHERE bucket > date_add('2023-11-22 00:00:00+00', INTERVAL '-5 days') GROUP BY bucket ) SELECT bucket as "time", average_y(rolling(block_stats) OVER (ORDER BY bucket RANGE '12 hours' PRECEDING)) AS "block weight", average_x(rolling(block_stats) OVER (ORDER BY bucket RANGE '12 hours' PRECEDING))*0.00000001 AS "mining fee" FROM stats ORDER BY 1; ``` The data you get back looks a bit like this: ```sql time | block weight | mining fee ------------------------+--------------------+--------------------- 2023-06-10 09:00:00+00 | 3991766.3333333335 | 0.5051264966666666 2023-06-10 10:00:00+00 | 3992424.5714285714 | 0.47238710285714286 2023-06-10 11:00:00+00 | 3992224 | 0.44353000909090906 2023-06-10 12:00:00+00 | 3992500.111111111 | 0.37056557222222225 2023-06-10 13:00:00+00 | 3992446.65 | 0.39728022799999996 ... ``` 3. **Visualize this in Grafana** 1. From the `Dashboards` page, click `New` and select `New dashboard`. 2. Click `Add visualization`, then select the data source that connects to your Tiger Cloud service. 3. In the `Queries` section, change the `Format` to `Time series` and select `Code`. 4. Type the query from the previous step and click `Run query`. 5. Under the panel options on the right, click `Add field override` > `Fields with name`, then choose `mining fee` in the dropdown. 6. Click `Add override property`, then select `Axis > Placement` and click `Right`. ![Visualizing block weight and mining fees](/docs/_astro/grafana-blockweight-rewards.C0vgyLN-_Z10Rj4p.webp) ### What's the average miner revenue per block? [Section titled “What's the average miner revenue per block?”](#whats-the-average-miner-revenue-per-block) In this final query, you analyze how much revenue miners actually generate by mining a new block on the blockchain, including fees and block rewards. To make the analysis more interesting, add the Bitcoin to US Dollar exchange rate, and increase the time range. 1. **Connect to the Tiger Cloud service that contains the Bitcoin dataset** 2. **Query average miner revenue per block with a 12-hour moving average** At the psql prompt, use this query: ```sql SELECT bucket as "time", average_y(rolling(stats_miner_revenue) OVER (ORDER BY bucket RANGE '12 hours' PRECEDING))*0.00000001 AS "revenue in BTC", average_x(rolling(stats_miner_revenue) OVER (ORDER BY bucket RANGE '12 hours' PRECEDING)) AS "revenue in USD" FROM one_hour_coinbase WHERE bucket > date_add('2023-11-22 00:00:00+00', INTERVAL '-5 days') ORDER BY 1; ``` The data you get back looks a bit like this: ```sql time | revenue in BTC | revenue in USD ------------------------+--------------------+-------------------- 2023-06-09 14:00:00+00 | 6.6732841925 | 176922.1133 2023-06-09 15:00:00+00 | 6.785046736363636 | 179885.1576818182 2023-06-09 16:00:00+00 | 6.7252952905 | 178301.02735000002 2023-06-09 17:00:00+00 | 6.716377454814815 | 178064.5978074074 2023-06-09 18:00:00+00 | 6.7784206471875 | 179709.487309375 ... ``` 3. **Visualize this in Grafana** 1. From the `Dashboards` page, click `New` and select `New dashboard`. 2. Click `Add visualization`, then select the data source that connects to your Tiger Cloud service. 3. In the `Queries` section, change the `Format` to `Time series` and select `Code`. 4. Type the query from the previous step and click `Run query`. 5. Under the panel options on the right, click `Add field override` > `Fields with name`, then choose `revenue in USD` in the dropdown. 6. Click `Add override property`, then select `Axis > Placement` and click `Right`. ![Visualizing block revenue over time](/docs/_astro/grafana-blockweight-revenue.YbvMr5KI_2qie38.webp) # Analyze energy consumption > Get insights from energy consumption data with Grafana and Tiger Cloud Energy providers understand that customers tend to lose patience when there is not enough power for them to complete day-to-day activities. Task one is keeping the lights on. If you are transitioning to renewable energy, it helps to know when you need to produce energy so you can choose a suitable energy source. Real-time analytics refers to the process of collecting, analyzing, and interpreting data instantly as it is generated. This approach enables you to track and monitor activity, make the decisions based on real-time insights on data stored in a Tiger Cloud service and keep those lights on. [Grafana](https://grafana.com/docs/) is a popular data visualization tool that enables you to create customizable dashboards and effectively monitor your systems and applications. ![Grafana dashboard showing energy consumption data from a continuous aggregate](/docs/_astro/use-case-rta-grafana-timescale-energy-cagg.BnE5uNhv_Eg8x0.webp) This page shows you how to integrate Grafana with a Tiger Cloud service and make insights based on visualization of data optimized for size and speed in the columnstore. ## Prerequisites for this tutorial To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service). These steps use Tiger Cloud, but the same approach applies to a [self-hosted TimescaleDB](/docs/get-started/choose-your-path/install-timescaledb) instance. - A [self-managed Grafana](https://grafana.com/get/?tab=self-managed) instance, or a [Grafana Cloud](https://grafana.com/get/) account. ## Optimize time-series data in hypertables Hypertables are PostgreSQL tables in TimescaleDB that automatically partition your time-series data by time. Time-series data represents the way a system, process, or behavior changes over time. Hypertables enable TimescaleDB to work efficiently with time-series data. Each hypertable is made up of child tables called chunks. Each chunk is assigned a range of time, and only contains data from that range. When you run a query, TimescaleDB identifies the correct chunk and runs the query on it, instead of going through the entire table. [Hypercore](/docs/learn/columnar-storage/understand-hypercore) is the hybrid row-columnar storage engine in TimescaleDB used by hypertables. Traditional databases force a trade-off between fast inserts (row-based storage) and efficient analytics (columnar storage). Hypercore eliminates this trade-off, allowing real-time analytics without sacrificing transactional capabilities. Hypercore dynamically stores data in the most efficient format for its lifecycle: ![Move from rowstore to columstore in hypercore](/docs/_astro/hypercore_intro.DmLiqZGV_1VM5z7.webp) ![Move from rowstore to columstore in hypercore](/docs/_astro/hypercore_intro-dark.qPmjkBqW_19V2rF.webp) * **Row-based storage for recent data**: the most recent chunk (and possibly more) is always stored in the rowstore, ensuring fast inserts, updates, and low-latency single record queries. Additionally, row-based storage is used as a writethrough for inserts and updates to columnar storage. * **Columnar storage for analytical performance**: chunks are automatically compressed into the columnstore, optimizing storage efficiency and accelerating analytical queries. Unlike traditional columnar databases, hypercore allows data to be inserted or modified at any stage, making it a flexible solution for both high-ingest transactional workloads and real-time analytics, within a single database. Because TimescaleDB is 100% PostgreSQL, you can use all the standard PostgreSQL tables, indexes, stored procedures, and other objects alongside your hypertables. This makes creating and working with hypertables similar to standard PostgreSQL. 1. **Import time-series data into a hypertable** 1. Unzip [metrics.csv.gz](https://assets.timescale.com/docs/downloads/metrics.csv.gz) to a ``. This test dataset contains energy consumption data. To import up to 100GB of data directly from your current PostgreSQL based database, [migrate with downtime](/docs/migrate/migrate-with-downtime). To seamlessly import 100GB-10TB+ of data, use the [live migration](/docs/migrate/live-migration) tooling supplied by Tiger Data. To add data from non-PostgreSQL data sources, see [Import and ingest data](/docs/migrate/import-terminal). 2. In the terminal, navigate to `` and update the following string with [your connection details](/docs/integrate/find-connection-details) to connect to your service. ```bash psql -d "postgres://:@:/?sslmode=require" ``` 3. Create an optimized hypertable for your time-series data. Create a [hypertable](/docs/learn/hypertables/understand-hypertables) with [hypercore](/docs/learn/columnar-storage/understand-hypercore) enabled by default for your time-series data using [CREATE TABLE](/docs/reference/timescaledb). For [efficient queries](/docs/build/performance-optimization/secondary-indexes) on data in the columnstore, remember to `segmentby` the column you will use most often to filter your data. In your SQL client, run the following command: ```sql CREATE TABLE "metrics"( created timestamp with time zone default now() not null, type_id integer not null, value double precision not null ) WITH ( tsdb.hypertable, tsdb.segmentby = 'type_id', tsdb.orderby = 'created DESC' ); ``` When you create a hypertable using [CREATE TABLE ... WITH ...](/docs/reference/timescaledb/hypertables/create_table), the default partitioning column is automatically the first column with a timestamp data type. Also, TimescaleDB creates a [columnstore policy](/docs/reference/timescaledb/hypercore/add_columnstore_policy) that automatically converts your data to the columnstore, after an interval equal to the value of the [chunk\_interval](/docs/reference/timescaledb/hypertables/set_chunk_time_interval), defined through `after` in the policy. This columnar format enables fast scanning and aggregation, optimizing performance for analytical workloads while also saving significant storage space. In the columnstore conversion, hypertable chunks are compressed by up to 98%, and organized for efficient, large-scale queries. You can customize this policy later using [alter\_job](/docs/reference/timescaledb/jobs-automation/alter_job). However, to change `after` or `created_before`, the compression settings, or the hypertable the policy is acting on, you must [remove the columnstore policy](/docs/reference/timescaledb/hypercore/remove_columnstore_policy) and [add a new one](/docs/reference/timescaledb/hypercore/add_columnstore_policy). You can also manually [convert chunks](/docs/reference/timescaledb/hypercore/convert_to_columnstore) in a hypertable to the columnstore. 4. Upload the dataset to your service: ```sql \COPY metrics FROM metrics.csv CSV; ``` 2. **Have a quick look at your data** You query hypertables in exactly the same way as you would a relational PostgreSQL table. Use one of the following SQL editors to run a query and see the data you uploaded: * **Data view**: write queries, visualize data, and share your results in [Tiger Console](https://console.cloud.tigerdata.com/dashboard/services?popsql) for all your Tiger Cloud services. * **SQL editor**: write, fix, and organize SQL faster and more accurately in [Tiger Console](https://console.cloud.tigerdata.com/dashboard/services) for a Tiger Cloud service. * **psql**: easily run queries on your Tiger Cloud services or self-hosted TimescaleDB deployment from Terminal. ```sql SELECT time_bucket('1 day', created, 'Europe/Berlin') AS "time", round((last(value, created) - first(value, created)) * 100.) / 100. AS value FROM metrics WHERE type_id = 5 GROUP BY 1; ``` On this amount of data, this query on data in the rowstore takes about 3.6 seconds. You see something like: | Time | value | | ---------------------- | ----- | | 2023-05-29 22:00:00+00 | 23.1 | | 2023-05-28 22:00:00+00 | 19.5 | | 2023-05-30 22:00:00+00 | 25 | | 2023-05-31 22:00:00+00 | 8.1 | ## Write fast analytical queries [Section titled “Write fast analytical queries”](#write-fast-analytical-queries) Aggregation is a way of combining data to get insights from it. Average, sum, and count are all examples of simple aggregates. However, with large amounts of data aggregation slows things down, quickly. Continuous aggregates are a kind of hypertable that is refreshed automatically in the background as new data is added, or old data is modified. Changes to your dataset are tracked, and the hypertable behind the continuous aggregate is automatically updated in the background. By default, querying continuous aggregates provides you with real-time data. Pre-aggregated data from the materialized view is combined with recent data that hasn't been aggregated yet. This gives you up-to-date results on every query. You create continuous aggregates on uncompressed data in high-performance storage. They continue to work on [data in the columnstore](/docs/learn/tiger-cloud/tiger-cloud-essentials) and [rarely accessed data in tiered storage](/docs/build/data-management/storage/query-tiered-data). You can even create [continuous aggregates on top of your continuous aggregates](/docs/learn/continuous-aggregates/hierarchical-continuous-aggregates). 1. **Monitor energy consumption on a day-to-day basis** 1. Create a continuous aggregate called `kwh_day_by_day` for energy consumption: ```sql CREATE MATERIALIZED VIEW kwh_day_by_day(time, value) with (timescaledb.continuous) as SELECT time_bucket('1 day', created, 'Europe/Berlin') AS "time", round((last(value, created) - first(value, created)) * 100.) / 100. AS value FROM metrics WHERE type_id = 5 GROUP BY 1; ``` 2. Add a refresh policy to keep `kwh_day_by_day` up-to-date: ```sql SELECT add_continuous_aggregate_policy('kwh_day_by_day', start_offset => NULL, end_offset => INTERVAL '1 hour', schedule_interval => INTERVAL '1 hour'); ``` 2. **Monitor energy consumption on an hourly basis** 1. Create a continuous aggregate called `kwh_hour_by_hour` for energy consumption: ```sql CREATE MATERIALIZED VIEW kwh_hour_by_hour(time, value) with (timescaledb.continuous) as SELECT time_bucket('01:00:00', metrics.created, 'Europe/Berlin') AS "time", round((last(value, created) - first(value, created)) * 100.) / 100. AS value FROM metrics WHERE type_id = 5 GROUP BY 1; ``` 2. Add a refresh policy to keep the continuous aggregate up-to-date: ```sql SELECT add_continuous_aggregate_policy('kwh_hour_by_hour', start_offset => NULL, end_offset => INTERVAL '1 hour', schedule_interval => INTERVAL '1 hour'); ``` 3. **Analyze your data** Now you have made continuous aggregates, you can use them to perform analytics on your data. For example, to see how average energy consumption changes during weekdays over the last year, run the following query: ```sql WITH per_day AS ( SELECT time, value FROM kwh_day_by_day WHERE "time" at time zone 'Europe/Berlin' > date_trunc('month', time) - interval '1 year' ORDER BY 1 ), daily AS ( SELECT to_char(time, 'Dy') as day, value FROM per_day ), percentile AS ( SELECT day, approx_percentile(0.50, percentile_agg(value)) as value FROM daily GROUP BY 1 ORDER BY 1 ) SELECT d.day, d.ordinal, pd.value FROM unnest(array['Sun', 'Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat']) WITH ORDINALITY AS d(day, ordinal) LEFT JOIN percentile pd ON lower(pd.day) = lower(d.day); ``` You see something like: | day | ordinal | value | | --- | ------- | ------------------ | | Mon | 2 | 23.08078714975423 | | Sun | 1 | 19.511430831944395 | | Tue | 3 | 25.003118897837307 | | Wed | 4 | 8.09300571759772 | ## Connect Grafana to Tiger Cloud To visualize the results of your queries, enable Grafana to read the data in your service: 1. **Log in to Grafana** In your browser, log in to either: * Self-hosted Grafana: at `http://localhost:3000/`. The default credentials are `admin`, `admin`. * Grafana Cloud: use the URL and credentials you set when you created your account. 2. **Add your service as a data source** 1. Open `Connections` > `Data sources`, then click `Add new data source`. 2. Select `PostgreSQL` from the list. 3. Configure the connection: * `Host URL`, `Database name`, `Username`, and `Password`, configure using your [connection details](/docs/integrate/find-connection-details). `Host URL` is in the format `:`. * `TLS/SSL Mode`: select `require`. * `PostgreSQL options`: enable `TimescaleDB`. * Leave the default setting for all other fields. 4. Click `Save & test`. Grafana checks that your details are set correctly. ## Visualize energy consumption [Section titled “Visualize energy consumption”](#visualize-energy-consumption) A Grafana dashboard represents a view into the performance of a system, and each dashboard consists of one or more panels, which represent information about a specific metric related to that system. To visually monitor the volume of energy consumption over time: 1. **Create the dashboard** 1. On the `Dashboards` page, click `New` and select `New dashboard`. 2. Click `Add visualization`, then select the data source that connects to your Tiger Cloud service and the `Bar chart` visualization type in the top right. ![Configuring a Grafana dashboard with a TimescaleDB data source](/docs/_astro/use-case-rta-grafana-timescale-configure-dashboard.PNpiH0JO_204Gcy.webp) 3. In the `Queries` section, select `Code` on the right, then run the following query based on your continuous aggregate: ```sql WITH per_hour AS ( SELECT time, value FROM kwh_hour_by_hour WHERE "time" at time zone 'Europe/Berlin' > date_trunc('month', time) - interval '1 year' ORDER BY 1 ), hourly AS ( SELECT extract(HOUR FROM time) * interval '1 hour' as hour, value FROM per_hour ) SELECT hour, approx_percentile(0.50, percentile_agg(value)) as median, max(value) as maximum FROM hourly GROUP BY 1 ORDER BY 1; ``` This query averages the results for households in a specific time zone by hour and orders them by time. Because you use a continuous aggregate, this data is always correct in real time. ![Grafana dashboard showing energy consumption data from a continuous aggregate](/docs/_astro/use-case-rta-grafana-timescale-energy-cagg.BnE5uNhv_Eg8x0.webp) You see that energy consumption is highest in the evening and at breakfast time. You also know that the wind drops off in the evening. This data proves that you need to supply a supplementary power source for peak times, or plan to store energy during the day for peak times. 2. **Click `Save dashboard`** You have integrated Grafana with a Tiger Cloud service and made insights based on visualization of your data. # Analyze application events with UUIDv7 > Get started with TimescaleDB using application event data and UUIDv7 identifiers, with no separate timestamp column Get started with TimescaleDB using application event data that leverages UUIDv7 identifiers. This example demonstrates how to handle event logging and analytics using time-embedded UUIDs for partitioning, with no separate timestamp column needed. ## What you learn [Section titled “What you learn”](#what-you-learn) * Using UUIDv7 for time-ordered unique identifiers. * Efficient time-range queries with `to_uuidv7_boundary()`. * Session tracking and user analytics. * Event funnel and conversion analysis. ## Prerequisites for this tutorial To follow these steps, you'll need: * Docker installed * The `psql` PostgreSQL client ## Set up and query the data [Section titled “Set up and query the data”](#set-up-and-query-the-data) 1. **Start TimescaleDB** Start TimescaleDB using one of these options. * One-line install (recommended) The easiest way to get started: Important This script is intended for local development and testing only. Do not use it for production deployments. For production-ready installation options, see the [TimescaleDB installation guide](/docs/get-started/choose-your-path/install-timescaledb). On Linux or Mac: ```sh curl -sL https://tsdb.co/start-local | sh ``` This command: * Downloads and starts TimescaleDB, if not already downloaded. * Exposes PostgreSQL on port `6543`, a non-standard port that avoids conflicts with other PostgreSQL instances on port 5432. * Automatically tunes settings for your environment using timescaledb-tune. * Sets up a persistent data volume. * Manual Docker command Run TimescaleDB directly with Docker. This option also works on Windows: ```bash docker run -d --name timescaledb \ -p 6543:5432 \ -e POSTGRES_PASSWORD=password \ timescale/timescaledb-ha:pg18 ``` This example uses port `6543`, mapped to container port 5432, to avoid conflicts if you have other PostgreSQL instances running on the standard port 5432. Wait about 1 to 2 minutes for TimescaleDB to download and initialize. 2. **Connect to TimescaleDB** Connect using `psql`: ```bash psql -h localhost -p 6543 -U postgres # When prompted, enter password: password ``` You should see the PostgreSQL prompt. Verify TimescaleDB is installed: ```sql SELECT extname, extversion FROM pg_extension WHERE extname = 'timescaledb'; ``` Expected output: ```plaintext extname | extversion -------------+------------ timescaledb | 2.x.x ``` Tips Prefer a GUI? If you would rather use a graphical tool instead of the command line, you can download [pgAdmin](https://www.pgadmin.org/download/) and connect to TimescaleDB using the same connection details (host: `localhost`, port: `6543`, user: `postgres`, password: `password`). 3. **Create the schema** Create the optimized hypertable by running this SQL in your `psql` session: ```sql -- Create the app_events table with UUIDv7 partitioning -- Note: No separate timestamp column needed - the timestamp is embedded in the UUIDv7! -- Note: No PRIMARY KEY to allow direct compress during COPY CREATE TABLE IF NOT EXISTS app_events ( event_id UUID NOT NULL, user_id UUID NOT NULL, session_id UUID NOT NULL, event_type TEXT NOT NULL, event_name TEXT, device_type TEXT, country_code TEXT, category TEXT, page_path TEXT, referrer TEXT, viewport_width INTEGER, element_id TEXT, position_x INTEGER, position_y INTEGER, product_id TEXT, quantity INTEGER, revenue_cents INTEGER ) WITH ( tsdb.hypertable, tsdb.partition_column = 'event_id', tsdb.segmentby = 'user_id' ); -- Create index on event_id for lookups (not unique to allow direct compress) CREATE INDEX idx_app_events_event_id ON app_events(event_id); ``` This creates an `app_events` table with: * UUIDv7 partitioning on `event_id` (time is embedded in the UUID). * Segmentation by `user_id` for optimal compression. * No separate timestamp column needed. 4. **Load the sample data** First, download and decompress the sample data: ```bash # Download the sample data wget https://assets.timescale.com/timescaledb-datasets/events_uuid.csv.gz # Decompress the CSV file gunzip events_uuid.csv.gz # This will create events_uuid.csv ready for loading ``` Load the data using one of these approaches. * Direct to columnstore (recommended) This approach writes data directly to the columnstore, bypassing the rowstore entirely. You get instant analytical performance. From the command line: ```bash psql -h localhost -p 6543 -U postgres \ -v ON_ERROR_STOP=1 \ -c "SET timescaledb.enable_direct_compress_copy = on; COPY app_events FROM STDIN WITH (FORMAT csv, HEADER true);" \ < events_uuid.csv ``` This command reads the CSV file from your local filesystem and pipes it to PostgreSQL, which loads it directly into the columnstore. Verify the data loaded: ```sql SELECT COUNT(*) FROM app_events; ``` * Standard COPY (fallback) This approach loads data into the rowstore first. Data is converted to the columnstore by a background policy (12 to 24 hours) for faster querying. From the command line: ```bash psql -h localhost -p 6543 -U postgres \ -v ON_ERROR_STOP=1 \ -c "COPY app_events FROM STDIN WITH (FORMAT csv, HEADER true);" \ < events_uuid.csv ``` Verify the data loaded: ```sql SELECT COUNT(*) FROM app_events; ``` If you loaded data using standard copy, a background process converts your rowstore data to the columnstore in 12 to 24 hours. To get the best query performance immediately, you can convert it manually: ```sql DO $$ DECLARE ch TEXT; BEGIN FOR ch IN SELECT show_chunks('app_events') LOOP CALL convert_to_columnstore(ch); END LOOP; END $$; ``` 5. **Run sample queries** Now explore the data with some analytical queries. Run these in your `psql` session. Query 1: efficient time range query (chunk pruning). ```sql -- Uses the boundary function for chunk exclusion \timing on SELECT COUNT(*), event_type FROM app_events WHERE event_id >= to_uuidv7_boundary(now() - interval '7 days') GROUP BY event_type; ``` Why this is fast: the [`to_uuidv7_boundary()`](/docs/reference/timescaledb/uuid-functions/to_uuidv7_boundary) function creates a UUID boundary value that TimescaleDB can use to exclude entire chunks without scanning them. Query 2: inefficient query (anti-pattern). ```sql -- Scans ALL chunks, extracts timestamp from every row SELECT COUNT(*), event_type FROM app_events WHERE uuid_timestamp(event_id) >= now() - interval '7 days' GROUP BY event_type; ``` Why this is slow: the `uuid_timestamp()` function must be evaluated for every row, which prevents chunk exclusion. Query 3: SkipScan on a single column (distinct users). ```sql -- Demonstrates SkipScan optimization: uses the compression index to skip repeated values \timing on SELECT DISTINCT ON (user_id) user_id, event_type, uuid_timestamp(event_id) as event_time FROM app_events WHERE event_id >= to_uuidv7_boundary(now() - interval '30 days') ORDER BY user_id, event_id DESC LIMIT 50; ``` Why this uses SkipScan: because `user_id` is the `segmentby` column, TimescaleDB automatically creates a compression index on it. SkipScan can jump directly to the next unique `user_id` value instead of scanning all rows. The `WHERE` clause ensures chunk exclusion, so it only scans chunks with recent data, making it even faster. To verify SkipScan is used, check the query plan with `EXPLAIN`. You should see `Custom Scan (SkipScan)` on the compressed chunks instead of a sequential scan. Query 4: funnel analysis. ```sql WITH funnel AS ( SELECT user_id, session_id, MAX(CASE WHEN event_type = 'page_view' THEN 1 ELSE 0 END) as viewed, MAX(CASE WHEN event_type = 'add_to_cart' THEN 1 ELSE 0 END) as added_to_cart, MAX(CASE WHEN event_type = 'purchase' THEN 1 ELSE 0 END) as purchased FROM app_events WHERE event_id >= to_uuidv7_boundary(now() - interval '30 days') GROUP BY user_id, session_id ) SELECT SUM(viewed) as sessions_with_view, SUM(added_to_cart) as sessions_with_cart, SUM(purchased) as sessions_with_purchase, ROUND(100.0 * SUM(added_to_cart) / NULLIF(SUM(viewed), 0), 2) as view_to_cart_pct, ROUND(100.0 * SUM(purchased) / NULLIF(SUM(added_to_cart), 0), 2) as cart_to_purchase_pct FROM funnel; ``` Query 5: revenue by country (last 30 days). ```sql SELECT country_code, COUNT(*) as purchase_count, SUM(revenue_cents) / 100.0 as total_revenue, ROUND(AVG(revenue_cents) / 100.0, 2) as avg_order_value FROM app_events WHERE event_id >= to_uuidv7_boundary(now() - interval '30 days') AND event_type = 'purchase' GROUP BY country_code ORDER BY total_revenue DESC LIMIT 10; ``` ## What's happening behind the scenes? [Section titled “What's happening behind the scenes?”](#whats-happening-behind-the-scenes) ### UUIDv7 partitioning [Section titled “UUIDv7 partitioning”](#uuidv7-partitioning) When you create a table with `tsdb.partition_column = 'event_id'` where `event_id` is a UUIDv7: * TimescaleDB automatically partitions your data by the time-embedded UUID. * No separate timestamp column is needed, because the timestamp is embedded in the UUID itself. * chunk exclusion works with `to_uuidv7_boundary()` for efficient time-range queries. * Vectorized UUID compression provides 30% storage savings and 2x query performance. ### Columnstore compression [Section titled “Columnstore compression”](#columnstore-compression) With `tsdb.enable_columnstore=true`: * Data is stored in a hybrid row-columnar format. * Analytical queries only scan the columns they need, for a large speedup. * Typical compression ratios: 90% or more for time-series data. * Compression happens transparently, with no changes to your queries. ### Direct to columnstore [Section titled “Direct to columnstore”](#direct-to-columnstore) When you use `SET timescaledb.enable_direct_compress_copy = on`: * Data loads directly into compressed columnstore format. * Bypasses the rowstore entirely. * Provides instant analytical performance, with no waiting for background compression. ### Segmentation [Section titled “Segmentation”](#segmentation) The `tsdb.segmentby='user_id'` setting: * Groups data by user within each chunk. * Improves compression ratios by keeping similar data together. * Speeds up queries that filter by `user_id`. * Is better for user-based analytics. ### UUIDv7 functions [Section titled “UUIDv7 functions”](#uuidv7-functions) TimescaleDB provides comprehensive UUIDv7 functionality across all supported PostgreSQL versions (including PostgreSQL 15, 16, and 17), while PostgreSQL only provides UUIDv7 support in PostgreSQL 18. For complete documentation on all available UUIDv7 functions, see the [UUIDv7 functions reference](/docs/reference/timescaledb/uuid-functions). ## Continuous aggregates (Advanced) [Section titled “Continuous aggregates (Advanced)”](#continuous-aggregates-advanced) For real-time dashboards, you can create continuous aggregates that automatically update: ```sql -- Create a continuous aggregate for hourly event statistics CREATE MATERIALIZED VIEW app_events_hourly WITH (timescaledb.continuous) AS SELECT time_bucket('1 hour', uuid_timestamp(event_id)) AS hour, event_type, COUNT(*) as event_count, COUNT(DISTINCT user_id) as unique_users, SUM(revenue_cents) / 100.0 as total_revenue FROM app_events GROUP BY hour, event_type; -- Add a refresh policy to keep it updated SELECT add_continuous_aggregate_policy('app_events_hourly', start_offset => INTERVAL '2 hours', end_offset => INTERVAL '1 hour', schedule_interval => INTERVAL '1 hour'); ``` Now you can query `app_events_hourly` for instant results on pre-aggregated data. ## Troubleshooting [Section titled “Troubleshooting”](#troubleshooting) ### Data didn't load [Section titled “Data didn't load”](#data-didnt-load) * Check the CSV file path is correct. * Ensure the CSV header matches the schema columns. * Try loading a few rows first to test: `LIMIT 10` in your data file. ### Direct to columnstore not working [Section titled “Direct to columnstore not working”](#direct-to-columnstore-not-working) * Verify TimescaleDB version 2.24 or later: `SELECT extversion FROM pg_extension WHERE extname = 'timescaledb';`. * Ensure you ran `SET timescaledb.enable_direct_compress_copy = on;` in the same session. * Check for error messages in the output. ### Queries seem slow [Section titled “Queries seem slow”](#queries-seem-slow) * Verify you're using `to_uuidv7_boundary()` for time-range queries, not `uuid_timestamp()`. * Check if data is compressed: `SELECT * FROM timescaledb_information.chunks WHERE hypertable_name = 'app_events';`. * Ensure chunk exclusion is working: use `EXPLAIN ANALYZE` to see chunk pruning. ### UUIDv7 functions not found [Section titled “UUIDv7 functions not found”](#uuidv7-functions-not-found) * Verify your TimescaleDB version supports UUIDv7: `SELECT extversion FROM pg_extension WHERE extname = 'timescaledb';`. * UUIDv7 support requires TimescaleDB 2.24 or later with the uuidv7 extension enabled. ## Use cases [Section titled “Use cases”](#use-cases) This application events example demonstrates patterns applicable to: * **SaaS analytics**: user behavior tracking, feature usage, and conversion funnels. * **E-commerce**: shopping cart analysis, purchase patterns, and product recommendations. * **Application monitoring**: error tracking, performance metrics, and user sessions. * **Audit logging**: security events, compliance tracking, and change history. * **Event-driven architectures**: microservices event sourcing and message queues. * **A/B testing**: experiment tracking, variant analysis, and statistical significance. ## Clean up [Section titled “Clean up”](#clean-up) When you're done experimenting, remove the container and data. If you used the one-line install: ```bash # Stop the container docker stop timescaledb-ha-pg18-quickstart # Remove the container docker rm timescaledb-ha-pg18-quickstart # Remove the persistent data volume docker volume rm timescaledb_data # (Optional) Remove the Docker image docker rmi timescale/timescaledb-ha:pg18 ``` If you used the manual Docker command: ```bash # Stop the container docker stop timescaledb # Remove the container docker rm timescaledb # (Optional) Remove the Docker image docker rmi timescale/timescaledb-ha:pg18 ``` If you created a named volume with the manual Docker command, you can remove it with `docker volume rm `. # Visualize financial tick data with Grafana > Store financial tick data, create candlestick views, and visualize price changes in Grafana The financial industry is extremely data-heavy and relies on real-time and historical data for decision-making, risk assessment, fraud detection, and market analysis. Tiger Data simplifies management of these large volumes of data, while also providing you with meaningful analytical insights and optimizing storage costs. To analyze financial data, you can chart the open, high, low, close, and volume (OHLCV) information for a financial asset. Using this data, you can create candlestick charts that make it easier to analyze the price changes of financial assets over time. You can use candlestick charts to examine trends in stock, cryptocurrency, or NFT prices. In this tutorial, you use real raw financial data provided by [Twelve Data](https://twelvedata.com/), create an aggregated candlestick view, query the aggregated data, and visualize the data in Grafana. This tutorial covers: 1. **Ingest data into a service**: load data from [Twelve Data](https://twelvedata.com/) into your TimescaleDB database. 2. **Query your dataset**: create candlestick views, query the aggregated data, and visualize the data in Grafana. ## Prerequisites for this tutorial To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service). These steps use Tiger Cloud, but the same approach applies to a [self-hosted TimescaleDB](/docs/get-started/choose-your-path/install-timescaledb) instance. - A [self-managed Grafana](https://grafana.com/get/?tab=self-managed) instance, or a [Grafana Cloud](https://grafana.com/get/) account. This tutorial uses a dataset that contains second-by-second trade data for the most-traded crypto-assets. You optimize this time-series data in a hypertable called `crypto_ticks`. You also create a separate table of asset symbols in a regular PostgreSQL table named `crypto_assets`. ## OHLCV data and candlestick charts [Section titled “OHLCV data and candlestick charts”](#ohlcv-data-and-candlestick-charts) The financial sector regularly uses [candlestick charts](https://www.investopedia.com/terms/c/candlestick.asp) to visualize the price change of an asset. Each candlestick represents a time period, such as one minute or one hour, and shows how the asset's price changed during that time. Candlestick charts are generated from the open, high, low, close, and volume data for each financial asset during the time period. This is often abbreviated as OHLCV: * Open: opening price * High: highest price * Low: lowest price * Close: closing price * Volume: volume of transactions ![Candlestick chart showing intraday stock price movements in Tiger Cloud](/docs/_astro/candlestick_fig.CkWcWeSI_ZSubFP.webp) TimescaleDB is well suited to storing and analyzing financial candlestick data, and many Tiger Data community members use it for exactly this purpose. Check out these stories from some Tiger Data community members: * [How Trading Strategy built a data stack for crypto quant trading](https://www.tigerdata.com/blog/how-trading-strategy-built-a-data-stack-for-crypto-quant-trading) * [How Messari uses data to open the cryptoeconomy to everyone](https://www.tigerdata.com/blog/how-messari-uses-data-to-open-the-cryptoeconomy-to-everyone) * [How I power a (successful) crypto trading bot with TimescaleDB](https://www.tigerdata.com/blog/how-i-power-a-successful-crypto-trading-bot-with-timescaledb) ## Optimize time-series data in a hypertable Hypertables are PostgreSQL tables in TimescaleDB that automatically partition your time-series data by time. Time-series data represents the way a system, process, or behavior changes over time. Hypertables enable TimescaleDB to work efficiently with time-series data. Each hypertable is made up of child tables called chunks. Each chunk is assigned a range of time, and only contains data from that range. When you run a query, TimescaleDB identifies the correct chunk and runs the query on it, instead of going through the entire table. [Hypercore](/docs/learn/columnar-storage/understand-hypercore) is the hybrid row-columnar storage engine in TimescaleDB used by hypertables. Traditional databases force a trade-off between fast inserts (row-based storage) and efficient analytics (columnar storage). Hypercore eliminates this trade-off, allowing real-time analytics without sacrificing transactional capabilities. Hypercore dynamically stores data in the most efficient format for its lifecycle: ![Move from rowstore to columstore in hypercore](/docs/_astro/hypercore_intro.DmLiqZGV_1VM5z7.webp) ![Move from rowstore to columstore in hypercore](/docs/_astro/hypercore_intro-dark.qPmjkBqW_19V2rF.webp) * **Row-based storage for recent data**: the most recent chunk (and possibly more) is always stored in the rowstore, ensuring fast inserts, updates, and low-latency single record queries. Additionally, row-based storage is used as a writethrough for inserts and updates to columnar storage. * **Columnar storage for analytical performance**: chunks are automatically compressed into the columnstore, optimizing storage efficiency and accelerating analytical queries. Unlike traditional columnar databases, hypercore allows data to be inserted or modified at any stage, making it a flexible solution for both high-ingest transactional workloads and real-time analytics, within a single database. Because TimescaleDB is 100% PostgreSQL, you can use all the standard PostgreSQL tables, indexes, stored procedures, and other objects alongside your hypertables. This makes creating and working with hypertables similar to standard PostgreSQL. 1. **Connect to your Tiger Cloud service** In [Tiger Console](https://console.cloud.tigerdata.com/dashboard/services) open an [SQL editor](/docs/build/data-management/run-queries-from-tiger-console). You can also connect to your service using [psql](/docs/integrate/query-administration/psql). 2. **Create a hypertable to store the real-time cryptocurrency data** Create a [hypertable](/docs/learn/hypertables/understand-hypertables) for your time-series data using [CREATE TABLE](/docs/reference/timescaledb). For [efficient queries](/docs/build/performance-optimization/secondary-indexes) on data in the columnstore, remember to `segmentby` the column you will use most often to filter your data: ```sql CREATE TABLE crypto_ticks ( "time" TIMESTAMPTZ, symbol TEXT, price DOUBLE PRECISION, day_volume NUMERIC ) WITH ( tsdb.hypertable, tsdb.segmentby='symbol', tsdb.orderby='time DESC' ); ``` When you create a hypertable using [CREATE TABLE ... WITH ...](/docs/reference/timescaledb/hypertables/create_table), the default partitioning column is automatically the first column with a timestamp data type. Also, TimescaleDB creates a [columnstore policy](/docs/reference/timescaledb/hypercore/add_columnstore_policy) that automatically converts your data to the columnstore, after an interval equal to the value of the [chunk\_interval](/docs/reference/timescaledb/hypertables/set_chunk_time_interval), defined through `after` in the policy. This columnar format enables fast scanning and aggregation, optimizing performance for analytical workloads while also saving significant storage space. In the columnstore conversion, hypertable chunks are compressed by up to 98%, and organized for efficient, large-scale queries. You can customize this policy later using [alter\_job](/docs/reference/timescaledb/jobs-automation/alter_job). However, to change `after` or `created_before`, the compression settings, or the hypertable the policy is acting on, you must [remove the columnstore policy](/docs/reference/timescaledb/hypercore/remove_columnstore_policy) and [add a new one](/docs/reference/timescaledb/hypercore/add_columnstore_policy). You can also manually [convert chunks](/docs/reference/timescaledb/hypercore/convert_to_columnstore) in a hypertable to the columnstore. ## Create a standard PostgreSQL table for relational data When you have relational data that enhances your time-series data, store that data in standard PostgreSQL relational tables. 1. **Add a table to store the asset symbol and name in a relational table** ```sql CREATE TABLE crypto_assets ( symbol TEXT UNIQUE, "name" TEXT ); ``` You now have two tables within your Tiger Cloud service. A hypertable named `crypto_ticks`, and a normal PostgreSQL table named `crypto_assets`. ## Load financial data This tutorial uses real-time cryptocurrency data, also known as tick data, from [Twelve Data](https://twelvedata.com/). To ingest data into the tables that you created, you need to download the dataset, then upload the data to your Tiger Cloud service. 1. **Download and unzip crypto\_sample.zip** Unzip [crypto\_sample.zip](https://assets.timescale.com/docs/downloads/candlestick/crypto_sample.zip) to a ``. This test dataset contains second-by-second trade data for the most-traded crypto-assets and a regular table of asset symbols and company names. To import up to 100GB of data directly from your current PostgreSQL-based database, [migrate with downtime](/docs/migrate/migrate-with-downtime) using native PostgreSQL tooling. To seamlessly import 100GB-10TB+ of data, use the [live migration](/docs/migrate/live-migration) tooling supplied by Tiger Data. To add data from non-PostgreSQL data sources, see [Import and ingest data](/docs/migrate). 2. **In Terminal, navigate to `` and connect to your service.** ```bash psql -d "postgres://:@:/" ``` The connection information for a service is available in the file you downloaded when you created it. 3. **Use the COPY command to transfer data into your Tiger Cloud service** At the `psql` prompt, if the `.csv` files aren't in your current directory, specify the file paths in these commands: ```sql \COPY crypto_ticks FROM 'tutorial_sample_tick.csv' CSV HEADER; ``` ```sql \COPY crypto_assets FROM 'tutorial_sample_assets.csv' CSV HEADER; ``` Because there are millions of rows of data, the `COPY` process could take a few minutes depending on your internet connection and local client resources. ## Query the data [Section titled “Query the data”](#query-the-data) Turning raw, real-time tick data into aggregated candlestick views is a common task for users who work with financial data. TimescaleDB includes [hyperfunctions](/docs/reference/timescaledb/hyperfunctions) that you can use to store and query your financial data more easily. Hyperfunctions are SQL functions within TimescaleDB that make it easier to manipulate and analyze time-series data in PostgreSQL with fewer lines of code. There are three hyperfunctions that are essential for calculating candlestick values: [`time_bucket()`](/docs/learn/data-lifecycle/time-buckets/about-time-buckets), [`FIRST()`](/docs/reference/timescaledb/hyperfunctions), and [`LAST()`](/docs/reference/timescaledb/hyperfunctions). The `time_bucket()` hyperfunction helps you aggregate records into buckets of arbitrary time intervals based on the timestamp value. `FIRST()` and `LAST()` help you calculate the opening and closing prices. To calculate highest and lowest prices, you can use the standard PostgreSQL aggregate functions `MIN` and `MAX`. In TimescaleDB, the most efficient way to create candlestick views is to use [continuous aggregates](/docs/learn/continuous-aggregates). In this tutorial, you create a continuous aggregate for a candlestick time bucket, and then query the aggregate with different refresh policies. Finally, you can use Grafana to visualize your data as a candlestick chart. ### Create a continuous aggregate [Section titled “Create a continuous aggregate”](#create-a-continuous-aggregate) To look at OHLCV values, the most effective way is to create a continuous aggregate. In this tutorial, you create a continuous aggregate to aggregate data for each day. You then set the aggregate to refresh every day, and to aggregate the last two days' worth of data. 1. **Connect to the Tiger Cloud service** Connect to the Tiger Cloud service that contains the Twelve Data cryptocurrency dataset. 2. **Create the continuous aggregate** At the psql prompt, create the continuous aggregate to aggregate data every day: ```sql CREATE MATERIALIZED VIEW one_day_candle WITH (timescaledb.continuous) AS SELECT time_bucket('1 day', time) AS bucket, symbol, FIRST(price, time) AS "open", MAX(price) AS high, MIN(price) AS low, LAST(price, time) AS "close", LAST(day_volume, time) AS day_volume FROM crypto_ticks GROUP BY bucket, symbol; ``` When you create the continuous aggregate, it refreshes by default. 3. **Set a refresh policy** Set a refresh policy to update the continuous aggregate every day, if there is new data available in the hypertable for the last two days: ```sql SELECT add_continuous_aggregate_policy('one_day_candle', start_offset => INTERVAL '3 days', end_offset => INTERVAL '1 day', schedule_interval => INTERVAL '1 day'); ``` ### Query the continuous aggregate [Section titled “Query the continuous aggregate”](#query-the-continuous-aggregate) When you have your continuous aggregate set up, you can query it to get the OHLCV values. 1. **Connect to the Tiger Cloud service** Connect to the Tiger Cloud service that contains the Twelve Data cryptocurrency dataset. 2. **Query Bitcoin OHLCV data** At the psql prompt, use this query to select all Bitcoin OHLCV data for the past 14 days, by time bucket: ```sql SELECT * FROM one_day_candle WHERE symbol = 'BTC/USD' AND bucket >= NOW() - INTERVAL '14 days' ORDER BY bucket; ``` The result of the query looks like this: ```sql bucket | symbol | open | high | low | close | day_volume ------------------------+---------+---------+---------+---------+---------+------------ 2022-11-24 00:00:00+00 | BTC/USD | 16587 | 16781.2 | 16463.4 | 16597.4 | 21803 2022-11-25 00:00:00+00 | BTC/USD | 16597.4 | 16610.1 | 16344.4 | 16503.1 | 20788 2022-11-26 00:00:00+00 | BTC/USD | 16507.9 | 16685.5 | 16384.5 | 16450.6 | 12300 ``` ## Connect Grafana to Tiger Cloud To visualize the results of your queries, enable Grafana to read the data in your service: 1. **Log in to Grafana** In your browser, log in to either: * Self-hosted Grafana: at `http://localhost:3000/`. The default credentials are `admin`, `admin`. * Grafana Cloud: use the URL and credentials you set when you created your account. 2. **Add your service as a data source** 1. Open `Connections` > `Data sources`, then click `Add new data source`. 2. Select `PostgreSQL` from the list. 3. Configure the connection: * `Host URL`, `Database name`, `Username`, and `Password`, configure using your [connection details](/docs/integrate/find-connection-details). `Host URL` is in the format `:`. * `TLS/SSL Mode`: select `require`. * `PostgreSQL options`: enable `TimescaleDB`. * Leave the default setting for all other fields. 4. Click `Save & test`. Grafana checks that your details are set correctly. ## Graph OHLCV data When you have extracted the raw OHLCV data, you can use it to graph the result in a candlestick chart, using Grafana. 1. **Create a new dashboard** In Grafana, from the `Dashboards` page, click `New` and select `New dashboard`. 2. **Add a Candlestick visualization** Click `Add visualization`, then select the data source that connects to your Tiger Cloud service and the `Candlestick` visualization type in the top right. 3. **Paste the OHLCV query** In the `Queries` section, select `Code` and paste the query you used to get the OHLCV values: ```sql SELECT * FROM one_day_candle WHERE symbol = 'BTC/USD' AND bucket >= NOW() - INTERVAL '14 days' ORDER BY bucket; ``` 4. **Apply and save your graph** Adjust elements of the table as required, and click `Apply` to save your graph to the dashboard. ![Candlestick chart in Grafana using 1-day OHLCV tick data](/docs/_astro/Grafana_candlestick_1day.BS8xfMii_Z14b0su.webp) # Analyze NYC taxi data > Get started with TimescaleDB using New York City taxi trip data, with location-based analytics and time-series aggregations Get started with TimescaleDB using New York City taxi trip data. This example demonstrates how to handle high-volume transportation data with location-based analytics and time-series aggregations. If you would rather visualize a taxi-style dataset on a live dashboard, see [Visualize transport and geospatial data with Grafana](/docs/build/examples/analyze-transport-data). ## What you learn [Section titled “What you learn”](#what-you-learn) * How to model high-volume transportation data with latitude and longitude coordinates. * Time-series aggregations with `time_bucket()`. * Optimal segmentation strategies for compression. * Revenue and usage pattern analysis. * Loading data with direct to columnstore for instant performance. ## Prerequisites for this tutorial To follow these steps, you'll need: * Docker installed * The `psql` PostgreSQL client ## Set up and query the data [Section titled “Set up and query the data”](#set-up-and-query-the-data) 1. **Start TimescaleDB** Start TimescaleDB using one of these options. * One-line install (recommended) The easiest way to get started: Important This script is intended for local development and testing only. Do not use it for production deployments. For production-ready installation options, see the [TimescaleDB installation guide](/docs/get-started/choose-your-path/install-timescaledb). On Linux or Mac: ```sh curl -sL https://tsdb.co/start-local | sh ``` This command: * Downloads and starts TimescaleDB, if not already downloaded. * Exposes PostgreSQL on port `6543`, a non-standard port that avoids conflicts with other PostgreSQL instances on port 5432. * Automatically tunes settings for your environment using timescaledb-tune. * Sets up a persistent data volume. * Manual Docker command Run TimescaleDB directly with Docker. This option also works on Windows: ```bash docker run -d --name timescaledb \ -p 6543:5432 \ -e POSTGRES_PASSWORD=password \ timescale/timescaledb-ha:pg18 ``` This example uses port `6543`, mapped to container port 5432, to avoid conflicts if you have other PostgreSQL instances running on the standard port 5432. Wait about 1 to 2 minutes for TimescaleDB to download and initialize. 2. **Connect to TimescaleDB** Connect using `psql`: ```bash psql -h localhost -p 6543 -U postgres # When prompted, enter password: password ``` You should see the PostgreSQL prompt. Verify TimescaleDB is installed: ```sql SELECT extname, extversion FROM pg_extension WHERE extname = 'timescaledb'; ``` Expected output: ```plaintext extname | extversion -------------+------------ timescaledb | 2.x.x ``` Tips Prefer a GUI? If you would rather use a graphical tool instead of the command line, you can download [pgAdmin](https://www.pgadmin.org/download/) and connect to TimescaleDB using the same connection details (host: `localhost`, port: `6543`, user: `postgres`, password: `password`). 3. **Create the schema** Create the optimized hypertable by running this SQL in your `psql` session: ```sql -- Create the hypertable with optimal settings for NYC Taxi data -- This automatically enables columnstore for fast analytical queries CREATE TABLE trips ( vendor_id TEXT, pickup_boroname VARCHAR, pickup_datetime TIMESTAMP WITHOUT TIME ZONE NOT NULL, dropoff_datetime TIMESTAMP WITHOUT TIME ZONE NOT NULL, passenger_count NUMERIC, trip_distance NUMERIC, pickup_longitude NUMERIC, pickup_latitude NUMERIC, rate_code INTEGER, dropoff_longitude NUMERIC, dropoff_latitude NUMERIC, payment_type VARCHAR, fare_amount NUMERIC, extra NUMERIC, mta_tax NUMERIC, tip_amount NUMERIC, tolls_amount NUMERIC, improvement_surcharge NUMERIC, total_amount NUMERIC ) WITH ( tsdb.hypertable, tsdb.partition_column='pickup_datetime', tsdb.enable_columnstore=true, tsdb.segmentby='pickup_boroname', tsdb.orderby='pickup_datetime DESC' ); -- Create indexes CREATE INDEX idx_trips_pickup_time ON trips (pickup_datetime DESC); CREATE INDEX idx_trips_borough_time ON trips (pickup_boroname, pickup_datetime DESC); ``` This creates a `trips` table with: * Automatic time-based partitioning on `pickup_datetime`. * Columnstore enabled for fast analytical queries. * Segmentation by `pickup_boroname` for optimal compression (6 boroughs). * Full trip details including fares, distances, and coordinates. You can also download the schema, with column comments, as a runnable script: [`nyc-taxi-schema.sql`](/docs/files/nyc-taxi/nyc-taxi-schema.sql). 4. **Load the sample data** First, download and decompress the sample data: ```bash # Download the sample data wget https://assets.timescale.com/timescaledb-datasets/nyc_taxi_sample_nov_dec_2015.csv.gz # Decompress the CSV file gunzip nyc_taxi_sample_nov_dec_2015.csv.gz # This will create nyc_taxi_sample_nov_dec_2015.csv ready for loading ``` Load the data using one of these approaches. * Direct to columnstore (recommended) This approach writes data directly to the columnstore, bypassing the rowstore entirely. You get instant analytical performance. From the command line: ```bash psql -h localhost -p 6543 -U postgres \ -v ON_ERROR_STOP=1 \ -c "SET timescaledb.enable_direct_compress_copy = on; COPY trips FROM STDIN WITH (FORMAT csv, HEADER true);" \ < nyc_taxi_sample_nov_dec_2015.csv ``` This command reads the CSV file from your local filesystem and pipes it to PostgreSQL, which loads it directly into the columnstore. Verify the data loaded: ```sql SELECT COUNT(*) FROM trips; ``` * Standard COPY (fallback) This approach loads data into the rowstore first. Data is converted to the columnstore by a background policy (12 to 24 hours) for faster querying. From the command line: ```bash psql -h localhost -p 6543 -U postgres \ -v ON_ERROR_STOP=1 \ -c "COPY trips FROM STDIN WITH (FORMAT csv, HEADER true);" \ < nyc_taxi_sample_nov_dec_2015.csv ``` Verify the data loaded: ```sql SELECT COUNT(*) FROM trips; ``` If you loaded data using standard copy, a background process converts your rowstore data to the columnstore in 12 to 24 hours. To get the best query performance immediately, you can convert it manually: ```sql DO $$ DECLARE ch TEXT; BEGIN FOR ch IN SELECT show_chunks('trips') LOOP CALL convert_to_columnstore(ch); END LOOP; END $$; ``` 5. **Run sample queries** Now explore the data with some analytical queries. Run these in your `psql` session. Query 1: overall statistics. ```sql \timing on SELECT COUNT(*) as total_trips, ROUND(SUM(fare_amount)::numeric, 2) as total_revenue, ROUND(AVG(fare_amount)::numeric, 2) as avg_fare, ROUND(AVG(trip_distance)::numeric, 2) as avg_distance FROM trips; ``` Query 2: breakdown by vendor. ```sql SELECT vendor_id, COUNT(*) as trips, ROUND(AVG(fare_amount)::numeric, 2) as avg_fare, ROUND(AVG(tip_amount)::numeric, 2) as avg_tip, ROUND(AVG(passenger_count)::numeric, 2) as avg_passengers FROM trips GROUP BY vendor_id ORDER BY trips DESC; ``` Query 3: hourly patterns using `time_bucket`. ```sql SELECT time_bucket('1 hour', pickup_datetime) AS hour, COUNT(*) as trips, ROUND(AVG(fare_amount)::numeric, 2) as avg_fare, ROUND(SUM(tip_amount)::numeric, 2) as total_tips FROM trips GROUP BY hour ORDER BY hour DESC LIMIT 24; ``` Query 4: payment type analysis. ```sql SELECT payment_type, COUNT(*) as trip_count, ROUND(SUM(fare_amount)::numeric, 2) as total_revenue, ROUND(AVG(trip_distance)::numeric, 2) as avg_distance, ROUND(AVG(tip_amount)::numeric, 2) as avg_tip FROM trips GROUP BY payment_type ORDER BY total_revenue DESC; ``` Query 5: daily statistics by borough. ```sql SELECT time_bucket('1 day', pickup_datetime) AS day, pickup_boroname, COUNT(*) as trips, ROUND(AVG(fare_amount)::numeric, 2) as avg_fare, ROUND(MAX(fare_amount)::numeric, 2) as max_fare FROM trips GROUP BY day, pickup_boroname ORDER BY day DESC, pickup_boroname LIMIT 20; ``` Query 6: trips by distance category. ```sql SELECT CASE WHEN trip_distance < 1 THEN 'Short (< 1 mile)' WHEN trip_distance < 5 THEN 'Medium (1-5 miles)' WHEN trip_distance < 10 THEN 'Long (5-10 miles)' ELSE 'Very Long (> 10 miles)' END as distance_category, COUNT(*) as trips, ROUND(AVG(fare_amount)::numeric, 2) as avg_fare, ROUND(AVG(tip_amount)::numeric, 2) as avg_tip FROM trips GROUP BY distance_category ORDER BY trips DESC; ``` ## Sample queries explained [Section titled “Sample queries explained”](#sample-queries-explained) You can download the complete set of queries as a runnable script: [`nyc-taxi-queries.sql`](/docs/files/nyc-taxi/nyc-taxi-queries.sql). Run it with `psql -f nyc-taxi-queries.sql`. Each query demonstrates: 1. Total trips and revenue: simple aggregations across all data. 2. Breakdown by vendor: segmentation analysis by taxi vendor. 3. Hourly patterns: using `time_bucket()` for time-based aggregation. 4. Payment type analysis: analyzing payment methods. 5. Daily statistics: multi-dimensional aggregation (time and borough). 6. Distance categories: `CASE` statement with aggregations. ## What's happening behind the scenes? [Section titled “What's happening behind the scenes?”](#whats-happening-behind-the-scenes) ### Hypertables [Section titled “Hypertables”](#hypertables) When you create a table with `tsdb.hypertable`, TimescaleDB automatically: * Partitions your data into time-based chunks (default: 7 days per chunk). * Manages chunk lifecycle automatically. * Optimizes queries to scan only relevant chunks. ### Columnstore compression [Section titled “Columnstore compression”](#columnstore-compression) With `tsdb.enable_columnstore=true`: * Data is stored in a hybrid row-columnar format. * Analytical queries only scan the columns they need, for a large speedup. * Typical compression ratios: 90% or more for time-series data. * Compression happens transparently, with no changes to your queries. ### Direct to columnstore [Section titled “Direct to columnstore”](#direct-to-columnstore) When you use `SET timescaledb.enable_direct_compress_copy = on`: * Data loads directly into compressed columnstore format. * Bypasses the rowstore entirely. * Provides instant analytical performance, with no waiting for background compression. * Works well for bulk data loads and migrations. ### Segmentation [Section titled “Segmentation”](#segmentation) The `tsdb.segmentby='pickup_boroname'` setting: * Groups data by pickup borough within each chunk (6 unique values: Manhattan, Brooklyn, Queens, Bronx, Staten Island, EWR). * Improves compression ratios by keeping similar data together. * Speeds up queries that filter by `pickup_boroname`. * Provides better cardinality than `vendor_id` (6 values versus 2) for optimal compression. * Is automatically optimized without manual tuning. ### time\_bucket() function [Section titled “time\_bucket() function”](#time_bucket-function) The `time_bucket()` function is like PostgreSQL's `date_trunc()` but more powerful: * Works with any interval: `5 minutes`, `1 hour`, `1 day`, and so on. * Is optimized for time-series queries. * Integrates seamlessly with continuous aggregates. * Is essential for time-series analytics. ## Schema design choices [Section titled “Schema design choices”](#schema-design-choices) Why these settings? **partition\_column='pickup\_datetime'** * Time is the natural partition key for time-series data. * Enables automatic chunk pruning for time-range queries. * The default chunk interval (7 days) works well for taxi data. **segmentby='pickup\_boroname'** * Optimal cardinality with 6 borough values (Manhattan, Brooklyn, Queens, Bronx, Staten Island, EWR). * Frequently used in `WHERE` clauses and `GROUP BY` for location-based analytics. * Improves compression by grouping geographically similar trips. * Better than `vendor_id` (only 2 values) for compression efficiency. **orderby='pickup\_datetime DESC'** * Most queries want recent data first. * Optimizes for latest-trips queries. * Improves query performance for time-range scans. ## Continuous aggregates (Advanced) [Section titled “Continuous aggregates (Advanced)”](#continuous-aggregates-advanced) For real-time dashboards, you can create continuous aggregates that automatically update: ```sql -- Create a continuous aggregate for hourly statistics by borough CREATE MATERIALIZED VIEW trips_hourly WITH (timescaledb.continuous) AS SELECT time_bucket('1 hour', pickup_datetime) AS hour, pickup_boroname, COUNT(*) as trip_count, AVG(fare_amount) as avg_fare, SUM(fare_amount) as total_revenue, AVG(trip_distance) as avg_distance FROM trips GROUP BY hour, pickup_boroname; -- Add a refresh policy to keep it updated SELECT add_continuous_aggregate_policy('trips_hourly', start_offset => INTERVAL '2 hours', end_offset => INTERVAL '1 hour', schedule_interval => INTERVAL '1 hour'); ``` Now you can query `trips_hourly` for instant results on pre-aggregated data. ## Troubleshooting [Section titled “Troubleshooting”](#troubleshooting) ### Data didn't load [Section titled “Data didn't load”](#data-didnt-load) * Check the CSV file path is correct. * Ensure the CSV header matches the schema columns. * Try loading a few rows first to test: `LIMIT 10` in your data file. ### Direct to columnstore not working [Section titled “Direct to columnstore not working”](#direct-to-columnstore-not-working) * Verify TimescaleDB version 2.24 or later: `SELECT extversion FROM pg_extension WHERE extname = 'timescaledb';`. * Ensure you ran `SET timescaledb.enable_direct_compress_copy = on;` in the same session. * Check for error messages in the output. ### Queries seem slow [Section titled “Queries seem slow”](#queries-seem-slow) * Verify the columnstore is enabled: `SELECT * FROM timescaledb_information.hypertables WHERE hypertable_name = 'trips';`. * Check if data is compressed: `SELECT * FROM timescaledb_information.chunks WHERE hypertable_name = 'trips';`. * Ensure you're querying with time ranges, which enables chunk exclusion. ### Out of memory during load [Section titled “Out of memory during load”](#out-of-memory-during-load) * Reduce batch size in the `COPY` command. * Increase Docker memory allocation. * Consider loading data in smaller time-range batches. ## Use cases [Section titled “Use cases”](#use-cases) This NYC Taxi example demonstrates patterns applicable to: * **Ride-sharing platforms**: track trips, drivers, and pricing. * **Fleet management**: vehicle tracking and route optimization. * **Delivery services**: order tracking, delivery times, and driver analytics. * **Public transportation**: route analysis, passenger counts, and schedule optimization. * **Urban planning**: traffic patterns, popular routes, and demand forecasting. * **Logistics**: shipment tracking, route efficiency, and cost analysis. ## Clean up [Section titled “Clean up”](#clean-up) When you're done experimenting, remove the container and data. If you used the one-line install: ```bash # Stop the container docker stop timescaledb-ha-pg18-quickstart # Remove the container docker rm timescaledb-ha-pg18-quickstart # Remove the persistent data volume docker volume rm timescaledb_data # (Optional) Remove the Docker image docker rmi timescale/timescaledb-ha:pg18 ``` If you used the manual Docker command: ```bash # Stop the container docker stop timescaledb # Remove the container docker rm timescaledb # (Optional) Remove the Docker image docker rmi timescale/timescaledb-ha:pg18 ``` If you created a named volume with the manual Docker command, you can remove it with `docker volume rm `. # Analyze stock market data > Analyze financial tick and candlestick data with TimescaleDB using S&P 500 stock prices This example demonstrates financial tick and candlestick data analysis with TimescaleDB. The dataset corresponds to the stocks listed in the [S\&P 500 index](https://en.wikipedia.org/wiki/List_of_S%26P_500_companies), with fictional prices and movements. The tick cadence is per second, over three business days, containing approximately 35 million records in total and 503 tickers. If you would rather build candlestick charts on a live dashboard, see [Visualize financial tick data with Grafana](/docs/build/examples/analyze-financial-tick-data), which uses a crypto dataset. ## Use cases [Section titled “Use cases”](#use-cases) Trading platforms, market data analysis, portfolio analytics, algorithmic trading. ## What you learn [Section titled “What you learn”](#what-you-learn) * OHLCV (open, high, low, close, volume) data modeling. * Candlestick aggregations at multiple intervals. * continuous aggregates for different timeframes (1 minute, 5 minutes, 1 hour). * Real-time market analysis queries. ## Dataset preview [Section titled “Dataset preview”](#dataset-preview) The dataset contains the following columns: * timestamp * ticker * price * price delta * change percentage * volume ```csv 2025-11-12 14:30:00+00:00,NVDA,38.25,0.25,0.65359,5095712 2025-11-12 14:30:00+00:00,AAPL,152.03,0.03,0.01973,6466554 2025-11-12 14:30:00+00:00,MSFT,129.23,0.23,0.17798,4417848 2025-11-12 14:30:00+00:00,GOOG,174.93,-0.07,-0.04002,19602229 2025-11-12 14:30:00+00:00,GOOGL,71.21,0.21,0.2949,3149482 2025-11-12 14:30:00+00:00,AMZN,95.95,-0.05,-0.05211,12150474 2025-11-12 14:30:00+00:00,AVGO,196.15,0.15,0.07647,6166047 2025-11-12 14:30:00+00:00,META,133.22,0.22,0.16514,12230004 2025-11-12 14:30:00+00:00,TSLA,82.0,0.0,0.0,10298937 2025-11-12 14:30:00+00:00,BRK.B,56.84,-0.16,-0.28149,10980047 ``` ## Prerequisites for this tutorial To follow these steps, you'll need: * Docker installed * The `psql` PostgreSQL client ## Set up and query the data [Section titled “Set up and query the data”](#set-up-and-query-the-data) First, download the dataset: ```sh curl -L https://assets.timescale.com/timescaledb-datasets/sp500_stock_prices_3d_1s.tar.gz | tar -xzf - ``` 1. **Start TimescaleDB** Start TimescaleDB using one of these options. * One-line install (recommended) The easiest way to get started: Important This script is intended for local development and testing only. Do not use it for production deployments. For production-ready installation options, see the [TimescaleDB installation guide](/docs/get-started/choose-your-path/install-timescaledb). On Linux or Mac: ```sh curl -sL https://tsdb.co/start-local | sh ``` This command: * Downloads and starts TimescaleDB, if not already downloaded. * Exposes PostgreSQL on port `6543`, a non-standard port that avoids conflicts with other PostgreSQL instances on port 5432. * Automatically tunes settings for your environment using timescaledb-tune. * Sets up a persistent data volume. * Manual Docker command Run TimescaleDB directly with Docker. This option also works on Windows: ```bash docker run -d --name timescaledb \ -p 6543:5432 \ -e POSTGRES_PASSWORD=password \ timescale/timescaledb-ha:pg18 ``` This example uses port `6543`, mapped to container port 5432, to avoid conflicts if you have other PostgreSQL instances running on the standard port 5432. Wait about 1 to 2 minutes for TimescaleDB to download and initialize. 2. **Connect to TimescaleDB** Connect using `psql`: ```sh psql "postgres://postgres:password@localhost:6543/postgres" ``` You should see the PostgreSQL prompt. Verify TimescaleDB is installed: ```sql SELECT extname, extversion FROM pg_extension WHERE extname = 'timescaledb'; ``` Expected output: ```plaintext extname | extversion -------------+------------ timescaledb | 2.x.x ``` Tips Prefer a GUI? If you would rather use a graphical tool instead of the command line, you can download [pgAdmin](https://www.pgadmin.org/download/) and connect to TimescaleDB using the same connection details (host: `localhost`, port: `6543`, user: `postgres`, password: `password`). 3. **Create the schema** Create the optimized hypertable by running this SQL in your `psql` session: ```sql -- Create the stock_prices table with the column ts as partitioning -- Note: for optimized query performance grouping on ticker, we select this column to segment by CREATE TABLE stock_prices ( ts TIMESTAMPTZ NOT NULL, ticker TEXT NOT NULL, price DOUBLE PRECISION NOT NULL, change_delta DOUBLE PRECISION NOT NULL, change_percentage DOUBLE PRECISION NOT NULL, volume BIGINT NOT NULL CHECK (volume >= 0) ) WITH ( timescaledb.hypertable, timescaledb.segmentby='ticker' ); ``` This creates a `stock_prices` table with: * Partitioning by timestamp on column `ts`. * Segmentation by `ticker` for optimal compression and query performance. 4. **Load the sample data** This approach writes data directly to the columnstore, bypassing the rowstore entirely. You get instant analytical performance. From `psql`: ```sql -- Enable direct to columnstore for this session SET timescaledb.enable_direct_compress_copy = on; -- Load data directly into columnstore \COPY stock_prices FROM 'sp500_stock_prices_3d_1s.csv' WITH (FORMAT csv, HEADER true); -- Verify data loaded SELECT COUNT(*) FROM stock_prices; ``` 5. **Run sample queries** Now explore the data with some analytical queries. Run these in your `psql` session. Activate time measuring: ```sql \timing on ``` Query 1: OHLCV per hour of AAPL. Aggregate raw 1-second tick data into hourly candlesticks (open, high, low, close, volume) of the ticker `AAPL`. ```sql SELECT time_bucket('1 hour', ts) AS hour_bucket, ticker, FIRST(price, ts) AS open_price, MAX(price) AS high_price, MIN(price) AS low_price, LAST(price, ts) AS close_price, AVG(price) AS avg_price, SUM(volume) AS sum_volume FROM stock_prices WHERE ticker = 'AAPL' GROUP BY hour_bucket, ticker ORDER BY hour_bucket DESC; ``` Query 2: trend analysis with a simple moving average (SMA) of MSFT. Calculate a smoothing line to see trends over noise for the ticker `MSFT` over 4 hours. ```sql WITH candles AS ( SELECT time_bucket('1 hour', ts) AS bucket, ticker, LAST(price, ts) AS close_price FROM stock_prices WHERE ticker = 'MSFT' GROUP BY bucket, ticker ) SELECT bucket, ticker, close_price, AVG(close_price) OVER ( PARTITION BY ticker ORDER BY bucket ROWS BETWEEN 3 PRECEDING AND CURRENT ROW ) AS sma_4hours FROM candles ORDER BY bucket DESC; ``` Query 3: hour-over-hour return. Compare the current price to the price exactly one hour ago to calculate percentage growth. ```sql WITH hourly_close AS ( SELECT time_bucket('1 hour', ts) AS bucket, ticker, LAST(price, ts) AS closing_price FROM stock_prices GROUP BY bucket, ticker ) SELECT bucket, ticker, closing_price, LAG(closing_price, 1) OVER (PARTITION BY ticker ORDER BY bucket) AS prev_close, ((closing_price - LAG(closing_price, 1) OVER (PARTITION BY ticker ORDER BY bucket)) / LAG(closing_price, 1) OVER (PARTITION BY ticker ORDER BY bucket)) * 100 AS hourly_return_pct FROM hourly_close; ``` Query 4: price volatility. ```sql SELECT ticker, AVG(price) AS avg_price, STDDEV(price) AS price_volatility, MAX(price) - MIN(price) AS price_spread FROM stock_prices WHERE ts > NOW() - INTERVAL '7 days' GROUP BY ticker HAVING count(*) > 10 ORDER BY price_volatility DESC; ``` ## What's happening behind the scenes? [Section titled “What's happening behind the scenes?”](#whats-happening-behind-the-scenes) ### Columnstore compression [Section titled “Columnstore compression”](#columnstore-compression) With `tsdb.enable_columnstore=true`: * Data is stored in a hybrid row-columnar format. * Analytical queries only scan the columns they need, for a large speedup. * Typical compression ratios: 90% or more for time-series data. * Compression happens transparently, with no changes to your queries. ### Direct to columnstore [Section titled “Direct to columnstore”](#direct-to-columnstore) When you use `SET timescaledb.enable_direct_compress_copy = on`: * Data loads directly into compressed columnstore format. * Bypasses the rowstore entirely. * Provides instant analytical performance, with no waiting for background compression. ### Segmentation [Section titled “Segmentation”](#segmentation) The `tsdb.segmentby='ticker'` setting: * Groups data by ticker within each chunk. * Improves compression ratios by keeping similar data together. * Speeds up queries that filter by ticker. * Is better for ticker-based analytics. ## Continuous aggregates (Advanced) [Section titled “Continuous aggregates (Advanced)”](#continuous-aggregates-advanced) For real-time dashboards, you can create continuous aggregates that automatically update: ```sql -- Create a continuous aggregate for hourly candlesticks CREATE MATERIALIZED VIEW candlesticks_hourly WITH (timescaledb.continuous) AS SELECT time_bucket('1 hour', ts) AS hour, ticker, FIRST(price, ts) AS open_price, MAX(price) AS high_price, MIN(price) AS low_price, LAST(price, ts) AS close_price, AVG(price) AS avg_price, SUM(volume) AS sum_volume FROM stock_prices GROUP BY hour, ticker ORDER BY hour DESC, ticker ASC; -- Add a refresh policy to keep it updated SELECT add_continuous_aggregate_policy('candlesticks_hourly', start_offset => INTERVAL '2 hours', end_offset => INTERVAL '1 hour', schedule_interval => INTERVAL '1 hour'); ``` Now you can query `candlesticks_hourly` for instant results on pre-aggregated data: ```sql SELECT * from candlesticks_hourly WHERE ticker = 'NFLX'; ``` # Visualize transport and geospatial data with Grafana > Simulate and analyze a transport dataset in Tiger Cloud and visualize it in Grafana This walkthrough uses NYC taxi-style geospatial time series in Tiger Cloud service, then layers Grafana on top so you can see movement and hotspots as they change instead of after the fact. ![Grafana heatmap showing NYC taxi trip pickup locations over time](/docs/_astro/use-case-rta-grafana-heatmap.Brk4prT1_ZqnYIW.webp) This page shows you how to integrate [Grafana](https://grafana.com/docs/) with a Tiger Cloud service and make insights based on visualization of data optimized for size and speed in the columnstore. ## Prerequisites for this tutorial To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service). These steps use Tiger Cloud, but the same approach applies to a [self-hosted TimescaleDB](/docs/get-started/choose-your-path/install-timescaledb) instance. - A [self-managed Grafana](https://grafana.com/get/?tab=self-managed) instance, or a [Grafana Cloud](https://grafana.com/get/) account. ## Optimize time-series data in hypertables Hypertables are PostgreSQL tables in TimescaleDB that automatically partition your time-series data by time. Time-series data represents the way a system, process, or behavior changes over time. Hypertables enable TimescaleDB to work efficiently with time-series data. Each hypertable is made up of child tables called chunks. Each chunk is assigned a range of time, and only contains data from that range. When you run a query, TimescaleDB identifies the correct chunk and runs the query on it, instead of going through the entire table. [Hypercore](/docs/learn/columnar-storage/understand-hypercore) is the hybrid row-columnar storage engine in TimescaleDB used by hypertables. Traditional databases force a trade-off between fast inserts (row-based storage) and efficient analytics (columnar storage). Hypercore eliminates this trade-off, allowing real-time analytics without sacrificing transactional capabilities. Hypercore dynamically stores data in the most efficient format for its lifecycle: ![Move from rowstore to columstore in hypercore](/docs/_astro/hypercore_intro.DmLiqZGV_1VM5z7.webp) ![Move from rowstore to columstore in hypercore](/docs/_astro/hypercore_intro-dark.qPmjkBqW_19V2rF.webp) * **Row-based storage for recent data**: the most recent chunk (and possibly more) is always stored in the rowstore, ensuring fast inserts, updates, and low-latency single record queries. Additionally, row-based storage is used as a writethrough for inserts and updates to columnar storage. * **Columnar storage for analytical performance**: chunks are automatically compressed into the columnstore, optimizing storage efficiency and accelerating analytical queries. Unlike traditional columnar databases, hypercore allows data to be inserted or modified at any stage, making it a flexible solution for both high-ingest transactional workloads and real-time analytics, within a single database. Because TimescaleDB is 100% PostgreSQL, you can use all the standard PostgreSQL tables, indexes, stored procedures, and other objects alongside your hypertables. This makes creating and working with hypertables similar to standard PostgreSQL. 1. **Import time-series data into a hypertable** 1. Unzip [nyc\_data.tar.gz](https://assets.timescale.com/docs/downloads/nyc_data.tar.gz) to a ``. This test dataset contains historical data from New York's yellow taxi network. To import up to 100GB of data directly from your current PostgreSQL-based database, [migrate with downtime](/docs/migrate/migrate-with-downtime) using native PostgreSQL tooling. To seamlessly import 100GB-10TB+ of data, use the [live migration](/docs/migrate/live-migration) tooling supplied by Tiger Data. To add data from non-PostgreSQL data sources, see [Import and ingest data](/docs/migrate/import-terminal). 2. In the terminal, navigate to `` and update the following string with [your connection details](/docs/integrate/find-connection-details) to connect to your service. ```bash psql -d "postgres://:@:/?sslmode=require" ``` 3. Create an optimized hypertable for your time-series data: 1. Create a [hypertable](/docs/learn/hypertables/understand-hypertables) with [hypercore](/docs/learn/columnar-storage/understand-hypercore) enabled by default for your time-series data using [CREATE TABLE](/docs/reference/timescaledb). For [efficient queries](/docs/build/performance-optimization/secondary-indexes) on data in the columnstore, remember to `segmentby` the column you will use most often to filter your data. In your SQL client, run the following command: ```sql CREATE TABLE "rides"( vendor_id TEXT, pickup_datetime TIMESTAMP WITHOUT TIME ZONE NOT NULL, dropoff_datetime TIMESTAMP WITHOUT TIME ZONE NOT NULL, passenger_count NUMERIC, trip_distance NUMERIC, pickup_longitude NUMERIC, pickup_latitude NUMERIC, rate_code INTEGER, dropoff_longitude NUMERIC, dropoff_latitude NUMERIC, payment_type INTEGER, fare_amount NUMERIC, extra NUMERIC, mta_tax NUMERIC, tip_amount NUMERIC, tolls_amount NUMERIC, improvement_surcharge NUMERIC, total_amount NUMERIC ) WITH ( tsdb.hypertable, tsdb.create_default_indexes=false, tsdb.segmentby='vendor_id', tsdb.orderby='pickup_datetime DESC' ); ``` When you create a hypertable using [CREATE TABLE ... WITH ...](/docs/reference/timescaledb/hypertables/create_table), the default partitioning column is automatically the first column with a timestamp data type. Also, TimescaleDB creates a [columnstore policy](/docs/reference/timescaledb/hypercore/add_columnstore_policy) that automatically converts your data to the columnstore, after an interval equal to the value of the [chunk\_interval](/docs/reference/timescaledb/hypertables/set_chunk_time_interval), defined through `after` in the policy. This columnar format enables fast scanning and aggregation, optimizing performance for analytical workloads while also saving significant storage space. In the columnstore conversion, hypertable chunks are compressed by up to 98%, and organized for efficient, large-scale queries. You can customize this policy later using [alter\_job](/docs/reference/timescaledb/jobs-automation/alter_job). However, to change `after` or `created_before`, the compression settings, or the hypertable the policy is acting on, you must [remove the columnstore policy](/docs/reference/timescaledb/hypercore/remove_columnstore_policy) and [add a new one](/docs/reference/timescaledb/hypercore/add_columnstore_policy). You can also manually [convert chunks](/docs/reference/timescaledb/hypercore/convert_to_columnstore) in a hypertable to the columnstore. 2. Add another dimension to partition your hypertable more efficiently: ```sql SELECT add_dimension('rides', by_hash('payment_type', 2)); ``` 3. Create an index to support efficient queries by vendor, rate code, and passenger count: ```sql CREATE INDEX ON rides (vendor_id, pickup_datetime DESC); CREATE INDEX ON rides (rate_code, pickup_datetime DESC); CREATE INDEX ON rides (passenger_count, pickup_datetime DESC); ``` 4. Create PostgreSQL tables for relational data: 1. Add a table to store the payment types data: ```sql CREATE TABLE IF NOT EXISTS "payment_types"( payment_type INTEGER, description TEXT ); INSERT INTO payment_types(payment_type, description) VALUES (1, 'credit card'), (2, 'cash'), (3, 'no charge'), (4, 'dispute'), (5, 'unknown'), (6, 'voided trip'); ``` 2. Add a table to store the rates data: ```sql CREATE TABLE IF NOT EXISTS "rates"( rate_code INTEGER, description TEXT ); INSERT INTO rates(rate_code, description) VALUES (1, 'standard rate'), (2, 'JFK'), (3, 'Newark'), (4, 'Nassau or Westchester'), (5, 'negotiated fare'), (6, 'group ride'); ``` 5. Upload the dataset to your service: ```sql \COPY rides FROM nyc_data_rides.csv CSV; ``` 2. **Have a quick look at your data** You query hypertables in exactly the same way as you would a relational PostgreSQL table. Use one of the following SQL editors to run a query and see the data you uploaded: * **Data view**: write queries, visualize data, and share your results in [Tiger Console](https://console.cloud.tigerdata.com/dashboard/services?popsql) for all your Tiger Cloud services. * **SQL editor**: write, fix, and organize SQL faster and more accurately in [Tiger Console](https://console.cloud.tigerdata.com/dashboard/services) for a Tiger Cloud service. * **psql**: easily run queries on your Tiger Cloud services or self-hosted TimescaleDB deployment from Terminal. For example: * Display the number of rides for each fare type: ```sql SELECT rate_code, COUNT(vendor_id) AS num_trips FROM rides WHERE pickup_datetime < '2016-01-08' GROUP BY rate_code ORDER BY rate_code; ``` This simple query runs in 3 seconds. You see something like: | rate\_code | num\_trips | | ---------- | ---------- | | 1 | 2266401 | | 2 | 54832 | | 3 | 4126 | | 4 | 967 | | 5 | 7193 | | 6 | 17 | | 99 | 42 | * To select all rides taken in the first week of January 2016, and return the total number of trips taken for each rate code: ```sql SELECT rates.description, COUNT(vendor_id) AS num_trips FROM rides JOIN rates ON rides.rate_code = rates.rate_code WHERE pickup_datetime < '2016-01-08' GROUP BY rates.description ORDER BY LOWER(rates.description); ``` On this large amount of data, this analytical query on data in the rowstore takes about 59 seconds. You see something like: | description | num\_trips | | --------------------- | ---------- | | group ride | 17 | | JFK | 54832 | | Nassau or Westchester | 967 | | negotiated fare | 7193 | | Newark | 4126 | | standard rate | 2266401 | ## Connect Grafana to Tiger Cloud To visualize the results of your queries, enable Grafana to read the data in your service: 1. **Log in to Grafana** In your browser, log in to either: * Self-hosted Grafana: at `http://localhost:3000/`. The default credentials are `admin`, `admin`. * Grafana Cloud: use the URL and credentials you set when you created your account. 2. **Add your service as a data source** 1. Open `Connections` > `Data sources`, then click `Add new data source`. 2. Select `PostgreSQL` from the list. 3. Configure the connection: * `Host URL`, `Database name`, `Username`, and `Password`, configure using your [connection details](/docs/integrate/find-connection-details). `Host URL` is in the format `:`. * `TLS/SSL Mode`: select `require`. * `PostgreSQL options`: enable `TimescaleDB`. * Leave the default setting for all other fields. 4. Click `Save & test`. Grafana checks that your details are set correctly. ## Monitor performance over time [Section titled “Monitor performance over time”](#monitor-performance-over-time) A Grafana dashboard represents a view into the performance of a system, and each dashboard consists of one or more panels, which represent information about a specific metric related to that system. To visually monitor the volume of taxi rides over time: 1. **Create the dashboard** 1. On the `Dashboards` page, click `New` and select `New dashboard`. 2. Click `Add visualization`. 3. Select the data source that connects to your Tiger Cloud service. The `Time series` visualization is chosen by default. ![Configuring a Grafana dashboard with a TimescaleDB data source](/docs/_astro/use-case-rta-grafana-timescale-configure-dashboard.PNpiH0JO_204Gcy.webp) 4. In the `Queries` section, select `Code`, then select `Time series` in `Format`. 5. Select the data range for your visualization: the data set is from 2016. Click the date range above the panel and set: * From: `2016-01-01 01:00:00` * To: `2016-01-30 01:00:00` 2. **Combine TimescaleDB and Grafana functionality to analyze your data** Combine a TimescaleDB [time\_bucket](/docs/learn/data-lifecycle/time-buckets/about-time-buckets), with the Grafana `$__timefilter()` function to set the `pickup_datetime` column as the filtering range for your visualizations. ```sql SELECT time_bucket('1 day', pickup_datetime) AS "time", COUNT(*) FROM rides WHERE $__timeFilter(pickup_datetime) GROUP BY time ORDER BY time; ``` This query groups the results by day and orders them by time. ![Completed Grafana dashboard showing NYC taxi ride count and geospatial data](/docs/_astro/use-case-rta-grafana-timescale-final-dashboard.CXJQRRVZ_oS2UG.webp) 3. **Click `Save dashboard`** ## Optimize revenue potential [Section titled “Optimize revenue potential”](#optimize-revenue-potential) Having all this data is great but how do you use it? Monitoring data is useful to check what has happened, but how can you analyse this information to your advantage? This section explains how to create a visualization that shows how you can maximize potential revenue. ### Set up your data for geospatial queries To add geospatial analysis to your ride count visualization, you need geospatial data to work out which trips originated where. As TimescaleDB is compatible with all PostgreSQL extensions, use [PostGIS](http://postgis.net/) to slice data by time and location. 1. **Connect to your Tiger Cloud service and add the PostGIS extension:** ```sql CREATE EXTENSION postgis; ``` 2. **Add geometry columns for pick up and drop off locations:** ```sql ALTER TABLE rides ADD COLUMN pickup_geom geometry(POINT,2163); ALTER TABLE rides ADD COLUMN dropoff_geom geometry(POINT,2163); ``` 3. **Convert the latitude and longitude points into geometry coordinates that work with PostGIS:** ```sql UPDATE rides SET pickup_geom = ST_Transform(ST_SetSRID(ST_MakePoint(pickup_longitude,pickup_latitude),4326),2163), dropoff_geom = ST_Transform(ST_SetSRID(ST_MakePoint(dropoff_longitude,dropoff_latitude),4326),2163); ``` This updates 10,906,860 rows of data on both columns, it takes a while. Coffee is your friend. You might run into this error while the update happens `Error: tuple decompression limit exceeded by operation Error Code: 53400 Details: current limit: 100000, tuples decompressed: 10906860 Hint: Consider increasing timescaledb.max_tuples_decompressed_per_dml_transaction or set to 0 (unlimited).` To fix this, use ```sql SET timescaledb.max_tuples_decompressed_per_dml_transaction TO 0; ``` ### Visualize the area where you can make the most money In this section you visualize a query that returns rides longer than 5 miles for trips taken within 2 km of Times Square. The data includes the distance travelled and is `GROUP BY` `trip_distance` and location so that Grafana can plot the data properly. This enables you to see where a taxi driver is most likely to pick up a passenger who wants a longer ride, and make more money. 1. **Create a geolocalization dashboard** 1. In Grafana, create a new dashboard that is connected to your Tiger Cloud service data source with a Geomap visualization. 2. In the `Queries` section, select `Code`, then select the Time series `Format`. ![Configuring a Grafana dashboard with a TimescaleDB data source](/docs/_astro/use-case-rta-grafana-timescale-configure-dashboard.PNpiH0JO_204Gcy.webp) 3. To find rides longer than 5 miles in Manhattan, paste the following query: ```sql SELECT time_bucket('5m', rides.pickup_datetime) AS time, rides.trip_distance AS value, rides.pickup_latitude AS latitude, rides.pickup_longitude AS longitude FROM rides WHERE rides.pickup_datetime BETWEEN '2016-01-01T01:41:55.986Z' AND '2016-01-01T07:41:55.986Z' AND ST_Distance(pickup_geom, ST_Transform(ST_SetSRID(ST_MakePoint(-73.9851,40.7589),4326),2163) ) < 2000 GROUP BY time, rides.trip_distance, rides.pickup_latitude, rides.pickup_longitude ORDER BY time LIMIT 500; ``` You see a world map with a dot on New York. 4. Zoom into your map to see the visualization clearly. 2. **Customize the visualization** In the Geomap options, under `Map Layers`, click `+ Add layer` and select `Heatmap`. You now see the areas where a taxi driver is most likely to pick up a passenger who wants a longer ride, and make more money. ![Grafana heatmap showing NYC taxi trip pickup locations](/docs/_astro/use-case-rta-grafana-heatmap.Brk4prT1_ZqnYIW.webp) You have integrated Grafana with a Tiger Cloud service and made insights based on visualization of your data. # Tiger Data cookbook > Browse code examples from the Tiger community that resolve common issues and provide solutions for non-standard tasks The [Tiger Data Community](https://timescaledb.slack.com/) shared these snippets for odd problems and non-standard setups. Treat them as patterns to adapt, not as fully supported product recipes. ## Prerequisites for this page To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service). These steps use Tiger Cloud, but the same approach applies to a [self-hosted TimescaleDB](/docs/get-started/choose-your-path/install-timescaledb) instance. ## Hypertable tutorials This section contains short tutorials about hypertables. ### Remove duplicates from an existing hypertable Looking to remove duplicates from an existing hypertable? One method is to run a `PARTITION BY` query to get `ROW_NUMBER()` and then the `ctid` of rows where `row_number>1`. You then delete these rows. However, you need to check `tableoid` and `ctid`. This is because `ctid` is not unique and might be duplicated in different chunks. The following code example took 17 hours to process a table with 40 million rows: ```sql CREATE OR REPLACE FUNCTION deduplicate_chunks(ht_name TEXT, partition_columns TEXT, bot_id INT DEFAULT NULL) RETURNS TABLE ( chunk_schema name, chunk_name name, deleted_count INT ) AS $$ DECLARE chunk RECORD; where_clause TEXT := ''; deleted_count INT; BEGIN IF bot_id IS NOT NULL THEN where_clause := FORMAT('WHERE bot_id = %s', bot_id); END IF; FOR chunk IN SELECT c.chunk_schema, c.chunk_name FROM timescaledb_information.chunks c WHERE c.hypertable_name = ht_name LOOP EXECUTE FORMAT(' WITH cte AS ( SELECT ctid, ROW_NUMBER() OVER (PARTITION BY %s ORDER BY %s ASC) AS row_num, * FROM %I.%I %s ) DELETE FROM %I.%I WHERE ctid IN ( SELECT ctid FROM cte WHERE row_num > 1 ) RETURNING 1; ', partition_columns, partition_columns, chunk.chunk_schema, chunk.chunk_name, where_clause, chunk.chunk_schema, chunk.chunk_name) INTO deleted_count; RETURN QUERY SELECT chunk.chunk_schema, chunk.chunk_name, COALESCE(deleted_count, 0); END LOOP; END $$ LANGUAGE plpgsql; SELECT * FROM deduplicate_chunks('nudge_events', 'bot_id, session_id, nudge_id, time', 2540); ``` Shoutout to **Mathias Ose** and **Christopher Piggott** for this tutorial. ### Get faster JOIN queries with Common Table Expressions Imagine there is a query that joins a hypertable to another table on a shared key: ```sql SELECT timestamp, FROM hypertable as h JOIN related_table as rt ON rt.id = h.related_table_id WHERE h.timestamp BETWEEN '2024-10-10 00:00:00' AND '2024-10-17 00:00:00' ``` If you run `EXPLAIN` on this query, you see that the query planner performs a `NestedJoin` between these two tables, which means querying the hypertable multiple times. Even if the hypertable is well indexed, if it is also large, the query will be slow. How do you force a once-only lookup? Use materialized Common Table Expressions (CTEs). If you split the query into two parts using CTEs, you can `materialize` the hypertable lookup and force PostgreSQL to perform it only once. ```sql WITH cached_query AS materialized ( SELECT * FROM hypertable WHERE BETWEEN '2024-10-10 00:00:00' AND '2024-10-17 00:00:00' ) SELECT * FROM cached_query as c JOIN related_table as rt ON rt.id = h.related_table_id ``` Now if you run `EXPLAIN` once again, you see that this query performs only one lookup. Depending on the size of your hypertable, this could result in a multi-hour query taking mere seconds. Shoutout to **Rowan Molony** for this tutorial. ## IoT tutorials This section contains short tutorials for common IoT scenarios: ### Work with columnar IoT data Narrow and medium width tables are a great way to store IoT data. A lot of reasons are outlined in [Designing Your Database Schema: Wide vs. Narrow PostgreSQL Tables](https://www.tigerdata.com/learn/designing-your-database-schema-wide-vs-narrow-postgres-tables). One of the key advantages of narrow tables is that the schema does not have to change when you add new sensors. Another big advantage is that each sensor can sample at different rates and times. This helps support things like hysteresis, where new values are written infrequently unless the value changes by a certain amount. #### Narrow table format example Working with narrow table data structures presents a few challenges. In the IoT world one concern is that many data analysis approaches - including machine learning as well as more traditional data analysis - require that your data is resampled and synchronized to a common time basis. Fortunately, TimescaleDB provides you with [hyperfunctions](/docs/learn/hyperfunctions/about-hyperfunctions) and other tools to help you work with this data. An example of a narrow table format is: | ts | sensor\_id | value | | ----------------------- | ---------- | ----- | | 2024-10-31 11:17:30.000 | 1007 | 23.45 | Typically you would couple this with a sensor table: | sensor\_id | sensor\_name | units | | ---------- | ------------- | ------------------------ | | 1007 | temperature | degreesC | | 1012 | heat\_mode | on/off | | 1013 | cooling\_mode | on/off | | 1041 | occupancy | number of people in room | A medium table retains the generic structure but adds columns of various types so that you can use the same table to store float, int, bool, or even JSON (jsonb) data: | ts | sensor\_id | d | i | b | t | j | | ----------------------- | ---------- | ----- | ---- | ---- | ---- | ---- | | 2024-10-31 11:17:30.000 | 1007 | 23.45 | null | null | null | null | | 2024-10-31 11:17:47.000 | 1012 | null | null | TRUE | null | null | | 2024-10-31 11:18:01.000 | 1041 | null | 4 | null | null | null | To remove all-null entries, use an optional constraint such as: ```sql CONSTRAINT at_least_one_not_null CHECK ((d IS NOT NULL) OR (i IS NOT NULL) OR (b IS NOT NULL) OR (j IS NOT NULL) OR (t IS NOT NULL)) ``` #### Get the last value of every sensor There are several ways to get the latest value of every sensor. The following examples use the structure defined in [Narrow table format example](#narrow-table-format-example) as a reference ##### SELECT DISTINCT ON If you have a list of sensors, the easy way to get the latest value of every sensor is to use `SELECT DISTINCT ON`: ```sql WITH latest_data AS ( SELECT DISTINCT ON (sensor_id) ts, sensor_id, d FROM iot_data WHERE d is not null AND ts > CURRENT_TIMESTAMP - INTERVAL '1 week' -- important ORDER BY sensor_id, ts DESC ) SELECT sensor_id, sensors.name, ts, d FROM latest_data LEFT OUTER JOIN sensors ON latest_data.sensor_id = sensors.id WHERE latest_data.d is not null ORDER BY sensor_id, ts; -- Optional, for displaying results ordered by sensor_id ``` The common table expression (CTE) used above is not strictly necessary. However, it is an elegant way to join to the sensor list to get a sensor name in the output. If this is not something you care about, you can leave it out: ```sql SELECT DISTINCT ON (sensor_id) ts, sensor_id, d FROM iot_data WHERE d is not null AND ts > CURRENT_TIMESTAMP - INTERVAL '1 week' -- important ORDER BY sensor_id, ts DESC ``` It is important to take care when down-selecting this data. In the previous examples, the time that the query would scan back was limited. However, if there any sensors that have either not reported in a long time or in the worst case, never reported, this query devolves to a full table scan.\ In a database with 1000+ sensors and 41 million rows, an unconstrained query takes over an hour. #### JOIN LATERAL An alternative to [SELECT DISTINCT ON](#select-distinct-on) is to use a `JOIN LATERAL`. By selecting your entire sensor list from the sensors table rather than pulling the IDs out using `SELECT DISTINCT`, `JOIN LATERAL` can offer some improvements in performance: ```sql SELECT sensor_list.id, latest_data.ts, latest_data.d FROM sensors sensor_list -- Add a WHERE clause here to downselect the sensor list, if you wish LEFT JOIN LATERAL ( SELECT ts, d FROM iot_data raw_data WHERE sensor_id = sensor_list.id ORDER BY ts DESC LIMIT 1 ) latest_data ON true WHERE latest_data.d is not null -- only pulling out float values ("d" column) in this example AND latest_data.ts > CURRENT_TIMESTAMP - interval '1 week' -- important ORDER BY sensor_list.id, latest_data.ts; ``` Limiting the time range is important, especially if you have a lot of data. Best practice is to use these kinds of queries for dashboards and quick status checks. To query over a much larger time range, encapsulate the previous example into a materialized query that refreshes infrequently, perhaps once a day. Shoutout to **Christopher Piggott** for this tutorial. # Create Tiger Cloud services with the Terraform provider > Use the Tiger Data Terraform provider to deploy, manage, and destroy Tiger Cloud services using infrastructure as code. ![](https://avatars.githubusercontent.com/mathisve?s=128) Mathis Van Eetvelde Software Engineer [GitHub](https://github.com/mathisve) In this tutorial, you learn how to use the [Tiger Data Terraform provider](https://registry.terraform.io/providers/timescale/timescale/latest/docs) to create and manage Tiger Cloud services using infrastructure as code (IaC). This tutorial is for developers and DevOps engineers who want to automate their Tiger Cloud deployments. It assumes you have basic knowledge of: * Terraform concepts (providers, resources, outputs) * Command-line interfaces * Tiger Cloud services By the end, you'll be able to: * Configure the Tiger Data Terraform provider with credentials * Deploy a Tiger Cloud service using a Terraform configuration * Retrieve service connection details, including the password, using Terraform outputs ## Background [Section titled “Background”](#background) Terraform is an infrastructure-as-code tool that lets you declaratively configure cloud resources. Your cloud infrastructure becomes a one-to-one representation of your configuration files, which improves security, auditability, and accountability. One of the biggest practical benefits is deployment speed, instead of manually clicking through a console, you can deploy, reuse, and recycle configurations in seconds. Terraform's idempotent nature also makes recovery straightforward. If something goes wrong during a deployment, re-running `terraform apply` brings your infrastructure back to the desired state without manual intervention. The Tiger Data Terraform provider brings these benefits to Tiger Cloud, letting you create, configure, and destroy services entirely from code, no UI required. Tips As of the Terraform provider v1.2.0, you can also create read replicas with Terraform. This means you can automatically provision or remove read replicas, for example, to spin up a replica for a business analyst to run BI queries against. ## What is the tech stack we're working with? [Section titled “What is the tech stack we're working with?”](#what-is-the-tech-stack-were-working-with) * **Terraform:** An open-source infrastructure-as-code tool by HashiCorp that lets you define and provision cloud infrastructure using declarative configuration files. Terraform's idempotent nature means you can re-run deployments safely: if something goes wrong, re-running `terraform apply` brings your infrastructure back to the desired state. * **Tiger Data Terraform provider:** The [official provider](https://registry.terraform.io/providers/timescale/timescale/latest/docs) in the Terraform Registry (`timescale/timescale`) that lets you manage Tiger Cloud services, VPC connections, and read replicas. ## Prerequisites for this tutorial To follow these steps, you'll need: * A [Tiger Cloud account](/docs/get-started/quickstart/create-service) * [Terraform installed](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli) on your local machine * Your [Tiger Cloud project ID](/docs/integrate/find-connection-details#find-your-project-and-service-id) and [client credentials](/docs/integrate/find-connection-details#create-client-credentials) Store your credentials securely. Never commit them to version control. For production use, consider HashiCorp Vault, Terraform environment variables, or a cloud KMS solution. ## Configure the Terraform provider [Section titled “Configure the Terraform provider”](#configure-the-terraform-provider) Set up the Tiger Data Terraform provider and authenticate with your Tiger Cloud project. 1. **Declare the provider** Create a `main.tf` file and add the provider declaration. Set the source to `timescale/timescale` to use the official provider from the Terraform Registry: ```hcl terraform { required_providers { timescale = { source = "timescale/timescale" version = "~> 1.0" } } } ``` Using the `~>` operator pins to the latest compatible minor version. You can also omit the `version` argument entirely to always use the latest release. 2. **Configure provider credentials** Add the provider configuration block to `main.tf`: ```hcl provider "timescale" { project_id = var.ts_project_id access_key = var.ts_access_key secret_key = var.ts_secret_key } ``` 3. **Define the credential variables** Create a `variables.tf` file to declare the input variables: ```hcl variable "ts_project_id" { type = string description = "Tiger Cloud project ID" } variable "ts_access_key" { type = string description = "Tiger Cloud access key" } variable "ts_secret_key" { type = string sensitive = true description = "Tiger Cloud secret key" } ``` Then create a `terraform.tfvars` file or export environment variables to supply the values: ```bash export TF_VAR_ts_project_id="" export TF_VAR_ts_access_key="" export TF_VAR_ts_secret_key="" ``` Warning Never share your secret key with anyone. It grants full ability to create and delete Tiger Cloud services in your project until the credentials are revoked. ## Create a service [Section titled “Create a service”](#create-a-service) Define and deploy a Tiger Cloud service using your Terraform configuration. 1. **Add the service resource** Add a `timescale_service` resource to your `main.tf` file. Configuring a service through Terraform is similar to the advanced configuration menu in Tiger Console: ```hcl resource "timescale_service" "my_service" { name = "my-service" milli_cpu = 1000 memory_gb = 4 region_code = "us-east-1" lifecycle { prevent_destroy = true } } ``` Here's what each field does: * **`name`**: The display name for your service. This doesn't need to match the Terraform resource name (`my_service` in this example). * **`milli_cpu`** and **`memory_gb`**: CPU and memory allocation. This example uses 1 CPU (1000 milli-CPU) and 4 GB of memory. Only specific combinations are supported: consult the [provider documentation](https://registry.terraform.io/providers/timescale/timescale/latest/docs) for valid options. * **`region_code`**: The cloud region for your service. This example uses `us-east-1`, but you can choose any [supported region](https://registry.terraform.io/providers/timescale/timescale/latest/docs). Tiger Cloud automatically allocates and charges only for the storage you use, so you don't need to pre-provision a specific storage size. Warning The `prevent_destroy` lifecycle rule is highly recommended. If a Tiger Cloud service is destroyed, the data it stores is destroyed too. Remove or comment out the lifecycle block only when you intentionally want to delete the service and its data. 2. **Initialize and deploy** Run the following commands in order: ```bash terraform init terraform plan terraform apply ``` Terraform shows you the planned changes, including the resource attributes that will be created. Enter `yes` when prompted to confirm: ```plaintext Terraform will perform the following actions: # timescale_service.my_service will be created + resource "timescale_service" "my_service" { + enable_ha_replica = false + hostname = (known after apply) + id = (known after apply) + memory_gb = 4 + milli_cpu = 1000 + name = "my-service" + password = (sensitive value) + port = (known after apply) + region_code = "us-east-1" + username = (known after apply) } Plan: 1 to add, 0 to change, 0 to destroy. Do you want to perform these actions? Terraform will perform the actions described above. Only 'yes' will be accepted to approve. Enter a value: yes timescale_service.my_service: Creating... timescale_service.my_service: Creation complete after 38s Apply complete! Resources: 1 added, 0 changed, 0 destroyed. ``` Notice that values like `hostname`, `port`, `username`, and `password` are marked as `(known after apply)`, these are only available after the service is created. You retrieve them using Terraform outputs in the next section. ## Retrieve service connection details [Section titled “Retrieve service connection details”](#retrieve-service-connection-details) After deploying the service, your first instinct might be to go to Tiger Console and copy the connection string. However, the Tiger Console connection string doesn't include the password. The only way to retrieve the password for a Terraform-managed service is through Terraform outputs. 1. **Define the outputs** Create an `outputs.tf` file: ```hcl output "service_hostname" { value = timescale_service.my_service.hostname } output "service_port" { value = timescale_service.my_service.port } output "service_username" { value = timescale_service.my_service.username } output "service_password" { value = timescale_service.my_service.password sensitive = true } ``` 2. **Apply and retrieve the outputs** Re-apply and then view the outputs: ```bash terraform apply terraform output ``` Terraform shows the hostname, port, and username in plain text, but keeps the password hidden: ```plaintext service_hostname = "abc123.tsdb.cloud.tigerdata.com" service_password = service_port = "30211" service_username = "tsdbadmin" ``` This is because the password is marked `sensitive`, Terraform avoids printing it to the terminal since you never know where your logs might end up. To retrieve the password, use the `-json` flag to output all values in JSON format: ```bash terraform output -json ``` This prints all outputs including sensitive values. To extract just the password, pipe the output through `jq`: ```bash terraform output -json | jq -r ".service_password.value" ``` 3. **(Optional) Synthesize a full connection string** You can compose the full connection string as a Terraform output. This is useful when other Terraform resources need to connect to your Tiger Cloud service: ```hcl output "service_url" { value = format( "postgres://%s:%s@%s:%s/tsdb?sslmode=require", timescale_service.my_service.username, timescale_service.my_service.password, timescale_service.my_service.hostname, timescale_service.my_service.port ) sensitive = true } ``` ## Summary [Section titled “Summary”](#summary) In this tutorial, you learned how to: * Configure the Tiger Data Terraform provider with project credentials * Deploy a Tiger Cloud service with CPU, memory, and region settings * Use `prevent_destroy` to protect against accidental data loss * Retrieve connection details and passwords using Terraform outputs ## Next steps [Section titled “Next steps”](#next-steps) ### Learn more [Section titled “Learn more”](#learn-more) [Integrate Terraform with Tiger Cloud](/docs/integrate/configuration-deployment/terraform) [Full reference for configuring Terraform with Tiger Cloud and self-hosted deployments.](/docs/integrate/configuration-deployment/terraform) [Create a Tiger Cloud service](/docs/get-started/quickstart/create-service) [Create your first service using Tiger Console.](/docs/get-started/quickstart/create-service) ### Other examples [Section titled “Other examples”](#other-examples) [Simulate an IoT sensor dataset](/docs/build/examples/simulate-iot-sensor-data) [Generate sample IoT time-series data for testing and development.](/docs/build/examples/simulate-iot-sensor-data) [Ingest real-time financial data](/docs/build/examples/ingest-real-time-financial-data) [Stream live cryptocurrency data into Tiger Cloud.](/docs/build/examples/ingest-real-time-financial-data) # Build hybrid search with BM25 and vector similarity > Combine keyword search and semantic vector search in PostgreSQL using pg_textsearch and pgvectorscale, fused with Reciprocal Rank Fusion. Ever searched for something and thought "I *know* it's in here somewhere, but the search bar just isn't getting it"? That's because most search systems only do one thing well. Hybrid search fixes that by combining the best of both worlds: * **Full-text search** (BM25) excels at exact keyword matching and linguistic features like stemming and ranking. It's the reliable friend who finds exactly what you asked for. * **Vector similarity search** captures semantic meaning, finding results that are conceptually related even without keyword overlap. It's the clever friend who knows what you *meant*. In this tutorial, you combine both approaches in PostgreSQL using [`pg_textsearch`](/docs/learn/search/using-pg-textsearch) and [`pgvectorscale`](/docs/learn/search/pgvector-pgvectorsearch), then fuse the results with Reciprocal Rank Fusion (RRF). The result? Search that's both precise *and* smart. By the end of this tutorial, you'll be able to: * Create a PostgreSQL table with both text and vector embedding columns * Generate embeddings using OpenAI's API * Run BM25 keyword search with `pg_textsearch` * Run vector similarity search with `pgvectorscale`'s StreamingDiskANN index * Combine both methods into a single ranked list using Reciprocal Rank Fusion ## Prerequisites for this tutorial To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service) running PostgreSQL 17+, or a [self-hosted PostgreSQL instance](/docs/get-started/choose-your-path/install-timescaledb) with `pg_textsearch` and `pgvectorscale` installed * [Docker](https://docs.docker.com/get-started/get-docker/) installed and running, if you plan to use the Docker setup option. Install [Docker Desktop](https://docs.docker.com/desktop/) (macOS, Windows, Linux GUI) or [Docker Engine](https://docs.docker.com/engine/install/) (Linux server) * [Python 3.9+](https://www.python.org/downloads/) with a package manager ([uv](https://docs.astral.sh/uv/), [pip](https://pip.pypa.io/en/stable/installation/), or [conda](https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html)) * An [OpenAI API key](https://platform.openai.com/api-keys) for generating embeddings * A SQL client such as `psql`, pgAdmin, or the Tiger Cloud SQL editor *** Tips The complete source code, setup scripts, and sample data for this tutorial are available in the [cookbook-search repository on GitHub](https://github.com/timescale/cookbook-search/tree/main/Hybrid-search). You can clone the repository to get started quickly, or follow this tutorial step by step to build everything from scratch. ## Step 1: Set up your database [Section titled “Step 1: Set up your database”](#step-1-set-up-your-database) First things first: you need a PostgreSQL database with `pg_textsearch` and `pgvectorscale` installed. Pick your adventure: **Option 1: Use Tiger Cloud (recommended for beginners)** This is the fastest way to get started, no installation required. Tiger Cloud services running PostgreSQL 17+ already have `pg_textsearch` and `pgvectorscale` installed and ready to use. You can manage your Tiger Cloud service through the web console, the Tiger CLI, or the Tiger MCP from your AI assistant. The steps below cover all three options. 1. **Install the Tiger CLI (optional but recommended)** The Tiger CLI lets you manage Tiger Cloud resources from your terminal. Install it with the following command: ```bash # macOS brew install timescale/tap/tiger # Linux / WSL curl -sL https://assets.tigerdata.com/releases/install-tiger-cli.sh | sh ``` After installing, authenticate with your Tiger Cloud account: ```bash tiger auth login ``` This opens a browser window for you to log in. Once authorized, verify the connection by listing your services: ```bash tiger service list ``` For the full setup guide, see [Get started with the command line](/docs/get-started/quickstart/cli-rest-api). 2. **Set up Tiger MCP for your AI assistant (optional)** Tiger MCP gives your AI assistant (Claude Code, Cursor, Windsurf, and others) direct access to Tiger Cloud so you can manage services and run queries using natural language. It's bundled with the Tiger CLI you just installed. Run the following in your terminal: ```bash tiger mcp install ``` Choose your AI assistant from the list (for example, `claude-code`, `cursor`, `windsurf`) and press Enter. After installation, start your AI assistant and ask: *"Is the Tiger MCP server active?"* to confirm it's connected. For the full setup guide, see [Integrate Tiger Cloud with your AI assistant](/docs/get-started/quickstart/mcp-cli). 3. **Create or select a Tiger Cloud service** Sign up or log in at the [Tiger Cloud console](https://console.cloud.timescale.com) and [create a new service](/docs/get-started/quickstart/create-service), or use an existing one running PostgreSQL 17+. Alternatively, if you installed the Tiger CLI, create a service from your terminal: ```bash tiger service create ``` 4. **Connect to your database** Run queries directly in the Tiger Cloud SQL editor, or connect with `psql` using your service credentials. If you're using the Tiger CLI, you can also connect directly: ```bash tiger service connect ``` Replace `` with the ID shown in `tiger service list`. **Option 2: Use Docker (recommended for local development)** Run everything locally with the [`timescaledb-docker-ha`](https://github.com/timescale/timescaledb-docker-ha) image. It ships with PostgreSQL, TimescaleDB, pgvector, `pgvectorscale`, and `pg_textsearch` pre-installed. 1. **Pull and run the container** Run the following command in your terminal to download the Docker image and start a PostgreSQL 17 container with the extensions pre-installed: ```bash docker run -d --name hybrid-search \ -p 5432:5432 \ -e POSTGRES_PASSWORD=password \ timescale/timescaledb-ha:pg17 ``` This creates a container named `hybrid-search`, maps port 5432 on your machine to the database inside the container, and sets the `postgres` user password to `password`. Note The image supports `arm64`, so it runs natively on Apple Silicon Macs. No Rosetta needed. 2. **Connect to your database** In your terminal, use `psql` to connect to the PostgreSQL instance running inside the container: ```bash psql -h localhost -U postgres ``` Enter the password you set in the previous step (`password`) when prompted. 3. **Create the extensions** Once connected to `psql`, run the following two SQL statements to enable `pg_textsearch` for keyword search and `pgvectorscale` for vector search: ```sql CREATE EXTENSION pg_textsearch; CREATE EXTENSION vectorscale CASCADE; ``` You should see `CREATE EXTENSION` printed after each command. The `CASCADE` keyword in the second statement automatically installs pgvector as a dependency. Tips `shared_preload_libraries` is already configured in the Docker image, so there's no need to edit `postgresql.conf` or restart the server. **Option 3: Manual install** If you prefer to run PostgreSQL locally without Docker, install the extensions yourself. 1. **Download the extensions** Download the pre-built packages from GitHub: * `pg_textsearch`: [GitHub releases](https://github.com/timescale/pg_textsearch/releases) * `pgvectorscale`: [GitHub releases](https://github.com/timescale/pgvectorscale/releases) Follow the installation instructions included with each release for your operating system. 2. **Update your PostgreSQL configuration** `pg_textsearch` needs to be loaded when PostgreSQL starts up. First, find the location of your `postgresql.conf` file by running this query in your SQL client: ```sql SHOW config_file; ``` Open the `postgresql.conf` file in a text editor and find the `shared_preload_libraries` line. Update it to include `pg_textsearch`: ```ini shared_preload_libraries = 'pg_textsearch' ``` If other extensions are already listed, add `pg_textsearch` to the comma-separated list. After saving the file, restart PostgreSQL from your terminal. A config reload is not sufficient; a full restart is required: ```bash # Linux (systemd) sudo systemctl restart postgresql # macOS (Homebrew) brew services restart postgresql@17 ``` 3. **Create the extensions** Connect to your database using `psql` or any SQL client and run the following two statements to enable `pg_textsearch` and `pgvectorscale`: ```sql CREATE EXTENSION pg_textsearch; CREATE EXTENSION vectorscale CASCADE; ``` The `CASCADE` keyword automatically installs pgvector as a dependency. To confirm everything is installed correctly, run this verification query in the same SQL session: ```sql SELECT extname, extversion FROM pg_extension WHERE extname IN ('pg_textsearch', 'vectorscale', 'vector'); ``` You should see three rows: ```plaintext extname | extversion ---------------+------------ vector | 0.8.0 pg_textsearch | 1.0.0 vectorscale | 0.7.0 ``` Your version numbers may differ, and that's fine as long as all three appear. For more details on configuring `pg_textsearch`, see the [pg\_textsearch deploy guide](/docs/deploy/tiger-cloud/tiger-cloud-aws/tiger-cloud-extensions/pg-textsearch). ## Step 2: Create a table with text and embeddings [Section titled “Step 2: Create a table with text and embeddings”](#step-2-create-a-table-with-text-and-embeddings) Now for the fun part: let's give your database something to search through. Tips **Quick start:** To skip the manual steps in Steps 2–4 and run all the setup at once (extensions, table, data, and indexes), run the following command in your terminal. It executes the [`setup.sql`](https://github.com/timescale/cookbook-search/blob/main/Hybrid-search/setup.sql) file from the repository against your database: ```bash psql -h localhost -U postgres -f setup.sql ``` If you use this shortcut, skip ahead to [Step 3: Generate embeddings](#step-3-generate-embeddings) to create the embedding vectors. This tutorial uses episode data from [Conduit](https://www.relay.fm/conduit), a productivity podcast by Jay Miller and Kathy Campbell on Relay FM. Transcripts are from the [conduit-transcripts](https://github.com/kjaymiller/conduit-transcripts) repository (MIT License, Jay Miller). 1. **Create the episodes table** Run the following SQL in your SQL client (the Tiger Cloud SQL editor, `psql`, or pgAdmin) to create a table that stores podcast episode metadata alongside a vector embedding column: ```sql CREATE TABLE episodes ( id bigserial PRIMARY KEY, title text, description text, pub_date date, url text, embedding vector(1536) -- OpenAI text-embedding-3-small dimensions ); ``` The `embedding` column uses the `vector` type from pgvector to store 1536-dimensional vectors. This column starts empty, and you populate it with embeddings in the next step. 2. **Insert sample data** Run this `INSERT` statement in the same SQL session to load 12 sample episodes into the table. Each row includes a title, description, publication date, and URL: ```sql INSERT INTO episodes (title, description, pub_date, url) VALUES ('1: Our Systems: The Unicorn & Silk Sonic Methods', 'For most people, productivity starts with their system. Jay and Kathy talk about their own brand of productivity and what their personal systems look like.', '2021-07-15', 'https://www.relay.fm/conduit/1'), ('5: Sustained Progress: Over Being Overwhelmed', 'Millennial Falcon wants to know how to make SUSTAINED progress on projects that feel more like a marathon, not a sprint. Kathy just made a big move and gives us some of the tips that she used to make this challenge a bit more manageable.', '2021-09-09', 'https://www.relay.fm/conduit/5'), ('13: Happiness First, Productivity Second', 'Kathy has lots to be thankful for, Jay is unfortunately unwell, but Rosemary was on standby! Time to review the end of the year and how you finish things or let them go, before getting started on the next new adventure.', '2021-12-30', 'https://www.relay.fm/conduit/13'), ('19: Eating the Devil''s Spaghetti: Combating Imposter Syndrome', 'How do we learn to shut up and take the compliment? How about with a fresh bowl of imp-pasta!', '2022-03-24', 'https://www.relay.fm/conduit/19'), ('48: Long Projects: Remove the Concept of Time', 'We''ve got a longer than usual period between our next live recording so we''re taking the time to think about some longer connections. Tune in to hear how we''re going about it and longer projects in general.', '2023-05-04', 'https://www.relay.fm/conduit/48'), ('57: I Need Help to Get the Help', 'Kathy and Jay need help to meet the demands of those around them. They need help getting help!', '2023-09-07', 'https://www.relay.fm/conduit/57'), ('61: The Conduit Burnout Candle', 'Kathy and Jay are feeling the burn(out) well maybe the steps before the burnout. We''ve taken blowtorches to our candles and now we''re telling you the warning signs we see that this next season might be a little tough.', '2023-11-02', 'https://www.relay.fm/conduit/61'), ('81: Brett''s Mental Health (and Tech) Corner', 'Kathy is still on a secret mission so Jay is joined by Brett Terpstra the Internet''s mad scientist to talk mental health''s link to productivity.', '2024-08-08', 'https://www.relay.fm/conduit/81'), ('100: It''s Episode 100!!', 'Grab your tissues, it''s our most guest filled episode ever. We also discuss what Conduit is, what it means to us, and how it has affected our lives.', '2025-05-01', 'https://www.relay.fm/conduit/100'), ('107: Bored as a Benefit', 'Jay and Kathy explore the idea that boredom isn''t the enemy of productivity - it might actually be the secret ingredient.', '2025-08-07', 'https://www.relay.fm/conduit/107'), ('115: Productivity Inside Systems You Don''t Control', 'Kathy is joined by the Nameless of the Show, Nameless, to talk about how to be productive when the system is one you don''t control.', '2025-11-21', 'https://www.relay.fm/conduit/115'), ('117: The Year to be Selfish', 'Kathy and Jay discuss end-of-year planning. Kathy''s 2026 theme: "The Year to Be Selfish." Jay commits to boundaries and self-preservation. They cover nonprofit transitions and preparing for the annual systems check.', '2025-12-18', 'https://www.relay.fm/conduit/117'); ``` ## Step 3: Generate embeddings [Section titled “Step 3: Generate embeddings”](#step-3-generate-embeddings) Here's where things get interesting. An **embedding** is a list of numbers (a vector) that represents the *meaning* of a piece of text. Texts about similar topics end up with similar vectors. Think of it as teaching your database to understand vibes, not just vocabulary. The `embedding` column is currently empty. You use a Python script to generate embeddings for each episode using OpenAI's `text-embedding-3-small` model and write them back to the database. **Already cloned the cookbook repository?** Skip ahead to run the script If you cloned the [cookbook-search repository](https://github.com/timescale/cookbook-search) earlier, you already have all the files you need. Run the following commands in your terminal to set up and run the embedding script: ```bash cd cookbook-search/Hybrid-search uv venv source .venv/bin/activate # On Windows: .venv\Scripts\activate uv pip install -r requirements.txt cd .. cp .env.example .env cd Hybrid-search ``` Open the `.env` file, add your `OPENAI_API_KEY` and any database connection variables, then skip to [Run the embedding script](#run-the-embedding-script) below. Follow the steps below to create the embedding script and its dependencies from scratch. 1. **Install uv (if you don't have it)** Run the appropriate command in your terminal to install `uv`, a fast Python package manager: ```bash # macOS / Linux curl -LsSf https://astral.sh/uv/install.sh | sh # Windows powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex" ``` Note Using a different package manager? Substitute the `uv` commands below: * **pip:** `pip install -r requirements.txt` * **conda:** `conda install openai psycopg2-binary python-dotenv` 2. **Create your project directory** Run the following in your terminal to create a working directory and navigate into it: ```bash mkdir hybrid-search-tutorial cd hybrid-search-tutorial ``` 3. **Create the requirements file** Create a file named `requirements.txt` in your project directory with the Python dependencies. These three packages handle the OpenAI API calls, database connections, and environment variable loading: ```txt openai>=1.0.0 psycopg2-binary>=2.9.0 python-dotenv>=1.0.0 ``` 4. **Install dependencies** Run the following commands in your terminal to create a virtual environment and install the packages listed in `requirements.txt`: ```bash uv venv source .venv/bin/activate # On Windows: .venv\Scripts\activate uv pip install -r requirements.txt ``` 5. **Create the embedding script** Create a file named `embed.py` in the same directory with the following Python code. This script connects to your database, fetches episodes that don't have embeddings yet, generates embeddings using OpenAI's API, and writes the vectors back to the `embedding` column: ```python import os from pathlib import Path from dotenv import load_dotenv import psycopg2 from openai import OpenAI load_dotenv(Path(__file__).resolve().parent / ".env") EMBEDDING_MODEL = "text-embedding-3-small" # 1536 dimensions DB_CONFIG = { "host": os.getenv("PGHOST", "localhost"), "port": os.getenv("PGPORT", "5432"), "user": os.getenv("PGUSER", "postgres"), "password": os.getenv("PGPASSWORD", "password"), "dbname": os.getenv("PGDATABASE", "postgres"), } def get_episodes_without_embeddings(conn): """Fetch episodes that don't have embeddings yet.""" with conn.cursor() as cur: cur.execute( "SELECT id, title, description FROM episodes WHERE embedding IS NULL" ) return cur.fetchall() def generate_embeddings(texts): """Call OpenAI to generate embeddings for a list of texts.""" client = OpenAI() response = client.embeddings.create(model=EMBEDDING_MODEL, input=texts) return [item.embedding for item in response.data] def update_embeddings(conn, episode_ids, embeddings): """Write embeddings back to the database.""" with conn.cursor() as cur: for episode_id, embedding in zip(episode_ids, embeddings): cur.execute( "UPDATE episodes SET embedding = %s WHERE id = %s", (str(embedding), episode_id), ) conn.commit() def main(): if not os.getenv("OPENAI_API_KEY"): print("Error: OPENAI_API_KEY is not set.") print("Create a .env file and add your key:") print(" OPENAI_API_KEY=your-key-here") raise SystemExit(1) conn = psycopg2.connect(**DB_CONFIG) try: episodes = get_episodes_without_embeddings(conn) if not episodes: print("All episodes already have embeddings. Nothing to do.") return print(f"Found {len(episodes)} episodes without embeddings.") ids = [row[0] for row in episodes] texts = [f"{row[1]}: {row[2]}" for row in episodes] print(f"Generating embeddings with {EMBEDDING_MODEL}...") embeddings = generate_embeddings(texts) print("Updating database...") update_embeddings(conn, ids, embeddings) print(f"Done! Embedded {len(embeddings)} episodes.") finally: conn.close() if __name__ == "__main__": main() ``` 6. **Configure your environment variables** Create a `.env` file in the same directory as `embed.py`. This file stores your secrets and database connection details. The embedding script reads these at runtime. Warning The `.env` file contains secrets like API keys and database passwords. Add `.env` to your `.gitignore` file so it is never committed to version control. At minimum, add your OpenAI API key. The embedding script reads this variable to authenticate with the OpenAI API: ```ini OPENAI_API_KEY=your-key-here ``` If you are connecting to a Tiger Cloud service or a non-default PostgreSQL instance, also add the following database connection variables to the same `.env` file. The embedding script uses these to connect to your database and write the generated vectors: ```ini PGHOST=your-host PGPORT=5432 PGUSER=tsdbadmin PGPASSWORD=your-password PGDATABASE=tsdb ``` ### Run the embedding script [Section titled “Run the embedding script”](#run-the-embedding-script) From your project directory in your terminal, run the Python script. It reads each episode's title and description from the database, sends them to OpenAI's `text-embedding-3-small` model, and writes the resulting 1536-dimensional vectors back to the `embedding` column: ```bash python embed.py ``` You should see output like: ```plaintext Found 12 episodes without embeddings. Generating embeddings with text-embedding-3-small... Updating database... Done! Embedded 12 episodes. ``` The script is idempotent: it only embeds episodes where `embedding IS NULL`, so you can safely re-run it if you add more data later. ### Verify the embeddings [Section titled “Verify the embeddings”](#verify-the-embeddings) Switch back to your SQL client and run the following query to confirm the embeddings were written. It selects the first 3 episodes and truncates the embedding vector to 40 characters for readability: ```sql SELECT id, title, left(embedding::text, 40) AS embedding_preview FROM episodes LIMIT 3; ``` You should see a truncated vector (a string of numbers starting with `[`) for each row instead of `NULL`. ## Step 4: Create indexes [Section titled “Step 4: Create indexes”](#step-4-create-indexes) You've got data and embeddings, but without indexes, every query scans every row. That's fine for 12 episodes, but it won't scale. Let's teach PostgreSQL to search smarter, not harder. 1. **Create a BM25 index for keyword search** Run the following SQL in your SQL client to create a BM25 index on the `description` column. This powers the keyword search queries in later steps: ```sql CREATE INDEX episodes_bm25_idx ON episodes USING bm25(description) WITH (text_config = 'english'); ``` The `text_config = 'english'` setting enables English stemming (so "productivity" also matches "productive") and removes common stopwords like "the" and "is." For more on BM25 index configuration, see [Understand pg\_textsearch and BM25 search](/docs/learn/search/using-pg-textsearch). 2. **Create a StreamingDiskANN index for vector search** Run the following SQL in the same session to create a vector similarity index on the `embedding` column. This powers the semantic search queries in later steps: ```sql CREATE INDEX episodes_embedding_idx ON episodes USING diskann (embedding vector_cosine_ops); ``` This creates a `pgvectorscale` StreamingDiskANN index using cosine distance. Unlike pgvector's built-in HNSW index, DiskANN stores the graph on disk rather than requiring the entire index to fit in RAM, which is a big advantage for large embedding sets. For more on StreamingDiskANN index types and tuning, see the [`pgvectorscale` repository](https://github.com/timescale/pgvectorscale). ## Step 5: Try keyword search (BM25) [Section titled “Step 5: Try keyword search (BM25)”](#step-5-try-keyword-search-bm25) Time to take your new indexes for a spin. Start with keyword search to see how BM25 works on its own. `pg_textsearch` uses the `<@>` operator to score how well a row matches your search terms. Scores are negative so that PostgreSQL's default ascending `ORDER BY` puts the most relevant results first (a score of -15.3 is more relevant than -8.2). **Basic keyword search:** run the following query in your SQL client to find episodes that match "burnout productivity." The `<@>` operator scores each row against the search terms, and `ORDER BY` sorts the results by relevance: ```sql SELECT title, description <@> 'burnout productivity' AS score FROM episodes ORDER BY description <@> 'burnout productivity' LIMIT 10; ``` You should see episodes like "The Conduit Burnout Candle" and "Happiness First, Productivity Second" near the top because they contain the words you searched for. **Keyword search with a date filter:** run this query in your SQL client to search only episodes from 2023 onward. PostgreSQL detects the `<@>` operator and uses the BM25 index automatically, so you can freely combine it with `WHERE` clauses: ```sql SELECT title, description <@> 'help' AS score FROM episodes WHERE pub_date >= '2023-01-01' ORDER BY description <@> 'help' LIMIT 10; ``` **What BM25 is good at:** finding exact keyword matches. If someone searches for "imposter syndrome," BM25 finds it. **Where it falls short:** if someone searches for "feeling like a fraud at work," BM25 won't match the imposter syndrome episode because none of those exact words appear in its description. That's where vector search comes in. ## Step 6: Try vector search (semantic similarity) [Section titled “Step 6: Try vector search (semantic similarity)”](#step-6-try-vector-search-semantic-similarity) Now let's see the other side of the coin. Vector search uses the embeddings you generated earlier to find episodes by *meaning* rather than keywords. Two pieces of text about the same topic have similar embeddings, even if they use completely different words. The `<=>` operator computes cosine distance between two vectors. Lower values mean more similar (0 = identical, 1 = completely unrelated). **Semantic search:** run the following query in your SQL client to find episodes closest in meaning to a query vector. Replace `$1` with an embedding generated from your search text (see the note below). The `<=>` operator computes cosine distance, and `ORDER BY` sorts by closest match: ```sql SELECT title, embedding <=> $1 AS distance FROM episodes ORDER BY embedding <=> $1 LIMIT 10; ``` Note To get a query vector, generate an embedding for your search text the same way you embedded the episodes, by calling the OpenAI embeddings API. The [`embed.py`](https://github.com/timescale/cookbook-search/blob/main/Hybrid-search/embed.py) script shows how. In a real application, your app generates the query embedding at search time and passes it as a parameter. **What vector search is good at:** finding semantically related content. A search for "feeling like a fraud at work" surfaces the imposter syndrome episode, even though those exact words don't appear anywhere in its description. **Where it falls short:** it can miss results that match on specific terms. If someone searches for "episode 100," BM25 finds it instantly, but vector search might rank it lower because "episode 100" doesn't carry strong semantic meaning. Each method has blind spots, which is exactly why you combine them in the next step. ## Step 7: Combine results with hybrid search (RRF) [Section titled “Step 7: Combine results with hybrid search (RRF)”](#step-7-combine-results-with-hybrid-search-rrf) This is the grand finale. Instead of trying to compare raw scores across different systems (which use different scales), **Reciprocal Rank Fusion** only looks at **rank position**. An episode ranked #1 by either method gets a high score. An episode ranked #1 by *both* methods gets an even higher score. The formula for each result is `1 / (k + rank)`, where `k` is a smoothing constant (typically 60). You sum this across all the search methods and sort by the total. **Hybrid search query:** run the following query in your SQL client. It executes both BM25 keyword search and vector similarity search as Common Table Expressions (CTEs), then joins and scores the results using the RRF formula. Replace `$1` with your query embedding vector: ```sql -- Get the top 20 BM25 keyword matches WITH bm25_results AS ( SELECT id, ROW_NUMBER() OVER ( ORDER BY description <@> 'mental health boundaries' ) AS rank FROM episodes ORDER BY description <@> 'mental health boundaries' LIMIT 20 ), -- Get the top 20 vector similarity matches vector_results AS ( SELECT id, ROW_NUMBER() OVER ( ORDER BY embedding <=> $1 -- $1 is the query embedding vector ) AS rank FROM episodes ORDER BY embedding <=> $1 LIMIT 20 ) -- Fuse the two ranked lists using RRF SELECT d.id, d.title, COALESCE(1.0 / (60 + b.rank), 0) + COALESCE(1.0 / (60 + v.rank), 0) AS rrf_score FROM episodes d LEFT JOIN bm25_results b ON d.id = b.id LEFT JOIN vector_results v ON d.id = v.id WHERE b.id IS NOT NULL OR v.id IS NOT NULL ORDER BY rrf_score DESC LIMIT 10; ``` **How it works:** 1. **`bm25_results`** runs a keyword search for "mental health boundaries" and assigns each result a rank (1 = best match) 2. **`vector_results`** runs a vector search using the query embedding and assigns ranks the same way 3. **The final `SELECT`** joins both result sets by episode ID and computes an RRF score for each. The `COALESCE(..., 0)` ensures that episodes found by only one method still get a score. The `60` is the standard smoothing constant that prevents top-ranked results from dominating too heavily. **An example to make it concrete:** imagine a user searches for "how do I deal with feeling like a fraud at work?" * **BM25 finds:** "Eating the Devil's Spaghetti: Combating Imposter Syndrome" because it matches on "imposter" after stemming * **Vector search finds:** episodes about burnout, boundaries, and mental health - semantically related even though the words are different * **RRF fuses both:** the imposter syndrome episode ranks highest (found by both methods), while related episodes about self-doubt and mental health also surface higher than they would with either method alone That's hybrid search: the precision of keywords and the recall of semantic similarity in a single ranked list. Not bad for a SQL query, right? ## Going further [Section titled “Going further”](#going-further) You've got a working hybrid search system, congrats! If you're the kind of person who reads the bonus chapters, here are tips and techniques for taking it into production. ### Search across multiple columns [Section titled “Search across multiple columns”](#search-across-multiple-columns) Each BM25 index covers a single text column. To search across both title and description, run the following two SQL statements in your SQL client. The first adds a generated column that automatically concatenates the title and description. The second creates a BM25 index on that combined column: ```sql ALTER TABLE episodes ADD COLUMN search_text text GENERATED ALWAYS AS (title || ' ' || description) STORED; CREATE INDEX ON episodes USING bm25(search_text) WITH (text_config = 'english'); ``` Now queries against `search_text` match words in either the title or description. ### Highlight matched terms in results [Section titled “Highlight matched terms in results”](#highlight-matched-terms-in-results) Use PostgreSQL's built-in `ts_headline()` to show which words matched, with surrounding context. This is useful for building search result snippets in a UI. Run this query in your SQL client to see highlighted results for the term "productivity": ```sql SELECT title, ts_headline('english', description, to_tsquery('english', 'productivity')), description <@> 'productivity' AS score FROM episodes ORDER BY description <@> 'productivity' LIMIT 10; ``` The `ts_headline()` function returns the description with matching terms wrapped in `` tags (configurable), while the `<@>` operator handles the BM25 ranking. ### Phrase search workaround [Section titled “Phrase search workaround”](#phrase-search-workaround) `pg_textsearch` doesn't support native phrase queries (matching exact multi-word sequences). Work around this by over-fetching from the BM25 index and post-filtering with `ILIKE`. Run this query in your SQL client. The inner query uses BM25 to find the top 100 candidates, and the outer query filters down to rows containing the exact phrase: ```sql SELECT * FROM ( SELECT * FROM episodes ORDER BY description <@> 'year end planning' LIMIT 100 -- over-fetch to compensate for the post-filter ) sub WHERE description ILIKE '%end-of-year%' LIMIT 10; ``` ### Tune vector search accuracy [Section titled “Tune vector search accuracy”](#tune-vector-search-accuracy) `pgvectorscale`'s StreamingDiskANN index uses smart defaults. If you need higher accuracy at the cost of slightly slower queries, run the following in your SQL client to increase the rescore parameter for the current transaction, then execute your vector search query: ```sql SET LOCAL diskann.query_rescore = 150; SELECT * FROM episodes ORDER BY embedding <=> $1 LIMIT 10; ``` Higher `query_rescore` values mean more candidates are re-scored for accuracy. The `SET LOCAL` scope means this setting only applies to the current transaction. ### Speed up index builds for large tables [Section titled “Speed up index builds for large tables”](#speed-up-index-builds-for-large-tables) For tables with millions of rows, run the following session-level settings in your SQL client before creating indexes. These tell PostgreSQL to use 4 parallel workers and allocate 256 MB of memory for the index build, significantly reducing build time: ```sql SET max_parallel_maintenance_workers = 4; SET maintenance_work_mem = '256MB'; CREATE INDEX ON large_table USING bm25(content) WITH (text_config = 'english'); CREATE INDEX ON large_table USING diskann (embedding vector_cosine_ops); ``` Replace `large_table` and `content`/`embedding` with your actual table and column names. ### Compact the BM25 index after bulk inserts [Section titled “Compact the BM25 index after bulk inserts”](#compact-the-bm25-index-after-bulk-inserts) If you bulk-insert a lot of data, the BM25 index may have multiple segments from repeated writes. Run the following in your SQL client to merge them into a single segment for faster query performance: ```sql SELECT bm25_force_merge('episodes_bm25_idx'); ``` Replace `'episodes_bm25_idx'` with your actual index name. This is a one-time operation. Run it after large bulk loads, not after every insert. ## Current limitations [Section titled “Current limitations”](#current-limitations) No tool is perfect (yet). Here's what to be aware of in `pg_textsearch`: * **No phrase queries**: the index stores term frequencies but not positions. Use the over-fetch + post-filter pattern shown above * **OR-only query semantics**: all query terms are implicitly OR'd. AND/OR/NOT operators are planned for a post-1.0 release * **No highlighting from the index**: use PostgreSQL's built-in `ts_headline()` on the result set * **One expression per index**: to search several fields at once, index an expression that concatenates them (for example `(title || ' ' || description)`) or use a generated column * **PL/pgSQL requires explicit index names**: use `to_bm25query('query', 'index_name')` inside PL/pgSQL, DO blocks, or stored procedures ## Next steps [Section titled “Next steps”](#next-steps) [Understand pg\_textsearch and BM25 search](/docs/learn/search/using-pg-textsearch) [Learn why BM25 full-text search matters, how pg\_textsearch ranks results, and when to use it](/docs/learn/search/using-pg-textsearch) [Understand pgvector and pgvectorscale](/docs/learn/search/pgvector-pgvectorsearch) [Learn how vector embeddings power semantic search with pgvector and pgvectorscale on Tiger Cloud](/docs/learn/search/pgvector-pgvectorsearch) [pgvectorscale on GitHub](https://github.com/timescale/pgvectorscale) [StreamingDiskANN index types, filtered search, and build or query parameters](https://github.com/timescale/pgvectorscale) [pg\_textsearch deploy guide](/docs/deploy/tiger-cloud/tiger-cloud-aws/tiger-cloud-extensions/pg-textsearch) [Install, configure, and optimize pg\_textsearch on Tiger Cloud](/docs/deploy/tiger-cloud/tiger-cloud-aws/tiger-cloud-extensions/pg-textsearch) # Ingest real-time financial data > Set up a data pipeline to get data from different financial APIs The financial industry is extremely data-heavy and relies on real-time and historical data for decision-making, risk assessment, fraud detection, and market analysis. Tiger Data simplifies management of these large volumes of data, while also providing you with meaningful analytical insights and optimizing storage costs. This tutorial shows you how to ingest real-time time-series data into TimescaleDB using a websocket connection. The tutorial sets up a data pipeline to ingest real-time data from our data partner, [Twelve Data](https://twelvedata.com). Twelve Data provides a number of different financial APIs, including stock, cryptocurrencies, foreign exchanges, and ETFs. It also supports websocket connections in case you want to update your database frequently. With websockets, you need to connect to the server, subscribe to symbols, and you can start receiving data in real-time during market hours. When you complete this tutorial, you'll have a data pipeline set up that ingests real-time financial data into your Tiger Cloud. This tutorial uses Python and the API [wrapper library](https://github.com/twelvedata/twelvedata-python) provided by Twelve Data. This tutorial covers: 1. **Set up your dataset**: connect to the Twelve Data websocket server, create hypertables, and ingest real-time cryptocurrency data. 2. **Query your data**: create continuous aggregates to aggregate OHLCV data, query the aggregated data, and visualize the data in Grafana. ## Prerequisites for this tutorial To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service). These steps use Tiger Cloud, but the same approach applies to a [self-hosted TimescaleDB](/docs/get-started/choose-your-path/install-timescaledb) instance. - A [self-managed Grafana](https://grafana.com/get/?tab=self-managed) instance, or a [Grafana Cloud](https://grafana.com/get/) account. - Python 3 installed. - A [Twelve Data](https://twelvedata.com/pricing) account. The free tier is perfect for this tutorial. - Your Twelve Data [API key](https://twelvedata.com/account/api-keys). ## About OHLCV data and candlestick charts [Section titled “About OHLCV data and candlestick charts”](#about-ohlcv-data-and-candlestick-charts) The financial sector regularly uses [candlestick charts](https://www.investopedia.com/terms/c/candlestick.asp) to visualize the price change of an asset. Each candlestick represents a time period, such as one minute or one hour, and shows how the asset's price changed during that time. Candlestick charts are generated from the open, high, low, close, and volume data for each financial asset during the time period. This is often abbreviated as OHLCV: * Open: opening price * High: highest price * Low: lowest price * Close: closing price * Volume: volume of transactions ![Candlestick chart showing intraday stock price movements](/docs/_astro/candlestick_fig.CkWcWeSI_ZSubFP.webp) TimescaleDB is well suited to storing and analyzing financial candlestick data, and many Tiger Data community members use it for exactly this purpose. ## Ingest data into a Tiger Cloud service [Section titled “Ingest data into a Tiger Cloud service”](#ingest-data-into-a-tiger-cloud-service) This tutorial uses a dataset that contains second-by-second cryptocurrency trade data, in a hypertable named `crypto_ticks`. It also includes a separate table of cryptocurrency symbols and names, in a regular PostgreSQL table named `crypto_assets`. ### Connect to the websocket server [Section titled “Connect to the websocket server”](#connect-to-the-websocket-server) When you connect to the Twelve Data API through a websocket, you create a persistent connection between your computer and the websocket server. You set up a Python environment, and pass two arguments to create a websocket object and establish the connection. #### Set up a new Python environment [Section titled “Set up a new Python environment”](#set-up-a-new-python-environment) Create a new Python virtual environment for this project and activate it. All the packages you need to complete for this tutorial are installed in this environment. 1. **Create and activate a Python virtual environment** ```bash virtualenv env source env/bin/activate ``` 2. **Install the Twelve Data Python wrapper library with websocket support** This [library](https://github.com/twelvedata/twelvedata-python) allows you to make requests to the API and maintain a stable websocket connection. ```bash pip install twelvedata websocket-client ``` 3. **Install Psycopg2** [Psycopg2](https://www.psycopg.org/docs/) enables you to connect to TimescaleDB from your Python script: ```bash pip install psycopg2-binary ``` #### Create the websocket connection [Section titled “Create the websocket connection”](#create-the-websocket-connection) A persistent connection between your computer and the websocket server is used to receive data for as long as the connection is maintained. You need to pass two arguments to create a websocket object and establish connection. **Websocket arguments** * `on_event` This argument needs to be a function that is invoked whenever there's a new data record is received from the websocket: ```python def on_event(event): print(event) # prints out the data record (dictionary) ``` This is where you want to implement the ingestion logic so whenever there's new data available you insert it into the database. * `symbols` This argument needs to be a list of stock ticker symbols (for example, `MSFT`) or crypto trading pairs (for example, `BTC/USD`). When using a websocket connection you always need to subscribe to the events you want to receive. You can do this by using the `symbols` argument or if your connection is already created you can also use the `subscribe()` function to get data for additional symbols. **Connect to the websocket server** 1. **Create a new Python file called `websocket_test.py`** Connect to the Twelve Data servers using the ``: ```python import time from twelvedata import TDClient messages_history = [] def on_event(event): print(event) # prints out the data record (dictionary) messages_history.append(event) td = TDClient(apikey="") ws = td.websocket(symbols=["BTC/USD", "ETH/USD"], on_event=on_event) ws.subscribe(['ETH/BTC', 'AAPL']) ws.connect() while True: print('messages received: ', len(messages_history)) ws.heartbeat() time.sleep(10) ``` 2. **Run the Python script** ```bash python websocket_test.py ``` 3. **Verify the connection** When you run the script, you receive a response from the server about the status of your connection: ```bash {'event': 'subscribe-status', 'status': 'ok', 'success': [ {'symbol': 'BTC/USD', 'exchange': 'Coinbase Pro', 'mic_code': 'Coinbase Pro', 'country': '', 'type': 'Digital Currency'}, {'symbol': 'ETH/USD', 'exchange': 'Huobi', 'mic_code': 'Huobi', 'country': '', 'type': 'Digital Currency'} ], 'fails': None } ``` When you have established a connection to the websocket server, wait a few seconds, and you can see data records, like this: ```bash {'event': 'price', 'symbol': 'BTC/USD', 'currency_base': 'Bitcoin', 'currency_quote': 'US Dollar', 'exchange': 'Coinbase Pro', 'type': 'Digital Currency', 'timestamp': 1652438893, 'price': 30361.2, 'bid': 30361.2, 'ask': 30361.2, 'day_volume': 49153} {'event': 'price', 'symbol': 'BTC/USD', 'currency_base': 'Bitcoin', 'currency_quote': 'US Dollar', 'exchange': 'Coinbase Pro', 'type': 'Digital Currency', 'timestamp': 1652438896, 'price': 30380.6, 'bid': 30380.6, 'ask': 30380.6, 'day_volume': 49157} {'event': 'heartbeat', 'status': 'ok'} {'event': 'price', 'symbol': 'ETH/USD', 'currency_base': 'Ethereum', 'currency_quote': 'US Dollar', 'exchange': 'Huobi', 'type': 'Digital Currency', 'timestamp': 1652438899, 'price': 2089.07, 'bid': 2089.02, 'ask': 2089.03, 'day_volume': 193818} {'event': 'price', 'symbol': 'BTC/USD', 'currency_base': 'Bitcoin', 'currency_quote': 'US Dollar', 'exchange': 'Coinbase Pro', 'type': 'Digital Currency', 'timestamp': 1652438900, 'price': 30346.0, 'bid': 30346.0, 'ask': 30346.0, 'day_volume': 49167} ``` Each price event gives you multiple data points about the given trading pair such as the name of the exchange, and the current price. You can also occasionally see `heartbeat` events in the response; these events signal the health of the connection over time. At this point the websocket connection is working successfully to pass data. ## Optimize time-series data in a hypertable Hypertables are PostgreSQL tables in TimescaleDB that automatically partition your time-series data by time. Time-series data represents the way a system, process, or behavior changes over time. Hypertables enable TimescaleDB to work efficiently with time-series data. Each hypertable is made up of child tables called chunks. Each chunk is assigned a range of time, and only contains data from that range. When you run a query, TimescaleDB identifies the correct chunk and runs the query on it, instead of going through the entire table. [Hypercore](/docs/learn/columnar-storage/understand-hypercore) is the hybrid row-columnar storage engine in TimescaleDB used by hypertables. Traditional databases force a trade-off between fast inserts (row-based storage) and efficient analytics (columnar storage). Hypercore eliminates this trade-off, allowing real-time analytics without sacrificing transactional capabilities. Hypercore dynamically stores data in the most efficient format for its lifecycle: ![Move from rowstore to columstore in hypercore](/docs/_astro/hypercore_intro.DmLiqZGV_1VM5z7.webp) ![Move from rowstore to columstore in hypercore](/docs/_astro/hypercore_intro-dark.qPmjkBqW_19V2rF.webp) * **Row-based storage for recent data**: the most recent chunk (and possibly more) is always stored in the rowstore, ensuring fast inserts, updates, and low-latency single record queries. Additionally, row-based storage is used as a writethrough for inserts and updates to columnar storage. * **Columnar storage for analytical performance**: chunks are automatically compressed into the columnstore, optimizing storage efficiency and accelerating analytical queries. Unlike traditional columnar databases, hypercore allows data to be inserted or modified at any stage, making it a flexible solution for both high-ingest transactional workloads and real-time analytics, within a single database. Because TimescaleDB is 100% PostgreSQL, you can use all the standard PostgreSQL tables, indexes, stored procedures, and other objects alongside your hypertables. This makes creating and working with hypertables similar to standard PostgreSQL. 1. **Connect to your Tiger Cloud service** In [Tiger Console](https://console.cloud.tigerdata.com/dashboard/services) open an [SQL editor](/docs/build/data-management/run-queries-from-tiger-console). You can also connect to your service using [psql](/docs/integrate/query-administration/psql). 2. **Create a hypertable to store the real-time cryptocurrency data** Create a [hypertable](/docs/learn/hypertables/understand-hypertables) for your time-series data using [CREATE TABLE](/docs/reference/timescaledb). For [efficient queries](/docs/build/performance-optimization/secondary-indexes) on data in the columnstore, remember to `segmentby` the column you will use most often to filter your data: ```sql CREATE TABLE crypto_ticks ( "time" TIMESTAMPTZ, symbol TEXT, price DOUBLE PRECISION, day_volume NUMERIC ) WITH ( tsdb.hypertable, tsdb.segmentby='symbol', tsdb.orderby='time DESC' ); ``` When you create a hypertable using [CREATE TABLE ... WITH ...](/docs/reference/timescaledb/hypertables/create_table), the default partitioning column is automatically the first column with a timestamp data type. Also, TimescaleDB creates a [columnstore policy](/docs/reference/timescaledb/hypercore/add_columnstore_policy) that automatically converts your data to the columnstore, after an interval equal to the value of the [chunk\_interval](/docs/reference/timescaledb/hypertables/set_chunk_time_interval), defined through `after` in the policy. This columnar format enables fast scanning and aggregation, optimizing performance for analytical workloads while also saving significant storage space. In the columnstore conversion, hypertable chunks are compressed by up to 98%, and organized for efficient, large-scale queries. You can customize this policy later using [alter\_job](/docs/reference/timescaledb/jobs-automation/alter_job). However, to change `after` or `created_before`, the compression settings, or the hypertable the policy is acting on, you must [remove the columnstore policy](/docs/reference/timescaledb/hypercore/remove_columnstore_policy) and [add a new one](/docs/reference/timescaledb/hypercore/add_columnstore_policy). You can also manually [convert chunks](/docs/reference/timescaledb/hypercore/convert_to_columnstore) in a hypertable to the columnstore. ## Create a standard PostgreSQL table for relational data When you have relational data that enhances your time-series data, store that data in standard PostgreSQL relational tables. 1. **Add a table to store the asset symbol and name in a relational table** ```sql CREATE TABLE crypto_assets ( symbol TEXT UNIQUE, "name" TEXT ); ``` You now have two tables within your Tiger Cloud service. A hypertable named `crypto_ticks`, and a normal PostgreSQL table named `crypto_assets`. When you ingest data into a transactional database like TimescaleDB, it is more efficient to insert data in batches rather than inserting data row-by-row. Using one transaction to insert multiple rows can significantly increase the overall ingest capacity and speed of your Tiger Cloud service. ## Batching in memory [Section titled “Batching in memory”](#batching-in-memory) A common practice to implement batching is to store new records in memory first, then after the batch reaches a certain size, insert all the records from memory into the database in one transaction. The perfect batch size isn't universal, but you can experiment with different batch sizes (for example, 100, 1000, 10000, and so on) and see which one fits your use case better. Using batching is a fairly common pattern when ingesting data into TimescaleDB from Kafka, Kinesis, or websocket connections. To ingest the data into your Tiger Cloud service, you need to implement the `on_event` function. After the websocket connection is set up, you can use the `on_event` function to ingest data into the database. This is a data pipeline that ingests real-time financial data into your Tiger Cloud service. You can implement a batching solution in Python with Psycopg2. You can implement the ingestion logic within the `on_event` function that you can then pass over to the websocket object. This function needs to: 1. Check if the item is a data item, and not websocket metadata. 2. Adjust the data so that it fits the database schema, including the data types, and order of columns. 3. Add it to the in-memory batch, which is a list in Python. 4. If the batch reaches a certain size, insert the data, and reset or empty the list. ## Ingest data in real-time [Section titled “Ingest data in real-time”](#ingest-data-in-real-time) 1. **Update the Python script** Update the script to print out the current batch size, so you can follow when data gets ingested from memory into your database. Use the ``, ``, and `` details for the Tiger Cloud service where you want to ingest the data and your API key from Twelve Data: ```python import time import psycopg2 from twelvedata import TDClient from psycopg2.extras import execute_values from datetime import datetime class WebsocketPipeline(): # name of the hypertable DB_TABLE = "crypto_ticks" # columns in the hypertable in the correct order DB_COLUMNS=["time", "symbol", "price", "day_volume"] # batch size used to insert data in batches MAX_BATCH_SIZE=100 def __init__(self, conn): """Connect to the Twelve Data web socket server and stream data into the database. Args: conn: psycopg2 connection object """ self.conn = conn self.current_batch = [] self.insert_counter = 0 def _insert_values(self, data): if self.conn is not None: cursor = self.conn.cursor() sql = f""" INSERT INTO {self.DB_TABLE} ({','.join(self.DB_COLUMNS)}) VALUES %s;""" execute_values(cursor, sql, data) self.conn.commit() def _on_event(self, event): """This function gets called whenever there's a new data record coming back from the server. Args: event (dict): data record """ if event["event"] == "price": # data record timestamp = datetime.utcfromtimestamp(event["timestamp"]) data = (timestamp, event["symbol"], event["price"], event.get("day_volume")) # add new data record to batch self.current_batch.append(data) print(f"Current batch size: {len(self.current_batch)}") # ingest data if max batch size is reached then reset the batch if len(self.current_batch) == self.MAX_BATCH_SIZE: self._insert_values(self.current_batch) self.insert_counter += 1 print(f"Batch insert #{self.insert_counter}") self.current_batch = [] def start(self, symbols): """Connect to the web socket server and start streaming real-time data into the database. Args: symbols (list of symbols): List of stock/crypto symbols """ td = TDClient(apikey="") ws = td.websocket(on_event=self._on_event) ws.subscribe(symbols) ws.connect() while True: ws.heartbeat() time.sleep(10) conn = psycopg2.connect(database="tsdb", host="", user="tsdbadmin", password="", port="") symbols = ["BTC/USD", "ETH/USD", "MSFT", "AAPL"] websocket = WebsocketPipeline(conn) websocket.start(symbols=symbols) ``` 2. **Run the script** ```bash python websocket_test.py ``` You can even create separate Python scripts to start multiple websocket connections for different types of symbols, for example, one for stock, and another one for cryptocurrency prices. ### Troubleshooting [Section titled “Troubleshooting”](#troubleshooting) If you see an error message similar to this: ```bash 2022-05-13 18:51:41,976 - ws-twelvedata - ERROR - TDWebSocket ERROR: Handshake status 200 OK ``` Then check that you use a proper API key received from Twelve Data. ## Query the data [Section titled “Query the data”](#query-the-data) To look at OHLCV values, the most effective way is to create a continuous aggregate. You can create a continuous aggregate to aggregate data for each day, then set the aggregate to refresh every day, and aggregate the last two days' worth of data. ### Creating a continuous aggregate [Section titled “Creating a continuous aggregate”](#creating-a-continuous-aggregate) 1. **Connect to the Tiger Cloud service** Connect to the service that contains the Twelve Data cryptocurrency dataset. 2. **Create the continuous aggregate to aggregate data every day** ```sql CREATE MATERIALIZED VIEW one_day_candle WITH (timescaledb.continuous) AS SELECT time_bucket('1 day', time) AS bucket, symbol, FIRST(price, time) AS "open", MAX(price) AS high, MIN(price) AS low, LAST(price, time) AS "close", LAST(day_volume, time) AS day_volume FROM crypto_ticks GROUP BY bucket, symbol; ``` When you create the continuous aggregate, it refreshes by default. 3. **Set a refresh policy to update the continuous aggregate every day** This refreshes if there is new data available in the hypertable for the last two days: ```sql SELECT add_continuous_aggregate_policy('one_day_candle', start_offset => INTERVAL '3 days', end_offset => INTERVAL '1 day', schedule_interval => INTERVAL '1 day'); ``` ### Query the continuous aggregate [Section titled “Query the continuous aggregate”](#query-the-continuous-aggregate) When you have your continuous aggregate set up, you can query it to get the OHLCV values. 1. **Connect to the Tiger Cloud service** Connect to the service that contains the Twelve Data cryptocurrency dataset. 2. **Select all Bitcoin OHLCV data for the past 14 days** At the psql prompt, use this query to select by time bucket: ```sql SELECT * FROM one_day_candle WHERE symbol = 'BTC/USD' AND bucket >= NOW() - INTERVAL '14 days' ORDER BY bucket; ``` The result of the query looks like this: ```sql bucket | symbol | open | high | low | close | day_volume ------------------------+---------+---------+---------+---------+---------+------------ 2022-11-24 00:00:00+00 | BTC/USD | 16587 | 16781.2 | 16463.4 | 16597.4 | 21803 2022-11-25 00:00:00+00 | BTC/USD | 16597.4 | 16610.1 | 16344.4 | 16503.1 | 20788 2022-11-26 00:00:00+00 | BTC/USD | 16507.9 | 16685.5 | 16384.5 | 16450.6 | 12300 ``` ## Connect Grafana to Tiger Cloud To visualize the results of your queries, enable Grafana to read the data in your service: 1. **Log in to Grafana** In your browser, log in to either: * Self-hosted Grafana: at `http://localhost:3000/`. The default credentials are `admin`, `admin`. * Grafana Cloud: use the URL and credentials you set when you created your account. 2. **Add your service as a data source** 1. Open `Connections` > `Data sources`, then click `Add new data source`. 2. Select `PostgreSQL` from the list. 3. Configure the connection: * `Host URL`, `Database name`, `Username`, and `Password`, configure using your [connection details](/docs/integrate/find-connection-details). `Host URL` is in the format `:`. * `TLS/SSL Mode`: select `require`. * `PostgreSQL options`: enable `TimescaleDB`. * Leave the default setting for all other fields. 4. Click `Save & test`. Grafana checks that your details are set correctly. ## Graph OHLCV data When you have extracted the raw OHLCV data, you can use it to graph the result in a candlestick chart, using Grafana. 1. **Create a new dashboard** In Grafana, from the `Dashboards` page, click `New` and select `New dashboard`. 2. **Add a Candlestick visualization** Click `Add visualization`, then select the data source that connects to your Tiger Cloud service and the `Candlestick` visualization type in the top right. 3. **Paste the OHLCV query** In the `Queries` section, select `Code` and paste the query you used to get the OHLCV values: ```sql SELECT * FROM one_day_candle WHERE symbol = 'BTC/USD' AND bucket >= NOW() - INTERVAL '14 days' ORDER BY bucket; ``` 4. **Apply and save your graph** Adjust elements of the table as required, and click `Apply` to save your graph to the dashboard. ![Candlestick chart in Grafana using 1-day OHLCV tick data](/docs/_astro/Grafana_candlestick_1day.BS8xfMii_Z14b0su.webp) # Simulate an IoT sensor dataset > Simulate an IoT sensor dataset for testing and development with Tiger Cloud The Internet of Things (IoT) describes a trend where computing capabilities are embedded into IoT devices. That is, physical objects, ranging from light bulbs to oil wells. Many IoT devices collect sensor data about their environment and generate time-series datasets with relational metadata. It is often necessary to simulate IoT datasets. For example, when you are testing a new system. This tutorial shows how to simulate a basic dataset in your Tiger Cloud service, and then run simple queries on it. To simulate a more advanced dataset, see [Time-series Benchmarking Suite (TSBS)](https://github.com/timescale/tsbs). ## Prerequisites for this tutorial To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service), or a running instance of [self-hosted TimescaleDB](/docs/get-started/choose-your-path/install-timescaledb). - Your [connection details](/docs/integrate/find-connection-details). ## Simulate a dataset [Section titled “Simulate a dataset”](#simulate-a-dataset) To simulate a dataset, run the following queries: 1. **Create the sensors table** ```sql CREATE TABLE sensors( id SERIAL PRIMARY KEY, type VARCHAR(50), location VARCHAR(50) ); ``` 2. **Create the sensor\_data hypertable** ```sql CREATE TABLE sensor_data ( time TIMESTAMPTZ NOT NULL, sensor_id INTEGER, temperature DOUBLE PRECISION, cpu DOUBLE PRECISION, FOREIGN KEY (sensor_id) REFERENCES sensors (id) ) WITH ( tsdb.hypertable ); ``` When you create a hypertable using [CREATE TABLE ... WITH ...](/docs/reference/timescaledb/hypertables/create_table), the default partitioning column is automatically the first column with a timestamp data type. Also, TimescaleDB creates a [columnstore policy](/docs/reference/timescaledb/hypercore/add_columnstore_policy) that automatically converts your data to the columnstore, after an interval equal to the value of the [chunk\_interval](/docs/reference/timescaledb/hypertables/set_chunk_time_interval), defined through `after` in the policy. This columnar format enables fast scanning and aggregation, optimizing performance for analytical workloads while also saving significant storage space. In the columnstore conversion, hypertable chunks are compressed by up to 98%, and organized for efficient, large-scale queries. You can customize this policy later using [alter\_job](/docs/reference/timescaledb/jobs-automation/alter_job). However, to change `after` or `created_before`, the compression settings, or the hypertable the policy is acting on, you must [remove the columnstore policy](/docs/reference/timescaledb/hypercore/remove_columnstore_policy) and [add a new one](/docs/reference/timescaledb/hypercore/add_columnstore_policy). You can also manually [convert chunks](/docs/reference/timescaledb/hypercore/convert_to_columnstore) in a hypertable to the columnstore. 3. **Populate the sensors table** ```sql INSERT INTO sensors (type, location) VALUES ('a','floor'), ('a', 'ceiling'), ('b','floor'), ('b', 'ceiling'); ``` 4. **Verify that the sensors have been added correctly** ```sql SELECT * FROM sensors; ``` Sample output: ```plaintext id | type | location ----+------+---------- 1 | a | floor 2 | a | ceiling 3 | b | floor 4 | b | ceiling (4 rows) ``` 5. **Generate and insert a dataset for all sensors** ```sql INSERT INTO sensor_data (time, sensor_id, cpu, temperature) SELECT time, sensor_id, random() AS cpu, random()*100 AS temperature FROM generate_series(now() - interval '24 hour', now(), interval '5 minute') AS g1(time), generate_series(1,4,1) AS g2(sensor_id); ``` 6. **Verify the simulated dataset** ```sql SELECT * FROM sensor_data ORDER BY time; ``` Sample output: ```plaintext time | sensor_id | temperature | cpu -------------------------------+-----------+--------------------+--------------------- 2020-03-31 15:56:25.843575+00 | 1 | 6.86688972637057 | 0.682070567272604 2020-03-31 15:56:40.244287+00 | 2 | 26.589260622859 | 0.229583469685167 2030-03-31 15:56:45.653115+00 | 3 | 79.9925176426768 | 0.457779890391976 2020-03-31 15:56:53.560205+00 | 4 | 24.3201029952615 | 0.641885648947209 2020-03-31 16:01:25.843575+00 | 1 | 33.3203678019345 | 0.0159163917414844 2020-03-31 16:01:40.244287+00 | 2 | 31.2673618085682 | 0.701185956597328 2020-03-31 16:01:45.653115+00 | 3 | 85.2960689924657 | 0.693413889966905 2020-03-31 16:01:53.560205+00 | 4 | 79.4769988860935 | 0.360561791341752 ... ``` ## Run basic queries [Section titled “Run basic queries”](#run-basic-queries) After you simulate a dataset, you can run some basic queries on it. For example: * Average temperature and CPU by 30-minute windows: ```sql SELECT time_bucket('30 minutes', time) AS period, AVG(temperature) AS avg_temp, AVG(cpu) AS avg_cpu FROM sensor_data GROUP BY period; ``` Sample output: ```plaintext period | avg_temp | avg_cpu ------------------------+------------------+------------------- 2020-03-31 19:00:00+00 | 49.6615830013373 | 0.477344429974134 2020-03-31 22:00:00+00 | 58.8521540844037 | 0.503637770501276 2020-03-31 16:00:00+00 | 50.4250325243144 | 0.511075591299838 2020-03-31 17:30:00+00 | 49.0742547437549 | 0.527267253802468 2020-04-01 14:30:00+00 | 49.3416377226822 | 0.438027751864865 ... ``` * Average and last temperature, average CPU by 30-minute windows: ```sql SELECT time_bucket('30 minutes', time) AS period, AVG(temperature) AS avg_temp, last(temperature, time) AS last_temp, AVG(cpu) AS avg_cpu FROM sensor_data GROUP BY period; ``` Sample output: ```plaintext period | avg_temp | last_temp | avg_cpu ------------------------+------------------+------------------+------------------- 2020-03-31 19:00:00+00 | 49.6615830013373 | 84.3963081017137 | 0.477344429974134 2020-03-31 22:00:00+00 | 58.8521540844037 | 76.5528806950897 | 0.503637770501276 2020-03-31 16:00:00+00 | 50.4250325243144 | 43.5192013625056 | 0.511075591299838 2020-03-31 17:30:00+00 | 49.0742547437549 | 22.740753274411 | 0.527267253802468 2020-04-01 14:30:00+00 | 49.3416377226822 | 59.1331578791142 | 0.438027751864865 ... ``` * Query the metadata: ```sql SELECT sensors.location, time_bucket('30 minutes', time) AS period, AVG(temperature) AS avg_temp, last(temperature, time) AS last_temp, AVG(cpu) AS avg_cpu FROM sensor_data JOIN sensors on sensor_data.sensor_id = sensors.id GROUP BY period, sensors.location; ``` Sample output: ```plaintext location | period | avg_temp | last_temp | avg_cpu ----------+------------------------+------------------+-------------------+------------------- ceiling | 2020-03-31 15:30:00+00 | 25.4546818090603 | 24.3201029952615 | 0.435734559316188 floor | 2020-03-31 15:30:00+00 | 43.4297036845237 | 79.9925176426768 | 0.56992522883229 ceiling | 2020-03-31 16:00:00+00 | 53.8454438598516 | 43.5192013625056 | 0.490728285357666 floor | 2020-03-31 16:00:00+00 | 47.0046211887772 | 23.0230117216706 | 0.53142289724201 ceiling | 2020-03-31 16:30:00+00 | 58.7817596504465 | 63.6621567420661 | 0.488188337767497 floor | 2020-03-31 16:30:00+00 | 44.611586847653 | 2.21919436007738 | 0.434762630766879 ceiling | 2020-03-31 17:00:00+00 | 35.7026890735142 | 42.9420990403742 | 0.550129583687522 floor | 2020-03-31 17:00:00+00 | 62.2794370166957 | 52.6636955793947 | 0.454323202022351 ... ``` You have now successfully simulated and run queries on an IoT dataset. # Basic compression with hypercore > Compress data into the columnstore for up to 98% storage savings and faster analytical queries Hypercore is TimescaleDB's hybrid row-columnar storage engine. It automatically compresses chunks from the rowstore into the columnstore, typically reducing storage by 90–98% while making analytical queries significantly faster. ## How it works [Section titled “How it works”](#how-it-works) When you create a hypertable with `segmentby` and `orderby` options, TimescaleDB automatically creates a [columnstore policy](/docs/reference/timescaledb/hypercore/add_columnstore_policy) that converts older chunks into columnar format. Recent data stays in the rowstore for fast inserts, while older data is compressed in the columnstore for efficient scans. ## Enable hypercore [Section titled “Enable hypercore”](#enable-hypercore) ## Prerequisites for this tutorial To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service), or a running instance of [self-hosted TimescaleDB](/docs/get-started/choose-your-path/install-timescaledb). - Any client that can run SQL (Tiger Console, `psql`, or your app's SQL driver). Create a hypertable with hypercore enabled using `CREATE TABLE`: ```sql CREATE TABLE conditions ( time TIMESTAMPTZ NOT NULL, location TEXT NOT NULL, device TEXT NOT NULL, temperature DOUBLE PRECISION NULL, humidity DOUBLE PRECISION NULL ) WITH ( tsdb.hypertable, tsdb.segmentby = 'device', tsdb.orderby = 'time DESC' ); ``` This automatically: * Creates a hypertable partitioned by `time` * Sets `device` as the segment column for efficient filtering * Orders data by `time DESC` for optimal compression and scan performance * Creates a columnstore policy that compresses chunks after one chunk interval Tips Choose `segmentby` based on how you filter data (for example, by device, location, or user). Choose `orderby` based on your most common sort order (usually time descending). Lower cardinality `segmentby` columns give better compression. ## Check compression results [Section titled “Check compression results”](#check-compression-results) After the columnstore policy runs, check how much space you've saved: ```sql SELECT hypertable_size('conditions') AS total_size, pg_size_pretty(hypertable_size('conditions')) AS pretty_size; ``` For detailed per-chunk information: ```sql SELECT * FROM chunks_detailed_size('conditions'); ``` ## Benefits [Section titled “Benefits”](#benefits) * Storage savings: 90–98% compression ratios are common. * Query performance: Analytical queries run faster on columnar data thanks to vectorized execution. * Cost reduction: Less storage means lower cloud costs. * Transparent: Queries work the same on both rowstore and columnstore data. ## Next steps [Section titled “Next steps”](#next-steps) * [Understand hypercore](/docs/learn/columnar-storage/understand-hypercore): How the hybrid storage engine works. * [Set up hypercore](/docs/build/columnar-storage/setup-hypercore): Optimize `segmentby`, `orderby`, and policies. * [Compression methods in hypercore](/docs/learn/columnar-storage/compression-methods): Algorithms used for each data type. # Your first hypertable > Learn what hypertables are, why they matter, and create your first one step by step You will create a hypertable from scratch, see why time-based partitioning matters, and finish with queries you can reuse. Expect the same SQL surface you already know, with chunk-aware planning underneath. ## What is a hypertable? [Section titled “What is a hypertable?”](#what-is-a-hypertable) A hypertable is the main abstraction TimescaleDB uses for time-series data. To your application it looks and behaves like a regular PostgreSQL table: you `INSERT`, `SELECT`, `UPDATE`, and `DELETE` using standard SQL. Under the hood, the database automatically partitions your rows by time into storage units called chunks. Each chunk holds a time range of data (for example, one day or one week). When you run a query, TimescaleDB figures out which chunks are relevant and only touches those, instead of scanning the whole table. That is why hypertables can scale to very large volumes while keeping queries fast. For a deeper look at structure, partitioning, and indexes, see [Understand hypertables](/docs/learn/hypertables/understand-hypertables). ## Why use hypertables? [Section titled “Why use hypertables?”](#why-use-hypertables) Time-series data (sensors, metrics, events, logs) usually grows quickly and is queried by time range. A normal table can become slow and hard to manage as it gets large. Hypertables are built for that workload: they give you automatic time-based partitioning, efficient time-range queries, and scalable data management (for example, compression and retention) without changing how you write SQL. You get the familiarity of a PostgreSQL table with the performance and scalability of a purpose-built time-series store. ## Advantages of hypertables [Section titled “Advantages of hypertables”](#advantages-of-hypertables) * Efficient data management: Data is split into chunks by time. You can [tune chunk intervals](/docs/build/performance-optimization/improve-hypertable-performance#optimize-hypertable-chunk-intervals) to match how you ingest and query (for example, one chunk per day or per week). * Better performance: A time-descending index is created by default; you can add more indexes (including [unique indexes](/docs/build/performance-optimization/hypertables-and-unique-indexes)) as needed. Queries use these indexes and only open relevant chunks. * Chunk skipping: The planner skips chunks that don't match your query (for example, a time filter), which [reduces work and speeds up queries](/docs/build/performance-optimization/improve-hypertable-performance#enable-chunk-skipping). * Advanced analysis: Hyperfunctions and related features let you aggregate and analyze large amounts of time-series data efficiently. See [About TimescaleDB hyperfunctions](/docs/learn/hyperfunctions/about-hyperfunctions). * Familiar interface: You interact with a hypertable exactly like a normal table; partitioning and optimization are handled for you. ## How are hypertables used? [Section titled “How are hypertables used?”](#how-are-hypertables-used) You use a hypertable like any other table: connect with your usual client, run `INSERT` and `SELECT` (and `UPDATE`/`DELETE` where needed). Typical use cases include: * IoT and sensors: Device readings with a timestamp (temperature, humidity, location). * Metrics and monitoring: Server or app metrics (CPU, latency, errors) over time. * Events and logs: User or system events with a time component. For creating, altering, and managing hypertables in real projects, see [Understand hypertables](/docs/learn/hypertables/understand-hypertables). For issues such as unique indexes, compression, or dropping chunks, see [Troubleshoot hypertables](/docs/build/tips-and-tricks/troubleshoot-hypertables). ## Make your first hypertable [Section titled “Make your first hypertable”](#make-your-first-hypertable) You will create a simple hypertable for sensor-style data: a `conditions` table with time, location, device, temperature, and humidity. When you run the command, the table is created and automatically set up as a hypertable partitioned by the time column. ## Prerequisites for this procedure To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service), or a running instance of [self-hosted TimescaleDB](/docs/get-started/choose-your-path/install-timescaledb). - Your [connection details](/docs/integrate/find-connection-details). - Any client that can run SQL (Tiger Console, `psql`, or your app's SQL driver). 1. **Create the hypertable** Run the following SQL. The important part is the `WITH (tsdb.hypertable, ...)` clause: it tells the database to treat this table as a hypertable. The first column with a timestamp type (`time`) is used as the partitioning column by default. We use `tsdb.segmentby = 'device'` so that rows are grouped by `device` inside chunks, which helps both query and compression when you filter by device. We use `tsdb.orderby = 'time DESC'` so that recent data is stored in an order that matches typical time-range queries. ```sql CREATE TABLE conditions ( time TIMESTAMPTZ NOT NULL, location TEXT NOT NULL, device TEXT NOT NULL, temperature DOUBLE PRECISION NULL, humidity DOUBLE PRECISION NULL ) WITH ( tsdb.hypertable, tsdb.segmentby = 'device', tsdb.orderby = 'time DESC' ); ``` You should see a confirmation that the table was created (exact message depends on your client). The table is now a hypertable: new chunks will be created automatically as you insert data into new time ranges. 2. **Insert a few rows** Insert some sample rows so we can query the hypertable: ```sql INSERT INTO conditions (time, location, device, temperature, humidity) VALUES (NOW() - INTERVAL '2 hours', 'office', 'sensor-1', 22.1, 45.0), (NOW() - INTERVAL '1 hour', 'office', 'sensor-1', 22.3, 44.8), (NOW(), 'office', 'sensor-1', 22.2, 45.1); ``` 3. **Query the hypertable** Query it like a normal table: ```sql SELECT * FROM conditions ORDER BY time DESC; ``` You should see your three rows, with the most recent first. Notice that you didn't have to mention chunks or partitioning, you just use standard SQL. The database routes the query to the right chunk(s) for you. You have built your first hypertable and used it for inserts and a time-ordered query. From here you can add more data, add indexes, or enable features like the columnstore. The [Understand hypertables](/docs/learn/hypertables/understand-hypertables) guide and [Hypertable operations](/docs/learn/hypertables/optimize-data-in-hypertables) cover those next steps. ## Next steps [Section titled “Next steps”](#next-steps) * [Understand chunks](/docs/learn/chunks/understanding-chunks): What chunks are, why they matter, and how to use them effectively. * [Basic compression with hypercore](/docs/build/how-to/basic-compression): Compress data into the columnstore for up to 98% storage savings and faster analytical queries. * [Hypertable operations](/docs/learn/hypertables/optimize-data-in-hypertables): Create, alter, and drop hypertables, and speed up data ingest. # Performance optimization > Diagnose and optimize query performance Optimize your hypertables and queries with indexes, constraints, schema tuning, and TimescaleDB-specific features like chunk skipping and columnstore ordering. ## Schema fundamentals [Section titled “Schema fundamentals”](#schema-fundamentals) * [Accelerate queries using indexes](/docs/build/performance-optimization/indexing) * [Ensure data integrity with constraints](/docs/build/performance-optimization/ensure-data-integrity-with-constraints) ## Schema management [Section titled “Schema management”](#schema-management) * [Alter and update table schemas](/docs/build/performance-optimization/alter-update-table-schema) * [Handle semi-structured data with JSON](/docs/build/performance-optimization/handle-semi-structured-data-with-json) ## Hypertable and query optimization [Section titled “Hypertable and query optimization”](#hypertable-and-query-optimization) * [Enforce constraints with unique indexes](/docs/build/performance-optimization/hypertables-and-unique-indexes) * [Improve query and upsert performance](/docs/build/performance-optimization/secondary-indexes) * [Improve hypertable performance](/docs/build/performance-optimization/improve-hypertable-performance) ## Advanced [Section titled “Advanced”](#advanced) * [Improve storage performance using tablespaces](/docs/build/performance-optimization/manage-tablespaces) * [Automate tasks with triggers](/docs/build/performance-optimization/automate-tasks-with-triggers) * [Query external data sources with FDW](/docs/build/performance-optimization/query-external-data-sources-with-fdw) # Alter and update table schemas > Modify the schema of an existing hypertable with the ALTER TABLE command in PostgreSQL Schema changes on a hypertable use normal PostgreSQL `ALTER TABLE` syntax, and TimescaleDB propagates those changes to every underlying chunk. Note Schema changes to a continuous aggregate are limited. You can rename the view or its columns, set a schema, change the owner, and adjust parameters. Since TimescaleDB 2.28.0, you can also add a new aggregate with [`ALTER MATERIALIZED VIEW ... ADD COLUMN`](/docs/reference/timescaledb/continuous-aggregates/alter_materialized_view). Other schema changes are not supported. For example, to add a new column called `address` to a table called `distributors`: ```sql ALTER TABLE distributors ADD COLUMN address varchar(30); ``` This creates the new column, with all existing entries recording `NULL` for the new column. Changing the schema can, in some cases, consume a lot of resources. This is especially true if it requires underlying data to be rewritten. If you want to check your schema change before you apply it, you can use a `CHECK` constraint, like this: ```sql ALTER TABLE distributors ADD CONSTRAINT zipchk CHECK (char_length(zipcode) = 5); ``` This scans the table to verify that existing rows meet the constraint, but does not require a table rewrite. ## Altering hypertables with columnstore enabled [Section titled “Altering hypertables with columnstore enabled”](#altering-hypertables-with-columnstore-enabled) Most common schema modifications work on hypertables with columnstore enabled, including adding columns, renaming columns, dropping columns, adding constraints, setting NOT NULL, and changing defaults. However, some operations are blocked, the most common of them being: * **Changing column data type** (`ALTER COLUMN ... TYPE`) * **Changing column storage** (`ALTER COLUMN ... SET STORAGE`) * **Dropping orderby or segmentby columns** * **Row-level security operations** (`ENABLE/DISABLE ROW SECURITY`) When you attempt a blocked operation, you receive an error: ```plaintext ERROR: operation not supported on hypertables that have columnstore enabled ``` If you encounter this error, you need to: 1. Stop any columnstore policy 2. Convert the affected chunks back into rowstore 3. Disable columnstore 4. Perform the schema change 5. Re-enable columnstore and restart the policy ### Example: change the column type on a hypertable with columnstore enabled [Section titled “Example: change the column type on a hypertable with columnstore enabled”](#example-change-the-column-type-on-a-hypertable-with-columnstore-enabled) This example shows how to change a column's data type on a hypertable with columnstore enabled, which requires conversion to rowstore: 1. **Check if you have a columnstore policy and note its settings** Query the jobs catalog to find the `job_id` and configuration for any existing columnstore policy: ```sql SELECT job_id, config FROM timescaledb_information.jobs WHERE proc_name = 'policy_compression' AND hypertable_name = 'conditions'; ``` 2. **Pause the columnstore policy** If a policy exists, pause it so it does not convert chunks while you are modifying the schema: ```sql SELECT alter_job(, scheduled => false); ``` 3. **Convert all chunks back to rowstore** Use [`convert_to_rowstore`](/docs/reference/timescaledb/hypercore/convert_to_rowstore) to move each chunk out of the columnstore: ```sql DO $$ DECLARE chunk regclass; BEGIN FOR chunk IN SELECT show_chunks('conditions') LOOP CALL convert_to_rowstore(chunk); END LOOP; END $$; ``` 4. **Disable columnstore** Some operations, such as changing a column type, require columnstore to be fully disabled: ```sql ALTER TABLE conditions SET (timescaledb.columnstore = false); ``` 5. **Perform the schema modification** ```sql ALTER TABLE conditions ALTER COLUMN temperature TYPE double precision; ``` 6. **Re-enable columnstore with original settings** ```sql ALTER TABLE conditions SET ( timescaledb.columnstore = true, timescaledb.orderby = 'time DESC', timescaledb.segmentby = 'device_id' ); ``` 7. **Restart the columnstore policy** ```sql SELECT alter_job(, scheduled => true); ``` 8. **Optionally, convert chunks to columnstore immediately** Without this step, chunks are converted by the policy on its next scheduled run. To convert them now, use [`convert_to_columnstore`](/docs/reference/timescaledb/hypercore/convert_to_columnstore): ```sql DO $$ DECLARE chunk regclass; BEGIN FOR chunk IN SELECT show_chunks('conditions') LOOP CALL convert_to_columnstore(chunk); END LOOP; END $$; ``` For more information about PostgreSQL `ALTER TABLE` operations, see the [PostgreSQL `ALTER TABLE` documentation](https://www.postgresql.org/docs/current/sql-altertable.html). # Automate tasks with triggers > Use PostgreSQL triggers to automatically execute functions when data changes occur in TimescaleDB hypertables TimescaleDB supports PostgreSQL triggers on a hypertable the same way you use them on a regular table. When you create, alter, drop, enable, or disable a trigger on a hypertable — including via `ALTER TABLE … ENABLE/DISABLE TRIGGER` with the `ALL`, `USER`, `ALWAYS`, and `REPLICA` variants — TimescaleDB propagates the change to every chunk. ## Create a trigger [Section titled “Create a trigger”](#create-a-trigger) This example creates a new table called `error_conditions` with the same schema as `conditions`, but that only stores records which are considered errors. An error, in this case, is when an application sends a `temperature` or `humidity` reading with a value that is greater than or equal to 1000. 1. **Create a function that inserts erroneous data into the `error_conditions` table** ```sql CREATE OR REPLACE FUNCTION record_error() RETURNS trigger AS $record_error$ BEGIN IF NEW.temperature >= 1000 OR NEW.humidity >= 1000 THEN INSERT INTO error_conditions VALUES(NEW.time, NEW.location, NEW.temperature, NEW.humidity); END IF; RETURN NEW; END; $record_error$ LANGUAGE plpgsql; ``` 2. **Create a trigger that calls this function whenever a new row is inserted into the hypertable** ```sql CREATE TRIGGER record_error BEFORE INSERT ON conditions FOR EACH ROW EXECUTE FUNCTION record_error(); ``` 3. **Verify that error rows are captured** All data is inserted into the `conditions` table, but rows that contain errors are also added to the `error_conditions` table. For all supported trigger types and clauses, see the [PostgreSQL `CREATE TRIGGER` docs](https://www.postgresql.org/docs/current/sql-createtrigger.html). ## Limitations [Section titled “Limitations”](#limitations) * Triggers are not supported on continuous aggregates. * `ROW`-level triggers with transition tables are not supported on hypertables. * `DELETE` triggers with transition tables are not supported. # Automatically route queries to continuous aggregates > Have TimescaleDB transparently route matching queries to a continuous aggregate instead of the raw hypertable Since [2.27.0](https://github.com/timescale/timescaledb/releases/tag/2.27.0) Warning Experimental features could have bugs. They might not be backwards compatible, and could be removed in future releases. Use these features at your own risk, and do not use any experimental features in production. When you enable `timescaledb.enable_cagg_rewrites`, TimescaleDB can transparently rewrite a query against a hypertable to read from a matching continuous aggregate instead. If a continuous aggregate aggregates exactly the columns and time buckets the query asks for, TimescaleDB replaces the hypertable scan with a continuous aggregate scan, which is much faster on large datasets. ## How continuous aggregate query rewrites work [Section titled “How continuous aggregate query rewrites work”](#how-continuous-aggregate-query-rewrites-work) When you query a hypertable with `SELECT … GROUP BY time_bucket(…)`, TimescaleDB looks for a continuous aggregate on that hypertable whose definition matches the query's grouping columns, `time_bucket` width, and aggregates. If one matches, the planner replaces the hypertable scan with a scan of the continuous aggregate's materialization — the rewrite is invisible to the application. The match must be exact. TimescaleDB does not reaggregate (for example, it does not roll up 1-hour continuous aggregate buckets into 2-hour query buckets). ## Use continuous aggregate query rewrites [Section titled “Use continuous aggregate query rewrites”](#use-continuous-aggregate-query-rewrites) Turn the rewriting on: ```sql SET timescaledb.enable_cagg_rewrites = on; ``` A second setting controls diagnostics: ```sql SET timescaledb.cagg_rewrites_debug_info = on; ``` With `cagg_rewrites_debug_info` on, TimescaleDB prints whether each query is eligible for rewriting and, if not, why. Set this on while `enable_cagg_rewrites` is off to inspect which queries would be rewritten without changing plans. ## Limitations [Section titled “Limitations”](#limitations) * **Exact match required.** The query must aggregate the same columns with the same `time_bucket` width as the target continuous aggregate. TimescaleDB does not roll up smaller buckets into larger ones to match a query. * **Real-time continuous aggregates only.** A continuous aggregate with the default `timescaledb.materialized_only = true` is not eligible. Set `materialized_only = false` on the continuous aggregate you want the rewriter to use. * **PostgreSQL 16 or later.** Rewrites are not supported on PostgreSQL 15. # Ensure data integrity with constraints > Prevent invalid data by applying rules to your table columns using PostgreSQL constraints Constraints reject invalid rows at insert or update time. On hypertables, TimescaleDB mirrors those rules to underlying chunks and keeps related indexes consistent when you add, change, or drop a constraint. hypertables support all standard PostgreSQL constraint types. For foreign keys in particular, the following is supported: * Foreign key constraints from a hypertable referencing a regular table * Foreign key constraints from a regular table referencing a hypertable Foreign keys from a hypertable referencing another hypertable **are not supported**. For example, you can create a table that only allows positive device IDs, and non-null temperature readings. You can also check that time values for all devices are unique. To create this table, with the constraints, use this command: ```sql CREATE TABLE conditions ( time TIMESTAMPTZ, temp FLOAT NOT NULL, device_id INTEGER CHECK (device_id > 0), location INTEGER REFERENCES locations (id), PRIMARY KEY(time, device_id) ) WITH ( tsdb.hypertable ); ``` This example also references values in another `locations` table using a foreign key constraint. Note Time columns used for partitioning must not allow `NULL` values. A `NOT NULL` constraint is added by default to these columns if it doesn't already exist. For more information on how to manage constraints, see the [PostgreSQL docs](https://www.postgresql.org/docs/current/ddl-constraints.html). # Handle semi-structured data with JSON > Use JSON or JSONB to store semi-structured data, such as user-defined fields, and learn how to index the JSONB structure and individual fields JSON and JSONB fit payloads where the shape changes per tenant or device. Keep identifiers and timestamps as typed columns, and tuck the long tail of optional fields into JSON so you can index what matters. ```sql CREATE TABLE metrics ( time TIMESTAMPTZ, user_id INT, device_id INT, data JSONB ); ``` That example keeps `time`, `user_id`, and `device_id` as real columns because predicates and joins hit them far more often than rarely used JSON keys. You should also use the JSONB data type, that is, JSON stored in a binary format, rather than JSON data type. JSONB data types are more efficient in both storage overhead and lookup performance. Note Use JSONB for user-defined data rather than sparse data. This works best for most data sets. For sparse data, use NULLable fields and, if possible, run on top of a compressed file system like ZFS. This will work better than a JSONB data type, unless the data is extremely sparse, for example, more than 95% of fields for a row are empty. ## Index the JSONB structure [Section titled “Index the JSONB structure”](#index-the-jsonb-structure) When you index JSONB data across all fields, it is usually best to use a GIN (generalized inverted) index. In most cases, you can use the default GIN operator, like this: ```sql CREATE INDEX idxgin ON metrics USING GIN (data); ``` For more information about GIN indexes, see the [PostgreSQL documentation](https://www.postgresql.org/docs/current/datatype-json.html#JSON-INDEXING). This index only optimizes queries where the `WHERE` clause uses the `?`, `?&`, `?|`, or `@>` operator. For more information about these operators, see the [PostgreSQL documentation](https://www.postgresql.org/docs/current/functions-json.html#FUNCTIONS-JSONB-OP-TABLE). ## Index individual fields [Section titled “Index individual fields”](#index-individual-fields) JSONB columns sometimes have common fields containing values that are useful to index individually. Indexes like this can be useful for ordering operations on field values, [multicolumn indexes](https://www.postgresql.org/docs/current/indexes-multicolumn.html), and indexes on specialized types, such as a postGIS geography type. Another advantage of indexes on individual field values is that they are often smaller than GIN indexes on the entire JSONB field. To create an index like this, it is usually best to use a [partial index](https://www.postgresql.org/docs/current/indexes-partial.html) on an [expression](https://www.postgresql.org/docs/current/indexes-expressional.html) accessing the field. For example: ```sql CREATE INDEX idxcpu ON metrics(((data->>'cpu')::double precision)) WHERE data ? 'cpu'; ``` In this example, the expression being indexed is the `cpu` field inside the `data` JSONB object, cast to a double. The cast reduces the size of the index by storing the much smaller double, instead of a string. The `WHERE` clause ensures that the only rows included in the index are those that contain a `cpu` field, because the `data ? 'cpu'` returns `true`. This also serves to reduce the size of the index by not including rows without a `cpu` field. Note that in order for a query to use the index, it must have `data ? 'cpu'` in the WHERE clause. This expression can also be used with a multi-column index, for example, by adding `time DESC` as a leading column. Note, however, that to enable index-only scans, you need `data` as a column, not the full expression `((data->>'cpu')::double precision)`. # Enforce constraints with unique indexes > Having a unique index on your hypertable simplifies lookup, speeds up aggregation, and makes JOINs more efficient. Learn to create a unique index and what the related limitations are. You use unique indexes on a hypertable to enforce [constraints](https://www.postgresql.org/docs/current/ddl-constraints.html). If you have a primary key, you have a unique index. In PostgreSQL, a primary key is a unique index with a `NOT NULL` constraint. You do not need to have a unique index on your hypertables. When you create a unique index, it must contain all the partitioning columns of the hypertable. ## Create a hypertable and add unique indexes [Section titled “Create a hypertable and add unique indexes”](#create-a-hypertable-and-add-unique-indexes) To create a unique index on a hypertable: 1. **Determine the partitioning columns** Before you create a unique index, you need to determine which unique indexes are allowed on your hypertable. Begin by identifying your partitioning columns. TimescaleDB traditionally uses the following columns to partition hypertables: * The `time` column used to create the hypertable. Every TimescaleDB hypertable is partitioned by time. * Any space-partitioning columns. Space partitions are optional and not included in every hypertable. 2. **Create a hypertable** Create a [hypertable](/docs/learn/hypertables/understand-hypertables) for your time-series data using [CREATE TABLE](/docs/reference/timescaledb/hypertables/create_table). For [efficient queries](/docs/build/performance-optimization/secondary-indexes) on data in the columnstore, remember to `segmentby` the column you will use most often to filter your data. For example: ```sql CREATE TABLE hypertable_example( time TIMESTAMPTZ, user_id BIGINT, device_id BIGINT, value FLOAT ) WITH ( tsdb.hypertable, tsdb.segmentby = 'device_id', tsdb.orderby = 'time DESC' ); ``` When you create a hypertable using [CREATE TABLE ... WITH ...](/docs/reference/timescaledb/hypertables/create_table), the default partitioning column is automatically the first column with a timestamp data type. Also, TimescaleDB creates a [columnstore policy](/docs/reference/timescaledb/hypercore/add_columnstore_policy) that automatically converts your data to the columnstore, after an interval equal to the value of the [chunk\_interval](/docs/reference/timescaledb/hypertables/set_chunk_time_interval), defined through `after` in the policy. This columnar format enables fast scanning and aggregation, optimizing performance for analytical workloads while also saving significant storage space. In the columnstore conversion, hypertable chunks are compressed by up to 98%, and organized for efficient, large-scale queries. You can customize this policy later using [alter\_job](/docs/reference/timescaledb/jobs-automation/alter_job). However, to change `after` or `created_before`, the compression settings, or the hypertable the policy is acting on, you must [remove the columnstore policy](/docs/reference/timescaledb/hypercore/remove_columnstore_policy) and [add a new one](/docs/reference/timescaledb/hypercore/add_columnstore_policy). You can also manually [convert chunks](/docs/reference/timescaledb/hypercore/convert_to_columnstore) in a hypertable to the columnstore. 3. **Create a unique index on the hypertable** When you create a unique index on a hypertable, it must contain all the partitioning columns. It may contain other columns as well, and they may be arranged in any order. You cannot create a unique index without `time`, because `time` is a partitioning column. For example: * Create a unique index on `time` and `device_id` with a call to `CREATE UNIQUE INDEX`: ```sql CREATE UNIQUE INDEX idx_deviceid_time ON hypertable_example(device_id, time); ``` * Create a unique index on `time`, `user_id`, and `device_id`. `device_id` is not a partitioning column, but this still works: ```sql CREATE UNIQUE INDEX idx_userid_deviceid_time ON hypertable_example(user_id, device_id, time); ``` Note This restriction is necessary to guarantee global uniqueness in the index. ## Create a hypertable from an existing table with unique indexes [Section titled “Create a hypertable from an existing table with unique indexes”](#create-a-hypertable-from-an-existing-table-with-unique-indexes) If you create a unique index on a table before turning it into a hypertable, the same restrictions apply in reverse. You can only partition the table by columns in your unique index. 1. **Create a relational table** ```sql CREATE TABLE another_hypertable_example( time TIMESTAMPTZ, user_id BIGINT, device_id BIGINT, value FLOAT ); ``` 2. **Create a unique index on the table** For example, on `device_id` and `time`: ```sql CREATE UNIQUE INDEX idx_deviceid_time ON another_hypertable_example(device_id, time); ``` 3. **Turn the table into a partitioned hypertable** * On `time` alone: ```sql SELECT * from create_hypertable('another_hypertable_example', by_range('time')); ``` * On `time` and `device_id`: ```sql SELECT * FROM create_hypertable('another_hypertable_example', by_range('time')); SELECT * FROM add_dimension('another_hypertable_example', by_hash('device_id', 4)); ``` You get an error if you try to turn the relational table into a hypertable partitioned by `time` and `user_id`. This is because `user_id` is not part of the `UNIQUE INDEX`. To fix the error, add `user_id` to your unique index. # Improve hypertable and query performance > Use chunk skipping to optimize hypertable performance and make sure your analytical queries are as fast as they can be Hypertables are PostgreSQL tables that help you improve insert and query performance by automatically partitioning your data by time. Each hypertable is made up of child tables called chunks. Each chunk is assigned a range of time, and only contains data from that range. When you run a query, TimescaleDB identifies the correct chunk and runs the query on it, instead of going through the entire table. This page shows you how to tune hypertables to increase performance even more. * [Optimize hypertable chunk intervals](#optimize-hypertable-chunk-intervals): choose the optimum chunk size for your data - manually or automatically * [Enable chunk skipping](#enable-chunk-skipping): skip chunks on non-partitioning columns in hypertables when you query your data * [Analyze your hypertables](#analyze-your-hypertables): use PostgreSQL `ANALYZE` to create the best query plan ## Optimize hypertable chunk intervals [Section titled “Optimize hypertable chunk intervals”](#optimize-hypertable-chunk-intervals) Tuning your hypertable chunk interval can improve performance in your database, especially as your workload evolves over time. With TimescaleDB, you can do this manually or automatically. ### Manual tuning [Section titled “Manual tuning”](#manual-tuning) To manually adjust the chunk size in your database, take the following steps: 1. **Choose an optimum chunk interval** PostgreSQL builds the index on the fly during ingestion. That means that to build a new entry on the index, a significant portion of the index needs to be traversed during every row insertion. When the index does not fit into memory, it is constantly flushed to disk and read back. This wastes IO resources which would otherwise be used for writing the heap/WAL data to disk. The default chunk interval is 7 days. However, best practice is to set `chunk_interval` so that prior to processing, the indexes for chunks currently being ingested into fit within 25% of main memory. For example, on a system with 64 GB of memory, if index growth is approximately 2 GB per day, a 1-week chunk interval is appropriate. If index growth is around 10 GB per day, use a 1-day interval. You set `chunk_interval` when you [create a hypertable](/docs/reference/timescaledb/hypertables/create_table), or by calling [`set_chunk_time_interval`](/docs/reference/timescaledb/hypertables/set_chunk_time_interval) on an existing hypertable. In the following example you create a table called `conditions` that stores time values in the `time` column and has chunks that store data for a `chunk_interval` of one day: ```sql CREATE TABLE conditions ( time TIMESTAMPTZ NOT NULL, location TEXT NOT NULL, device TEXT NOT NULL, temperature DOUBLE PRECISION NULL, humidity DOUBLE PRECISION NULL ) WITH ( tsdb.hypertable, tsdb.chunk_interval='1 day' ); ``` 2. **Check current setting for chunk intervals** Query the TimescaleDB catalog for a hypertable. For example: ```sql SELECT * FROM timescaledb_information.dimensions WHERE hypertable_name = 'conditions'; ``` The result looks like: ```sql hypertable_schema | hypertable_name | dimension_number | column_name | column_type | dimension_type | time_interval | integer_interval | integer_now_func | num_partitions -------------------+-----------------+------------------+-------------+--------------------------+----------------+---------------+------------------+------------------+---------------- public | metrics | 1 | recorded | timestamp with time zone | Time | 1 day | | | ``` Time-based interval lengths are reported in microseconds. 3. **Change the chunk interval length on an existing hypertable** To change the chunk interval on an already existing hypertable, call [`set_chunk_time_interval()`](/docs/reference/timescaledb/hypertables/set_chunk_time_interval). ```sql SELECT set_chunk_time_interval('conditions', INTERVAL '24 hours'); ``` The updated chunk interval only applies to new chunks. This means setting an overly long interval might take a long time to correct. For example, if you set `chunk_interval` to 1 year and start inserting data, you can no longer shorten the chunk for that year. If you need to correct this situation, create a new hypertable and migrate your data. While chunk turnover does not degrade performance, chunk creation does take longer lock time than a normal `INSERT` operation into a chunk that has already been created. This means that if multiple chunks are being created at the same time, the transactions block each other until the first transaction is completed. If you use expensive index types, such as some PostGIS geospatial indexes, take care to check the total size of the chunk and its index using [`chunks_detailed_size()`](/docs/reference/timescaledb/hypertables/chunks_detailed_size). ### Automated tuning [Section titled “Automated tuning”](#automated-tuning) Automated chunk tuning manages chunk intervals for you. Tiger Cloud continuously evaluates your hypertable and adjusts the interval when it detects the current setting is no longer optimal. Updated chunk intervals apply to **new chunks only**; existing historical chunks are not affected. When you enable automated chunk tuning on a hypertable, the interval is evaluated and updated within 5 minutes if necessary. The interval is changed incrementally rather than jumping directly to the target interval: * **Large increase**: if the recommended interval is more than double the current one, the interval is only doubled each evaluation cycle rather than jumping directly to the target. For example, if the current interval is 1 day and the recommended interval is 8 days, the interval increases to 2 days first, then 4 days, then 8 days over successive evaluations. * **Large decrease**: if the recommended interval is less than half the current one, the interval is only halved each evaluation cycle rather than jumping directly to the target. For example, if the current interval is 8 days and the recommended interval is 1 day, the interval decreases to 4 days first, then 2 days, then 1 day over successive evaluations. * **Small change**: if the recommended interval is within 2x of the current one, bigger or smaller, the new interval is applied directly. For example, if the current interval is 4 days and the recommended interval is 6 days, the interval changes to 6 days immediately. If a columnstore policy is configured on the hypertable, automated chunk tuning respects it. The chunk interval is never increased beyond the `after` value. For example, if your compression policy compresses data older than 1 day, the maximum chunk interval is `1 day`. You can track all interval changes in the `Activity` tab. Automated tuning is **off by default** and is configured at the individual hypertable level. Even when disabled, Tiger Console shows the recommended chunk interval for every hypertable so you can evaluate whether a change is warranted before opting in. #### Manage chunk tuning [Section titled “Manage chunk tuning”](#manage-chunk-tuning) To enable automated chunk tuning: 1. **In Tiger Console, select your service** 2. **Click `Explorer` and select your hypertable** ![Chunk auto-tuning in Tiger Console](/docs/_astro/automated-chunk-tuning.fHKpGxep_bJbUP.webp) 3. **Toggle on `Automated chunk tuning`** The recommended chunk interval is visible regardless of whether automated tuning is enabled, so you can preview the suggested value before turning it on. To disable automated chunk tuning: 1. **In Tiger Console, select your service** 2. **Click `Explorer` and select your hypertable** 3. **Toggle off `Automated chunk tuning`** 4. **Optionally set your preferred interval by clicking `Change` under the interval size** The next new chunk will use the interval you specify. #### Limitations [Section titled “Limitations”](#limitations) Automated chunk tuning is not supported for: * Hypertables not owned by `tsdbadmin`. * Hypertables that do not use a time-based partition column. ## Enable chunk skipping [Section titled “Enable chunk skipping”](#enable-chunk-skipping) Early access [2.17.1](https://github.com/timescale/timescaledb/releases/tag/2.17.1) One of the key purposes of hypertables is to make your analytical queries run with the lowest latency possible. When you execute a query on a hypertable, you do not parse the whole table; you only access the chunks necessary to satisfy the query. This works well when the `WHERE` clause of a query uses the column by which a hypertable is partitioned. For example, in a hypertable where every day of the year is a separate chunk, a query for September 1 accesses only the chunk for that day. However, many queries use columns other than the partitioning one. For example, a satellite company might have a table with two columns: one for when data was gathered by a satellite and one for when it was added to the database. If you partition by the date of gathering, a query by the date of adding accesses all chunks in the hypertable and slows the performance. To improve query performance, TimescaleDB enables you to skip chunks on non-partitioning columns in hypertables. Important Chunk skipping only works on chunks converted to the columnstore **after** you `enable_chunk_skipping`. ### How chunk skipping works [Section titled “How chunk skipping works”](#how-chunk-skipping-works) You enable chunk skipping on a column in a hypertable. TimescaleDB tracks the minimum and maximum values for that column in each chunk. These ranges are stored in the start (inclusive) and end (exclusive) format in the `chunk_column_stats` catalog table. TimescaleDB uses these ranges for dynamic chunk exclusion when the `WHERE` clause of an SQL query specifies ranges on the column. ![How chunk skipping narrows query execution to relevant chunks](/docs/_astro/hypertable-with-chunk-skipping.CWpD6uFb_y7uoW.webp) ![How chunk skipping narrows query execution to relevant chunks](/docs/_astro/hypertable-with-chunk-skipping-dark.BVJfg_rN_1tX4eK.webp) You can enable chunk skipping on hypertables compressed into the columnstore for `smallint`, `int`, `bigint`, `serial`, `bigserial`, `date`, `timestamp`, or `timestamptz` type columns. ### When to enable chunk skipping [Section titled “When to enable chunk skipping”](#when-to-enable-chunk-skipping) You can enable chunk skipping on as many columns as you need. However, best practice is to enable it on columns that are both: * Correlated, that is, related to the partitioning column in some way. * Referenced in the `WHERE` clauses of the queries. In the satellite example, the time of adding data to a database inevitably follows the time of gathering. Sequential IDs and the creation timestamp for both entities also increase synchronously. This means those two columns are correlated. For a more in-depth look on chunk skipping, see [our blog post](https://www.tigerdata.com/blog/boost-postgres-performance-by-7x-with-chunk-skipping-indexes). ### Enable chunk skipping [Section titled “Enable chunk skipping”](#enable-chunk-skipping-1) To enable chunk skipping on a column, call [`enable_chunk_skipping()`](/docs/reference/timescaledb/hypertables/enable_chunk_skipping) on a hypertable for a `column_name`. For example, the following query enables chunk skipping on the `order_id` column in the `orders` table: ```sql SELECT enable_chunk_skipping('orders', 'order_id'); ``` For more details on how to implement chunk skipping, see the [API reference](/docs/reference/timescaledb/hypertables/enable_chunk_skipping). ## Analyze your hypertables [Section titled “Analyze your hypertables”](#analyze-your-hypertables) You can use the PostgreSQL `ANALYZE` command to query all chunks in your hypertable. The statistics collected by the `ANALYZE` command are used by the PostgreSQL planner to create the best query plan. For more information about the `ANALYZE` command, see the [PostgreSQL documentation](https://www.postgresql.org/docs/current/sql-analyze.html). # Accelerate queries using indexes > Add indexes to speed up queries and DML on your hypertables — different mechanisms apply to the rowstore and the columnstore Indexes shrink scan work for point lookups and selective filters. TimescaleDB uses two complementary mechanisms: * **Regular PostgreSQL indexes on the rowstore** — B-tree indexes on recent, non-columnstore chunks, configured the same way as on a regular PostgreSQL table. * **Sparse indexes on the columnstore** — `bloom`, `minmax`, and `firstlast` summaries per columnstore batch that let the engine skip batches that can't match a query. See [Sparse indexes on the columnstore](#sparse-indexes-on-the-columnstore). For background on how the columnstore stores compressed data, see [Compression methods](/docs/learn/columnar-storage/compression-methods). ## Indexes on the rowstore [Section titled “Indexes on the rowstore”](#indexes-on-the-rowstore) You can create an index on any combination of columns. To define an index as `UNIQUE` or `PRIMARY KEY`, it must include the partitioning column (this is usually the time column). When you create a hypertable, set the datatype for the `time` column as `timestamptz` and not `timestamp`. For more information, see [PostgreSQL timestamp best practices](https://wiki.postgresql.org/wiki/Don't_Do_This#Don.27t_use_timestamp_.28without_time_zone.29). Note While it is possible to add an index that does not include the `time` column, doing so results in very slow ingest speeds. For time-series data, indexing on the time column allows one index to be created per chunk. ### Choose effective index columns [Section titled “Choose effective index columns”](#choose-effective-index-columns) Consider a simple example with temperatures collected from two locations named `office` and `garage`: An index on `(location, time DESC)` is organized like this: ```sql garage-0940 garage-0930 garage-0920 garage-0910 office-0930 office-0920 office-0910 ``` An index on `(time DESC, location)` is organized like this: ```sql 0940-garage 0930-garage 0930-office 0920-garage 0920-office 0910-garage 0910-office ``` A good rule of thumb with indexes is to think in layers. Start by choosing the columns that you typically want to run equality operators on, such as `location = garage`. Then finish by choosing columns you want to use range operators on, such as `time > 0930`. ### Composite indexes [Section titled “Composite indexes”](#composite-indexes) As a more complex example, imagine you have a number of devices tracking 1,000 different retail stores. You have 100 devices per store, and 5 different types of devices. All of these devices report metrics as `float` values, and you decide to store all the metrics in the same table, like this: ```sql CREATE TABLE devices ( time timestamptz, device_id int, device_type int, store_id int, value float ); ``` When you create this table, an index is automatically generated on the time column, making it faster to query your data based on time. If you want to query your data on something other than time, you can create different indexes. For example, you might want to query data from the last month for just a given `device_id`. Or you could query all data for a single `store_id` for the last three months. You want to keep the index on time so that you can quickly filter for a given time range, and add another index on `device_id` and `store_id`. This creates a composite index. A composite index on `(store_id, device_id, time)` orders by `store_id` first. Each unique `store_id`, will then be sorted by `device_id` in order. And each entry with the same `store_id` and `device_id` are then ordered by `time`. To create this index, use this command: ```sql CREATE INDEX ON devices (store_id, device_id, time DESC); ``` When you have this composite index on your hypertable, you can run a range of different queries. Here are some examples: ```sql SELECT * FROM devices WHERE store_id = x ``` This queries the portion of the list with a specific `store_id`. The index is effective for this query, but could be a bit bloated; an index on just `store_id` would probably be more efficient. ```sql SELECT * FROM devices WHERE store_id = x AND time > 10 ``` This query is not effective, because it would need to scan multiple sections of the list. This is because the part of the list that contains data for `time > 10` for one device would be located in a different section than for a different device. In this case, consider building an index on `(store_id, time)` instead. ```sql SELECT * FROM devices WHERE device_id = M AND time > 10 ``` The index in the example is useless for this query, because the data for `device M` is located in a completely different section of the list for each `store_id`. ```sql SELECT * FROM devices WHERE store_id = M AND device_id = M AND time > 10 ``` This is an effective query for this index. It narrows down the list to a very specific portion. ### Create indexes [Section titled “Create indexes”](#create-indexes) You can create an index using the `CREATE INDEX` command. For example, to create an index that sorts first by `location`, then by `time`, in descending order: ```sql CREATE INDEX ON conditions (location, time DESC); ``` Tips Monitor index creation progress on a hypertable through PostgreSQL's [`pg_stat_progress_create_index`](https://www.postgresql.org/docs/current/progress-reporting.html#CREATE-INDEX-PROGRESS-REPORTING) view. Since 2.27.0, TimescaleDB reports per-chunk progress so you can track long-running builds. If you do not want to lock the entire hypertable for the duration of the build, use [`CREATE INDEX … WITH (timescaledb.transaction_per_chunk)`](/docs/reference/timescaledb/hypertables/create_index), which creates the index in a separate transaction per chunk. Other operations on the hypertable can run concurrently between chunk transactions. ### Default indexes [Section titled “Default indexes”](#default-indexes) When you create a hypertable with [`CREATE TABLE`](/docs/reference/timescaledb/hypertables/create_table), TimescaleDB creates a time index automatically. If you want to create the time index manually, run: ```sql CREATE INDEX ON conditions (time DESC); ``` You can also create an additional index on another column and time. For example: ```sql CREATE INDEX ON conditions (location, time DESC); ``` To opt out of default index creation, set `tsdb.create_default_indexes` to `false` when you create the hypertable: ```sql CREATE TABLE conditions ( time TIMESTAMPTZ NOT NULL, location TEXT NOT NULL, device TEXT NOT NULL, temperature DOUBLE PRECISION NULL, humidity DOUBLE PRECISION NULL ) WITH ( tsdb.hypertable, tsdb.create_default_indexes=false ); ``` When you create a hypertable using [CREATE TABLE ... WITH ...](/docs/reference/timescaledb/hypertables/create_table), the default partitioning column is automatically the first column with a timestamp data type. Also, TimescaleDB creates a [columnstore policy](/docs/reference/timescaledb/hypercore/add_columnstore_policy) that automatically converts your data to the columnstore, after an interval equal to the value of the [chunk\_interval](/docs/reference/timescaledb/hypertables/set_chunk_time_interval), defined through `after` in the policy. This columnar format enables fast scanning and aggregation, optimizing performance for analytical workloads while also saving significant storage space. In the columnstore conversion, hypertable chunks are compressed by up to 98%, and organized for efficient, large-scale queries. You can customize this policy later using [alter\_job](/docs/reference/timescaledb/jobs-automation/alter_job). However, to change `after` or `created_before`, the compression settings, or the hypertable the policy is acting on, you must [remove the columnstore policy](/docs/reference/timescaledb/hypercore/remove_columnstore_policy) and [add a new one](/docs/reference/timescaledb/hypercore/add_columnstore_policy). You can also manually [convert chunks](/docs/reference/timescaledb/hypercore/convert_to_columnstore) in a hypertable to the columnstore. ### Best practices [Section titled “Best practices”](#best-practices) If you have sparse data, with columns that are often NULL, you can add a clause to the index, saying `WHERE column IS NOT NULL`. This prevents the index from indexing NULL data, which can lead to a more compact and efficient index. For example: ```sql CREATE INDEX ON conditions (time DESC, humidity) WHERE humidity IS NOT NULL; ``` Time-series data uses `UNIQUE` indexes more rarely than relational data. When you do need one, the index must include the partitioning column. For example, on a hypertable partitioned only on `time`, a `UNIQUE` index on `(time, location)` is valid; a `UNIQUE` index on `(location)` alone is not. ## Sparse indexes on the columnstore [Section titled “Sparse indexes on the columnstore”](#sparse-indexes-on-the-columnstore) Once chunks are converted to the columnstore, regular B-tree indexes don't apply. Instead, TimescaleDB maintains *sparse indexes* — small per-batch summaries that let the engine skip batches that can't match the query before reading or moving them back to the rowstore. You configure two index types explicitly: * **`bloom(, ...)`** — probabilistic filter for equality predicates. Lets the engine answer "could this batch contain `column = X`?" without inspecting the batch's compressed data. Accepts one or more columns; a [composite bloom](#composite-bloom-filters) tests combined equality predicates across the listed columns. * **`minmax()`** — stores per-batch min and max for the column. Lets the engine answer range and equality predicates by comparing the predicate against the batch's bounds. A third type, **`firstlast`**, is maintained automatically on `tsdb.orderby` columns and isn't configurable. See [Default sparse indexes](#default-sparse-indexes). Sparse indexes accelerate `SELECT`, `UPSERT`, `UPDATE`, and `DELETE` on columnstore data. For DML, when a batch can't contain matching rows, TimescaleDB skips it without moving it back to the rowstore for checking — this is what delivers the headline speedups (more than 2× for upserts, up to 160× for `UPDATE`/`DELETE` with selective bloom matches). ### Default sparse indexes [Section titled “Default sparse indexes”](#default-sparse-indexes) By default ([`timescaledb.auto_sparse_indexes`](/docs/reference/timescaledb/configuration/gucs)), TimescaleDB chooses sparse indexes for you, based on the hypertable's existing B-tree indexes and column data types. Setting `tsdb.orderby` always adds an implicit sparse index on the orderby column. Since 2.28.0, that implicit index is a `firstlast` index on chunks newly converted to the columnstore. It stores the first and last orderby value in each batch, which does two things: * Skips batches that can't match a range or equality filter, the same as `minmax`. * Answers latest- or earliest-value queries straight from the metadata, without scanning the full batches. This covers `max()` and `min()` on the orderby column, and `first(value, time)` / `last(value, time)`. chunks converted before 2.28.0 keep their `minmax` orderby metadata until they are reconverted to the columnstore. ### Configure sparse indexes [Section titled “Configure sparse indexes”](#configure-sparse-indexes) Configure sparse indexes when you [`CREATE TABLE`](/docs/reference/timescaledb/hypertables/create_table#arguments), and change them at any time with [`ALTER TABLE`](/docs/reference/timescaledb/hypercore/alter_table#arguments). Use `tsdb.sparse_index` with a comma-separated list: ```sql CREATE TABLE metrics ( time TIMESTAMPTZ NOT NULL, device_id INT NOT NULL, region TEXT, cpu DOUBLE PRECISION, PRIMARY KEY (time, device_id) ) WITH ( tsdb.hypertable, tsdb.orderby = 'time DESC', tsdb.segmentby = 'region', tsdb.sparse_index = 'bloom(device_id), minmax(cpu)' ); ``` ### Restrictions [Section titled “Restrictions”](#restrictions) * **Bloom on `tsdb.segmentby` columns**: not allowed, single-column or composite. A bloom whose column list includes any segmentby column is rejected. * **Single-column bloom on a `tsdb.orderby` column**: not allowed. The orderby column gets an implicit sparse index automatically; use that for range filters. A composite `bloom(orderby_col, other_col)` that includes the orderby column **is** allowed. * **Minmax on `tsdb.segmentby` columns**: not allowed. * **Minmax on a `tsdb.orderby` column**: accepted, but on 2.28.0 and later it replaces the orderby column's automatic `firstlast` index with a legacy `minmax` one. Leave it unset unless you specifically need `minmax` there. * **One single-column sparse index per column**: you cannot have two indexes on the same single column (`bloom(region), minmax(region)` is rejected). Composite indexes are not affected by this rule (`bloom(region, dev), bloom(dev)` is valid). ### Composite bloom filters [Section titled “Composite bloom filters”](#composite-bloom-filters) When a query filters on multiple columns that have `bloom` indexes, TimescaleDB automatically combines the per-column bloom checks at query time — no extra configuration needed. For known multi-column query patterns, you can also configure an explicit composite bloom by listing the columns inside a single `bloom()` call. This stores one combined bloom per batch, which can be tighter than separate per-column blooms: ```sql ALTER TABLE metrics SET ( timescaledb.sparse_index = 'bloom(region, device)' ); ``` You can mix single-column and composite bloom indexes on the same hypertable. When multiple bloom indexes apply to a query, the planner picks the most-selective one. ### Verify with EXPLAIN [Section titled “Verify with EXPLAIN”](#verify-with-explain) `EXPLAIN ANALYZE` exposes counters for sparse-index pruning. For `SELECT`, `UPDATE`, and `DELETE`: * **Compressed batches filtered** — batches skipped by sparse indexes without moving them back to the rowstore. * **Batches filtered after decompression** — batches that had to be processed before being filtered out. For `UPSERT`, additional bloom-specific counters: * **Batches checked by bloom**, **Batches pruned by bloom**, **Batches without bloom**, **Batches bloom false positives**. Use these to confirm bloom and minmax indexes are doing useful work for your workload. If `Compressed batches filtered` stays at zero on selective queries, your sparse indexes are not on the right columns. # Improve storage performance using tablespaces > Store database objects in specific physical locations on disk using tablespaces for PostgreSQL hypertable chunks Tablespaces pin tables and indexes to specific disk paths so you can put hot data on fast volumes and colder data on cheaper ones. A hypertable stores each chunk in a tablespace you choose, so one logical table can span many disks. New chunks pick a tablespace according to your configuration. You can attach and detach tablespaces on a hypertable. When a disk runs out of space, you can [detach](/docs/reference/timescaledb/hypertables/detach_tablespace) the full tablespace from the hypertable, and then [attach](/docs/reference/timescaledb/hypertables/attach_tablespace) a tablespace associated with a new disk. To see the tablespaces for your hypertable, use the [`show_tablespaces`](/docs/reference/timescaledb/hypertables/show_tablespaces) command. ## How hypertable chunks are assigned tablespaces [Section titled “How hypertable chunks are assigned tablespaces”](#how-hypertable-chunks-are-assigned-tablespaces) A hypertable can be partitioned in multiple dimensions, but only one of the dimensions is used to determine the tablespace assigned to a particular hypertable chunk. If a hypertable has one or more hash-partitioned, or space, dimensions, it uses the first hash-partitioned dimension. Otherwise, it uses the first time dimension. This strategy ensures that hash-partitioned hypertables have chunks co-located according to hash partition, as long as the list of tablespaces attached to the hypertable remains the same. Modulo calculation is used to pick a tablespace, so there can be more partitions than tablespaces. For example, if there are two tablespaces, partition number three uses the first tablespace. Hypertables that are only time-partitioned add new partitions continuously, and therefore have chunks assigned to tablespaces in a way similar to round-robin. Note It is possible to attach more tablespaces than there are partitions for the hypertable. In this case, some tablespaces remain unused until others are detached or additional partitions are added. This is especially true for hash-partitioned tables. # Query external data sources with FDW > Query other Tiger Cloud services or external PostgreSQL databases using foreign data wrappers You use PostgreSQL foreign data wrappers (FDWs) to query external data sources from a Tiger Cloud service. These external data sources can be one of the following: * Other Tiger Cloud services * PostgreSQL databases outside of Tiger Cloud If you are using VPC peering, you can create FDWs in your Customer VPC to query a service in your Tiger Cloud project. However, you can't create FDWs in your Tiger Cloud services to query a data source in your Customer VPC. This is because Tiger Cloud VPC peering uses AWS PrivateLink for increased security. See [VPC peering documentation](/docs/deploy/tiger-cloud/tiger-cloud-aws/security/vpc) for additional details. PostgreSQL FDWs are particularly useful if you manage multiple Tiger Cloud services with different capabilities, and need to seamlessly access and merge regular and time-series data. ## Prerequisites for this procedure To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service), or a running instance of [self-hosted TimescaleDB](/docs/get-started/choose-your-path/install-timescaledb). - Your [connection details](/docs/integrate/find-connection-details). ## Query another data source To query another data source: * Tiger Cloud You create PostgreSQL FDWs with the `postgres_fdw` extension, which is enabled by default in Tiger Cloud. 1. **Connect to your service** See [how to connect](/docs/build/data-management/run-queries-from-tiger-console). 2. **Create a server** Run the following command using your [connection details](/docs/integrate/find-connection-details): ```sql CREATE SERVER myserver FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '', dbname 'tsdb', port ''); ``` 3. **Create user mapping** Run the following command using your [connection details](/docs/integrate/find-connection-details): ```sql CREATE USER MAPPING FOR tsdbadmin SERVER myserver OPTIONS (user 'tsdbadmin', password ''); ``` 4. **Import a foreign schema (recommended) or create a foreign table** * Import the whole schema: ```sql CREATE SCHEMA foreign_stuff; IMPORT FOREIGN SCHEMA public FROM SERVER myserver INTO foreign_stuff ; ``` * Alternatively, import a limited number of tables: ```sql CREATE SCHEMA foreign_stuff; IMPORT FOREIGN SCHEMA public LIMIT TO (table1, table2) FROM SERVER myserver INTO foreign_stuff; ``` * Create a foreign table. Skip if you are importing a schema: ```sql CREATE FOREIGN TABLE films ( code char(5) NOT NULL, title varchar(40) NOT NULL, did integer NOT NULL, date_prod date, kind varchar(10), len interval hour to minute ) SERVER film_server; ``` A user with the `tsdbadmin` role assigned already has the required `USAGE` permission to create PostgreSQL FDWs. You can enable another user, without the `tsdbadmin` role assigned, to query foreign data. To do so, explicitly grant the permission. For example, for a new `grafana` user: ```sql CREATE USER grafana; GRANT grafana TO tsdbadmin; CREATE SCHEMA fdw AUTHORIZATION grafana; CREATE SERVER db1 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '', dbname 'tsdb', port ''); CREATE USER MAPPING FOR grafana SERVER db1 OPTIONS (user 'tsdbadmin', password ''); GRANT USAGE ON FOREIGN SERVER db1 TO grafana; SET ROLE grafana; IMPORT FOREIGN SCHEMA public FROM SERVER db1 INTO fdw; ``` * Self-hosted TimescaleDB You create PostgreSQL FDWs with the `postgres_fdw` extension. See [documentation](https://www.postgresql.org/docs/current/postgres-fdw.html) on how to enable it. 1. **Connect to your database** Use [`psql`](/docs/integrate/query-administration/psql) to connect to your database. 2. **Create a server** Run the following command using your [connection details](/docs/integrate/find-connection-details): ```sql CREATE SERVER myserver FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '', dbname '', port ''); ``` 3. **Create user mapping** Run the following command using your [connection details](/docs/integrate/find-connection-details): ```sql CREATE USER MAPPING FOR postgres SERVER myserver OPTIONS (user 'postgres', password ''); ``` 4. **Import a foreign schema (recommended) or create a foreign table** * Import the whole schema: ```sql CREATE SCHEMA foreign_stuff; IMPORT FOREIGN SCHEMA public FROM SERVER myserver INTO foreign_stuff ; ``` * Alternatively, import a limited number of tables: ```sql CREATE SCHEMA foreign_stuff; IMPORT FOREIGN SCHEMA public LIMIT TO (table1, table2) FROM SERVER myserver INTO foreign_stuff; ``` * Create a foreign table. Skip if you are importing a schema: ```sql CREATE FOREIGN TABLE films ( code char(5) NOT NULL, title varchar(40) NOT NULL, did integer NOT NULL, date_prod date, kind varchar(10), len interval hour to minute ) SERVER film_server; ``` # Retrofit chunk intervals on a production hypertable > Increase the chunk_interval of an existing hypertable without blocking ingestion, using a table-swap migration and batched backfill. Selecting the optimal `chunk_interval` before your system reaches production scale is a hard architectural decision. As a rule of thumb, it is safer to provision larger chunks than smaller ones: larger chunks compress better in the columnstore and reduce planner overhead. If you find yourself operating a hypertable with an undersized interval, this page provides a production-ready methodology for retrofitting the `chunk_interval` in hindsight, without blocking ongoing ingestion. Warning This is a generalized approach. It does not account for every schema — particularly the full lifecycle of constraints. Apply your own technical diligence to adapt these steps safely and test in a staging environment first. ## Identify the symptoms of undersized chunks [Section titled “Identify the symptoms of undersized chunks”](#identify-the-symptoms-of-undersized-chunks) When a hypertable is fragmented into too many tiny chunks, the PostgreSQL planner works exponentially harder. You typically see: * **Degrading query performance**: maintaining over 1,000 chunks per hypertable is an anti-pattern. Beyond that threshold, scanning catalog tables adds significant planning overhead. * **Increased OOM errors**: out-of-memory errors occur more frequently due to the volume of chunk metadata the database must hold in memory during wide time-series scans. For background on why chunk sizing matters, see [Size hypertable chunks](/docs/learn/hypertables/sizing-hypertable-chunks). Warning The table-swap methodology is **unsafe** for hypertables that enforce unique constraints. Because the new table's index starts empty, incoming live data is not checked against historical tuples in the temporary table, causing fatal duplicate-key violations during backfill. To maintain data integrity, pause all ingestion until the historical migration is complete. ## The scenario [Section titled “The scenario”](#the-scenario) Consider a system storing temperatures from 100 distinct sensors, configured with a `chunk_interval` of 15 minutes: ```sql CREATE TABLE temperature_readings ( time TIMESTAMPTZ NOT NULL, sensor_id INTEGER NOT NULL, temperature DOUBLE PRECISION NULL ) WITH ( tsdb.hypertable, tsdb.segmentby='sensor_id', tsdb.chunk_interval='15 minutes' ); ``` Each sensor emits a reading every 5 minutes, creating tiny chunks holding merely 300 tuples each. Three months in, the hypertable has accumulated nearly 3,000 chunks, causing noticeable performance degradation: ```sql SELECT show_chunks('temperature_readings'); -- 2,977 chunks ``` To resolve this, drastically reduce the chunk count by increasing the `chunk_interval`. The rest of this page walks through a "table swap" strategy: create a new hypertable, swap the underlying relations, and backfill historical data. ## Retrofit the chunk interval [Section titled “Retrofit the chunk interval”](#retrofit-the-chunk-interval) 1. **Provision the target hypertable** Create a structurally identical table, appending a `_new` suffix to the name. Define this table with your newly optimized `chunk_interval` — in this case, **1 day**: ```sql CREATE TABLE temperature_readings_new ( time TIMESTAMPTZ NOT NULL, sensor_id INTEGER NOT NULL, temperature DOUBLE PRECISION NULL ) WITH ( tsdb.hypertable, tsdb.segmentby='sensor_id', tsdb.chunk_interval='1 day' ); ``` To preserve referential integrity, transfer all constraints — foreign keys, primary keys, unique, check, and exclusion — from `temperature_readings` to the new hypertable: ```sql DO $$ DECLARE constraint_record RECORD; drop_stmt TEXT; add_stmt TEXT; -- Define your tables here source_table TEXT := 'temperature_readings'; target_table TEXT := 'temperature_readings_new'; BEGIN -- Loop through ALL constraints on the source table FOR constraint_record IN SELECT conname AS constraint_name, pg_get_constraintdef(oid) AS constraint_definition FROM pg_constraint WHERE conrelid = source_table::regclass -- No contype filter: captures 'p', 'u', 'c', 'f', and 'x' constraint types LOOP drop_stmt := format('ALTER TABLE %I DROP CONSTRAINT %I;', source_table, constraint_record.constraint_name); EXECUTE drop_stmt; RAISE NOTICE 'Dropped: %', drop_stmt; add_stmt := format('ALTER TABLE %I ADD CONSTRAINT %I %s;', target_table, constraint_record.constraint_name, constraint_record.constraint_definition); EXECUTE add_stmt; RAISE NOTICE 'Added: %', add_stmt; END LOOP; END $$; ``` 2. **Swap the tables** To avoid blocking ingestion, swap the table names inside a single transaction. This guarantees that any new incoming data is immediately routed to the optimized hypertable: ```sql BEGIN; ALTER TABLE temperature_readings RENAME TO temperature_readings_temp; ALTER TABLE temperature_readings_new RENAME TO temperature_readings; COMMIT; ``` Warning The moment this transaction commits, the active `temperature_readings` table is empty. Live ingestion continues uninterrupted, but historical queries return no data until the backfill completes. If downstream dependencies rely on strict historical read consistency, run this during a scheduled maintenance window. 3. **Backfill the historical data** Migrate historical data from the temporary table into the new hypertable. To avoid overwhelming memory or the transaction log, use a `PL/pgSQL` block to move data in manageable 1-day batches: ```sql DO $$ DECLARE start_time timestamptz; end_time timestamptz; batch_interval interval := INTERVAL '1 day'; -- Adjust batch size as needed min_time timestamptz; max_time timestamptz; BEGIN -- Find the total time range of your historical data SELECT min(time), max(time) INTO min_time, max_time FROM temperature_readings_temp; start_time := min_time; WHILE start_time <= max_time LOOP end_time := start_time + batch_interval; RAISE NOTICE 'Moving data from % to %', start_time, end_time; INSERT INTO temperature_readings SELECT * FROM temperature_readings_temp WHERE time >= start_time AND time < end_time ORDER BY time ASC; -- Commit the current batch to the database immediately COMMIT; start_time := end_time; END LOOP; END $$; ``` 4. **Verify the new chunk count** Once the backfill completes, verify the new chunk count. The hypertable should now reflect a significantly healthier architecture — for example, dropping from \~3,000 chunks to roughly 32: ```sql SELECT show_chunks('temperature_readings'); ``` Optionally, force compression of newly generated chunks into the columnstore rather than waiting for the next scheduled run of your compression policy: ```sql SELECT compress_chunk(i) FROM show_chunks('temperature_readings', older_than => INTERVAL '1 day') i; ``` See [`show_chunks()`](/docs/reference/timescaledb/hypertables/show_chunks) and [`convert_to_columnstore()`](/docs/reference/timescaledb/hypercore/convert_to_columnstore) for details. 5. **Clean up the legacy table** Once you have verified that all data is present and the application is functioning as expected, drop the legacy temporary table to reclaim disk space: ```sql DROP TABLE temperature_readings_temp; ``` ## Testing resources [Section titled “Testing resources”](#testing-resources) To rehearse this migration in a staging environment, use the following scripts to generate a mock dataset. ### Generate 3 months of historical data [Section titled “Generate 3 months of historical data”](#generate-3-months-of-historical-data) ```sql INSERT INTO temperature_readings (time, sensor_id, temperature) SELECT t.time, s.sensor_id, -- Generates a random temperature between 10.0 and 40.0 random() * 30 + 10 AS temperature FROM -- Generate a timestamp every 5 minutes for the last 3 months generate_series( NOW() - INTERVAL '3 month', NOW(), INTERVAL '5 minutes' ) AS t(time) CROSS JOIN -- Generate sensor IDs from 1 to 100 generate_series(1, 100) AS s(sensor_id); -- Run the compression policy, compress everything older than one day SELECT compress_chunk(i) FROM show_chunks('temperature_readings', older_than => INTERVAL '1 day') i; ``` ### Simulate continuous ingestion [Section titled “Simulate continuous ingestion”](#simulate-continuous-ingestion) Use this `bash` script to continuously trickle live data into the database while you rehearse the migration: ```bash #!/bin/bash CONN_STR="" # Change to your connection string SENSOR_ID=1 echo "Starting data generation... (Press Ctrl+C to stop)" while true; do TEMP=$(awk -v min=10 -v max=40 'BEGIN{srand(); print min+rand()*(max-min)}') psql $CONN_STR -c "INSERT INTO temperature_readings (time, sensor_id, temperature) VALUES (NOW(), $SENSOR_ID, $TEMP);" > /dev/null 2>&1 echo "Inserted: Sensor $SENSOR_ID | Temp $TEMP °C" SENSOR_ID=$(( (SENSOR_ID % 100) + 1 )) sleep 5 done ``` ## Learn more [Section titled “Learn more”](#learn-more) * [Size hypertable chunks](/docs/learn/hypertables/sizing-hypertable-chunks): conceptual background on why chunk sizing matters. * [Improve hypertable and query performance](/docs/build/performance-optimization/improve-hypertable-performance): tune chunk intervals on new chunks and enable chunk skipping. * [`set_chunk_time_interval()`](/docs/reference/timescaledb/hypertables/set_chunk_time_interval): change the chunk interval for newly created chunks. * [`show_chunks()`](/docs/reference/timescaledb/hypertables/show_chunks): inspect existing chunks and their ranges. # Improve query and upsert performance > Use segmenting and ordering data in the columnstore to make lookup queries and upserts faster Wide analytical scans are only half the story; many workloads still need fast point lookups, uniqueness checks, and upserts on compressed data. Segmenting and ordering columnstore data is how you tune those paths without giving up analytics performance. To improve query performance using indexes, see [About indexes](/docs/learn/data-model/understand-database-schemas) and [Indexing data](/docs/build/performance-optimization/indexing). ## Segmenting and ordering data [Section titled “Segmenting and ordering data”](#segmenting-and-ordering-data) To optimize query performance, TimescaleDB enables you to explicitly control the way your data is physically organized in the columnstore. By structuring data effectively, queries can minimize disk reads and execute more efficiently, using vectorized execution for parallel batch processing where possible. ![Columnstore chunk organized with SEGMENTBY and ORDERBY: rows grouped into batches per segment key and ordered by time within each batch](/docs/_astro/columnstore-segmentby-light.Csu9SmJQ_2lxGlQ.webp) ![Columnstore chunk organized with SEGMENTBY and ORDERBY: rows grouped into batches per segment key and ordered by time within each batch](/docs/_astro/columnstore-segmentby-dark.Bxa3vqFc_ZldADl.webp) * **Group related data together to improve scan efficiency**: organizing rows into logical segments ensures that queries filtering by a specific value only scan relevant data sections. For example, in the above, querying for a specific ID is particularly fast. * **Sort data within segments to accelerate range queries**: defining a consistent order reduces the need for post-query sorting, making time-based queries and range scans more efficient. * **Reduce disk reads and maximize vectorized execution**: a well-structured storage layout enables efficient batch processing (Single Instruction, Multiple Data, or SIMD vectorization) and parallel execution, optimizing query performance. By combining segmentation and ordering, TimescaleDB ensures that columnar queries are not only fast but also resource-efficient, enabling high-performance real-time analytics. ### Improve performance in the columnstore by segmenting and ordering data [Section titled “Improve performance in the columnstore by segmenting and ordering data”](#improve-performance-in-the-columnstore-by-segmenting-and-ordering-data) Ordering data in the columnstore has a large impact on the compression ratio and performance of your queries. Rows that change over a dimension should be close to each other. As hypertables contain time-series data, they are partitioned by time. This makes the time column a perfect candidate for ordering your data since the measurements evolve as time goes on. If you use `orderby` as your only columnstore setting, you get a good enough compression ratio to save a lot of storage and your queries are faster. However, if you only use `orderby`, you always have to access your data using the time dimension, then filter the rows returned on other criteria. Accessing the data effectively depends on your use case and your queries. You segment data in the columnstore to match the way you want to access it. That is, in a way that makes it easier for your queries to fetch the right data at the right time. When you segment your data to access specific columns, your queries are optimized and yield even better performance. For example, to access information about a single device with a specific `device_id`, you segment on the `device_id` column. This enables you to run analytical queries on compressed data in the columnstore much faster. To illustrate, run the same query on a hypertable, first without, then with optimizations: 1. **Create a hypertable** Create a `metrics` hypertable with the following command: ```sql CREATE TABLE metrics ( time TIMESTAMPTZ, user_id INT, device_id INT, cpu DOUBLE PRECISION, disk_io DOUBLE PRECISION ) WITH ( tsdb.hypertable ); ``` When you create a hypertable using [CREATE TABLE ... WITH ...](/docs/reference/timescaledb/hypertables/create_table), the default partitioning column is automatically the first column with a timestamp data type. Also, TimescaleDB creates a [columnstore policy](/docs/reference/timescaledb/hypercore/add_columnstore_policy) that automatically converts your data to the columnstore, after an interval equal to the value of the [chunk\_interval](/docs/reference/timescaledb/hypertables/set_chunk_time_interval), defined through `after` in the policy. This columnar format enables fast scanning and aggregation, optimizing performance for analytical workloads while also saving significant storage space. In the columnstore conversion, hypertable chunks are compressed by up to 98%, and organized for efficient, large-scale queries. You can customize this policy later using [alter\_job](/docs/reference/timescaledb/jobs-automation/alter_job). However, to change `after` or `created_before`, the compression settings, or the hypertable the policy is acting on, you must [remove the columnstore policy](/docs/reference/timescaledb/hypercore/remove_columnstore_policy) and [add a new one](/docs/reference/timescaledb/hypercore/add_columnstore_policy). You can also manually [convert chunks](/docs/reference/timescaledb/hypercore/convert_to_columnstore) in a hypertable to the columnstore. 2. **Execute a query on the hypertable without optimizations** ```sql SELECT device_id, AVG(cpu) AS avg_cpu, AVG(disk_io) AS avg_disk_io FROM metrics WHERE time >= '2024-03-01 00:00:00+01' AND time < '2024-03-02 00:00:00+01' AND device_id = 5 GROUP BY device_id; ``` Gives the following result: ```sql device_id | avg_cpu | avg_disk_io -----------+--------------------+--------------------- 5 | 0.4954351575883885 | 0.49725603413909114 (1 row) Time: 29.216 ms ``` 3. **Execute a query on the same data segmented and ordered in the columnstore** 1. Control the way your data is ordered and segmented in the columnstore: ```sql ALTER TABLE metrics SET ( timescaledb.orderby = 'time', timescaledb.segmentby = 'device_id' ); ``` 2. Query your data: ```sql SELECT device_id, AVG(cpu) AS avg_cpu, AVG(disk_io) AS avg_disk_io FROM metrics WHERE time >= '2024-03-01 00:00:00+01' AND time < '2024-03-02 00:00:00+01' AND device_id = 5 GROUP BY device_id; ``` Gives the following result: ```sql device_id | avg_cpu | avg_disk_io -----------+--------------------+--------------------- 5 | 0.4954351575883885 | 0.49725603413909114 (1 row) Time: 1.828 ms ``` As you see, using `orderby` and `segmentby` not only reduces the amount of space taken by your data, but also vastly improves query speed. The number of rows that are compressed together in a single batch (like the ones we see above) is 1000. If your chunk does not contain enough data to create big enough batches, your compression ratio will be reduced. This needs to be taken into account when you define your columnstore settings. # Get faster DISTINCT queries with SkipScan > Speed up DISTINCT queries with the SkipScan optimization Since [2.2.0](https://github.com/timescale/timescaledb/releases/tag/2.2.0) Tiger Data SkipScan dramatically speeds up `DISTINCT` queries. It jumps directly to the first row of each distinct value in an index instead of scanning all rows. First introduced for the rowstore hypertables and relational tables, SkipScan now extends to columnstore hypertables, distinct aggregates like `COUNT(DISTINCT)`, and even multiple columns. ## Speed up `DISTINCT` queries [Section titled “Speed up DISTINCT queries”](#speed-up-distinct-queries) You use `DISTINCT` queries to get only the unique values in your data. For example, the IDs of customers who placed orders, the countries where your users are located, or the devices reporting into an IoT system. You might also have graphs and alarms that repeatedly query the most recent values for every device or service. As your tables get larger, `DISTINCT` queries tend to get slower. Even when your index matches the exact order and columns for these kinds of queries, PostgreSQL (without SkipScan) has to scan the entire index and then run deduplication. As the table grows, this operation keeps getting slower. SkipScan is an optimization for `DISTINCT` and `DISTINCT ON` queries, including multi-column `DISTINCT`. SkipScan allows queries to incrementally jump from one ordered value to the next, without reading the rows in between. Conceptually, SkipScan is a regular IndexScan that skips across an index looking for the next value that is greater than the current value. When you issue a query that uses SkipScan, the `EXPLAIN` output includes a new `Custom Scan (SkipScan)` operator, or node, that can quickly return distinct items from a properly ordered index. As it locates one item, the SkipScan node quickly restarts the search for the next item. This is a much more efficient way of finding distinct items in an ordered index. SkipScan cost is based on the ratio of distinct tuples to total tuples. If the number of distinct tuples is close to the total number of tuples, SkipScan is unlikely to be used due to its higher estimated cost. Multi-column SkipScan is supported for queries that do not produce NULL distinct values. For example: ```sql CREATE INDEX ON metrics(region, device, metric_type, time DESC); -- All distinct columns have filters which don't allow NULLs: can use SkipScan SELECT DISTINCT ON (region, device, metric_type) * FROM metrics WHERE region IN ('UK','EU','JP') AND device > 1 AND metric_type IS NOT NULL ORDER BY region, device, metric_type, time DESC; -- Distinct columns are declared NOT NULL: can use SkipScan with index on (region, device) CREATE TABLE metrics(region TEXT NOT NULL, device INT NOT NULL, ...); SELECT DISTINCT ON (region, device) * FROM metrics ORDER BY region, device, time DESC; ``` For benchmarking information on how SkipScan compares to regular `DISTINCT` queries, see the [SkipScan blog post](https://www.tigerdata.com/blog/skipscan-in-timescaledb-why-distinct-was-slow-how-we-built-it-and-how-you-can-use-it). ## Use SkipScan queries [Section titled “Use SkipScan queries”](#use-skipscan-queries) Design your layout: * Rowstore: create an index starting with the `DISTINCT` columns, followed by your time sort. If the `DISTINCT` columns are not the first in your index, ensure any leading columns are used as constraints in your query. This means that if you are asking a question such as "retrieve a list of unique IDs in order" and "retrieve the last reading of each ID," you need at least one index like this: ```sql CREATE INDEX "cpu_customer_tags_id_time_idx" ON readings (customer_id, tags_id, time DESC) ``` * Columnstore: set `tsdb.segmentby` to the distinct columns and `tsdb.orderby` to match your query's sort. Convert your historical chunks to the columnstore. With your index set up correctly, you should start to see immediate benefit for `DISTINCT` queries. When SkipScan is chosen for your query, the `EXPLAIN ANALYZE` output shows one or more `Custom Scan (SkipScan)` nodes, like this: ```sql -> Unique -> Merge Append Sort Key: _hyper_8_79_chunk.tags_id, _hyper_8_79_chunk."time" DESC -> Custom Scan (SkipScan) on _hyper_8_79_chunk -> Index Only Scan using _hyper_8_79_chunk_cpu_tags_id_time_idx on _hyper_8_79_chunk Index Cond: (tags_id > NULL::integer) -> Custom Scan (SkipScan) on _hyper_8_80_chunk -> Index Only Scan using _hyper_8_80_chunk_cpu_tags_id_time_idx on _hyper_8_80_chunk Index Cond: (tags_id > NULL::integer) ``` # Troubleshooting > Diagnose and fix common Tiger Data problems ## Troubleshoot by feature [Section titled “Troubleshoot by feature”](#troubleshoot-by-feature) * [Continuous aggregates](/docs/build/tips-and-tricks/troubleshoot-continuous-aggregates) * [Hypertables](/docs/build/tips-and-tricks/troubleshoot-hypertables) * [Hypercore](/docs/build/tips-and-tricks/troubleshoot-hypercore) * [Import and ingest](/docs/build/tips-and-tricks/troubleshoot-import-ingest) * [Queries](/docs/build/tips-and-tricks/troubleshoot-query-data) * [Schema management](/docs/build/tips-and-tricks/troubleshoot-schema-management) * [Time buckets](/docs/build/tips-and-tricks/troubleshoot-time-buckets) * [Data retention](/docs/build/tips-and-tricks/troubleshoot-data-retention) * [Data tiering](/docs/build/tips-and-tricks/troubleshoot-data-tiering) * [Jobs](/docs/build/tips-and-tricks/troubleshoot-jobs) * [Hyperfunctions](/docs/build/tips-and-tricks/troubleshoot-hyperfunctions) If you need a walkthrough instead, use [guided projects](/docs/build/examples) or [quickstarts](/docs/build/how-to/your-first-hypertable). ## Still stuck? [Section titled “Still stuck?”](#still-stuck) Browse the full [Build](/docs/build) section for more how-tos, or see [Deploy](/docs/deploy) for platform-specific runbooks. # Troubleshoot continuous aggregates > Troubleshoot common problems with continuous aggregates ## Continuous aggregate watermark is in the future Continuous aggregates use a watermark to indicate which time buckets have already been materialized. When you query a continuous aggregate, your query returns materialized data from before the watermark. It returns real-time, non-materialized data from after the watermark. In certain cases, the watermark might be in the future. If this happens, all buckets, including the most recent bucket, are materialized and below the watermark. No real-time data is returned. This might happen if you refresh your continuous aggregate over the time window `, NULL`, which materializes all recent data. It might also happen if you create a continuous aggregate using the `WITH DATA` option. This also implicitly refreshes your continuous aggregate with a window of `NULL, NULL`. To fix this, create a new continuous aggregate using the `WITH NO DATA` option. Then use a policy to refresh this continuous aggregate over an explicit time window. ### Creating a new continuous aggregate with an explicit refresh window 1. **Create the continuous aggregate with WITH NO DATA** ```sql CREATE MATERIALIZED VIEW WITH (timescaledb.continuous) AS SELECT time_bucket('', ), , ... FROM GROUP BY bucket, WITH NO DATA; ``` 2. **Add a refresh policy with an explicit `end_offset`** ```sql SELECT add_continuous_aggregate_policy('', start_offset => INTERVAL '30 day', end_offset => INTERVAL '1 hour', schedule_interval => INTERVAL '1 hour'); ``` 3. **Check the watermark** Get the ID for the materialization hypertable that contains the actual continuous aggregate data: ```sql SELECT id FROM _timescaledb_catalog.hypertable WHERE table_name=( SELECT materialization_hypertable_name FROM timescaledb_information.continuous_aggregates WHERE view_name='' ); ``` 4. **Query for the watermark timestamp** Use the returned ID to query for the watermark's timestamp: For TimescaleDB >= 2.12: ```sql SELECT COALESCE( _timescaledb_functions.to_timestamp(_timescaledb_functions.cagg_watermark()), '-infinity'::timestamp with time zone ); ``` For TimescaleDB < 2.12: ```sql SELECT COALESCE( _timescaledb_internal.to_timestamp(_timescaledb_internal.cagg_watermark()), '-infinity'::timestamp with time zone ); ``` Warning If you choose to delete your old continuous aggregate after creating a new one, beware of historical data loss. If your old continuous aggregate contained data that you dropped from your original hypertable, for example through a data retention policy, the dropped data is not included in your new continuous aggregate. ## Hierarchical continuous aggregate fails with incompatible bucket width ```sql ERROR: cannot create continuous aggregate with incompatible bucket width DETAIL: Time bucket width of "" [1 year] should be multiple of the time bucket width of "" [1 day]. ``` If you attempt to create a hierarchical continuous aggregate, you must use compatible time buckets. You can't create a continuous aggregate with a fixed-width time bucket on top of a continuous aggregate with a variable-width time bucket. For more information, see the restrictions section in [hierarchical continuous aggregates](/docs/learn/continuous-aggregates/hierarchical-continuous-aggregates#restrictions). ## Hypertable retention policy isn't applying to continuous aggregates A retention policy set on a hypertable does not apply to any continuous aggregates made from the hypertable. This allows you to set different retention periods for raw and summarized data. To apply a retention policy to a continuous aggregate, set the policy on the continuous aggregate itself. ## Continuous aggregate doesn't refresh with newly inserted historical data Materialized views are generally used with ordered data. If you insert historic data, or data that is not related to the current time, you need to refresh policies and reevaluate the values that are dragging from past to present. You can set up an after insert rule for your hypertable or upsert to trigger something that can validate what needs to be refreshed as the data is merged. Let's say you inserted ordered timeframes named A, B, D, and F, and you already have a continuous aggregation looking for this data. If you now insert E, you need to refresh E and F. However, if you insert C you need to refresh C, D, E and F. For example: 1. A, B, D, and F are already materialized in a view with all data. 2. To insert C, split the data into `AB` and `DEF` subsets. 3. `AB` are consistent and the materialized data is too; you only need to reuse it. 4. Insert C, `DEF`, and refresh policies after C. This can use a lot of resources to process, especially if you have any important data in the past that also needs to be brought to the present. Consider an example where you have 300 columns on a single hypertable and use, for example, five of them in a continuous aggregation. In this case, it could be hard to refresh and would make more sense to isolate these columns in another hypertable. Alternatively, you might create one hypertable per metric and refresh them independently. ## Permissions error when migrating a continuous aggregate [Section titled “Permissions error when migrating a continuous aggregate”](#permissions-error-when-migrating-a-continuous-aggregate) You might get a permissions error when migrating a continuous aggregate from old to new format using `cagg_migrate`. The user performing the migration must have the following permissions: * Select, insert, and update permissions on the tables `_timescale_catalog.continuous_agg_migrate_plan` and `_timescale_catalog.continuous_agg_migrate_plan_step` * Usage permissions on the sequence `_timescaledb_catalog.continuous_agg_migrate_plan_step_step_id_seq` To solve the problem, change to a user capable of granting permissions, and grant the following permissions to the user performing the migration: ```sql GRANT SELECT, INSERT, UPDATE ON TABLE _timescaledb_catalog.continuous_agg_migrate_plan TO ; GRANT SELECT, INSERT, UPDATE ON TABLE _timescaledb_catalog.continuous_agg_migrate_plan_step TO ; GRANT USAGE ON SEQUENCE _timescaledb_catalog.continuous_agg_migrate_plan_step_step_id_seq TO ; ``` ## Queries fail when defining continuous aggregates but work on regular tables Continuous aggregates do not work on all queries. For example, TimescaleDB does not support window functions on continuous aggregates. If you use an unsupported function, you see the following error: ```sql ERROR: invalid continuous aggregate view SQL state: 0A000 ``` The following table summarizes the aggregate functions supported in continuous aggregates: | Function, clause, or feature | TimescaleDB 2.6 and earlier | TimescaleDB 2.7, 2.8, and 2.9 | TimescaleDB 2.10 and later | | --------------------------------------------------------------------------------------------------------------------- | --------------------------- | ----------------------------- | -------------------------- | | Parallelizable aggregate functions | ✅ | ✅ | ✅ | | [Non-parallelizable SQL aggregates](https://www.postgresql.org/docs/current/parallel-plans.html#PARALLEL-AGGREGATION) | ❌ | ✅ | ✅ | | `ORDER BY` | ❌ | ✅ | ✅ | | Ordered-set aggregates | ❌ | ✅ | ✅ | | Hypothetical-set aggregates | ❌ | ✅ | ✅ | | `DISTINCT` in aggregate functions | ❌ | ✅ | ✅ | | `FILTER` in aggregate functions | ❌ | ✅ | ✅ | | `FROM` clause supports `JOINS` | ❌ | ❌ | ✅ | DISTINCT works in aggregate functions, not in the query definition. For example, for the table: ```sql CREATE TABLE public.candle( symbol_id uuid NOT NULL, symbol text NOT NULL, "time" timestamp with time zone NOT NULL, open double precision NOT NULL, high double precision NOT NULL, low double precision NOT NULL, close double precision NOT NULL, volume double precision NOT NULL ); ``` * The following works: ```sql CREATE MATERIALIZED VIEW candles_start_end WITH (timescaledb.continuous) AS SELECT time_bucket('1 hour', "time"), COUNT(DISTINCT symbol), first(time, time) as first_candle, last(time, time) as last_candle FROM candle GROUP BY 1; ``` * This does not: ```sql CREATE MATERIALIZED VIEW candles_start_end WITH (timescaledb.continuous) AS SELECT DISTINCT ON (symbol) symbol,symbol_id, first(time, time) as first_candle, last(time, time) as last_candle FROM candle GROUP BY symbol_id; ``` ## Updates to previously materialized regions aren't shown in real-time aggregates Real-time aggregates automatically add the most recent data when you query your continuous aggregate. In other words, they include data *more recent than* your last materialized bucket. If you add new *historical* data to an already-materialized bucket, it won't be reflected in a real-time aggregate. You should wait for the next scheduled refresh, or manually refresh by calling `refresh_continuous_aggregate`. You can think of real-time aggregates as being eventually consistent for historical data. The following example shows how this works: 1. **Create the hypertable** ```sql CREATE TABLE conditions( day DATE NOT NULL, city text NOT NULL, temperature INT NOT NULL ) WITH ( tsdb.hypertable, tsdb.partition_column='day', tsdb.chunk_interval='1 day' ); ``` For TimescaleDB [v2.23.0](https://github.com/timescale/timescaledb/releases/tag/2.23.0) and higher, the table is automatically partitioned on the first column in the table with a timestamp data type. If multiple columns are suitable candidates as a partitioning column, TimescaleDB throws an error and asks for an explicit definition. For earlier versions, set `partition_column` to a time column. If you are self-hosting TimescaleDB [v2.20.0](https://github.com/timescale/timescaledb/releases/tag/2.23.0) to [v2.22.1](https://github.com/timescale/timescaledb/releases/tag/2.23.0), to convert your data to the columnstore after a specific time interval, you have to call [add\_columnstore\_policy](/docs/reference/timescaledb/hypercore/add_columnstore_policy) after you call [CREATE TABLE](/docs/reference/timescaledb/hypertables/create_table) If you are self-hosting TimescaleDB [v2.19.3](https://github.com/timescale/timescaledb/releases/tag/2.19.3) and below, create a [PostgreSQL relational table](https://www.postgresql.org/docs/current/sql-createtable.html), then convert it using [create\_hypertable](/docs/reference/timescaledb/hypertables/create_hypertable). You then enable hypercore with a call to [ALTER TABLE](/docs/reference/timescaledb/hypercore/alter_table). 2. **Add data** ```sql INSERT INTO conditions (day, city, temperature) VALUES ('2021-06-14', 'Moscow', 26), ('2021-06-15', 'Moscow', 22), ('2021-06-16', 'Moscow', 24), ('2021-06-17', 'Moscow', 24), ('2021-06-18', 'Moscow', 27), ('2021-06-19', 'Moscow', 28), ('2021-06-20', 'Moscow', 30), ('2021-06-21', 'Moscow', 31), ('2021-06-22', 'Moscow', 34), ('2021-06-23', 'Moscow', 34), ('2021-06-24', 'Moscow', 34), ('2021-06-25', 'Moscow', 32), ('2021-06-26', 'Moscow', 32), ('2021-06-27', 'Moscow', 31); ``` 3. **Create a continuous aggregate with no materialized data** 1. Create the continuous aggregate: ```sql CREATE MATERIALIZED VIEW conditions_summary WITH (timescaledb.continuous, timescaledb.materialized_only=false) AS SELECT city, time_bucket('7 days', day) AS bucket, MIN(temperature), MAX(temperature) FROM conditions GROUP BY city, bucket WITH NO DATA; ``` 2. Check your data: ```sql SELECT * FROM conditions_summary ORDER BY bucket; ``` The query fetches data directly from the hypertable: | city | bucket | min | max | | ------ | ---------- | --- | --- | | Moscow | 2021-06-14 | 22 | 30 | | Moscow | 2021-06-21 | 31 | 34 | 4. **Materialize the data** 1. Refresh the continuous aggregate: ```sql CALL refresh_continuous_aggregate('conditions_summary', '2021-06-14', '2021-06-21'); ``` 2. Check your data: ```sql SELECT * FROM conditions_summary ORDER BY bucket; ``` The same data is returned, but this time from the underlying materialized table: | city | bucket | min | max | | ------ | ---------- | --- | --- | | Moscow | 2021-06-14 | 22 | 30 | | Moscow | 2021-06-21 | 31 | 34 | 5. **Update data in a previously materialized bucket** 1. Update the data in your hypertable: ```sql UPDATE conditions SET temperature = 35 WHERE day = '2021-06-14' and city = 'Moscow'; ``` 2. Check your data: ```sql SELECT * FROM conditions_summary ORDER BY bucket; ``` The updated data is not yet visible. This is because the changes have not been materialized: | city | bucket | min | max | | ------ | ---------- | --- | --- | | Moscow | 2021-06-14 | 22 | 30 | | Moscow | 2021-06-21 | 31 | 34 | 6. **Refresh to update the materialized region** 1. Refresh the data: ```sql CALL refresh_continuous_aggregate('conditions_summary', '2021-06-14', '2021-06-21'); ``` 2. Check your data: ```sql SELECT * FROM conditions_summary ORDER BY bucket; ``` The update is now visible: | city | bucket | min | max | | ------ | ---------- | --- | --- | | Moscow | 2021-06-14 | 22 | 35 | | Moscow | 2021-06-21 | 31 | 34 | ## Queries using `locf()` don't treat NULL values as missing When you have a query that uses a last observation carried forward (locf) function, the query carries forward NULL values by default. If you want the function to ignore NULL values instead, you can set `treat_null_as_missing=TRUE` as the second parameter in the query. For example: ```sql dev=# select * FROM (select time_bucket_gapfill(4, time,-5,13), locf(avg(v)::int,treat_null_as_missing:=true) FROM (VALUES (0,0),(8,NULL)) v(time, v) WHERE time BETWEEN 0 AND 10 GROUP BY 1) i ORDER BY 1 DESC; time_bucket_gapfill | locf ---------------------+------ 12 | 0 8 | 0 4 | 0 0 | 0 -4 | -8 | (6 rows) ``` ## Scheduled jobs stop running Your scheduled jobs might stop running for various reasons. On self-hosted TimescaleDB, you can fix this by restarting background workers: * TimescaleDB >= 2.12 ```sql SELECT _timescaledb_functions.start_background_workers(); ``` * TimescaleDB < 2.12 ```sql SELECT _timescaledb_internal.start_background_workers(); ``` On Tiger Cloud and Managed Service for TimescaleDB, restart background workers by doing one of the following: * Run `SELECT timescaledb_pre_restore()`, followed by `SELECT timescaledb_post_restore()`. * Power the service off and on again. This might cause a downtime of a few minutes while the service restores from backup and replays the write-ahead log. # Troubleshoot data retention > Find the solutions to common errors when working with data retention policies ## Hypertable retention policy isn't applying to continuous aggregates A retention policy set on a hypertable does not apply to any continuous aggregates made from the hypertable. This allows you to set different retention periods for raw and summarized data. To apply a retention policy to a continuous aggregate, set the policy on the continuous aggregate itself. ## Dropping chunks times out When you drop a chunk, it requires an exclusive lock. If a chunk is being accessed by another session, you cannot drop the chunk at the same time. If a drop chunk operation can't get the lock on the chunk, then it times out and the process fails. To resolve this problem, check what is locking the chunk. In some cases, this could be caused by a continuous aggregate or other process accessing the chunk. When the drop chunk operation can get an exclusive lock on the chunk, it completes as expected. For more information about locks, see the [PostgreSQL lock monitoring documentation](https://wiki.postgresql.org/wiki/Lock_Monitoring). ## Reindex hypertables to fix large indexes **Message:** ```plaintext ERROR: invalid attribute number -6 for _hyper_2_839_chunk CONTEXT: SQL function "hypertable_local_size" statement 1 PL/pgSQL function hypertable_detailed_size(regclass) line 26 at RETURN QUERY SQL function "hypertable_size" statement 1 SQL state: XX000 ``` You might see this error if your hypertable indexes have become very large. To resolve the problem, reindex your hypertables with this command: ```sql REINDEX TABLE _timescaledb_internal._hyper_2_1523284_chunk; ``` For more information, see the [hypertable documentation](/docs/learn/hypertables/understand-hypertables). ## Scheduled jobs stop running Your scheduled jobs might stop running for various reasons. On self-hosted TimescaleDB, you can fix this by restarting background workers: * TimescaleDB >= 2.12 ```sql SELECT _timescaledb_functions.start_background_workers(); ``` * TimescaleDB < 2.12 ```sql SELECT _timescaledb_internal.start_background_workers(); ``` On Tiger Cloud and Managed Service for TimescaleDB, restart background workers by doing one of the following: * Run `SELECT timescaledb_pre_restore()`, followed by `SELECT timescaledb_post_restore()`. * Power the service off and on again. This might cause a downtime of a few minutes while the service restores from backup and replays the write-ahead log. # Troubleshoot data tiering > Solutions to common errors when working with tiered storage ## Slow tiering of chunks Chunks are tiered asynchronously. Chunks are selected to be tiered to the object storage tier one at a time ordered by their enqueue time. To see the chunks waiting to be tiered, query the `timescaledb_osm.chunks_queued_for_tiering` view: ```sql select count(*) from timescaledb_osm.chunks_queued_for_tiering ``` Processing all the chunks in the queue may take considerable time if a large quantity of data is being migrated to the object storage tier. # Troubleshoot hypercore > Resolve common issues with hypercore, columnstore conversion, and compression ## Temporary file size limit exceeded when converting chunks to the columnstore **Message:** ```plaintext ERROR: temporary file size exceeds temp_file_limit ``` When you try to convert a chunk to the columnstore, especially if the chunk is very large, you could get this error. Compression operations write files to a new compressed chunk table, which is written in temporary memory. The maximum amount of temporary memory available is determined by the `temp_file_limit` parameter. You can work around this problem by adjusting the `temp_file_limit` and `maintenance_work_mem` parameters. ## Inefficient `compress_chunk_time_interval` configuration When you configure `compress_chunk_time_interval` but do not set the primary dimension as the first column in `orderby`, TimescaleDB converts chunks back to the rowstore before merging. This makes merging less efficient. Set the primary dimension of the chunk as the first column in `orderby` to improve efficiency. ## Tuple decompression limit exceeded by operation **Message:** ```plaintext ERROR: tuple decompression limit exceeded by operation ``` When inserting, updating, or deleting tuples from chunks in the columnstore, it might be necessary to convert tuples to the rowstore. This happens either when you are updating existing tuples or have constraints that need to be verified during insert time. If you happen to trigger a lot of rowstore conversion with a single command, you may end up running out of storage space. For this reason, a limit has been put in place on the number of tuples you can decompress into the rowstore for a single command. The limit can be increased or turned off (set to 0) like so: ```sql -- set limit to a million tuples SET timescaledb.max_tuples_decompressed_per_dml_transaction TO 1000000; -- disable limit by setting to 0 SET timescaledb.max_tuples_decompressed_per_dml_transaction TO 0; ``` ## Out of memory errors after enabling the columnstore ```plaintext Error out of memory DETAIL: Failed on request of size 16777216 in memory context "ErrorContext". ``` By default, columnstore policies move all uncompressed chunks to the columnstore. However, before converting a large backlog of chunks from the rowstore to the columnstore, best practice is to set `maxchunks_to_compress` and limit the amount of chunks to be converted. For example: ```sql SELECT alter_job(j.job_id, config => j.config || '{"maxchunks_to_compress": 10}'::jsonb) FROM timescaledb_information.jobs j WHERE j.proc_name = 'policy_compression' AND j.hypertable_name = ''; ``` When all chunks have been converted to the columnstore, reset `maxchunks_to_compress` to `0` (unlimited): ```sql SELECT alter_job(j.job_id, config => j.config || '{"maxchunks_to_compress": 0}'::jsonb) FROM timescaledb_information.jobs j WHERE j.proc_name = 'policy_compression' AND j.hypertable_name = ''; ``` ## User permissions do not allow chunks to be converted to columnstore or rowstore **Message:** ```plaintext ERROR: must be owner of hypertable "HYPERTABLE_NAME" ``` You might get this error if you attempt to compress a chunk into the columnstore, or decompress it back into rowstore with a non-privileged user account. To compress or decompress a chunk, your user account must have permissions that allow it to perform `CREATE INDEX` on the chunk. You can check the permissions of the current user with this command at the `psql` command prompt: ```sql \dn+ ``` To resolve this problem, grant your user account the appropriate privileges with this command: ```sql GRANT PRIVILEGES ON TABLE TO ; ``` For more information about the `GRANT` command, see the [PostgreSQL documentation](https://www.postgresql.org/docs/current/sql-grant.html). ## Reindex hypertables to fix large indexes **Message:** ```plaintext ERROR: invalid attribute number -6 for _hyper_2_839_chunk CONTEXT: SQL function "hypertable_local_size" statement 1 PL/pgSQL function hypertable_detailed_size(regclass) line 26 at RETURN QUERY SQL function "hypertable_size" statement 1 SQL state: XX000 ``` You might see this error if your hypertable indexes have become very large. To resolve the problem, reindex your hypertables with this command: ```sql REINDEX TABLE _timescaledb_internal._hyper_2_1523284_chunk; ``` For more information, see the [hypertable documentation](/docs/learn/hypertables/understand-hypertables). ## Low compression rate Low compression rates are often caused by [high cardinality](https://www.tigerdata.com/blog/what-is-high-cardinality) of the segment key. This means that the column you selected for grouping the rows during compression has too many unique values. This makes it impossible to group a lot of rows in a batch. To achieve better compression results, choose a segment key with lower cardinality. # Troubleshoot hyperfunctions > Solutions to common errors with hyperfunctions and TimescaleDB Toolkit ## Queries using `locf()` don't treat NULL values as missing When you have a query that uses a last observation carried forward (locf) function, the query carries forward NULL values by default. If you want the function to ignore NULL values instead, you can set `treat_null_as_missing=TRUE` as the second parameter in the query. For example: ```sql dev=# select * FROM (select time_bucket_gapfill(4, time,-5,13), locf(avg(v)::int,treat_null_as_missing:=true) FROM (VALUES (0,0),(8,NULL)) v(time, v) WHERE time BETWEEN 0 AND 10 GROUP BY 1) i ORDER BY 1 DESC; time_bucket_gapfill | locf ---------------------+------ 12 | 0 8 | 0 4 | 0 0 | 0 -4 | -8 | (6 rows) ``` # Troubleshoot hypertables > Solutions to common errors and fixes for hypertables, chunks, compression, and permissions. When something breaks around partitioning, compression, permissions, or chunk lifecycle, these fixes cover the usual failures on hypertables, chunks, and the columnstore. ## Can't create unique index on hypertable, or can't create hypertable with unique index **Message:** ```plaintext ERROR: cannot create a unique index without the column "" (used in partitioning) ``` You might get a unique index and partitioning column error in 2 situations: * When creating a primary key or unique index on a hypertable * When creating a hypertable from a table that already has a unique index or primary key For more information on how to fix this problem, see the [section on creating unique indexes on hypertables](/docs/build/performance-optimization/hypertables-and-unique-indexes). ## Reindex hypertables to fix large indexes **Message:** ```plaintext ERROR: invalid attribute number -6 for _hyper_2_839_chunk CONTEXT: SQL function "hypertable_local_size" statement 1 PL/pgSQL function hypertable_detailed_size(regclass) line 26 at RETURN QUERY SQL function "hypertable_size" statement 1 SQL state: XX000 ``` You might see this error if your hypertable indexes have become very large. To resolve the problem, reindex your hypertables with this command: ```sql REINDEX TABLE _timescaledb_internal._hyper_2_1523284_chunk; ``` For more information, see the [hypertable documentation](/docs/learn/hypertables/understand-hypertables). ## Temporary file size limit exceeded when converting chunks to the columnstore **Message:** ```plaintext ERROR: temporary file size exceeds temp_file_limit ``` When you try to convert a chunk to the columnstore, especially if the chunk is very large, you could get this error. Compression operations write files to a new compressed chunk table, which is written in temporary memory. The maximum amount of temporary memory available is determined by the `temp_file_limit` parameter. You can work around this problem by adjusting the `temp_file_limit` and `maintenance_work_mem` parameters. ## Dropping chunks times out When you drop a chunk, it requires an exclusive lock. If a chunk is being accessed by another session, you cannot drop the chunk at the same time. If a drop chunk operation can't get the lock on the chunk, then it times out and the process fails. To resolve this problem, check what is locking the chunk. In some cases, this could be caused by a continuous aggregate or other process accessing the chunk. When the drop chunk operation can get an exclusive lock on the chunk, it completes as expected. For more information about locks, see the [PostgreSQL lock monitoring documentation](https://wiki.postgresql.org/wiki/Lock_Monitoring). ## Tuple decompression limit exceeded by operation **Message:** ```plaintext ERROR: tuple decompression limit exceeded by operation ``` When inserting, updating, or deleting tuples from chunks in the columnstore, it might be necessary to convert tuples to the rowstore. This happens either when you are updating existing tuples or have constraints that need to be verified during insert time. If you happen to trigger a lot of rowstore conversion with a single command, you may end up running out of storage space. For this reason, a limit has been put in place on the number of tuples you can decompress into the rowstore for a single command. The limit can be increased or turned off (set to 0) like so: ```sql -- set limit to a million tuples SET timescaledb.max_tuples_decompressed_per_dml_transaction TO 1000000; -- disable limit by setting to 0 SET timescaledb.max_tuples_decompressed_per_dml_transaction TO 0; ``` ## User permissions do not allow chunks to be converted to columnstore or rowstore **Message:** ```plaintext ERROR: must be owner of hypertable "HYPERTABLE_NAME" ``` You might get this error if you attempt to compress a chunk into the columnstore, or decompress it back into rowstore with a non-privileged user account. To compress or decompress a chunk, your user account must have permissions that allow it to perform `CREATE INDEX` on the chunk. You can check the permissions of the current user with this command at the `psql` command prompt: ```sql \dn+ ``` To resolve this problem, grant your user account the appropriate privileges with this command: ```sql GRANT PRIVILEGES ON TABLE TO ; ``` For more information about the `GRANT` command, see the [PostgreSQL documentation](https://www.postgresql.org/docs/current/sql-grant.html). ## Low compression rate Low compression rates are often caused by [high cardinality](https://www.tigerdata.com/blog/what-is-high-cardinality) of the segment key. This means that the column you selected for grouping the rows during compression has too many unique values. This makes it impossible to group a lot of rows in a batch. To achieve better compression results, choose a segment key with lower cardinality. ## Permission denied when changing ownership of tables and hypertables **Message:** ```plaintext ERROR: permission denied for schema _timescaledb_internal ``` You might see this error when using the `ALTER TABLE` command to change the ownership of tables or hypertables. This use of `ALTER TABLE` is blocked because the `tsdbadmin` user is not a superuser. To change table ownership, use the [`REASSIGN`](https://www.postgresql.org/docs/current/sql-reassign-owned.html) command instead: ```sql REASSIGN OWNED BY TO ; ``` # Troubleshoot import and ingest > Solutions to common errors when importing data and migrating databases Imports and migrations often fail on version skew, encoding edge cases, or bulk-load limits. Start with these answers before you spend time debugging alone. Live import from a database Unsupported in live migration Live migration tooling is currently experimental. You may run into the following shortcomings: * Live migration does not yet support mutable columnstore compression (`INSERT`, `UPDATE`, `DELETE` on data in the columnstore). * By default, numeric fields containing `NaN`/`+Inf`/`-Inf` values are not correctly replicated, and will be converted to `NULL`. A workaround is available, but is not enabled by default. Should you run into any problems, please open a support request before losing any time debugging issues. You can open a support request directly from Tiger Console, or by email to . Where can I find logs for processes running during live migration? Live migration involves several background processes to manage different stages of the migration. The logs of these processes can be helpful for troubleshooting unexpected behavior. You can find these logs in the `/logs` directory. Source and target databases have different TimescaleDB versions When you migrate a self-hosted or Managed Service for TimescaleDB database to Tiger Cloud, the source database and the destination Tiger Cloud service must run the same version of TimescaleDB. Before you start live migration: 1. Check the version of TimescaleDB running on the source database and the target Tiger Cloud service: 2. If the version of TimescaleDB on the source database is lower than your Tiger Cloud service, either: * **Downgrade**: reinstall an older version of TimescaleDB on your Tiger Cloud service that matches the source database: 1. Connect to your Tiger Cloud service and check the versions of TimescaleDB available: 2. If an available TimescaleDB release matches your source database: 1. Uninstall TimescaleDB from your Tiger Cloud service: 2. Reinstall the correct version of TimescaleDB: You may need to reconnect to your Tiger Cloud service using `psql -X` when you’re creating the TimescaleDB extension. * **Upgrade**: for self-hosted databases, upgrade TimescaleDB to match your Tiger Cloud service. Why does live migration log “no tuple identifier” warning? Live migration logs a warning `WARNING: no tuple identifier for UPDATE in table` when it cannot determine which specific rows should be updated after receiving an `UPDATE` statement from the source database during replication. This occurs when tables in the source database that receive `UPDATE` statements lack either a `PRIMARY KEY` or a `REPLICA IDENTITY` setting. For live migration to successfully replicate `UPDATE` and `DELETE` statements, tables must have either a `PRIMARY KEY` or `REPLICA IDENTITY` set as a prerequisite. Set REPLICA IDENTITY on PostgreSQL partitioned tables If your PostgreSQL tables use native partitioning, setting `REPLICA IDENTITY` on the root (parent) table will not automatically apply it to the partitioned child tables. You must manually set `REPLICA IDENTITY` on each partitioned child table. Can I use read/failover replicas as source database for live migration? Live migration does not support replication from read or failover replicas. You must provide a connection string that points directly to your source database for live migration. Can I use live migration with a PostgreSQL connection pooler like PgBouncer? Live migration does not support connection poolers. You must provide a connection string that points directly to your source and target databases for live migration to work smoothly. Can I use Tiger Cloud instance as source for live migration? No, Tiger Cloud cannot be used as a source database for live migration. How can I exclude a schema/table from being replicated in live migration? At present, live migration does not allow for excluding schemas or tables from replication, but this feature is expected to be added in future releases. However, a workaround is available for skipping table data using the `--skip-table-data` flag. For more information, please refer to the help text under the `migrate` subcommand. Large migrations blocked Tiger Cloud automatically manages the underlying disk volume. Due to platform limitations, it is only possible to resize the disk once every six hours. Depending on the rate at which you’re able to copy data, you may be affected by this restriction. Affected instances are unable to accept new data and error with: `FATAL: terminating connection due to administrator command`. If you intend on migrating more than 400 GB of data to Tiger Cloud, open a support request requesting the required storage to be pre-allocated in your Tiger Cloud service. You can open a support request directly from Tiger Console, or by email to . Import from a database with downtime Dumping and locks When `pg_dump` starts, it takes an `ACCESS SHARE` lock on all tables which it dumps. This ensures that tables aren’t dropped before `pg_dump` is able to drop them. A side effect of this is that any query which tries to take an `ACCESS EXCLUSIVE` lock on a table is be blocked by the `ACCESS SHARE` lock. A number of Tiger Cloud-internal processes require taking `ACCESS EXCLUSIVE` locks to ensure consistency of the data. The following is a non-exhaustive list of potentially affected operations: * converting a chunk into the columnstore/rowstore and back * continuous aggregate refresh (before 2.12) * create hypertable with foreign keys, truncate hypertable * enable hypercore on a hypertable * drop chunks The most likely impact of the above is that background jobs for retention policies, columnstore compression policies, and continuous aggregate refresh policies are blocked for the duration of the `pg_dump` command. This may have unintended consequences for your database performance. Dumping with concurrency When using the `pg_dump` directory format, it is possible to use concurrency to use multiple connections to the source database to dump data. This speeds up the dump process. Due to the fact that there are multiple connections, it is possible for `pg_dump` to end up in a deadlock situation. When it detects a deadlock it aborts the dump. In principle, any query which takes an `ACCESS EXCLUSIVE` lock on a table causes such a deadlock. As mentioned above, some common operations which take an `ACCESS EXCLUSIVE` lock are: * retention policies * columnstore compression policies * continuous aggregate refresh policies If you would like to use concurrency nonetheless, turn off all background jobs in the source database before running `pg_dump`, and turn them on once the dump is complete. If the dump procedure takes longer than the continuous aggregate refresh policy’s window, you must manually refresh the continuous aggregate in the correct time range. For more information, consult the [refresh policies documentation](/docs/build/continuous-aggregates/refresh-policies). To turn off the jobs: To turn on the jobs: Restoring with concurrency If the directory format is used for `pg_dump` and `pg_restore`, concurrency can be employed to speed up the process. Unfortunately, loading the tables in the `timescaledb_catalog` schema concurrently causes errors. Furthermore, the `tsdbadmin` user does not have sufficient privileges to turn off triggers in this schema. To get around this limitation, load this schema serially, and then load the rest of the database concurrently. Ownership of background jobs The `_timescaledb_config` schema is an internal schema not part of the public API. For normal job management, use the [`timescaledb_information.jobs`](/docs/reference/timescaledb/informational-views/jobs) informational view and job management functions. Direct modification of internal tables should only be done in migration scenarios as described below. The `_timescaledb_config.bgw_jobs` table is used to manage background jobs. This includes custom jobs, columnstore compression policies, retention policies, and continuous aggregate refresh policies. On Tiger Cloud, this table has a trigger which ensures that no database user can create or modify jobs owned by another database user. This trigger can provide an obstacle for migrations. If the `--no-owner` flag is used with `pg_dump` and `pg_restore`, all objects in the target database are owned by the user that ran `pg_restore`, likely `tsdbadmin`. If all the background jobs in the source database were owned by a user of the same name as the user running the restore (again likely `tsdbadmin`), then loading the `_timescaledb_config.bgw_jobs` table should work. If the background jobs in the source were owned by the `postgres` user, they are be automatically changed to be owned by the `tsdbadmin` user. In this case, one just needs to verify that the jobs do not make use of privileges that the `tsdbadmin` user does not possess. If background jobs are owned by one or more users other than the user employed in restoring, then there could be issues. To work around this issue, do not dump this table with `pg_dump`. Provide either `--exclude-table-data='_timescaledb_config.bgw_job'` or `--exclude-table='_timescaledb_config.bgw_job'` to `pg_dump` to skip this table. Then, use `psql` and the `COPY` command to dump and restore this table with modified values for the `owner` column. Once the table has been loaded and the restore completed, you may then use SQL to adjust the ownership of the jobs and/or the associated stored procedures and functions as you wish. Common issues Extension availability There are a vast number of PostgreSQL extensions available in the wild. Tiger Cloud supports many of the most popular extensions, but not all extensions. Before migrating, check that the extensions you are using are supported on Tiger Cloud. Consult the list of supported extensions. TimescaleDB extension in the public schema When self-hosting, the TimescaleDB extension may be installed in an arbitrary schema. Tiger Cloud only supports installing the TimescaleDB extension in the `public` schema. How to go about resolving this depends heavily on the particular details of the source schema and the migration approach chosen. Tablespaces Tiger Cloud does not support using custom tablespaces. Providing the `--no-tablespaces` flag to `pg_dump` and `pg_restore` when dumping/restoring the schema results in all objects being in the default tablespace as desired. Only one database per instance While PostgreSQL clusters can contain many databases, Tiger Cloud services are limited to a single database. When migrating a cluster with multiple databases to Tiger Cloud, one can either migrate each source database to a separate Tiger Cloud service or “merge” source databases to target schemas. Superuser privileges The `tsdbadmin` database user is the most powerful available on Tiger Cloud, but it is not a true superuser. Review your application for use of superuser privileged operations and mitigate before migrating. Migrate partial continuous aggregates In order to improve the performance and compatibility of continuous aggregates, TimescaleDB v2.7 replaces *partial* continuous aggregates with *finalized* continuous aggregates. To test your database for partial continuous aggregates, run the following query: If you have partial continuous aggregates in your database, migrate them from partial to finalized before you migrate your database. If you accidentally migrate partial continuous aggregates across PostgreSQL versions, you see the following error when you query any continuous aggregates: # Troubleshoot jobs > Solutions to common errors when working with scheduled jobs ## Scheduled jobs stop running Your scheduled jobs might stop running for various reasons. On self-hosted TimescaleDB, you can fix this by restarting background workers: * TimescaleDB >= 2.12 ```sql SELECT _timescaledb_functions.start_background_workers(); ``` * TimescaleDB < 2.12 ```sql SELECT _timescaledb_internal.start_background_workers(); ``` On Tiger Cloud and Managed Service for TimescaleDB, restart background workers by doing one of the following: * Run `SELECT timescaledb_pre_restore()`, followed by `SELECT timescaledb_post_restore()`. * Power the service off and on again. This might cause a downtime of a few minutes while the service restores from backup and replays the write-ahead log. ## Failed to start a background worker You might see this error message in the logs if background workers aren't properly configured: ```bash "": failed to start a background worker ``` To fix this error, make sure that `max_worker_processes`, `max_parallel_workers`, and `timescaledb.max_background_workers` are properly set. `timescaledb.max_background_workers` should equal the number of databases plus the number of concurrent background workers. `max_worker_processes` should equal the sum of `timescaledb.max_background_workers` and `max_parallel_workers`. For more information, see the [configuration docs](/docs/deploy/self-hosted/configuration/about-configuration). ## Background jobs failing with "out of shared memory" You might see this error when running compression, continuous aggregate refresh, or other background jobs that touch many chunks: ```text FATAL: out of shared memory HINT: You might need to increase max_locks_per_transaction. ``` Despite the wording, this error is **not caused by insufficient RAM**. It means the PostgreSQL lock table is full. TimescaleDB acquires a lock on every chunk involved in a query or background job. When a hypertable has many chunks, these locks can exceed the default `max_locks_per_transaction` limit (usually 64). ### Diagnose the issue 1. **Check the current setting:** ```sql SELECT name, setting FROM pg_settings WHERE name = 'max_locks_per_transaction'; ``` 2. **Count chunks per hypertable:** ```sql SELECT hypertable_name, num_chunks FROM timescaledb_information.hypertables ORDER BY num_chunks DESC; ``` ### Calculate the right value For most workloads, use this formula: ```text max_locks_per_transaction = (2 × max_chunks_in_any_hypertable) / max_connections ``` The factor of 2 accounts for index locks. Round up and add headroom for future growth, because changing this parameter requires a database restart. ### Apply the fix Tips **Tiger Cloud:** Adjust `max_locks_per_transaction` from Tiger Console under **Database configuration → Advanced parameters**. Search for the parameter, edit the value, and click **Apply changes and restart**. See [Advanced parameters](/docs/deploy/tiger-cloud/tiger-cloud-aws/configuration/advanced-parameters) for details. **Self-hosted:** Set `max_locks_per_transaction` in the `postgresql.conf` configuration file, then restart PostgreSQL. For more information, see [Transaction lock settings](/docs/reference/timescaledb/configuration/tiger-postgres#transaction-lock-settings) and the [PostgreSQL lock management documentation](https://www.postgresql.org/docs/current/runtime-config-locks.html). # Troubleshoot queries > Resolve common issues with querying data in TimescaleDB ## A particular query executes more slowly than expected PostgreSQL's EXPLAIN feature allows users to understand the underlying query plan that PostgreSQL uses to execute a query. There are multiple ways that PostgreSQL can execute a query: for example, a query might be fulfilled using a slow sequence scan or a much more efficient index scan. The choice of plan depends on what indexes are created on the table, the statistics that PostgreSQL has about your data, and various planner settings. The EXPLAIN output lets you know which plan PostgreSQL is choosing for a particular query. PostgreSQL has an [in-depth explanation](https://www.postgresql.org/docs/current/using-explain.html) of this feature. To understand the query performance on a hypertable, first make sure that the planner statistics and table maintenance is up-to-date on the hypertable by running `VACUUM ANALYZE ;`. Then, run the following version of EXPLAIN: ```sql EXPLAIN (ANALYZE on, BUFFERS on) ; ``` If you suspect that your performance issues are due to slow IOs from disk, you can get even more information by enabling the [track\_io\_timing](https://www.postgresql.org/docs/current/runtime-config-statistics.html#GUC-TRACK-IO-TIMING) variable with `SET track_io_timing = 'on';` before running the above EXPLAIN. # Troubleshoot schema management > Resolve common issues with database schemas and indexes ## Reindex hypertables to fix large indexes **Message:** ```plaintext ERROR: invalid attribute number -6 for _hyper_2_839_chunk CONTEXT: SQL function "hypertable_local_size" statement 1 PL/pgSQL function hypertable_detailed_size(regclass) line 26 at RETURN QUERY SQL function "hypertable_size" statement 1 SQL state: XX000 ``` You might see this error if your hypertable indexes have become very large. To resolve the problem, reindex your hypertables with this command: ```sql REINDEX TABLE _timescaledb_internal._hyper_2_1523284_chunk; ``` For more information, see the [hypertable documentation](/docs/learn/hypertables/understand-hypertables). # Troubleshoot time buckets > Troubleshoot common problems with time buckets ## Hierarchical continuous aggregate fails with incompatible bucket width ```sql ERROR: cannot create continuous aggregate with incompatible bucket width DETAIL: Time bucket width of "" [1 year] should be multiple of the time bucket width of "" [1 day]. ``` If you attempt to create a hierarchical continuous aggregate, you must use compatible time buckets. You can't create a continuous aggregate with a fixed-width time bucket on top of a continuous aggregate with a variable-width time bucket. For more information, see the restrictions section in [hierarchical continuous aggregates](/docs/learn/continuous-aggregates/hierarchical-continuous-aggregates#restrictions). # Deploy Tiger Data products > Choose your deployment option The guides in this section show you how to **deploy and configure** Tiger Data products - Tiger Cloud on AWS and Azure, self-hosted TimescaleDB, and Managed Service for TimescaleDB. Choose the deployment option that's right for you. ## Deployment options [Section titled “Deployment options”](#deployment-options) [Tiger Cloud](/docs/deploy/tiger-cloud/tiger-cloud-aws/service-management) [Fully managed cloud service - fastest way to get started](/docs/deploy/tiger-cloud/tiger-cloud-aws/service-management) [Self-hosted TimescaleDB](/docs/deploy/self-hosted) [Run TimescaleDB on your own infrastructure](/docs/deploy/self-hosted) [Managed Service (MST)](/docs/deploy/mst) [Managed Service for TimescaleDB on Aiven](/docs/deploy/mst) ## Comparison [Section titled “Comparison”](#comparison) | Feature | Tiger Cloud | Self-hosted | MST | | ----------- | ------------- | ------------ | --------- | | Management | Fully managed | Self-managed | Managed | | Setup time | Minutes | Hours | Minutes | | Scaling | Automatic | Manual | Automatic | | Maintenance | Included | DIY | Included | Tips **Not sure which to choose?** Start with Tiger Cloud for the fastest path to production. You can always migrate later. ## Quick links [Section titled “Quick links”](#quick-links) * [Tiger Cloud quickstart](/docs/get-started/quickstart/quickstart-5-minutes) * [Self-hosted installation](/docs/get-started/choose-your-path/install-timescaledb) * [MST overview](/docs/deploy/mst) # Tiger Cloud > Robust elastic cloud platform for startups and enterprise [Begin using Tiger Cloud in minutes. Create your first service, run queries from the console, connect from your applications, and deploy your production workloads.](/docs/get-started/quickstart/create-service) [Learn how Tiger Cloud works. Explore the architecture, understand the implementation workflow, review pricing and account management, and stay updated with the latest changes.](/docs/learn/tiger-cloud) [Optimize storage costs with tiered storage. Manage hot and cold data tiers, query tiered data efficiently, work with replicas and forks, and connect to external data sources using foreign data wrappers.](/docs/build/data-management/storage/manage-storage) [Build AI-powered applications with vector search. Add RAG (Retrieval Augmented Generation) capabilities to your service and speed up similarity search with advanced indexing.](https://github.com/timescale/pgai/blob/main/docs/vectorizer/overview.md) [Keep track of your database performance. Monitor from the console, export metrics to CloudWatch, Datadog, or Prometheus, configure alerts, and understand telemetry data.](/docs/deploy/tiger-cloud/tiger-cloud-aws/monitoring) [Protect your data with enterprise-grade security features. Configure high availability, enable read scaling, set up rapid recovery, manage backups, and implement point-in-time restore.](/docs/deploy/tiger-cloud/tiger-cloud-aws/high-availability/high-availability) [Control who can access your services and how. Configure role-based access, set up user authentication, connect securely from any cloud, use stricter SSL, enable VPC peering and AWS PrivateLink, and set up IP allow lists.](/docs/deploy/tiger-cloud/tiger-cloud-aws/security/members) [Optimize your service for peak performance. Tune database parameters, adjust memory settings, and configure your service for your specific workload requirements.](/docs/deploy/tiger-cloud/tiger-cloud-aws/configuration/customize-configuration) [Resolve issues quickly with our troubleshooting guides. Find solutions to common problems, diagnose performance issues, and get help with configuration challenges.](/docs/deploy/tiger-cloud/troubleshoot) [Access comprehensive reference information. Explore available regions, learn how to contribute, discover available PostgreSQL extensions, and review configuration parameters.](/docs/learn/tiger-cloud/regions) # Limitations > Current limitations of Tiger Data products While Tiger Data generally offers capabilities that go beyond what PostgreSQL offers, there are some limitations. ## Hypertable limitations * Time dimensions (columns) used for partitioning cannot have NULL values. * Unique indexes must include all columns that are partitioning dimensions. * `UPDATE` statements that move values between partitions (chunks) are not supported. This includes upserts (`INSERT ... ON CONFLICT UPDATE`). * Foreign key constraints from a hypertable referencing another hypertable are not supported. * PostgreSQL [table inheritance](https://www.postgresql.org/docs/current/ddl-inherit.html) is not supported for hypertables and may lead to unexpected behavior. # Managed Service for TimescaleDB > Install, configure, and manage TimescaleDB managed by Aiven ## Getting started [Section titled “Getting started”](#getting-started) [About MST](/docs/deploy/mst/about-mst) [Learn about Managed Service for TimescaleDB](/docs/deploy/mst/about-mst) [Ingest data](/docs/deploy/mst/ingest-data) [Load data into your MST service](/docs/deploy/mst/ingest-data) [Migrate to MST](/docs/deploy/mst/migrate-to-mst) [Migrate from self-hosted TimescaleDB](/docs/deploy/mst/migrate-to-mst) [Create an MST service](/docs/deploy/mst/create-mst-service) [Set up your first MST service](/docs/deploy/mst/create-mst-service) ## Administration [Section titled “Administration”](#administration) [User management](/docs/deploy/mst/user-management) [Manage users and permissions](/docs/deploy/mst/user-management) [Connection pools](/docs/deploy/mst/connection-pools) [Configure connection pooling](/docs/deploy/mst/connection-pools) [Backups](/docs/deploy/mst/manage-backups) [Manage backups and restore data](/docs/deploy/mst/manage-backups) [Maintenance](/docs/deploy/mst/maintenance) [Configure maintenance windows](/docs/deploy/mst/maintenance) [Read replicas](/docs/deploy/mst/postgresql-read-replica) [Create a read-only replica](/docs/deploy/mst/postgresql-read-replica) [Failover](/docs/deploy/mst/failover) [Understand high availability and failover](/docs/deploy/mst/failover) [Service logs](/docs/deploy/mst/viewing-service-logs) [View and manage service logs](/docs/deploy/mst/viewing-service-logs) ## Security and networking [Section titled “Security and networking”](#security-and-networking) [Security](/docs/deploy/mst/security) [Configure security and IP filters](/docs/deploy/mst/security) [VPC peering](/docs/deploy/mst/vpc-peering) [Set up private network connections](/docs/deploy/mst/vpc-peering) ## Extensions and tools [Section titled “Extensions and tools”](#extensions-and-tools) [Supported extensions](/docs/deploy/mst/extensions) [Available PostgreSQL extensions](/docs/deploy/mst/extensions) [dblink extension](/docs/deploy/mst/dblink-extension) [Use the PostgreSQL dblink extension](/docs/deploy/mst/dblink-extension) [Integrations](/docs/deploy/mst/integrations) [Connect with Grafana, Datadog, and more](/docs/deploy/mst/integrations) [Aiven client](/docs/deploy/mst/aiven-client) [Manage services from the command line](/docs/deploy/mst/aiven-client) [REST API](/docs/deploy/mst/restapi) [Manage services using the Aiven API](/docs/deploy/mst/restapi) ## Troubleshooting and maintenance [Section titled “Troubleshooting and maintenance”](#troubleshooting-and-maintenance) [Index issues](/docs/deploy/mst/identify-index-issues) [Identify and repair PostgreSQL indexes](/docs/deploy/mst/identify-index-issues) [Troubleshooting](/docs/deploy/mst/troubleshooting) [Resolve common MST issues](/docs/deploy/mst/troubleshooting) # About MST > Set up and manage your services in Managed Service for TimescaleDB [Managed Service for TimescaleDB (MST)](https://www.tigerdata.com/mst-signup) is hosted [TimescaleDB](https://github.com/timescale/timescaledb) offered in partnership with Aiven. Note Tiger Cloud is a high-performance, developer-focused cloud that provides PostgreSQL services enhanced with our blazing fast vector search, data tiering, monitoring suite, and other features. [Create a Tiger Cloud service](/docs/get-started/quickstart/create-service) and try for free. Your Managed Service for TimescaleDB account has three main components: projects, MST services, and databases. ## Projects [Section titled “Projects”](#projects) When you [sign up for Managed Service for TimescaleDB](https://www.tigerdata.com/mst-signup), an empty project is created for you automatically. Projects are the highest organization level, and they contain all your services and databases. You can use projects to organize groups of MST services. Each project can also have its own billing settings. To create a new project: In [MST Console](https://portal.managed.timescale.com/login), click `Projects` > `Create project`. ![Creating a project in MST portal](/docs/_astro/create-project.wHxd2v5l_Z2fWDdy.webp) ## Managed Service for TimescaleDB services [Section titled “Managed Service for TimescaleDB services”](#managed-service-for-timescaledb-services) Each project contains one or more MST services. You can have multiple MST services under each project, and each MST service corresponds to a cloud service provider tier. You can access all your MST services from the `Services` tab within your projects. ![MST services list](/docs/_astro/services.BVeCL2CX_ZbUMnm.webp) For more information about getting your first MST service up and running, see the [Managed Service for TimescaleDB installation section](/docs/deploy/mst/create-mst-service). Warning When you have created, and named, a new Managed Service for TimescaleDB service, you cannot rename it. If you need to have your service running under a different name, you need to create a new service, and manually migrate the data. For more information about migrating data, see [migrating your data](/docs/deploy/mst/migrate-to-mst). For information about billing on Managed Service for TimescaleDB, see the [billing section](/docs/deploy/mst/billing). ## Databases [Section titled “Databases”](#databases) Each service can contain one or more databases. To view existing databases, or to create a new database, select a service in the services list, click `Databases`, then click `Create database`. ![Creating a database in MST portal](/docs/_astro/create-database.BkwX7Jm7_ZF86va.webp) ## Service level agreement [Section titled “Service level agreement”](#service-level-agreement) Managed Service for TimescaleDB is provided through a partnership with Aiven. This provides you with a service commitment to deliver 99.99% availability. For more information, see the [Aiven Service Level Agreement policy](https://aiven.io/sla). ## Service configuration plans [Section titled “Service configuration plans”](#service-configuration-plans) When you create a new service, you need to select a configuration plan. The plan determines the number of VMs the service runs in, the high availability configuration, the number of CPU cores, and size of RAM and storage volumes. The plans are: * Basic Plans: include 2 days of backups and automatic backup and restore if your instance fails. * Dev Plans: include 1 day of backups and automatic backup and restore if your instance fails. * Pro Plans: include 3 days of backups and automatic failover to a hot standby if your instance fails. The Basic and Dev plans are serviced by a single virtual machine (VM) node. This means that if the node fails, the service is unavailable until a new VM is built. This can result in data loss, if some of the latest changes to the data weren't backed up before the failure. Sometimes, it can also take a long time to return the service back to normal operation, because a new VM needs to be created and restored from backups before the service can resume. The time to recover depends on the amount of data you have to restore. The Pro plans are much more resilient to failures. A single node failure causes no data loss, and the possible downtime is minimal. If an acting TimescaleDB master node fails, an up-to-date replica node is automatically promoted to become the new master. This means there is only a small outage while applications reconnect to the database and access the new master. You can upgrade your plan while the service is running. The service is reconfigured to run on larger VMs in the background and when the reconfiguration is complete, the DNS names are pointed to the new hosts. This can cause a short disruption to your service while DNS changes are propagated. Within each configuration plan option, there are several plan types available: * `IO-Optimized` and `Compute-Optimized` These configurations are optimized for input/output (I/O) performance, using SSD storage media. * `Storage-Optimized`: These configurations usually have larger amounts of overall storage, using HDD storage media. * `Dev-Only`: These configurations are typically smaller footprints, and lower cost, designed for development and testing scenarios. ![Selecting an MST service configuration plan](/docs/_astro/service-plans.Cs7X531X_Z2sVbPg.webp) ## High availability [Section titled “High availability”](#high-availability) Most minor failures are handled automatically without making any changes to your service deployment. This includes failures such as service process crashes, or a temporary loss of network access. The service automatically restores normal operation when the crashed process restarts automatically or when the network access is restored. However, more severe failure modes, such as losing a single node entirely, require more drastic recovery measures. Losing an entire node or a virtual machine could happen for example due to hardware failure or a severe software failure. A failing node is automatically detected by the MST monitoring infrastructure. Either the node starts reporting that its own self-diagnostics is reporting problems or the node stops communicating entirely. The monitoring infrastructure automatically schedules a new replacement node to be created when this happens. Note In case of database failover, the service URL of your service remains the same. Only the IP address changes to point at the new master node. Managed Service for TimescaleDB availability features differ based on the service plan: * Basic and Dev plans: These are single-node plans. Basic plans include a two-day backup history, and Dev plans include a one-day backup history. * Pro plans: These are two-node plans with a master and a standby for higher availability, and three-day backup histories. ### Single node [Section titled “Single node”](#single-node) In the Basic and Dev plans, if you lose the only node from the service, it immediately starts the automatic process of creating a new replacement node. The new node starts up, restores its state from the latest available backup, and resumes the service. Because there was just a single node providing the service, the service is unavailable for the duration of the restore operation. Also, any writes made since the backup of the latest write-ahead log (WAL) file is lost. Typically this time window is limited to either five minutes, or one WAL file. ### Highly available nodes [Section titled “Highly available nodes”](#highly-available-nodes) In Pro plans, if a PostgreSQL standby fails, the master node keeps running normally and provides normal service level to the client applications. When the new replacement standby node is ready and synchronized with the master, it starts replicating the master in real time and normal operation resumes. If the PostgreSQL master fails, the combined information from the MST monitoring infrastructure and the standby node is used to make a failover decision. On the nodes, the open source monitoring daemon `PGLookout`, in combination with the information from the MST system infrastructure, reports the failover. If the master node is down completely, the standby node promotes itself as the new master node and immediately starts serving clients. A new replacement node is automatically scheduled and becomes the new standby node. If both master and standby nodes fail at the same time, two new nodes are automatically scheduled for creation and become the new master and standby nodes respectively. The master node restores itself from the latest available backup, which means that there can be some degree of data loss involved. For example, any writes made since the backup of the latest write-ahead log (WAL) file can be lost. The amount of time it takes to replace a failed node depends mainly on the cloud region and the amount of data that needs to be restored. However, in the case of services with two-node Pro plans, the surviving node keeps serving clients even during the recreation of the other node. This process is entirely automatic and requires no manual intervention. For backups and restoration, Managed Service for TimescaleDB uses the open source backup daemon `PGHoard` that MST maintains. It makes real-time copies of write-ahead log (WAL) files to an object store in a compressed and encrypted format. ## Connection limits [Section titled “Connection limits”](#connection-limits) Managed Service for TimescaleDB limits the maximum number of connections to each service. The maximum number of allowed connections depends on your service plan. To see the current connection limit for your service, navigate to the service `Overview` tab and locate the `Connection Limit` section. If you have a lot of clients or client threads connecting to your database, use connection pooling to limit the number of connections. For more information about connection pooling, see the [connection pooling section](/docs/deploy/mst/connection-pools). Note If you have a high number of connections to your database, your service might run more slowly, and could run out of memory. Remain aware of how many open connections your have to your database at any given time. ## Service termination protection [Section titled “Service termination protection”](#service-termination-protection) You can protect your services from accidentally being terminated, by enabling service termination protection. When termination protection is enabled, you cannot power down the service from the web console, the REST API, or with a command-line client. To power down a protected service, you need to turn off termination protection first. Termination protection does not interrupt service migrations or upgrades. To enable service termination protection, navigate to the service `Overview` tab. Locate the `Termination protection` section, and toggle to enable protection. Warning If you run out of free sign-up credit, and have not entered a valid credit card for payment, your service is powered down, even if you have enabled termination protection. ## Idle connections [Section titled “Idle connections”](#idle-connections) Managed Service for TimescaleDB uses the default keep alive settings for TCP connections. The default settings are: * `tcp_keepalives_idle`: 7200 * `tcp_keepalive_count`: 9 * `tcp_keepalives_interval`: 75 If you have long idle database connection sessions, you might need to adjust these settings to ensure that your TCP connection remains stable. If you experience a broken TCP connection, when you reconnect make sure that your client resolves the DNS address correctly, as the underlying address changes during automatic failover. For more information about adjusting keep alive settings, see the [PostgreSQL documentation](https://www.postgresql.org/docs/9.5/libpq-connect.html#LIBPQ-KEEPALIVES). ## Long running queries [Section titled “Long running queries”](#long-running-queries) Managed Service for TimescaleDB does not cancel database queries. If you have created a query that is taking a very long time, or that has hung, it could lock resources on your service, and could prevent database administration tasks from being performed. You can find out if you have any long-running queries by navigating to the service `Current Queries` tab. You can also cancel long running queries from this tab. Alternatively, you can use your connection client to view running queries with this command: ```sql SELECT * FROM pg_stat_activity WHERE state <> 'idle'; ``` Cancel long-running queries using this command, with the PID of the query you want to cancel: ```sql SELECT pg_terminate_backend(); ``` If you want to automatically cancel any query that runs over a specified length of time, you can use this command: ```sql SET statement_timeout = ``` # Aiven Client > Manage your services in the Managed Service for TimescaleDB with a command-line tool You can use Aiven Client to manage your MST services in Managed Service for TimescaleDB. You can use the Aiven Client tool to: * Connect to Managed Service for TimescaleDB * Create a MST service * Create a fork * Add authentication plugins to your attached Grafana service Instructions: * [Install and configure the Aiven client](#install-and-configure-the-aiven-client) * [Fork services with Aiven client](#fork-services-with-aiven-client) * [Configure Grafana authentication plugins](#configure-grafana-authentication-plugins) * [Send Grafana emails](#send-grafana-emails) * [Create a read-only replica with the Aiven client](#create-a-read-only-replica-with-aiven-client) ## Install and configure the Aiven client [Section titled “Install and configure the Aiven client”](#install-and-configure-the-aiven-client) Aiven Client is a command line tool for fully managed services. To use Aiven Client, you first need to create an authentication token. Then, you configure the client to connect to your Managed Service for TimescaleDB using the command line. ### Create an authentication token in Managed Service for TimescaleDB [Section titled “Create an authentication token in Managed Service for TimescaleDB”](#create-an-authentication-token-in-managed-service-for-timescaledb) To connect to Managed Service for TimescaleDB using Aiven Client, create an authentication token. 1. **Open user information** In [Managed Service for TimescaleDB](https://portal.managed.timescale.com/login), click `User Information` in the top right corner. 2. **Generate a token** In the `User Profile` page, navigate to the `Authentication` tab. Click `Generate Token`. 3. **Save the token** In the `Generate access token` dialog, type a descriptive name for the token. Leave the rest of the fields blank. Copy the generated authentication token and save it. ### Install the Aiven Client [Section titled “Install the Aiven Client”](#install-the-aiven-client) The [Aiven Client](https://github.com/aiven/aiven-client) is provided as a Python package. If you've already installed Python, you can install the client on Linux, MacOS, or Windows systems using `pip`: ```bash pip install aiven-client ``` For more information about installing the Aiven Client, see the [Aiven](https://github.com/aiven/aiven-client) documentation. ### Configure Aiven Client to connect to Managed Service for TimescaleDB [Section titled “Configure Aiven Client to connect to Managed Service for TimescaleDB”](#configure-aiven-client-to-connect-to-managed-service-for-timescaledb) To access Managed Service for TimescaleDB with the Aiven Client, you need an authentication token. Aiven Client uses this to access your MST services on Managed Service for TimescaleDB. 1. **Open the configuration directory** Change to the install directory that contains the configuration files: ```bash cd ~/.config/aiven/ ``` 2. **Update the credentials file** Open the `aiven-credentials.json` using any editor and update these lines with your Managed Service for TimescaleDB `User email`, and the `authentication token` that you generated: ```bash { "auth_token": "ABC1+123...TOKEN==", "user_email": "your.email@timescale.com" } ``` Save the `aiven-credentials.json` file. 3. **Verify access** To verify that you can access your services on Managed Service for TimescaleDB, type: ```bash avn project list ``` This command shows a list of all your projects: ```bash PROJECT_NAME DEFAULT_CLOUD CREDIT_CARD ============= ======================= =================== project-xxxx timescale-aws-us-east-1 xxxx-xxxx-xxxx-xxxx project-yyyy timescale-aws-us-east-1 xxxx-xxxx-xxxx-xxxx project-zzzz timescale-aws-us-east-1 xxxx-xxxx-xxxx-xxxx ``` ## Fork services with Aiven client [Section titled “Fork services with Aiven client”](#fork-services-with-aiven-client) When you fork a MST service, you create an exact copy of the MST service, including the underlying database. You can use a fork of your MST service to: * Create a development copy of your production environment. * Set up a snapshot to analyze an issue or test an upgrade. * Create an instance in a different cloud, geographical location, or under a different plan. For more information about projects, plans, and other details about services, see [About Managed Service for TimescaleDB](/docs/deploy/mst/about-mst). 1. **Connect to your service** In the Aiven client, connect to your Managed Service for TimescaleDB service. 2. **Switch to the project** Switch to the project that contains the MST service you want to fork: ```bash avn project switch ``` 3. **List the services** List the MST services in the project, and make a note of the MST service that you want to fork, listed under `SERVICE_NAME` column in the output. ```bash avn service list ``` 4. **Get the service details** Get the details of the MST service that you want to fork: ```bash avn service get ``` 5. **Create the fork** Create the fork: ```bash avn service create --project \ -t --plan --cloud \ -c service_to_fork_from= ``` ### Example [Section titled “Example”](#example) To create a fork named `grafana-fork` for a MST service named `grafana` with these parameters: * PROJECT\_ID: `project-fork` * CLOUD\_NAME: `timescale-aws-us-east-1` * PLAN\_TYPE: `dashboard-1` ```bash avn service create grafana-fork --project project-fork -t grafana --plan dashboard-1 --cloud timescale-aws-us-east-1 -c service_to_fork_from=grafana ``` You can switch to `project-fork` and view the newly created `grafana-fork` using: ```bash avn service list ``` ## Configure Grafana authentication plugins [Section titled “Configure Grafana authentication plugins”](#configure-grafana-authentication-plugins) Grafana supports multiple authentication plugins, in addition to built-in username and password authentication. On Managed Service for TimescaleDB, Grafana supports Google, GitHub, and GitLab authentication. You can configure authentication integration using the Aiven command-line client. ### Integrating the Google authentication plugin [Section titled “Integrating the Google authentication plugin”](#integrating-the-google-authentication-plugin) To integrate Google authentication with Grafana service on Managed Service for TimescaleDB, you need to create your [Google OAuth keys](https://grafana.com/docs/grafana/v9.0/setup-grafana/configure-security/configure-authentication/google/). Copy your client ID and client secret to a secure location. 1. **Connect and switch to the project** In the Aiven Client, connect to your Managed Service for TimescaleDB service. Switch to the project that contains the Grafana service you want to integrate: ```bash avn switch ``` 2. **Find the Grafana service** List the services in the project. Make a note of the Grafana service that you want to integrate, listed under `SERVICE_NAME` column in the output. ```bash avn service list ``` Get the details of the service that you want to integrate: ```bash avn service get ``` 3. **Integrate the plugin** Integrate the plugin with your services using the `` and `` from your Google developer console: ```bash avn service update -c auth_google.allowed_domains=\ -c auth_google.client_id=\ -c auth_google.client_secret= ``` 4. **Verify the plugin** Log in to Grafana with your service credentials. Navigate to `Configuration` → `Plugins` and verify that the Google OAuth application is listed as a plugin. Note When you allow sign-ups using the `-c auth_google.allow_sign_up=true` option, by default each new user is created with `viewer` permissions and added to their own newly created organizations. To specify different permissions, use `-c user_auto_assign_org_role=ROLE_NAME`. To add all new users to the main organization, use the `-c user_auto_assign_org=true` option. ### Integrating the GitHub authentication plugin [Section titled “Integrating the GitHub authentication plugin”](#integrating-the-github-authentication-plugin) To integrate GitHub authentication with Grafana service on Managed Service for TimescaleDB, you need to create your [GitHub OAuth application](https://grafana.com/docs/grafana/v9.0/setup-grafana/configure-security/configure-authentication/github/). Store your client ID and client secret in a secure location. 1. **Connect and switch to the project** In the Aiven Client, connect to your Managed Service for TimescaleDB service. Switch to the project that contains the Grafana service you want to integrate: ```bash avn switch ``` 2. **Find the Grafana service** List the services in the project, and make a note of the Grafana service that you want to integrate, listed under `SERVICE_NAME` column in the output. ```bash avn service list ``` Get the details of the service that you want to integrate: ```bash avn service get ``` 3. **Integrate the plugin** Integrate the plugin with your service using the ``, and `` from your GitHub OAuth application: ```bash avn service update -c auth_github.client_id=\ -c auth_github.client_secret= ``` 4. **Verify the plugin** Log in to Grafana with your service credentials. Navigate to `Configuration` → `Plugins`. The Plugins page lists GitHub OAuth application for the Grafana instance. Note When you allow sign-ups using the `-c auth_github.allow_sign_up=true` option, by default each new user is created with `viewer` permission and added to their own newly created organizations. To specify different permissions, use `-c user_auto_assign_org_role=ROLE_NAME`. To add all new users to the main organization, use the `-c user_auto_assign_org=true` option. ### Integrating the GitLab authentication plugin [Section titled “Integrating the GitLab authentication plugin”](#integrating-the-gitlab-authentication-plugin) To integrate the GitLab authentication with Grafana service on Managed Service for TimescaleDB, you need to create your [GitLab OAuth application](https://grafana.com/docs/grafana/v9.0/setup-grafana/configure-security/configure-authentication/gitlab/). Copy your client ID, client secret, and GitLab groups name to a secure location. If you use your own instance of GitLab instead of gitlab.com, then you need to set the following: * `auth_gitlab.api_url` * `auth_github.auth_url` * `auth_github.token_url` 1. **Connect and switch to the project** In the Aiven Client, connect to your Managed Service for TimescaleDB service. Switch to the project that contains the Grafana service you want to integrate: ```bash avn project switch ``` 2. **Find the Grafana service** List the services in the project. Note the Grafana service that you want to integrate, listed under `SERVICE_NAME` column in the output. ```bash avn service list ``` Get the details of the service that you want to integrate: ```bash avn service get ``` 3. **Integrate the plugin** Integrate the plugin with your service using the ``, ``, and `` from your GitLab OAuth application: ```bash avn service update -c auth_gitlab.client_id=\ -c auth_gitlab.client_secret=\ -c auth_gitlab.allowed_groups= ``` 4. **Verify the plugin** Log in to Grafana with your service credentials. Navigate to `Configuration` → `Plugins`. The Plugins page lists GitLab OAuth application for the Grafana instance. Note When you allow sign-ups using the `-c auth_gitlab.allow_sign_up=true` option, by default each new user is created with `viewer` permission and added to their own newly created organizations. To specify different permissions, use `-c user_auto_assign_org_role=ROLE_NAME`. To add all new users to the main organization, use the `-c user_auto_assign_org=true` option. ## Send Grafana emails [Section titled “Send Grafana emails”](#send-grafana-emails) Use the Aiven client to configure the Simple Mail Transfer Protocol (SMTP) server settings and send emails from Managed Service for TimescaleDB for Grafana. This includes invite emails, reset password emails, and alert messages. ### Before you begin [Section titled “Before you begin”](#before-you-begin) * (Optional): Made a note of these values in the SMTP server: `IP or hostname`, `SMTP server port`, `Username`, `Password`, `Sender email address`, and `Sender name`. 1. **Connect and switch to the project** In the Aiven client, connect to your Managed Service for TimescaleDB service. Switch to the project that contains the Grafana service you want to integrate: ```bash avn project switch ``` 2. **Find the Grafana service** List the services in the project. Note the Grafana service that you want to configure, listed under `SERVICE_NAME` column in the output. ```bash avn service list ``` Get the details of the service that you want to integrate: ```bash avn service get ``` 3. **Configure the SMTP settings** Configure the Grafana service using the SMTP values: ```bash avn service update --project \ -c smtp_server.host=smtp.example.com \ -c smtp_server.port=465 \ -c smtp_server.username=emailsenderuser \ -c smtp_server.password=emailsenderpass \ -c smtp_server.from_address="grafana@yourcompany.com" ``` 4. **Review custom options (optional)** Review all available custom options, and configure: ```bash avn service types -v ``` You can now send emails for your Grafana service on MST. ## Create a read-only replica with Aiven client [Section titled “Create a read-only replica with Aiven client”](#create-a-read-only-replica-with-aiven-client) Read-only replicas enable you to perform read-only queries against the replica and reduce the load on the primary server. They are also a good way to optimize query response times across different geographical locations. You can achieve this by placing the replicas in different regions or even different cloud providers. 1. **Connect and switch to the project** In the Aiven client, connect to your Managed Service for TimescaleDB service. Switch to the project that contains the Managed Service for TimescaleDB service you want to create a read-only replica for: ```bash avn project switch ``` 2. **Find the service** List the MST services in the project. Note the service for which you will create a read-only replica. You can find it listed under the `SERVICE_NAME` column in the output: ```bash avn service list ``` Get the details of the MST service that you want to fork: ```bash avn service get ``` 3. **Create a read-only replica** Create a read-only replica: ```bash avn service create --project \ -t pg --plan --cloud timescale-aws-us-east-1\ -c pg_read_replica=true\ -c service_to_fork_from=\ -c pg_version=11 -c variant=timescale ``` ### Example [Section titled “Example”](#example-1) To create a fork named `replica-fork` for a MST service named `timescaledb` with these parameters: * PROJECT\_ID: `fork-project` * CLOUD\_NAME: `timescale-aws-us-east-1` * PLAN\_TYPE: `timescale-basic-100-compute-optimized` ```bash avn service create replica-fork --project fork-project\ -t pg --plan timescale-basic-100-compute-optimized\ --cloud timescale-aws-us-east-1 -c pg_read_replica=true\ -c service_to_fork_from=timescaledb -c\ pg_version=11 -c variant=timescale ``` You can switch to `project-fork` and view the newly created `replica-fork` using: ```bash avn service list ``` # Billing > Billing and account management in Managed Service for TimescaleDB By default, all new Managed Service for TimescaleDB services require a credit card, which is charged at the end of the month for all charges accrued over that month. Each project is charged separately. Your credit card statement records the transaction as coming from Aiven, as Aiven provides billing services for Managed Service for TimescaleDB. Managed Service for TimescaleDB uses hourly billing. This charge is automatically calculated, based on the MST services you are running in your project. The price charged for your project includes: * Virtual machine * Networking * Backups * Setting up Note Managed Service for TimescaleDB does not charge you for network traffic used by your service. However, your application cloud service provider might charge you for the network traffic going to or from your service. Terminating or powering a MST service down stops the accumulation of new charges immediately. However, the minimum hourly charge unit is one hour. For example, if you launch a MST service and shut it down after 40 minutes, you are charged for one full hour. Migrating to different service plan levels does not incur extra charges for the migration itself. Note, though, that some service plan levels are more costly per hour, and your new MST service is charged at the new rate. Migrating a MST service to another cloud region or different cloud provider does not incur extra charges. Note All prices listed for Managed Service for TimescaleDB are inclusive of credit card and processing fees. However, in some cases, your credit card provider might charge additional fees, such as an international transaction fee. These fees are not charged by Tiger Data or Aiven. ## Billing groups [Section titled “Billing groups”](#billing-groups) Create billing groups to set up common billing profiles for projects within an organization. Billing groups make it easier to manage your costs since you receive a consolidated invoice for all projects assigned to a billing group and can pay with one saved payment method. Billing groups can only be used in one organization. Credits are assigned per billing group and are automatically used to cover charges of any project assigned to that group. You can track spending by exporting cost information to business intelligence tools using the [invoice API](https://api.aiven.io/doc/#tag/BillingGroup). To access billing groups in [MST Console](https://portal.managed.timescale.com), you must be a super admin or account owner. ### Create a billing group [Section titled “Create a billing group”](#create-a-billing-group) 1. **Open billing groups** In [MST Console](https://portal.managed.timescale.com), click **Billing** > **Billing groups** > **Create billing group**. 2. **Name the billing group** Enter a name for the billing group and click **Continue**. 3. **Enter billing details** Enter the billing details. You can copy these details from another billing group by selecting it from the list. Click **Continue**. 4. **Add projects** Select the projects to add to this billing group and click **Continue**. You can skip this step and add projects later. 5. **Review and confirm** Check the information in the **Summary** step. To make changes to any section, click **Edit**. When you have confirmed everything is correct, click **Create & Assign**. ### Manage billing groups [Section titled “Manage billing groups”](#manage-billing-groups) To view and update your billing groups, take the following steps: * Rename billing groups: 1. **Find the billing group** In [MST Console](https://portal.managed.timescale.com), go to **Billing** > **Billing groups** and find the billing group to rename. 2. **Rename** Click **Actions > Rename**. Enter the new name and click **Rename**. * Update your billing information: 1. **Open the billing group** In [MST Console](https://portal.managed.timescale.com), go to **Billing** > **Billing groups** and click on the name of the group to update. 2. **Edit billing information** Open the **Billing information** tab and click **Edit** to update the details for each section. * Delete billing groups: 1. **Select the billing group** In [MST Console](https://portal.managed.timescale.com), open **Billing** > **Billing groups** and select the group to delete. 2. **Remove projects** On the **Projects** tab, confirm that the billing group has no projects. If there are projects listed, move them to a different billing group. 3. **Delete** Go back to the list of billing groups and click **Actions** > **Delete** next to the group to be deleted. ### Assign and unassign projects [Section titled “Assign and unassign projects”](#assign-and-unassign-projects) To manage projects in billing groups, take the following steps. * Assign projects to a billing group: 1. **Open billing groups** In [MST Console](https://portal.managed.timescale.com), go to **Billing > Billing groups**. 2. **Select the billing group** Select the billing group to assign the project to. 3. **Assign projects** On the **Projects** tab, click **Assign projects**. Select the projects and click **Assign projects**. Click **Cancel** to close the dialog box. Note Assigning a project that is already assigned to another billing group will unassign it from that billing group. * Move a project to another billing group: 1. **Open billing groups** In [MST Console](https://portal.managed.timescale.com), go to **Billing > Billing groups**. 2. **Find the project** Click on the name of the billing group that the project is currently assigned to. On the **Projects** tab, find the project to move. 3. **Move the project** Click the three dots for that project and select the billing group to move it to. ## Taxation [Section titled “Taxation”](#taxation) Aiven provides billing services for Managed Service for TimescaleDB. These services are provided by Aiven Ltd, a private limited company incorporated in Finland. If you are within the European Union, Finnish law requires that you are charged a value-added tax (VAT). The VAT percentage depends on where you are domiciled. For business customers in EU countries other than Finland, you can use the reverse charge mechanism of 2006/112/EC article 196, by entering a valid VAT ID into the billing information of your project. If you are within the United States, no tax is withheld from your payments. In most cases, you do not require a W-8 form to confirm this, however, if you require a `W-8BEN-E` form describing this status, you can [request one](https://www.tigerdata.com/contact/). If you are elsewhere in the world, no taxes are applied to your account, according to the Value-Added Tax Act of Finland, section 69 h. ## Corporate billing [Section titled “Corporate billing”](#corporate-billing) If you prefer to pay by invoice, or if you are unable to provide a credit card for billing, you can switch your project to corporate billing instead. Under this model, invoices are generated at the end of the month based on actual usage, and are sent in `.pdf` format by email to the billing email addresses you configured in your dashboard. Payment terms for corporate invoices are 14 days net, by bank transfer, to the bank details provided on the invoice. By default, services are charged in US Dollars (USD), but you can request your invoices be sent in either Euros (EUR) or Pounds Sterling (GBP) at the invoice date's currency exchange rates. To switch from credit card to corporate billing, make sure your billing profile and email address is correct in your project's billing settings, and send a message to the [Tiger Data support team](https://www.tigerdata.com/contact/) asking to be changed to corporate billing. # Connection pools > Pool client connections to your service in Managed Service for TimescaleDB to use less server resources When you connect to your database, you consume server resources. If you have a lot of connections to your database, you can consume a lot of server resources. One way to mitigate this is to use connection pooling, which allows you to have high numbers of connections, but keep your server resource use low. The more client connections you have to your database, the more useful connection pooling becomes. By default, PostgreSQL creates a separate backend process for each connection to the server. Connection pooling uses a tool called PGBouncer to pool multiple connections to a single backend process. PGBouncer automatically interleaves the client queries to use a limited number of backend connections more efficiently, leading to lower resource use on the server and better total performance. Without connection pooling, the database connections are handled directly by PostgreSQL backend processes, one process per connection: ![Connection flow without connection pooling](/docs/_astro/pgbouncer-pooling-none.BlFrO1gt_2uhBaj.webp) ![Connection flow without connection pooling](/docs/_astro/pgbouncer-pooling-none-dark.Dd9DxYrk_2ivwKv.webp) When you add connection pooling, fewer backend connections are required. This frees up server resources for other tasks, such as disk caching: ![Connection flow with PgBouncer connection pooling enabled](/docs/_astro/pgbouncer-pooling-enabled.D-Mkqkqu_ZxnUmk.webp) ![Connection flow with PgBouncer connection pooling enabled](/docs/_astro/pgbouncer-pooling-enabled-dark.B7pKBgJv_4OrRH.webp) Connection pooling allows you to handle up to 5000 database client connections simultaneously. You can calculate how many connections you can handle by the number of CPU cores you have available. You should have at least one connection per core, but make sure you are not overloading each core. A good number of connections to aim for is three to five times the available CPU cores, depending on your workload. ## Connection pooling modes [Section titled “Connection pooling modes”](#connection-pooling-modes) There are several different pool modes: * Transaction (default) * Session * Statement ### Transaction pooling mode [Section titled “Transaction pooling mode”](#transaction-pooling-mode) This is the default pooling mode. It allows each client connection to take turns using a backend connection during a single transaction. When the transaction is committed, the backend connection is returned back into the pool and the next waiting client connection reuses the same connection immediately. This provides quick response times for queries as long as the most transactions are performed quickly. This is the most commonly used mode. ### Session pooling mode [Section titled “Session pooling mode”](#session-pooling-mode) This mode holds a client connection until the client disconnects. When the client disconnects, the server connection is returned back into the connection pool free connection list, to wait for the next client connection. Client connections are accepted at TCP level, but their queries only proceed when another client disconnects and frees up the backend connection back into the pool. This mode is useful when you require a wait queue for incoming connections, while keeping the server memory usage low. However, it is not useful in most common scenarios because the backend connections are recycled very slowly. ### Statement pooling mode [Section titled “Statement pooling mode”](#statement-pooling-mode) This mode is similar to the transaction pool mode, except that instead of allowing a full transaction to be run, it cycles the server side connections after each and every database statement (SELECT, INSERT, UPDATE, DELETE, for example). Transactions containing multiple SQL statements are not allowed in this mode. This mode is best suited to specialized workloads that use sharding front-end proxies. ## Set up a connection pool [Section titled “Set up a connection pool”](#set-up-a-connection-pool) You can set up a connection pool from the Managed Service for TimescaleDB Console. Make sure you have already created a MST service that you want to add connection pooling to. 1. **Navigate to `Pools`** In [MST Console](https://portal.managed.timescale.com/login), navigate to the `Services` list, and click the name of the MST service you want to add connection pooling to. In the `Service overview` page, navigate to the `Pools` tab. 2. **Add a new pool** Click `Add Pool` to create a new pool. 3. **Configure the pool** In the `Create New Connection Pool` dialog, use these settings: * In the `Pool name` field, type a name for your new pool. This name becomes the database `dbname` connection parameter for your pooled client connections. * In the `Database` field, select a database to connect to. Each pool can only connect to one database. * In the `Pool Mode` field, select which [pool mode](#connection-pooling-modes) to use. * In the `Pool Size` field, select the maximum number of server connections this pool can use at any one time. * In the `Username` field, select which database username to connect to the database with. 4. **Create the pool** Click `Create` to create the pool, and see the details of the new pool in the list. You can click `Info` next to the pool details to see more information, including the URI and port details. Note Pooled servers use a different port number than regular servers. This allows you to use both pooled and un-pooled connections at the same time. # Create an MST service > Create and connect to a service in Managed Service for TimescaleDB [Managed Service for TimescaleDB (MST)](https://www.tigerdata.com/mst-signup) is hosted [TimescaleDB](https://github.com/timescale/timescaledb) offered in partnership with Aiven. ## Create your first service [Section titled “Create your first service”](#create-your-first-service) A service in Managed Service for TimescaleDB is a cloud instance on your chosen cloud provider, which you can install your database on. 1. **Sign in to your MST Console** [Sign in](https://portal.managed.timescale.com/login) to your Managed Service for TimescaleDB Console. 2. **Click `Create service` and choose `TimescaleDB`** Update your preferences: ![Creating a new MST service](/docs/_astro/new-service.Bvjfe18u_Z1D5562.webp) * In the `Select Your Cloud Service Provider` field, click your preferred provider. * In the `Select Your Cloud Service Region` field, click your preferred server location. This is often the server that's physically closest to you. * In the `Select Your Service Plan` field, click your preferred plan, based on the hardware configuration you require. If you are in your trial period, and just want to try the MST service out, or develop a proof of concept, we recommend the `Dev` plan, because it is the most cost-effective during your trial period. 3. **Review your settings and click `Create Service`** In the information bar on the right of the screen, review the settings you have selected for your MST service. The MST service takes a few minutes to provision. ## Connect to your MST service from the command prompt [Section titled “Connect to your MST service from the command prompt”](#connect-to-your-mst-service-from-the-command-prompt) When you have an MST service up and running, you can connect to it from your local system using the `psql` command-line utility. This is the same tool you might have used to connect to PostgreSQL before, but if you haven't installed it yet, check out the [installing psql](/docs/integrate/query-administration/psql) section. 1. **Sign in to your MST Console** [Sign in](https://portal.managed.timescale.com/login) to your MST Console. 2. **Find your service** In the `Services` tab, find the MST service you want to connect to, and check it is marked as `Running`. 3. **Get the connection details** Click the name of the MST service you want to connect to see the connection information. Take a note of the `host`, `port`, and `password`. 4. **Connect with psql** On your local system, at the command prompt, connect to the MST service, using your own MST service details: ```bash psql -x "postgres://tsdbadmin:@:/defaultdb?sslmode=require" ``` If your connection is successful, you'll see a message like this, followed by the `psql` prompt: ```bash psql (13.3, server 13.4) SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off) Type "help" for help. defaultdb=> ``` ## Check that you have the TimescaleDB extension [Section titled “Check that you have the TimescaleDB extension”](#check-that-you-have-the-timescaledb-extension) TimescaleDB is provided as an extension to your PostgreSQL database, and it is enabled by default when you create a new MST service on Managed Service for TimescaleDB. You can check that the TimescaleDB extension is installed by using the `\dx` command at the `psql` prompt. It looks like this: ```sql defaultdb=> \dx List of installed extensions -[ RECORD 1 ]------------------------------------------------------------------ Name | plpgsql Version | 1.0 Schema | pg_catalog Description | PL/pgSQL procedural language -[ RECORD 2 ]------------------------------------------------------------------ Name | timescaledb Version | 2.5.1 Schema | public Description | Enables scalable inserts and complex queries for time-series data defaultdb=> ``` ## Install and update TimescaleDB Toolkit [Section titled “Install and update TimescaleDB Toolkit”](#install-and-update-timescaledb-toolkit) Run this command on each database you want to use Toolkit with: ```sql CREATE EXTENSION timescaledb_toolkit; ``` Update an installed version of Toolkit using this command: ```sql ALTER EXTENSION timescaledb_toolkit UPDATE; ``` ## Where to next [Section titled “Where to next”](#where-to-next) Now that you have your first MST service up and running, check out the rest of the docs in this section, to find out what you can do with it. If you want to work through some tutorials to help you get up and running with TimescaleDB and time-series data, check out [Learn](/docs/learn). You can always [contact us](https://www.tigerdata.com/contact) for any questions you might have. # Use the Postgres dblink extension > Use the dblink extension and connect to other Postgres databases The `dblink` [PostgreSQL extension](https://www.postgresql.org/docs/current/dblink.html) allows you to connect to other PostgreSQL databases and to run arbitrary queries. You can use [foreign data wrappers](https://www.postgresql.org/docs/current/postgres-fdw.html) (FDWs) to define a remote `foreign server` to access its data. The database connection details such as hostnames are kept in a single place, and you only need to create a `user mapping` to store remote connections credentials. ## Before you begin [Section titled “Before you begin”](#before-you-begin) Sign in to your Managed Service for TimescaleDB service, navigate to the `Overview` tab, and take a note of these parameters for the PostgreSQL remote server. Alternatively, you can use the `avn service get` command in the Aiven client: * `HOSTNAME`: The remote database hostname * `PORT`: The remote database port * `USER`: The remote database user to connect. The default user is `tsdbadmin`. * `PASSWORD`: The remote database password for the `USER` * `DATABASE_NAME`: The remote database name. The default database name is `defaultdb`. ## Enable the dblink extension [Section titled “Enable the dblink extension”](#enable-the-dblink-extension) To enable the `dblink` extension on an MST PostgreSQL MST service: 1. **Connect to the database** Connect to the database as the `tsdbadmin` user: ```bash psql -x "postgres://tsdbadmin:@:/defaultdb?sslmode=require" ``` 2. **Create the dblink extension and test data** 1. Create the `dblink` extension: ```sql CREATE EXTENSION dblink; ``` 2. Create a table named `inventory`: ```sql CREATE TABLE inventory (id int); ``` 3. Insert data into the `inventory` table: ```sql INSERT INTO inventory (id) VALUES (100), (200), (300); ``` ## Create a foreign data wrapper using dblink\_fdw [Section titled “Create a foreign data wrapper using dblink\_fdw”](#create-a-foreign-data-wrapper-using-dblink_fdw) 1. **Create a user** Create a user `user1` who can access the `dblink`: ```sql CREATE USER user1 PASSWORD 'secret1' ``` 2. **Create a remote server definition** Create a remote server definition named `mst_remote`, using `dblink_fdw` and the connection details of the Managed Service for TimescaleDB service: ```sql CREATE SERVER mst_remote FOREIGN DATA WRAPPER dblink_fdw OPTIONS ( host 'HOST', dbname 'DATABASE_NAME', port 'PORT' ); ``` 3. **Create a user mapping** Create a user mapping for `user1` to automatically authenticate as `tsdbadmin` when using the `dblink`: ```sql CREATE USER MAPPING FOR user1 SERVER mst_remote OPTIONS ( user 'tsdbadmin', password 'PASSWORD' ); ``` 4. **Grant usage on the remote server** Enable `user1` to use the remote PostgreSQL connection `mst_remote`: ```sql GRANT USAGE ON FOREIGN SERVER mst_remote TO user1; ``` ## Query data using a foreign data wrapper [Section titled “Query data using a foreign data wrapper”](#query-data-using-a-foreign-data-wrapper) To query a foreign data wrapper, you must be a database user with the necessary permissions on the remote server. 1. **Connect to the service** Connect to the Managed Service for TimescaleDB service as `user1` with necessary grants to the remote server. 2. **Establish the dblink connection** Establish the `dblink` connection to the remote target server: ```sql SELECT dblink_connect('my_new_conn', 'mst_remote'); ``` 3. **Query using the foreign server definition** Query using the foreign server definition as parameter: ```sql SELECT * FROM dblink('my_new_conn','SELECT * FROM inventory') AS t(a int); ``` Output is similar to: ```sql a ----- 100 200 300 (3 rows) ``` # Supported extensions > Add an extension and see a list of available extensions in Managed Service for TimescaleDB Managed Service for TimescaleDB supports many PostgreSQL extensions. See [available extensions](#available-extensions) for a full list. ## Add an extension [Section titled “Add an extension”](#add-an-extension) You can add a supported extension to your database from the command line. Warning Some extensions have dependencies. When adding these, make sure to create them in the proper order. Note Some extensions require disconnecting and reconnecting the client connection before they are fully available. 1. **Connect to your database** Connect to your database as the `tsdbadmin` user. 2. **Create the extension** Run `CREATE EXTENSION IF NOT EXISTS `. ## Available extensions [Section titled “Available extensions”](#available-extensions) These extensions are available on Managed Service for TimescaleDB: * address\_standardizer * address\_standardizer\_data\_us * aiven\_extras * amcheck * anon * autoinc * bloom * bool\_plperl * btree\_gin * btree\_gist * citext * cube * dblink * dict\_int * dict\_xsyn * earthdistance * file\_fdw * fuzzystrmatch * h3 * h3\_postgis * hll * hstore * hstore\_plperl * insert\_username * intagg * intarray * isn * jsonb\_plperl * lo * ltree * moddatetime * pageinspect * pg\_buffercache * pg\_cron * pg\_freespacemap * pg\_prewarm * pg\_repack * pg\_similarity * pg\_stat\_monitor * pg\_stat\_statements * pg\_surgery * pg\_trgm * pg\_visibility * pg\_walinspect * pgaudit * pgcrypto * pgrouting * pgrowlocks * pgstattuple * plperl * plpgsql * postgis * postgis\_raster * postgis\_sfcgal * postgis\_tiger\_geocoder * postgis\_topology * postgres\_fdw * refint * rum * seg * sslinfo * tablefunc * tcn * timescaledb * tsm\_system\_rows * tsm\_system\_time * unaccent * unit * uuid-ossp * vector * vectorscale * xml2 * timescaledb\_toolkit Note The `postgis_legacy` extension is not packaged or supported as an extension by the PostGIS project. Tiger Data provides the extension package for Managed Service for TimescaleDB. ## Request an extension [Section titled “Request an extension”](#request-an-extension) You can request an extension not on the list by contacting Support. In your request, specify the database service and user database where you want to use the extension. Untrusted language extensions are not supported. This restriction preserves our ability to offer the highest possible service level. An example of an untrusted language extension is `plpythonu`. Note You can contact Support directly from Managed Service for TimescaleDB. Click the life-preserver icon in the upper-right corner of your dashboard. # Failover > Learn how failover to a read-only replica ensures business continuity for your services running in Managed Service for TimescaleDB One standby read-only replica server is configured, for each Managed Service for TimescaleDB service on a Pro plan. You can query a read-only replica server, but cannot write to a read-only replica server. When a master server fails, the standby replica server is automatically promoted as master. If you manually created a read-only replica service, then if a master server fails, the read-only replica services are not promoted as master servers. The two distinct cases during which failovers occur are: * When the master or replica fails unexpectedly, for example because the hardware hosting the virtual machine fails. * When controlled failover happens because of upgrades. ## Uncontrolled master or replica fail [Section titled “Uncontrolled master or replica fail”](#uncontrolled-master-or-replica-fail) When a replica server fails unexpectedly, there is no way to know whether the server really failed, or whether there is a temporary network glitch with the cloud provider's network. There is a 300 second timeout before Managed Service for TimescaleDB automatically decides the server is gone and spins up a new replica server. During these 300 seconds, `replica.servicename.timescaledb.io` points to a server that may not serve queries anymore. The DNS record pointing to the master server `servicename.timescaledb.io` continues to serve the queries. If the replica server does not come back up within 300 seconds, `replica.servicename.timescaledb.io` points to the master server, until a new replica server is built. When the master server fails, a replica server waits for 60 seconds before promoting itself as master. During this 60-second timeout, the master server `servicename.timescaledb.io` remains unavailable and does not respond. However, `replica.servicename.timescaledb.io` works in read-only mode. After the replica server promotes itself as master, `servicename.timescaledb.io` points to the new master server, and `replica.servicename.timescaledb.io` continues to point to the new master server. A new replica server is built automatically, and after it is in sync, `replica.servicename.timescaledb.io` points to the new replica server. ## Controlled failover during upgrades [Section titled “Controlled failover during upgrades”](#controlled-failover-during-upgrades) When applying upgrades or plan changes on business or premium plans, the standby server is replaced: A new server is started, the backup is restored, and the new server starts following the old master server. After the new server is up and running, `replica.servicename.timescaledb.io` is updated, and the old replica server is deleted. For premium plans, this step is executed for both replica servers before the master server is replaced. Two new servers are started, a backup is restored, and one new server is synced up to the old master server. When it is time to switch the master to a new server, the old master is terminated and one of the new replica servers is immediately promoted as a master. At this point, `servicename.timescaledb.io` is updated to point at the new master server. Similarly, the new master is removed from the `replica.servicename.timescaledb.io` record. # Identify and repair issues with PostgreSQL indexes with REINDEX > Identify and resolve issues with Postgres indexes in Managed Service for TimescaleDB PostgreSQL indexes can be corrupted for a variety of reasons, including software bugs, hardware failures, or unexpected duplicated data. `REINDEX` allows you to rebuild the index in such situations. ## Rebuild non-unique indexes [Section titled “Rebuild non-unique indexes”](#rebuild-non-unique-indexes) You can rebuild corrupted indexes that do not have `UNIQUE` in their definition. You can run the `REINDEX` command for all indexes of a table (`REINDEX TABLE`), and for all indexes in the entire database (`REINDEX DATABASE`). For more information on the `REINDEX` command, see the [PostgreSQL documentation](https://www.postgresql.org/docs/current/sql-reindex.html). This command creates a new index that replaces the old one: ```sql REINDEX INDEX ; ``` Warning When you use `REINDEX`, the tables are locked and you may not be able to use the database, until the operation is complete. In some cases, you might need to manually build a second index concurrently with the old index, and then remove the old index: ```sql CREATE INDEX CONCURRENTLY test_index_new ON table_a (...); DROP INDEX CONCURRENTLY test_index_old; ALTER INDEX test_index_new RENAME TO test_index; ``` ## Rebuild unique indexes [Section titled “Rebuild unique indexes”](#rebuild-unique-indexes) A `UNIQUE` index works on one or more columns where the combination is unique in the table. When the index is corrupted or disabled, duplicated physical rows appear in the table, breaking the uniqueness constraint of the index. When you try to rebuild an index that is not unique, the `REINDEX` command fails. To resolve this issue, first remove the duplicate rows from the table and then rebuild the index. ### Identify conflicting duplicated rows [Section titled “Identify conflicting duplicated rows”](#identify-conflicting-duplicated-rows) To identify conflicting duplicate rows, you need to run a query that counts the number of rows for each combination of columns included in the index definition. For example, this `route` table has a `unique_route_index` index defining unique rows based on the combination of the `source` and `destination` columns: ```sql CREATE TABLE route( source TEXT, destination TEXT, description TEXT ); CREATE UNIQUE INDEX unique_route_index ON route (source, destination); ``` If the `unique_route_index` is corrupt, you can find duplicated rows in the `route` table using this query: ```sql SELECT source, destination, count FROM (SELECT source, destination, COUNT(*) AS count FROM route GROUP BY source, destination) AS foo WHERE count > 1; ``` The query groups the data by the same `source` and `destination` fields defined in the index, and filters any entries with more than one occurrence. Resolve the problematic entries in the rows by manually deleting or merging the entries until no duplicates exist. After all duplicate entries are removed, you can use the `REINDEX` command to rebuild the index. # Ingest data > Ingest data into your services running in Managed Service for TimescaleDB There are several different ways of ingesting your data into Managed Service for TimescaleDB. This section contains instructions to: * Bulk upload [from a `.csv` file](#bulk-upload-from-csv-files) * Insert data [directly using a client driver](#insert-data-directly-using-a-client-driver), such as JDBC, ODBC, or Node.js * Insert data [directly using a message queue](#insert-data-directly-using-a-message-queue), such as Kafka Before you begin, make sure you have [created your Managed Service for TimescaleDB service](/docs/deploy/mst/create-mst-service), and can connect to it using `psql`. ## Preparing your new database [Section titled “Preparing your new database”](#preparing-your-new-database) 1. **Connect to your MST service** Use `psql` to connect to your MST service. ```sql psql -h -p -U -W -d ``` You retrieve the MST service URL, port, and login credentials from the MST service overview in the [MST dashboard](https://portal.managed.timescale.com/login). 2. **Create a new database** Create a new database for your data. In this example, the new database is called `new_db`: ```sql CREATE DATABASE new_db; \c new_db; ``` 3. **Create a table** Create a new SQL table in your database. The columns you create for the table must match the columns in your source data. In this example, the table is storing weather condition data, and has columns for the timestamp, location, and temperature: ```sql CREATE TABLE conditions ( time TIMESTAMPTZ NOT NULL, location text NOT NULL, temperature DOUBLE PRECISION NULL ); ``` 4. **Load the TimescaleDB extension** Load the `timescaledb` PostgreSQL extension: ```sql CREATE EXTENSION timescaledb; \dx ``` 5. **Convert to a hypertable** Convert the SQL table into a hypertable: ```sql SELECT create_hypertable('conditions', by_range('time')); ``` Note The `by_range` dimension builder is an addition to TimescaleDB 2.13. When you have successfully set up your new database, you can ingest data using one of these methods. ## Bulk upload from CSV files [Section titled “Bulk upload from CSV files”](#bulk-upload-from-csv-files) If you have a dataset stored in a `.csv` file, you can import it into an empty hypertable. You need to begin by creating the new table, before you import the data. Important Before you begin, make sure you have [prepared your new database](#preparing-your-new-database). 1. **Insert data using timescaledb-parallel-copy** Insert data into the new hypertable using the `timescaledb-parallel-copy` tool. You should already have the tool installed, but you can install it manually from [our GitHub repository](https://github.com/timescale/timescaledb-parallel-copy) if you need to. In this example, we are inserting the data using four workers: ```sql timescaledb-parallel-copy \ --connection '' \ --table conditions \ --file ~/Downloads/example.csv \ --workers 4 \ --copy-options "CSV" \ --skip-header ``` We recommend that you set the number of workers lower than the number of available CPU cores on your client machine or server, to prevent the workers having to compete for resources. This helps your ingest go faster. 2. **(Optional) Use PostgreSQL COPY instead** If you don't want to use the `timescaledb-parallel-copy` tool, or if you have a very small dataset, you can use the PostgreSQL `COPY` command instead: ```sql psql '/new_db?sslmode=require' -c "\copy conditions FROM WITH (FORMAT CSV, HEADER)" ``` ## Insert data directly using a client driver [Section titled “Insert data directly using a client driver”](#insert-data-directly-using-a-client-driver) You can use a client driver such as JDBC, Python, or Node.js, to insert data directly into your new database. See the [PostgreSQL instructions](https://odbc.postgresql.org/) for using the ODBC driver. See the [Code Quick Starts](/docs/get-started/quickstart/connect-your-app) for using various languages, including Python and node.js. ## Insert data directly using a message queue [Section titled “Insert data directly using a message queue”](#insert-data-directly-using-a-message-queue) If you have data stored in a message queue, you can import it into your MST service. This section provides instructions on using the Kafka Connect PostgreSQL connector. This connector deploys PostgreSQL change events from Kafka Connect to a runtime service. It monitors one or more schemas in a Managed Service for TimescaleDB service, and writes all change events to Kafka topics, which can then be independently consumed by one or more clients. Kafka Connect can be distributed to provide fault tolerance, which ensures the connectors are running and continually keeping up with changes in the database. You can also use the PostgreSQL connector as a library without Kafka or Kafka Connect. This allows applications and services to directly connect to MST and obtain the ordered change events. In this environment, the application must record the progress of the connector so that when it is restarted, the connect can continue where it left off. This approach can be useful for less critical use cases. However, for production use cases, we recommend that you use the connector with Kafka and Kafka Connect. See [these instructions](https://github.com/debezium/debezium/tree/main/debezium-connector-postgres) for using the Kafka connector. # Integrations > Integrate Managed Service for TimescaleDB with other services Managed Service for TimescaleDB integrates with the other tools you are already using. You can combine your Managed Service for TimescaleDB services with third-party tools and build a complete cloud data platform. You can integrate Managed Service for TimescaleDB with: * [Grafana](/docs/deploy/mst/integrations/grafana-mst) * [Loggly](/docs/deploy/mst/integrations/logging) * [Datadog](/docs/deploy/mst/integrations/metrics-datadog) * [Prometheus](/docs/deploy/mst/integrations/prometheus-mst) * [Google Data Studio](/docs/deploy/mst/integrations/google-data-studio-mst) * Syslog * External Elasticsearch * External OpenSearch # Visualize data with Google Data Studio > Integrate Google Data Studio with Managed Service for TimescaleDB You can create reports or perform some analysis on data you have in Managed Service for TimescaleDB using Google Data Studio. You can use Data Studio to integrate other data sources, such as YouTube Analytics, MySQL, BigQuery, AdWords, and others. ## Before you begin [Section titled “Before you begin”](#before-you-begin) * You should also have a Google account. * In the overview page of your Managed Service for TimescaleDB service: * Download the CA certificate named `ca.pem` for your MST service. * Make a note of the `Host`, `Port`, `Database name`, `User`, and `Password` fields for the MST service. ## Connecting to a data source from Data Studio [Section titled “Connecting to a data source from Data Studio”](#connecting-to-a-data-source-from-data-studio) 1. **Create a new data source** Log in to Google and open [Google Data Studio](https://lookerstudio.google.com/overview). Click the `Create +` button and choose `Data source`. Select `PostgreSQL` as the Google Connector. 2. **Enter database details and enable SSL** In the `Database Authentication` tab, type details for the `Host Name`, `Port`, `Database`, `Username`, and `Password` fields. Select `Enable SSL` and upload your server certificate file, `ca.pem`. 3. **Authenticate and connect** Click `AUTHENTICATE`. Choose the table to be queried, or select `CUSTOM QUERY` to create an SQL query. Click `CONNECT`. # Visualize data with Grafana > Integrate Managed Service for TimescaleDB as a data source in Grafana to visualize your data You can integrate Managed Service for TimescaleDB with Grafana to visualize your data. Grafana service in MST has built-in Prometheus, PostgreSQL, Jaeger, and other data source plugins that allow you to query and visualize data from a compatible database. ## Before you begin [Section titled “Before you begin”](#before-you-begin) * Created a Managed Service for TimescaleDB service * Created a Grafana service ## Configure Managed Service for TimescaleDB as a data source [Section titled “Configure Managed Service for TimescaleDB as a data source”](#configure-managed-service-for-timescaledb-as-a-data-source) You can configure a Managed Service for TimescaleDB service as a data source to a Grafana service to query and visualize the data from the database. 1. **Set up the Datasource integration** In [MST Console](https://portal.managed.timescale.com/login), click the MST service that you want to add as a data source for the Grafana service. In the `Overview` tab, go to the `Service Integrations` section and click `Set up integration`. In the `Available service integrations for TimescaleDB` dialog, click the `Use Integration` button for `Datasource`. 2. **Enable the Grafana service** In the dialog that appears, choose the Grafana service in the drop-down menu, and click the `Enable` button. 3. **Open and log in to Grafana** In the `Services` view, click the Grafana service to which you added the MST MST service as a data source. In the `Overview` tab for the Grafana service, make a note of the `User` and `Password` fields, then click the link in the `Service URI` field to open Grafana. Log in with your service credentials. 4. **Verify the data source** Navigate to `Configuration` → `Data sources`. The data sources page lists Managed Service for TimescaleDB as a configured data source for the Grafana instance. When you have configured Managed Service for TimescaleDB as a data source in Grafana, you can create panels that are populated with data using SQL. # Logging > Retrieve logging information on Managed Service for TimescaleDB There are a number of different ways to review logs and metrics for your Managed Service for TimescaleDB services. You can use the native logging tool in MST Console, retrieve detailed logs using the Aiven CLI tool, or integrate a third-party service, such as [SolarWinds Loggly](https://www.loggly.com/). ## Native logging [Section titled “Native logging”](#native-logging) To see the most recent logged events for your MST service: 1. **Find the service** In [MST Console](https://portal.managed.timescale.com/login), in the `Services` tab, find the MST service you want to review, and check it is marked as `Running`. 2. **View the logs** Navigate to the `Logs` tab to see a constantly updated list of logged events. ![Viewing native logs in MST portal](/docs/_astro/view-logs.BrFHeiDr_txUPV.webp) ## Dump logs to a text file with the Aiven CLI [Section titled “Dump logs to a text file with the Aiven CLI”](#dump-logs-to-a-text-file-with-the-aiven-cli) If you want to dump your Managed Service for TimescaleDB logs to a text file or an archive for use later on, you can use the Aiven CLI. Sign in to your Managed Service for TimescaleDB account from the Aiven CLI tool, and use this command to dump your logs to a text file called `tslogs.txt`: ```bash avn service logs -S desc -f --project > tslogs.txt ``` For more information about the Aiven CLI tool, see the [Aiven Client section](/docs/deploy/mst/aiven-client). ## Logging integrations [Section titled “Logging integrations”](#logging-integrations) If you need to access logs for your MST services regularly, or if you need more detailed logging than Managed Service for TimescaleDB can provide in MST Console, you can connect your Managed Service for TimescaleDB to a logging service such as [SolarWinds Loggly](https://www.loggly.com/). This section covers how to create a service integration to Loggly with Managed Service for TimescaleDB. 1. **Create or log in to Loggly** Navigate to [SolarWinds Loggly](https://www.loggly.com/) and create or log in to your account. 2. **Create a customer token** From the Loggly Home screen, navigate to `Logs`→`Source Setup`. Click `Customer Tokens` from the top menu bar. On the `Customer Tokens` page, click `Add New` to create a new token. Give your token a name, and click `Save`. Copy your new token to your clipboard. 3. **Add a new Syslog endpoint** Log in to your Managed Service for TimescaleDB account, and navigate to `Service Integrations`. Navigate to `Syslog`, and click `Add new endpoint`. 4. **Configure the Syslog endpoint** In the `Create new syslog endpoint` dialog, complete these fields: * In the `Endpoint name` field, type a name for your endpoint. * In the `Server` field, type `logs-01.loggly.com`. * In the `Port` field, type `514`. * Uncheck the `TLS` checkbox. * In the `Format` field, select `rfc5425`. * In the `Structured Data` field, type `@41058`, using the Loggly token you copied earlier. You can also add a tag here, which you can use to more easily search for your logs in Loggly. For example, `8480330f5-aa09-46b0-b220-a0efa372b17b@41058 TAG="example-tag"`. Click `Create` to create the endpoint. When the endpoint has been created, it shows as an enabled service integration, with a green `active` indicator. 5. **View incoming logs in Loggly** In the Loggly dashboard, navigate to `Search` to see your incoming logs. From here, you can create custom dashboards and view reports for your logs. ![Viewing incoming MST logs in Loggly](/docs/_astro/loggly-view-logs.D5Fjee05_Kfm05.webp) # Send metrics to Datadog > Collect Datadog metrics on your Managed Service for TimescaleDB service Datadog is a popular cloud-based monitoring service. You can send metrics to Datadog using a metrics collection agent for graphing, service dashboards, alerting, and logging. Managed Service for TimescaleDB (MST) can send data directly to Datadog for monitoring. Datadog integrations are provided free of charge on Managed Service for TimescaleDB. You need to create a Datadog API key, and use the key to enable metrics for your Managed Service for TimescaleDB service. Note Datadog logging is not currently supported on MST. ## Before you begin [Section titled “Before you begin”](#before-you-begin) * Created a Managed Service for TimescaleDB service. * Signed up for [Datadog](https://app.datadoghq.com/), and can log in to your Datadog dashboard. * Created an API key in your Datadog account. For more information about creating a Datadog API key, see [Datadog API and Application Keys](https://docs.datadoghq.com/account_management/api-app-keys/). ## Upload a Datadog API key [Section titled “Upload a Datadog API key”](#upload-a-datadog-api-key) To integrate MST with Datadog you need to upload the API key that you generated in your Datadog account to MST. 1. **Navigate to `Integration Endpoints`** In [MST Console](https://portal.managed.timescale.com/login), choose the project you want to connect to Datadog, and click `Integration Endpoints`. 2. **Create a new Datadog endpoint** Select `Datadog`, then choose `Create new`. In `Add new Datadog service integration`, complete these details: * In the `Endpoint integration` section, give your endpoint a name, and paste the API key from your Datadog dashboard. Ensure you choose the site location that matches where your Datadog service is hosted. * *Optional*: In the `Endpoint tags` section, you can add custom tags to help you manage your integrations. Click `Add endpoint` to save the integration. ![Adding a Datadog integration endpoint in MST](/docs/_astro/add-datadog-integration.wqEgQ9WN_Zxc0Nt.webp) ## Activate Datadog integration for a MST service [Section titled “Activate Datadog integration for a MST service”](#activate-datadog-integration-for-a-mst-service) When you have successfully added the endpoint, you can set up one of your MST services to send data to Datadog. 1. **Select the service** Sign in to MST Console, navigate to `Services`, and select the MST service you want to monitor. 2. **Enable Datadog Metrics** In the `Integrations` tab, go to `External integrations` section and select `Datadog Metrics`. In the `Datadog integration` dialog, select the Datadog endpoint that you created and click `Enable`. The Datadog endpoint is listed under `Enabled integrations` for the Managed Service for TimescaleDB service. ## Datadog dashboards [Section titled “Datadog dashboards”](#datadog-dashboards) When you have your Datadog integration set up successfully, you can use the Datadog dashboard editor to configure your visualizations. For more information, see the [Datadog Dashboard documentation](https://docs.datadoghq.com/dashboards/). # Set up a Prometheus endpoint > Use Prometheus to monitor your Managed Service for TimescaleDB You can get more insights into the performance of your Managed Service for TimescaleDB service by monitoring it using [Prometheus](https://prometheus.io), a popular open source metrics-based systems monitoring solution. ## Before you begin [Section titled “Before you begin”](#before-you-begin) * Create a Managed Service for TimescaleDB service. * Make a note of the `Port` and `Host` for your MST service. ## Enabling Prometheus service integration [Section titled “Enabling Prometheus service integration”](#enabling-prometheus-service-integration) 1. **Create a new Prometheus endpoint** 1. In [MST Console](https://portal.managed.timescale.com/login), choose a project and navigate to `Integration Endpoints`. Navigate to `Prometheus`, and click `Create new`. 2. In the `Create new Prometheus endpoint` dialog, complete these fields: * In the `Endpoint name` field, type a name for your endpoint. * In the `Username` field, type your username. * In the `Password` field, type your password. 3. Click `Create` to create the endpoint. These details are used when setting up your Prometheus installation, in the `prometheus.yml` configuration file. This allows you to make this Managed Service for TimescaleDB endpoint a target for Prometheus to scrape. 2. **Configure Prometheus** Use this sample configuration file to set up your Prometheus installation, by substituting ``, ``, ``, and `` with those of your Managed Service for TimescaleDB service: ```yaml global: scrape_interval: 10s evaluation_interval: 10s scrape_configs: - job_name: prometheus scheme: https static_configs: - targets: [':'] tls_config: insecure_skip_verify: true basic_auth: username: password: remote_write: - url: "http://:9201/write" remote_read: - url: "http://:9201/read" ``` 3. **Enable Prometheus for a service** 1. In the MST Console, navigate to `Services` and select the MST service you want to monitor. 2. In the `Integrations` tab, go to `External integrations` section and select `Prometheus`. 3. In the `Prometheus integrations` dialog, select the Prometheus endpoint that you created and click `Enable`. The Prometheus endpoint is listed under `Enabled integrations` for the Managed Service for TimescaleDB service. # Maintenance > Learn how maintenance is automatically handled on Managed Service for TimescaleDB On Managed Service for TimescaleDB, software updates are handled automatically, and you do not need to perform any actions to keep up to date. Non-critical software updates are applied during a maintenance window that you can define to suit your workload. If a security vulnerability is found that affects you, maintenance might be performed outside of your scheduled maintenance window. After maintenance updates have been applied, if a new version of the TimescaleDB binary has been installed, you need to update the extension to use the new version. To do this, use this command: ```sql ALTER EXTENSION timescaledb UPDATE; ``` Note After a maintenance update, the DNS name remains the same, but the IP address it points to changes. ## Non-critical maintenance updates [Section titled “Non-critical maintenance updates”](#non-critical-maintenance-updates) Non-critical upgrades are made available before the upgrade is performed automatically. During this time you can click `Apply upgrades` to start the upgrade at any time. However, after the time expires, usually around a week, the upgrade is triggered automatically in the next available maintenance window for your service. You can configure the maintenance window so that these upgrades are started only at a particular time, on a set day of the week. If there are no pending upgrades available during a regular maintenance window, no changes are performed. When you are considering your maintenance window schedule, you might prefer to choose a day and time that usually has very low activity, such as during the early hours of the morning, or over the weekend. This can help minimize the impact of a short service interruption. Alternatively, you might prefer to have your maintenance window occur during office hours, so that you can monitor your system during the upgrade. ### Adjusting your maintenance window [Section titled “Adjusting your maintenance window”](#adjusting-your-maintenance-window) 1. **Select the service** In [Managed Service for TimescaleDB Console](https://portal.managed.timescale.com/login), click the MST service that you want to manage the maintenance window for. 2. **Change the maintenance window** Click the ellipses (...) to the right of `Maintenance`, then click `Change maintenance window`. 3. **Set the schedule** In the `Service Maintenance Window` dialog, select the day of the week and the time (in Universal Coordinated Time) you want the maintenance window to start. Maintenance windows can run for up to four hours. ![Adjusting the MST maintenance window](/docs/_astro/change-service-mainenence-window.DV3bWPrI_1nt6wG.webp) 4. **Save changes** Click `Save Changes`. ## Critical updates [Section titled “Critical updates”](#critical-updates) Critical upgrades and security fixes are installed outside normal maintenance windows when necessary, and sometimes require a short outage. Upgrades are performed as rolling upgrades where completely new server instances are built alongside the old ones. When the new instances are up and running they are synchronized with the old servers, and a controlled automatic failover is performed to switch the service to the new upgraded servers. The old servers are retired automatically after the new servers have taken over. The controlled failover is a very quick and safe operation and it takes less than a minute to get clients connected again. In most cases, there is five to ten second outage during this process. # Backups > Back up your services running in the Managed Service for TimescaleDB with alternative tools Managed Service for TimescaleDB services are automatically backed up, with full backups daily, and write-ahead log (WAL) continuously recorded. All backups are [encrypted](https://aiven.io/docs/platform/concepts/cloud-security#data-encryption). Managed Service for TimescaleDB uses [`pghoard`](https://github.com/aiven/pghoard), a PostgreSQL backup daemon and restore tool, to store backup data in cloud object stores. The number of backups stored and the retention time of the backup depend on the service plan. Note The size of logical backups can be different from the size of the Managed Service for TimescaleDB backup that appears on the web console. In some cases, the difference is significant. Backup sizes that appear in the MST Console are for daily backups, before encryption and compression. To view the size of each database, including space consumed by indexes, you can use the `\l+` command at the psql prompt. ## Logical and binary backups [Section titled “Logical and binary backups”](#logical-and-binary-backups) The two types of backups are binary backups and logical backups. Full backups are version-specific binary backups which, when combined with WAL, allow consistent recovery to a point in time (PITR). You can create a logical backup with the `pg_dump` command. This table lists the differences between binary and logical backups when backing up indexes, transactions, and data: | Type | Binary | Logical | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- | | Index | Contains all data from indexes | Does not contain index data, it contains only queries used to recreate indexes from other data | | Transactions | Contains uncommitted transactions | Does not contain uncommitted transactions | | Data | Contains deleted and updated rows which have not been cleaned up by PostgreSQL VACUUM process, and all databases, including templates | Does not contain any data already deleted, and depending on the options given, the output might be compressed | ## Restore a service [Section titled “Restore a service”](#restore-a-service) Managed Service for TimescaleDB provides a point-in-time recovery (PITR). To restore your service from a backup, click the `Restore` button in the `Backups` tab for your service. The backups are taken automatically by Managed Service for TimescaleDB and retained for a few days depending on your plan type. | Plan type | Backup retention period | | --------- | ----------------------- | | Dev | 1 day | | Basic | 2 days | | Pro | 3 days | ## Manually creating a backup [Section titled “Manually creating a backup”](#manually-creating-a-backup) You can use `pg_dump` to create a backup manually. The `pg_dump` command allows you to create backups that can be directly restored elsewhere if required. Typical parameters for the command `pg_dump` include: ```bash pg_dump '' -f '' -j '' -F '' ``` The `pg_dump` command can also be run against one of the standby nodes. For example, use this command to create a backup in directory format using two concurrent jobs. The results are stored to a directory named `backup`: ```bash pg_dump 'postgres://tsdbadmin:password@mypg-myproject.a.timescaledb.io:26882/defaultdb?sslmode=require' -f backup -j 2 -F directory ``` You can put all backup files to single tar file and upload to Amazon S3. For example: ```bash export BACKUP_NAME=backup-$(date -I).tartar -cf $BACKUP_NAME backup/s3cmd put $BACKUP_NAME s3://pg-backups/$BACKUP_NAME ``` # Migrate data from self-hosted TimescaleDB to MST > Migrate a self-hosted TimescaleDB database to Managed Service for TimescaleDB You can migrate your data from self-hosted TimescaleDB to Managed Service for TimescaleDB and automate most of the common operational tasks. Each MST service has a database named `defaultdb`, and a default user account named `tsdbadmin`. You use MST Console to create additional users and databases using the `Users` and `Databases` tabs. You can switch between different plan sizes in Managed Service for TimescaleDB. However, during the migration process, choose a plan size that has the same storage size or slightly larger than the currently allocated plan. This allows you to limit the downtime during the migration process and have sufficient compute and storage resources. Warning Depending on your database size and network speed, migration can take a very long time. During this time, any new writes that happen during the migration process are not included. To prevent data loss, turn off all the writes to the source self-hosted TimescaleDB database before you start migration. Before migrating for production, do a cold run without turning off writes to the source self-hosted TimescaleDB database. This gives you an estimate of the time the migration process takes, and helps you to practice migrating without causing downtime to your customers. If you prefer the features of Tiger Cloud, you can easily [migrate your data](/docs/migrate/live-migration) from an MST service to a Tiger Cloud service. ## Before you begin [Section titled “Before you begin”](#before-you-begin) * Set up the migration machine: You run the migration commands on the migration machine. It must have enough disk space to hold the dump file. * Install the PostgreSQL [`pg_dump`](https://www.postgresql.org/docs/current/app-pgdump.html) and [`pg_restore`](https://www.postgresql.org/docs/current/app-pgrestore.html) utilities on a migration machine. * Install a client to connect to self-hosted TimescaleDB and Managed Service for TimescaleDB. These instructions use [`psql`](https://www.postgresql.org/docs/current/app-psql.html), but any client works. * [Create a target MST service](/docs/deploy/mst/create-mst-service): Provision your target MST service with enough space for all your data. * On the source self-hosted TimescaleDB and the target MST service, ensure that you are running: * The same major version of PostgreSQL. For information, see [upgrade PostgreSQL](/docs/deploy/self-hosted/upgrades/upgrade-pg). * The same major version of TimescaleDB. For more information, see [upgrade TimescaleDB to a major version](/docs/deploy/self-hosted/upgrades/major-upgrade). ## Migrate your data to an MST service [Section titled “Migrate your data to an MST service”](#migrate-your-data-to-an-mst-service) To move your data from self-hosted TimescaleDB instance to a MST service, run the following commands from your migration machine: 1. **Take offline the applications that connect to the source** The duration of migration is proportional to the amount of data stored in your database. By disconnecting your app from your database, you avoid possible data loss. 2. **Set your connection strings** These variables hold the connection information for the source self-hosted TimescaleDB instance and the target MST service: ```bash export SOURCE="postgres://:@:/" export TARGET="postgres://tsdbadmin:@:/defaultdb?sslmode=require" ``` 3. **Dump the data from your source service** ```bash pg_dump -d "$SOURCE" --no-owner -Fc -v -f dump.bak ``` 4. **Restore the data to your target service** 1. Put your target MST service in the right state for restoring: ```bash psql -d "$TARGET" -c "SELECT timescaledb_pre_restore();" ``` 2. Upload your data to the target MST service: ```bash pg_restore -d "$TARGET" --jobs 4 -Fc dump.bak ``` The `--jobs` option specifies the number of CPUs to use to dump and restore the database concurrently. 3. Return your target MST service to normal operations: ```bash psql -d "$TARGET" -c "SELECT timescaledb_post_restore();" ``` 5. **Update table statistics** Connect to your new database and update your table statistics by running [`ANALYZE`](https://www.postgresql.org/docs/current/sql-analyze.html) on your entire dataset: ```bash psql -d "$TARGET" -c "ANALYZE;" ``` To migrate from multiple databases, you repeat this migration procedure one database after another. ## Troubleshooting [Section titled “Troubleshooting”](#troubleshooting) If you see the following errors during migration, you can safely ignore them. The migration still runs successfully. * For `pg_dump`: ```bash pg_dump: warning: there are circular foreign-key constraints on this table: pg_dump: hypertable pg_dump: You might not be able to restore the dump without using --disable-triggers or temporarily dropping the constraints. pg_dump: Consider using a full dump instead of a --data-only dump to avoid this problem. pg_dump: NOTICE: hypertable data are in the chunks, no data will be copied DETAIL: Data for hypertables are stored in the chunks of a hypertable so COPY TO of a hypertable will not copy any data. HINT: Use "COPY (SELECT * FROM ) TO ..." to copy all data in hypertable, or copy each chunk individually. ``` * For `pg_restore`: ```bash pg_restore: while PROCESSING TOC: pg_restore: from TOC entry 4142; 0 0 COMMENT EXTENSION timescaledb pg_restore: error: could not execute query: ERROR: must be owner of extension timescaledb Command was: COMMENT ON EXTENSION timescaledb IS 'Enables scalable inserts and complex queries for time-series data'; ``` # Create a read-only replica > Create and use a read-only replica for your services in Managed Service for TimescaleDB PostgreSQL read-only replicas allow you to perform read-only queries against the replica and reduce the load on the primary server. You can optimize query response times across different geographical locations because the replica can be created in different regions or on different cloud providers. For information about creating a read-only replica using the Aiven client, see the documentation on [creating a read replica using the CLI](/docs/deploy/mst/aiven-client). Note If you are running a Managed Service for TimescaleDB [Pro plan](/docs/deploy/mst/about-mst), you have standby nodes available in a high availability setup. The standby nodes support read-only queries to reduce the effect of slow queries on the primary node. ## Creating a replica [Section titled “Creating a replica”](#creating-a-replica) 1. **Select the service** In [Managed Service for TimescaleDB Console](https://portal.managed.timescale.com/login), click the service you want to create a remote replica for. 2. **Create a read replica** In `Overview`, click `Create a read replica`. 3. **Configure and create** In `Create a PostgreSQL read replica`, type a name for the remote replica, select the cloud provider, location, plan that you want to use, and click `Create`. When the read-only replica is created it is listed as a service in your project. The `Overview` tab of the replica also lists the name of the primary service for the replica. To promote a read-only replica as a master database, click the `Promote to master` button. ## Using read-only replica for the service on MST [Section titled “Using read-only replica for the service on MST”](#using-read-only-replica-for-the-service-on-mst) 1. **Copy the Service URI** In the `Overview` page of the read-only replica for the service on MST, copy the `Service URI`. 2. **Connect to the read-only service** At the psql prompt, connect to the read-only service: ```sql psql ``` 3. **Check the connection type** To check whether you are connected to a primary or replica node: ```sql SELECT * FROM pg_is_in_recovery(); ``` If the output is `TRUE` you are connected to the replica, and if the output is `FALSE` you are connected to the primary server. Note Managed Service for TimescaleDB uses asynchronous replication, so some lag is expected. When you run an `INSERT` operation on the primary node, a small delay of less than a second is expected for the change to propagate to the replica. # REST API > Use REST API for integration and automation in Managed Service for TimescaleDB Managed Service for TimescaleDB has an API for integration and automation tasks. For information about using the endpoints, see the [API Documentation](https://api.aiven.io/doc/). MST offers an HTTP API with token authentication and JSON-formatted data. You can use the API for all the tasks that can be performed using the MST Console. To get started you need to first create an authentication token, and then use the token in the header to use the API endpoints. ## Create an authentication token [Section titled “Create an authentication token”](#create-an-authentication-token) 1. **Open user information** In [Managed Service for TimescaleDB](https://portal.managed.timescale.com/login), click `User Information` in the top right corner. 2. **Generate a token** In the `User Profile` page, navigate to the `Authentication` tab. Click `Generate Token`. 3. **Save the token** In the `Generate access token` dialog, type a descriptive name for the token and leave the rest of the fields blank. Copy the generated authentication token and save it. ## Using cURL to get your details [Section titled “Using cURL to get your details”](#using-curl-to-get-your-details) 1. **Set the API token** Set the environment variable `MST_API_TOKEN` with the access token that you generated: ```bash export MST_API_TOKEN="access token" ``` 2. **Query the /me endpoint** To get the details about the current user session using the `/me` endpoint: ```bash curl -s -H "Authorization: aivenv1 $MST_API_TOKEN" https://api.aiven.io/v1/me|json_pp ``` The output looks similar to this: ```bash { "user": { "auth": [], "create_time": "string", "features": { }, "intercom": {}, "invitations": [], "project_membership": {}, "project_memberships": {}, "projects": [], "real_name": "string", "state": "string", "token_validity_begin": "string", "user": "string", "user_id": "string" } } ``` # Security > Learn how your Managed Service for TimescaleDB instance is secured This section covers how Managed Service for TimescaleDB handles security of your data while it is stored. ## Cloud provider accounts [Section titled “Cloud provider accounts”](#cloud-provider-accounts) Managed Service for TimescaleDB services are hosted by cloud provider accounts controlled by Tiger Data. These accounts are managed only by Tiger Data and Aiven operations personnel. Members of the public cannot directly access the cloud provider account resources. ## Virtual machines [Section titled “Virtual machines”](#virtual-machines) Your MST services are located on one or more virtual machines. Each virtual machine is dedicated to a single customer, and is never multi-tenanted. Customer data never leaves the virtual machine, except when uploaded to an offsite backup location. When you create a new service, you need to select a cloud region. When the virtual machine is launched, it does so in the cloud region you have chosen. Your data never leaves the chosen cloud region. If a cloud region has multiple Availability Zones, or a similar high-availability mechanism, the virtual machines are distributed evenly across the zones. This provides the best possible service if an Availability Zone becomes unavailable. Access to the virtual machine providing your service is restricted. Software that is accessing your database needs to run on a different virtual machine. To reduce latency, it is best for it to be using a virtual machine provided by the same cloud provider, and in the same region, if possible. Virtual machines are not reused. They are terminated and wiped when you upgrade or delete your service. ## Project security [Section titled “Project security”](#project-security) Every Managed Service for TimescaleDB project has its own certificate authority. This certificate authority is used to sign certificates used internally by your services to communicate between different cluster nodes and to management systems. You can download your project certificate authority in Managed Service for TimescaleDB Console. In the `Services` tab, click the service you want to find the certificate for. In the service `Overview` tab, under `Connection information`, locate the `CA Certificate` section, and click `Show` to see the certificate. It is recommended that you set up your browser or client to trust that certificate. All server certificates are signed by the project certificate authority of Managed Service for TimescaleDB Console. ## Data encryption [Section titled “Data encryption”](#data-encryption) Managed Service for TimescaleDB at-rest data encryption covers both active service instances as well as service backups in cloud object storage. Service instances and the underlying virtual machines use full volume encryption. The encryption method uses LUKS, with a randomly generated ephemeral key per each instance, and per volume. The keys are never re-used, and are disposed of when the instance is destroyed. This means that a natural key rotation occurs with roll-forward upgrades. By default, the LUKS mode is `aes-xts-plain64:sha256`, with a 512-bit key. Backups are encrypted with a randomly generated key per file. These keys are in turn encrypted with an RSA key-encryption key-pair, and stored in the header section of each backup segment. The file encryption is performed with AES-256 in CTR mode, with HMAC-SHA256 for integrity protection. The RSA key-pair is randomly generated for each service. The key lengths are 256-bit for block encryption, 512-bit for the integrity protection, and 3072-bits for the RSA key. Encrypted backup files are stored in the object storage in the same region that the virtual machines are located for the service. ## Networking security [Section titled “Networking security”](#networking-security) Access to provided services is only provided over TLS encrypted connections. TLS ensures that third-parties can't eavesdrop or modify the data while it's in transit between your service and the clients accessing your service. You cannot use unencrypted plain text connections. Communication between virtual machines within Managed Service for TimescaleDB is secured with either TLS or IPsec. You cannot use unencrypted plaintext connections. Virtual machines network interfaces are protected by a dynamically configured firewall based on iptables, which only allows connections from specific addresses. This is used for network traffic from the internal network to other VMs in the same service, and for external public network, to client connections. By default, new services accept incoming traffic from all sources, which is used to simplify initial set up of your service. It is highly recommended that you restrict the IP addresses that are allowed to establish connections to your services. ### Configure allowed incoming IP addresses for your service [Section titled “Configure allowed incoming IP addresses for your service”](#configure-allowed-incoming-ip-addresses-for-your-service) 1. **Select the service** In [MST Console](https://portal.managed.timescale.com/login), select the service to update. 2. **Check the port number** In `Overview` check the `Port` number. This is the port that you are managing inbound access for. 3. **Set public IP filters** In `Network`, check `IP filters`. The default value is `Open for all`. Click the ellipsis (...) to the right of Network, then select `Set public IP filters`. 4. **Set allowed inbound IP addresses** Set the `Allowed inbound IP addresses`: ![Setting allowed IP addresses for MST services](/docs/_astro/set-allowed-ip-addresses.BscecODh_22lu0h.webp) ## Networking with VPC peering [Section titled “Networking with VPC peering”](#networking-with-vpc-peering) When you set up VPC peering, you cannot access your services using public internet-based access. Service addresses are published in the public DNS record, but they can only be connected to from your peered VPC network using private network addresses. The virtual machines providing your service are hosted by cloud provider accounts controlled by Tiger Data. ## Customer data privacy [Section titled “Customer data privacy”](#customer-data-privacy) Customer data privacy is of utmost importance at Tiger Data. Tiger Data works with Aiven to provide Managed Service for TimescaleDB. In most cases, all the resources required for providing your services are automatically created, maintained, and terminated by the Managed Service for TimescaleDB infrastructure, with no manual operator intervention required. The Tiger Data Operations Team are able to securely log in to your service Virtual Machines, for the purposes of troubleshooting, as required. Tiger Data operators never access customer data unless you explicitly request them to do so, to troubleshoot a technical issue. This access is logged and audited. There is no ability for any customer or member of the public to access any virtual machines used in Managed Service for TimescaleDB. Managed Service for TimescaleDB services are periodically assessed and penetration tested for any security issues by an independent professional cyber-security vendor. Aiven is fully GDPR-compliant, and has executed data processing agreements (DPAs) with relevant cloud infrastructure providers. If you require a DPA, or if you want more information about information security policies, [contact Tiger Data](https://www.tigerdata.com/contact/). # Troubleshooting > Troubleshoot common issues with Managed Service for TimescaleDB ## Permission denied when changing ownership of tables and hypertables **Message:** ```plaintext ERROR: permission denied for schema _timescaledb_internal ``` You might see this error when using the `ALTER TABLE` command to change the ownership of tables or hypertables. This use of `ALTER TABLE` is blocked because the `tsdbadmin` user is not a superuser. To change table ownership, use the [`REASSIGN`](https://www.postgresql.org/docs/current/sql-reassign-owned.html) command instead: ```sql REASSIGN OWNED BY TO ; ``` ## Corrupted unique index has duplicated rows **Message:** ```plaintext ERROR: could not create unique index DETAIL: Table contains duplicated values. ``` When you try to rebuild an index with `REINDEX` it fails because of conflicting duplicated rows. To identify conflicting duplicate rows, you need to run a query that counts the number of rows for each combination of columns included in the index definition. For example, this `route` table has a `unique_route_index` index defining unique rows based on the combination of the `source` and `destination` columns: ```sql CREATE TABLE route( source TEXT, destination TEXT, description TEXT ); CREATE UNIQUE INDEX unique_route_index ON route (source, destination); ``` If the `unique_route_index` is corrupt, you can find duplicated rows in the `route` table using this query: ```sql SELECT source, destination, count FROM (SELECT source, destination, COUNT(*) AS count FROM route GROUP BY source, destination) AS foo WHERE count > 1; ``` The query groups the data by the same `source` and `destination` fields defined in the index, and filters any entries with more than one occurrence. Resolve the problematic entries in the rows by manually deleting or merging the entries until no duplicates exist. After all duplicate entries are removed, you can use the `REINDEX` command to rebuild the index. ## Reset password It happens to us all, you want to log in to MST Console, and the password is somewhere next to your keys, wherever they are. To reset your password: 1. Open [MST Console](https://portal.managed.timescale.com/login). 2. Click `Forgot password`. 3. Enter your email address, then click `Reset password`. A secure reset password link is sent to the email associated with this account. Click the link and update your password. ## Service is running low on disk, memory, or CPU **Message:** ```plaintext Your Managed Service for TimescaleDB service, in project "ExampleAccount", is running low on CPU. Running low on CPU affects performance and could affect service availability. Please either optimize your usage pattern or reduce the workload, and consider upgrading to a larger plan to avoid service outage. ``` When your database reaches 90% of your allocated disk, memory, or CPU resources, an automated message with the text above is sent to your email address. You can resolve this by logging in to your Managed Service for TimescaleDB account and changing your plan to increase your available resources: 1. **Select the service** From the Managed Service for TimescaleDB Dashboard, select the MST service that you want to increase resources for. 2. **Change the plan** Click `Service Settings` > `Service summary` > `Change Plan`. Select the suitable plan and click `Change Plan`. If you run out of resources regularly, you might need to consider using your resources more efficiently. Consider enabling [hypercore](/docs/build/columnar-storage/setup-hypercore), using [continuous aggregates](/docs/build/continuous-aggregates/create-a-continuous-aggregate), or [configuring data retention](/docs/learn/data-lifecycle/data-retention/about-data-retention) to reduce the amount of resources your database uses. ## Problem resolving DNS Managed Service for TimescaleDB services require a DNS record. When you launch a new MST service the DNS record is created, and it can take some time for the new name to propagate to DNS servers around the world. If you move an existing service to a new Cloud provider or region, the MST service is rebuilt in the new region in the background. When the MST service has been rebuilt in the new region, the DNS records are updated. This could cause a short interruption to your MST service while the DNS changes are propagated. If you are unable to resolve DNS, wait a few minutes and try again. ## PostgreSQL transaction ID wraparound The transaction control mechanism in PostgreSQL assigns a transaction ID to every row that is modified in the database; these IDs control the visibility of that row to other concurrent transactions. The transaction ID is a 32-bit number where two billion IDs are always in the visible past and the remaining IDs are reserved for future transactions and are not visible to the running transaction. To avoid a transaction wraparound of old rows, PostgreSQL requires occasional cleanup and freezing of old rows. This ensures that existing rows are visible when more transactions are created. You can manually freeze the old rows by executing `VACUUM FREEZE`. It can also be done automatically using the `autovacuum` daemon when a configured number of transactions has been created since the last freeze point. In Managed Service for TimescaleDB, the transaction limit is set according to the size of the database, up to 1.5 billion transactions. This ensures 500 million transaction IDs are available before a forced freeze and avoids churning stable data in existing tables. To check your transaction freeze limits, you can execute `show autovacuum_freeze_max_age` in your PostgreSQL instance. When the limit is reached, `autovacuum` starts freezing the old rows. Some applications do not automatically adjust the configuration when the PostgreSQL settings change, which can result in unnecessary warnings. For example, PGHero's default settings alert when 500 million transactions have been created instead of alerting after 1.5 billion transactions. To avoid this, change the value of the `transaction_id_danger` setting from 1,500,000,000 to 500,000,000, to receive warnings when the transaction limit reaches 1.5 billion. # User management > Add and manage users in your Managed Service for TimescaleDB project You can add new users, and manage existing users, in Managed Service for TimescaleDB Console. New users can be added to an entire project, or a single MST service. ## Project members [Section titled “Project members”](#project-members) You can invite new users to join your project as project members. There are several roles available for project members: | Role | Invite more users | Modify billing information | Manage existing services | Start and stop services | View service information | | --------- | ----------------- | -------------------------- | ------------------------ | ----------------------- | ------------------------ | | Admin | ✅ | ✅ | ✅ | ✅ | ✅ | | Operator | ❌ | ❌ | ✅ | ✅ | ✅ | | Developer | ✅ | ❌ | ✅ | ❌ | ✅ | | Read-only | ❌ | ❌ | ❌ | ❌ | ✅ | Users who can manage existing services can create databases and connect to them, on a service that already exists. To create a new service, users need the start and stop services permission. ### Adding project members [Section titled “Adding project members”](#adding-project-members) 1. **Sign in to the MST Console** [Sign in](https://portal.managed.timescale.com/login) to your Managed Service for TimescaleDB Console. 2. **Navigate to `Members`** Check that you are in the project that you want to change the members for, and click `Members`. 3. **Add a new member** In the `Project members` page, type the email address of the member you want to add, and select a role for the member. 4. **Send the invitation** Click `Send invitation`. The new user is sent an email inviting them to the project, and the invite shows in the `Pending invitations` list. You can click `Withdraw invitation` to remove an invitation before it has been accepted. 5. **Manage members** When they accept the invitation, the user details show in the `Members` list. You can edit a member role by selecting a new role in the list. You can delete a member by clicking the delete icon in the list. ## Service users [Section titled “Service users”](#service-users) By default, when you create a new service, a new `tsdbadmin` user is created. This is the user that you use to connect to your new service. The `tsdbadmin` user is the owner of the database, but is not a superuser. To access features requiring a superuser, log in as the `postgres` user instead. The `tsdbadmin` user for Managed Service for TimescaleDBs can: * Create a database * Create a role * Perform replication * Bypass row level security (RLS) This allows you to use the `tsdbadmin` user to create another user with any other roles. For a complete list of roles available, see the [PostgreSQL role attributes documentation](https://www.postgresql.org/docs/current/role-attributes.html). Important Your service must be running before you can manage users. ### Adding service users [Section titled “Adding service users”](#adding-service-users) 1. **Sign in to the MST Console** [Sign in](https://portal.managed.timescale.com/login) to Managed Service for TimescaleDB Console. By default, you start in the `Services` view, showing any services you currently have in your project. 2. **Select the service** Click the name of the service that you want to add users to. 3. **Add a service user** Select `Users`, then click `Add service user`: ![Adding a new MST service user](/docs/_astro/create-service-user.BqFwUdaM_ZGTk3U.webp) 4. **Configure the user** In the `Username` field, type a name for your user. If you want to allow the user to be replicated, toggle `Allow replication`. Click `Add service user` to save the user. 5. **View the new user** The new user shows in the `Username` list. To view the password, click the eye icon. Use the options in the list to change the replication setting and password, or delete the user. ## Multi-factor user authentication [Section titled “Multi-factor user authentication”](#multi-factor-user-authentication) You can use multi-factor authentication (MFA) to log in to Managed Service for TimescaleDB Console. This requires an authentication code, provided by the Google Authenticator app on your mobile device. You can see which authentication method is in use by each member of your Managed Service for TimescaleDB project. From the dashboard, navigate to the `Members` section. Each member is listed in the table with an authentication method of either `Password` or `Two-Factor`. Before you begin, install the Google Authenticator app on your mobile device. For more information, and installation instructions, see [the Google Authenticator documentation](https://support.google.com/accounts/answer/1066447). ### Configuring multi-factor authentication [Section titled “Configuring multi-factor authentication”](#configuring-multi-factor-authentication) 1. **Sign in to the MST Console** [Sign in](https://portal.managed.timescale.com/login) to Managed Service for TimescaleDB Console. 2. **Open `User profile`** Click the `User information` icon in the top-right of the dashboard to go to the `User profile` section. 3. **Enable two-factor authentication** In the `Authentication` tab, toggle `Two-factor authentication` to `Enabled`, and enter your password. 4. **Scan the QR code** On your mobile device, open the Google Authenticator app, tap `+` and select `Scan a QR code`. Scan the QR code provided by Managed Service for TimescaleDB. 5. **Confirm the code** In your MST dashboard, enter the confirmation code provided by the Google Authenticator app, and click `Enable Two-Factor Auth`. Warning If you lose access to the mobile device you use for multi-factor authentication, you cannot sign in to your Managed Service for TimescaleDB account. To regain access to your account, on the login screen, click `Forgot password?` and follow the step to reset your password. When you have regained access to your account, reconfigure multi-factor authentication. ## User authentication tokens [Section titled “User authentication tokens”](#user-authentication-tokens) Every time a registered user logs in, Managed Service for TimescaleDB creates a new authentication token. This occurs for login events using the portal, and using the API. By default, authentication tokens expire after 30 days, but the expiry date is adjusted every time the token is used. This means that tokens can be used indefinitely, if the user logs in at least every 30 days. You can see the list of all current authentication tokens in the Managed Service for TimescaleDB dashboard. Sign in to your account, and click the `User information` icon in the top-right of the dashboard to go to the `User profile` section. In the `Authentication` tab, the table lists all current authentication tokens. When you make authentication changes, such as enabling two factor authentication or resetting a password, all existing tokens are revoked. In some cases, a new token is immediately created so that the web console session remains valid. You can also manually revoke authentication tokens from the `User profile` page individually, or click `Revoke all tokens` to revoke all current tokens. Additionally, you can click `Generate token` to create a new token. When you generate a token on this page, you can provide a description, maximum age, and an extension policy. Generating authentication tokens in this way allows you to use them with monitoring applications that make automatic API calls to Managed Service for TimescaleDB. Warning There is a limit to how many valid authentication tokens are allowed per user. This limit is different for tokens that are created as a result of a sign in operation, and for tokens created explicitly. For automatically created tokens, the system automatically deletes the oldest tokens as new ones are created. For explicitly created tokens, older tokens are not deleted unless they expire or are manually revoked. This can result in explicitly created tokens that stop working, even though they haven't expired or been revoked. To avoid this, make sure you sign out at the end of every user session, instead of just discarding your authentication token. This is especially important for automation tools that automatically sign in. # Viewing service logs > View logs for your services in Managed Service for TimescaleDB by using MST Console or the command-line tool Occasionally there is a need to inspect logs from Managed Service for TimescaleDB. For example, to debug query performance or inspecting errors caused by a specific workload. There are different built-in ways to inspect service logs at Managed Service for TimescaleDB: * When you select a specific service, navigate to the `Logs` tab to see recent events. Logs can be browsed back in time. * Download logs using the [command-line client](https://github.com/aiven/aiven-client) by running: ```bash avn service logs -S desc -f --project ``` * [REST API](/docs/deploy/mst/restapi) endpoint is available for fetching the same information the two above methods output, in case programmatic access is needed. Service logs included on the normal service price are stored only for a few days. Unless you are using logs integration to another service, older logs are not accessible. # VPC peering > Secure your Managed Service for TimescaleDB service with VPC peering Virtual Private Cloud (VPC) peering is a method of connecting separate Cloud private networks to each other. It makes it possible for the virtual machines in the different VPCs to talk to each other directly without going through the public internet. VPC peering is limited to VPCs that share the same Cloud provider. VPC peering setup is a per project and per region setting. This means that all MST services created and running utilize the same VPC peering connection. If needed, you can have multiple projects that peer with different connections. Note Managed Service for TimescaleDB services are only accessible using your VPC's internal network. They are not accessible from the public internet. TLS certificates for VPC peered MST services are signed by the MST project CA and cannot be validated against a public CA (Let's Encrypt). You can choose whether you want to run on a VPC peered network or on the public internet for every MST service. You can set up VPC peering on: * [Amazon Web Services (AWS)](/docs/deploy/mst/vpc-peering/vpc-peering-aws) * [Google Cloud Platform (GCP)](/docs/deploy/mst/vpc-peering/vpc-peering-gcp) * [Microsoft Azure](/docs/deploy/mst/vpc-peering/vpc-peering-azure) # Configure VPC peering > Configure VPC peering for your services running in the Managed Service for TimescaleDB Console You can configure VPC peering for your Managed Service for TimescaleDB project, using the VPC section of the dashboard for your project. VPC peering setup is a per project and per region setting. This means that all MST services created and running utilize the same VPC peering connection. If needed, you can have multiple projects that peer with different connections. Only Admin and operator user roles can create a VPC. 1. **Open the VPC section** In [MST Console](https://portal.managed.timescale.com/login), click `VPC`. 2. **Create a new VPC** Click `Create VPC`. 3. **Choose a cloud provider** Choose a cloud provider in the `Cloud` list. 4. **Set the IP range** In the `IP range` field, type the IP range that you want to use for the VPC connection. Use an IP range that does not overlap with any networks that you want to connect through VPC peering. For example, if your own networks use the range 10.0.0.0/8, you could set the range for your Managed Service for TimescaleDB project VPC to 192.168.0.0/24. 5. **Confirm** Click `Create VPC`. The state of the VPC is listed in the table. # VPC peering on AWS > Secure your Managed Service for TimescaleDB instance with VPC peering on AWS You can configure VPC peering for your Managed Service for TimescaleDB project using VPC on AWS. ## Before you begin [Section titled “Before you begin”](#before-you-begin) * Set up a VPC peering for your project in MST. See [Configure VPC peering](/docs/deploy/mst/vpc-peering/vpc-peering). * In your AWS console, go to `My Account` and make a note of your `account ID`. * In your AWS console, go to `Peering connections`, find the VPC that you want to connect, and make a note of the ID for that VPC. ## Configuring a VPC peering [Section titled “Configuring a VPC peering”](#configuring-a-vpc-peering) 1. **Select the VPC connection** In [MST Console](https://portal.managed.timescale.com/login), click `VPC` and select the VPC connection that you created. 2. **Enter your AWS account ID** Type the account ID of your AWS account in `AWS Account ID`. 3. **Enter your AWS VPC ID** Type the ID of the VPC in AWS in `AWS VPC ID`. 4. **Add the peering connection** Click `Add peering connection`. A new connection with a status of `Pending Acceptance` is listed in your AWS console. Verify that the account ID and VPC ID match those listed in MST Console. 5. **Accept the request in AWS** In the AWS console, go to `Actions` and select `Accept Request`. Update your AWS route tables to match your Aiven CIDR settings. After you accept the request in AWS Console, the peering connection is active in the MST portal. # AWS Transit Gateway > Secure your Managed Service for TimescaleDB instance with Transit Gateway on AWS AWS Transit Gateway (TGW) enables transitive routing from on-premises networks through VPN and from other VPC. By creating a Transit Gateway VPC attachment, MST services in an MST Project VPC can route traffic to all other networks attached - directly or indirectly - to the Transit Gateway. ## Before you begin [Section titled “Before you begin”](#before-you-begin) * Set up a [VPC peering for your project in MST](/docs/deploy/mst/vpc-peering/vpc-peering). * In your AWS console, go to `My Account` and make a note of your `account ID`. * In your AWS console, go to `Transit Gateways`, find the transit gateway that you want to attach, and make a note of the ID. ## Attaching a VPC to an AWS Transit Gateway [Section titled “Attaching a VPC to an AWS Transit Gateway”](#attaching-a-vpc-to-an-aws-transit-gateway) 1. **Select the VPC connection** In [MST Console](https://portal.managed.timescale.com/login), click `VPC` and select the VPC connection that you created. 2. **Select `Transit Gateway VPC Attachment`** In the `VPC Peering connections` page select `Transit Gateway VPC Attachment`. 3. **Enter your AWS account ID** Type the account ID of your AWS account in `AWS Account ID`. 4. **Enter the Transit Gateway ID** Type the ID of the Transit Gateway of AWS in `Transit Gateway ID`. 5. **Set the IP range** Type the IP range in the `Network cidrs` field. Each Transit Gateway has a route table of its own, and by default routes traffic to each attached network directly to attached VPCs or indirectly through VPN attachments. The attached VPCs' route tables need to be updated to include the TGW as a target for any IP range (CIDR) that should be routed using the VPC attachment. These IP ranges must be configured when creating the attachment for an MST Project VPC. 6. **Add the peering connection** Click `Add peering connection`. A new connection with a status of `Pending Acceptance` is listed in your AWS console. Verify that the account ID and transit gateway ID match those listed in MST Console. 7. **Accept the request in AWS** In the AWS console, go to `Actions` and select `Accept Request`. Update your AWS route tables to match your Managed Service for TimescaleDB CIDR settings. After you accept the request in AWS Console, the peering connection is active in the MST Console. # VPC peering on Azure > Secure your Managed Service for TimescaleDB instance with VPC peering on Azure You can configure VPC peering for your Managed Service for TimescaleDB project using VPC on Azure. ## Before you begin [Section titled “Before you begin”](#before-you-begin) * Install the [Aiven Client](/docs/deploy/mst/aiven-client). * Sign in to Managed Service for TimescaleDB Console. * Set up a VPC peering for your project in MST. See [Configure VPC peering](/docs/deploy/mst/vpc-peering/vpc-peering). ## Configuring a VPC peering on Azure [Section titled “Configuring a VPC peering on Azure”](#configuring-a-vpc-peering-on-azure) 1. **Log in with an Azure admin account** Log in with an Azure administration account, using the Azure CLI: ```bash az account clear az login ``` This should open a window in your browser prompting you to choose an Azure account to log in with. You need an account with at least the Application administrator role to create VPC peering. If you manage multiple Azure subscriptions, configure the Azure CLI to default to the correct subscription using the command: ```bash az account set --subscription ``` 2. **Create an application object** Create an application object in your AD tenant, using the Azure CLI: ```bash az ad app create --display-name "" --sign-in-audience AzureADMultipleOrgs --key-type Password ``` This creates an entity to your AD that can be used to log into multiple AD tenants (`--sign-in-audience AzureADMultipleOrgs`), but only the home tenant (the tenant the app was created in) has the credentials to authenticate the app. Save the `appId` field from the output - this is referred to as `$user_app_id`. 3. **Create a service principal** Create a service principal for your app object. Ensure that the service principal is created to the Azure subscription containing the VNet you wish to peer: ```bash az ad sp create --id $user_app_id ``` This creates a service principal to your subscription that may have permissions to peer your VNet. Save the `objectId` field from the output - this is referred to as `$user_sp_id`. 4. **Set a password for your app object** ```bash az ad app credential reset --id $user_app_id ``` Save the password field from the output - this is referred to as `$user_app_secret`. 5. **Find your virtual network ID properties** ```bash az network vnet list ``` Make a note of these: * The `id` field, which is referred to as `$user_vnet_id` * The Azure Subscription ID, which is the part after `/subscriptions/` in the resource ID. This is referred to as `$user_subscription_id`. * The resource group name or the `resourceGroup` field in the output. This is referred to as `$user_resource_group`. * The VNet name or the `name` field from the output as `$user_vnet_name`. The `$user_vnet_id` should have the format: `/subscriptions/$user_subscription_id/resourceGroups/$user_resource_group/providers/Microsoft.Network/virtualNetworks/$user_vnet_name`. 6. **Grant your service principal permissions to peer** The service principal that you created needs to be assigned a role that has permission for the `Microsoft.Network/virtualNetworks/virtualNetworkPeerings/write` action on the scope of your VNet. To limit the permissions granted to the app object and service principal, you can create a custom role with just that permission. The built-in `Network Contributor` role includes that permission, and can be found using `az role definition list --name "Network Contributor"`. The `id` field from the output is used as `$network_contributor_role_id` to assign the service principal that role: ```bash az role assignment create --role $network_contributor_role_id --assignee-object-id $user_sp_id --scope $user_vnet_id ``` This allows the application object to manage the network in the `--scope`. Because you control the application object, it may also be given permission for the scope of an entire resource group, or the whole subscription to allow creating other peerings later without assigning the role again for each VNet separately. 7. **Create a service principal for the MST application object** The Managed Service for TimescaleDB AD tenant contains an application object similar to the one you created, and Managed Service for TimescaleDB uses it to create a peering from the Project VPC VNet in Managed Service for TimescaleDB to the VNet in Azure. For this, the Managed Service for TimescaleDB app object needs a service principal in your subscription: ```bash az ad sp create --id ``` Save the `objectId` field from the output - it is referred to as `$aiven_sp_id`. If this fails with the error "When using this permission, the backing application of the service principal being created must in the local tenant" then your account does not have the correct permissions. Use an account with at least the Application administrator role assigned. 8. **Create a custom role for the MST application object** The Managed Service for TimescaleDB application now has a service principal that can be given permissions. In order to target a network in your subscription with a peering and nothing else, you can create a custom role definition, with only a single action allowing to do that and only that: ```bash az role definition create --role-definition '{"Name": "", "Description": "Allows creating a peering to vnets in scope (but not from)", "Actions": ["Microsoft.Network/virtualNetworks/peer/action"], "AssignableScopes": ["/subscriptions/'$user_subscription_id'"]}' ``` Creating a custom role must include your subscription's id in `AssignableScopes`. This in itself does not give permissions to your subscription - it merely restricts which scopes a role assignment can include. Save the `id` field from the output - this is referred to as `$aiven_role_id`. 9. **Assign the custom role to the MST service principal** Assign the role that you created in the previous step to the Managed Service for TimescaleDB service principal with the scope of your VNet: ```bash az role assignment create --role $aiven_role_id --assignee-object-id $aiven_sp_id --scope $user_vnet_id ``` 10. **Get your Azure AD tenant ID** ```bash az account list ``` Make note of the `tenantId` field from the output. It is referred to as `$user_tenant_id`. 11. **Create a peering connection using Aiven CLI** ```bash avn vpc peering-connection create --project-vpc-id $aiven_project_vpc_id --peer-cloud-account $user_subscription_id --peer-resource-group $user_resource_group --peer-vpc $user_vnet_name --peer-azure-app-id $user_app_id --peer-azure-tenant-id $user_tenant_id ``` `$aiven_project_vpc_id` is the ID of the Managed Service for TimescaleDB project VPC, and can be found using the `avn vpc list` command. Managed Service for TimescaleDB creates a peering from the VNet in the Managed Service for TimescaleDB Project VPC to the VNet in your subscription. In addition, it creates a service principal for the application object in your tenant `--peer-azure-app-id $user_app_id`, giving it permission to target the Managed Service for TimescaleDB subscription VNet with a peering. Your AD tenant ID is also needed in order for the Managed Service for TimescaleDB application object to authenticate with your tenant to give it access to the service principal that you created `--peer-azure-tenant-id $user_tenant_id`. Ensure that the arguments starting with `$user_` are in lower case. Azure resource names are case-agnostic, but the Aiven API currently only accepts names in lower case. If no error is shown, the peering connection is being set up by Managed Service for TimescaleDB. 12. **Wait for PENDING\_PEER state** Run the following command until the state is no longer `APPROVED`, but `PENDING_PEER`: ```bash avn vpc peering-connection get -v --project-vpc-id $aiven_project_vpc_id --peer-cloud-account $user_subscription_id --peer-resource-group $user_resource_group --peer-vpc $user_vnet_name ``` A state such as `INVALID_SPECIFICATION` or `REJECTED_BY_PEER` may be shown if the VNet specified did not exist, or the Managed Service for TimescaleDB app object wasn't given permissions to peer with it. If that occurs, check your configuration and then recreate the peering connection. If everything went as expected, the state changes to `PENDING_PEER` within a couple of minutes showing details to set up the peering connection from your VNet to the Project VPC's VNet in Managed Service for TimescaleDB. Save the `to-tenant-id` field in the output. It is referred to as the `aiven_tenant_id`. The `to-network-id` field from the output is referred to as the `$aiven_vnet_id`. 13. **Log out and log in as the application object** 1. Log out the Azure user: ```bash az account clear ``` 2. Log in the application object you created to your AD tenant: ```bash az login --service-principal -u $user_app_id -p $user_app_secret --tenant $user_tenant_id ``` 3. Log in the same application object to the Managed Service for TimescaleDB AD tenant: ```bash az login --service-principal -u $user_app_id -p $user_app_secret --tenant $aiven_tenant_id ``` Now your application object has a session with both AD tenants. 14. **Create a peering from your VNet** ```bash az network vnet peering create --name --remote-vnet $aiven_vnet_id --vnet-name $user_vnet_name --resource-group $user_resource_group --subscription $user_subscription_id --allow-vnet-access ``` If you do not specify `--allow-vnet-access` no traffic is allowed to flow from the peered VNet and Managed Service for TimescaleDB services cannot be reached through the peering. After the peering has been created, the peering should be in the state `connected`. In case you get an authorization error, it's possible the role assignment hasn't taken effect yet. If that is the case, try logging in again and creating the peering again after waiting a bit by repeating the commands in this step. If the error message persists, check the role assignment was correct. 15. **Verify the peering connection is active** In the Aiven CLI, check if the peering connection is `ACTIVE`: ```bash avn vpc peering-connection get -v --project-vpc-id $aiven_project_vpc_id --peer-cloud-account $user_subscription_id --peer-resource-group $user_resource_group --peer-vpc $user_vnet_name ``` Managed Service for TimescaleDB polls peering connections in state `PENDING_PEER` regularly to see if your subscription has created a peering connection to the Managed Service for TimescaleDB Project VPC's VNet. After this is detected, the state changes from `PENDING_PEER` to `ACTIVE`. After this, services in the Project VPC can be reached through the peering. # VPC peering on GCP > Secure your Managed Service for TimescaleDB instance with VPC peering on GCP You can configure VPC peering for your Managed Service for TimescaleDB project using VPC provided by GCP. ## Before you begin [Section titled “Before you begin”](#before-you-begin) * Set up a VPC peering for your project in MST. See [Configure VPC peering](/docs/deploy/mst/vpc-peering/vpc-peering). * In your GCP console, click the project name and make a note of the `Project ID`. * In your GCP console, go to `VPC Networks`, find the VPC that you want to connect, and make a note of the network name for that VPC. ## Configuring a VPC peering on GCP [Section titled “Configuring a VPC peering on GCP”](#configuring-a-vpc-peering-on-gcp) 1. **Select the VPC connection** In [MST Console](https://portal.managed.timescale.com/login), click `VPC` and select the VPC connection that you created. 2. **Enter your GCP project ID** Type the project ID of your GCP project in `GCP Project ID`. 3. **Enter your GCP VPC network name** Type the network name of the VPC in GCP in `GCP VPC network name`. 4. **Add the peering connection** Click `Add peering connection`. A new connection with a status of `Pending Peer` is listed in your GCP console. Make a note of the project name and the network name. 5. **Create the peering in GCP** In the GCP console, go to `VPC` > `VPC network peering` and select `Create Connection`. Type a name for the peering connection and type the project ID and network name that you made a note of. Click `Create`. After the peering is successful, it is active in both MST Console and your GCP console. # Self-hosted TimescaleDB > Install, configure, and manage self-hosted TimescaleDB on your own infrastructure ## Getting started [Section titled “Getting started”](#getting-started) [Install TimescaleDB](/docs/get-started/choose-your-path/install-timescaledb) [Install self-hosted TimescaleDB on your platform](/docs/get-started/choose-your-path/install-timescaledb) [Configure](/docs/deploy/self-hosted/configuration) [Configure TimescaleDB and PostgreSQL](/docs/deploy/self-hosted/configuration) [Migrate to TimescaleDB](/docs/deploy/self-hosted/migration) [Migrate data from PostgreSQL, another TimescaleDB instance, or InfluxDB](/docs/deploy/self-hosted/migration) ## Administration [Section titled “Administration”](#administration) [Upgrade](/docs/deploy/self-hosted/upgrades) [Upgrade your self-hosted TimescaleDB installation](/docs/deploy/self-hosted/upgrades) [Uninstall](/docs/deploy/self-hosted/uninstall) [Remove TimescaleDB from your system](/docs/deploy/self-hosted/uninstall) [Manage storage](/docs/deploy/self-hosted/manage-storage) [Manage storage using tablespaces](/docs/deploy/self-hosted/manage-storage) [Additional tooling](/docs/deploy/self-hosted/tooling) [TimescaleDB Tune and Toolkit](/docs/deploy/self-hosted/tooling) [Backup and restore](/docs/deploy/self-hosted/backup-and-restore) [Backup and restore your database](/docs/deploy/self-hosted/backup-and-restore) [Replication and HA](/docs/deploy/self-hosted/replication-and-ha) [Configure replication and high availability](/docs/deploy/self-hosted/replication-and-ha) ## Troubleshooting [Section titled “Troubleshooting”](#troubleshooting) [Troubleshoot](/docs/deploy/self-hosted/troubleshooting) [Resolve common self-hosted issues](/docs/deploy/self-hosted/troubleshooting) # Backup and restore > Back up and restore your self-hosted TimescaleDB instance using logical or physical backups TimescaleDB takes advantage of the reliable backup and restore functionality provided by PostgreSQL. There are a few different mechanisms you can use to back up your self-hosted TimescaleDB database: * [Logical backup](/docs/deploy/self-hosted/backup-and-restore/logical-backup) with `pg_dump` and `pg_restore`. * [Physical backup](/docs/deploy/self-hosted/backup-and-restore/physical) with `pg_basebackup` or another tool. # Logical backup with pg_dump and pg_restore > Back up and restore a hypertable or an entire database using native PostgreSQL commands You back up and restore each self-hosted PostgreSQL database with TimescaleDB enabled using the native PostgreSQL [`pg_dump`](https://www.postgresql.org/docs/current/app-pgdump.html) and [`pg_restore`](https://www.postgresql.org/docs/current/app-pgrestore.html) commands. This also works for compressed hypertables, you don't have to decompress the chunks before you begin. If you are using `pg_dump` to backup regularly, make sure you keep track of the versions of PostgreSQL and TimescaleDB you are running. This page shows you how to: * [Back up and restore an entire database](#back-up-and-restore-an-entire-database) * [Back up and restore individual hypertables](#back-up-and-restore-individual-hypertables) You can also [upgrade between different versions of TimescaleDB](/docs/deploy/self-hosted/upgrades). ## Prerequisites for this procedure To follow these steps, you'll need: * A source database to backup from, and a target database to restore to. * The `psql` and `pg_dump` PostgreSQL client tools installed on your migration machine. ## Back up and restore an entire database [Section titled “Back up and restore an entire database”](#back-up-and-restore-an-entire-database) You backup and restore an entire database using `pg_dump` and `psql`. 1. **Set your connection strings** These variables hold the connection information for the source database to backup from and the target database to restore to: ```bash export SOURCE=postgres://:@:/ export TARGET=postgres://:@: ``` 2. **Back up your database** ```bash pg_dump -d "$SOURCE" \ -Fc -f .bak ``` You may see some errors while `pg_dump` is running. These can usually be safely ignored. 3. **Restore your database from the backup** 1. Connect to your target database: ```bash psql -d "$TARGET" ``` 2. Create a new database and enable TimescaleDB: ```sql CREATE DATABASE ; \c CREATE EXTENSION IF NOT EXISTS timescaledb; ``` 3. Put your database in the right state for restoring: ```sql SELECT timescaledb_pre_restore(); ``` 4. Restore the database: ```sql pg_restore -Fc -d .bak ``` 5. Return your database to normal operations: ```sql SELECT timescaledb_post_restore(); ``` Do not use `pg_restore` with the `-j` option. This option does not correctly restore the TimescaleDB catalogs. ## Back up and restore individual hypertables [Section titled “Back up and restore individual hypertables”](#back-up-and-restore-individual-hypertables) `pg_dump` provides flags that allow you to specify tables or schemas to back up. However, using these flags means that the dump lacks necessary information that TimescaleDB requires to understand the relationship between them. Even if you explicitly specify both the hypertable and all of its constituent chunks, the dump would still not contain all the information it needs to recreate the hypertable on restore. To backup individual hypertables, backup the database schema, then backup only the tables you need. You also use this method to backup individual plain tables. 1. **Set your connection strings** These variables hold the connection information for the source database to backup from and the target database to restore to: ```bash export SOURCE=postgres://:@:/ export TARGET=postgres://:@:/ ``` 2. **Back up the database schema and individual tables** 1. Back up the hypertable schema: ```bash pg_dump -s -d $SOURCE --table > schema.sql ``` 2. Back up hypertable data to a CSV file. For each hypertable to backup: ```bash psql -d $SOURCE \ -c "\COPY (SELECT * FROM ) TO .csv DELIMITER ',' CSV" ``` 3. **Restore the schema to the target database** ```bash psql -d $TARGET < schema.sql ``` 4. **Restore hypertables from the backup** For each hypertable to backup: 1. Recreate the hypertable: ```bash psql -d $TARGET -c "SELECT create_hypertable(, )" ``` When you [create the new hypertable](/docs/reference/timescaledb/hypertables/create_hypertable), you do not need to use the same parameters as existed in the source database. This can provide a good opportunity for you to re-organize your hypertables if you need to. For example, you can change the partitioning key, the number of partitions, or the chunk interval sizes. 2. Restore the data: ```bash psql -d $TARGET -c "\COPY FROM .csv CSV" ``` The standard `COPY` command in PostgreSQL is single threaded. If you have a lot of data, you can speed up the copy using the [timescaledb-parallel-copy](https://github.com/timescale/timescaledb-parallel-copy). Best practice is to back up and restore a database at a time. However, if you have superuser access to PostgreSQL instance with TimescaleDB installed, you can use `pg_dumpall` to back up all PostgreSQL databases in a cluster, including global objects that are common to all databases, namely database roles, tablespaces, and privilege grants. You restore the PostgreSQL instance using `psql`. For more information, see the [PostgreSQL documentation](https://www.postgresql.org/docs/17/backup-dump.html#BACKUP-DUMP-ALL). # Physical backups > Make physical backups of your entire TimescaleDB instance with pg_basebackup or external backup and restore managers For full instance physical backups (which are especially useful for starting up new [replicas](/docs/deploy/self-hosted/replication-and-ha)), [`pg_basebackup`](https://www.postgresql.org/docs/current/app-pgbasebackup.html) works with all TimescaleDB installation types. You can also use any of several external backup and restore managers such as [`pg_backrest`](https://pgbackrest.org/), or [`barman`](https://pgbarman.org/). These tools all allow you to take online, physical backups of your entire instance, and many offer incremental backups and other automation options. # Configuration > Configure your self-hosted TimescaleDB instance By default, TimescaleDB uses the default PostgreSQL server configuration settings. However, in some cases, these settings are not appropriate, especially if you have larger servers that use more hardware resources such as CPU, memory, and storage. * [Learn about configuration](/docs/deploy/self-hosted/configuration/about-configuration) to understand how it works before you begin using it. * Use the [TimescaleDB tune tool](/docs/deploy/self-hosted/configuration/timescaledb-tune). * Manually edit the `postgresql.conf` [configuration file](/docs/deploy/self-hosted/configuration/postgres-config). * If you run TimescaleDB in a Docker container, configure [within Docker](/docs/deploy/self-hosted/configuration/docker-config). * Find out more about the [data that we collect](/docs/deploy/self-hosted/configuration/telemetry). # About configuration in TimescaleDB > Configure your TimescaleDB instance including settings related to memory, workers, disk writes, and transactional locks By default, TimescaleDB uses the default PostgreSQL server configuration settings. However, in some cases, these settings are not appropriate, especially if you have larger servers that use more hardware resources such as CPU, memory, and storage. This section explains some of the settings you are most likely to need to adjust. Some of these settings are PostgreSQL settings, and some are TimescaleDB specific settings. For most changes, you can use the [tuning tool](/docs/deploy/self-hosted/configuration/timescaledb-tune) to adjust your configuration. For more advanced configuration settings, or to change settings that aren't included in the `timescaledb-tune` tool, you can [manually adjust](/docs/deploy/self-hosted/configuration/postgres-config) the `postgresql.conf` configuration file. ## Memory [Section titled “Memory”](#memory) Settings: * `shared_buffers` * `effective_cache_size` * `work_mem` * `maintenance_work_mem` * `max_connections` You can adjust each of these to match the machine's available memory. To make it easier, you can use the [PgTune](https://pgtune.leopard.in.ua/) site to work out what settings to use: enter your machine details, and select the `data warehouse` DB type to see the suggested parameters. Tips You can adjust these settings with `timescaledb-tune`. ## Workers [Section titled “Workers”](#workers) Settings: * `timescaledb.max_background_workers` * `max_parallel_workers` * `max_worker_processes` PostgreSQL uses worker pools to provide workers for live queries and background jobs. If you do not configure these settings, your queries and background jobs could run more slowly. TimescaleDB background workers are configured with `timescaledb.max_background_workers`. Each database needs a background worker allocated to schedule jobs. Additional workers run background jobs as required. This setting should be the sum of the total number of databases and the total number of concurrent background workers you want running at any one time. By default, `timescaledb-tune` sets `timescaledb.max_background_workers` to 16. You can change this setting directly, use the `--max-bg-workers` flag, or adjust the `TS_TUNE_MAX_BG_WORKERS` [Docker environment variable](/docs/deploy/self-hosted/configuration/docker-config). TimescaleDB parallel workers are configured with `max_parallel_workers`. For larger queries, PostgreSQL automatically uses parallel workers if they are available. Increasing this setting can improve query performance for large queries that trigger the use of parallel workers. By default, this setting corresponds to the number of CPUs available. You can change this parameter directly, by adjusting the `--cpus` flag, or by using the `TS_TUNE_NUM_CPUS` [Docker environment variable](/docs/deploy/self-hosted/configuration/docker-config). The `max_worker_processes` setting defines the total pool of workers available to both background and parallel workers, as well a small number of built-in PostgreSQL workers. It should be at least the sum of `timescaledb.max_background_workers` and `max_parallel_workers`. Tips You can adjust these settings with `timescaledb-tune`. ## Disk writes [Section titled “Disk writes”](#disk-writes) Settings: * `synchronous_commit` By default, disk writes are performed synchronously, so each transaction must be completed and a success message sent, before the next transaction can begin. You can change this to asynchronous to increase write throughput by setting `synchronous_commit = 'off'`. Note that disabling synchronous commits could result in some committed transactions being lost. To help reduce the risk, do not also change `fsync` setting. For more information about asynchronous commits and disk write speed, see the [PostgreSQL documentation](https://www.postgresql.org/docs/current/wal-async-commit.html). Tips You can adjust these settings in the `postgresql.conf` configuration file. ## Transaction locks [Section titled “Transaction locks”](#transaction-locks) Settings: * `max_locks_per_transaction` TimescaleDB relies on table partitioning to scale time-series workloads. A hypertable needs to acquire locks on many chunks during queries, which can exhaust the default limits for the number of allowed locks held. In some cases, you might see a warning like this: ```sql psql: FATAL: out of shared memory HINT: You might need to increase max_locks_per_transaction. ``` To avoid this issue, you can increase the `max_locks_per_transaction` setting from the default value, which is usually 64. This parameter limits the average number of object locks used by each transaction; individual transactions can lock more objects as long as the locks of all transactions fit in the lock table. For most workloads, choose a number equal to double the maximum number of chunks you expect to have in a hypertable divided by `max_connections`. This takes into account that the number of locks used by a hypertable query is roughly equal to the number of chunks in the hypertable if you need to access all chunks in a query, or double that number if the query uses an index. You can see how many chunks you currently have using the [`timescaledb_information.hypertables`](/docs/reference/timescaledb/informational-views/hypertables) view. Changing this parameter requires a database restart, so make sure you pick a larger number to allow for some growth. For more information about lock management, see the [PostgreSQL documentation](https://www.postgresql.org/docs/current/runtime-config-locks.html). Tips **Tiger Cloud:** Adjust `max_locks_per_transaction` from Tiger Console under **Database configuration → Advanced parameters**. Search for the parameter, edit the value, and click **Apply changes and restart** (this parameter requires a restart). See [Advanced parameters](/docs/deploy/tiger-cloud/tiger-cloud-aws/configuration/advanced-parameters) for details. **Self-hosted:** Adjust this setting in the `postgresql.conf` configuration file and restart the database. # Configuration with Docker > Configure a TimescaleDB instance running in a Docker container If you are running TimescaleDB in a Docker container, there are two different ways to modify your PostgreSQL configuration. You can edit the PostgreSQL configuration file inside the Docker container, or you can set parameters at the command prompt. ## Edit the PostgreSQL configuration file inside Docker [Section titled “Edit the PostgreSQL configuration file inside Docker”](#edit-the-postgresql-configuration-file-inside-docker) You can start the Docker container, and then use a text editor to edit the PostgreSQL configuration file directly. The configuration file requires one parameter per line. Blank lines are ignored, and you can use a `#` symbol at the beginning of a line to denote a comment. 1. **Start your Docker instance** ```bash docker start timescaledb ``` 2. **Open a shell** ```bash docker exec -i -t timescaledb /bin/bash ``` 3. **Open the configuration file in your preferred text editor** ```bash vi /var/lib/postgresql/data/postgresql.conf ``` 4. **Restart the container to reload the configuration** ```bash docker restart timescaledb ``` ## Setting parameters at the command prompt [Section titled “Setting parameters at the command prompt”](#setting-parameters-at-the-command-prompt) If you don't want to open the configuration file to make changes, you can also set parameters directly from the command prompt inside your Docker container, using the `-c` option. For example: ```bash docker run -i -t timescale/timescaledb:latest-pg18 postgres -c max_wal_size=2GB ``` # Manual PostgreSQL configuration and tuning > Manually configure your self-hosted database using the PostgreSQL configuration file If you prefer to tune settings yourself, or for settings not covered by `timescaledb-tune`, you can manually configure your installation using the PostgreSQL configuration file. For some common configuration settings you might want to adjust, see the [about configuration](/docs/deploy/self-hosted/configuration/about-configuration) page. For more information about the PostgreSQL configuration page, see the [PostgreSQL documentation](https://www.postgresql.org/docs/current/config-setting.html). ## Edit the PostgreSQL configuration file [Section titled “Edit the PostgreSQL configuration file”](#edit-the-postgresql-configuration-file) The location of the PostgreSQL configuration file depends on your operating system and installation. 1. **Find the location of the config file for your instance** 1. Connect to your database: ```shell psql -d "postgres://:@:/" ``` 2. Retrieve the database file location from the database internal configuration. ```sql SHOW config_file; ``` PostgreSQL returns the path to your configuration file. For example: ```sql -------------------------------------------- /home/postgres/pgdata/data/postgresql.conf (1 row) ``` 2. **Open the config file, then edit your PostgreSQL configuration** See the [PostgreSQL documentation](https://www.postgresql.org/docs/current/config-setting.html) for details on available settings. ```shell vi /home/postgres/pgdata/data/postgresql.conf ``` 3. **Save your updated configuration** When you have saved the changes you make to the configuration file, the new configuration is not applied immediately. The configuration file is automatically reloaded when the server receives a `SIGHUP` signal. To manually reload the file, use the `pg_ctl` command. ## Setting parameters at the command prompt [Section titled “Setting parameters at the command prompt”](#setting-parameters-at-the-command-prompt) If you don't want to open the configuration file to make changes, you can also set parameters directly from the command prompt, using the `postgres` command. For example: ```sql postgres -c log_connections=yes -c log_destination='syslog' ``` # Telemetry and version checking > Learn what telemetry TimescaleDB collects and how to disable it TimescaleDB collects anonymous usage data to help us better understand and assist our users. It also helps us provide some services, such as automated version checking. In particular, the `UUID` (user ID) fields contain no identifying information, but are randomly generated by appropriately seeded random number generators. This is an example of the JSON data file that is sent for a specific deployment: Example JSON telemetry data file ```json { "access_methods": { "btree": { "instances": 214, "pages": 566 }, "heap": { "instances": 142, "pages": 620 } }, "build_architecture": "x86_64", "build_architecture_bit_size": 64, "build_os_name": "Linux", "build_os_version": "6.8.0-45-generic", "data_volume": 381903727, "db_metadata": {}, "db_telemetry_events": [], "db_uuid": "860c2be4-59a3-43b5-b895-5d9e0dd44551", "errors_by_sqlerrcode": {}, "exported_db_uuid": "5730161f-0d18-42fb-a800-45df33494c21", "functions_used": { "pg_catalog.count()": 1689, "pg_catalog.count(pg_catalog.\"any\")": 1600, "pg_catalog.current_database()": 2489, "pg_catalog.now()": 6, "pg_catalog.version()": 6852 }, "install_method": "docker", "installed_time": "2026-05-20T12:12:28+00", "license": { "edition": "community" }, "num_compression_policies": 1, "num_compression_policies_fixed": 0, "num_continuous_aggs_policies": 24, "num_continuous_aggs_policies_fixed": 0, "num_reorder_policies": 0, "num_reorder_policies_fixed": 0, "num_retention_policies": 0, "num_retention_policies_fixed": 0, "num_user_defined_actions": 1, "num_user_defined_actions_fixed": 0, "os_name": "Linux", "os_name_pretty": "Ubuntu 22.04.5 LTS", "os_release": "6.8.0-45-generic", "os_version": "#45-Ubuntu SMP PREEMPT_DYNAMIC Fri Aug 30 12:02:04 UTC 2024", "postgresql_version": "18.4", "related_extensions": { "ai": false, "pg_prometheus": false, "postgis": false, "promscale": false, "timescale_analytics": false, "timescaledb_toolkit": true, "vector": false, "vectorscale": false }, "relations": { "continuous_aggregates": { "compression": { "compressed_heap_size": 0, "compressed_indexes_size": 0, "compressed_row_count": 0, "compressed_row_count_frozen_immediately": 0, "compressed_toast_size": 0, "num_compressed_caggs": 0, "num_compressed_chunks": 0, "uncompressed_heap_size": 0, "uncompressed_indexes_size": 0, "uncompressed_row_count": 0, "uncompressed_toast_size": 0 }, "heap_size": 122404864, "indexes_size": 165044224, "num_caggs_finalized": 24, "num_caggs_nested": 0, "num_caggs_using_real_time_aggregation": 24, "num_children": 760, "num_relations": 24, "num_reltuples": 914704, "toast_size": 6225920 }, "hypertables": { "compression": { "compressed_heap_size": 3522560, "compressed_indexes_size": 11993088, "compressed_row_count": 4392, "compressed_row_count_frozen_immediately": 0, "compressed_toast_size": 20365312, "num_compressed_chunks": 366, "num_compressed_hypertables": 1, "uncompressed_heap_size": 41951232, "uncompressed_indexes_size": 42696704, "uncompressed_row_count": 421368, "uncompressed_toast_size": 2998272 }, "heap_size": 3522560, "indexes_size": 18022400, "num_children": 366, "num_relations": 2, "num_reltuples": 421368, "toast_size": 23379968 }, "materialized_views": { "heap_size": 0, "indexes_size": 0, "num_relations": 0, "num_reltuples": 0, "toast_size": 0 }, "partitioned_tables": { "heap_size": 0, "indexes_size": 0, "num_children": 0, "num_relations": 0, "num_reltuples": 0, "toast_size": 0 }, "tables": { "heap_size": 32768, "indexes_size": 98304, "num_relations": 4, "num_reltuples": 12, "toast_size": 16384 }, "views": { "num_relations": 5 } }, "replication": { "is_wal_receiver": false, "num_wal_senders": 0 }, "stats_by_job_type": { "policy_job_stat_history_retention": { "max_consecutive_crashes": 0, "max_consecutive_failures": 0, "total_crashes": 0, "total_duration": "00:00:00.517221", "total_duration_failures": "00:00:00", "total_failures": 0, "total_runs": 49, "total_successes": 49 }, "policy_telemetry": { "max_consecutive_crashes": 0, "max_consecutive_failures": 0, "total_crashes": 0, "total_duration": "00:00:01.280382", "total_duration_failures": "00:00:00", "total_failures": 0, "total_runs": 23, "total_successes": 23 } }, "telemetry_version": 2, "timescaledb_version": "2.27.0" } ``` If you want to see the exact JSON data file that is sent, use the [`get_telemetry_report`](/docs/reference/timescaledb/administration/get_telemetry_report) API call. Your use of either TimescaleDB Apache 2 Edition or TimescaleDB Community Edition is governed by our [Privacy Policy](https://www.tigerdata.com/legal/privacy). ## Change what is included in the telemetry report [Section titled “Change what is included in the telemetry report”](#change-what-is-included-in-the-telemetry-report) If you want to adjust which metadata is included or excluded from the telemetry report, you can do so in the `_timescaledb_catalog.metadata` table. Metadata which has `include_in_telemetry` set to `true`, and a value of `timescaledb_telemetry.cloud`, is included in the telemetry report. ## Version checking [Section titled “Version checking”](#version-checking) Telemetry reports are sent periodically in the background. In response to the telemetry report, the database receives the most recent version of TimescaleDB available for installation. This version is recorded in your server logs, along with any applicable out-of-date version warnings. You do not have to update immediately to the newest release, but we highly recommend that you do so, to take advantage of performance improvements and bug fixes. ## Disable telemetry [Section titled “Disable telemetry”](#disable-telemetry) It is highly recommended that you leave telemetry enabled, as it provides useful features for you, and helps to keep improving Timescale. However, you can turn off telemetry if you need to for a specific database, or for an entire instance. Note If you turn off telemetry, the version checking feature is also turned off. ### Disabling telemetry [Section titled “Disabling telemetry”](#disabling-telemetry) 1. **Open your PostgreSQL configuration file** Locate the `timescaledb.telemetry_level` parameter. See the [PostgreSQL configuration file](/docs/deploy/self-hosted/configuration/postgres-config) instructions for locating and opening the file. 2. **Change the parameter setting to off** ```yaml timescaledb.telemetry_level=off ``` 3. **Reload the configuration file** ```bash pg_ctl ``` Alternatively, you can use this command at the `psql` prompt, as the root user: ```sql ALTER [SYSTEM | DATABASE | USER] { *db_name* | *role_specification* } SET timescaledb.telemetry_level=off ``` This command disables telemetry for the specified system, database, or user. ### Enabling telemetry [Section titled “Enabling telemetry”](#enabling-telemetry) 1. **Open your PostgreSQL configuration file** Locate the `timescaledb.telemetry_level` parameter. See the [PostgreSQL configuration file](/docs/deploy/self-hosted/configuration/postgres-config) instructions for locating and opening the file. 2. **Change the parameter setting to basic** ```yaml timescaledb.telemetry_level=basic ``` 3. **Reload the configuration file** ```bash pg_ctl ``` Alternatively, you can use this command at the `psql` prompt, as the root user: ```sql ALTER [SYSTEM | DATABASE | USER] { *db_name* | *role_specification* } SET timescaledb.telemetry_level=basic ``` This command enables telemetry for the specified system, database, or user. # TimescaleDB configuration and tuning > Configure TimescaleDB settings related to policies, query planning and execution, and administration Just as you can tune settings in PostgreSQL, TimescaleDB provides a number of configuration settings that may be useful to your specific installation and performance needs. These can also be set within the `postgresql.conf` file or as command-line parameters when starting PostgreSQL. ## Query planning and execution [Section titled “Query planning and execution”](#query-planning-and-execution) ### `timescaledb.enable_chunkwise_aggregation (bool)` [Section titled “timescaledb.enable\_chunkwise\_aggregation (bool)”](#timescaledbenable_chunkwise_aggregation-bool) If enabled, aggregations are converted into partial aggregations during query planning. The first part of the aggregation is executed on a per-chunk basis. Then, these partial results are combined and finalized. Splitting aggregations decreases the size of the created hash tables and increases data locality, which speeds up queries. ### `timescaledb.vectorized_aggregation (bool)` [Section titled “timescaledb.vectorized\_aggregation (bool)”](#timescaledbvectorized_aggregation-bool) Enables or disables the vectorized optimizations in the query executor. For example, the `sum()` aggregation function on compressed chunks can be optimized in this way. ### `timescaledb.enable_merge_on_cagg_refresh (bool)` [Section titled “timescaledb.enable\_merge\_on\_cagg\_refresh (bool)”](#timescaledbenable_merge_on_cagg_refresh-bool) Set to `ON` to dramatically decrease the amount of data written on a continuous aggregate in the presence of a small number of changes, reduce the i/o cost of refreshing a continuous aggregate, and generate fewer Write-Ahead Logs (WAL). Only works for continuous aggregates that don't have compression enabled. Please refer to the [Grand Unified Configuration (GUC) parameters](/docs/reference/timescaledb/configuration/gucs) for a complete list. ## Policies [Section titled “Policies”](#policies) ### `timescaledb.max_background_workers (int)` [Section titled “timescaledb.max\_background\_workers (int)”](#timescaledbmax_background_workers-int) Max background worker processes allocated to TimescaleDB. Set to at least 1 + the number of databases loaded with the TimescaleDB extension in a PostgreSQL instance. Default value is 16. ## Self-hosted TimescaleDB tuning [Section titled “Self-hosted TimescaleDB tuning”](#self-hosted-timescaledb-tuning) ### `timescaledb.disable_load (bool)` [Section titled “timescaledb.disable\_load (bool)”](#timescaledbdisable_load-bool) Disable the loading of the actual extension. ## Administration [Section titled “Administration”](#administration) ### `timescaledb.restoring (bool)` [Section titled “timescaledb.restoring (bool)”](#timescaledbrestoring-bool) Set TimescaleDB in restoring mode. It is disabled by default. ### `timescaledb.license (string)` [Section titled “timescaledb.license (string)”](#timescaledblicense-string) Change access to features based on the TimescaleDB license in use. For example, setting `timescaledb.license` to `apache` limits TimescaleDB to features that are implemented under the Apache 2 license. The default value is `timescale`, which allows access to all features. To check which edition you are running, see [Compare TimescaleDB editions](/docs/get-started/choose-your-path/timescaledb-editions#check-your-installed-edition). ### `timescaledb.telemetry_level (enum)` [Section titled “timescaledb.telemetry\_level (enum)”](#timescaledbtelemetry_level-enum) Telemetry settings level. Level used to determine which telemetry to send. Can be set to `off` or `basic`. Defaults to `basic`. ### `timescaledb.last_tuned (string)` [Section titled “timescaledb.last\_tuned (string)”](#timescaledblast_tuned-string) Records last time `timescaledb-tune` ran. ### `timescaledb.last_tuned_version (string)` [Section titled “timescaledb.last\_tuned\_version (string)”](#timescaledblast_tuned_version-string) Version of `timescaledb-tune` used to tune when it runs. # TimescaleDB tuning tool > Use timescaledb-tune to automatically configure your TimescaleDB instance based on your system's resources To help make configuring TimescaleDB a little easier, you can use the [`timescaledb-tune`](https://github.com/timescale/timescaledb-tune) tool. This tool handles setting the most common parameters to good values based on your system. It accounts for memory, CPU, and PostgreSQL version. `timescaledb-tune` is packaged with the TimescaleDB binary releases as a dependency, so if you installed TimescaleDB from a binary release (including Docker), you should already have access to the tool. Alternatively, you can use the `go install` command to install it: ```bash go install github.com/timescale/timescaledb-tune/cmd/timescaledb-tune@latest ``` The `timescaledb-tune` tool reads your system's `postgresql.conf` file and offers interactive suggestions for your settings. Here is an example of the tool running: ```bash Using postgresql.conf at this path: /usr/local/var/postgres/postgresql.conf Is this correct? [(y)es/(n)o]: y Writing backup to: /var/folders/cr/example/T/timescaledb_tune.backup202101071520 shared_preload_libraries needs to be updated Current: #shared_preload_libraries = 'timescaledb' Recommended: shared_preload_libraries = 'timescaledb' Is this okay? [(y)es/(n)o]: y success: shared_preload_libraries will be updated Tune memory/parallelism/WAL and other settings? [(y)es/(n)o]: y Recommendations based on 8.00 GB of available memory and 4 CPUs for PostgreSQL 12 Memory settings recommendations Current: shared_buffers = 128MB #effective_cache_size = 4GB #maintenance_work_mem = 64MB #work_mem = 4MB Recommended: shared_buffers = 2GB effective_cache_size = 6GB maintenance_work_mem = 1GB work_mem = 26214kB Is this okay? [(y)es/(s)kip/(q)uit]: ``` When you have answered the questions, the changes are written to your `postgresql.conf` and take effect when you next restart. If you are starting on a fresh instance and don't want to approve each group of changes, you can automatically accept and append the suggestions to the end of your `postgresql.conf` by using some additional flags when you run the tool: ```bash timescaledb-tune --quiet --yes --dry-run >> /path/to/postgresql.conf ``` # Manage storage using tablespaces > Save on data storage by moving older chunks to cheaper, slower storage, while still using faster, more expensive storage for frequently accessed data If you are running TimescaleDB on your own hardware, you can save storage by moving chunks between tablespaces. By moving older chunks to cheaper, slower storage, you can save on storage costs while still using faster, more expensive storage for frequently accessed data. Moving infrequently accessed chunks can also improve performance, because it isolates historical data from the continual read-and-write workload of more recent data. Note Using tablespaces is one way to manage data storage costs with TimescaleDB. You can also use [compression](/docs/learn/columnar-storage/understand-hypercore) and [data retention](/docs/learn/data-lifecycle/data-retention/about-data-retention) to reduce your storage requirements. ## Move data [Section titled “Move data”](#move-data) To move chunks to a new tablespace, you first need to create the new tablespace and set the storage mount point. You can then use the [`move_chunk`](/docs/reference/timescaledb/hypertables/move_chunk) API call to move individual chunks from the default tablespace to the new tablespace. The `move_chunk` command also allows you to move indexes belonging to those chunks to an appropriate tablespace. Additionally, `move_chunk` allows you reorder the chunk during the migration. This can be used to make your queries faster, and works in a similar way to the [`reorder_chunk` command](/docs/reference/timescaledb/hypertables/reorder_chunk). Important You must be logged in as a super user, such as the `postgres` user, to use the `move_chunk()` API call. 1. **Create a new tablespace** In this example, the tablespace is called `history`, it is owned by the `postgres` super user, and the mount point is `/mnt/history`: ```sql CREATE TABLESPACE history OWNER postgres LOCATION '/mnt/history'; ``` 2. **List chunks that you want to move** In this example, chunks that contain data that is older than two days: ```sql SELECT show_chunks('conditions', older_than => INTERVAL '2 days'); ``` 3. **Move a chunk and its index to the new tablespace** You can also reorder the data in this step. In this example, the chunk called `_timescaledb_internal._hyper_1_4_chunk` is moved to the `history` tablespace, and is reordered based on its time index: ```sql SELECT move_chunk( chunk => '_timescaledb_internal._hyper_1_4_chunk', destination_tablespace => 'history', index_destination_tablespace => 'history', reorder_index => '_timescaledb_internal._hyper_1_4_chunk_netdata_time_idx', verbose => TRUE ); ``` 4. **Verify that the chunk now resides in the correct tablespace** Query `pg_tables` to list all of the chunks on the tablespace: ```sql SELECT tablename from pg_tables WHERE tablespace = 'history' and tablename like '_hyper_%_%_chunk'; ``` You can also verify that the index is in the correct location: ```sql SELECT indexname FROM pg_indexes WHERE tablespace = 'history'; ``` ## Move data in bulk [Section titled “Move data in bulk”](#move-data-in-bulk) To move several chunks at once, select the chunks you want to move by using `FROM show_chunks(...)`. For example, to move chunks containing data between 1 and 3 weeks old, in a hypertable named `example`: ```sql SELECT move_chunk( chunk => i, destination_tablespace => '') FROM show_chunks('example', now() - INTERVAL '1 week', now() - INTERVAL '3 weeks') i; ``` ## Examples [Section titled “Examples”](#examples) After moving a chunk to a slower tablespace, you can move it back to the default, faster tablespace: ```sql SELECT move_chunk( chunk => '_timescaledb_internal._hyper_1_4_chunk', destination_tablespace => 'pg_default', index_destination_tablespace => 'pg_default', reorder_index => '_timescaledb_internal._hyper_1_4_chunk_netdata_time_idx' ); ``` You can move a data chunk to the slower tablespace, but keep the chunk's indexes on the default, faster tablespace: ```sql SELECT move_chunk( chunk => '_timescaledb_internal._hyper_1_4_chunk', destination_tablespace => 'history', index_destination_tablespace => 'pg_default', reorder_index => '_timescaledb_internal._hyper_1_4_chunk_netdata_time_idx' ); ``` You can also keep the data in `pg_default` but move the index to `history`. Alternatively, you can set up a third tablespace called `history_indexes`, and move the data to `history` and the indexes to `history_indexes`. In TimescaleDB v2.0 and later, you can use `move_chunk` with the job scheduler framework. For more information, see the [jobs section](/docs/reference/timescaledb/jobs-automation). # Migrate your PostgreSQL database to self-hosted TimescaleDB > Choose how to migrate your existing PostgreSQL database to your self-hosted TimescaleDB installation You can migrate your existing PostgreSQL database to self-hosted TimescaleDB. There are several methods for migrating your data: * If the database you want to migrate is smaller than 100 GB, [migrate your entire database at once](/docs/deploy/self-hosted/migration/entire-database): This method directly transfers all data and schemas, including Timescale-specific features. Your hypertables, continuous aggregates, and policies are automatically available in the new self-hosted TimescaleDB instance. * For databases larger than 100GB, [migrate your schema and data separately](/docs/deploy/self-hosted/migration/schema-then-data): With this method, you migrate your tables one by one for easier failure recovery. If migration fails mid-way, you can restart from the failure point rather than from the beginning. However, Timescale-specific features won't be automatically migrated. Follow the instructions to restore your hypertables, continuous aggregates, and policies. * If you need to move data from PostgreSQL tables into hypertables within an existing self-hosted TimescaleDB instance, [migrate within the same database](/docs/deploy/self-hosted/migration/same-db): This method assumes that you have TimescaleDB set up in the same database instance as your existing table. * If you have data in an InfluxDB database, [migrate using Outflux](/docs/deploy/self-hosted/migration/migrate-influxdb): Outflux pipes exported data directly to your self-hosted TimescaleDB instance, and manages schema discovery, validation, and creation. Outflux works with earlier versions of InfluxDB. It does not work with InfluxDB version 2 and later. ## Choose a migration method [Section titled “Choose a migration method”](#choose-a-migration-method) Which method you choose depends on your database size, network upload and download speeds, existing continuous aggregates, and tolerance for failure recovery. Note If you are migrating from an Amazon RDS service, Amazon charges for the amount of data transferred out of the service. You could be charged by Amazon for all data egressed, even if the migration fails. If your database is smaller than 100 GB, choose to migrate your entire database at once. You can also migrate larger databases using this method, but the copying process must keep running, potentially over days or weeks. If the copy is interrupted, the process needs to be restarted. If you think an interruption in the copy is possible, choose to migrate your schema and data separately instead. Warning Migrating your schema and data separately does not retain continuous aggregates calculated using already-deleted data. For example, if you delete raw data after a month but retain downsampled data in a continuous aggregate for a year, the continuous aggregate loses any data older than a month upon migration. If you must keep continuous aggregates calculated using deleted data, migrate your entire database at once regardless of database size. If you aren't sure which method to use, try copying the entire database at once to estimate the time required. If the time estimate is very long, stop the migration and switch to the other method. ## Migrate an active database [Section titled “Migrate an active database”](#migrate-an-active-database) If your database is actively ingesting data, take precautions to ensure that your self-hosted TimescaleDB instance contains the data that is ingested while the migration is happening. Begin by running ingest in parallel on the source and target databases. This ensures that the newest data is written to both databases. Then backfill your data with one of the two migration methods. # Migrate the entire database at once > Migrate a small database to self-hosted TimescaleDB in one go with pg_dump and pg_restore Migrate smaller databases by dumping and restoring the entire database at once. This method works best on databases smaller than 100 GB. For larger databases, consider [migrating your schema and data separately](/docs/deploy/self-hosted/migration/schema-then-data). Warning Depending on your database size and network speed, migration can take a very long time. You can continue reading from your source database during this time, though performance could be slower. To avoid this problem, fork your database and migrate your data from the fork. If you write to tables in your source database during the migration, the new writes might not be transferred to TimescaleDB. To avoid this problem, see [Live migration](/docs/migrate/live-migration). ## Prerequisites for this procedure To follow these steps, you'll need: * The PostgreSQL [`pg_dump`](https://www.postgresql.org/docs/current/app-pgdump.html) and [`pg_restore`](https://www.postgresql.org/docs/current/app-pgrestore.html) utilities installed. * A client for connecting to PostgreSQL installed. These instructions use `psql`, but any client works. * A new empty database in your self-hosted TimescaleDB instance, provisioned with enough space for all your data. * Any other PostgreSQL extensions you use confirmed compatible with TimescaleDB. * The same major version of PostgreSQL on both your target and source databases. For information about upgrading PostgreSQL on your source database, see the [upgrade instructions](/docs/deploy/self-hosted/upgrades/upgrade-pg). * The same major version of TimescaleDB on both your target and source databases. For more information, see [upgrade self-hosted TimescaleDB](/docs/deploy/self-hosted/upgrades/major-upgrade). Note To speed up migration, compress your data into the columnstore. You can compress any chunks where data is not currently inserted, updated, or deleted. When you finish the migration, you can decompress chunks back to the rowstore as needed for normal operation. 1. **Dump all the data from your source database** Dump into a `dump.bak` file, using your source database connection details. If you are prompted for a password, use your source database credentials: ```bash pg_dump -U -W \ -h -p -Fc -v \ -f dump.bak ``` 2. **Connect to your self-hosted TimescaleDB instance** ```bash psql "postgres://:@:/?sslmode=require" ``` 3. **Prepare your self-hosted TimescaleDB instance for data restoration** Use [`timescaledb_pre_restore`](/docs/reference/timescaledb/administration/timescaledb_pre_restore) to stop background workers: ```sql SELECT timescaledb_pre_restore(); ``` 4. **Restore the dumped data** At the command prompt, restore the dumped data from the `dump.bak` file into your self-hosted TimescaleDB instance, using your connection details. To avoid permissions errors, include the `--no-owner` flag: ```bash pg_restore -U tsdbadmin -W \ -h -p --no-owner \ -Fc -v -d tsdb dump.bak ``` 5. **Return your self-hosted TimescaleDB instance to normal operations** Use [`timescaledb_post_restore`](/docs/reference/timescaledb/administration/timescaledb_post_restore): ```sql SELECT timescaledb_post_restore(); ``` 6. **Update your table statistics** Run `ANALYZE` on your entire dataset: ```sql ANALYZE; ``` # Migrate data to TimescaleDB from InfluxDB > Migrate data into your self-hosted TimescaleDB installation using the Outflux tool You can migrate data to TimescaleDB from InfluxDB using the Outflux tool. [Outflux](https://github.com/timescale/outflux) is an open source tool built by Tiger Data for fast, seamless migrations. It pipes exported data directly to self-hosted TimescaleDB, and manages schema discovery, validation, and creation. Important Outflux works with earlier versions of InfluxDB. It does not work with InfluxDB version 2 and later. ## Prerequisites for this procedure To follow these steps, you'll need: * A running instance of InfluxDB and a means to connect to it. * A self-hosted TimescaleDB instance and a means to connect to it. * Data in your InfluxDB instance. ## Procedures [Section titled “Procedures”](#procedures) To import data from Outflux, follow these procedures: 1. [Install Outflux](#install-outflux) 2. [Discover, validate, and transfer schema](#discover-validate-and-transfer-schema) to self-hosted TimescaleDB (optional) 3. [Migrate data to TimescaleDB](#migrate-data-to-timescaledb) ## Install Outflux [Section titled “Install Outflux”](#install-outflux) Install Outflux from the GitHub repository. There are builds for Linux, Windows, and MacOS. 1. **Go to the releases section of the Outflux repository** Open the [Outflux releases page](https://github.com/timescale/outflux/releases). 2. **Download the latest compressed tarball for your platform** 3. **Extract it to a preferred location** Note If you prefer to build Outflux from source, see the [Outflux README](https://github.com/timescale/outflux/blob/master/README.md) for instructions. To get help with Outflux, run `./outflux --help` from the directory where you installed it. ## Discover, validate, and transfer schema [Section titled “Discover, validate, and transfer schema”](#discover-validate-and-transfer-schema) Outflux can: * Discover the schema of an InfluxDB measurement * Validate whether a table exists that can hold the transferred data * Create a new table to satisfy the schema requirements if no valid table exists Note Outflux's `migrate` command does schema transfer and data migration in one step. For more information, see the [migrate](#migrate-data-to-timescaledb) section. Use this section if you want to validate and transfer your schema independently of data migration. To transfer your schema from InfluxDB to Timescale, run `outflux schema-transfer`: ```bash outflux schema-transfer \ --input-server=http://localhost:8086 \ --output-conn="dbname=tsdb user=tsdbadmin" ``` To transfer all measurements from the database, leave out the measurement name argument. Note This example uses the `postgres` user and database to connect to the self-hosted TimescaleDB instance. For other connection options and configuration, see the [Outflux GitHub repo](https://github.com/timescale/outflux#connection). ### Schema transfer options [Section titled “Schema transfer options”](#schema-transfer-options) Outflux's `schema-transfer` can use 1 of 4 schema strategies: * `ValidateOnly`: checks that self-hosted TimescaleDB is installed and that the specified database has a properly partitioned hypertable with the correct columns, but doesn't perform modifications * `CreateIfMissing`: runs the same checks as `ValidateOnly`, and creates and properly partitions any missing hypertables * `DropAndCreate`: drops any existing table with the same name as the measurement, and creates a new hypertable and partitions it properly * `DropCascadeAndCreate`: performs the same action as `DropAndCreate`, and also executes a cascade table drop if there is an existing table with the same name as the measurement You can specify your schema strategy by passing a value to the `--schema-strategy` option in the `schema-transfer` command. The default strategy is `CreateIfMissing`. By default, each tag and field in InfluxDB is treated as a separate column in your TimescaleDB tables. To transfer tags and fields as a single JSONB column, use the flag `--tags-as-json`. ## Migrate data to TimescaleDB [Section titled “Migrate data to TimescaleDB”](#migrate-data-to-timescaledb) Transfer your schema and migrate your data all at once with the `migrate` command. For example, run: ```bash outflux migrate \ --input-server=http://localhost:8086 \ --output-conn="dbname=tsdb user=tsdbadmin" ``` The schema strategy and connection options are the same as for `schema-transfer`. For more information, see [Discover, validate, and transfer schema](#discover-validate-and-transfer-schema). In addition, `outflux migrate` also takes the following flags: * `--limit`: Pass a number, `N`, to `--limit` to export only the first `N` rows, ordered by time. * `--from` and `to`: Pass a timestamp to `--from` or `--to` to specify a time window of data to migrate. * `chunk-size`: Changes the size of data chunks transferred. Data is pulled from the InfluxDB server in chunks of default size 15 000. * `batch-size`: Changes the number of rows in an insertion batch. Data is inserted into a self-hosted TimescaleDB database in batches that are 8000 rows by default. For more flags, see the [GitHub documentation for `outflux migrate`](https://github.com/timescale/outflux#migrate). Alternatively, see the command line help: ```bash outflux migrate --help ``` # Migrate data to TimescaleDB from the same PostgreSQL instance > Migrate data into a TimescaleDB hypertable from a regular PostgreSQL table You can migrate data into a TimescaleDB hypertable from a regular PostgreSQL table. This method assumes that you have TimescaleDB set up in the same database instance as your existing table. ## Prerequisites for this procedure To follow these steps, you'll need: * A [self-hosted TimescaleDB](/docs/get-started/choose-your-path/install-timescaledb) instance. * A table with existing data. In this example, the source table is named `old_table`. Replace the table name with your actual table name. The example also names the destination table `new_table`, but you might want to use a more descriptive name. ## Migrate data [Section titled “Migrate data”](#migrate-data) Migrate your data into TimescaleDB from within the same database. 1. **Call `CREATE TABLE` to make a new table based on your existing table** You can create your indexes at the same time, so you don't have to recreate them manually. Or you can create the table without indexes, which makes data migration faster. * With indexes ```sql CREATE TABLE new_table ( LIKE old_table INCLUDING DEFAULTS INCLUDING CONSTRAINTS INCLUDING INDEXES ) WITH ( tsdb.hypertable, tsdb.partition_column='' ); ``` * Without indexes ```sql CREATE TABLE new_table ( LIKE old_table INCLUDING DEFAULTS INCLUDING CONSTRAINTS EXCLUDING INDEXES ) WITH ( tsdb.hypertable, tsdb.partition_column='' ); ``` When you create a hypertable using `CREATE TABLE ... WITH ...`, the default partitioning column is automatically the first column with a timestamp data type. Also, TimescaleDB creates a [columnstore policy](/docs/reference/timescaledb/hypercore/add_columnstore_policy) that automatically converts your data to the columnstore, after an interval equal to the value of the chunk interval, defined through `compress_after` in the policy. This columnar format enables fast scanning and aggregation, optimizing performance for analytical workloads while also saving significant storage space. In the columnstore conversion, hypertable chunks are compressed by up to 98%, and organized for efficient, large-scale queries. You can customize this policy later using [alter\_job](/docs/reference/timescaledb/jobs-automation/alter_job). However, to change `after` or `created_before`, the compression settings, or the hypertable the policy is acting on, you must [remove the columnstore policy](/docs/reference/timescaledb/hypercore/remove_columnstore_policy) and [add a new one](/docs/reference/timescaledb/hypercore/add_columnstore_policy). You can also manually [convert chunks](/docs/reference/timescaledb/hypercore/convert_to_columnstore) in a hypertable to the columnstore. 2. **Insert data from the old table to the new table** ```sql INSERT INTO new_table SELECT * FROM old_table; ``` 3. **Recreate indexes if needed** If you created your new table without indexes, recreate your indexes now. # Migrate schema and data separately > Migrate your data and schema to self-hosted TimescaleDB separately for easier failure recovery Migrate larger databases by migrating your schema first, then migrating the data. This method copies each table or chunk separately, which allows you to restart midway if one copy operation fails. Note For smaller databases, it may be more convenient to migrate your entire database at once. For more information, see the section on [choosing a migration method](/docs/deploy/self-hosted/migration). Warning This method does not retain continuous aggregates calculated using already-deleted data. For example, if you delete raw data after a month but retain downsampled data in a continuous aggregate for a year, the continuous aggregate loses any data older than a month upon migration. If you must keep continuous aggregates calculated using deleted data, migrate your entire database at once. For more information, see the section on [choosing a migration method](/docs/deploy/self-hosted/migration). The procedure to migrate your database requires these steps: * [Migrate schema pre-data](#migrate-schema-pre-data) * [Restore hypertables in TimescaleDB](#restore-hypertables-in-your-self-hosted-timescaledb-instance) * [Copy data from the source database](#copy-data-from-the-source-database) * [Restore data into TimescaleDB](#restore-data-into-timescaledb) * [Migrate schema post-data](#migrate-schema-post-data) * [Recreate continuous aggregates](#recreate-continuous-aggregates) (optional) * [Recreate policies](#recreate-policies) (optional) * [Update table statistics](#update-table-statistics) Warning Depending on your database size and network speed, steps that involve copying data can take a very long time. You can continue reading from your source database during this time, though performance could be slower. To avoid this problem, fork your database and migrate your data from the fork. If you write to the tables in your source database during the migration, the new writes might not be transferred to TimescaleDB. To avoid this problem, see the section on [migrating an active database](/docs/deploy/self-hosted/migration#migrate-an-active-database). ## Prerequisites for this procedure To follow these steps, you'll need: * The PostgreSQL [`pg_dump`](https://www.postgresql.org/docs/current/app-pgdump.html) and [`pg_restore`](https://www.postgresql.org/docs/current/app-pgrestore.html) utilities installed. * A client for connecting to PostgreSQL installed. These instructions use `psql`, but any client works. * A new empty database in a self-hosted TimescaleDB instance, provisioned with enough space for all your data. * Any other PostgreSQL extensions you use confirmed compatible with TimescaleDB. * The same major version of PostgreSQL on both your self-hosted TimescaleDB instance and your source database. For information about upgrading PostgreSQL on your source database, see the [upgrade instructions for self-hosted TimescaleDB](/docs/deploy/self-hosted/upgrades/upgrade-pg) and [Managed Service for TimescaleDB](/docs/deploy/mst/maintenance). * The same major version of TimescaleDB on both your target and source database. For more information, see [upgrading TimescaleDB](/docs/deploy/self-hosted/upgrades/major-upgrade). ## Migrate schema pre-data [Section titled “Migrate schema pre-data”](#migrate-schema-pre-data) Migrate your pre-data from your source database to self-hosted TimescaleDB. This includes table and schema definitions, as well as information on sequences, owners, and settings. This doesn't include TimescaleDB-specific schemas. 1. **Dump the schema pre-data from your source database into a `dump_pre_data.bak` file** Exclude TimescaleDB-specific schemas. If you are prompted for a password, use your source database credentials: ```bash pg_dump -U -W \ -h -p -Fc -v \ --section=pre-data --exclude-schema="_timescaledb*" \ -f dump_pre_data.bak ``` 2. **Restore the dumped data from the dump\_pre\_data.bak file into your self-hosted TimescaleDB instance** To avoid permissions errors, include the `--no-owner` flag: ```bash pg_restore -U tsdbadmin -W \ -h -p --no-owner -Fc \ -v -d tsdb dump_pre_data.bak ``` ## Restore hypertables in your self-hosted TimescaleDB instance [Section titled “Restore hypertables in your self-hosted TimescaleDB instance”](#restore-hypertables-in-your-self-hosted-timescaledb-instance) After pre-data migration, your hypertables from your source database become regular PostgreSQL tables in TimescaleDB. Recreate your hypertables in your self-hosted TimescaleDB instance to restore them. 1. **Connect to your self-hosted TimescaleDB instance** ```sql psql "postgres://:@:/?sslmode=require" ``` 2. **Restore the hypertable** ```sql SELECT create_hypertable( '', by_range('', INTERVAL '') ); ``` Note The `by_range` dimension builder is an addition to TimescaleDB 2.13. ## Copy data from the source database [Section titled “Copy data from the source database”](#copy-data-from-the-source-database) After restoring your hypertables, return to your source database to copy your data, table by table. 1. **Connect to your source database** ```bash psql "postgres://:@:/?sslmode=require" ``` 2. **Dump the data from the first table into a .csv file** ```sql \COPY (SELECT * FROM ) TO .csv CSV ``` Repeat for each table and hypertable you want to migrate. Note If your tables are very large, you can migrate each table in multiple pieces. Split each table by time range, and copy each range individually. For example: ```sql \COPY (SELECT * FROM WHERE time > '2021-11-01' AND time < '2011-11-02') TO .csv CSV ``` ## Restore data into TimescaleDB [Section titled “Restore data into TimescaleDB”](#restore-data-into-timescaledb) When you have copied your data into `.csv` files, you can restore it to self-hosted TimescaleDB by copying from the `.csv` files. There are two methods: using regular PostgreSQL [`COPY`](https://www.postgresql.org/docs/current/sql-copy.html), or using the TimescaleDB [`timescaledb-parallel-copy`](https://github.com/timescale/timescaledb-parallel-copy) function. In tests, `timescaledb-parallel-copy` is 16% faster. The `timescaledb-parallel-copy` tool is not included by default. You must install the function. Warning Because `COPY` decompresses data, any compressed data in your source database is now stored uncompressed in your `.csv` files. If you provisioned your self-hosted TimescaleDB storage for your compressed data, the uncompressed data may take too much storage. To avoid this problem, periodically recompress your data as you copy it in. ### Restoring data with timescaledb-parallel-copy [Section titled “Restoring data with timescaledb-parallel-copy”](#restoring-data-with-timescaledb-parallel-copy) 1. **Install timescaledb-parallel-copy** ```bash go get github.com/timescale/timescaledb-parallel-copy/cmd/timescaledb-parallel-copy ``` 2. **Import data into your self-hosted TimescaleDB instance** Set `` to twice the number of CPUs in your database. For example, if you have 4 CPUs, `` should be `8`. ```bash timescaledb-parallel-copy \ --connection "host= \ user=tsdbadmin password= \ port= \ dbname=tsdb \ sslmode=require " \ --table \ --file .csv \ --workers \ --reporting-period 30s ``` Repeat for each table and hypertable you want to migrate. ### Restoring data into self-hosted TimescaleDB with COPY [Section titled “Restoring data into self-hosted TimescaleDB with COPY”](#restoring-data-into-self-hosted-timescaledb-with-copy) 1. **Connect to your self-hosted TimescaleDB instance** ```sql psql "postgres://tsdbadmin:@:/tsdb?sslmode=require" ``` 2. **Restore the data to your self-hosted TimescaleDB instance** ```sql \copy FROM '.csv' WITH (FORMAT CSV); ``` Repeat for each table and hypertable you want to migrate. ## Migrate schema post-data [Section titled “Migrate schema post-data”](#migrate-schema-post-data) When you have migrated your table and hypertable data, migrate your PostgreSQL schema post-data. This includes information about constraints. 1. **Dump the schema post-data from your source database into a `dump_post_data.dump` file** Exclude Timescale-specific schemas. If you are prompted for a password, use your source database credentials: ```bash pg_dump -U -W \ -h -p -Fc -v \ --section=post-data --exclude-schema="_timescaledb*" \ -f dump_post_data.dump ``` 2. **Restore the dumped schema post-data from the dump\_post\_data.dump file into your self-hosted TimescaleDB instance** Using your connection details. To avoid permissions errors, include the `--no-owner` flag: ```bash pg_restore -U tsdbadmin -W \ -h -p --no-owner -Fc \ -v -d tsdb dump_post_data.dump ``` ### Troubleshooting [Section titled “Troubleshooting”](#troubleshooting) If you see these errors during the migration process, you can safely ignore them. The migration still occurs successfully. ```plaintext pg_restore: error: could not execute query: ERROR: relation "" already exists ``` ```plaintext pg_restore: error: could not execute query: ERROR: trigger "ts_insert_blocker" for relation "" already exists ``` ## Recreate continuous aggregates [Section titled “Recreate continuous aggregates”](#recreate-continuous-aggregates) Continuous aggregates aren't migrated by default when you transfer your schema and data separately. You can restore them by recreating the continuous aggregate definitions and recomputing the results on your self-hosted TimescaleDB instance. The recomputed continuous aggregates only aggregate existing data in your self-hosted TimescaleDB instance. They don't include deleted raw data. 1. **Connect to your source database** ```bash psql "postgres://:@:/?sslmode=require" ``` 2. **Get a list of your existing continuous aggregate definitions** ```sql SELECT view_name, view_definition FROM timescaledb_information.continuous_aggregates; ``` This query returns the names and definitions for all your continuous aggregates. For example: ```sql view_name | view_definition ----------------+-------------------------------------------------------------------------------------------------------- avg_fill_levels | SELECT round(avg(fill_measurements.fill_level), 2) AS avg_fill_level, + | time_bucket('01:00:00'::interval, fill_measurements."time") AS bucket, + | fill_measurements.sensor_id + | FROM fill_measurements + | GROUP BY (time_bucket('01:00:00'::interval, fill_measurements."time")), fill_measurements.sensor_id; (1 row) ``` 3. **Connect to your self-hosted TimescaleDB instance** ```bash psql "postgres://tsdbadmin:@:/tsdb?sslmode=require" ``` 4. **Recreate each continuous aggregate definition** ```sql CREATE MATERIALIZED VIEW WITH (timescaledb.continuous) AS ``` ## Recreate policies [Section titled “Recreate policies”](#recreate-policies) By default, policies aren't migrated when you transfer your schema and data separately. Recreate them on your self-hosted TimescaleDB instance. 1. **Connect to your source database** ```bash psql "postgres://:@:/?sslmode=require" ``` 2. **Get a list of your existing policies** This query returns a list of all your policies, including continuous aggregate refresh policies, retention policies, compression policies, and reorder policies: ```sql SELECT application_name, schedule_interval, retry_period, config, hypertable_name FROM timescaledb_information.jobs WHERE owner = ''; ``` 3. **Connect to your self-hosted TimescaleDB instance** ```sql psql "postgres://tsdbadmin:@:/tsdb?sslmode=require" ``` 4. **Recreate each policy** For more information about recreating policies, see the sections on [continuous-aggregate refresh policies](/docs/build/continuous-aggregates/refresh-policies), [retention policies](/docs/learn/data-lifecycle/data-retention/about-data-retention), and [reorder policies](/docs/reference/timescaledb/hypertables/add_reorder_policy). ## Update table statistics [Section titled “Update table statistics”](#update-table-statistics) Update your table statistics by running [`ANALYZE`](https://www.postgresql.org/docs/current/sql-analyze.html) on your entire dataset. This might take some time depending on the size of your database: ```sql ANALYZE; ``` ### Troubleshooting [Section titled “Troubleshooting”](#troubleshooting-1) If you see errors of the following form when you run `ANALYZE`, you can safely ignore them: ```plaintext WARNING: skipping "" --- only superuser can analyze it ``` The skipped tables and indexes correspond to system catalogs that can't be accessed. Skipping them does not affect statistics on your data. # Replication and high availability > Set up replication and high availability for self-hosted TimescaleDB PostgreSQL relies on replication for high availability, failover, and balancing read loads across multiple nodes. Replication ensures that data written to the primary PostgreSQL database is mirrored on one or more nodes. By virtue of having multiple nodes with an exact copy of the primary database available, the primary database can be replaced with a replica node in the event of a failure or outage on the primary server. Replica nodes can also be used as read only databases, also called read replicas, allowing reads to be horizontally scaled by spreading the read query volume across multiple nodes. * [Learn about high availability](/docs/deploy/self-hosted/replication-and-ha/about-ha) to understand how it works before you begin using it. * [Configure replication](/docs/deploy/self-hosted/replication-and-ha/configure-replication). # About high availability > Increase the redundancy and resilience of your TimescaleDB database with backup and replication High availability (HA) is achieved by increasing redundancy and resilience. To increase redundancy, parts of the system are replicated, so that they are on standby in the event of a failure. To increase resilience, recovery processes switch between these standby resources as quickly as possible. ## Backups [Section titled “Backups”](#backups) For some systems, recovering from backup alone can be a suitable availability strategy. For more information about backups in self-hosted TimescaleDB, see the [backup and restore section](/docs/deploy/self-hosted/backup-and-restore) in the TimescaleDB documentation. ## Storage redundancy [Section titled “Storage redundancy”](#storage-redundancy) Storage redundancy refers to having multiple copies of a database's data files. If the storage currently attached to a PostgreSQL instance corrupts or otherwise becomes unavailable, the system can replace its current storage with one of the copies. ## Instance redundancy [Section titled “Instance redundancy”](#instance-redundancy) Instance redundancy refers to having replicas of your database running simultaneously. In the case of a database failure, a replica is an up-to-date, running database that can take over immediately. ## Zonal redundancy [Section titled “Zonal redundancy”](#zonal-redundancy) While the public cloud is highly reliable, entire portions of the cloud can be unavailable at times. TimescaleDB does not protect against Availability Zone failures unless the user is using HA replicas. We do not currently offer multi-cloud solutions or protection from an AWS Regional failure. ## Replication [Section titled “Replication”](#replication) TimescaleDB supports replication using PostgreSQL's built-in [streaming replication](https://www.postgresql.org/docs/current/warm-standby.html#STREAMING-REPLICATION). Using [logical replication](https://www.postgresql.org/docs/current/logical-replication.html) with TimescaleDB is not recommended, as it requires schema synchronization between the primary and replica nodes and replicating partition root tables, which are [not currently supported](https://www.postgresql.org/docs/current/logical-replication-restrictions.html). For migrating or replicating data from any PostgreSQL or TimescaleDB database to TimescaleDB, you can use the [source PostgreSQL connector](/docs/migrate/livesync-for-postgresql). The source PostgreSQL connector is built on top of PostgreSQL's logical replication protocol and extends it for TimescaleDB use cases, providing continuous, low-latency data synchronization. PostgreSQL achieves streaming replication by having replicas continuously stream the WAL from the primary database. See the official [replication documentation](https://www.postgresql.org/docs/current/warm-standby.html#STREAMING-REPLICATION) for details. For more information about how PostgreSQL implements Write-Ahead Logging, see their [WAL Documentation](https://www.postgresql.org/docs/current/wal-intro.html). ## Failover [Section titled “Failover”](#failover) PostgreSQL offers failover functionality where a replica is promoted to primary in the event of a failure on the primary. This is done using [pg\_ctl](https://www.postgresql.org/docs/current/app-pg-ctl.html) or the `trigger_file`, but it does not provide out-of-the-box support for automatic failover. Read more in the PostgreSQL [failover documentation](https://www.postgresql.org/docs/current/warm-standby-failover.html). [Patroni](https://github.com/zalando/patroni) offers a configurable high availability solution with automatic failover functionality. # Configure replication > Set up asynchronous streaming replication on one or more database replicas in your self-hosted TimescaleDB installation This section outlines how to set up asynchronous streaming replication on one or more database replicas. Before you begin, make sure you have at least two separate instances of TimescaleDB running. If you installed TimescaleDB using a Docker container, use a [PostgreSQL entry point script](https://hub.docker.com/_/postgres/) to run the configuration. For more advanced examples, see the [TimescaleDB Helm Charts repository](https://github.com/timescale/helm-charts/tree/main/charts/timescaledb-single). To configure replication on self-hosted TimescaleDB, you need to perform these procedures: 1. [Configure the primary database](#configure-the-primary-database) 2. [Configure replication parameters](#configure-replication-parameters) 3. [Create replication slots](#create-replication-slots) 4. [Configure host-based authentication parameters](#configure-host-based-authentication-parameters) 5. [Create a base backup on the replica](#create-a-base-backup-on-the-replica) 6. [Configure replication and recovery settings](#configure-replication-and-recovery-settings) 7. [Verify that the replica is working](#verify-that-the-replica-is-working) ## Configure the primary database [Section titled “Configure the primary database”](#configure-the-primary-database) To configure the primary database, you need a PostgreSQL user with a role that allows it to initialize streaming replication. This is the user each replica uses to stream from the primary database. 1. **Set the password encryption level** On the primary database, as a user with superuser privileges, such as the `postgres` user, set the password encryption level to `scram-sha-256`: ```sql SET password_encryption = 'scram-sha-256'; ``` 2. **Create a new user called repuser** ```sql CREATE ROLE repuser WITH REPLICATION PASSWORD '' LOGIN; ``` Note The [scram-sha-256](https://www.postgresql.org/docs/current/sasl-authentication.html#SASL-SCRAM-SHA-256) encryption level is the most secure password-based authentication available in PostgreSQL. It is only available in PostgreSQL 10 and later. ## Configure replication parameters [Section titled “Configure replication parameters”](#configure-replication-parameters) There are several replication settings that need to be added or edited in the `postgresql.conf` configuration file. 1. **Set the `synchronous_commit` parameter to `off`** 2. **Set the `max_wal_senders` parameter** Set it to the total number of concurrent connections from replicas or backup clients. As a minimum, this should equal the number of replicas you intend to have. 3. **Set the `wal_level` parameter** Set it to the amount of information written to the PostgreSQL write-ahead log (WAL). For replication to work, there needs to be enough data in the WAL to support archiving and replication. The default value is usually appropriate. 4. **Set the `max_replication_slots` parameter** Set it to the total number of replication slots the primary database can support. 5. **Set the `listen_addresses` parameter** Set it to the address of the primary database. Do not leave this parameter as the local loopback address, because the remote replicas must be able to connect to the primary to stream the WAL. 6. **Restart PostgreSQL to pick up the changes** This must be done before you create replication slots. The most common streaming replication use case is asynchronous replication with one or more replicas. In this example, the WAL is streamed to the replica, but the primary server does not wait for confirmation that the WAL has been written to disk on either the primary or the replica. This is the most performant replication configuration, but it does carry the risk of a small amount of data loss in the event of a system failure. It also makes no guarantees that the replica is fully up to date with the primary, which could cause inconsistencies between read queries on the primary and the replica. The example configuration for this use case: ```yaml listen_addresses = '*' wal_level = replica max_wal_senders = 2 max_replication_slots = 2 synchronous_commit = off ``` If you need stronger consistency on the replicas, or if your query load is heavy enough to cause significant lag between the primary and replica nodes in asynchronous mode, consider a synchronous replication configuration instead. For more information about the different replication modes, see the [replication modes section](#replication-modes). ## Create replication slots [Section titled “Create replication slots”](#create-replication-slots) When you have configured `postgresql.conf` and restarted PostgreSQL, you can create a [replication slot](https://www.postgresql.org/docs/current/static/warm-standby.html#STREAMING-REPLICATION-SLOTS) for each replica. Replication slots ensure that the primary does not delete segments from the WAL until they have been received by the replicas. This is important in case a replica goes down for an extended time. The primary needs to verify that a WAL segment has been consumed by a replica, so that it can safely delete data. You can use [archiving](https://www.postgresql.org/docs/current/continuous-archiving.html) for this purpose, but replication slots provide the strongest protection for streaming replication. 1. **Create the first replication slot** At the `psql` prompt, create the first replication slot. The name of the slot is arbitrary. In this example, it is called `replica_1_slot`: ```sql SELECT * FROM pg_create_physical_replication_slot('replica_1_slot', true); ``` 2. **Repeat for each required replication slot** ## Configure host-based authentication parameters [Section titled “Configure host-based authentication parameters”](#configure-host-based-authentication-parameters) There are several replication settings that need to be added or edited to the `pg_hba.conf` configuration file. In this example, the settings restrict replication connections to traffic coming from `REPLICATION_HOST_IP` as the PostgreSQL user `repuser` with a valid password. `REPLICATION_HOST_IP` can initiate streaming replication from that machine without additional credentials. You can change the `address` and `method` values to match your security and network settings. For more information about `pg_hba.conf`, see the [`pg_hba` documentation](https://www.postgresql.org/docs/current/auth-pg-hba-conf.html). 1. **Open the `pg_hba.conf` configuration file** Add or edit this line: ```yaml TYPE DATABASE USER ADDRESS METHOD AUTH_METHOD host replication repuser /32 scram-sha-256 ``` 2. **Restart PostgreSQL to pick up the changes** ## Create a base backup on the replica [Section titled “Create a base backup on the replica”](#create-a-base-backup-on-the-replica) Replicas work by streaming the primary server's WAL log and replaying its transactions in PostgreSQL recovery mode. To do this, the replica needs to be in a state where it can replay the log. You can do this by restoring the replica from a base backup of the primary instance. 1. **Stop PostgreSQL services** 2. **Delete the existing data if applicable** If the replica database already contains data, delete it before you run the backup, by removing the PostgreSQL data directory: ```bash rm -rf /* ``` If you don't know the location of the data directory, find it with the `show data_directory;` command. 3. **Restore from the base backup** Use the IP address of the primary database and the replication username: ```bash pg_basebackup -h \ -D \ -U repuser -vP -W ``` The -W flag prompts you for a password. If you are using this command in an automated setup, you might need to use a [pgpass file](https://www.postgresql.org/docs/current/libpq-pgpass.html). 4. **Create a standby.signal file** When the backup is complete, create a [standby.signal](https://www.postgresql.org/docs/current/runtime-config-wal.html#RUNTIME-CONFIG-WAL-ARCHIVE-RECOVERY) file in your data directory. When PostgreSQL finds a `standby.signal` file in its data directory, it starts in recovery mode and streams the WAL through the replication protocol: ```bash touch /standby.signal ``` ## Configure replication and recovery settings [Section titled “Configure replication and recovery settings”](#configure-replication-and-recovery-settings) When you have successfully created a base backup and a `standby.signal` file, you can configure the replication and recovery settings. 1. **Add primary server details to the replica** In the replica's `postgresql.conf` file, add details for communicating with the primary server. If you are using streaming replication, the `application_name` in `primary_conninfo` should be the same as the name used in the primary's `synchronous_standby_names` settings: ```yaml primary_conninfo = 'host= port=5432 user=repuser password= application_name=r1' primary_slot_name = 'replica_1_slot' ``` 2. **Mirror the configuration of the primary database** If you are using asynchronous replication, use these settings: ```yaml hot_standby = on wal_level = replica max_wal_senders = 2 max_replication_slots = 2 synchronous_commit = off ``` The `hot_standby` parameter must be set to `on` to allow read-only queries on the replica. In PostgreSQL 10 and later, this setting is `on` by default. 3. **Restart PostgreSQL to pick up the changes** ## Verify that the replica is working [Section titled “Verify that the replica is working”](#verify-that-the-replica-is-working) At this point, your replica should be fully synchronized with the primary database and prepared to stream from it. You can verify that it is working properly by checking the logs on the replica, which should look like this: ```txt LOG: database system was shut down in recovery at 2018-03-09 18:36:23 UTC LOG: entering standby mode LOG: redo starts at 0/2000028 LOG: consistent recovery state reached at 0/3000000 LOG: database system is ready to accept read only connections LOG: started streaming WAL from primary at 0/3000000 on timeline 1 ``` Any client can perform reads on the replica. You can verify this by running inserts, updates, or other modifications to your data on the primary database, and then querying the replica to ensure they have been properly copied over. ## Replication modes [Section titled “Replication modes”](#replication-modes) In most cases, asynchronous streaming replication is sufficient. However, you might require greater consistency between the primary and replicas, especially if you have a heavy workload. Under heavy workloads, replicas can lag far behind the primary, providing stale data to clients reading from the replicas. Additionally, in cases where any data loss is fatal, asynchronous replication might not provide enough of a durability guarantee. The PostgreSQL [`synchronous_commit`](https://www.postgresql.org/docs/current/runtime-config-wal.html#GUC-SYNCHRONOUS-COMMIT) feature has several options with varying consistency and performance tradeoffs. In the `postgresql.conf` file, set the `synchronous_commit` parameter to: * `on`: This is the default value. The server does not return `success` until the WAL transaction has been written to disk on the primary and any replicas. * `off`: The server returns `success` when the WAL transaction has been sent to the operating system to write to the WAL on disk on the primary, but does not wait for the operating system to actually write it. This can cause a small amount of data loss if the server crashes when some data has not been written, but it does not result in data corruption. Turning `synchronous_commit` off is a well-known PostgreSQL optimization for workloads that can withstand some data loss in the event of a system crash. * `local`: Enforces `on` behavior only on the primary server. * `remote_write`: The database returns `success` to a client when the WAL record has been sent to the operating system for writing to the WAL on the replicas, but before confirmation that the record has actually been persisted to disk. This is similar to asynchronous commit, except it waits for the replicas as well as the primary. In practice, the extra wait time incurred waiting for the replicas significantly decreases replication lag. * `remote_apply`: Requires confirmation that the WAL records have been written to the WAL and applied to the databases on all replicas. This provides the strongest consistency of any of the `synchronous_commit` options. In this mode, replicas always reflect the latest state of the primary, and replication lag is nearly non-existent. Note If `synchronous_standby_names` is empty, the settings `on`, `remote_apply`, `remote_write` and `local` all provide the same synchronization level, and transaction commits wait for the local flush to disk. This matrix shows the level of consistency provided by each mode: | Mode | WAL Sent to OS (Primary) | WAL Persisted (Primary) | WAL Sent to OS (Primary & Replicas) | WAL Persisted (Primary & Replicas) | Transaction Applied (Primary & Replicas) | | ------------ | ------------------------ | ----------------------- | ----------------------------------- | ---------------------------------- | ---------------------------------------- | | Off | ✅ | ❌ | ❌ | ❌ | ❌ | | Local | ✅ | ✅ | ❌ | ❌ | ❌ | | Remote Write | ✅ | ✅ | ✅ | ❌ | ❌ | | On | ✅ | ✅ | ✅ | ✅ | ❌ | | Remote Apply | ✅ | ✅ | ✅ | ✅ | ✅ | The `synchronous_standby_names` setting is a complementary setting to `synchronous_commit`. It lists the names of all replicas the primary database supports for synchronous replication, and configures how the primary database waits for them. The `synchronous_standby_names` setting supports these formats: * `FIRST num_sync (replica_name_1, replica_name_2)`: This waits for confirmation from the first `num_sync` replicas before returning `success`. The list of `replica_names` determines the relative priority of the replicas. Replica names are determined by the `application_name` setting on the replicas. * `ANY num_sync (replica_name_1, replica_name_2)`: This waits for confirmation from `num_sync` replicas in the provided list, regardless of their priority or position in the list. This is works as a quorum function. Synchronous replication modes force the primary to wait until all required replicas have written the WAL, or applied the database transaction, depending on the `synchronous_commit` level. This could cause the primary to hang indefinitely if a required replica crashes. When the replica reconnects, it replays any of the WAL it needs to catch up. Only then is the primary able to resume writes. To mitigate this, provision more than the amount of nodes required under the `synchronous_standby_names` setting and list them in the `FIRST` or `ANY` clauses. This allows the primary to move forward as long as a quorum of replicas have written the most recent WAL transaction. Replicas that were out of service are able to reconnect and replay the missed WAL transactions asynchronously. ## Replication diagnostics [Section titled “Replication diagnostics”](#replication-diagnostics) The PostgreSQL [pg\_stat\_replication](https://www.postgresql.org/docs/current/monitoring-stats.html#MONITORING-PG-STAT-REPLICATION-VIEW) view provides information about each replica. This view is particularly useful for calculating replication lag, which measures how far behind the primary the current state of the replica is. The `replay_lag` field gives a measure of the seconds between the most recent WAL transaction on the primary, and the last reported database commit on the replica. Coupled with `write_lag` and `flush_lag`, this provides insight into how far behind the replica is. The `*_lsn` fields also provide helpful information. They allow you to compare WAL locations between the primary and the replicas. The `state` field is useful for determining exactly what each replica is currently doing; the available modes are `startup`, `catchup`, `streaming`, `backup`, and `stopping`. To see the data, on the primary database, run this command: ```sql SELECT * FROM pg_stat_replication; ``` The output looks like this: ```sql -[ RECORD 1 ]----+------------------------------ pid | 52343 usesysid | 16384 usename | repuser application_name | r2 client_addr | 10.0.13.6 client_hostname | client_port | 59610 backend_start | 2018-02-07 19:07:15.261213+00 backend_xmin | state | streaming sent_lsn | 16B/43DB36A8 write_lsn | 16B/43DB36A8 flush_lsn | 16B/43DB36A8 replay_lsn | 16B/43107C28 write_lag | 00:00:00.009966 flush_lag | 00:00:00.03208 replay_lag | 00:00:00.43537 sync_priority | 2 sync_state | sync -[ RECORD 2 ]----+------------------------------ pid | 54498 usesysid | 16384 usename | repuser application_name | r1 client_addr | 10.0.13.5 client_hostname | client_port | 43402 backend_start | 2018-02-07 19:45:41.410929+00 backend_xmin | state | streaming sent_lsn | 16B/43DB36A8 write_lsn | 16B/43DB36A8 flush_lsn | 16B/43DB36A8 replay_lsn | 16B/42C3B9C8 write_lag | 00:00:00.019736 flush_lag | 00:00:00.044073 replay_lag | 00:00:00.644004 sync_priority | 1 sync_state | sync ``` ## Failover [Section titled “Failover”](#failover) PostgreSQL provides some failover functionality, where the replica is promoted to primary in the event of a failure. This is provided using the [pg\_ctl](https://www.postgresql.org/docs/current/static/app-pg-ctl.html) command or the `trigger_file`. However, PostgreSQL does not provide support for automatic failover. For more information, see the [PostgreSQL failover documentation](https://www.postgresql.org/docs/current/static/warm-standby-failover.html). If you require a configurable high availability solution with automatic failover functionality, check out [Patroni](https://github.com/zalando/patroni). # Additional tooling > Get the most out of TimescaleDB with open-source tools that help you perform common tasks * Automatically configure your TimescaleDB instance with [`timescaledb-tune`](/docs/deploy/self-hosted/tooling/about-timescaledb-tune) * Install [TimescaleDB Toolkit](/docs/deploy/self-hosted/tooling/install-toolkit) to access more hyperfunctions and function pipelines # About timescaledb-tune > Automatically tune your TimescaleDB database to match your system resources and PostgreSQL version Get better performance by tuning your TimescaleDB database to match your system resources and PostgreSQL version. `timescaledb-tune` is an open source command line tool that analyzes and adjusts your database settings. ## Install timescaledb-tune [Section titled “Install timescaledb-tune”](#install-timescaledb-tune) `timescaledb-tune` is packaged with binary releases of TimescaleDB. If you installed TimescaleDB from any binary release, including Docker, you already have access. For more install instructions, see the [GitHub repository](https://github.com/timescale/timescaledb-tune). ## Tune your database with timescaledb-tune [Section titled “Tune your database with timescaledb-tune”](#tune-your-database-with-timescaledb-tune) Run `timescaledb-tune` from the command line. The tool analyzes your `postgresql.conf` file to provide recommendations for memory, parallelism, write-ahead log, and other settings. These changes are written to your `postgresql.conf`. They take effect on the next restart. 1. **Run timescaledb-tune** At the command line, run `timescaledb-tune`. To accept all recommendations automatically, include the `--yes` flag. ```bash timescaledb-tune ``` 2. **Respond to prompts** If you didn't use the `--yes` flag, respond to each prompt to accept or reject the recommendations. The changes are written to your `postgresql.conf`. For detailed instructions and other options, see the documentation in the [Github repository](https://github.com/timescale/timescaledb-tune). # Install, update, and uninstall TimescaleDB Toolkit > Install, update, and uninstall the TimescaleDB Toolkit extension to access more hyperfunctions and function pipelines Some hyperfunctions are included by default in TimescaleDB. For additional hyperfunctions, you need to install the TimescaleDB Toolkit PostgreSQL extension. If you're hosting the TimescaleDB extension on your self-hosted database, you can install Toolkit by: * Using the TimescaleDB high-availability Docker image * Using a package manager such as `yum`, `apt`, or `brew` on platforms where pre-built binaries are available * Building from source. For more information, see the [Toolkit developer documentation](https://github.com/timescale/timescaledb-toolkit#-installing-from-source) Note From Toolkit 1.23.0, the extension is `trusted`. A role with the `CREATE` privilege on the database, such as the database owner, can run `CREATE EXTENSION timescaledb_toolkit` without superuser access. Earlier versions require a superuser to create the extension. * Docker ## Install TimescaleDB Toolkit [Section titled “Install TimescaleDB Toolkit”](#install-timescaledb-toolkit) The Toolkit extension is pre-installed and pre-enabled when you install the TimescaleDB HA docker image from a Docker container. Once you start the container, the extension is already active in your database and ready to use. No additional installation steps are required. ## Update TimescaleDB Toolkit [Section titled “Update TimescaleDB Toolkit”](#update-timescaledb-toolkit) To get the latest version of Toolkit, update the TimescaleDB HA docker image. ## Uninstall TimescaleDB Toolkit [Section titled “Uninstall TimescaleDB Toolkit”](#uninstall-timescaledb-toolkit) Toolkit is included in the TimescaleDB HA Docker image and cannot be uninstalled separately. To remove Toolkit, you need to remove the entire TimescaleDB container. See [Uninstall TimescaleDB from Docker](/docs/deploy/self-hosted/uninstall). To remove the extension from a specific database without removing the container: 1. **Connect to your database** ```bash docker exec -it timescaledb psql -U postgres -d ``` 2. **Drop the Toolkit extension** ```sql DROP EXTENSION IF EXISTS timescaledb_toolkit CASCADE; ``` Warning Using `CASCADE` drops all objects that depend on the Toolkit extension. Ensure you have backed up any data you want to keep. * Kubernetes ## Install TimescaleDB Toolkit [Section titled “Install TimescaleDB Toolkit”](#install-timescaledb-toolkit-1) The Toolkit extension is pre-installed and pre-enabled when you install TimescaleDB on Kubernetes. Once you start the container, the extension is already active in your database and ready to use. No additional installation steps are required. ## Update TimescaleDB Toolkit [Section titled “Update TimescaleDB Toolkit”](#update-timescaledb-toolkit-1) To get the latest version of Toolkit, update the TimescaleDB HA Docker image used by your Kubernetes StatefulSet: 1. **Update the StatefulSet to use the latest image** Replace `` with the desired version tag. For example, `pg18`. You find the available tags on the [TimescaleDB HA Docker Hub page](https://hub.docker.com/r/timescale/timescaledb-ha/tags). ```shell kubectl set image statefulset/timescaledb timescaledb=timescale/timescaledb-ha: -n tigerdata ``` Kubernetes performs a rolling update, replacing the pod with the new image version. 2. **Verify the new image is running** ```shell kubectl get statefulset timescaledb -n tigerdata -o jsonpath='{.spec.template.spec.containers[0].image}' ``` 3. **Connect to your TimescaleDB pod and verify the extension versions** ```shell kubectl exec -it test-pod -- bash -c "psql -h \$PGHOST -U \$PGUSER -d \$PGDATABASE" \dx ``` ## Uninstall TimescaleDB Toolkit [Section titled “Uninstall TimescaleDB Toolkit”](#uninstall-timescaledb-toolkit-1) Toolkit is included in the TimescaleDB HA Docker image and cannot be uninstalled separately. To remove Toolkit, you need to remove the entire TimescaleDB deployment. See [Uninstall TimescaleDB from Kubernetes](/docs/deploy/self-hosted/uninstall). If you only want to remove the extension from a specific database without removing the deployment: 1. **Connect to your TimescaleDB pod** ```shell kubectl exec -it test-pod -- bash -c "psql -h \$PGHOST -U \$PGUSER -d \$PGDATABASE" ``` 2. **Drop the Toolkit extension** At the `psql` prompt: ```sql DROP EXTENSION timescaledb_toolkit; ``` Warning Using `CASCADE` drops all objects that depend on the Toolkit extension. Ensure you have backed up any data you want to keep. * Linux * Debian ## Install TimescaleDB Toolkit [Section titled “Install TimescaleDB Toolkit”](#install-timescaledb-toolkit-2) These instructions use the `apt` package manager. 1. **Update your local repository list** ```bash sudo apt update ``` 2. **Install TimescaleDB Toolkit** ```bash sudo apt install timescaledb-toolkit-postgresql-18 ``` 3. **Connect to the database where you want to use Toolkit** ```bash psql -d "postgres://:@:/" ``` 4. **Create the Toolkit extension in the database** ```sql CREATE EXTENSION timescaledb_toolkit; ``` ## Update TimescaleDB Toolkit [Section titled “Update TimescaleDB Toolkit”](#update-timescaledb-toolkit-2) Update Toolkit by installing the latest version and running `ALTER EXTENSION`. 1. **Update your local repository list** ```bash apt update ``` 2. **Install the latest version of TimescaleDB Toolkit** ```bash apt install timescaledb-toolkit-postgresql-18 ``` 3. **Connect to the database where you want to use Toolkit** ```bash psql -d "postgres://:@:/" ``` 4. **Update the Toolkit extension in the database** ```sql ALTER EXTENSION timescaledb_toolkit UPDATE; ``` Note For some Toolkit versions, you might need to disconnect and reconnect active sessions. ## Uninstall TimescaleDB Toolkit [Section titled “Uninstall TimescaleDB Toolkit”](#uninstall-timescaledb-toolkit-2) If you no longer need Toolkit, you can remove it without uninstalling TimescaleDB or PostgreSQL. 1. **Drop the Toolkit extension from your databases** Connect to each database where Toolkit is enabled and remove the extension: ```bash psql -d "postgres://:@:/" ``` At the `psql` prompt: ```sql DROP EXTENSION IF EXISTS timescaledb_toolkit CASCADE; ``` Repeat this for all databases with Toolkit enabled. To exit psql, type `\q`. Warning Using `CASCADE` will drop all objects that depend on the Toolkit extension. Ensure you have backed up any data you want to keep. 2. **Uninstall the Toolkit package** ```bash sudo apt remove timescaledb-toolkit-postgresql-18 ``` Replace `18` with your PostgreSQL version if different. 3. **(Optional) Remove dependencies** To also remove unused dependencies: ```bash sudo apt autoremove ``` * Ubuntu ## Install TimescaleDB Toolkit [Section titled “Install TimescaleDB Toolkit”](#install-timescaledb-toolkit-3) These instructions use the `apt` package manager. 1. **Update your local repository list** ```bash sudo apt update ``` 2. **Install TimescaleDB Toolkit** ```bash sudo apt install timescaledb-toolkit-postgresql-18 ``` 3. **Connect to the database where you want to use Toolkit** ```bash psql -d "postgres://:@:/" ``` 4. **Create the Toolkit extension in the database** ```sql CREATE EXTENSION timescaledb_toolkit; ``` ## Update TimescaleDB Toolkit [Section titled “Update TimescaleDB Toolkit”](#update-timescaledb-toolkit-3) Update Toolkit by installing the latest version and running `ALTER EXTENSION`. 1. **Update your local repository list** ```bash apt update ``` 2. **Install the latest version of TimescaleDB Toolkit** ```bash apt install timescaledb-toolkit-postgresql-18 ``` 3. **Connect to the database where you want to use Toolkit** ```bash psql -d "postgres://:@:/" ``` 4. **Update the Toolkit extension in the database** ```sql ALTER EXTENSION timescaledb_toolkit UPDATE; ``` Note For some Toolkit versions, you might need to disconnect and reconnect active sessions. ## Uninstall TimescaleDB Toolkit [Section titled “Uninstall TimescaleDB Toolkit”](#uninstall-timescaledb-toolkit-3) If you no longer need Toolkit, you can remove it without uninstalling TimescaleDB or PostgreSQL. 1. **Drop the Toolkit extension from your databases** Connect to each database where Toolkit is enabled and remove the extension: ```bash psql -d "postgres://:@:/" ``` At the `psql` prompt: ```sql DROP EXTENSION IF EXISTS timescaledb_toolkit CASCADE; ``` Repeat this for all databases with Toolkit enabled. To exit psql, type `\q`. Warning Using `CASCADE` will drop all objects that depend on the Toolkit extension. Ensure you have backed up any data you want to keep. 2. **Uninstall the Toolkit package** ```bash sudo apt remove timescaledb-toolkit-postgresql-18 ``` Replace `18` with your PostgreSQL version if different. 3. **(Optional) Remove dependencies** To also remove unused dependencies: ```bash sudo apt autoremove ``` * Red Hat ## Install TimescaleDB Toolkit [Section titled “Install TimescaleDB Toolkit”](#install-timescaledb-toolkit-4) These instructions use the `yum` package manager. 1. **Update your local repository list** ```bash sudo yum update ``` 2. **Install TimescaleDB Toolkit** ```bash sudo yum install timescaledb-toolkit-postgresql-18 ``` 3. **Connect to the database where you want to use Toolkit** ```bash psql -d "postgres://:@:/" ``` 4. **Create the Toolkit extension in the database** ```sql CREATE EXTENSION timescaledb_toolkit; ``` ## Update TimescaleDB Toolkit [Section titled “Update TimescaleDB Toolkit”](#update-timescaledb-toolkit-4) Update Toolkit by installing the latest version and running `ALTER EXTENSION`. 1. **Update your local repository list** ```bash yum update ``` 2. **Install the latest version of TimescaleDB Toolkit** ```bash yum install timescaledb-toolkit-postgresql-18 ``` 3. **Connect to the database where you want to use Toolkit** ```bash psql -d "postgres://:@:/" ``` 4. **Update the Toolkit extension in the database** ```sql ALTER EXTENSION timescaledb_toolkit UPDATE; ``` Note For some Toolkit versions, you might need to disconnect and reconnect active sessions. ## Uninstall TimescaleDB Toolkit [Section titled “Uninstall TimescaleDB Toolkit”](#uninstall-timescaledb-toolkit-4) If you no longer need Toolkit, you can remove it without uninstalling TimescaleDB or PostgreSQL. 1. **Drop the Toolkit extension from your databases** Connect to each database where Toolkit is enabled and remove the extension: ```bash psql -d "postgres://:@:/" ``` At the `psql` prompt: ```sql DROP EXTENSION IF EXISTS timescaledb_toolkit CASCADE; ``` Repeat this for all databases with Toolkit enabled. To exit psql, type `\q`. Warning Using `CASCADE` will drop all objects that depend on the Toolkit extension. Ensure you have backed up any data you want to keep. 2. **Uninstall the Toolkit package** ```bash sudo yum remove timescaledb-toolkit-postgresql-18 ``` Replace `18` with your PostgreSQL version if different. 3. **(Optional) Remove dependencies** To also remove unused dependencies: ```bash sudo yum autoremove ``` * Fedora ## Install TimescaleDB Toolkit [Section titled “Install TimescaleDB Toolkit”](#install-timescaledb-toolkit-5) These instructions use the `yum` package manager. 1. **Update your local repository list** ```bash sudo yum update ``` 2. **Install TimescaleDB Toolkit** ```bash sudo yum install timescaledb-toolkit-postgresql-18 ``` 3. **Connect to the database where you want to use Toolkit** ```bash psql -d "postgres://:@:/" ``` 4. **Create the Toolkit extension in the database** ```sql CREATE EXTENSION timescaledb_toolkit; ``` ## Update TimescaleDB Toolkit [Section titled “Update TimescaleDB Toolkit”](#update-timescaledb-toolkit-5) Update Toolkit by installing the latest version and running `ALTER EXTENSION`. 1. **Update your local repository list** ```bash yum update ``` 2. **Install the latest version of TimescaleDB Toolkit** ```bash yum install timescaledb-toolkit-postgresql-18 ``` 3. **Connect to the database where you want to use Toolkit** ```bash psql -d "postgres://:@:/" ``` 4. **Update the Toolkit extension in the database** ```sql ALTER EXTENSION timescaledb_toolkit UPDATE; ``` Note For some Toolkit versions, you might need to disconnect and reconnect active sessions. ## Uninstall TimescaleDB Toolkit [Section titled “Uninstall TimescaleDB Toolkit”](#uninstall-timescaledb-toolkit-5) If you no longer need Toolkit, you can remove it without uninstalling TimescaleDB or PostgreSQL. 1. **Drop the Toolkit extension from your databases** Connect to each database where Toolkit is enabled and remove the extension: ```bash psql -d "postgres://:@:/" ``` At the `psql` prompt: ```sql DROP EXTENSION IF EXISTS timescaledb_toolkit CASCADE; ``` Repeat this for all databases with Toolkit enabled. To exit psql, type `\q`. Warning Using `CASCADE` will drop all objects that depend on the Toolkit extension. Ensure you have backed up any data you want to keep. 2. **Uninstall the Toolkit package** ```bash sudo yum remove timescaledb-toolkit-postgresql-18 ``` Replace `18` with your PostgreSQL version if different. 3. **(Optional) Remove dependencies** To also remove unused dependencies: ```bash sudo yum autoremove ``` * RockyLinux ## Install TimescaleDB Toolkit [Section titled “Install TimescaleDB Toolkit”](#install-timescaledb-toolkit-6) These instructions use the `dnf` package manager. 1. **Update your local repository list** ```bash sudo dnf update ``` 2. **Install TimescaleDB Toolkit** ```bash sudo dnf install timescaledb-toolkit-postgresql-18 ``` 3. **Connect to the database where you want to use Toolkit** ```bash psql -d "postgres://:@:/" ``` 4. **Create the Toolkit extension in the database** ```sql CREATE EXTENSION timescaledb_toolkit; ``` ## Update TimescaleDB Toolkit [Section titled “Update TimescaleDB Toolkit”](#update-timescaledb-toolkit-6) Update Toolkit by installing the latest version and running `ALTER EXTENSION`. 1. **Update your local repository list** ```bash sudo dnf update ``` 2. **Install the latest version of TimescaleDB Toolkit** ```bash sudo dnf install timescaledb-toolkit-postgresql-18 ``` 3. **Connect to the database where you want to use Toolkit** ```bash psql -d "postgres://:@:/" ``` 4. **Update the Toolkit extension in the database** ```sql ALTER EXTENSION timescaledb_toolkit UPDATE; ``` Note For some Toolkit versions, you might need to disconnect and reconnect active sessions. ## Uninstall TimescaleDB Toolkit [Section titled “Uninstall TimescaleDB Toolkit”](#uninstall-timescaledb-toolkit-6) If you no longer need Toolkit, you can remove it without uninstalling TimescaleDB or PostgreSQL. 1. **Drop the Toolkit extension from your databases** Connect to each database where Toolkit is enabled and remove the extension: ```bash psql -d "postgres://:@:/" ``` At the `psql` prompt: ```sql DROP EXTENSION IF EXISTS timescaledb_toolkit CASCADE; ``` Repeat this for all databases with Toolkit enabled. To exit psql, type `\q`. Warning Using `CASCADE` will drop all objects that depend on the Toolkit extension. Ensure you have backed up any data you want to keep. 2. **Uninstall the Toolkit package** ```bash sudo dnf remove timescaledb-toolkit-postgresql-18 ``` Replace `18` with your PostgreSQL version if different. 3. **(Optional) Remove dependencies** To also remove unused dependencies: ```bash sudo dnf autoremove ``` * macOS ## Install TimescaleDB Toolkit [Section titled “Install TimescaleDB Toolkit”](#install-timescaledb-toolkit-2) These instructions use the `apt` package manager. 1. **Update your local repository list** ```bash sudo apt update ``` 2. **Install TimescaleDB Toolkit** ```bash sudo apt install timescaledb-toolkit-postgresql-18 ``` 3. **Connect to the database where you want to use Toolkit** ```bash psql -d "postgres://:@:/" ``` 4. **Create the Toolkit extension in the database** ```sql CREATE EXTENSION timescaledb_toolkit; ``` ## Update TimescaleDB Toolkit [Section titled “Update TimescaleDB Toolkit”](#update-timescaledb-toolkit-2) Update Toolkit by installing the latest version and running `ALTER EXTENSION`. 1. **Update your local repository list** ```bash apt update ``` 2. **Install the latest version of TimescaleDB Toolkit** ```bash apt install timescaledb-toolkit-postgresql-18 ``` 3. **Connect to the database where you want to use Toolkit** ```bash psql -d "postgres://:@:/" ``` 4. **Update the Toolkit extension in the database** ```sql ALTER EXTENSION timescaledb_toolkit UPDATE; ``` Note For some Toolkit versions, you might need to disconnect and reconnect active sessions. ## Uninstall TimescaleDB Toolkit [Section titled “Uninstall TimescaleDB Toolkit”](#uninstall-timescaledb-toolkit-2) If you no longer need Toolkit, you can remove it without uninstalling TimescaleDB or PostgreSQL. 1. **Drop the Toolkit extension from your databases** Connect to each database where Toolkit is enabled and remove the extension: ```bash psql -d "postgres://:@:/" ``` At the `psql` prompt: ```sql DROP EXTENSION IF EXISTS timescaledb_toolkit CASCADE; ``` Repeat this for all databases with Toolkit enabled. To exit psql, type `\q`. Warning Using `CASCADE` will drop all objects that depend on the Toolkit extension. Ensure you have backed up any data you want to keep. 2. **Uninstall the Toolkit package** ```bash sudo apt remove timescaledb-toolkit-postgresql-18 ``` Replace `18` with your PostgreSQL version if different. 3. **(Optional) Remove dependencies** To also remove unused dependencies: ```bash sudo apt autoremove ``` * Source ## Install TimescaleDB Toolkit [Section titled “Install TimescaleDB Toolkit”](#install-timescaledb-toolkit-3) These instructions use the `apt` package manager. 1. **Update your local repository list** ```bash sudo apt update ``` 2. **Install TimescaleDB Toolkit** ```bash sudo apt install timescaledb-toolkit-postgresql-18 ``` 3. **Connect to the database where you want to use Toolkit** ```bash psql -d "postgres://:@:/" ``` 4. **Create the Toolkit extension in the database** ```sql CREATE EXTENSION timescaledb_toolkit; ``` ## Update TimescaleDB Toolkit [Section titled “Update TimescaleDB Toolkit”](#update-timescaledb-toolkit-3) Update Toolkit by installing the latest version and running `ALTER EXTENSION`. 1. **Update your local repository list** ```bash apt update ``` 2. **Install the latest version of TimescaleDB Toolkit** ```bash apt install timescaledb-toolkit-postgresql-18 ``` 3. **Connect to the database where you want to use Toolkit** ```bash psql -d "postgres://:@:/" ``` 4. **Update the Toolkit extension in the database** ```sql ALTER EXTENSION timescaledb_toolkit UPDATE; ``` Note For some Toolkit versions, you might need to disconnect and reconnect active sessions. ## Uninstall TimescaleDB Toolkit [Section titled “Uninstall TimescaleDB Toolkit”](#uninstall-timescaledb-toolkit-3) If you no longer need Toolkit, you can remove it without uninstalling TimescaleDB or PostgreSQL. 1. **Drop the Toolkit extension from your databases** Connect to each database where Toolkit is enabled and remove the extension: ```bash psql -d "postgres://:@:/" ``` At the `psql` prompt: ```sql DROP EXTENSION IF EXISTS timescaledb_toolkit CASCADE; ``` Repeat this for all databases with Toolkit enabled. To exit psql, type `\q`. Warning Using `CASCADE` will drop all objects that depend on the Toolkit extension. Ensure you have backed up any data you want to keep. 2. **Uninstall the Toolkit package** ```bash sudo apt remove timescaledb-toolkit-postgresql-18 ``` Replace `18` with your PostgreSQL version if different. 3. **(Optional) Remove dependencies** To also remove unused dependencies: ```bash sudo apt autoremove ``` * Debian ## Install TimescaleDB Toolkit [Section titled “Install TimescaleDB Toolkit”](#install-timescaledb-toolkit-4) These instructions use the `yum` package manager. 1. **Update your local repository list** ```bash sudo yum update ``` 2. **Install TimescaleDB Toolkit** ```bash sudo yum install timescaledb-toolkit-postgresql-18 ``` 3. **Connect to the database where you want to use Toolkit** ```bash psql -d "postgres://:@:/" ``` 4. **Create the Toolkit extension in the database** ```sql CREATE EXTENSION timescaledb_toolkit; ``` ## Update TimescaleDB Toolkit [Section titled “Update TimescaleDB Toolkit”](#update-timescaledb-toolkit-4) Update Toolkit by installing the latest version and running `ALTER EXTENSION`. 1. **Update your local repository list** ```bash yum update ``` 2. **Install the latest version of TimescaleDB Toolkit** ```bash yum install timescaledb-toolkit-postgresql-18 ``` 3. **Connect to the database where you want to use Toolkit** ```bash psql -d "postgres://:@:/" ``` 4. **Update the Toolkit extension in the database** ```sql ALTER EXTENSION timescaledb_toolkit UPDATE; ``` Note For some Toolkit versions, you might need to disconnect and reconnect active sessions. ## Uninstall TimescaleDB Toolkit [Section titled “Uninstall TimescaleDB Toolkit”](#uninstall-timescaledb-toolkit-4) If you no longer need Toolkit, you can remove it without uninstalling TimescaleDB or PostgreSQL. 1. **Drop the Toolkit extension from your databases** Connect to each database where Toolkit is enabled and remove the extension: ```bash psql -d "postgres://:@:/" ``` At the `psql` prompt: ```sql DROP EXTENSION IF EXISTS timescaledb_toolkit CASCADE; ``` Repeat this for all databases with Toolkit enabled. To exit psql, type `\q`. Warning Using `CASCADE` will drop all objects that depend on the Toolkit extension. Ensure you have backed up any data you want to keep. 2. **Uninstall the Toolkit package** ```bash sudo yum remove timescaledb-toolkit-postgresql-18 ``` Replace `18` with your PostgreSQL version if different. 3. **(Optional) Remove dependencies** To also remove unused dependencies: ```bash sudo yum autoremove ``` * Ubuntu ## Install TimescaleDB Toolkit [Section titled “Install TimescaleDB Toolkit”](#install-timescaledb-toolkit-5) These instructions use the `yum` package manager. 1. **Update your local repository list** ```bash sudo yum update ``` 2. **Install TimescaleDB Toolkit** ```bash sudo yum install timescaledb-toolkit-postgresql-18 ``` 3. **Connect to the database where you want to use Toolkit** ```bash psql -d "postgres://:@:/" ``` 4. **Create the Toolkit extension in the database** ```sql CREATE EXTENSION timescaledb_toolkit; ``` ## Update TimescaleDB Toolkit [Section titled “Update TimescaleDB Toolkit”](#update-timescaledb-toolkit-5) Update Toolkit by installing the latest version and running `ALTER EXTENSION`. 1. **Update your local repository list** ```bash yum update ``` 2. **Install the latest version of TimescaleDB Toolkit** ```bash yum install timescaledb-toolkit-postgresql-18 ``` 3. **Connect to the database where you want to use Toolkit** ```bash psql -d "postgres://:@:/" ``` 4. **Update the Toolkit extension in the database** ```sql ALTER EXTENSION timescaledb_toolkit UPDATE; ``` Note For some Toolkit versions, you might need to disconnect and reconnect active sessions. ## Uninstall TimescaleDB Toolkit [Section titled “Uninstall TimescaleDB Toolkit”](#uninstall-timescaledb-toolkit-5) If you no longer need Toolkit, you can remove it without uninstalling TimescaleDB or PostgreSQL. 1. **Drop the Toolkit extension from your databases** Connect to each database where Toolkit is enabled and remove the extension: ```bash psql -d "postgres://:@:/" ``` At the `psql` prompt: ```sql DROP EXTENSION IF EXISTS timescaledb_toolkit CASCADE; ``` Repeat this for all databases with Toolkit enabled. To exit psql, type `\q`. Warning Using `CASCADE` will drop all objects that depend on the Toolkit extension. Ensure you have backed up any data you want to keep. 2. **Uninstall the Toolkit package** ```bash sudo yum remove timescaledb-toolkit-postgresql-18 ``` Replace `18` with your PostgreSQL version if different. 3. **(Optional) Remove dependencies** To also remove unused dependencies: ```bash sudo yum autoremove ``` * Red Hat ## Install TimescaleDB Toolkit [Section titled “Install TimescaleDB Toolkit”](#install-timescaledb-toolkit-6) These instructions use the `dnf` package manager. 1. **Update your local repository list** ```bash sudo dnf update ``` 2. **Install TimescaleDB Toolkit** ```bash sudo dnf install timescaledb-toolkit-postgresql-18 ``` 3. **Connect to the database where you want to use Toolkit** ```bash psql -d "postgres://:@:/" ``` 4. **Create the Toolkit extension in the database** ```sql CREATE EXTENSION timescaledb_toolkit; ``` ## Update TimescaleDB Toolkit [Section titled “Update TimescaleDB Toolkit”](#update-timescaledb-toolkit-6) Update Toolkit by installing the latest version and running `ALTER EXTENSION`. 1. **Update your local repository list** ```bash sudo dnf update ``` 2. **Install the latest version of TimescaleDB Toolkit** ```bash sudo dnf install timescaledb-toolkit-postgresql-18 ``` 3. **Connect to the database where you want to use Toolkit** ```bash psql -d "postgres://:@:/" ``` 4. **Update the Toolkit extension in the database** ```sql ALTER EXTENSION timescaledb_toolkit UPDATE; ``` Note For some Toolkit versions, you might need to disconnect and reconnect active sessions. ## Uninstall TimescaleDB Toolkit [Section titled “Uninstall TimescaleDB Toolkit”](#uninstall-timescaledb-toolkit-6) If you no longer need Toolkit, you can remove it without uninstalling TimescaleDB or PostgreSQL. 1. **Drop the Toolkit extension from your databases** Connect to each database where Toolkit is enabled and remove the extension: ```bash psql -d "postgres://:@:/" ``` At the `psql` prompt: ```sql DROP EXTENSION IF EXISTS timescaledb_toolkit CASCADE; ``` Repeat this for all databases with Toolkit enabled. To exit psql, type `\q`. Warning Using `CASCADE` will drop all objects that depend on the Toolkit extension. Ensure you have backed up any data you want to keep. 2. **Uninstall the Toolkit package** ```bash sudo dnf remove timescaledb-toolkit-postgresql-18 ``` Replace `18` with your PostgreSQL version if different. 3. **(Optional) Remove dependencies** To also remove unused dependencies: ```bash sudo dnf autoremove ``` * Fedora ## Install TimescaleDB Toolkit [Section titled “Install TimescaleDB Toolkit”](#install-timescaledb-toolkit-7) These instructions use the `brew` package manager. For more information on installing or using Homebrew, see [the `brew` homepage](https://brew.sh). 1. **Tap the Tiger Data formula repository** This also contains formulae for TimescaleDB and `timescaledb-tune`. ```bash brew tap timescale/tap ``` 2. **Update your local brew installation** ```bash brew update ``` 3. **Install TimescaleDB Toolkit** ```bash brew install timescaledb-toolkit ``` 4. **Connect to PostgreSQL** ```bash psql postgres ``` 5. **Create the Toolkit extension in the database** ```sql CREATE EXTENSION timescaledb_toolkit; ``` Note If you encounter an error like `could not access file "$libdir/timescaledb_toolkit-X.XX.X"`, you may need to create a symlink from `.so` to `.dylib` format: ```bash ln -sf $(pg_config --pkglibdir)/timescaledb_toolkit-*.so \ $(pg_config --pkglibdir)/timescaledb_toolkit-*.dylib ``` This is due to a known issue in the Homebrew formula where the library is installed with a `.so` extension, but PostgreSQL on macOS expects a `.dylib` extension. ## Update TimescaleDB Toolkit [Section titled “Update TimescaleDB Toolkit”](#update-timescaledb-toolkit-7) Update Toolkit by installing the latest version and running `ALTER EXTENSION`. 1. **Update your local repository list** ```bash brew update ``` 2. **Install the latest version of TimescaleDB Toolkit** ```bash brew upgrade timescaledb-toolkit ``` 3. **Connect to PostgreSQL** ```bash psql postgres ``` 4. **Update the Toolkit extension in the database** ```sql ALTER EXTENSION timescaledb_toolkit UPDATE; ``` Note For some Toolkit versions, you might need to disconnect and reconnect active sessions. ## Uninstall TimescaleDB Toolkit [Section titled “Uninstall TimescaleDB Toolkit”](#uninstall-timescaledb-toolkit-7) If you no longer need Toolkit, you can remove it without uninstalling TimescaleDB or PostgreSQL. 1. **Drop the Toolkit extension from your databases** Connect to each database where Toolkit is enabled and remove the extension: ```bash psql -d "postgres://:@:/" ``` At the `psql` prompt: ```sql DROP EXTENSION IF EXISTS timescaledb_toolkit CASCADE; ``` Repeat this for all databases with Toolkit enabled. To exit psql, type `\q`. Warning Using `CASCADE` will drop all objects that depend on the Toolkit extension. Ensure you have backed up any data you want to keep. 2. **Uninstall the Toolkit package** ```bash brew uninstall timescaledb-toolkit ``` 3. **(Optional) Remove manual symlinks** If you created a manual `.dylib` symlink as a workaround during installation, remove it: ```bash rm $(pg_config --pkglibdir)/timescaledb_toolkit-*.dylib 2>/dev/null || true ``` * RockyLinux ## Install TimescaleDB Toolkit [Section titled “Install TimescaleDB Toolkit”](#install-timescaledb-toolkit-8) To install Toolkit from source: 1. **Build the extension from the latest source code** Follow the build instructions in the [TimescaleDB Toolkit GitHub repository](https://github.com/timescale/timescaledb-toolkit#-installing-from-source) to download and build the latest version. 2. **Connect to your database** ```bash psql -d "postgres://:@:/" ``` 3. **Create the Toolkit extension in the database** ```sql CREATE EXTENSION timescaledb_toolkit; ``` ## Update TimescaleDB Toolkit [Section titled “Update TimescaleDB Toolkit”](#update-timescaledb-toolkit-8) To update Toolkit installed from source: 1. **Build the extension from the latest source code** Follow the build instructions in the [TimescaleDB Toolkit GitHub repository](https://github.com/timescale/timescaledb-toolkit#-installing-from-source) to download and build the latest version. 2. **Connect to your database** ```bash psql -d "postgres://:@:/" ``` 3. **Update the Toolkit extension in the database** ```sql ALTER EXTENSION timescaledb_toolkit UPDATE; ``` ## Uninstall TimescaleDB Toolkit [Section titled “Uninstall TimescaleDB Toolkit”](#uninstall-timescaledb-toolkit-8) If you installed Toolkit by building from source, you can uninstall it without removing TimescaleDB or PostgreSQL. 1. **Drop the Toolkit extension from your databases** Connect to each database where Toolkit is enabled and remove the extension: ```bash psql -d "postgres://:@:/" ``` At the `psql` prompt: ```sql DROP EXTENSION IF EXISTS timescaledb_toolkit CASCADE; ``` Repeat this for all databases with Toolkit enabled. To exit psql, type `\q`. Warning Using `CASCADE` will drop all objects that depend on the Toolkit extension. Ensure you have backed up any data you want to keep. # Troubleshooting > Troubleshoot common problems with self-hosted TimescaleDB ## Failed to start a background worker You might see this error message in the logs if background workers aren't properly configured: ```bash "": failed to start a background worker ``` To fix this error, make sure that `max_worker_processes`, `max_parallel_workers`, and `timescaledb.max_background_workers` are properly set. `timescaledb.max_background_workers` should equal the number of databases plus the number of concurrent background workers. `max_worker_processes` should equal the sum of `timescaledb.max_background_workers` and `max_parallel_workers`. For more information, see the [configuration docs](/docs/deploy/self-hosted/configuration/about-configuration). ## Log error: could not access file "timescaledb" If your PostgreSQL logs have this error preventing it from starting up, you should double-check that the TimescaleDB files have been installed to the correct location. The installation methods use `pg_config` to get PostgreSQL's location. However, if you have multiple versions of PostgreSQL installed on the same machine, the location `pg_config` points to may not be for the version you expect. To check which version of TimescaleDB is used: ```bash $ pg_config --version PostgreSQL 12.3 ``` If that is the correct version, double-check that the installation path is the one you'd expect. For example, for PostgreSQL 11.0 installed via Homebrew on macOS it should be `/usr/local/Cellar/postgresql/11.0/bin`: ```bash $ pg_config --bindir /usr/local/Cellar/postgresql/11.0/bin ``` If either of those steps is not the version you are expecting, you need to either uninstall the incorrect version of PostgreSQL if you can, or update your `PATH` environmental variable to have the correct path of `pg_config` listed first, that is, by prepending the full path: ```bash export PATH = /usr/local/Cellar/postgresql/11.0/bin:$PATH ``` Then, reinstall TimescaleDB and it should find the correct installation path. ## Warnings during `pg_dump` migration You might see these warnings during the migration process: ```bash pg_dump: warning: there are circular foreign-key constraints on this table: pg_dump: hypertable pg_dump: You might not be able to restore the dump without using --disable-triggers or temporarily dropping the constraints. pg_dump: Consider using a full dump instead of a --data-only dump to avoid this problem. ``` ```bash pg_dump: NOTICE: hypertable data are in the chunks, no data will be copied DETAIL: Data for hypertables are stored in the chunks of a hypertable so COPY TO of a hypertable will not copy any data. HINT: Use "COPY (SELECT * FROM ) TO ..." to copy all data in hypertable, or copy each chunk individually. ``` ```bash WARNING: skipping "
" --- only superuser can analyze it ``` You can safely ignore these errors. The migration still occurs successfully. ## Permission error during pg\_restore migration The `pg_restore` function tries to apply the TimescaleDB extension when it copies your schema. This can cause a permissions error: ```bash pg_restore: creating EXTENSION "timescaledb" pg_restore: creating COMMENT "EXTENSION timescaledb" pg_restore: while PROCESSING TOC: pg_restore: from TOC entry 6239; 0 0 COMMENT EXTENSION timescaledb pg_restore: error: could not execute query: ERROR: must be owner of extension timescaledb ``` If you already have the TimescaleDB extension installed, you can safely ignore this. ## Permission denied for table `job_errors` when `running pg_dump` You might see this error when running `pg_dump`: ```bash pg_dump: error: query failed: ERROR: permission denied for table job_errors pg_dump: detail: Query was: LOCK TABLE _timescaledb_internal.job_errors IN ACCESS SHARE MODE ``` When the `pg_dump` tool tries to acquire a lock on the `job_errors` table, if the user doesn't have the required SELECT permission, it results in this error. To resolve this issue, use a superuser account to grant the necessary permissions to the user requiring the `pg_dump` tool. Use this command to grant permissions to ``: ```sql GRANT SELECT ON TABLE _timescaledb_internal.job_errors TO ; ``` ## Errors when running `pg_dump` You might see this notice when running `pg_dump`: ```bash pg_dump: NOTICE: hypertable data are in the chunks, no data will be copied DETAIL: Data for hypertables are stored in the chunks of a hypertable so COPY TO of a hypertable will not copy any data. HINT: Use "COPY (SELECT * FROM ) TO ..." to copy all data in hypertable, or copy each chunk individually. ``` You can safely ignore this. Your hypertable data is still accurately copied. ## Errors after restoring from file dump You might see this error after restoring from a file dump: ```bash org.postgresql.util.PSQLException: ERROR: invalid INSERT on the root table of hypertable "_hyper_1_10_chunk. ``` When loading from a logical dump make sure that you set `timescaledb.restoring` to `true` before loading the dump. ## Version mismatch when dumping and restoring a database The PostgreSQL `pg_dump` command does not allow you to specify which version of the extension to use when backing up. This can create problems if you have a more recent version installed. For example, if you create the backup using an older version of TimescaleDB, and when you restore it uses the current version, without giving you an opportunity to upgrade first. You can work around this problem when you are restoring from backup by making sure the new PostgreSQL instance has the same extension version as the original database before you perform the restore. After the data is restored, you can upgrade the version of TimescaleDB. ## Install or upgrade of TimescaleDB Toolkit fails In some cases, when you create the TimescaleDB Toolkit extension, or upgrade it with the `ALTER EXTENSION timescaledb_toolkit UPDATE` command, it might fail with this error: ```sql ERROR: extension "timescaledb_toolkit" has no update path from version "1.2" to version "1.3" ``` This occurs if the list of available extensions does not include the version you are trying to upgrade to, and it can occur if the package was not installed correctly in the first place. To correct the problem, install the upgrade package, restart PostgreSQL, verify the version, and then attempt the update again. 1. **Check your package repository** If you're installing Toolkit from a package, check your package manager's local repository list. Make sure the TimescaleDB repository is available and contains Toolkit. Update your local repository list with `apt update` or `yum update`. 2. **Restart PostgreSQL** Restart your PostgreSQL service. 3. **Verify the available version** Check that the right version of Toolkit is among your available extensions: ```sql SELECT * FROM pg_available_extensions WHERE name = 'timescaledb_toolkit'; ``` The result should look like this: ```bash -[ RECORD 1 ]-----+-------------------------------------------------------------------------------------- name | timescaledb_toolkit default_version | 1.6.0 installed_version | 1.6.0 comment | Library of analytical hyperfunctions, time-series pipelining, and other SQL utilities ``` 4. **Retry the command** Retry `CREATE EXTENSION` or `ALTER EXTENSION`. ## Upgrade fails with "old version has already been loaded" When you use the `ALTER EXTENSION timescaledb UPDATE` command to upgrade, you might see this error: ```sql ERROR: extension "timescaledb" cannot be updated after the old version has already been loaded ``` This occurs if you don't run `ALTER EXTENSION timescaledb UPDATE` as the first command after starting a new session using psql, or if you use tab completion when running the command. Tab completion triggers metadata queries in the background which prevents the alter extension from being the first command. To correct the problem, execute the ALTER EXTENSION command like this: ```sql psql -X -c 'ALTER EXTENSION timescaledb UPDATE;' ``` ## Could not access file "timescaledb-\": No such file or directory If this error occurs immediately after updating your version of TimescaleDB and the file mentioned is from the previous version, it is probably due to an incomplete update process. Within the greater PostgreSQL server instance, each database that has TimescaleDB installed needs to be updated with the SQL command `ALTER EXTENSION timescaledb UPDATE;` while connected to that database. Otherwise, the database looks for the previous version of the `timescaledb` files. See the [upgrade docs](/docs/deploy/self-hosted/upgrades) for more info. ## Error loading the TimescaleDB extension on Windows If you see a message saying that PostgreSQL cannot load the TimescaleDB library `timescaledb-.dll`: ```sql ERROR: could not load library "C:/Program Files/PostgreSQL/16/lib/timescaledb-2.14.2.dll": The specified module could not be found. ``` Start a new psql session to your self-hosted instance and create the `timescaledb` extension as the first command: ```bash psql -X -d "postgres://:@:/" -c "CREATE EXTENSION IF NOT EXISTS timescaledb;" ``` ## Upgrade fails with no update path When you use the `ALTER EXTENSION timescaledb UPDATE` command to upgrade, it might fail with this error: ```sql ERROR: extension "timescaledb" has no update path from version "X.Y" to version "X.Z" ``` This occurs if the list of available extensions does not include the version you are trying to upgrade to, and it can occur if the package was not installed correctly in the first place. To correct the problem, install the upgrade package, restart PostgreSQL, verify the version, and then attempt the upgrade again. ## Background jobs failing with "out of shared memory" You might see this error when running compression, continuous aggregate refresh, or other background jobs that touch many chunks: ```text FATAL: out of shared memory HINT: You might need to increase max_locks_per_transaction. ``` Despite the wording, this error is **not caused by insufficient RAM**. It means the PostgreSQL lock table is full. TimescaleDB acquires a lock on every chunk involved in a query or background job. When a hypertable has many chunks, these locks can exceed the default `max_locks_per_transaction` limit (usually 64). ### Diagnose the issue 1. **Check the current setting:** ```sql SELECT name, setting FROM pg_settings WHERE name = 'max_locks_per_transaction'; ``` 2. **Count chunks per hypertable:** ```sql SELECT hypertable_name, num_chunks FROM timescaledb_information.hypertables ORDER BY num_chunks DESC; ``` ### Calculate the right value For most workloads, use this formula: ```text max_locks_per_transaction = (2 × max_chunks_in_any_hypertable) / max_connections ``` The factor of 2 accounts for index locks. Round up and add headroom for future growth, because changing this parameter requires a database restart. ### Apply the fix Tips **Tiger Cloud:** Adjust `max_locks_per_transaction` from Tiger Console under **Database configuration → Advanced parameters**. Search for the parameter, edit the value, and click **Apply changes and restart**. See [Advanced parameters](/docs/deploy/tiger-cloud/tiger-cloud-aws/configuration/advanced-parameters) for details. **Self-hosted:** Set `max_locks_per_transaction` in the `postgresql.conf` configuration file, then restart PostgreSQL. For more information, see [Transaction lock settings](/docs/reference/timescaledb/configuration/tiger-postgres#transaction-lock-settings) and the [PostgreSQL lock management documentation](https://www.postgresql.org/docs/current/runtime-config-locks.html). # Uninstall TimescaleDB > Uninstall TimescaleDB without uninstalling PostgreSQL You can uninstall TimescaleDB without uninstalling PostgreSQL. Choose your platform below. * Docker If you installed TimescaleDB using Docker, you can completely remove the TimescaleDB container, image, and optionally the data volumes. 1. **Stop the running container** ```bash docker stop timescaledb ``` If you named your container differently when you created it, replace `timescaledb` with your container name. 2. **Remove the container** ```bash docker rm timescaledb ``` This removes the container but preserves the data volume and the Docker image. 3. **List and remove the Docker image** 1. See which TimescaleDB images you have installed: ```bash docker images | grep timescale ``` You see something like: ```bash timescale/timescaledb-ha pg18 1ec79f20f47d 9 hours ago 6.2GB ``` 2. Remove the TimescaleDB image: * TimescaleDB-HA ```bash docker rmi timescale/timescaledb-ha:pg18 ``` * TimescaleDB light ```bash docker rmi timescale/timescaledb:latest-pg18 ``` Replace `pg18` with your PostgreSQL version. 4. **(Optional) Remove the data** Warning This step permanently deletes all your database data. Only proceed if you're sure you no longer need this data or have backed it up. List all Docker volumes: ```bash docker volume ls ``` If you used a named volume when creating your container, remove it: ```bash docker volume rm ``` If you used a host directory mount (with the `-v :/pgdata` flag), you can manually delete that directory: ```bash rm -rf ``` * Kubernetes ```bash docker rmi timescale/timescaledb-ha:pg18 ``` * Linux ```bash docker rmi timescale/timescaledb:latest-pg18 ``` * macOS If you deployed TimescaleDB on Kubernetes, you can completely remove all associated resources including the StatefulSet, Service, PersistentVolumeClaim, Secret, and application deployments. 1. **Back up any important data from your TimescaleDB instance** ```shell kubectl exec -it timescaledb-0 -- pg_dump -U postgres postgres > backup.sql ``` 2. **If you created a test pod during installation, delete it** ```shell kubectl delete pod test-pod ``` 3. **Remove any application deployments that connect to TimescaleDB** ```shell kubectl delete deployment tigerdata-app ``` 4. **Remove the service that exposes TimescaleDB within the cluster** ```shell kubectl delete service timescaledb ``` 5. **Delete the StatefulSet managing the TimescaleDB pods** ```shell kubectl delete statefulset timescaledb ``` This terminates the TimescaleDB pod. 6. **Delete the PersistentVolumeClaim** Warning Deleting the PersistentVolumeClaim permanently deletes all your database data. Ensure you have backed up any data you need before proceeding. ```shell kubectl delete pvc tigerdata-pvc ``` 7. **Remove the Kubernetes secret containing database credentials** ```shell kubectl delete secret tigerdata-secret ``` 8. **(Optional) Delete the namespace** If you created a dedicated namespace for TimescaleDB, you can remove it as well. Warning Only delete the namespace if you're certain no other resources are using it. This will delete all resources in the namespace. ```shell kubectl delete namespace tigerdata ``` * Windows Take the following steps to uninstall TimescaleDB based on your distribution: * Debian 1. **Drop the TimescaleDB extension from your databases** 1. Connect to each database where TimescaleDB is enabled and remove the extension: ```bash sudo -u postgres psql -d ``` 2. At the `psql` prompt: ```sql DROP EXTENSION timescaledb CASCADE; ``` Repeat this for all databases with TimescaleDB enabled. To exit psql, type `\q`. Warning Using `CASCADE` will drop all TimescaleDB-specific objects including hypertables, continuous aggregates, and retention policies. Ensure you have backed up any data you want to keep. 2. **Remove TimescaleDB from shared\_preload\_libraries** Edit the PostgreSQL configuration file: ```bash sudo vi /etc/postgresql/18/main/postgresql.conf ``` Find the line with `shared_preload_libraries` and remove `timescaledb` from the list. For example, change: ```plaintext shared_preload_libraries = 'timescaledb' ``` to: ```plaintext shared_preload_libraries = '' ``` If there are other extensions in the list, keep them and only remove `timescaledb`. 3. **Restart PostgreSQL** ```bash sudo systemctl restart postgresql ``` 4. **Uninstall the TimescaleDB package** ```bash sudo apt remove timescaledb-2-postgresql-18 ``` Replace `18` with your PostgreSQL version if different. 5. **Remove the TimescaleDB repository configuration** ```bash sudo rm /etc/apt/sources.list.d/timescaledb.list sudo rm /etc/apt/trusted.gpg.d/timescaledb.gpg sudo apt update ``` 6. **Remove dependencies** Remove unused dependencies installed with TimescaleDB: Warning This step may also remove PostgreSQL as a dependency. ```bash sudo apt autoremove ``` * Ubuntu 1. **Drop the TimescaleDB extension from your databases** 1. Connect to each database where TimescaleDB is enabled and remove the extension: ```bash sudo -u postgres psql -d ``` 2. At the `psql` prompt: ```sql DROP EXTENSION timescaledb CASCADE; ``` Repeat this for all databases with TimescaleDB enabled. To exit psql, type `\q`. Warning Using `CASCADE` will drop all TimescaleDB-specific objects including hypertables, continuous aggregates, and retention policies. Ensure you have backed up any data you want to keep. 2. **Remove TimescaleDB from shared\_preload\_libraries** Edit the PostgreSQL configuration file: ```bash sudo vi /etc/postgresql/18/main/postgresql.conf ``` Find the line with `shared_preload_libraries` and remove `timescaledb` from the list. For example, change: ```plaintext shared_preload_libraries = 'timescaledb' ``` to: ```plaintext shared_preload_libraries = '' ``` If there are other extensions in the list, keep them and only remove `timescaledb`. 3. **Restart PostgreSQL** ```bash sudo systemctl restart postgresql ``` 4. **Uninstall the TimescaleDB package** ```bash sudo apt remove timescaledb-2-postgresql-18 ``` Replace `18` with your PostgreSQL version if different. 5. **Remove the TimescaleDB repository configuration** ```bash sudo rm /etc/apt/sources.list.d/timescaledb.list sudo rm /etc/apt/trusted.gpg.d/timescaledb.gpg sudo apt update ``` 6. **Remove dependencies** Remove unused dependencies installed with TimescaleDB: Warning This step may also remove PostgreSQL as a dependency. ```bash sudo apt autoremove ``` * Red Hat 1. **Drop the TimescaleDB extension from your databases** 1. Connect to each database where TimescaleDB is enabled and remove the extension: ```bash sudo -u postgres psql -d ``` 2. At the `psql` prompt: ```sql DROP EXTENSION timescaledb CASCADE; ``` Repeat this for all databases with TimescaleDB enabled. To exit psql, type `\q`. Warning Using `CASCADE` will drop all TimescaleDB-specific objects including hypertables, continuous aggregates, and retention policies. Ensure you have backed up any data you want to keep. 2. **Remove TimescaleDB from shared\_preload\_libraries** Edit the PostgreSQL configuration file: ```bash sudo vi /var/lib/pgsql/18/data/postgresql.conf ``` Press `/` to search, type `shared_preload_libraries`, and press Enter. Press `i` to edit. Find the line with `shared_preload_libraries` and remove `timescaledb` from the list. For example, change: ```plaintext shared_preload_libraries = 'timescaledb' ``` to: ```plaintext shared_preload_libraries = '' ``` If there are other extensions in the list, keep them and only remove `timescaledb`. Press `Esc`, then type `:wq` and press Enter to save. 3. **Restart PostgreSQL** ```bash sudo systemctl restart postgresql-18 ``` Replace `18` with your PostgreSQL version if different. 4. **Uninstall the TimescaleDB package** ```bash sudo dnf remove timescaledb-2-postgresql-18 ``` Replace `18` with your PostgreSQL version if different. 5. **Remove the TimescaleDB repository configuration** ```bash sudo rm /etc/yum.repos.d/timescale_timescaledb.repo ``` 6. **Remove dependencies** Remove unused dependencies installed with TimescaleDB: Warning This step may also remove PostgreSQL as a dependency. ```bash sudo dnf autoremove ``` * Fedora 1. **Drop the TimescaleDB extension from your databases** 1. Connect to each database where TimescaleDB is enabled and remove the extension: ```bash sudo -u postgres psql -d ``` 2. At the `psql` prompt: ```sql DROP EXTENSION timescaledb CASCADE; ``` Repeat this for all databases with TimescaleDB enabled. To exit psql, type `\q`. Warning Using `CASCADE` will drop all TimescaleDB-specific objects including hypertables, continuous aggregates, and retention policies. Ensure you have backed up any data you want to keep. 2. **Remove TimescaleDB from shared\_preload\_libraries** Edit the PostgreSQL configuration file: ```bash sudo vi /var/lib/pgsql/18/data/postgresql.conf ``` Press `/` to search, type `shared_preload_libraries`, and press Enter. Press `i` to edit. Find the line with `shared_preload_libraries` and remove `timescaledb` from the list. For example, change: ```plaintext shared_preload_libraries = 'timescaledb' ``` to: ```plaintext shared_preload_libraries = '' ``` If there are other extensions in the list, keep them and only remove `timescaledb`. 3. **Restart PostgreSQL** ```bash sudo systemctl restart postgresql-18 ``` Replace `18` with your PostgreSQL version if different. 4. **Uninstall the TimescaleDB package** ```bash sudo dnf remove timescaledb-2-postgresql-18 ``` Replace `18` with your PostgreSQL version if different. 5. **Remove the TimescaleDB repository configuration** ```bash sudo rm /etc/yum.repos.d/timescale_timescaledb.repo ``` 6. **Remove dependencies** Remove unused dependencies installed with TimescaleDB: Warning This step may also remove PostgreSQL as a dependency. ```bash sudo dnf autoremove ``` * Rocky 1. **Drop the TimescaleDB extension from your databases** 1. Connect to each database where TimescaleDB is enabled and remove the extension: ```bash sudo -u postgres psql -d ``` 2. At the `psql` prompt: ```sql DROP EXTENSION timescaledb CASCADE; ``` Repeat this for all databases with TimescaleDB enabled. To exit psql, type `\q`. Warning Using `CASCADE` will drop all TimescaleDB-specific objects including hypertables, continuous aggregates, and retention policies. Ensure you have backed up any data you want to keep. 2. **Remove TimescaleDB from shared\_preload\_libraries** Edit the PostgreSQL configuration file: ```bash sudo vi /var/lib/pgsql/18/data/postgresql.conf ``` Press `/` to search, type `shared_preload_libraries`, and press Enter. Press `i` to edit. Find the line with `shared_preload_libraries` and remove `timescaledb` from the list. For example, change: ```plaintext shared_preload_libraries = 'timescaledb' ``` to: ```plaintext shared_preload_libraries = '' ``` If there are other extensions in the list, keep them and only remove `timescaledb`. Press `Esc`, then type `:wq` and press Enter to save. 3. **Restart PostgreSQL** ```bash sudo systemctl restart postgresql-18 ``` Replace `18` with your PostgreSQL version if different. 4. **Uninstall the TimescaleDB package** ```bash sudo dnf remove timescaledb-2-postgresql-18 ``` Replace `18` with your PostgreSQL version if different. 5. **Remove the TimescaleDB repository configuration** ```bash sudo rm /etc/yum.repos.d/timescale_timescaledb.repo ``` 6. **Remove dependencies** Remove unused dependencies installed with TimescaleDB: Warning This step may also remove PostgreSQL as a dependency. ```bash sudo dnf autoremove ``` * TimescaleDB-HA 1. **Drop the TimescaleDB extension from your databases** 1. Connect to each database where TimescaleDB is enabled and remove the extension: ```bash sudo -u postgres psql -d ``` 2. At the `psql` prompt: ```sql DROP EXTENSION timescaledb CASCADE; ``` Repeat this for all databases with TimescaleDB enabled. To exit psql, type `\q`. Warning Using `CASCADE` will drop all TimescaleDB-specific objects including hypertables, continuous aggregates, and retention policies. Ensure you have backed up any data you want to keep. 2. **Remove TimescaleDB from shared\_preload\_libraries** Edit the PostgreSQL configuration file: ```bash sudo vi /etc/postgresql/18/main/postgresql.conf ``` Find the line with `shared_preload_libraries` and remove `timescaledb` from the list. For example, change: ```plaintext shared_preload_libraries = 'timescaledb' ``` to: ```plaintext shared_preload_libraries = '' ``` If there are other extensions in the list, keep them and only remove `timescaledb`. 3. **Restart PostgreSQL** ```bash sudo systemctl restart postgresql ``` 4. **Uninstall the TimescaleDB package** ```bash sudo apt remove timescaledb-2-postgresql-18 ``` Replace `18` with your PostgreSQL version if different. 5. **Remove the TimescaleDB repository configuration** ```bash sudo rm /etc/apt/sources.list.d/timescaledb.list sudo rm /etc/apt/trusted.gpg.d/timescaledb.gpg sudo apt update ``` 6. **Remove dependencies** Remove unused dependencies installed with TimescaleDB: Warning This step may also remove PostgreSQL as a dependency. ```bash sudo apt autoremove ``` * TimescaleDB light 1. **Drop the TimescaleDB extension from your databases** 1. Connect to each database where TimescaleDB is enabled and remove the extension: ```bash sudo -u postgres psql -d ``` 2. At the `psql` prompt: ```sql DROP EXTENSION timescaledb CASCADE; ``` Repeat this for all databases with TimescaleDB enabled. To exit psql, type `\q`. Warning Using `CASCADE` will drop all TimescaleDB-specific objects including hypertables, continuous aggregates, and retention policies. Ensure you have backed up any data you want to keep. 2. **Remove TimescaleDB from shared\_preload\_libraries** Edit the PostgreSQL configuration file: ```bash sudo vi /etc/postgresql/18/main/postgresql.conf ``` Find the line with `shared_preload_libraries` and remove `timescaledb` from the list. For example, change: ```plaintext shared_preload_libraries = 'timescaledb' ``` to: ```plaintext shared_preload_libraries = '' ``` If there are other extensions in the list, keep them and only remove `timescaledb`. 3. **Restart PostgreSQL** ```bash sudo systemctl restart postgresql ``` 4. **Uninstall the TimescaleDB package** ```bash sudo apt remove timescaledb-2-postgresql-18 ``` Replace `18` with your PostgreSQL version if different. 5. **Remove the TimescaleDB repository configuration** ```bash sudo rm /etc/apt/sources.list.d/timescaledb.list sudo rm /etc/apt/trusted.gpg.d/timescaledb.gpg sudo apt update ``` 6. **Remove dependencies** Remove unused dependencies installed with TimescaleDB: Warning This step may also remove PostgreSQL as a dependency. ```bash sudo apt autoremove ``` * Debian 1. **Drop the TimescaleDB extension from your databases** 1. Connect to each database where TimescaleDB is enabled and remove the extension: ```bash sudo -u postgres psql -d ``` 2. At the `psql` prompt: ```sql DROP EXTENSION timescaledb CASCADE; ``` Repeat this for all databases with TimescaleDB enabled. To exit psql, type `\q`. Warning Using `CASCADE` will drop all TimescaleDB-specific objects including hypertables, continuous aggregates, and retention policies. Ensure you have backed up any data you want to keep. 2. **Remove TimescaleDB from shared\_preload\_libraries** Edit the PostgreSQL configuration file: ```bash sudo vi /var/lib/pgsql/18/data/postgresql.conf ``` Press `/` to search, type `shared_preload_libraries`, and press Enter. Press `i` to edit. Find the line with `shared_preload_libraries` and remove `timescaledb` from the list. For example, change: ```plaintext shared_preload_libraries = 'timescaledb' ``` to: ```plaintext shared_preload_libraries = '' ``` If there are other extensions in the list, keep them and only remove `timescaledb`. Press `Esc`, then type `:wq` and press Enter to save. 3. **Restart PostgreSQL** ```bash sudo systemctl restart postgresql-18 ``` Replace `18` with your PostgreSQL version if different. 4. **Uninstall the TimescaleDB package** ```bash sudo dnf remove timescaledb-2-postgresql-18 ``` Replace `18` with your PostgreSQL version if different. 5. **Remove the TimescaleDB repository configuration** ```bash sudo rm /etc/yum.repos.d/timescale_timescaledb.repo ``` 6. **Remove dependencies** Remove unused dependencies installed with TimescaleDB: Warning This step may also remove PostgreSQL as a dependency. ```bash sudo dnf autoremove ``` * Ubuntu 1. **Drop the TimescaleDB extension from your databases** 1. Connect to each database where TimescaleDB is enabled and remove the extension: ```bash sudo -u postgres psql -d ``` 2. At the `psql` prompt: ```sql DROP EXTENSION timescaledb CASCADE; ``` Repeat this for all databases with TimescaleDB enabled. To exit psql, type `\q`. Warning Using `CASCADE` will drop all TimescaleDB-specific objects including hypertables, continuous aggregates, and retention policies. Ensure you have backed up any data you want to keep. 2. **Remove TimescaleDB from shared\_preload\_libraries** Edit the PostgreSQL configuration file: ```bash sudo vi /var/lib/pgsql/18/data/postgresql.conf ``` Press `/` to search, type `shared_preload_libraries`, and press Enter. Press `i` to edit. Find the line with `shared_preload_libraries` and remove `timescaledb` from the list. For example, change: ```plaintext shared_preload_libraries = 'timescaledb' ``` to: ```plaintext shared_preload_libraries = '' ``` If there are other extensions in the list, keep them and only remove `timescaledb`. 3. **Restart PostgreSQL** ```bash sudo systemctl restart postgresql-18 ``` Replace `18` with your PostgreSQL version if different. 4. **Uninstall the TimescaleDB package** ```bash sudo dnf remove timescaledb-2-postgresql-18 ``` Replace `18` with your PostgreSQL version if different. 5. **Remove the TimescaleDB repository configuration** ```bash sudo rm /etc/yum.repos.d/timescale_timescaledb.repo ``` 6. **Remove dependencies** Remove unused dependencies installed with TimescaleDB: Warning This step may also remove PostgreSQL as a dependency. ```bash sudo dnf autoremove ``` * Red Hat 1. **Drop the TimescaleDB extension from your databases** 1. Connect to each database where TimescaleDB is enabled and remove the extension: ```bash sudo -u postgres psql -d ``` 2. At the `psql` prompt: ```sql DROP EXTENSION timescaledb CASCADE; ``` Repeat this for all databases with TimescaleDB enabled. To exit psql, type `\q`. Warning Using `CASCADE` will drop all TimescaleDB-specific objects including hypertables, continuous aggregates, and retention policies. Ensure you have backed up any data you want to keep. 2. **Remove TimescaleDB from shared\_preload\_libraries** Edit the PostgreSQL configuration file: ```bash sudo vi /var/lib/pgsql/18/data/postgresql.conf ``` Press `/` to search, type `shared_preload_libraries`, and press Enter. Press `i` to edit. Find the line with `shared_preload_libraries` and remove `timescaledb` from the list. For example, change: ```plaintext shared_preload_libraries = 'timescaledb' ``` to: ```plaintext shared_preload_libraries = '' ``` If there are other extensions in the list, keep them and only remove `timescaledb`. Press `Esc`, then type `:wq` and press Enter to save. 3. **Restart PostgreSQL** ```bash sudo systemctl restart postgresql-18 ``` Replace `18` with your PostgreSQL version if different. 4. **Uninstall the TimescaleDB package** ```bash sudo dnf remove timescaledb-2-postgresql-18 ``` Replace `18` with your PostgreSQL version if different. 5. **Remove the TimescaleDB repository configuration** ```bash sudo rm /etc/yum.repos.d/timescale_timescaledb.repo ``` 6. **Remove dependencies** Remove unused dependencies installed with TimescaleDB: Warning This step may also remove PostgreSQL as a dependency. ```bash sudo dnf autoremove ``` * Fedora Uninstall self-hosted TimescaleDB with Homebrew or MacPorts. * Homebrew 1. **Drop the TimescaleDB extension from your databases** At the `psql` prompt, remove the TimescaleDB extension: ```sql DROP EXTENSION timescaledb CASCADE; ``` Repeat this for all databases with TimescaleDB enabled. To exit psql, type `\q`. 2. **Remove TimescaleDB from shared\_preload\_libraries** Edit the PostgreSQL configuration file: ```bash sudo vi /opt/homebrew/var/postgresql@17/postgresql.conf ``` Find the line with `shared_preload_libraries` and remove `timescaledb` from the list. For example, change: ```plaintext shared_preload_libraries = 'timescaledb' ``` to: ```plaintext shared_preload_libraries = '' ``` If there are other extensions in the list, keep them and only remove `timescaledb`. 3. **Restart PostgreSQL** ```bash brew services restart postgresql@17 ``` 4. **Uninstall TimescaleDB** ```bash brew uninstall timescaledb-tools timescaledb ``` 5. **Remove all the dependencies and related files** ```bash brew untap timescale/tap ``` * MacPorts 1. **Drop the TimescaleDB extension from your databases** At the `psql` prompt, remove the TimescaleDB extension: ```sql DROP EXTENSION timescaledb CASCADE; ``` 2. **Remove TimescaleDB from shared\_preload\_libraries** At the command prompt, remove `timescaledb` from `shared_preload_libraries` in the `postgresql.conf` configuration file: ```bash nano /opt/homebrew/var/postgresql@14/postgresql.conf shared_preload_libraries = '' ``` 3. **Save the changes to the postgresql.conf file** 4. **Restart PostgreSQL** ```bash port reload postgresql ``` 5. **Uninstall TimescaleDB and the related dependencies** ```bash port uninstall timescaledb --follow-dependencies ``` * Rocky 1. **Drop the TimescaleDB extension from your databases** At the `psql` prompt, remove the TimescaleDB extension: ```sql DROP EXTENSION timescaledb CASCADE; ``` Repeat this for all databases with TimescaleDB enabled. To exit psql, type `\q`. 2. **Remove TimescaleDB from shared\_preload\_libraries** Edit the PostgreSQL configuration file: ```bash sudo vi /opt/homebrew/var/postgresql@17/postgresql.conf ``` Find the line with `shared_preload_libraries` and remove `timescaledb` from the list. For example, change: ```plaintext shared_preload_libraries = 'timescaledb' ``` to: ```plaintext shared_preload_libraries = '' ``` If there are other extensions in the list, keep them and only remove `timescaledb`. 3. **Restart PostgreSQL** ```bash brew services restart postgresql@17 ``` 4. **Uninstall TimescaleDB** ```bash brew uninstall timescaledb-tools timescaledb ``` 5. **Remove all the dependencies and related files** ```bash brew untap timescale/tap ``` * Homebrew 1. **Drop the TimescaleDB extension from your databases** At the `psql` prompt, remove the TimescaleDB extension: ```sql DROP EXTENSION timescaledb CASCADE; ``` 2. **Remove TimescaleDB from shared\_preload\_libraries** At the command prompt, remove `timescaledb` from `shared_preload_libraries` in the `postgresql.conf` configuration file: ```bash nano /opt/homebrew/var/postgresql@14/postgresql.conf shared_preload_libraries = '' ``` 3. **Save the changes to the postgresql.conf file** 4. **Restart PostgreSQL** ```bash port reload postgresql ``` 5. **Uninstall TimescaleDB and the related dependencies** ```bash port uninstall timescaledb --follow-dependencies ``` * MacPorts If you installed TimescaleDB on Windows using a package manager, you can uninstall it without removing PostgreSQL. 1. **Drop the TimescaleDB extension from your databases** 1. Open Command Prompt or PowerShell and connect to each database where TimescaleDB is enabled: ```bash psql -U postgres -d ``` 2. At the `psql` prompt, remove the extension: ```sql DROP EXTENSION timescaledb CASCADE; ``` Repeat this for all databases with TimescaleDB enabled. To exit psql, type `\q`. Warning Using `CASCADE` will drop all TimescaleDB-specific objects including hypertables, continuous aggregates, and retention policies. Ensure you have backed up any data you want to keep. 2. **Remove TimescaleDB from shared\_preload\_libraries** 1. Open your PostgreSQL configuration file. The default location is: ```plaintext C:\Program Files\PostgreSQL\\data\postgresql.conf ``` You may need to run the editor as Administrator. 2. Find the line with `shared_preload_libraries` and remove `timescaledb` from the list. For example, change: ```plaintext shared_preload_libraries = 'timescaledb' ``` to: ```plaintext shared_preload_libraries = '' ``` If there are other extensions in the list, keep them and only remove `timescaledb`. 3. Save the file. 3. **Restart PostgreSQL** ```powershell Restart-Service postgresql-x64-18 ``` Replace `18` with your PostgreSQL version if different. 4. **Remove TimescaleDB binaries** Manually remove the TimescaleDB files from your PostgreSQL installation directory. 1. Remove TimescaleDB library files. You may need to run PowerShell as Administrator: ```powershell Remove-Item "C:\Program Files\PostgreSQL\18\lib\timescaledb*.dll" ``` Replace `18` with your PostgreSQL version if different. 2. Remove TimescaleDB extension files: ```powershell Remove-Item -Recurse "C:\Program Files\PostgreSQL\18\share\extension\timescaledb*" ``` Replace `18` with your PostgreSQL version if different. Note If you encounter permission errors, ensure you're running PowerShell as Administrator. # Upgrade TimescaleDB > Upgrade your self-hosted TimescaleDB installation to a minor or major version in-place, or revert to the previous minor A major upgrade is when you update from TimescaleDB `X.` to `Y.`. A minor upgrade is when you update from TimescaleDB `.x`, to TimescaleDB `.y`. You upgrade your self-hosted TimescaleDB installation in-place. This section shows you how to: * Upgrade self-hosted TimescaleDB to a new [minor version](/docs/deploy/self-hosted/upgrades/minor-upgrade). * Upgrade self-hosted TimescaleDB to a new [major version](/docs/deploy/self-hosted/upgrades/major-upgrade). * Upgrade self-hosted TimescaleDB running in a [Docker container](/docs/deploy/self-hosted/upgrades/upgrade-docker) to a new minor version. * Upgrade [PostgreSQL](/docs/deploy/self-hosted/upgrades/upgrade-pg) to a new version. * Downgrade self-hosted TimescaleDB to the [previous minor version](/docs/deploy/self-hosted/upgrades/downgrade). # Downgrade to a previous version of TimescaleDB > Downgrade your self-hosted TimescaleDB installation to the previous minor version If you upgrade to a new TimescaleDB version and encounter problems, you can roll back to a previously installed version. This works in the same way as a minor upgrade. Downgrading is not supported for all versions. Generally, downgrades between patch versions and between consecutive minor versions are supported. For example, you can downgrade from TimescaleDB 2.5.2 to 2.5.1, or from 2.5.0 to 2.4.2. To check whether you can downgrade from a specific version, see the [release notes](https://github.com/timescale/timescaledb/releases). ## Plan your downgrade [Section titled “Plan your downgrade”](#plan-your-downgrade) You can downgrade your on-premise TimescaleDB installation in-place. This means that you do not need to dump and restore your data. However, it is still important that you plan for your downgrade ahead of time. Before you downgrade: * Read [the release notes](https://github.com/timescale/timescaledb/releases) for the TimescaleDB version you are downgrading to. * Check which PostgreSQL version you are currently running. You might need to [upgrade to the latest PostgreSQL version](/docs/deploy/self-hosted/upgrades/upgrade-pg) before you begin your TimescaleDB downgrade. * [Perform a backup](/docs/deploy/self-hosted/backup-and-restore) of your database. While TimescaleDB downgrades are performed in-place, downgrading is an intrusive operation. Always make sure you have a backup on hand, and that the backup is readable in the case of disaster. ## Downgrade TimescaleDB to a previous minor version [Section titled “Downgrade TimescaleDB to a previous minor version”](#downgrade-timescaledb-to-a-previous-minor-version) This downgrade uses the PostgreSQL `ALTER EXTENSION` function to downgrade to a previous version of the TimescaleDB extension. TimescaleDB supports having different extension versions on different databases within the same PostgreSQL instance. This allows you to upgrade and downgrade extensions independently on different databases. Run the `ALTER EXTENSION` function on each database to downgrade them individually. Warning The downgrade script is tested and supported for single-step downgrades. That is, downgrading from the current version, to the previous minor version. Downgrading might not work if you have made changes to your database between upgrading and downgrading. 1. **Set your connection string** This variable holds the connection information for the database to upgrade: ```bash export SOURCE="postgres://:@:/" ``` 2. **Connect to your database instance** ```shell psql -X -d $SOURCE ``` The `-X` flag prevents any `.psqlrc` commands from accidentally triggering the load of a previous TimescaleDB version on session startup. 3. **Downgrade the TimescaleDB extension** This must be the first command you execute in the current session: ```sql ALTER EXTENSION timescaledb UPDATE TO ''; ``` For example: ```sql ALTER EXTENSION timescaledb UPDATE TO '2.17.0'; ``` 4. **Check that you have downgraded to the correct version of TimescaleDB** ```sql \dx timescaledb; ``` PostgreSQL returns something like: ```shell Name | Version | Schema | Description -------------+---------+--------+--------------------------------------------------------------------------------------- timescaledb | 2.17.0 | public | Enables scalable inserts and complex queries for time-series data (Community Edition) ``` # Major TimescaleDB upgrades > Upgrade your self-hosted TimescaleDB installation to a new major version A major upgrade is when you update from TimescaleDB `X.` to `Y.`. A minor upgrade is when you update from TimescaleDB `.x`, to TimescaleDB `.y`. You can run different versions of TimescaleDB on different databases within the same PostgreSQL instance. This process uses the PostgreSQL `ALTER EXTENSION` function to upgrade TimescaleDB independently on different databases. When you perform a major upgrade, new policies are automatically configured based on your current configuration. In order to verify your policies post upgrade, in this upgrade process you export your policy settings before upgrading. This page shows you how to perform a major upgrade. For minor upgrades, see [Upgrade TimescaleDB to a minor version](/docs/deploy/self-hosted/upgrades/minor-upgrade). ## Prerequisites for this procedure To follow these steps, you'll need: * The PostgreSQL client tools installed on your migration machine. This includes `psql`, and `pg_dump`. * Familiarity with [the release notes](https://github.com/timescale/timescaledb/releases) for the version of TimescaleDB that you are upgrading to. * A [recent backup](/docs/deploy/self-hosted/backup-and-restore) of your database. While TimescaleDB upgrades are performed in-place, upgrading is an intrusive operation. Always make sure you have a backup on hand, and that the backup is readable in the case of disaster. ## Check the TimescaleDB and PostgreSQL versions [Section titled “Check the TimescaleDB and PostgreSQL versions”](#check-the-timescaledb-and-postgresql-versions) 1. **Set your connection string** This variable holds the connection information for the database to upgrade: ```bash export SOURCE="postgres://:@:/" ``` 2. **Retrieve the version of PostgreSQL that you are running** ```shell psql -X -d $SOURCE -c "SELECT version();" ``` PostgreSQL returns something like: ```shell ----------------------------------------------------------------------------------------------------------------------------------------- PostgreSQL 17.2 (Ubuntu 17.2-1.pgdg22.04+1) on aarch64-unknown-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, 64-bit (1 row) ``` 3. **Retrieve the version of TimescaleDB that you are running** ```sql psql -X -d $SOURCE -c "\dx timescaledb;" ``` PostgreSQL returns something like: ```shell Name | Version | Schema | Description -------------+---------+------------+--------------------------------------------------------------------- timescaledb | 2.17.2 | public | Enables scalable inserts and complex queries for time-series data (1 row) ``` ## Plan your upgrade path [Section titled “Plan your upgrade path”](#plan-your-upgrade-path) Best practice is to always use the latest version of TimescaleDB. Subscribe to our releases on GitHub or use Tiger Cloud and always get latest update without any hassle. Check the following support matrix against the versions of TimescaleDB and PostgreSQL that you are running currently and the versions you want to update to, then choose your upgrade path. For example, to upgrade from TimescaleDB 1.7 on PostgreSQL 12 to TimescaleDB 2.17.2 on PostgreSQL 15 you need to: 1. Upgrade TimescaleDB to 2.10 2. Upgrade PostgreSQL to 15 3. Upgrade TimescaleDB to 2.17.2. You may need to [upgrade to the latest PostgreSQL version](/docs/deploy/self-hosted/upgrades/upgrade-pg#upgrade-your-postgresql-instance) before you upgrade TimescaleDB. Warning TimescaleDB 2.28.x is the final minor version to support PostgreSQL 15. Support for PostgreSQL 15 is dropped in 2.29.0; patch releases in the 2.28.x cycle continue to support it. | TimescaleDB version | PostgreSQL 18 | PostgreSQL 17 | PostgreSQL 16 | PostgreSQL 15 | PostgreSQL 14 | PostgreSQL 13 | PostgreSQL 12 | PostgreSQL 11 | PostgreSQL 10 | | ------------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | | 2.28.x | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | 2.27.x | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | 2.26.x | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | 2.25.x | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | 2.24.x | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | 2.23.x | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | 2.22.x | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | 2.21.x | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | 2.20.x | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | 2.17 - 2.19 | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | | 2.16.x | ❌ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | | 2.13 - 2.15 | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | | 2.12.x | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | | 2.10.x | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | | 2.5 - 2.9 | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | | 2.4 | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | | 2.1 - 2.3 | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ❌ | | 2.0 | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | | 1.7 | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | We recommend not using TimescaleDB with PostgreSQL 17.1, 16.5, 15.9, 14.14, 13.17, 12.21.\ These minor versions [introduced a breaking binary interface change](https://www.postgresql.org/about/news/postgresql-172-166-1510-1415-1318-and-1222-released-2965/) that, once identified, was reverted in subsequent minor PostgreSQL versions 17.2, 16.6, 15.10, 14.15, 13.18, and 12.22. When you build from source, best practice is to build with PostgreSQL 17.2, 16.6, etc and higher. Users of [Tiger Cloud](https://console.cloud.tigerdata.com/) and platform packages for Linux, Windows, MacOS, Docker, and Kubernetes are unaffected. ## Check for failed retention policies [Section titled “Check for failed retention policies”](#check-for-failed-retention-policies) When you upgrade from TimescaleDB 1 to TimescaleDB 2, scripts automatically configure updated features to work as expected with the new version. However, not everything works in exactly the same way as previously. Before you begin this major upgrade, check the database log for errors related to failed retention policies that could have occurred in TimescaleDB 1. You can either remove the failing policies entirely, or update them to be compatible with your existing continuous aggregates. If incompatible retention policies are present when you perform the upgrade, the `ignore_invalidation_older_than` setting is automatically turned off, and a notice is shown. ## Export your policy settings [Section titled “Export your policy settings”](#export-your-policy-settings) 1. **Set your connection string** This variable holds the connection information for the database to upgrade: ```bash export SOURCE="postgres://:@:/" ``` 2. **Connect to your PostgreSQL deployment** ```bash psql -d $SOURCE ``` 3. **Save your policy statistics settings to a .csv file** ```sql COPY (SELECT * FROM timescaledb_information.policy_stats) TO policy_stats.csv csv header ``` 4. **Save your continuous aggregates settings to a .csv file** ```sql COPY (SELECT * FROM timescaledb_information.continuous_aggregate_stats) TO continuous_aggregate_stats.csv csv header ``` 5. **Save your drop chunk policies to a .csv file** ```sql COPY (SELECT * FROM timescaledb_information.drop_chunks_policies) TO drop_chunk_policies.csv csv header ``` 6. **Save your reorder policies to a .csv file** ```sql COPY (SELECT * FROM timescaledb_information.reorder_policies) TO reorder_policies.csv csv header ``` 7. **Exit your psql session** ```sql \q; ``` ## Implement your upgrade path [Section titled “Implement your upgrade path”](#implement-your-upgrade-path) 1. **Upgrade TimescaleDB** ```sql psql -X -d $SOURCE -c "ALTER EXTENSION timescaledb UPDATE TO '';" ``` 2. **If your migration path dictates it, upgrade PostgreSQL** Follow the procedure in [Upgrade PostgreSQL](/docs/deploy/self-hosted/upgrades/upgrade-pg). The version of TimescaleDB installed in your PostgreSQL deployment must be the same before and after the PostgreSQL upgrade. 3. **If your migration path dictates it, upgrade TimescaleDB again** ```sql psql -X -d $SOURCE -c "ALTER EXTENSION timescaledb UPDATE TO '';" ``` 4. **Check that you have upgraded to the correct version of TimescaleDB** ```sql psql -X -d $SOURCE -c "\dx timescaledb;" ``` PostgreSQL returns something like: ```shell Name | Version | Schema | Description -------------+---------+--------+--------------------------------------------------------------------------------------- timescaledb | 2.17.2 | public | Enables scalable inserts and complex queries for time-series data (Community Edition) ``` Note To upgrade TimescaleDB in a Docker container, see the [Docker container upgrades](/docs/deploy/self-hosted/upgrades/upgrade-docker) section. ## Verify the updated policy settings and jobs [Section titled “Verify the updated policy settings and jobs”](#verify-the-updated-policy-settings-and-jobs) 1. **Verify the continuous aggregate policy jobs** ```sql SELECT * FROM timescaledb_information.jobs WHERE application_name LIKE 'Refresh Continuous%'; ``` PostgreSQL returns something like: ```shell -[ RECORD 1 ]-----+-------------------------------------------------- job_id | 1001 application_name | Refresh Continuous Aggregate Policy [1001] schedule_interval | 01:00:00 max_runtime | 00:00:00 max_retries | -1 retry_period | 01:00:00 proc_schema | _timescaledb_internal proc_name | policy_refresh_continuous_aggregate owner | postgres scheduled | t config | {"start_offset": "20 days", "end_offset": "10 days", "mat_hypertable_id": 2} next_start | 2020-10-02 12:38:07.014042-04 hypertable_schema | _timescaledb_internal hypertable_name | _materialized_hypertable_2 ``` 2. **Verify the information for each policy type that you exported before you upgraded** For continuous aggregates, take note of the `config` information to verify that all settings were converted correctly. 3. **Verify that all jobs are scheduled and running as expected** ```sql SELECT * FROM timescaledb_information.job_stats WHERE job_id = 1001; ``` PostgreSQL returns something like: ```sql -[ RECORD 1 ]----------+------------------------------ hypertable_schema | _timescaledb_internal hypertable_name | _materialized_hypertable_2 job_id | 1001 last_run_started_at | 2020-10-02 09:38:06.871953-04 last_successful_finish | 2020-10-02 09:38:06.932675-04 last_run_status | Success job_status | Scheduled last_run_duration | 00:00:00.060722 next_scheduled_run | 2020-10-02 10:38:06.932675-04 total_runs | 1 total_successes | 1 total_failures | 0 ``` You are running a shiny new version of TimescaleDB. # Minor TimescaleDB upgrades > Upgrade your self-hosted TimescaleDB to a new minor version A minor upgrade is when you update from TimescaleDB `.x` to TimescaleDB `.y`. A major upgrade is when you update from TimescaleDB `X.` to `Y.`. You can run different versions of TimescaleDB on different databases within the same PostgreSQL instance. This process uses the PostgreSQL `ALTER EXTENSION` function to upgrade TimescaleDB independently on different databases. This page shows you how to perform a minor upgrade, for major upgrades, see [Upgrade TimescaleDB to a major version](/docs/deploy/self-hosted/upgrades/major-upgrade). ## Prerequisites for this procedure To follow these steps, you'll need: * The PostgreSQL client tools installed on your migration machine. This includes `psql`, and `pg_dump`. * Familiarity with [the release notes](https://github.com/timescale/timescaledb/releases) for the version of TimescaleDB that you are upgrading to. * A [recent backup](/docs/deploy/self-hosted/backup-and-restore) of your database. While TimescaleDB upgrades are performed in-place, upgrading is an intrusive operation. Always make sure you have a backup on hand, and that the backup is readable in the case of disaster. ## Check the TimescaleDB and PostgreSQL versions [Section titled “Check the TimescaleDB and PostgreSQL versions”](#check-the-timescaledb-and-postgresql-versions) 1. **Set your connection string** This variable holds the connection information for the database to upgrade: ```bash export SOURCE="postgres://:@:/" ``` 2. **Retrieve the version of PostgreSQL that you are running** ```shell psql -X -d $SOURCE -c "SELECT version();" ``` PostgreSQL returns something like: ```shell ----------------------------------------------------------------------------------------------------------------------------------------- PostgreSQL 17.2 (Ubuntu 17.2-1.pgdg22.04+1) on aarch64-unknown-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, 64-bit (1 row) ``` 3. **Retrieve the version of TimescaleDB that you are running** ```sql psql -X -d $SOURCE -c "\dx timescaledb;" ``` PostgreSQL returns something like: ```shell Name | Version | Schema | Description -------------+---------+------------+--------------------------------------------------------------------- timescaledb | 2.17.2 | public | Enables scalable inserts and complex queries for time-series data (1 row) ``` ## Plan your upgrade path [Section titled “Plan your upgrade path”](#plan-your-upgrade-path) Best practice is to always use the latest version of TimescaleDB. Subscribe to our releases on GitHub or use Tiger Cloud and always run the latest update without any hassle. Check the following support matrix against the versions of TimescaleDB and PostgreSQL that you are running currently and the versions you want to update to, then choose your upgrade path. For example, to upgrade from TimescaleDB 2.13.x on PostgreSQL 13 to TimescaleDB 2.25.x: 1. Upgrade TimescaleDB to 2.15.x (last version fully supporting PostgreSQL 13) 2. Upgrade PostgreSQL to 15, 16, 17, or 18 3. Upgrade TimescaleDB to 2.25.x Older TimescaleDB versions may not be available in package repositories. If you need a specific intermediate version that is no longer packaged, you can build from source or upgrade directly to the latest version that supports your current PostgreSQL version before upgrading PostgreSQL. You may need to upgrade to the latest PostgreSQL version before you upgrade TimescaleDB. Also, if you use [TimescaleDB Toolkit](/docs/deploy/self-hosted/tooling/install-toolkit), ensure the `timescaledb_toolkit` extension is >= v1.6.0 before you upgrade TimescaleDB extension. Warning TimescaleDB 2.28.x is the final minor version to support PostgreSQL 15. Support for PostgreSQL 15 is dropped in 2.29.0; patch releases in the 2.28.x cycle continue to support it. | TimescaleDB version | PostgreSQL 18 | PostgreSQL 17 | PostgreSQL 16 | PostgreSQL 15 | PostgreSQL 14 | PostgreSQL 13 | PostgreSQL 12 | PostgreSQL 11 | PostgreSQL 10 | | ------------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | | 2.28.x | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | 2.27.x | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | 2.26.x | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | 2.25.x | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | 2.24.x | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | 2.23.x | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | 2.22.x | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | 2.21.x | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | 2.20.x | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | 2.17 - 2.19 | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | | 2.16.x | ❌ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | | 2.13 - 2.15 | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | | 2.12.x | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | | 2.10.x | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | | 2.5 - 2.9 | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | | 2.4 | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | | 2.1 - 2.3 | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ❌ | | 2.0 | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | | 1.7 | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | We recommend not using TimescaleDB with PostgreSQL 17.1, 16.5, 15.9, 14.14, 13.17, 12.21.\ These minor versions [introduced a breaking binary interface change](https://www.postgresql.org/about/news/postgresql-172-166-1510-1415-1318-and-1222-released-2965/) that, once identified, was reverted in subsequent minor PostgreSQL versions 17.2, 16.6, 15.10, 14.15, 13.18, and 12.22. When you build from source, best practice is to build with PostgreSQL 17.2, 16.6, etc and higher. Users of [Tiger Cloud](https://console.cloud.tigerdata.com/) and platform packages for Linux, Windows, MacOS, Docker, and Kubernetes are unaffected. ## Implement your upgrade path [Section titled “Implement your upgrade path”](#implement-your-upgrade-path) 1. **Upgrade TimescaleDB** ```sql psql -X -d $SOURCE -c "ALTER EXTENSION timescaledb UPDATE TO '';" ``` 2. **If your migration path dictates it, upgrade PostgreSQL** Follow the procedure in [Upgrade PostgreSQL](/docs/deploy/self-hosted/upgrades/upgrade-pg). The version of TimescaleDB installed in your PostgreSQL deployment must be the same before and after the PostgreSQL upgrade. 3. **If your migration path dictates it, upgrade TimescaleDB again** ```sql psql -X -d $SOURCE -c "ALTER EXTENSION timescaledb UPDATE TO '';" ``` 4. **Check that you have upgraded to the correct version of TimescaleDB** ```sql psql -X -d $SOURCE -c "\dx timescaledb;" ``` PostgreSQL returns something like: ```shell Name | Version | Schema | Description -------------+---------+--------+--------------------------------------------------------------------------------------- timescaledb | 2.17.2 | public | Enables scalable inserts and complex queries for time-series data (Community Edition) ``` You are running a shiny new version of TimescaleDB. # Upgrade TimescaleDB running in Docker > Upgrade self-hosted TimescaleDB running in a Docker container to a new minor version If you originally installed TimescaleDB using Docker, you can upgrade from within the Docker container. This allows you to upgrade to the latest TimescaleDB version while retaining your data. The `timescale/timescaledb-ha*` images have the files necessary to run previous versions. Patch releases only contain bugfixes so should always be safe. Non-patch releases may rarely require some extra steps. These steps are mentioned in the [release notes](https://github.com/timescale/timescaledb/releases) for the version of TimescaleDB that you are upgrading to. After you upgrade the docker image, you run `ALTER EXTENSION` for all databases using TimescaleDB. The examples in this page use a Docker instance called `timescaledb`. If you have given your Docker instance a different name, replace it when you issue the commands. ## Determine the mount point type [Section titled “Determine the mount point type”](#determine-the-mount-point-type) When you start your upgraded Docker container, you need to be able to point the new Docker image to the location that contains the data from your previous version. To do this, you need to work out where the current mount point is. The current mount point varies depending on whether your container is using volume mounts, or bind mounts. 1. **Find the mount type used by your Docker container** ```bash docker inspect timescaledb --format='{{range .Mounts }}{{.Type}}{{end}}' ``` This returns either `volume` or `bind`. 2. **Note the volume or bind used by your container** * Volume ```bash docker inspect timescaledb --format='{{range .Mounts }}{{.Name}}{{end}}' ``` Docker returns the ``. You see something like this: ```plaintext 069ba64815f0c26783b81a5f0ca813227fde8491f429cf77ed9a5ae3536c0b2c ``` * Bind ```bash docker inspect timescaledb --format='{{range .Mounts }}{{.Source}}{{end}}' ``` Docker returns the ``. You see something like this: ```plaintext /path/to/data ``` You use this value when you perform the upgrade. ## Upgrade TimescaleDB within Docker [Section titled “Upgrade TimescaleDB within Docker”](#upgrade-timescaledb-within-docker) To upgrade TimescaleDB within Docker, you need to download the upgraded image, stop the old container, and launch the new container pointing to your existing data. * TimescaleDB-HA 1. **Pull the latest TimescaleDB image** This command pulls the latest version of TimescaleDB running on PostgreSQL 18: ```plaintext docker pull timescale/timescaledb-ha:pg18 ``` If you're using another version of PostgreSQL, look for the relevant tag in the [TimescaleDB HA](https://hub.docker.com/r/timescale/timescaledb-ha/tags) repository on Docker Hub. 2. **Stop the old container, and remove it** ```bash docker stop timescaledb docker rm timescaledb ``` 3. **Launch a new container with the upgraded Docker image** Launch based on your mount point type: * Volume mount ```bash docker run -v :/pgdata -e PGDATA=/pgdata -d --name timescaledb -p 5432:5432 timescale/timescaledb-ha:pg18 ``` * Bind mount ```bash docker run -v :/pgdata -e PGDATA=/pgdata -d --name timescaledb \ -p 5432:5432 timescale/timescaledb-ha:pg18 ``` 4. **Connect to the upgraded instance using psql with the -X flag** ```bash docker exec -it timescaledb psql -U postgres -X ``` 5. **At the `psql` prompt, use the `ALTER` command to upgrade the extension** ```plaintext ALTER EXTENSION timescaledb UPDATE; CREATE EXTENSION IF NOT EXISTS timescaledb_toolkit; ALTER EXTENSION timescaledb_toolkit UPDATE; ``` The [TimescaleDB Toolkit](/docs/deploy/self-hosted/tooling/install-toolkit) extension is packaged with TimescaleDB HA, it includes additional hyperfunctions to help you with queries and data analysis. Note If you have multiple databases, update each database separately. * TimescaleDB light ```bash docker run -v :/pgdata -e PGDATA=/pgdata -d --name timescaledb -p 5432:5432 timescale/timescaledb-ha:pg18 ``` * Volume mount ```bash docker run -v :/pgdata -e PGDATA=/pgdata -d --name timescaledb \ -p 5432:5432 timescale/timescaledb-ha:pg18 ``` * Bind mount 1. **Pull the latest TimescaleDB image** This command pulls the latest version of TimescaleDB running on PostgreSQL 18. ```plaintext docker pull timescale/timescaledb:latest-pg18 ``` If you're using another version of PostgreSQL, look for the relevant tag in the [TimescaleDB light](https://hub.docker.com/r/timescale/timescaledb) repository on Docker Hub. 2. **Stop the old container, and remove it** ```bash docker stop timescaledb docker rm timescaledb ``` 3. **Launch a new container with the upgraded Docker image** Launch based on your mount point type: * Volume mount ```bash docker run -v :/pgdata -e PGDATA=/pgdata \ -d --name timescaledb -p 5432:5432 timescale/timescaledb:latest-pg18 ``` * Bind mount ```bash docker run -v :/pgdata -e PGDATA=/pgdata -d --name timescaledb \ -p 5432:5432 timescale/timescaledb:latest-pg18 ``` 4. **Connect to the upgraded instance using psql with the -X flag** ```bash docker exec -it timescaledb psql -U postgres -X ``` 5. **At the `psql` prompt, use the `ALTER` command to upgrade the extension** ```sql ALTER EXTENSION timescaledb UPDATE; ``` Note If you have multiple databases, you need to update each database separately. * Volume mount ```bash docker run -v :/pgdata -e PGDATA=/pgdata \ -d --name timescaledb -p 5432:5432 timescale/timescaledb:latest-pg18 ``` * Bind mount ```bash docker run -v :/pgdata -e PGDATA=/pgdata -d --name timescaledb \ -p 5432:5432 timescale/timescaledb:latest-pg18 ``` # Upgrade PostgreSQL > Upgrade PostgreSQL to a new version compatible with your self-hosted TimescaleDB installation TimescaleDB is a PostgreSQL extension. Ensure that you upgrade to compatible versions of TimescaleDB and PostgreSQL. ## Prerequisites for this procedure To follow these steps, you'll need: * The PostgreSQL client tools installed on your migration machine. This includes `psql`, and `pg_dump`. * Familiarity with [the release notes](https://github.com/timescale/timescaledb/releases) for the version of TimescaleDB that you are upgrading to. * A [recent backup](/docs/deploy/self-hosted/backup-and-restore) of your database. While TimescaleDB upgrades are performed in-place, upgrading is an intrusive operation. Always make sure you have a backup on hand, and that the backup is readable in the case of disaster. ## Plan your upgrade path [Section titled “Plan your upgrade path”](#plan-your-upgrade-path) Best practice is to always use the latest version of TimescaleDB. Subscribe to our releases on GitHub or use Tiger Cloud and always run the latest update without any hassle. Check the following support matrix against the versions of TimescaleDB and PostgreSQL that you are running currently and the versions you want to update to, then choose your upgrade path. For example, to upgrade from TimescaleDB 2.13.x on PostgreSQL 13 to TimescaleDB 2.25.x: 1. Upgrade TimescaleDB to 2.15.x (last version fully supporting PostgreSQL 13) 2. Upgrade PostgreSQL to 15, 16, 17, or 18 3. Upgrade TimescaleDB to 2.25.x Older TimescaleDB versions may not be available in package repositories. If you need a specific intermediate version that is no longer packaged, you can build from source or upgrade directly to the latest version that supports your current PostgreSQL version before upgrading PostgreSQL. You may need to upgrade to the latest PostgreSQL version before you upgrade TimescaleDB. Also, if you use [TimescaleDB Toolkit](/docs/deploy/self-hosted/tooling/install-toolkit), ensure the `timescaledb_toolkit` extension is >= v1.6.0 before you upgrade TimescaleDB extension. Warning TimescaleDB 2.28.x is the final minor version to support PostgreSQL 15. Support for PostgreSQL 15 is dropped in 2.29.0; patch releases in the 2.28.x cycle continue to support it. | TimescaleDB version | PostgreSQL 18 | PostgreSQL 17 | PostgreSQL 16 | PostgreSQL 15 | PostgreSQL 14 | PostgreSQL 13 | PostgreSQL 12 | PostgreSQL 11 | PostgreSQL 10 | | ------------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | | 2.28.x | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | 2.27.x | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | 2.26.x | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | 2.25.x | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | 2.24.x | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | 2.23.x | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | 2.22.x | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | 2.21.x | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | 2.20.x | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | 2.17 - 2.19 | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | | 2.16.x | ❌ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | | 2.13 - 2.15 | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | | 2.12.x | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | | 2.10.x | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | | 2.5 - 2.9 | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | | 2.4 | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | | 2.1 - 2.3 | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ❌ | | 2.0 | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | | 1.7 | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | We recommend not using TimescaleDB with PostgreSQL 17.1, 16.5, 15.9, 14.14, 13.17, 12.21.\ These minor versions [introduced a breaking binary interface change](https://www.postgresql.org/about/news/postgresql-172-166-1510-1415-1318-and-1222-released-2965/) that, once identified, was reverted in subsequent minor PostgreSQL versions 17.2, 16.6, 15.10, 14.15, 13.18, and 12.22. When you build from source, best practice is to build with PostgreSQL 17.2, 16.6, etc and higher. Users of [Tiger Cloud](https://console.cloud.tigerdata.com/) and platform packages for Linux, Windows, MacOS, Docker, and Kubernetes are unaffected. ## Upgrade your PostgreSQL instance [Section titled “Upgrade your PostgreSQL instance”](#upgrade-your-postgresql-instance) You use [`pg_upgrade`](https://www.postgresql.org/docs/current/pgupgrade.html) to upgrade PostgreSQL in-place. `pg_upgrade` allows you to retain the data files of your current PostgreSQL installation while binding the new PostgreSQL binary runtime to them. 1. **Find the location of the PostgreSQL binary** Set the `OLD_BIN_DIR` environment variable to the folder holding the `postgres` binary. For example, `which postgres` returns something like `/usr/lib/postgresql/16/bin/postgres`. ```bash export OLD_BIN_DIR=/usr/lib/postgresql/16/bin ``` 2. **Set your connection string** This variable holds the connection information for the database to upgrade: ```bash export SOURCE="postgres://:@:/" ``` 3. **Retrieve the location of the PostgreSQL data folder** Set the `OLD_DATA_DIR` environment variable to the value returned by the following: ```shell psql -d "$SOURCE" -c "SHOW data_directory ;" ``` PostgreSQL returns something like: ```shell ---------------------------- /home/postgres/pgdata/data (1 row) ``` 4. **Choose the new locations for the PostgreSQL binary and data folders** For example: ```shell export NEW_BIN_DIR=/usr/lib/postgresql/17/bin export NEW_DATA_DIR=/home/postgres/pgdata/data-17 ``` 5. **Perform the upgrade using `pg_upgrade`** ```sql pg_upgrade -b $OLD_BIN_DIR -B $NEW_BIN_DIR -d $OLD_DATA_DIR -D $NEW_DATA_DIR ``` If you are moving data to a new physical instance of PostgreSQL, you can use `pg_dump` and `pg_restore` to dump your data from the old database, and then restore it into the new, upgraded, database. For more information, see the [backup and restore section](/docs/deploy/self-hosted/backup-and-restore). # About configuration in Tiger Cloud > Have your service auto-tuned or configure manually By default, Tiger Cloud uses the default PostgreSQL server configuration settings. Most configuration values for a Tiger Cloud service are initially set in accordance with best practices given the compute and storage settings of the service. Any time you increase or decrease the compute for a service, the most essential values are set to reflect the size of the new service. There are times, however, when your specific workload could require tuning some of the many available Tiger Cloud-specific and PostgreSQL parameters. By providing the ability to tune various runtime settings, Tiger Cloud provides the balance and flexibility you need when running your workloads in a hosted environment. You can use [service settings](../configuration/advanced-parameters) and [service operations](../configuration/customize-configuration) to customize Tiger Cloud configurations. # Advanced parameters > Tune advanced PostgreSQL parameters for your Tiger Cloud service in Tiger Console It is possible to configure a wide variety of Tiger Cloud service database parameters by navigating to the `Advanced parameters` tab under the `Database configuration` heading. The advanced parameters are displayed in a scrollable and searchable list. ![Database configuration advanced parameters](/docs/_astro/database-configuration-advanced-parameters.bUCg8l3U_Z1157EE.webp) As with the basic database configuration parameters, any changes are highlighted and the `Apply changes`, or `Apply changes and restart`, button is available, prompting you to confirm changes before the service is modified. ## Commonly tuned parameters These are some of the most frequently adjusted parameters available in the Advanced parameters tab: * **`max_locks_per_transaction`**: maximum number of object locks per transaction. Increase this if you see `out of shared memory` errors, especially with hypertables that have many chunks. See [Transaction lock settings](/docs/reference/timescaledb/configuration/tiger-postgres#transaction-lock-settings) for sizing guidance. * **`shared_buffers`**: amount of memory used for shared memory buffers. A good starting point is 25% of total system memory. * **`work_mem`**: amount of memory used by internal sort operations and hash tables before writing to temporary disk files. * **`maintenance_work_mem`**: maximum memory for maintenance operations such as `VACUUM` and `CREATE INDEX`. * **`max_connections`**: maximum number of concurrent connections to the database. * **`timescaledb.max_background_workers`**: maximum background workers allocated to TimescaleDB. Set to at least 1 + the number of databases in the instance. For a complete list of available parameters, see the [Grand Unified Configuration (GUC) parameters](/docs/reference/timescaledb/configuration/gucs) reference. For detailed tuning guidance, see [Configuration parameters](/docs/reference/timescaledb/configuration/tiger-postgres). ## Multiple databases To create more than one database, you need to create a new service for each database. Tiger Cloud does not support multiple databases within the same service. Having a separate service for each database affords each database its own isolated resources. You can also use [schemas](https://www.postgresql.org/docs/current/ddl-schemas.html) to organize tables into logical groups. A single database can contain multiple schemas, which in turn contain tables. The main difference between isolating with databases versus schemas is that a user can access objects in any of the schemas in the database they are connected to, so long as they have the corresponding privileges. Schemas can help isolate smaller use cases that do not warrant their own service. Please refer to the [Grand Unified Configuration (GUC) parameters](/docs/reference/timescaledb/configuration/gucs) for a complete list. ## Policies ### `timescaledb.max_background_workers (int)` Max background worker processes allocated to TimescaleDB. Set to at least 1 + the number of databases loaded with the TimescaleDB extension in a PostgreSQL instance. Default value is 16. ## Tiger Cloud service tuning ### `timescaledb.disable_load (bool)` Disable the loading of the actual extension # Configure database parameters > Customize PostgreSQL and Tiger Cloud database parameters for individual services Tiger Cloud allows you to customize many Tiger Cloud-specific and PostgreSQL configuration options for each service individually. Most configuration values for a service are initially set in accordance with best practices given the compute and storage settings of the service. Any time you increase or decrease the compute for a service, the most essential values are set to reflect the size of the new service. Warning You can modify most parameters without restarting the service. However, some changes do require a restart, resulting in some brief downtime that is usually about 30 seconds. An example of a change that needs a restart is modifying the compute resources of a running service. ## View service operation details To modify configuration parameters, first select the service that you want to modify. This displays the service details, with these tabs across the top: `Overview`, `Actions`, `Explorer`, `Monitoring`, `Connections`, `Operations`, and `AI`. Select `Operations`, then `Database parameters`. ![Database configuration parameters](/docs/_astro/tiger-console-service-configuration-parameters.B_9CNzSq_1YC4HB.webp) ### Modify basic parameters Under the `Common parameters` tab, you can modify a limited set of the parameters that are most often modified in a Tiger Cloud or PostgreSQL instance. To modify a configured value, hover over the value and click the revealed pencil icon. This reveals an editable field to apply your change. Clicking anywhere outside of that field saves the value to be applied. ![Change Tiger Cloud configuration parameters](/docs/_astro/tsc-settings-change.MDtFEN3R_1pf8ld.webp) ### Apply configuration changes When you have modified the configuration parameters that you would like to change, click `Apply changes`. For some changes, such as `timescaledb.max_background_workers`, the service needs to be restarted. In this case, the button reads `Apply changes and restart`. A confirmation dialog is displayed which indicates whether a restart is required. Click `Confirm` to apply the changes, and restart if necessary. # Back up and recover services > Protect your data with automatic backups and cross-region protection Tiger Cloud provides comprehensive backup and recovery solutions to protect your data, including automatic daily backups, cross-region protection, and point-in-time recovery. ## Automatic backups [Section titled “Automatic backups”](#automatic-backups) Tiger Cloud automatically handles backup for your Tiger Cloud services using the `pgBackRest` tool. You don't need to perform backups manually. What's more, with [cross-region backup](#enable-cross-region-backup), you are protected when an entire AWS region goes down. Tiger Cloud automatically creates one full backup every week, and incremental backups every day in the same region as your service. Additionally, all [Write-Ahead Log (WAL)](https://www.postgresql.org/docs/current/wal-intro.html) files are retained back to the oldest full backup. This means that you always have a full backup available for the current and previous week: ![Database backup and recovery architecture in Tiger Cloud](/docs/_astro/database-backup-recovery.DHoYmo-Y_Z1NJvh0.webp) On [Scale and Enterprise](/docs/deploy/tiger-cloud/tiger-cloud-aws/pricing-and-account-management) pricing plans, you can check the list of backups for the previous 14 days in Tiger Console. To do so, select your service, then click `Operations` > `Backup and restore` > `Backup history`. In the event of a storage failure, a service automatically recovers from a backup to the point of failure. If the whole availability zone goes down, your Tiger Cloud services are recovered in a different zone. In the event of a user error, you can [create a point-in-time recovery fork](../service-management/fork-services). ## Enable cross-region backup [Section titled “Enable cross-region backup”](#enable-cross-region-backup) Note Cross-region backup is available on [Enterprise](/docs/deploy/tiger-cloud/tiger-cloud-aws/pricing-and-account-management) pricing plan. For added reliability, you can enable cross-region backup. This protects your data when an entire AWS region goes down. In this case, you have two identical backups of your service at any time, but one of them is in a different AWS region. Cross-region backups are updated daily and weekly in the same way as a regular backup. You can have one cross-region backup for a service. You enable cross-region backup when you create a service, or configure it for an existing service in Tiger Console: 1. **Select your service and open backup settings** In [Console](https://console.cloud.tigerdata.com/dashboard/services), select your service and click `Operations` > `Backup & restore`. 2. **Enable cross-region backup** In `Cross-region backup`, select the region in the dropdown and click `Enable backup`. ![Creating a cross-region backup in Tiger Console](/docs/_astro/create-cross-region-backup-in-tiger-console.CGDp6LDi_2tHbvp.webp) You can now see the backup, its region, and creation date in a list. You can have one cross-region backup per service. To change the region of your backup: 1. **Open backup settings for your service** In [Console](https://console.cloud.tigerdata.com/dashboard/services), select your service and click `Operations` > `Backup & restore`. 2. **Disable the existing backup** Click the trash icon next to the existing backup to disable it. ![Cross-region backup list in Tiger Console](/docs/_astro/cross-region-backup-list-in-tiger-console.DtjFvbaZ_16nhNb.webp) 3. **Create a new backup in a different region** # Manage high availability > Configure HA replicas across availability zones to minimize downtime For Tiger Cloud services where every second of uptime matters, Tiger Cloud delivers High Availability (HA) replicas. These replicas safeguard your data and keep your service running smoothly, even in the face of unexpected failures. By minimizing downtime and protecting against data loss, HA replicas ensure business continuity and give you the confidence to operate without interruption, including during routine maintenance. ![High availability replicas in Tiger Cloud across multiple AWS availability zones](/docs/_astro/tiger-cloud-ha-architecture-diagram.eNwtvpxM_x1xwG.webp) ![High availability replicas in Tiger Cloud across multiple AWS availability zones](/docs/_astro/tiger-cloud-ha-architecture-diagram-dark.DnWxlufO_18jKoM.webp) This page shows you how to choose the best high availability option for your service. ## What is HA replication? HA replicas are exact, up-to-date copies of your database hosted in multiple AWS availability zones (AZ) within the same region as your primary node. They automatically take over operations if the original primary data node becomes unavailable. The primary node streams its write-ahead log (WAL) to the replicas to minimize the chances of data loss during failover. HA replicas can be synchronous and asynchronous. * Synchronous: the primary commits its next write once the replica confirms that the previous write is complete. There is no lag between the primary and the replica. They are in the same state at all times. This is preferable if you need the highest level of data integrity. However, this affects the primary ingestion time. * Asynchronous: the primary commits its next write without the confirmation of the previous write completion. The asynchronous HA replicas often have a lag, in both time and data, compared to the primary. This is preferable if you need the shortest primary ingest time. ![Sync and async replication](/docs/_astro/sync_async_replication_draft.owC364ei_Z2rKBfJ.webp) ![Sync and async replication](/docs/_astro/sync_async_replication_draft-dark.DgNPMrCW_Z1aUROl.webp) HA replicas have separate unique addresses that you can use to serve read-only requests in parallel to your primary data node. When your primary data node fails, Tiger Cloud automatically fails over to an HA replica within 30 seconds. During failover, the read-only address is unavailable while Tiger Cloud automatically creates a new HA replica. The time to make this replica depends on several factors, including the size of your data. Operations such as upgrading your service to a new major or minor version may necessitate a service restart. Restarts are run during the [maintenance window](../upgrades). To avoid any downtime, each data node is updated in turn. That is, while the primary data node is updated, a replica is promoted to primary. After the primary is updated and online, the same maintenance is performed on the HA replicas. To ensure that all services have minimum downtime and data loss in the most common failure scenarios and during maintenance, [rapid recovery](../high-availability/overview#rapid-recovery) is enabled by default for all services. ## Choose an HA strategy The following HA configurations are available in Tiger Cloud: * **Non-production**: no replica, best for developer environments. * **High availability**: a single async replica in a different AWS availability zone from your primary. Provides high availability with cost efficiency. Best for production apps. * **Highest availability**: two replicas in different AWS availability zones from your primary. Available replication modes are: * **High performance** - two async replicas. Provides the highest level of availability with two AZs and the ability to query the HA system. Best for apps where service availability is most critical. * **High data integrity** - one sync replica and one async replica. The sync replica is identical to the primary at all times. Best for apps that can tolerate no data loss. The following table summarizes the differences between these HA configurations: | | High availability (1 async) | High performance (2 async) | High data integrity (1 sync + 1 async) | | ------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Write flow | The primary streams its WAL to the async replica, which may have a slight lag compared to the primary, providing 99.9% uptime SLA. | The primary streams its writes to both async replicas, providing 99.9+% uptime SLA. | The primary streams its writes to the sync and async replicas. The async replica is never ahead of the sync one. | | Additional read replica | Recommended. Reads from the HA replica may cause availability and lag issues. | Not needed. You can still read from the HA replica even if one of them is down. Configure an additional read replica only if your read use case is significantly different from your write use case. | Highly recommended. If you run heavy queries on a sync replica, it may fall behind the primary. Specifically, if it takes too long for the replica to confirm a transaction, the next transaction is canceled. | | Choosing the replica to read from manually | Not applicable. | Not available. Queries are load-balanced against all available HA replicas. | Not available. Queries are load-balanced against all available HA replicas. | | Sync replication | Only async replicas are supported in this configuration. | Only async replicas are supported in this configuration. | Supported. | | Failover flow | * If the primary fails, the replica becomes the primary while a new node is created, with only seconds of downtime. * If the replica fails, a new async replica is created without impacting the primary. If you read from the async HA replica, those reads fail until the new replica is available. | - If the primary fails, one of the replicas becomes the primary while a new node is created, with the other one still available for reads. - If the replica fails, a new async replica is created in another AZ, without impacting the primary. The newly created replica is behind the primary and the original replica while it catches up. | * If the primary fails, the sync replica becomes the primary while a new node is created, with the async one still available for reads. * If the async replica fails, a new async replica is created. Heavy reads on the sync replica may delay the ingest time of the primary while a new async replica is created. Data integrity remains high but primary ingest performance may degrade. * If the sync replica fails, the async replica becomes the sync one, and a new async replica is created. The primary may experience some ingest performance degradation during this time. | | Cost composition | Primary + async (2x) | Primary + 2 async (3x) | Primary + 1 async + 1 sync (3x) | | Tier | Performance, Scale, and Enterprise | Scale and Enterprise | Scale and Enterprise | The `High` and `Highest` HA strategies are available with the [Scale and the Enterprise](../pricing-and-account-management) pricing plans. To enable HA for a service: 1. **In Tiger Console, select the service to enable replication for.** 2. **Click `Operations`, then select `High availability`.** 3. **Choose your replication strategy, then click `Change configuration`.** ![High availability replica configuration in Tiger Console](/docs/_astro/tiger-console-ha-replicas.BsuEJHbW_Znnyey.webp) 4. **In `Change high availability configuration`, click `Change config`.** To change your HA replica strategy, click `Change configuration`, choose a strategy and click `Change configuration`. To download the connection information for the HA replica, either click the link next to the replica `Active configuration`, or find the information in the `Overview` tab for this service. ## Test failover for your HA replicas To test the failover mechanism, you can trigger a switchover. A switchover is a safe operation that attempts a failover, and throws an error if the replica or primary is not in a state to safely switch. 1. **Connect to your primary node as `tsdbadmin` or another user that is part of the `tsdbowner` group** Note You can also connect to the HA replica and check its node using this procedure. 2. **At the `psql` prompt, connect to the `postgres` database** ```sql \c postgres ``` You should see `postgres=>` prompt. 3. **Check if your node is currently in recovery** ```sql select pg_is_in_recovery(); ``` 4. **Check which node is currently your primary** ```sql select * from pg_stat_replication; ``` Note the `application_name`. This is your service ID followed by the node. The important part is the `-an-0` or `-an-1`. 5. **Schedule a switchover** ```sql CALL tscloud.cluster_switchover(); ``` By default, the switchover occurs in 30 secs. You can change the time by passing an interval, like this: ```sql CALL tscloud.cluster_switchover('15 seconds'::INTERVAL); ``` 6. **Wait for the switchover to occur, then check which node is your primary** ```sql SELECT * FROM pg_stat_replication; ``` You should see a notice that your connection has been reset, like this: ```sql FATAL: terminating connection due to administrator command SSL connection has been closed unexpectedly The connection to the server was lost. Attempting reset: Succeeded. ``` 7. **Check the `application_name`** If your primary was `-an-1` before, it should now be `-an-0`. If it was `-an-0`, it should now be `-an-1`. # Overview > Reduce downtime risk and scale reads with database replication In Tiger Cloud, replicas are copies of the primary data instance in a Tiger Cloud service. If your primary becomes unavailable, Tiger Cloud automatically fails over to your HA replica. The replication strategies offered by Tiger Cloud are: * [High Availability (HA) replicas](../high-availability/high-availability): significantly reduce the risk of downtime and data loss due to system failure, and enable services to avoid downtime during routine maintenance. * [Read replicas](../high-availability/read-scaling): safely scale a service to power your read-intensive apps and business intelligence tooling and remove the load from the primary data instance. For MST, see [Failover in Managed Service for TimescaleDB](/docs/deploy/mst/failover). For self-hosted TimescaleDB, see [Replication and high availability](/docs/deploy/self-hosted/replication-and-ha/about-ha). ## Rapid recovery By default, all services have rapid recovery enabled. Because compute and storage are handled separately in Tiger Cloud, services recover quickly from compute failures, but usually need a full recovery from backup for storage failures. * **Compute failure**: the most common cause of database failure. Compute failures can be caused by hardware failing, or through things like unoptimized queries, causing increased load that maxes out the CPU usage. In these cases, data on disk is unaffected and only the compute and memory needs replacing. Tiger Cloud recovery immediately provisions new compute infrastructure for the service and mounts the existing storage to the new node. PostgreSQL crash recovery then replays any unapplied WAL from disk. This process typically only takes thirty seconds. However, depending on the amount of WAL that needs replaying this may take up to twenty minutes. Even in the worst-case scenario, Tiger Cloud recovery is an order of magnitude faster than a standard recovery from backup. * **Storage failure**: in the rare occurrence of disk failure, Tiger Cloud automatically [performs a full recovery from backup](../high-availability/backup-restore). If CPU usage for a service runs high for long periods of time, issues such as WAL archiving getting queued behind other processes can occur. This can cause a failure and could result in a larger data loss. To avoid data loss, services are monitored for this kind of scenario. # Read scaling > Scale reads horizontally with read replica sets and automated load balancing When read-intensive workloads compete with high ingest rates, your primary data instance can become a bottleneck. Spiky query traffic, analytical dashboards, and business intelligence tools risk slowing down ingest performance and disrupting critical write operations. With read replica sets in Tiger Cloud, you can scale reads horizontally and keep your applications responsive. By offloading queries to replicas, your service maintains high ingest throughput while serving large or unpredictable read traffic with ease. This approach not only protects write performance but also gives you confidence that your read-heavy apps and BI workloads will run smoothly, even under pressure. ![Horizontal read scaling in Tiger Cloud: writes go to a primary, while reads are distributed across read replicas through a single load-balanced endpoint](/docs/_astro/horizontal-read-scaling-in-tiger-cloud-light.FKvfNjnh_1Ew1eU.webp) ![Horizontal read scaling in Tiger Cloud: writes go to a primary, while reads are distributed across read replicas through a single load-balanced endpoint](/docs/_astro/horizontal-read-scaling-in-tiger-cloud-dark.CujqDL0m_ZAil5b.webp) This page shows you how to create and manage read replica sets in Tiger Console. ## What is read replication? A read replica is a read-only copy of your primary database instance. Queries on read replicas have minimal impact on the performance of the primary instance. This enables you to interact with up-to-date production data for analysis, or to scale out reads beyond the limits of your primary instance. Read replicas can be short-lived and deleted when a session of data analysis is complete, or long-running to power an application or a business intelligence tool. A read replica set in Tiger Cloud is a group of one or more read replica nodes that are accessed through the same endpoint. You query each set as a single replica. Tiger Cloud balances the load between the nodes in the set for you. You can create as many read replica sets as you need. For security and resource isolation, each read replica set has unique connection details. You use read replica sets for horizontal **read** scaling. To limit data loss for your Tiger Cloud services, use [HA replicas](../high-availability/high-availability). ## Prerequisites for this procedure To follow these steps, you'll need: * A Tiger Cloud service. * A [read-only user](../security/read-only-role#create-a-read-only-user) on the primary data instance. A user with read-only permissions cannot make changes in the primary database. This user is propagated to the read replica set when you create it. ## Create a read replica set To create a secure read replica set for your read-intensive apps: 1. **In Tiger Console, select your target service** 2. **Click `Operations` > `Read scaling` > `Add a read replica set`** 3. **Configure your replica set** Configure the number of nodes, compute size, connection pooling, and the name for your replica, then click `Create read replica set`. ![Create a set in](/docs/_astro/create-read-replica-set-tiger-console.BOwX6j3I_1MzWxL.webp) 4. **Save the connection information** The username and password of a read replica set are the same as the primary service. They cannot be changed independently. The connection information for each read replica set is unique. You can add or remove nodes from an existing set and the connection information of that set will remain the same. To find the connection information for an existing read replica set: 1. Select the primary service in Tiger Console. 2. Click `Operations` > `Read scaling`. 3. Click the link icon next to the replica set in the list. ## Edit a read replica set You can edit an existing read replica set to better handle your reads. This includes changing the number of nodes, compute size, storage, and IOPS, as well as configuring VPC and other features. To change the compute and storage configuration of your read replica set: 1. **In Tiger Console, expand and click the read replica set under your primary service** ![s in](/docs/_astro/read-replica-sets-tiger-console.DRE2fSvD_1NIiPt.webp) 2. **Click `Operations` > `Compute and storage`** ![compute and storage in](/docs/_astro/read-replica-set-config-tiger-console.BZJHSkD0_Z25HFhW.webp) 3. **Change the replica configuration and click `Apply`** ## Manage data lag for your read replica sets Read replica sets use asynchronous replication. This can cause a slight lag in data to the primary database instance. The lag is measured in bytes, against the current state of the primary instance. To check the status and lag for your read replica set: 1. **In Tiger Console, select your primary service** 2. **Click `Operations` > `Read scaling`** You see a list of configured read replica sets for this service, including their status and lag: ![sets](/docs/_astro/configured-replica-set-tiger-console.3fi7MRoJ_1013A9.webp) 3. **Configure the allowable lag** 1. Select the replica set in the list. 2. Click `Operations` > `Database parameters`. 3. Adjust `max_standby_streaming_delay` and `max_standby_archive_delay`. This is not recommended for cases where changes must be immediately represented, for example, for user credentials. ## Delete a read replica set To delete a replica set: 1. **In Tiger Console, select your primary service** 2. **Click `Operations` > `Read scaling`** 3. **Click the trash icon next to a replica set** Confirm the deletion when prompted. # Monitor your services > View logs, metrics, and performance insights for your Tiger Cloud services Get complete visibility into your service performance with Tiger Cloud's powerful monitoring suite. Whether you're optimizing for peak efficiency or troubleshooting unexpected behavior, Tiger Cloud gives you the tools to quickly identify and resolve issues. When something doesn't look right, Tiger Cloud provides a complete investigation workflow: ![Monitoring suite in Tiger Cloud](/docs/_astro/tiger-monitoring-workflow-diagram.ndjrElOs_KWOK2.webp) ![Monitoring suite in Tiger Cloud](/docs/_astro/tiger-monitoring-workflow-diagram-dark.CTYAlyS5_ZUKbEt.webp) 1. **Pinpoint the bottleneck**: check [**Metrics**](#metrics) to identify exactly when CPU, memory, or storage spiked. 2. **Find the root cause**: review [**Logs**](#logs) for errors or warnings that occurred during the incident. 3. **Identify the culprit**: examine [**Insights**](#insights) to see which queries were running at that time and how they impacted resources. 4. **Check background activity**: look at [**Jobs**](#jobs) to see if scheduled tasks triggered the issue. 5. **Investigate active connections**: use [**Connections**](#connections) to see what clients were connected and what queries they were running. Want to save some time? Check out [**Recommendations**](#recommendations) for alerts that may have already flagged the problem! This page explains what specific data you get at each point. ## Metrics Tiger Cloud shows you CPU, memory, and storage metrics for up to 30 previous days and with down to 10-second granularity. To access metrics, select your service in Tiger Console, then click `Monitoring` > `Metrics`: ![Service metrics](/docs/_astro/service-metrics-tiger-console.CVDYrWWU_11SHsc.webp) The following metrics are represented by graphs: * CPU, in mCPU * Memory, in GiB * Queries per second, in q/sec * Storage used, in GiB * Read and write storage I/O, in ops/sec * Read and write storage bandwidth, in MiB/sec When you hit the limits: * **For CPU and memory**: provision more for your service in `Operations` > `Compute and storage`. * **For storage, I/O, and bandwidth**: these resources depend on your storage type and I/O Boost settings. The standard high-performance storage gives you 16TB of compressed data on a single server, regardless of the number of hypertables in your service. See [About storage tiers](/docs/learn/data-lifecycle/storage/about-storage-tiers) for how to change the available storage, I/O, and bandwidth. Hover over the graph to view metrics for a specific time point. Select an area in the graph to zoom into a specific period. Gray bars indicate that metrics have not been collected for the period shown: ![Metrics not collected](/docs/_astro/tsc-metrics_graybar.C2YZ0u1f_SSgWP.webp) ### Understand high memory usage It is normal to observe high overall memory usage for your Tiger Cloud services, especially for workloads with active read and write. Tiger Cloud service run on Linux, and high memory usage is a particularity of the Linux page cache. The Linux kernel stores file-backed data in memory to speed up read operations. PostgreSQL, and by extension, Tiger Cloud services rely heavily on disk I/O to access tables, WALs, and indexes. When your service reads these files, the kernel caches them in memory to improve performance for future access. Page cache entries are not [locked memory](https://www.gnu.org/s/libc/manual/html_node/Locked-Memory-Details.html): they are evictable and are automatically reclaimed by the kernel when actual memory pressure arises. Therefore, high memory usage shown in the monitoring dashboards is often not due to service memory allocation, but the beneficial caching behavior in the Linux kernel. The trick is to distinguish between normal memory utilization and memory pressure. High memory usage does not necessarily mean a problem, especially on read replicas or after periods of activity. For a more accurate view of database memory consumption, look at PostgreSQL-specific metrics, such as shared\_buffers or memory context breakdowns. Only [take action](https://www.postgresql.org/docs/current/runtime-config-resource.html#RUNTIME-CONFIG-RESOURCE-MEMORY) if you see signs of real memory pressure, such as OOM (Out Of Memory) events or degraded performance. ### Service states Tiger Console gives you a visual representation of the state of your service. The following states are represented with the following colors: | State | Color | | ----------- | ------ | | Configuring | Yellow | | Deleted | Yellow | | Deleting | Yellow | | Optimizing | Green | | Paused | Grey | | Pausing | Grey | | Queued | Yellow | | Ready | Green | | Resuming | Yellow | | Unstable | Yellow | | Upgrading | Yellow | | Read-only | Red | ## Logs Tiger Cloud shows you detailed logs for your service, which you can filter by type, date, and time, and then export to CSV. To access logs, select your service in Tiger Console, then click `Monitoring` > `Logs`: ![Find logs faster](/docs/_astro/tiger-console-service-logs.DLosa5jQ_Z29cxVQ.webp) ## Insights Insights help you get a comprehensive understanding of how your queries perform over time, and make the most efficient use of your resources. To view insights, select your service, then click `Monitoring` > `Insights`. Search or filter queries by type, maximum execution time, and time frame. ![Insights](/docs/_astro/insights-overview-tiger-console.i9IYWTi-_Z1cSVHC.webp) Insights include `Metrics`, `Current lock contention`, and `Queries`. `Metrics` provides a visual representation of CPU, memory, and storage input/output usage over time. It also overlays the execution times of the top three queries matching your search. This helps correlate query executions with resource utilization. Select an area of the graph to zoom into a specific time frame. `Current lock contention` shows how many queries or transactions are currently waiting for locks held by other queries or transactions. `Queries` displays the top 50 queries matching your search. This includes a whole range of metrics: ![Queries](/docs/_astro/tiger-console-query-insights.DVctYj3R_rLYWM.webp) | Column | Description | | --------------------- | --------------------------------------------------------------------------------------------------------------- | | `Executions` | Number of times the query ran during the selected period. | | `Total rows` | Total rows scanned, inserted, or updated by the query during the selected period. | | `Total time` | Combined execution time across all runs of the query. | | `Time (P95)` | Execution time in the worst typical case. 95th percentile. | | `CPU User (P95)` | CPU spent on the query's own work — calculations, sorting, filtering. 95th percentile, in millicores. | | `CPU System (P95)` | CPU spent by the OS for the query — disk reads, memory management, network I/O. 95th percentile, in millicores. | | `Memory (P95)` | Memory used during query execution. 95th percentile, in MiB. | | `Storage read (P95)` | Disk read operations caused by the query. 95th percentile, in ops/sec. | | `Storage write (P95)` | Disk write operations caused by the query. 95th percentile, in ops/sec. | | `Hypertables` | Whether the query ran on a hypertable. | | `Columnar tables` | Whether the query read data from a chunk in the columnstore. | | `User name` | The database user who ran the query. | These metrics calculations are based on the entire period you've selected. For example, if you've selected six hours, all the metrics represent an aggregation of the previous six hours of executions. Note If you have just completed a query, it can take some minutes for it to show in the table. Wait a little, then refresh the page to see your query. Check out the last update value at the top of the query table to identify the timestamp from the last processed query stat. Click a query in the list to see the drill-down view. This view not only helps you identify spikes and unexpected behaviors, but also offers information to optimize your query. ![Queries drill-down view](/docs/_astro/query-drill-down-view-tiger-console.DQQSXNNW_S2949.webp) This view includes the following graphs: * `Execution time`: the median and P95 query execution times over the selected period. This is useful to understand the consistency and efficiency of your query's execution over time. * `EXPLAIN` plan: for queries that take more than 10 seconds to execute, there is an EXPLAIN plan collected automatically. * `Rows`: the impact of your query on rows over time. If it's a `SELECT` statement, it shows the number of rows retrieved, while for an `INSERT/UPDATE` statement, it reflects the rows inserted. * `Plans and executions`: the number of query plans and executions over time. You can use this to optimize query performance, helping you assess if you can benefit from prepared statements to reduce planning overhead. * `Shared buffers hit and miss`: shared buffers play a critical role in PostgreSQL's performance by caching data in memory. A shared buffer hit occurs when the required data block is found in the shared buffer memory, while a miss indicates that PostgreSQL couldn't locate the block in memory. A miss doesn't necessarily mean a disk read, because PostgreSQL may retrieve the data from the operating system's disk pages cache. If you observe a high number of shared buffer misses, your current shared buffers setting might be insufficient. Increasing the shared buffer size can improve cache hit rates and query speed. * `Cache hit ratio`: measures how much of your query's data is read from shared buffers. A 100% value indicates that all the data required by the query was found in the shared buffer, while a 0% value means none of the necessary data blocks were in the shared buffers. This metric provides a clear understanding of how efficiently your query leverages shared buffers, helping you optimize data access and database performance. * `CPU time (user)`: CPU time the query spends on its own work — parsing, sorting, filtering, computing results. A spike here means the query is doing more computation than expected. Check for missing indexes, unnecessary joins, or large sequential scans. * `CPU time (system)`: CPU time the OS spends servicing the query — disk reads, memory management, network I/O. High values point to heavy disk access or memory pressure rather than query logic. * `Memory allocated`: memory the query uses during execution. Sudden increases often come from inefficient joins, large sorts spilling to disk, or growing result sets. If usage stays near the service's limits, optimize the query or provision more resources. * `Storage IO bytes read`: data the query reads from disk over time. High read volumes point to full table scans or queries accessing more data than necessary. Adding indexes or narrowing the time range helps reduce reads. * `Storage IO bytes written`: data the query writes to disk over time. This includes inserts, updates, and temporary files from sorts or joins. Unusually high writes often mean large intermediate results are spilling to disk. ## Jobs Tiger Cloud summarizes all [jobs](/docs/build/data-management/about-automation) set up for your service along with their details like type, target object, and status. This includes native Tiger Cloud jobs as well as custom jobs you configure based on your specific needs. 1. **View jobs** To view jobs, select your service in Tiger Console, then click `Monitoring` > `Jobs`: ![Jobs](/docs/_astro/tiger-console-jobs-table.BJLLWbId_2w8JGD.webp) Click `Timeline` at the top right to see the recent job runs with their statuses. Select the timeframe from 1 to 24 hours: ![Jobs timeline](/docs/_astro/tiger-console-jobs-timeline-view.B9LcH-xP_Z2ii5eF.webp) 2. **View job details** Click a job ID in the list to view its config and run history: ![Job details](/docs/_astro/tiger-console-job-details.M9gynlga_1olJiS.webp) 3. **Edit the job config** Click the pencil icon to edit the job config: ![Update job config](/docs/_astro/tiger-console-edit-job.o--VQZER_1Ecg7.webp) ## Connections Tiger Cloud lists current and past connections to your service. This includes details like the corresponding query, connecting application, username, connection status, start time, and duration. To view connections, select your service in Tiger Console, then click `Monitoring` > `Connections`. Expand the query underneath each connection to see the full SQL. ![Connections](/docs/_astro/tiger-console-service-connections.B7ohsR73_ZXvDaW.webp) Click the trash icon next to a connection in the list to terminate it. A lock icon means that a connection cannot be terminated; hover over the icon to see the reason. ## Recommendations Tiger Cloud offers specific tips on configuring your service. This includes a wide range of actions, from finishing account setup to tuning your service for the best performance. For example, Tiger Cloud may recommend a more suitable chunk interval or draw your attention to consistently failing jobs. To view recommendations, select your service in Tiger Console, then click `Monitoring` > `Recommendations`: ![Recommendations](/docs/_astro/recommendations-tiger-cloud.DthX8HUN_Z1CcH5B.webp) ## Query-level statistics with `pg_stat_statements` You can also get query-level statistics for your services with the `pg_stat_statements` extension. This includes the time spent planning and executing each query; the number of blocks hit, read, and written; and more. `pg_stat_statements` comes pre-installed with Tiger Cloud. Note For more information about `pg_stat_statements`, see the [PostgreSQL documentation](https://www.postgresql.org/docs/current/pgstatstatements.html). Query the `pg_stat_statements` view as you would any PostgreSQL view. The full view includes superuser queries used by Tiger Cloud to manage your service in the background. To view only your queries, filter by the current user. [Connect](/docs/get-started/quickstart/create-service#connect-to-your-service) to your service and run the following command: ```sql SELECT * FROM pg_stat_statements WHERE pg_get_userbyid(userid) = current_user; ``` For example, to identify the top five longest-running queries by their mean execution time: ```sql SELECT calls, mean_exec_time, query FROM pg_stat_statements WHERE pg_get_userbyid(userid) = current_user ORDER BY mean_exec_time DESC LIMIT 5; ``` Or the top five queries with the highest relative variability in the execution time, expressed as a percentage: ```sql SELECT calls, stddev_exec_time/mean_exec_time*100 AS rel_std_dev, query FROM pg_stat_statements WHERE pg_get_userbyid(userid) = current_user ORDER BY rel_std_dev DESC LIMIT 5; ``` For more examples and detailed explanations, see the [blog post on identifying performance bottlenecks with `pg_stat_statements`](https://www.tigerdata.com/blog/using-pg-stat-statements-to-optimize-queries). # Billing and account management > Manage billing, pricing plans, and account settings for Tiger Cloud on AWS As we enhance our offerings and align them with your evolving needs, pricing plans provide more value, flexibility, and efficiency for your business. Whether you're a growing startup or a well-established enterprise, our plans are structured to support your journey towards greater success. ![pricing plans](/docs/_astro/tiger-pricing.D4VriqHo_Z1ITAcw.webp) ![pricing plans](/docs/_astro/tiger-pricing-dark.CAPZJelR_1rI8TC.webp) This page explains pricing plans for Tiger Cloud, and how to easily manage your Tiger Cloud account. Pricing plans give you: * **Enhanced performance**: with increased CPU and storage capacities, your apps run smoother and more efficiently, even under heavy loads. * **Improved scalability**: as your business grows, so do your demands. Pricing plans scale with you, they provide the resources and support you need at each stage of your growth. Scale up or down based on your current needs, ensuring that you only pay for what you use. * **Better support**: access to enhanced support options, including production support and dedicated account management, ensures you have the help you need when you need it. * **Greater flexibility**: we know that one size doesn't fit all. Pricing plans give you the flexibility to choose the features and support levels that best match your business and engineering requirements. The ability to add features like I/O Boost and customize your pricing plan means you can tailor Tiger Cloud services to fit your specific needs. * **Cost efficiency**: by aligning our pricing with the value delivered, we ensure that you get the most out of every dollar spent. Our goal is to help you achieve more with less. It’s that simple! You don't pay for automated backups or networking costs, such as data ingest or egress. There are no per-query fees, nor additional costs to read or write data. It's all completely transparent, easily understood, and up to you. Using self-hosted TimescaleDB and our open-source products is still free. If you create a Tiger Cloud account from AWS Marketplace, the pricing options are pay-as-you-go and annual commit. See [AWS pricing](#aws-marketplace-pricing) for details. ## Disaggregated, consumption-based compute and storage [Section titled “Disaggregated, consumption-based compute and storage”](#disaggregated-consumption-based-compute-and-storage) With Tiger Cloud, you are not limited to pre-set compute and storage. Get as much as you need when provisioning your services or later, as your needs grow. * **Compute**: pay only for the compute resources you run. Compute is metered on an hourly basis, and you can [scale it up to 64 CPU](/docs/deploy/tiger-cloud/tiger-cloud-aws/service-management/change-resources) at any time. You can also [scale out using replicas](/docs/deploy/tiger-cloud/tiger-cloud-aws/high-availability/read-scaling) as your application grows. We also provide services to help you lower your compute needs while improving query performance. Tiger Cloud is very efficient and generally needs less compute than other databases to deliver the same performance. The best way to size your needs is to sign up for a $1000 credit, 30-day free trial and test with a realistic workload. * **Storage**: pay only for the storage you consume. You have high-performance storage for more frequently accessed data, and [low-cost bottomless storage in S3](/docs/build/data-management/storage/manage-storage) for older or less frequently accessed data. The high-performance storage offers you up to 64 TB of compressed (typically 80-100 TB uncompressed) data and is metered on your average GB consumption per hour. We can help you compress your data by up to 98% so you pay even less. For low-cost storage, Tiger Data charges for data tiered based on its original uncompressed size in the high-performance storage tier. There are no additional expenses, such as data transfer or compute. For easy upgrades, each service stores the TimescaleDB binaries. This contributes up to 900 MB to overall storage, which amounts to less than $.80/month in additional storage costs. ## How your bill is calculated [Section titled “How your bill is calculated”](#how-your-bill-is-calculated) You are billed at the end of each month in arrears. Your monthly invoice includes an itemized cost accounting for each Tiger Cloud service and any additional charges. Tiger Cloud charges are based on consumption and your pricing plan: * **Compute**: billed and metered on an hourly basis. This means that you are billed for a full hour even if the actual consumption is less. You can scale compute up and down at any time. If the compute config changes mid-hour, you are billed for the config used at the end of that hour. * **Storage**: billed and metered on a quarter of an hour basis. Storage grows and shrinks automatically with your data. For example, over the last month your Tiger Cloud service has been running compute for 500 hours total: * 375 hours with 2 CPU * 125 hours 4 CPU and consumed high-performance storage for 720 hours total: * 200 hours with 100 GB * 520 hours with 150 GB **Compute cost** = (`375` x `hourly price for 2 CPU`) + (`125` x `hourly price for 4 CPU`) **High-performance storage cost** = (`200` x `100 GB` x `hourly price per GB`) + (`520` x `150 GB` x `hourly price per GB`) Some add-ons such as tiered storage, HA replicas, and connection pooling may incur additional charges. These charges are clearly marked in your billing snapshot in Tiger Console. Tips You are charged for all active standard services in your account, even if you are not actively using them. To reduce costs, pause or delete your unused services. ## Use Tiger Cloud for free [Section titled “Use Tiger Cloud for free”](#use-tiger-cloud-for-free) Are you just starting out with Tiger Cloud? Activate a $1000-credit, 30-day trial of our Performance or Scale plan (no credit card required). After 30 days, any remaining credits expire and we may remove your data unless you've added a payment method. After you have completed your trial, choose the [pricing plan](pricing-and-account-management#features-included-in-each-pricing-plan) that suits your business and engineering needs. Two free services are included for all plans. Note Free services are currently in beta. If you want to try out features in a higher pricing plan before upgrading, contact us. ## Upgrade or downgrade your pricing plans at any time [Section titled “Upgrade or downgrade your pricing plans at any time”](#upgrade-or-downgrade-your-pricing-plans-at-any-time) You can upgrade or downgrade between the Performance and Scale plans whenever you want using [Tiger Console](https://console.cloud.tigerdata.com/). To move to Enterprise, [get in touch with Tiger Data](https://www.tigerdata.com/contact/). If you switch your pricing plan mid-month, your prices are prorated to when you switch. Your services are not interrupted when you switch, so you can keep working without any hassle. ## Monitor usage and costs You keep track of your monthly usage in [Tiger Console](https://console.cloud.tigerdata.com/dashboard/billing/details). Console shows your resource usage and dashboards with performance insights. This allows you to closely monitor your services’ performance, and any need to scale your services or upgrade your pricing plan. Console also shows your month-to-date accrued charges, as well as a forecast of your expected month-end bill. Your previous invoices are also available as PDFs for download. Tips You are charged for all active standard services in your account, even if you are not actively using them. To reduce costs, pause or delete your unused services. ## Tiger Cloud support [Section titled “Tiger Cloud support”](#tiger-cloud-support) Tiger Data runs a global support organization with Customer Satisfaction (CSAT) scores above 99%. Support covers all timezones and is fully staffed at weekend hours. All paid pricing plans have free Developer Support through email with a target response time of 1 business day; we are often faster. If you need 24x7 responsiveness, talk to us about [Production Support](https://www.tigerdata.com/support). You can open, view, reply to, and close support tickets from the `Support` tab in Tiger Console: ![Manage support in Tiger Cloud](/docs/_astro/tiger-cloud-manage-support.oAio5bba_Z1Ot3bO.webp) ## Charging for HA and read replicas [Section titled “Charging for HA and read replicas”](#charging-for-ha-and-read-replicas) HA and read replicas are both charged at the same rate as your primary services, based on the compute and primary storage consumed by your replicas. Data tiered to our bottomless storage tier is shared by all database replicas; replicas accessing tiered storage do not add to your bill. ## Charging over regions [Section titled “Charging over regions”](#charging-over-regions) Storage is priced the same across all regions. However, compute prices vary depending on the region. This is because our cloud provider (AWS) prices infrastructure differently based on region. ## Features included in each pricing plan [Section titled “Features included in each pricing plan”](#features-included-in-each-pricing-plan) The available pricing plans are: * **Performance**: for cost-focused, smaller projects. No credit card required to start. * **Scale**: for developers handling critical and demanding apps. * **Enterprise**: for enterprises with mission-critical apps. The features included in each [pricing plan](https://www.tigerdata.com/pricing) are: | Feature | Performance | Scale | Enterprise | | -------------------------------------------------------------------------- | ------------------------------------ | ------------------------------------------------ | ------------------------------------------------ | | **Compute and storage** | | | | | Number of services | Up to 2 free and 4 standard services | Up to 2 free and and unlimited standard services | Up to 2 free and and unlimited standard services | | CPU limit per service | Up to 8 CPU | Up to 32 CPU | Up to 64 CPU | | Memory limit per service | Up to 32 GB | Up to 128 GB | Up to 256 GB | | Storage limit per service | Up to 64 TB | Up to 64 TB | Up to 64 TB | | Bottomless storage on S3 | | Unlimited | Unlimited | | Independently scale compute and storage | Standard services only | Standard services only | Standard services only | | **Data services and workloads** | | | | | Relational | ✓ | ✓ | ✓ | | Time-series | ✓ | ✓ | ✓ | | Vector search | ✓ | ✓ | ✓ | | Tiger Cloud SQL editor | 3 seats | 10 seats | 20 seats | | Charts | ✓ | ✓ | ✓ | | Dashboards | 2 | Unlimited | Unlimited | | **Storage and performance** | | | | | IOPS (autoscale) | 3,000 - 5,000 | 5,000 - 8,000 | 5,000 - 8,000 | | Bandwidth (autoscale) | 125 - 250 MB/s | 250 - 500 MB/s | 250 - 500 MB/s | | I/O Boost | | Add-on: Up to 40,000 IOPS, 1,500 MB/s BW | Add-on: Up to 80,000 IOPS, 2,000 MB/s BW | | **Availability and monitoring** | | | | | High-availability replicas (Automated multi-AZ failover) | ✓ | ✓ | ✓ | | Read replicas | | ✓ | ✓ | | Cross-region backup & restore | | | ✓ | | Backup reports | | 14 days | 14 days | | Point-in-time recovery and forking | 3 days | 14 days | 14 days | | Performance insights | ✓ | ✓ | ✓ | | Metrics and log exporters | | ✓ | ✓ | | **Security and compliance** | | | | | Role-based access | ✓ | ✓ | ✓ | | End-to-end encryption | ✓ | ✓ | ✓ | | Private Networking (VPC) | 1 multi-attach VPC | Unlimited multi-attach VPCs | Unlimited multi-attach VPCs | | AWS Transit Gateway | | ✓ | ✓ | | [HIPAA compliance](https://www.hhs.gov/hipaa/for-professionals/index.html) | | | ✓ | | IP address allow list | 1 list with up to 10 IP addresses | Up to 10 lists with up to 10 IP addresses each | Up to 10 lists with up to 100 IP addresses each | | Multi-factor authentication | ✓ | ✓ | ✓ | | Federated authentication (SAML) | | | ✓ | | SOC 2 Type 2 report | | ✓ | ✓ | | Penetration testing report | | | ✓ | | Security questionnaire and review | | | ✓ | | Pay by invoice | Available at minimum spend | Available at minimum spend | ✓ | | [Uptime SLAs](https://www.tigerdata.com/legal/terms-of-service) | Standard | Standard | Enterprise | | **Support and technical services** | | | | | Community support | ✓ | ✓ | ✓ | | Email support | ✓ | ✓ | ✓ | | Production support | Add-on | Add-on | ✓ | | Named account manager | | | ✓ | | JOIN services (Jumpstart Onboarding and INtegration) | | Available at minimum spend | ✓ | For a personalized quote, [get in touch with Tiger Data](https://www.tigerdata.com/contact/). ## Manage your Tiger Cloud pricing plan [Section titled “Manage your Tiger Cloud pricing plan”](#manage-your-tiger-cloud-pricing-plan) You handle all details about your Tiger Cloud project including updates to your pricing plan, payment methods, and add-ons in the [billing section in Tiger Console](https://console.cloud.tigerdata.com/dashboard/billing/details): ![Adding a payment method in Timescale](/docs/_astro/tiger-cloud-billing.DFbo-8aP_1nHIRb.webp) * **Details**: an overview of your pricing plan, usage, and payment details. You can add up to three credit cards to your `Wallet`. If you prefer to pay by invoice, [contact Tiger Data](https://www.tigerdata.com/contact/) and ask to change to corporate billing. You can also request early access to paying with crypto. * **Emails**: the addresses Tiger Data uses to communicate with you. Payment confirmations and alerts are sent to the email address you signed up with. Add another address to send details to other departments in your organization. * **History**: the list of your downloadable Tiger Cloud invoices. * **Plans**: choose the pricing plan supplying the [features](pricing-and-account-management#features-included-in-each-pricing-plan) that suit your business and engineering needs. * **Add-ons**: add `Production support` and improved database performance for mission-critical workloads. ## AWS Marketplace pricing [Section titled “AWS Marketplace pricing”](#aws-marketplace-pricing) When you get Tiger Cloud at AWS Marketplace, the following pricing options are available: * **Pay-as-you-go**: your consumption is calculated at the end of the month and included in your AWS invoice. No upfront costs, standard Tiger Cloud rates apply. * **Annual commit**: your consumption is calculated at the end of the month ensuring predictable pricing and seamless billing through your AWS account. We confirm the contract terms with you before finalizing the commitment. # Private endpoints > Connect your services to AWS PrivateLink endpoints to eliminate public internet exposure With Tiger Cloud, you can connect your services to AWS PrivateLink endpoints. A private endpoint in your VPC routes traffic to your service over the AWS backbone, without crossing the public internet or requiring VPC peering. This page describes how to authorize your AWS account in Tiger Console, create a private endpoint on the AWS side, and attach a service to it. ## Prerequisites for this procedure To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service). - An [AWS account](https://signin.aws.amazon.com/signup?request_type=register) with a [VPC](https://docs.aws.amazon.com/vpc/latest/userguide/create-vpc.html) and a subnet for the resources you will connect to Tiger Cloud. - [IAM permissions](https://docs.aws.amazon.com/vpc/latest/privatelink/identity-access-management.html) configured to create VPC endpoints. ## Set up a private endpoint connection [Section titled “Set up a private endpoint connection”](#set-up-a-private-endpoint-connection) Take the following steps to connect your Tiger Cloud service to a PrivateLink endpoint. 1. **Create an AWS account authorization** 1. In [Tiger Console](https://console.cloud.tigerdata.com/dashboard/aws-privatelink), select `Security` > `Private Endpoints` > `Configure Private Endpoint Connection`. ![Configure private endpoint connection in Tiger Cloud](/docs/_astro/private-endpoint-tiger-cloud.CsM1u4VA_Z2djjdM.webp) 2. In `Cloud provider`, select `AWS`. 3. In `Principal ID`, enter your AWS account ID. Give your authorization a name, for convenience. Warning Click the checkmark next to `Connection name` to save your authorization. Otherwise, your input is discarded. 4. Under `Alias`, copy the alias for the region in which you need to create the connection. Choose the region of your AWS resources. 5. Click `Done`. Tiger Cloud confirms your authorization. Once it is confirmed, you can connect multiple endpoints from the same authorized account. 2. **Create a VPC endpoint in AWS** 1. In [AWS Console](https://console.aws.amazon.com/), go to `VPC` > `Endpoints` > `Create endpoint`. 2. Optionally provide a name tag for your endpoint. 3. Under `Type`, select `PrivateLink Ready partner services`. ![AWS create endpoint settings](/docs/_astro/aws-private-endpoint-settings.BV5BZDys_Zmdxfq.webp) 4. Under `Service settings` > `Service name`, paste in the alias your copied in Tiger Console and click `Verify service`. Wait for the `Service name verified.` success message. ![AWS endpoint service settings](/docs/_astro/aws-private-endpoint-service-settings.5t4nz9qK_1GMB0a.webp) 5. Select the VPC that contains the resources you want to connect to Tiger Cloud, then choose one or more Availability Zones and subnets where the endpoint's private IPs will be created. For lowest latency and no cross-AZ data transfer charges, match the Availability Zones of your workloads; select multiple AZs for high availability. 6. Optionally configure the security groups and tags, then click `Create endpoint`. 7. Go to `EC2` > `Network interfaces` and copy the primary private IPv4 address of your endpoint. 3. **Sync the connection** 1. In [Tiger Console](https://console.cloud.tigerdata.com/dashboard/aws-privatelink) > `Security` > `Private Endpoints`, click `Refresh`. Tiger Cloud automatically approves connections from authorized accounts. Your connection appears in the list. ![Authorized Private Endpoint connection in Tiger Cloud](/docs/_astro/private-endpoint-connection-authorized.Bh_zZp9N_Z1nWA9c.webp) 2. Under `IP Address` click `Add IP` and paste the private endpoint IP address you have copied from AWS. 3. Under `Services` click `Attach service`. Select your service from the dropdown and click `Attach`. You can attach a service to one private endpoint. 4. From an EC2 instance inside your VPC, connect to your service using a connection string with your [connection details](/docs/integrate/find-connection-details). You should be able to connect successfully. ## Manage connections [Section titled “Manage connections”](#manage-connections) * To detach a service from a private endpoint connection, go to `Security` > `Private Endpoints`, expand the arrow in the `Services` column, and click the trash icon next to the service connection string. * To edit or remove an endpoint connection, go to `Security` > `Private Endpoints` and click the three dots next to the connection in the list. Select `Edit` or `Disconnect`, respectively. You need to detach all services from a private endpoint connection before deleting it. * To remove an authorization, click `Manage Authorizations` > trash bin icon. You need to disconnect all relevant endpoint connections before removing an authorization. # Client credentials > Create programmatic credentials to access Tiger Cloud without a password You can use client credentials to programmatically access resources instead of using your username and password. You can generate multiple client credentials for different applications or use cases rather than a single set of user credentials for everything. ## Create client credentials When you create client credentials, a public key and a private key are generated. These keys act as the username and password for programmatic client applications. It is important that you save these keys in a safe place. You can also delete these client credentials when the client applications no longer need access to Tiger Cloud resources. For more information about obtaining an access token programmatically, see the [Tiger Cloud Terraform provider documentation](https://registry.terraform.io/providers/timescale/timescale/latest/docs). 1. **Log in to Tiger Console and click `Settings`** 2. **Click `Create credentials`** 3. **In the `New client credentials` dialog, you can view the `Public key` and the `Secret Key`** Copy your secret key and store it in a secure place. You won't be able to view the `Secret Key` again in Tiger Console. 4. **Click `Done`** You can use these keys in your client applications to access Tiger Cloud resources inside the respective project. Tiger Cloud generates a default `Name` for the client credentials. 5. **Click the ⋮ menu and select `Rename credentials`** 6. **In the `Edit credential name` dialog, type the new name and click `Accept`** ## Delete client credentials 1. **Log in to Tiger Console and click `Settings`** 2. **Click the `⋮` menu of the client credential, and select `Delete`** 3. **In the `Are you sure` dialog, type the name of the client credential, and click `Delete`** # IP allow list > Restrict access to your services with IP address allow lists You can restrict access to your Tiger Cloud services to trusted IP addresses only. This prevents unauthorized connections without the need for a [Virtual Private Cloud](../security/vpc). Creating IP allow lists helps comply with security standards such as SOC 2 or HIPAA that require IP filtering. This is especially useful in regulated industries like finance, healthcare, and government. For a more fine-grained control, you create separate IP allow lists for [service and data management](/docs/get-started/quickstart/create-service). ## Create and attach an IP allow list for service management You create an IP allow list at the [project level](../security/members), then attach your service to it. Important You attach a service to either one VPC, or one IP allow list. You cannot attach a service to a VPC and an IP allow list at the same time. 1. **In Tiger Console, select `Security` > `IP Allow List`, then click `Create IP Allow List`** ![Create IP allow list](/docs/_astro/create-ip-allow-list-tiger-console.CQABvOHi_KyQo1.webp) 2. **Enter your trusted IP addresses** The number of IP addresses that you can include in one list depends on your [pricing plan](../pricing-and-account-management). ![Add IP addresses to allow list](/docs/_astro/add-ip-addresses-to-allow-list-tiger-console.Ds8zI430_217RCA.webp) 3. **Name your allow list and click `Create IP Allow List`** Click `+ Create IP Allow List` to create another list. The number of IP allow lists you can create depends on your [pricing plan](../pricing-and-account-management). 4. **Select a Tiger Cloud service, then click `Operations` > `Security` > `IP Allow List`** ![Attach IP allow list](/docs/_astro/attach-ip-allow-list-tiger-console.DAd9QaHC_CQzmd.webp) 5. **Select the list in the drop-down and click `Apply`** 6. **Type `Apply` in the confirmation popup** You have created and attached an IP allow list for service management operations. You can unattach or change the list attached to a service from the same tab. ## Create an IP allow list for querying your data You create an IP allow list in the Data view settings. 1. **In Tiger Console, toggle `Data view` at the top right** 2. **Click the cog icon to open `Preferences` > `Settings`** 3. **Log into PopSQL** 4. **Scroll down and toggle `IP Allowlist`** 5. **Add IP addresses** 1. Click `Add entry`. 2. Enter an IP address or a range of IP addresses. 3. Click `Add`. 4. When all the IP addresses have been added, click `Apply`. 5. Click `Confirm`. You have successfully added an IP allow list for querying your service in Data view. # Control user access to projects > Manage project roles, add and remove users, and transfer ownership When you sign up for a [$1000-credit, 30-day free trial](https://console.cloud.tigerdata.com/), Tiger Cloud creates a project with built-in role-based access. A single email address can create only one project. To work across multiple projects, ask an Owner to add you to their project. Role-based access includes the following roles: * **Owner**: Tiger Cloud assigns this role to you when your project is created. As the Owner, you can add and delete other users, transfer project ownership, administer services, and edit project settings. * **Admin**: the Owner assigns this role to other users in the project. A user with the Admin role has the same scope of rights as the Owner but cannot transfer project ownership. * **Developer**: the Owner and Admins assign this role to other users in the project. A Developer can build, deploy, and operate services across projects, but does not have administrative privileges over users, roles, or billing. A Developer can invite other users to the project, but only with the Viewer role. * **Viewer**: the Owner and Admins assign this role to other users in the project. A Viewer has limited, read-only access to Tiger Console. This means that a Viewer cannot modify services and their configurations in any way. A Viewer has no access to Data view and has read-queries-only access to SQL editor. ![Project users in Console](/docs/_astro/tiger-console-project-roles-overview.YQXi0kOj_2pBFfD.webp) If you have the [Enterprise pricing plan](../pricing-and-account-management), you can use your company [SAML](../security/saml) identity provider to log in to Console. Note User roles in a Tiger Cloud project do not overlap with the database-level roles for the individual services. This page describes the project roles available in Console. For the database-level user roles, see [Manage data security in your Tiger Cloud service](../security/read-only-role). ## Add a user to your project New users do not need to have a Tiger Cloud account before you add them, they are prompted to create one when they respond to the confirmation email. Existing users join a project in addition to the other projects they are already members of. To add a user to a project: 1. **In Tiger Console, click `Users`, then click `Add new user`** 2. **Type the email address of the person that you want to add, select their role, and click `Invite user`** [Enterprise pricing plan](../pricing-and-account-management) and SAML users receive a notification in Console. Users in the other pricing plans receive a confirmation email. The new user then [joins the project](#join-a-project). ## Join a project When you are asked to join a project, Tiger Console sends you an invitation email. Follow the instructions in the invitation email to join the project: * New Tiger Cloud user 1. **In the invitation email, click `Accept Invite`** Tiger Cloud opens. 2. **Follow the setup wizard and create a new account** You are added to the project you were invited to. * Existing Tiger Cloud user 1. **In the invitation email, click `Accept Invite`** Tiger Console opens, and you are added to the project. * Enterprise plan and SAML user 1. **Log in to Console using your company's identity provider** 2. **Click `Notifications`, then accept the invitation** Tiger Console opens, and you are added to the project. As you are now included in more than one project, you can easily [change projects](#change-your-current-project). ## Resend a project invitation Project invitations are valid for 7 days. To resend a project invitation: 1. **In Tiger Console, click `Users`** 2. **Next to the person you want to invite to your project, click `Resend invitation`** ## Change your current project Note A single email address can create only one project in Console. To work across multiple projects, ask an Owner to [add you to their project](#add-a-user-to-your-project). To change the project you are currently working in: 1. **In Tiger Console, click the arrows next to the project name in the top left** ![Change project in Console](/docs/_astro/tiger-console-change-project.D-LN4vwi_1pfWTk.webp) 2. **Select the project you want to use** ## Transfer project ownership Each Tiger Cloud project has one Owner. As the project Owner, you have rights to add and delete users, edit project settings, and transfer the Owner role to another user. When you transfer ownership to another user, you lose your ownership rights. To transfer project ownership: 1. **In Tiger Console, click `Users`** 2. **Next to the person you want to transfer project ownership to, click `⋮` > `Transfer project ownership`** If you are unable to transfer ownership, hover over the greyed out button to see the details. 3. **Enter your password, and click `Verify`** 4. **Complete the two-factor authentication challenge and click `Confirm`** If you have the [Enterprise pricing plan](../pricing-and-account-management), and log in to Tiger Cloud using [SAML authentication](../security/saml) or have not enabled [two-factor authentication](../security/multi-factor-authentication), [contact support](https://www.tigerdata.com/contact) to transfer project ownership. ## Leave a project To stop working in a project: 1. **In Tiger Console, click `Users`** 2. **Click `⋮` > `Leave project`, then click `Leave`** Your account is removed from the project immediately, you can no longer access this project. ## Change roles of other users in a project The Owner can change the roles of all users in the project. An Admin can change the roles of all users other than the Owner. Developer and Viewer cannot change the roles of other users. To change the role for another user: 1. **In Tiger Console, click `Users`** 2. **Next to the corresponding user, select another role in the dropdown** The user role is changed immediately. ## Remove users from a project To remove a user's access to a project: 1. **In Tiger Console, click `Users`** 2. **Next to the person you want to remove, click `⋮` > `Remove`** 3. **In `Remove user`, click `Remove`** The user is deleted immediately, they can no longer access your project. # Multi-factor authentication > Set up two-factor authentication for your Tiger Cloud account You can use two-factor authentication to log in to your Tiger Cloud account. Two-factor authentication, also known as two-step verification or 2FA, enables secure logins that require an authentication code in addition to your user password. The code is provided by an authenticator app on your mobile device. There are multiple authenticator apps available. ![2FA](/docs/_astro/tiger-console-2fa.BcA5vlKw_Mwqle.webp) This page describes how to configure two-factor authentication with Google Authenticator. ## Prerequisites for this procedure To follow these steps, you'll need: * The [Google Authenticator application](https://support.google.com/accounts/answer/1066447) installed on your mobile device. ## Configure two-factor authentication with Google Authenticator Take the following steps to configure two-factor authentication: 1. **Log in to Tiger Console with your username and password. 2FA is not available if you log in with Google SSO.** 2. **Click your user name in the top right of Tiger Console and select `Account settings`** 3. **Click `Add two-factor authentication`** 4. **On your mobile device, open Google Authenticator, tap `+`, and select `Scan a QR code`** 5. **Scan the QR code provided by Tiger Console in `Connect to an authenticator app` and click `Next`** 6. **In Tiger Console, enter the verification code provided by Google Authenticator, and click `Next`** 7. **In `Save your recovery codes`, copy, download, or print the recovery codes** These are used to recover your account if you lose your device. 8. **Verify that you have saved your recovery codes, by clicking `OK, I saved my recovery codes`** 9. **If two-factor authentication is enabled correctly, an email notification is sent to you** Warning If you lose access to the mobile device you use for multi-factor authentication, and you do not have access to your recovery codes, you cannot sign in to your Tiger Cloud account. To regain access to your account, contact . ## Regenerate recovery codes If you do not have access to your authenticator app and need to log in to Tiger Console, you can use your recovery codes. Recovery codes are single-use. If you've used all 10 recovery codes, or lost access to them, you can generate another list. Generating a new list invalidates all previously generated codes. 1. **Log in to Tiger Console with your username and password** 2. **Click your user name in the top right and select `Account settings`** 3. **In `Two-factor authentication`, click `Regenerate recovery codes`** 4. **Enter the verification code from your authenticator app** Alternatively, if you do not have access to the authenticator app, click `Use recovery code instead` to enter a recovery code. 5. **Click `Next`** 6. **In `Save your recovery codes`, copy, download, or print the recovery codes** These are used to recover your account if you lose your device. 7. **Verify that you have saved your recovery codes, by clicking `OK, I saved my recovery codes`** ## Remove two-factor authentication If you need to enroll a new device for two-factor authentication, you can remove two-factor authentication from your account and then add it again with your new device. 1. **Log in to Tiger Console with your username and password** 2. **Click your user name in the top right of Tiger Console and select `Account settings`** 3. **In `Two-factor authentication`, click `Remove two-factor authentication`** 4. **Enter the verification code from your authenticator app to confirm** Alternatively click `Use recovery code instead` to type the recovery code. 5. **Click `Remove`** # About security in Tiger Cloud > Learn how Tiger Cloud protects your PostgreSQL data with encryption, access controls, and network security Protecting data starts with secure software engineering. At Tiger Data, we embed security into every stage of development, from static code analysis and automated dependency scanning to rigorous code security reviews. To go even further, we developed [pgspot](https://github.com/timescale/pgspot), an open-source extension to identify security issues with PostgreSQL extensions, which strengthens the broader ecosystem as well as our own platform. Tiger Data products do not have any identified weaknesses. ![Tiger Cloud platform security architecture overview](/docs/_astro/tiger-platform-security-overview.DexpnyhU_Z2vdcGA.webp) ![Tiger Cloud platform security architecture overview](/docs/_astro/tiger-platform-security-overview-dark.DE5djrdr_794D7.webp) This page lists the additional things we do to ensure operational security and to lock down Tiger Cloud services. To see our security features at a glance, see [Tiger Data Security](https://www.tigerdata.com/security). ## Role-based access [Section titled “Role-based access”](#role-based-access) Tiger Cloud provides role-based access for you to: * Administer your Tiger Cloud project In Tiger Console, users with the Owner, Admin, and Viewer roles have different permissions to manage users and services in the project. * Manage data in each service To restrict access to your data on the database level, you can create other roles on top of the default tsdbadmin role. ## Data encryption [Section titled “Data encryption”](#data-encryption) Your data on Tiger Cloud is encrypted both in transit and at rest. Both active databases and backups are encrypted. Tiger Cloud uses AWS as its cloud provider, with all the security that AWS provides. Data encryption uses the industry-standard AES-256 algorithm. Cryptographic keys are managed by [AWS Key Management Service (AWS KMS)](https://aws.amazon.com/kms/). Keys are never stored in plaintext. For more information about AWS security, see the AWS documentation on security in [Amazon Elastic Compute Cloud](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/data-protection.html) and [Elastic Block Storage](https://docs.aws.amazon.com/ebs/latest/userguide/ebs-encryption.html). ## Networking security [Section titled “Networking security”](#networking-security) Customer access to Tiger Cloud services is only provided over TLS-encrypted connections. There is no option to use unencrypted plaintext connections. ## Networking with Virtual Private Cloud (VPC) peering [Section titled “Networking with Virtual Private Cloud (VPC) peering”](#networking-with-virtual-private-cloud-vpc-peering) When using VPC peering, **no public Internet-based access** is provided to the service. Service addresses are published in public DNS, but they can only be connected to from the customer's peered VPC using private network addresses. VPC peering only enables communication to be initiated from your Customer VPC to Tiger Cloud services running in the Tiger Cloud VPC. Tiger Cloud cannot initiate communication with your VPC. To learn how to set up VPC Peering, see [Secure your Tiger Cloud services with VPC Peering and AWS PrivateLink](/docs/deploy/tiger-cloud/tiger-cloud-aws/security/vpc). ## Private endpoints [Section titled “Private endpoints”](#private-endpoints) Tiger Cloud enables you to connect to your services through a private endpoint in AWS using AWS PrivateLink. Traffic routes over the AWS network, without crossing the public internet or requiring VPC peering. Once a service is attached, it is reachable only through the endpoint's private IP in your VPC, and communication can only be initiated from your VPC to Tiger Cloud. ## IP address allow lists [Section titled “IP address allow lists”](#ip-address-allow-lists) You can allow only trusted IP addresses to access your Tiger Cloud services. You do this by creating [IP address allow lists](/docs/deploy/tiger-cloud/tiger-cloud-aws/security/ip-allow-list) and attaching them to your services. ## Operator access [Section titled “Operator access”](#operator-access) Normally all the resources required for providing Tiger Cloud services are automatically created, maintained and terminated by the Tiger Cloud infrastructure. No manual operator intervention is required. However, the Tiger Data operations team has the capability to securely log in to the service virtual machines for troubleshooting purposes. These accesses are audit logged. No customer access to the virtual machine level is provided. ## GDPR compliance [Section titled “GDPR compliance”](#gdpr-compliance) Tiger Data complies with the European Union's General Data Protection Regulation (GDPR), and all practices are covered by our [Privacy Policy](https://www.tigerdata.com/legal/privacy) and the [Terms of Service](https://www.tigerdata.com/legal/terms-of-service). All customer data is processed in accordance with Tiger Data's GDPR-compliant [Data Processor Addendum](https://www.tigerdata.com/legal/timescale-cloud-data-processing-addendum), which applies to all Tiger Data customers. Tiger Data operators never access customer data, unless explicitly requested by the customer to troubleshoot a technical issue. The Tiger Data operations team has mandatory recurring training regarding the applicable policies. ## HIPAA compliance [Section titled “HIPAA compliance”](#hipaa-compliance) The Tiger Cloud [Enterprise plan](/docs/deploy/tiger-cloud/tiger-cloud-aws/pricing-and-account-management#features-included-in-each-pricing-plan) is Health Insurance Portability and Accountability Act (HIPAA) compliant. This allows organizations to securely manage and analyze sensitive healthcare data, ensuring they meet regulatory requirements while building compliant applications. ## SOC 2 compliance [Section titled “SOC 2 compliance”](#soc-2-compliance) Tiger Cloud is SOC 2 Type 2 compliant. This ensures that organizations can securely manage customer data in alignment with industry standards for security, availability, processing integrity, confidentiality, and privacy. It helps businesses meet trust requirements while confidently building applications that handle sensitive information. The annual SOC 2 report is available to customers on the Scale or Enterprise pricing plans. Open a [support ticket](https://console.cloud.tigerdata.com/dashboard/support) to get access to it. # Manage data security in your service > Restrict data access with read-only PostgreSQL roles When you create a service, Tiger Cloud assigns you the tsdmadmin role. This role has full permissions to modify data in your service. However, Tiger Cloud does not provide superuser access. tsdmadmin is not a superuser. As tsdmadmin, you can use standard PostgreSQL means to create other roles or assign individual permissions. This page shows you how to create a read-only role for your database. Adding a read-only role does not provide resource isolation. To restrict the access of a read-only user, as well as isolate resources, create a [read replica](../high-availability/read-scaling) instead. Note The database-level roles for the individual services in your project do not overlap with the Tiger Cloud project user roles. This page describes the database-level roles. For user roles available in Console, see [Control user access to Tiger Cloud projects](../security/members). ## Create a read-only user You can create a read-only user to provide limited access to your database. 1. **Connect to your service as the tsdbadmin user** 2. **Create the new role** ```sql CREATE ROLE readaccess; ``` 3. **Grant usage on the schema to allow access to objects within it** ```sql GRANT USAGE ON SCHEMA TO readaccess; ``` 4. **Grant the appropriate permissions for the role, as required** For example, to grant `SELECT` permissions to a specific table, use: ```sql GRANT SELECT ON TO readaccess; ``` To grant `SELECT` permissions to all tables in a specific schema, use: ```sql GRANT SELECT ON ALL TABLES IN SCHEMA TO readaccess; ``` 5. **Create a new user** ```sql CREATE USER read_user WITH PASSWORD 'read_password'; ``` 6. **Assign the role to the new user** ```sql GRANT readaccess TO read_user; ``` # SAML authentication > Configure SAML single sign-on for Enterprise accounts Tiger Cloud offers SAML authentication as part of its [Enterprise](https://www.tigerdata.com/enterprise) offering. SAML (Security Assertion Markup Language) is an open standard for exchanging authentication and authorization data between parties. With SAML enabled Tiger Cloud customers can log into their Tiger Cloud account using their existing SSO service provider credentials. Note Tiger Cloud supports most SAML providers that can handle IDP-initiated login ### SAML offers many benefits for the Enterprise including: * Improved security: SAML centralizes user authentication with an identity provider (IdP). This makes it more difficult for attackers to gain access to user accounts. * Reduced IT costs: SAML can help companies reduce IT costs by eliminating the need to manage multiple user accounts and passwords. * Improved user experience: SAML makes it easier for users to access multiple applications and resources. Reach out to your CSM/sales contact to get started. The connection process looks like the following: 1. **Configure the IdP to support SAML authentication** This will involve creating a new application and configuring the IdP with the settings provided by your contact. 2. **Provide your contact with the requested details about your IdP** 3. **Test the SAML authentication process to make sure that it is working correctly** # Connect with a stricter SSL mode > Use verify-ca or verify-full SSL modes for stricter connection security The default connection string for Tiger Cloud uses the Secure Sockets Layer (SSL) mode `require`. Users can choose not to use Transport Layer Security (TLS) while connecting to their databases, but connecting to production databases without encryption is strongly discouraged. To achieve even stronger security, clients may select to verify the identity of the server. If you want your connection client to verify the server's identity, you can connect with an [SSL mode](https://www.postgresql.org/docs/current/libpq-ssl.html#LIBPQ-SSL-SSLMODE-STATEMENTS) of `verify-ca` or `verify-full`. To do so, you need to store a copy of the certificate chain where your connection tool can find it. This section provides instructions for setting up a stricter SSL connection. ## SSL certificates As part of the secure connection protocol, the server proves its identity by providing clients with a certificate. This certificate should be issued and signed by a well-known and trusted Certificate Authority. Because requesting a certificate from a Certificate Authority takes some time, Tiger Cloud services are initialized with a self-signed certificate. This lets you start up a service immediately. After your service is started, a signed certificate is requested behind the scenes. The new certificate is usually received within 30 minutes. Your certificate is then replaced with almost no interruption. Connections are reset, and most clients reconnect automatically. Free services do not supply SSL certificates. With the signed certificate, you can switch your connections to a stricter SSL mode, such as `verify-ca` or `verify-full`. For more information on the different SSL modes, see the [PostgreSQL SSL mode descriptions](https://www.postgresql.org/docs/current/libpq-ssl.html#LIBPQ-SSL-SSLMODE-STATEMENTS). ## Connect to your database with a stricter SSL mode To set up a stricter SSL connection: 1. Generate a copy of your certificate chain and store it in the right location 2. Change your Tiger Cloud connection string 1) **Use the `openssl` tool to connect to your Tiger Cloud service and get the certificate bundle** Store the bundle in a file called `bundle.crt`. Replace `$SERVICE_URL_WITH_PORT` with your Tiger Cloud connection URL: ```shell openssl s_client -showcerts -partial_chain -starttls postgres \ -connect $SERVICE_URL_WITH_PORT < /dev/null 2>/dev/null | \ awk '/BEGIN CERTIFICATE/,/END CERTIFICATE/{ print }' > bundle.crt ``` 2) **Copy the bundle to your clipboard** * MacOS ```shell pbcopy < bundle.crt ``` * Linux ```shell xclip -sel clip < bundle.crt ``` * Windows ```shell clip.exe < bundle.crt ``` 3) **Navigate to https\://whatsmychaincert.com/** This online tool generates a full certificate chain, including the root Certificate Authority certificate, which is not included in the certificate bundle returned by the database. 4) **Paste your certificate bundle in the provided box** Check `Include Root Certificate`. Click `Generate Chain`. 5) **Save the downloaded certificate chain to `~/.postgresql/root.crt`** 6) **Change your Tiger Cloud connection string from `sslmode=require` to either `sslmode=verify-full` or `sslmode=verify-ca`** For example, to connect to your database with `psql`, run: ```shell psql "postgres://tsdbadmin@$SERVICE_URL_WITH_PORT/tsdb?sslmode=verify-full" ``` ## Verify the certificate type used by your database To check whether the certificate has been replaced yet, connect to your database instance and inspect the returned certificate. We are using two certificate providers - Google and ZeroSSL, that's why chances are you can have a certificate issued by either of those CAs: ```shell openssl s_client -showcerts -partial_chain -starttls postgres -connect : < /dev/null 2>/dev/null | grep "Google\|ZeroSSL" ``` # AWS Transit Gateway > Securely connect to your services from AWS, Google Cloud, Azure, or on-premise environments using AWS Transit Gateway [AWS Transit Gateway](https://aws.amazon.com/transit-gateway/) enables you to securely connect to your **Tiger Cloud on AWS** from Google Cloud, Microsoft Azure, AWS, or any other cloud or on-premise environment. You use AWS Transit Gateway as a traffic controller for your network. Instead of setting up multiple direct connections to different clouds, on-premise data centers, and other AWS services, you connect everything to AWS Transit Gateway. This simplifies your network and makes it easier to manage and scale. You can then create a peering connection between your Tiger Cloud services and AWS Transit Gateway in Tiger Cloud. This means that, no matter how big or complex your infrastructure is, you can connect securely to your Tiger Cloud services. For enhanced security, you can add peering connections to multiple Transit Gateways with overlapping CIDRs, Tiger Cloud creates a new isolated connection for every unique Transit Gateway ID. Otherwise, the existing connection is reused for your services in the same project and region. To configure this secure connection, you: 1. Connect your infrastructure to AWS Transit Gateway. 2. Create a Tiger Cloud Peering VPC with a peering connection to AWS Transit Gateway. 3. Accept and configure the peering connection on your side. 4. Attach individual services to the Peering VPC. AWS Transit Gateway enables you to connect from almost any environment, this page provides examples for the most common use cases. * Amazon Web Services 1. **Create a Peering VPC in Tiger Console** 1. In `Security` > `VPC`, click `Create a VPC`: ![Creating a new peering VPC in Tiger Console](/docs/_astro/add-peering-vpc-tiger-cloud.BHFW98a8_Hcuw1.webp) 2. Choose your region and IP range, name your VPC, then click `Create VPC`: ![Configuring VPC region, IP range, and name](/docs/_astro/configure-peering-vpc-tiger-cloud.DZom78te_Zspfuq.webp) Your service and Peering VPC must be in the same AWS region. The number of Peering VPCs you can create in your project depends on your [pricing plan](../../pricing-and-account-management). If you need another Peering VPC, either contact or change your plan in [Tiger Console](https://console.cloud.tigerdata.com/). 3. In the `VPC Peering` column, click `Add`. 4. Provide your AWS account ID, Transit Gateway ID, CIDR ranges, and AWS region. Tiger Cloud creates a new isolated connection for every unique Transit Gateway ID. ![Adding a peering connection with AWS account details](/docs/_astro/add-peering-tiger-cloud.9tTfnXvs_Z1YfI1i.webp) 5. Click `Add connection`. 2. **Accept and configure peering connection in your AWS account** Once your peering connection appears as `Processing`, you can accept and configure it in AWS: 1. Accept the peering request coming from Tiger Cloud. The request can take up to 5 min to arrive. Within 5 more minutes after accepting, the peering should appear as `Connected` in Tiger Console. 2. Configure at least the following in your AWS account networking: your subnet route table to route traffic to your Transit Gateway for the Peering VPC CIDRs; your Transit Gateway route table to route traffic to the newly created Transit Gateway peering attachment for the Peering VPC CIDRs; and security groups to allow outbound TCP 5432. 3. **Attach a Tiger Cloud service to the Peering VPC in Tiger Console** 1. Select the service you want to connect to the Peering VPC. 2. Click `Operations` > `Security` > `VPC`. 3. Select the VPC, then click `Attach VPC`. You cannot attach a Tiger Cloud service to multiple Tiger Cloud VPCs at the same time. * Google Cloud 1. **Connect your infrastructure to AWS Transit Gateway** Establish connectivity between Google Cloud and AWS. See [Connect HA VPN to AWS peer gateways](https://cloud.google.com/network-connectivity/docs/vpn/how-to/connect-ha-vpn-aws-peer-gateway). 2. **Create a Peering VPC in Tiger Console** 1. In `Security` > `VPC`, click `Create a VPC`: ![Creating a new peering VPC in Tiger Console](/docs/_astro/add-peering-vpc-tiger-cloud.BHFW98a8_Hcuw1.webp) 2. Choose your region and IP range, name your VPC, then click `Create VPC`: ![Configuring VPC region, IP range, and name](/docs/_astro/configure-peering-vpc-tiger-cloud.DZom78te_Zspfuq.webp) Your service and Peering VPC must be in the same AWS region. The number of Peering VPCs you can create in your project depends on your [pricing plan](../../pricing-and-account-management). If you need another Peering VPC, either contact or change your plan in [Tiger Console](https://console.cloud.tigerdata.com/). 3. In the `VPC Peering` column, click `Add`. 4. Provide your AWS account ID, Transit Gateway ID, CIDR ranges, and AWS region. Tiger Cloud creates a new isolated connection for every unique Transit Gateway ID. ![Adding a peering connection with AWS account details](/docs/_astro/add-peering-tiger-cloud.9tTfnXvs_Z1YfI1i.webp) 5. Click `Add connection`. 3. **Accept and configure peering connection in your AWS account** Once your peering connection appears as `Processing`, you can accept and configure it in AWS: 1. Accept the peering request coming from Tiger Cloud. The request can take up to 5 min to arrive. Within 5 more minutes after accepting, the peering should appear as `Connected` in Tiger Console. 2. Configure at least the following in your AWS account networking: your subnet route table to route traffic to your Transit Gateway for the Peering VPC CIDRs; your Transit Gateway route table to route traffic to the newly created Transit Gateway peering attachment for the Peering VPC CIDRs; and security groups to allow outbound TCP 5432. 4. **Attach a Tiger Cloud service to the Peering VPC in Tiger Console** 1. Select the service you want to connect to the Peering VPC. 2. Click `Operations` > `Security` > `VPC`. 3. Select the VPC, then click `Attach VPC`. You cannot attach a Tiger Cloud service to multiple Tiger Cloud VPCs at the same time. * On-premise 1. **Connect your infrastructure to AWS Transit Gateway** Establish connectivity between your on-premise infrastructure and AWS. See [Centralize network connectivity using AWS Transit Gateway](https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/centralize-network-connectivity-using-aws-transit-gateway.html). 2. **Create a Peering VPC in Tiger Console** 1. In `Security` > `VPC`, click `Create a VPC`: ![Creating a new peering VPC in Tiger Console](/docs/_astro/add-peering-vpc-tiger-cloud.BHFW98a8_Hcuw1.webp) 2. Choose your region and IP range, name your VPC, then click `Create VPC`: ![Configuring VPC region, IP range, and name](/docs/_astro/configure-peering-vpc-tiger-cloud.DZom78te_Zspfuq.webp) Your service and Peering VPC must be in the same AWS region. The number of Peering VPCs you can create in your project depends on your [pricing plan](../../pricing-and-account-management). If you need another Peering VPC, either contact or change your plan in [Tiger Console](https://console.cloud.tigerdata.com/). 3. In the `VPC Peering` column, click `Add`. 4. Provide your AWS account ID, Transit Gateway ID, CIDR ranges, and AWS region. Tiger Cloud creates a new isolated connection for every unique Transit Gateway ID. ![Adding a peering connection with AWS account details](/docs/_astro/add-peering-tiger-cloud.9tTfnXvs_Z1YfI1i.webp) 5. Click `Add connection`. 3. **Accept and configure peering connection in your AWS account** Once your peering connection appears as `Processing`, you can accept and configure it in AWS: 1. Accept the peering request coming from Tiger Cloud. The request can take up to 5 min to arrive. Within 5 more minutes after accepting, the peering should appear as `Connected` in Tiger Console. 2. Configure at least the following in your AWS account networking: your subnet route table to route traffic to your Transit Gateway for the Peering VPC CIDRs; your Transit Gateway route table to route traffic to the newly created Transit Gateway peering attachment for the Peering VPC CIDRs; and security groups to allow outbound TCP 5432. 4. **Attach a Tiger Cloud service to the Peering VPC in Tiger Console** 1. Select the service you want to connect to the Peering VPC. 2. Click `Operations` > `Security` > `VPC`. 3. Select the VPC, then click `Attach VPC`. You cannot attach a Tiger Cloud service to multiple Tiger Cloud VPCs at the same time. You can now securely access your services in Tiger Cloud. # VPC Peering > Secure your services with Virtual Private Cloud peering You use Virtual Private Cloud (VPC) peering to ensure that your Tiger Cloud services are only accessible through your secured AWS infrastructure. This reduces the potential attack vector surface and improves security. The data isolation architecture that ensures a highly secure connection between your apps and Tiger Cloud is: ![Tiger Cloud VPC isolation architecture](/docs/_astro/tsc-vpc-architecture.CCzyUivO_28dJGl.webp) ![Tiger Cloud VPC isolation architecture](/docs/_astro/tsc-vpc-architecture-dark.BjWSy3Rm_Z2me1uq.webp) Your customer apps run inside your AWS Customer VPC, your Tiger Cloud services always run inside the secure Tiger Cloud VPC. You control secure communication between apps in your VPC and your services using a dedicated Peering VPC. The AWS PrivateLink connecting Tiger Cloud VPC to the dedicated Peering VPC gives the same level of protection as using a direct AWS PrivateLink connection. It only enables communication to be initiated from your Customer VPC to services running in the Tiger Cloud VPC. Tiger Cloud cannot initiate communication with your Customer VPC. To configure this secure connection, you first create a Peering VPC with AWS PrivateLink in Tiger Console. After you have accepted and configured the peering connection to your Customer VPC, you use AWS Security Groups to restrict the apps in your Customer VPC that are visible to the Peering VPC. The last step is to attach individual services to the Peering VPC in Tiger Console. * You create each Peering VPC on a [Tiger Cloud project level](/docs/deploy/tiger-cloud/tiger-cloud-aws/security/members). * You **can attach**: * Up to 50 Customer VPCs to a Peering VPC. * A Tiger Cloud service to a single Peering VPC at a time. The service and the Peering VPC must be in the same AWS region. However, you can peer a Customer VPC and a Peering VPC that are in different regions. * Multiple Tiger Cloud services to the same Peering VPC. * You **cannot attach** a Tiger Cloud service to multiple Peering VPCs at the same time. The number of Peering VPCs you can create in your project depends on your [pricing plan](/docs/deploy/tiger-cloud/tiger-cloud-aws/pricing-and-account-management). If you need another Peering VPC, either contact or change your pricing plan in [Tiger Console](https://console.cloud.tigerdata.com/). ## Prerequisites for this procedure To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service). - Permissions in your AWS account to: * Accept VPC peering requests * Configure route table rules * Configure security group and firewall rules ## Set up a secured connection between Tiger Cloud and AWS [Section titled “Set up a secured connection between Tiger Cloud and AWS”](#set-up-a-secured-connection-between-tiger-cloud-and-aws) To connect to a Tiger Cloud service using VPC peering, your apps and infrastructure must be already running in an Amazon Web Services (AWS) VPC. You can peer your VPC from any AWS region. However, your Peering VPC must be within one of the [Cloud-supported regions](/docs/get-started/choose-your-path/supported-platforms). The stages to create a secured connection between Tiger Cloud services and your AWS infrastructure are: 1. [Create a Peering VPC in Tiger Console](#create-a-peering-vpc-in-tiger-console) 2. [Complete the VPC connection in your AWS](#complete-the-vpc-connection-in-aws) 3. [Set up security groups in your AWS](#set-up-security-groups-in-aws) 4. [Attach a Tiger Cloud service to the Peering VPC](#attach-a-tiger-cloud-service-to-the-peering-vpc) ### Create a peering VPC in Tiger Console [Section titled “Create a peering VPC in Tiger Console”](#create-a-peering-vpc-in-tiger-console) Create the VPC and the peering connection that enables you to securely route traffic between Tiger Cloud and your Customer VPC in a logically isolated virtual network. 1. **In Tiger Console > Security > VPC, click `Create a VPC`** ![Creating a new peering VPC in Tiger Console](/docs/_astro/add-peering-vpc-tiger-cloud.BHFW98a8_Hcuw1.webp) 2. **Choose your region and IP range, name your VPC, then click `Create VPC`** ![Configuring VPC region, IP range, and name](/docs/_astro/configure-peering-vpc-tiger-cloud.DZom78te_Zspfuq.webp) The IP ranges of the Peering VPC and Customer VPC should not overlap. 3. **For as many peering connections as you need** 1. In the `VPC Peering` column, click `Add`. 2. Enter information about your existing Customer VPC, then click `Add Connection`. ![Adding a peering connection with AWS account details](/docs/_astro/add-peering-tiger-cloud.9tTfnXvs_Z1YfI1i.webp) * You **can attach**: * Up to 50 Customer VPCs to a Peering VPC. * A Tiger Cloud service to a single Peering VPC at a time. The service and the Peering VPC must be in the same AWS region. However, you can peer a Customer VPC and a Peering VPC that are in different regions. * Multiple Tiger Cloud services to the same Peering VPC. * You **cannot attach** a Tiger Cloud service to multiple Peering VPCs at the same time. The number of Peering VPCs you can create in your project depends on your [pricing plan](/docs/deploy/tiger-cloud/tiger-cloud-aws/pricing-and-account-management). If you need another Peering VPC, either contact or change your pricing plan in [Tiger Console](https://console.cloud.tigerdata.com/). Tiger Cloud sends a peering request to your AWS account so you can [complete the VPC connection in AWS](#complete-the-vpc-connection-in-aws). ### Complete the VPC connection in AWS [Section titled “Complete the VPC connection in AWS”](#complete-the-vpc-connection-in-aws) When you receive the Tiger Cloud peering request in AWS, edit your routing table to match the `IP Range` and `CIDR block` between your Customer and Peering VPCs. When you peer a VPC with multiple CIDRs, all CIDRs are added to the Tiger Cloud rules automatically. After you have finished peering, further changes in your VPC's CIDRs are not detected automatically. If you need to refresh the CIDRs, recreate the peering connection. The request acceptance process is an important safety mechanism. Do not accept a peering request from an unknown account. 1. **In AWS > VPC Dashboard > Peering connections, select the peering connection request from Tiger Cloud** Copy the peering connection ID to the clipboard. The connection request starts with `pcx-`. 2. **In the peering connection, click `Route Tables`, then select the `Route Table ID` that corresponds to your VPC** 3. **In `Routes`, click `Edit routes`** You see the list of existing destinations. ![Adding a new VPC route in the AWS console](/docs/_astro/tsc-vpc-add-route.BPur-Qe3_ieNCz.webp) If you do not already have a destination that corresponds to the `IP range / CIDR block` of your Peering VPC: 1. Click `Add route`, and set: `Destination`: the CIDR block of your Peering VPC. For example: `10.0.0.7/17`. `Target`: the peering connection ID you copied to your clipboard. 2. Click `Save changes`. Network traffic is secured between your AWS account and Tiger Cloud for this project. ### Set up security groups in AWS [Section titled “Set up security groups in AWS”](#set-up-security-groups-in-aws) Security groups allow specific inbound and outbound traffic at the resource level. You can associate a VPC with one or more security groups, and each instance in your VPC may belong to a different set of security groups. The security group choices for your VPC are: * Create a security group to use for your Tiger Cloud VPC only. * Associate your VPC with an existing security group. * Do nothing, your VPC is automatically associated with the default one. To create a security group specific to your Tiger Cloud Peering VPC: 1. **In AWS > VPC Dashboard > Security Groups, click `Create security group`** 2. **Enter the rules for this security group** ![AWS VPC security group configuration](/docs/_astro/aws-vpc-securitygroup.CaesDIXY_Z1PJqnn.webp) * `VPC`: select the VPC that is peered with Tiger Cloud. * `Inbound rules`: leave empty. * `Outbound rules`: * `Type`: `Custom TCP` * `Protocol`: `TCP` * `Port range`: `5432` * `Destination`: `Custom` * `Info`: the CIDR block of your Tiger Cloud Peering VPC. 3. **Click `Add rule`, then click `Create security group`** ### Attach a Tiger Cloud service to the peering VPC [Section titled “Attach a Tiger Cloud service to the peering VPC”](#attach-a-tiger-cloud-service-to-the-peering-vpc) Now that Tiger Cloud is communicating securely with your AWS infrastructure, you can attach one or more services to the Peering VPC. After you attach a service to a Peering VPC, you can only access it through the peered AWS VPC. It is no longer accessible using the public internet. 1. **In Tiger Console > Services select the service you want to connect to the Peering VPC** 2. **Click `Operations` > `Security` > `VPC`** 3. **Select the VPC, then click `Attach VPC`** Your service is now securely communicating with your AWS account inside a VPC. ## Migrate a Tiger Cloud service between VPCs [Section titled “Migrate a Tiger Cloud service between VPCs”](#migrate-a-tiger-cloud-service-between-vpcs) To ensure that your applications continue to run without interruption, you keep service attached to the Peering VPC. However, you can change the Peering VPC your service is attached to, or disconnect from the Peering VPC and enable access to the service from the public internet. Note Tiger Cloud uses a different DNS for services that are attached to a Peering VPC. When you migrate a service between public access and a Peering VPC, you need to update your connection string. 1. **In Tiger Console > Services select the service to migrate** If you don't have a service, [create a new one](/docs/get-started/quickstart/create-service). 2. **Click `Operations` > `Security` > `VPC`** 3. **Select the VPC, then click `Attach VPC`** Migration takes a few minutes to complete and requires a change to DNS settings for the service. The service is not accessible during this time. If you receive a DNS error, allow some time for DNS propagation. # About Tiger Cloud services > Create, manage, and scale Tiger Cloud PostgreSQL services Tiger Cloud is the modern PostgreSQL data platform for all your applications. It enhances PostgreSQL to handle time series, events, real-time analytics, and vector search, all in a single database alongside transactional workloads. You get one system that handles live data ingestion, late and out-of-order updates, and low latency queries, with the performance, reliability, and scalability your app needs. Ideal for IoT, crypto, finance, SaaS, and a myriad other domains, Tiger Cloud allows you to build data-heavy, mission-critical apps while retaining the familiarity and reliability of PostgreSQL. A Tiger Cloud service is a single optimized PostgreSQL instance extended with innovations in the database engine and cloud infrastructure to deliver speed without sacrifice. A Tiger Cloud service is 10-1000x faster at scale! It is ideal for applications requiring strong data consistency, complex relationships, and advanced querying capabilities. Get ACID compliance, extensive SQL support, JSON handling, and extensibility through custom functions, data types, and extensions. Each service is associated with a project in Tiger Cloud. Each project can have multiple services. Each user is a [member of one or more projects](/docs/deploy/tiger-cloud/tiger-cloud-aws/security/members). You create free and standard services in Tiger Console. A standard service comes with the full range of features according to your pricing plan. A free service comes at zero cost and gives you limited features and resources. ![Tiger Cloud pricing plans: Performance, Scale, and Enterprise](/docs/_astro/tiger-pricing.D4VriqHo_Z1ITAcw.webp) ![Tiger Cloud pricing plans: Performance, Scale, and Enterprise](/docs/_astro/tiger-pricing-dark.CAPZJelR_1rI8TC.webp) To the PostgreSQL you know and love, Tiger Cloud adds the following capabilities: * **Standard services**: * *Real-time analytics*: store and query [time-series data](https://www.tigerdata.com/blog/time-series-database-an-explainer#what-is-a-time-series-database) at scale for real-time analytics and other use cases. Get faster time-based queries with hypertables, continuous aggregates, and columnar storage. Save money by compressing data into the columnstore, moving cold data to low-cost bottomless storage in Amazon S3 or Azure Blob storage, and deleting old data with automated policies. * *AI-focused*: build AI applications from start to scale. Get fast and accurate similarity search with the pgvector and pgvectorscale extensions. * *Hybrid applications*: get a full set of tools to develop applications that combine time-based data and AI. All standard Tiger Cloud services include the tooling you expect for production and developer environments: [live migration](/docs/migrate/live-migration), [automatic backups and PITR](/docs/deploy/tiger-cloud/tiger-cloud-aws/high-availability/backup-restore), [high availability](/docs/deploy/tiger-cloud/tiger-cloud-aws/high-availability/high-availability), [read replicas](/docs/deploy/tiger-cloud/tiger-cloud-aws/high-availability/read-scaling), [data forking](/docs/deploy/tiger-cloud/tiger-cloud-aws/service-management/fork-services), [connection pooling](/docs/deploy/tiger-cloud/tiger-cloud-aws/service-management/connection-pooling), [tiered storage](/docs/build/data-management/storage/manage-storage), [usage-based storage](/docs/deploy/tiger-cloud/tiger-cloud-aws/pricing-and-account-management#how-your-bill-is-calculated), secure in-Tiger Console [SQL editing](/docs/build/data-management/run-queries-from-tiger-console), service [metrics](/docs/deploy/tiger-cloud/tiger-cloud-aws/monitoring#metrics) and [insights](/docs/deploy/tiger-cloud/tiger-cloud-aws/monitoring#insights), streamlined maintenance, and much more. Tiger Cloud continuously monitors your services and prevents common PostgreSQL out-of-memory crashes. * **Free services**: *PostgreSQL with TimescaleDB and vector extensions* Free services come with pre-configured CPU and memory, are hosted in `us-east-1` region, and offer limited resources with a basic feature scope. This excludes connection pooling, replication, data tiering, exporters, metrics, VPC, and other features. Free services are perfect for schema explorations, syntax validations, and other small-scale testing. Once you reach the storage limit, a free service enters a read-only state. You can [convert your free service](/docs/deploy/tiger-cloud/tiger-cloud-aws/service-management/service-management#convert-a-free-service-to-a-standard-one) to a standard one at any time. Note Free services are currently in beta. You can [manage, pause, or delete](/docs/deploy/tiger-cloud/tiger-cloud-aws/service-management/service-management) your service at any time from Tiger Console. ## Learn more about Tiger Cloud Read about Tiger Cloud features in the documentation: * Create your first [hypertable](/docs/learn/hypertables/understand-hypertables). * Run your first query using [time\_bucket()](/docs/build/data-management/query-data/advanced-analytic-queries#group-data-into-time-buckets). * Trying more advanced time-series functions, starting with [gap filling](/docs/build/data-management/query-data/advanced-analytic-queries#fill-gaps-in-time-series-data) or [real-time aggregates](/docs/build/continuous-aggregates/create-a-continuous-aggregate). ## Keep testing during your free trial You're now on your way to a great start with Tiger Cloud. You have a $1000-credit, 30-day free trial with Tiger Cloud to continue to test your use case. Before the end of your trial, make sure you add your credit card information. This ensures a smooth transition after your trial period concludes. If you have any questions, you can [join our community Slack group](https://slack-login.timescale.com) or [contact us](https://www.tigerdata.com/contact) directly. ## Advanced configuration Tiger Cloud is a versatile hosting service that provides a growing list of advanced features for your PostgreSQL and time-series data workloads. For more information about customizing your database configuration, see the [Configuration section](../configuration/about-configuration). Note The [TimescaleDB Terraform provider](https://registry.terraform.io/providers/timescale/timescale/latest/) provides configuration management resources for Tiger Cloud. You can use it to create, rename, resize, delete, and import services. For more information about the supported service configurations and operations, see the [Terraform provider documentation](https://registry.terraform.io/providers/timescale/timescale/latest/docs). # Manually change compute resources > Scale compute resources for a Tiger Cloud service with minimal downtime Tiger Cloud charges are based on the amount of storage you use. You don't pay for fixed storage size, and you don't need to worry about scaling disk size as your data grows, we handle it all for you. To reduce your data costs further, combine [hypercore](/docs/reference/timescaledb/hypercore), a [data retention policy](/docs/learn/data-lifecycle/data-retention/about-data-retention), and [tiered storage](/docs/learn/data-lifecycle/storage/about-storage-tiers). You use [Tiger Console](https://console.cloud.tigerdata.com/) to resize the compute (CPU/RAM) resources available to your Tiger Cloud services at any time, with a short downtime. ## Update compute resources for a service You can change the CPU and memory allocation for your standard service at any time with minimal downtime, usually less than a minute. The new resources become available as soon as the service restarts. You can change the CPU and memory allocation up or down, as frequently as required. ![Change resources](/docs/_astro/tiger-console-update-cpu-manually.DRnZv6ev_Ze3Xpb.webp) To change the resources of a free service, first [convert it to a standard one](../service-management/service-management#convert-a-free-service-to-a-standard-one). Note that: * For the 48 CPU / 192 GiB option, 6 CPU / 14 GiB is reserved for platform operations. * For the 64 CPU / 256 GiB option, 6 CPU / 16 GiB is reserved for platform operations. There is momentary downtime while the new compute settings are applied. In most cases, this is less than a minute. However, before making changes to your service, best practice is to enable [HA replication](../high-availability/overview) on the service. When you resize a service with HA enabled, Tiger Cloud: 1. Resizes the replica. 2. Waits for the replica to catch up. 3. Performs a switchover to the resized replica. 4. Restarts the primary. HA reduce downtime in the case of resizes or maintenance window restarts, from a minute or so to a couple of seconds. When you change resource settings, the current and new charges are displayed immediately so that you can verify how the changes impact your costs. Warning Because compute changes require an interruption to your services, plan accordingly so that the settings are applied during an appropriate service window. 1. **In Console, choose the service to modify** 2. **Click `Operations` > `Compute and storage`** 3. **Select the new `CPU / Memory` allocation** You see the allocation and costs in the comparison chart. 4. **Click `Apply`** Your service goes down briefly while the changes are applied. ## Out of memory errors If you run intensive queries on your standard services, you might encounter out of memory (OOM) errors. This occurs if your query consumes more memory than is available. When this happens, an `OOM killer` process shuts down PostgreSQL processes using `SIGKILL` commands until the memory usage falls below the upper limit. Because this kills the entire server process, it usually requires a restart. To prevent service disruption caused by OOM errors, Tiger Cloud attempts to shut down only the query that caused the problem. This means that the problematic query does not run, but that your service continues to operate normally. * If the normal OOM killer is triggered, the error log looks like this: ```yml 2021-09-09 18:15:08 UTC [560567]:TimescaleDB: LOG: server process (PID 2351983) was terminated by signal 9: Killed ``` Wait for the service to come back online before reconnecting. * Tiger Cloud shuts the client connection only If Tiger Cloud successfully guards the service against the OOM killer, it shuts down only the client connection that was using too much memory. This prevents the entire service from shutting down, so you can reconnect immediately. The error log looks like this: ```yml 2022-02-03 17:12:04 UTC [2253150]:TimescaleDB: tsdbadmin@tsdb,app=psql [53200] ERROR: out of memory ``` If you encounter OOM errors on a free service, [convert it to a standard service](../service-management/service-management#convert-a-free-service-to-a-standard-one), then increase its CPU and memory allocation as needed to resolve the memory pressure. # Connection pooling > Create and manage connection poolers in Tiger Console to improve database performance You can scale your Tiger Cloud service connections and improve its performance by using connection poolers. Tiger Cloud uses `pgBouncer` for connection pooling. If your service needs a large number of short-lived connections, a connection pooler is a great way to improve performance. For example, web, serverless, and IoT applications often use an event-based architecture where data is read or written from the database for a very short amount of time. Your application rapidly opens and closes connections while the pooler maintains a set of long-running connections to the service. This improves performance because the pooler opens the connections in advance, allowing the application to open many short-lived connections, while the service opens few, long-lived connections. ## User authentication By default, the poolers have authentication to the service, so you can use any custom users you already have set up without further configuration. You can continue using the `tsdbadmin` user if that is your preferred method. However, you might need to add custom configurations for some cases such as `statement_timeout` for a pooler user. 1. **Connect to your service as the `tsdbadmin` user and create a new role** ```sql CREATE ROLE LOGIN PASSWORD ''; ``` 2. **Change the `statement_timeout` settings to 2 seconds for this user** ```sql ALTER ROLE my_app SET statement_timeout TO '2s'; ``` 3. **In a new terminal window, connect on the pooler with the new user ``** ```bash PGPASSWORD= psql 'postgres://my_app@service.project.tsdb.cloud.timescale.com:30477/tsdb?sslmode=require' ``` The output looks something like this: ```plaintext psql (15.3 (Homebrew), server 15.4 (Ubuntu 15.4-1.pgdg22.04+1)) SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, compression: off) Type "help" for help. ``` 4. **Check that the settings are correct** ```sql SELECT current_user; ┌──────────────┐ │ current_user │ ├──────────────┤ │ my_app │ └──────────────┘ (1 row) ``` Check the `statement_timeout` setting is correct for the `` user: ```plaintext tsdb=> show statement_timeout; ┌───────────────────┐ │ statement_timeout │ ├───────────────────┤ │ 2s │ └───────────────────┘ (1 row) ``` ## Pool types When you create a connection pooler, there are two pool types to choose from: session or transaction. Each pool type uses a different mode to handle connections. Session pools allocate a connection from the pool until they are closed by the application, similar to a regular PostgreSQL connection. When the application closes the connection, it is sent back to the pool. Transaction pool connections are allocated only for the duration of the transaction, releasing the connection back to the pool when the transaction ends. If your application opens and closes connections frequently, choose the transaction pool type. By default, the pooler supports both modes simultaneously. However, the connection string you use to connect your application is different, depending on whether you want a session or transaction pool type. When you create a connection pool in the Tiger Console, you are given the correct connection string for the mode you choose. For example, a connection string to connect directly to your service looks a bit like this: ```plaintext postgres://:@service.example.cloud.timescale.com:30133/tsdb?sslmode=require ``` A session pool connection string is the same, but uses a different port number, like this: ```plaintext postgres://:@service.example.cloud.timescale.com:29303/tsdb?sslmode=require ``` The transaction pool connection string uses the same port number as a session pool connection, but uses a different database name, like this: ```plaintext postgres://:@service.example.cloud.timescale.com:29303/tsdb_transaction?sslmode=require ``` Make sure you check the Tiger Console output for the correct connection string to use in your application. ## Connection pool sizes A connection pooler manages connections to both the service itself, and the client application. It keeps a fixed number of connections open with the service, while allowing clients to open and close connections. Clients can request a connection from the session pool or the transaction pool. The connection pooler will then allocate the connection if there is one free. The number of client connections allowed to each pool is proportional to the `max_connections` parameter set for the service. The session pool can have a maximum of `max_connections - 17` client connections, while the transaction pool can have a maximum of `(max_connections - 17) * 20` client connections. Of the 17 reserved connections that are not allocated to either pool, 12 are reserved for the database superuser by default, and another 5 for Tiger Cloud operations. For example, if `max_connections` is set to 500, the maximum number of client connections for your session pool is `483 (500 - 17)` and `9,660 (483 * 20)` for your transaction pool. The configurable range of `max_connections` depends on your service's compute size. See [Min and max connection ranges](#min-and-max-connection-ranges). ## Add a connection pooler When you create a new service, you can also create a connection pooler. Alternatively, you can add a connection pooler to an existing service in Console. 1. **Log in to Console and click the service you want to add a connection pooler to** 2. **In `Operations`, click `Connection pooling` > `Add pooler`** Your pooler connection details are displayed in the `Connection pooling` tab. Use this information to connect to your transaction or session pooler. For more information about the different pool types, see the [pool types](#pool-types) section. ## Remove a connection pooler If you no longer need a connection pooler, you can remove it in Console. When you have removed your connection pooler, make sure that you also update your application to adjust the port it uses to connect to your service. 1. **In Console, select the service you want to remove a connection pooler from** 2. **Select `Operations`, then `Connection pooling`** 3. **Click `Remove connection pooler`** Confirm that you want to remove the connection pooler. After you have removed a pooler, if you add it back in the future, it uses the same connection string and port that was used before. ## pgBouncer statistics commands 1. **Connect to your service** 2. **Switch to the `pgbouncer` database: `\\c pgbouncer`** 3. **Run any read-only command for the pgBouncer CLI (for example, `SHOW STATS;`)** For full options, see the pgBouncer [docs here](https://www.pgbouncer.org/usage.html). ## VPC and connection pooling VPCs are supported with connection pooling. It does not matter the order you add the pooler or connect to a VPC. Your connection strings will automatically be updated to use the VPC connection string. ## Min and max connection ranges You can set `max_connections` within the following ranges, based on your service memory and CPU configuration: | Configuration | Minimum | Maximum | | -------------------------------------- | ------- | ------- | | Shared memory and CPU | 25 | 25 | | From 0.5 CPU / 2 GiB to 4 CPU / 16 GiB | 25 | 500 | | 8 CPU / 32 GiB and more | 25 | 2000 | `max_connections` of the primary service cannot be higher than its replica's. # Fork services > Fork a service to create isolated database branches for testing and development Modern development is highly iterative. Developers and AI agents need safe spaces to test changes before deploying them to production. Forkable services make this natural and easy. Spin up a branch, run your test, throw it away, or merge it back. A fork is an exact copy of a service at a specific point in time, with its own independent data and configuration, including: * The database data and schema * Configuration * An admin `tsdbadmin` user with a new password Forks are fully independent. Changes to the fork don't affect the parent service. You can query them, run migrations, add indexes, or test new features against the fork without affecting the original service. Forks are a powerful way to share production-scale data safely. Testing, BI and data science teams often need access to real datasets to build models or generate insights. With forkable services, you easily create fast, zero-copy branches of a production service that are isolated from production, but contain all the data needed for analysis. Rapid fork creation dramatically reduces friction getting insights from live data. ## Understand service forks You can use service forks for disaster recovery, CI/CD automation, and testing and development. For example, you can automatically test a major PostgreSQL upgrade on a fork before applying it to your production service. Tiger Cloud offers the following fork strategies: * `now`: create a fresh fork of your database at the current time. Use when: * You need the absolute latest data * Recent changes must be included in the fork * `last-snapshot`: fork from the most recent [automatic backup](../high-availability/backup-restore). Because Tiger Cloud takes automatic backups daily, this is usually within the previous 24 hours of your service. To see when the latest backup was taken, check [`Backup history`](../high-availability/backup-restore#automatic-backups) in Tiger Console. Use when: * You want the fastest possible fork creation * Up to 24 hours behind current data is acceptable * `timestamp`: fork from a specific point in time, up to 14 days before, depending on your [pricing plan](../pricing-and-account-management#features-included-in-each-pricing-plan). Use when: * Disaster recovery from a known-good state * Investigating issues that occurred at a specific time * Testing "what-if" scenarios from historical data ### Fork creation speed Fork creation speed depends on the type of service you want to create: * Free services: \~30-90 seconds. Use a Copy-on-Write storage architecture with zero-copy between a fork and the parent. * Standard services: varies with the size of your service, typically 5-20+ minutes. Use traditional storage architecture with backup restore + WAL replay. ### Billing You can fork a free service to a free or a standard service. However, you cannot fork a standard service to a free service. Billing on storage works in the following way: * High-performance storage: * Copy-on-Write: you are only billed for storage for the chunks that diverge from the parent service. * Traditional: you are billed for storage for the whole service. * Object storage tier: * [Tiered data](/docs/build/data-management/storage/manage-storage) is shared across forks using copy-on-write and traditional storage: * Chunks in tiered storage are only billed once, regardless of the number of forks * Only new or modified chunks in a fork incur additional costs For details, see [Replicas and forks with tiered data](/docs/build/data-management/storage/tiered-data-replicas-forks). ## Prerequisites for this procedure To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service). ## Manage forks Important You cannot fork a service while it is being provisioned or otherwise transitioning state. Wait for the service to reach `Running` or `Paused` before you start forking. * Tiger CLI To manage development forks: 1. **Install Tiger CLI** Use the terminal to install the CLI: * macOS ```shell brew install --cask timescale/tap/tiger-cli ``` * Debian ```shell curl -s https://packagecloud.io/install/repositories/timescale/tiger-cli/script.deb.sh | sudo os=any dist=any bash sudo apt-get install tiger-cli ``` * Ubuntu ```shell curl -s https://packagecloud.io/install/repositories/timescale/tiger-cli/script.deb.sh | sudo os=any dist=any bash sudo apt-get install tiger-cli ``` * Red Hat ```shell curl -s https://packagecloud.io/install/repositories/timescale/tiger-cli/script.rpm.sh | sudo os=rpm_any dist=rpm_any bash sudo yum install tiger-cli ``` * Fedora ```shell curl -s https://packagecloud.io/install/repositories/timescale/tiger-cli/script.rpm.sh | sudo os=rpm_any dist=rpm_any bash sudo yum install tiger-cli ``` * Windows PowerShell ```shell irm https://cli.tigerdata.com/install.ps1 | iex ``` * x-platform ```shell curl -fsSL https://cli.tigerdata.com | sh ``` 2. **Authenticate Tiger CLI** 1. In an interactive terminal, start the login flow: ```shell tiger auth login ``` Tiger CLI opens Console in your browser to sign in with OAuth. Log in, or sign up if you don't have a Tiger Cloud account yet, then authorize Tiger CLI. If the browser does not open automatically, copy the `Auth URL` printed in the terminal. For non-interactive sessions such as scripts and CI, you can authenticate with [client credentials](/docs/integrate/find-connection-details#create-client-credentials) instead of the browser flow. 2. Select a Tiger Cloud project: ```text Auth URL is: https://console.cloud.tigerdata.com/oauth/authorize?client_id=...&code_challenge_method=S256&redirect_uri=http://localhost:PORT/callback&response_type=code&state=... Opening browser for authentication... Select a project: > 1. Tiger Project (tgrproject) 2. YourCompany (Company wide project) (cpnproject) 3. YourCompany Department (dptproject) Successfully logged in (project: tgrproject) 🎉 Next steps: • Install MCP server for your favorite AI coding tool: tiger mcp install • List existing services: tiger service list • Create a new service: tiger service create • Enable read-only mode: tiger config set read_only true ``` If only one project is associated with your account, this step is skipped. Tiger CLI stores the OAuth session in your system keychain, or in `~/.config/tiger/credentials` with restricted file permissions (600) if the keychain is unavailable, and refreshes it automatically. Tiger CLI stores your configuration in `~/.config/tiger/config.yaml`. To sign out and revoke the session, run `tiger auth logout`. 3. **Test your authenticated connection to Tiger Cloud by listing services** ```bash tiger service list ``` This call returns something like: * No services: ```txt 🏜️ No services found! Your project is looking a bit empty. 🚀 Ready to get started? Create your first service with: tiger service create ``` * One or more services: ```txt ┌────────────┬─────────────────────┬────────┬─────────────┬──────────────┬──────────────────┐ │ SERVICE ID │ NAME │ STATUS │ TYPE │ REGION │ CREATED │ ├────────────┼─────────────────────┼────────┼─────────────┼──────────────┼──────────────────┤ │ tgrservice │ tiger-agent-service │ READY │ TIMESCALEDB │ eu-central-1 │ 2025-09-25 16:09 │ └────────────┴─────────────────────┴────────┴─────────────┴──────────────┴──────────────────┘ ``` 4. **Fork the service** ```shell tiger service fork tgrservice --now --no-wait --name bob ``` You must specify exactly one timing option: `--now` (fork at current state), `--last-snapshot` (fork at last snapshot, faster), or `--to-timestamp` (fork at specific point in time). By default a fork matches the resources of the parent Tiger Cloud service. For standard services, specify `--cpu` and/or `--memory` for dedicated resources. You see something like: ```txt 🍴 Forking service 'tgrservice' to create 'bob' at current state... ✅ Fork request accepted! 📋 New Service ID: 🔐 Password saved to system keyring for automatic authentication 🎯 Set service '' as default service. ⏳ Service is being forked. Use 'tiger service list' to check status. ┌───────────────────┬──────────────────────────────────────────────────────────────────────────────────────────────────┐ │ PROPERTY │ VALUE │ ├───────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────┤ │ Service ID │ │ │ Name │ bob │ │ Status │ │ │ Type │ TIMESCALEDB │ │ Region │ eu-central-1 │ │ CPU │ 0.5 cores (500m) │ │ Memory │ 2 GB │ │ Direct Endpoint │ ..tsdb.cloud.timescale.com: │ │ Created │ 2025-10-08 13:58:07 UTC │ │ Connection String │ postgresql://tsdbadmin@..tsdb.cloud.timescale.com:/tsdb?sslmode=require │ └───────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────┘ ``` 5. **When you are done, delete your forked service** 1. Use the CLI to request service delete: ```shell tiger service delete ``` 2. Validate the service delete: ```text Are you sure you want to delete service ''? This operation cannot be undone. Type the service ID '' to confirm: ``` You see something like: ```text 🗑️ Delete request accepted for service ''. ✅ Service '' has been successfully deleted. ``` * Tiger Console ```shell brew install --cask timescale/tap/tiger-cli ``` * macOS ```shell curl -s https://packagecloud.io/install/repositories/timescale/tiger-cli/script.deb.sh | sudo os=any dist=any bash sudo apt-get install tiger-cli ``` * Debian ```shell curl -s https://packagecloud.io/install/repositories/timescale/tiger-cli/script.deb.sh | sudo os=any dist=any bash sudo apt-get install tiger-cli ``` * Ubuntu ```shell curl -s https://packagecloud.io/install/repositories/timescale/tiger-cli/script.rpm.sh | sudo os=rpm_any dist=rpm_any bash sudo yum install tiger-cli ``` * Red Hat ```shell curl -s https://packagecloud.io/install/repositories/timescale/tiger-cli/script.rpm.sh | sudo os=rpm_any dist=rpm_any bash sudo yum install tiger-cli ``` * Fedora ```shell irm https://cli.tigerdata.com/install.ps1 | iex ``` * Windows PowerShell ```shell curl -fsSL https://cli.tigerdata.com | sh ``` * x-platform Tiger Console lets you create development forks and point-in-time recovery (PITR) forks. ### Development forks Development forks use the [`now`](#understand-service-forks) strategy. 1. **In Tiger Console, navigate to `Operations` > `Service Management` and click `Fork service`** ![Fork a Tiger Cloud service](/docs/_astro/create-fork-tiger-console.BhBMTQ17_Z1ezrTS.webp) 2. **Configure the fork, then click `Fork service`** A fork of the service is created. The forked service shows in `Services` with a label specifying which service it has been forked from. 3. **Update the connection strings in your app to use the fork** ### PITR forks PITR forks use the [`timestamp`](#understand-service-forks) strategy. To recover from a destructive or unwanted action, create a fork at a specific point in time. To avoid paying compute for both the recovery fork and the original service, pause the original. 1. **Navigate to `Operations` > `Service management` and click `Create recovery fork`** ![Creating a recovery fork in Tiger Console](/docs/_astro/create-recovery-fork-tiger-console.B_ta8j9w_ZQcbm9.webp) 2. **Select the recovery point, ensuring the correct time zone (UTC offset)** 3. **Select the region** If you had [cross-region backup](../high-availability/backup-restore#enable-cross-region-backup) enabled for your service, select the region in which to create the fork. 4. **Configure the fork to match the original service** Best practice is to match the same configuration you had at the point you want to recover to. You can also add an HA replica, tag the fork, or add a connection pooler. 5. **Confirm by clicking `Create recovery fork`** 6. **Update the connection strings in your app to use the fork** ## Integrate service forks in your CI/CD pipeline To fork your Tiger Cloud service using GitHub actions: 1. **Store your Tiger Cloud API key as a GitHub Actions secret** 1. In [Tiger Console](https://console.cloud.tigerdata.com/dashboard/settings), click `Create credentials`. 2. Save the `Public key` and `Secret key` locally, then click `Done`. 3. In your GitHub repository, click `Settings`, open `Secrets and variables`, then click `Actions`. 4. Click `New repository secret`, then set `Name` to `TIGERDATA_API_KEY`. 5. Set `Secret` to your Tiger Cloud API key in the following format `:`, then click `Add secret`. 2. **Add the GitHub Actions Marketplace to your workflow YAML files** For example, the following workflow forks a service when a pull request is opened, running tests against the fork, then automatically cleans up. ```yaml name: Test on a service fork on: pull_request jobs: test: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Fork Database id: fork uses: timescale/fork-service@v1 with: project_id: ${{ secrets.TIGERDATA_PROJECT_ID }} service_id: ${{ secrets.TIGERDATA_SERVICE_ID }} api_key: ${{ secrets.TIGERDATA_API_KEY }} fork_strategy: last-snapshot cleanup: true name: pr-${{ github.event.pull_request.number }} - name: Run Integration Tests env: DATABASE_URL: postgresql://tsdbadmin:${{ steps.fork.outputs.initial_password }}@${{ steps.fork.outputs.host }}:${{ steps.fork.outputs.port }}/tsdb?sslmode=require run: | npm install npm test - name: Run Migrations env: DATABASE_URL: postgresql://tsdbadmin:${{ steps.fork.outputs.initial_password }}@${{ steps.fork.outputs.host }}:${{ steps.fork.outputs.port }}/tsdb?sslmode=require run: npm run migrate ``` For the full list of inputs, outputs, and configuration options, see the [Tiger Data - Fork Service](https://github.com/marketplace/actions/tiger-data-fork-service) in GitHub marketplace. # Service explorer > Monitor service health and performance with the Service Explorer dashboard Service explorer in Tiger Console provides a rich administrative dashboard for understanding the state of your database instance. The explorer gives you insight into the performance of your database, giving you greater confidence and control over your data. The explorer works like an operations center as you develop and run your applications with Tiger Cloud. It gives you quick access to the key properties of your database, like table sizes, schema definitions, and foreign key references, as well as information specific to Tiger Cloud, like information on your hypertables and continuous aggregates. To see the explorer, select your service in Console and click `Explorer`. ## General information In the `General information` section, you can see a high-level summary of your service, including all your hypertables and relational tables. It summarizes your overall compression ratios, and other policy and continuous aggregate data. And, if you aren't already using key features like continuous aggregates, columnstore compression, or other automation policies and actions, it provides pointers to tutorials and documentation to help you get started. ![Service explorer](/docs/_astro/tiger-console-service-explorer.CiAngqJU_Z14JOnR.webp) ## Tables You can have a detailed look into all your tables, including information about table schemas, table indexes, and foreign keys. For your hypertables, it shows details about chunks, continuous aggregates, and policies such as data retention policies and data reordering. You can also inspect individual hypertables, including their sizes, dimension ranges, and columnstore compression status. From this section, you can also set an automated policy to compress chunks into the columnstore. For more information, see the [hypercore documentation](/docs/learn/columnar-storage/understand-hypercore). When you inspect a hypertable, a timeline chart visualizes its chunks, showing their relative sizes and whether each is in the rowstore or the columnstore. By default, the chart shows the latest 100 chunks; click and drag to zoom in on a specific time range. The chart also shows the status of the latest runs for the hypertable's columnstore job, with a link to the job's deep dive page. ![Service explorer tables](/docs/_astro/tiger-console-columstore-data-savings.DzFnEbdZ_Z2h6ct0.webp) For more information about hypertables, see the [hypertables section](/docs/learn/hypertables/understand-hypertables). ## Continuous aggregates In the `Continuous aggregate` section, you can see all your continuous aggregates, including top-level information such as their size, whether they are configured for real-time aggregation, and their refresh periods. ![Service explorer caggs](/docs/_astro/tiger-console-service-explorer-caggs.DvufWqGS_1utudM.webp) For more information about continuous aggregates, see the [continuous aggregates section](/docs/build/continuous-aggregates/create-a-continuous-aggregate). # Service management > Manage your service lifecycle from the Operations dashboard From the `Operations` dashboard, you can reset the password, change the environment, convert a free service to a standard one, pause, or delete the service. To fork a service, see [Fork services](../service-management/fork-services). ## Reset your service password You can reset your service password from the `Operations` dashboard. This is the password you use to connect to your service, not the password for Tiger Console. To reset your Console password, navigate to the `Account` page. When you reset your service password, you are prompted for your Console password. When you have authenticated, you can create a new service password, ask Console to auto-generate a password, or switch your authentication type between SCRAM and MD5. SCRAM (salted challenge response authentication mechanism) and MD5 (message digest algorithm 5) are cryptographic authentication mechanisms. Tiger Console uses SCRAM by default. It is more secure and strongly recommended. The MD5 option is provided for compatibility with older clients. ## Change the service environment In Tiger Cloud, you create services for the following environments: * Development services: for less demanding tasks, such as ad hoc analytics, testing, or application building. Development services have no delete protection. * Production services: for mission-critical applications like client-facing dashboards or APIs. Production services have delete protection. Tiger Cloud applies upgrades differently to development and production services. To change the environment of an existing service: 1. **In Tiger Console, select the service to update** 2. **Click `Operations` > `Environment`, then change the tag under `Change service environment`** ## Convert a free service to a standard one You can convert a free service to a standard one, to increase its resources. Note that it is only available if you haven't reached the quota on the number of standard services according to your pricing plan. 1. **In Tiger Console, select the service to update** 2. **Click `Operations` > `Environment` > `Convert service to standard` > `Yes, convert to standard`** ## Pause a service You can pause a service if you want to stop it running temporarily. When you pause a service, you are no longer billed for compute resources. However, you do need to continue paying for any storage you are using. Pausing a service ensures that it is still available, and is ready to be restarted at any time. ## Delete a service You can delete a service to remove it completely. This removes the service and its underlying data from the server. You cannot recover a deleted service. # Tiger Console overview > Manage access, security, and day-to-day operations for your services You use Tiger Console to manage your services and data in a convenient, centralized manner. When you [log into Tiger Console](https://console.cloud.tigerdata.com/), you see `Services` and other project management tabs: ![Check is running](/docs/_astro/tiger-console-services-view.F-zo-hID_1rY9dD.webp) * `Services`: create a new [service](/docs/get-started/quickstart/create-service) in your project, and view the existing services and their statuses. Each service is an optimized [PostgreSQL](/docs/learn/glossary#postgresql) instance extended with database engine innovations such as [TimescaleDB](/docs/learn/glossary#timescaledb), managed for you by Tiger Cloud. * `Security`: create [VPCs](../security/overview) and [IP allowlists](../security/overview). * `Exporters`: [export metrics and logs](../monitoring) from your services. * `Activity`: [check your services](../monitoring) and project activity. * `Settings`: [create client credentials](/docs/integrate/find-connection-details#create-client-credentials), add AI model API keys, and request a SOC 2 report. * `CLI/MCP`: install Tiger CLI and set up Tiger MCP. * `Users`: [add and remove users](../security/members) in your project. * `Billing`: [check usage](../pricing-and-account-management#monitor-usage-and-costs), [change pricing plans](../pricing-and-account-management#upgrade-or-downgrade-your-pricing-plans-at-any-time), and manage payment methods. * `Support`: open and manage [support tickets](../pricing-and-account-management#tiger-cloud-support). When you select a Tiger Cloud service in the `Services` tab, you land in the Ops view. In this view, you manage your services. You see `Overview` and other related tabs: ![Select a service to edit](/docs/_astro/ops-mode-overview-tiger-console.BaKQ_e8S_Z1vnL4H.webp) * `Overview`: get an overview of your service details, infrastructure, [connection info](/docs/get-started/quickstart/create-service#connect-to-your-service), and performance optimization tips. * `Actions`: [connect and import data](/docs/migrate) into your service. * `Explorer`: [inspect objects and storage configuration](../service-management/service-explorer) for your service. * `Monitoring`: [get visibility](../monitoring) into your service performance. * `Connectors`: [sync or stream](/docs/migrate) directly in your service, so data from another source is continuously updated. * `Operations`: fully manage your service: [configure compute and storage](../service-management/change-resources), [set up replicas](../high-availability/overview), [back up](../security/overview), upgrade, and more. * `AI`: create vector embeddings and [manage AI extensions](../tiger-cloud-extensions) for your service. Use the toggle at the top to switch to Data view. In this view, you can [write queries with autocomplete](/docs/build/data-management/run-queries-from-tiger-console#data-view) for any service, save them in folders, share them, and create charts/dashboards with the added power of AI. To query your service from any tab, click `SQL Editor` at the bottom. The floating window expands and is ready for work. Click the window icon in its top right to change the editor position on the screen. # PostgreSQL extensions > Browse and enable PostgreSQL extensions available in Tiger Cloud The following PostgreSQL extensions are installed with each Tiger Cloud service: * [Tiger Data extensions](#tiger-data-extensions) * [PostgreSQL built-in extensions](#postgresql-built-in-extensions) * [Third-party extensions](#third-party-extensions) ## Tiger Data extensions | Extension | Description | Enabled by default | | ------------------------------------------------------------------------ | ----------------------------------------------------------------------- | ----------------------------------------------------- | | [pgai](https://github.com/timescale/pgai) | Helper functions for AI workflows | For [AI-focused](/docs/get-started) services | | [pg\_textsearch](../tiger-cloud-extensions/pg-textsearch) | [BM25](https://en.wikipedia.org/wiki/Okapi_BM25)-based full-text search | Production-ready (v1.0.0) | | [pgvector](https://github.com/pgvector/pgvector) | Vector similarity search for PostgreSQL | For [AI-focused](/docs/get-started) services | | [pgvectorscale](https://github.com/timescale/pgvectorscale) | Advanced indexing for vector data | For [AI-focused](/docs/get-started) services | | [timescaledb\_toolkit](https://github.com/timescale/timescaledb-toolkit) | TimescaleDB Toolkit | For [Real-time analytics](/docs/get-started) services | | [timescaledb](https://github.com/timescale/timescaledb) | TimescaleDB | For all services | ## PostgreSQL built-in extensions | Extension | Description | Enabled by default | | -------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------- | ------------------ | | [autoinc](https://www.postgresql.org/docs/current/contrib-spi.html#CONTRIB-SPI-AUTOINC) | Functions for autoincrementing fields | - | | [amcheck](https://www.postgresql.org/docs/current/amcheck.html) | Functions for verifying relation integrity | - | | [bloom](https://www.postgresql.org/docs/current/bloom.html) | Bloom access method - signature file-based index | - | | [bool\_plperl](https://www.postgresql.org/docs/current/plperl-funcs.html) | Transform between bool and plperl | - | | [btree\_gin](https://www.postgresql.org/docs/current/btree-gin.html) | Support for indexing common datatypes in GIN | - | | [btree\_gist](https://www.postgresql.org/docs/current/btree-gist.html) | Support for indexing common datatypes in GiST | - | | [citext](https://www.postgresql.org/docs/current/citext.html) | Data type for case-insensitive character strings | - | | [cube](https://www.postgresql.org/docs/current/cube.html) | Data type for multidimensional cubes | - | | [dict\_int](https://www.postgresql.org/docs/current/dict-int.html) | Text search dictionary template for integers | - | | [dict\_xsyn](https://www.postgresql.org/docs/current/dict-xsyn.html) | Text search dictionary template for extended synonym processing | - | | [earthdistance](https://www.postgresql.org/docs/current/earthdistance.html) | Calculate great-circle distances on the surface of the Earth | - | | [fuzzystrmatch](https://www.postgresql.org/docs/current/fuzzystrmatch.html) | Determine similarities and distance between strings | - | | [hstore](https://www.postgresql.org/docs/current/hstore.html) | Data type for storing sets of (key, value) pairs | - | | [hstore\_plperl](https://www.postgresql.org/docs/current/hstore.html) | Transform between hstore and plperl | - | | [insert\_username](https://www.postgresql.org/docs/current/contrib-spi.html#CONTRIB-SPI-INSERT-USERNAME) | Functions for tracking who changed a table | - | | [intagg](https://www.postgresql.org/docs/current/intagg.html) | Integer aggregator and enumerator (obsolete) | - | | [intarray](https://www.postgresql.org/docs/current/intarray.html) | Functions, operators, and index support for 1-D arrays of integers | - | | [isn](https://www.postgresql.org/docs/current/isn.html) | Data types for international product numbering standards | - | | [jsonb\_plperl](https://www.postgresql.org/docs/current/datatype-json.html#DATATYPE-JSON-TRANSFORMS) | Transform between jsonb and plperl | - | | [lo](https://www.postgresql.org/docs/current/lo.html) | Large object maintenance | - | | [ltree](https://www.postgresql.org/docs/current/ltree.html) | Data type for hierarchical tree-like structures | - | | [moddatetime](https://www.postgresql.org/docs/current/contrib-spi.html#CONTRIB-SPI-MODDATETIME) | Functions for tracking last modification time | - | | [old\_snapshot](https://www.postgresql.org/docs/16/oldsnapshot.html) | Utilities in support of `old_snapshot_threshold` | - | | [pgcrypto](../tiger-cloud-extensions/pgcrypto) | Cryptographic functions | - | | [pgrowlocks](https://www.postgresql.org/docs/current/pgrowlocks.html) | Show row-level locking information | - | | [pgstattuple](https://www.postgresql.org/docs/current/pgstattuple.html) | Obtain tuple-level statistics | - | | [pg\_freespacemap](https://www.postgresql.org/docs/current/pgfreespacemap.html) | Examine the free space map (FSM) | - | | [pg\_prewarm](https://www.postgresql.org/docs/current/pgprewarm.html) | Prewarm relation data | - | | [pg\_stat\_statements](https://www.postgresql.org/docs/current/pgstatstatements.html) | Track execution statistics of all SQL statements executed | For all services | | [pg\_trgm](https://www.postgresql.org/docs/current/pgtrgm.html) | Text similarity measurement and index searching based on trigrams | - | | [pg\_visibility](https://www.postgresql.org/docs/current/pgvisibility.html) | Examine the visibility map (VM) and page-level visibility info | - | | [plperl](https://www.postgresql.org/docs/current/plperl.html) | PL/Perl procedural language | - | | [plpgsql](https://www.postgresql.org/docs/current/plpgsql.html) | SQL procedural language | For all services | | [postgres\_fdw](https://www.postgresql.org/docs/current/postgres-fdw.html) | Foreign data wrappers | For all services | | [refint](https://www.postgresql.org/docs/current/contrib-spi.html) | Functions for implementing referential integrity (obsolete) | - | | [seg](https://www.postgresql.org/docs/current/seg.html) | Data type for representing line segments or floating-point intervals | - | | [sslinfo](https://www.postgresql.org/docs/current/sslinfo.html) | Information about SSL certificates | - | | [tablefunc](https://www.postgresql.org/docs/current/tablefunc.html) | Functions that manipulate whole tables, including crosstab | - | | [tcn](https://www.postgresql.org/docs/current/tcn.html) | Trigger change notifications | - | | [tsm\_system\_rows](https://www.postgresql.org/docs/current/tsm-system-rows.html) | `TABLESAMPLE` method which accepts the number of rows as a limit | - | | [tsm\_system\_time](https://www.postgresql.org/docs/current/tsm-system-time.html) | `TABLESAMPLE` method which accepts the time in milliseconds as a limit | - | | [unaccent](https://www.postgresql.org/docs/current/unaccent.html) | Text search dictionary that removes accents | - | | [uuid-ossp](https://www.postgresql.org/docs/current/uuid-ossp.html) | Generate universally unique identifiers (UUIDs) | - | ## Third-party extensions | Extension | Description | Enabled by default | | ------------------------------------------------------------------------------- | ----------------------------------------------------------------------- | ---------------------------------------------------- | | [h3](https://pgxn.org/dist/h3/) | H3 bindings for PostgreSQL | - | | [pgaudit](https://www.pgaudit.org/) | Detailed session and/or object audit logging | - | | [pgpcre](https://github.com/petere/pgpcre) | Perl-compatible RegEx | - | | [pg\_cron](https://github.com/citusdata/pg_cron) | SQL commands that you can schedule and run directly inside the database | [Contact us](mailto:support@tigerdata.com) to enable | | [pg\_repack](https://github.com/reorg/pg_repack) | Table reorganization in PostgreSQL with minimal locks | - | | [pgrouting](https://pgrouting.org/) | Geospatial routing functionality | - | | [postgis](../tiger-cloud-extensions/postgis) | PostGIS geometry and geography spatial types and functions | - | | [postgis\_raster](https://postgis.net/docs/RT_reference.html) | PostGIS raster types and functions | - | | [postgis\_sfcgal](https://postgis.net/docs/reference_sfcgal.html) | PostGIS SFCGAL functions | - | | [postgis\_tiger\_geocoder](https://postgis.net/docs/Extras.html#Tiger_Geocoder) | PostGIS Tiger Cloud geocoder and reverse geocoder | - | | [postgis\_topology](https://postgis.net/workshops/postgis-intro/topology.html) | PostGIS topology spatial types and functions | - | | [unit](https://github.com/df7cb/postgresql-unit) | SI units for PostgreSQL | - | # Optimize full text search with BM25 > Install and tune pg_textsearch (v1.1.0) for BM25 full-text search on Tiger Cloud and self-hosted deployments Note **pg\_textsearch v1.1.0** is **production-ready** on Tiger Cloud. See the [v1.1.0 release notes](https://github.com/timescale/pg_textsearch/releases/tag/v1.1.0). `pg_textsearch` supports **PostgreSQL 17 and 18**. For self-hosted installs, confirm versions against the [upstream compatibility table](https://github.com/timescale/pg_textsearch#postgresql-version-compatibility). PostgreSQL full-text search at scale consistently hits a wall where performance degrades catastrophically. Tiger Data's [pg\_textsearch](https://github.com/timescale/pg_textsearch) brings modern [BM25](https://en.wikipedia.org/wiki/Okapi_BM25)-based full-text search directly into PostgreSQL, with a memtable architecture for efficient indexing and ranking. `pg_textsearch` integrates seamlessly with SQL and provides better search quality and performance than the PostgreSQL built-in full-text search. With Block-Max WAND optimization, `pg_textsearch` delivers up to **4x faster top-k queries** compared to native BM25 implementations. Parallel index builds reduce indexing times by **4x or more** for large tables. Advanced compression using delta encoding and bitpacking reduces index sizes by **41%** while improving query performance by 10-20% for shorter queries. BM25 scores in `pg_textsearch` are returned as negative values, where lower (more negative) numbers indicate better matches. `pg_textsearch` implements the following: * **Corpus-aware ranking**: BM25 uses inverse document frequency to weight rare terms higher * **Term frequency saturation**: prevents documents with excessive term repetition from dominating results * **Length normalization**: adjusts scores based on document length relative to corpus average * **Relative ranking**: focuses on rank order rather than absolute score values This page shows you how to install `pg_textsearch`, configure BM25 indexes, and optimize your search capabilities using the following best practices: * **Parallel indexing**: enable parallel workers for faster index creation on large tables * **Language configuration**: choose appropriate text search configurations for your data language * **Hybrid search**: combine with pgvector or pgvectorscale for applications requiring both semantic and keyword search * **Query optimization**: use score thresholds to filter low-relevance results * **Index monitoring**: regularly check index usage and memory consumption ## Prerequisites for this procedure To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service). These steps use Tiger Cloud, but the same approach applies to a [self-hosted TimescaleDB](/docs/get-started/choose-your-path/install-timescaledb) instance. - Your [connection details](/docs/integrate/find-connection-details). ## Install pg\_textsearch To install this PostgreSQL extension: 1. **Connect to your Tiger Cloud service** In [Tiger Console](https://console.cloud.tigerdata.com/dashboard/services) open an [SQL editor](/docs/build/data-management/run-queries-from-tiger-console). You can also connect to your service using [psql](/docs/integrate/query-administration/psql#connect-to-your-service). 2. **Enable the extension on your Tiger Cloud service** * For new services, simply enable the extension: ```sql CREATE EXTENSION pg_textsearch; ``` * For existing services, update your instance, then enable the extension: The extension may not be available until after your next scheduled maintenance window. To pick up the update immediately, manually pause and restart your service. 3. **Verify the installation** ```sql SELECT * FROM pg_extension WHERE extname = 'pg_textsearch'; ``` You have installed `pg_textsearch` on Tiger Cloud. ## Self-hosted: load `pg_textsearch` before `CREATE EXTENSION` On self-managed PostgreSQL, the extension must be **preloaded** so the server loads it at startup: 1. Set `shared_preload_libraries` in `postgresql.conf` (append `pg_textsearch` to any existing list): ```ini shared_preload_libraries = 'pg_textsearch' ``` 2. Restart PostgreSQL. 3. In each database where you need search, run `CREATE EXTENSION pg_textsearch;`. Tiger Cloud manages preloading for you; you typically only run `CREATE EXTENSION` in the SQL editor or client. ## Create BM25 indexes on your data BM25 indexes provide modern relevance ranking that outperforms PostgreSQL's built-in ts\_rank functions by using corpus statistics and better algorithmic design. To create a BM25 index with `pg_textsearch`: 1. **Create a table with text content** ```sql CREATE TABLE products ( id serial PRIMARY KEY, name text, description text, category text, price numeric ); ``` 2. **Insert sample data** ```sql INSERT INTO products (name, description, category, price) VALUES ('Mechanical Keyboard', 'Durable mechanical switches with RGB backlighting for gaming and productivity', 'Electronics', 149.99), ('Ergonomic Mouse', 'Wireless mouse with ergonomic design to reduce wrist strain during long work sessions', 'Electronics', 79.99), ('Standing Desk', 'Adjustable height desk for better posture and productivity throughout the workday', 'Furniture', 599.99); ``` 3. **Create a BM25 index** ```sql CREATE INDEX products_search_idx ON products USING bm25(description) WITH (text_config='english'); ``` Each BM25 index covers a single text expression. That expression can be a plain column, or a computed expression that concatenates columns, extracts a JSONB field, or normalizes text. See [Index expressions, subsets of rows, and arrays](#index-expressions-subsets-of-rows-and-arrays). For optimal performance, load your data first, then create the index. You have created a BM25 index for full-text search. ## Index expressions, subsets of rows, and arrays Since [v1.1.0](https://github.com/timescale/pg_textsearch/releases/tag/v1.1.0) A BM25 index can cover more than a plain column. You can index a computed expression, restrict the index to a subset of rows, or index a `text[]` array. ### Index an expression Index a computed expression to search JSONB fields, combine multiple columns, or normalize text. The expression must return `text` and use only `IMMUTABLE` functions, and your queries must repeat the same expression in `ORDER BY`: ```sql -- Extract a JSONB field CREATE INDEX events_msg_idx ON events USING bm25 ((data->>'description')) WITH (text_config='english'); -- Combine title and body into one searchable index CREATE INDEX articles_idx ON articles USING bm25 ((coalesce(title, '') || ' ' || coalesce(body, ''))) WITH (text_config='english'); -- Normalize case before indexing CREATE INDEX docs_lower_idx ON docs USING bm25 ((lower(content))) WITH (text_config='simple'); ``` ### Index a subset of rows Add a `WHERE` clause to index only the rows you search. Partial indexes are smaller and faster when queries always target the same subset: ```sql CREATE INDEX docs_published_idx ON docs USING bm25 (content) WITH (text_config='english') WHERE status = 'published'; ``` Partial indexes require explicit index naming with `to_bm25query()`. The implicit `content <@> 'query'` syntax skips them: ```sql SELECT * FROM docs WHERE status = 'published' ORDER BY content <@> to_bm25query('search terms', 'docs_published_idx') LIMIT 10; ``` You can combine expression and partial indexes: ```sql CREATE INDEX events_error_idx ON events USING bm25 ((data->>'message')) WITH (text_config='english') WHERE (data->>'severity') = 'error'; ``` ### Search multilingual tables For tables that hold documents in several languages, create one partial index per language, each with the matching text search configuration. Each index then applies language-appropriate stemming and stop words: ```sql CREATE INDEX docs_en_idx ON docs USING bm25 (content) WITH (text_config='english') WHERE lang = 'en'; CREATE INDEX docs_de_idx ON docs USING bm25 (content) WITH (text_config='german') WHERE lang = 'de'; CREATE INDEX docs_fr_idx ON docs USING bm25 (content) WITH (text_config='french') WHERE lang = 'fr'; ``` Query with the matching predicate and index name: ```sql SELECT * FROM docs WHERE lang = 'en' ORDER BY content <@> to_bm25query('databases', 'docs_en_idx') LIMIT 10; ``` ### Index a text array `pg_textsearch` indexes `text[]` columns by scoring the array elements as a single flattened document. This is useful when you store a document as pre-split chunks: ```sql CREATE TABLE chunked_docs (id bigserial PRIMARY KEY, content text[]); CREATE INDEX chunked_docs_idx ON chunked_docs USING bm25(content) WITH (text_config='english'); ``` BM25 scores match what you would get by concatenating the elements into a single `text` value. ## Enable parallel indexing for faster index creation Since [v0.5.0](https://github.com/timescale/pg_textsearch/releases/tag/v0.5.0) `pg_textsearch` supports parallel index builds that can significantly reduce indexing times for large tables. PostgreSQL automatically uses parallel workers based on table size and available resources. 1. **Configure parallel workers (optional)** PostgreSQL uses server defaults, but you can adjust settings for your workload: ```sql -- Set number of parallel workers (uses CPU count by default) SET max_parallel_maintenance_workers = 4; -- Set memory for index builds (must be at least 64MB for parallel builds) SET maintenance_work_mem = '256MB'; ``` Note The planner requires `maintenance_work_mem >= 64MB` to enable parallel index builds. With insufficient memory, builds fall back to serial mode silently. 2. **Create index (parallel workers used automatically for large tables)** ```sql CREATE INDEX products_search_idx ON products USING bm25(description) WITH (text_config='english'); ``` When parallel build is used, you see a notice: ```plaintext NOTICE: parallel index build: launched 4 of 4 requested workers ``` 3. **Verify parallel execution in partitioned tables** For partitioned tables, each partition builds its index independently with parallel workers if the partition is large enough. This allows efficient indexing of very large partitioned datasets. You have configured parallel index builds for faster indexing. ## Merge BM25 segments after bulk loads After large batch inserts or sustained incremental writes, multiple on-disk **segments** can accumulate. Merging them into a single segment improves query speed by reducing how many segments a query must scan. This is analogous to Lucene’s `forceMerge(1)`. Use `bm25_force_merge()` **after bulk loads**, not during steady high write traffic: ```sql SELECT bm25_force_merge('products_search_idx'); ``` ## Optimize search queries for performance The `<@>` operator returns BM25-based scores as **negative** values: **lower (more negative) means a better match**. ### Ranked search with implicit query syntax (primary pattern) For typical top-k search, pass the query as a **string literal**. In `ORDER BY`, PostgreSQL can **automatically detect** the BM25 index from the indexed column: ```sql SELECT name, description, description <@> 'ergonomic work' AS score FROM products ORDER BY score LIMIT 3; ``` You see something like: ```sql name | description | score ----------------------------+-----------------------------------------------------------------------------------+--------------------- Ergonomic Mouse | Wireless mouse with ergonomic design to reduce wrist strain during long work sessions | -1.8132977485656738 Mechanical Keyboard | Durable mechanical switches with RGB backlighting for gaming and productivity | 0 Standing Desk | Adjustable height desk for better posture and productivity throughout the workday | 0 ``` ### The `bm25query` type and `to_bm25query()` Use an explicit **`bm25query`** value when you need the index name in the expression (for example in `WHERE`, or when the planner cannot infer the index from context): | Form | Use | | ------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------- | | `to_bm25query('query text')` | Query text only; use in **`ORDER BY`** with an index scan (implicit index detection from the column). | | `to_bm25query('query text', 'index_name')` | Query text **and** index name; required for **`WHERE`** filters and whenever you must name the index. | | `'index_name:query text'::bm25query` | Cast form (embedded index name). In PostgreSQL 18, the single-colon form helps the planner when expressions are evaluated early. | | `text <@> bm25query` → `double precision` | BM25 score (negative is better). | | `bm25query = bm25query` | Equality comparison. | ### Filter results by score threshold For `WHERE` clauses, use **`to_bm25query()`** with an explicit index name: ```sql SELECT name, description <@> to_bm25query('wireless', 'products_search_idx') AS score FROM products WHERE description <@> to_bm25query('wireless', 'products_search_idx') < -0.5; ``` You see something like: ```sql name | score ----------------+--------------------- Ergonomic Mouse | -0.9066488742828369 ``` ### Combine with standard SQL operations ```sql SELECT category, name, description <@> to_bm25query('ergonomic', 'products_search_idx') AS score FROM products WHERE price < 500 AND description <@> to_bm25query('ergonomic', 'products_search_idx') < -0.5 ORDER BY score LIMIT 5; ``` You see something like: ```sql category | name | score -------------+-----------------+--------------------- Electronics | Ergonomic Mouse | -0.9066488742828369 ``` ### Pre-filtering vs post-filtering How you combine BM25 with other predicates affects performance: * **Pre-filtering**: use another index (for example B-tree on `category_id`) so PostgreSQL restricts rows **before** BM25 scoring. Best when the filter is **selective** (for example matches a small fraction of rows); then `ORDER BY ... LIMIT` scores a smaller candidate set. * **Post-filtering**: the BM25 index scan produces top-k matches **first**; other `WHERE` conditions apply **after**. If the filter removes most of those rows, you can get **fewer rows than `LIMIT`**. Increase the inner `LIMIT` and re-limit in application code if needed. This mirrors tradeoffs familiar from approximate vector indexes: know whether your filter is shrinking the universe of rows before or after ranking. ### Verify plans with EXPLAIN (and sequential scans on small tables) ```sql EXPLAIN SELECT * FROM products ORDER BY description <@> 'ergonomic' LIMIT 5; ``` On **small** tables, the planner may choose a **sequential scan** instead of the BM25 index. To force index use while testing, you can run: ```sql SET enable_seqscan = off; ``` Note Even when `EXPLAIN` shows a sequential scan, `<@>` and `to_bm25query()` still use the BM25 index for **corpus statistics** (document counts, average length, and related metadata) required to score results. You have optimized your search queries for BM25 ranking. ## Build hybrid search with semantic and keyword search Combine `pg_textsearch` with `pgvector` or `pgvectorscale` to build powerful hybrid search systems that use both semantic vector search and keyword BM25 search. 1. **Enable the vectorscale extension on your Tiger Cloud service** ```sql CREATE EXTENSION IF NOT EXISTS vectorscale CASCADE; ``` 2. **Create a table with both text content and vector embeddings** ```sql CREATE TABLE articles ( id serial PRIMARY KEY, title text, content text, embedding vector(3) -- Using 3 dimensions for this example; use 1536 for OpenAI ada-002 ); ``` 3. **Insert sample data** ```sql INSERT INTO articles (title, content, embedding) VALUES ('Database Query Optimization', 'Learn how to optimize database query performance using indexes and query planning', '[0.1, 0.15, 0.2]'), ('Performance Tuning Guide', 'A comprehensive guide to performance tuning in distributed systems and databases', '[0.12, 0.18, 0.25]'), ('Introduction to Indexing', 'Understanding how database indexes improve query performance and data retrieval', '[0.09, 0.14, 0.19]'), ('Advanced SQL Techniques', 'Master advanced SQL techniques for complex data analysis and reporting', '[0.5, 0.6, 0.7]'), ('Data Warehousing Basics', 'Getting started with data warehousing and analytical query processing', '[0.8, 0.9, 0.85]'); ``` 4. **Create indexes for both search types** ```sql -- Vector index for semantic search CREATE INDEX articles_embedding_idx ON articles USING hnsw (embedding vector_cosine_ops); -- Keyword index for BM25 search CREATE INDEX articles_content_idx ON articles USING bm25(content) WITH (text_config='english'); ``` 5. **Perform hybrid search using reciprocal rank fusion** ```sql WITH vector_search AS ( SELECT id, ROW_NUMBER() OVER (ORDER BY embedding <=> '[0.1, 0.2, 0.3]'::vector) AS rank FROM articles ORDER BY embedding <=> '[0.1, 0.2, 0.3]'::vector LIMIT 20 ), keyword_search AS ( SELECT id, ROW_NUMBER() OVER (ORDER BY content <@> 'query performance') AS rank FROM articles ORDER BY content <@> 'query performance' LIMIT 20 ) SELECT a.id, a.title, COALESCE(1.0 / (60 + v.rank), 0.0) + COALESCE(1.0 / (60 + k.rank), 0.0) AS combined_score FROM articles a LEFT JOIN vector_search v ON a.id = v.id LEFT JOIN keyword_search k ON a.id = k.id WHERE v.id IS NOT NULL OR k.id IS NOT NULL ORDER BY combined_score DESC LIMIT 10; ``` You see something like: ```sql id | title | combined_score ----+----------------------------+-------------------- 3 | Introduction to Indexing | 0.0325224748810153 1 | Database Query Optimization| 0.0322664584959667 2 | Performance Tuning Guide | 0.0320020481310804 5 | Data Warehousing Basics | 0.0310096153846154 4 | Advanced SQL Techniques | 0.0310096153846154 ``` 6. **Adjust relative weights for different search types** ```sql WITH vector_search AS ( SELECT id, ROW_NUMBER() OVER (ORDER BY embedding <=> '[0.1, 0.2, 0.3]'::vector) AS rank FROM articles ORDER BY embedding <=> '[0.1, 0.2, 0.3]'::vector LIMIT 20 ), keyword_search AS ( SELECT id, ROW_NUMBER() OVER (ORDER BY content <@> 'query performance') AS rank FROM articles ORDER BY content <@> 'query performance' LIMIT 20 ) SELECT a.id, a.title, 0.7 * COALESCE(1.0 / (60 + v.rank), 0.0) + -- 70% weight to vectors 0.3 * COALESCE(1.0 / (60 + k.rank), 0.0) -- 30% weight to keywords AS combined_score FROM articles a LEFT JOIN vector_search v ON a.id = v.id LEFT JOIN keyword_search k ON a.id = k.id WHERE v.id IS NOT NULL OR k.id IS NOT NULL ORDER BY combined_score DESC LIMIT 10; ``` You see something like: ```sql id | title | combined_score ----+----------------------------+-------------------- 3 | Introduction to Indexing | 0.0163141195134849 2 | Performance Tuning Guide | 0.0160522273425499 1 | Database Query Optimization| 0.0160291438979964 4 | Advanced SQL Techniques | 0.0155528846153846 5 | Data Warehousing Basics | 0.0154567307692308 ``` You have implemented hybrid search combining semantic and keyword search. ## Configuration options Customize `pg_textsearch` behavior for your specific use case and data characteristics. 1. **Configure memory and performance settings** `pg_textsearch` keeps in-memory inverted indexes (memtables) in PostgreSQL dynamic shared memory. `memory_limit` bounds their growth: as usage approaches the cap, memtables spill to on-disk segments automatically, and if usage still exceeds the cap, inserts fail with an error rather than risking an out-of-memory kill. Tips **Crash recovery:** the memtable is rebuilt from the heap on startup, so you do **not** lose indexed data if PostgreSQL crashes before a spill to disk completes. Two settings can be changed in any session, by any role: ```sql -- Default query limit when no LIMIT clause is present (default 1000) SET pg_textsearch.default_limit = 5000; -- Segment compression with delta encoding and bitpacking (enabled by default). -- Reduces index size by ~41% with a 10-20% query performance improvement for shorter queries. SET pg_textsearch.compress_segments = on; ``` Since [v0.4.0](https://github.com/timescale/pg_textsearch/releases/tag/v0.4.0) The remaining settings are **server-level** or **superuser-only**, so a plain `SET` in a regular session does not apply them: | Setting | Default | Notes | | ----------------------------------- | -------- | ---------------------------------------------------------------------------------------- | | `pg_textsearch.memory_limit` | `2GB` | Server-level; takes effect on a configuration reload. A value of `0` disables the limit. | | `pg_textsearch.bulk_load_threshold` | `100000` | Superuser. Terms per transaction before the memtable spills to disk. | | `pg_textsearch.segments_per_level` | `8` | Superuser. Segments per level before automatic compaction (range 2–64). | | `pg_textsearch.log_bmw_stats` | `off` | Superuser. Logs Block-Max WAND block-skip statistics. | | `pg_textsearch.log_scores` | `off` | Superuser. Logs BM25 scores during index scans. | * On Tiger Cloud, change these from the Tiger Console: open your service, then `Operations` > `Database parameters`. See [Configure database parameters](/docs/deploy/tiger-cloud/tiger-cloud-aws/configuration/customize-configuration). The `tsdbadmin` role cannot apply them with `SET` or `ALTER SYSTEM`. * On self-hosted PostgreSQL, set `memory_limit` in `postgresql.conf`, or run `ALTER SYSTEM SET pg_textsearch.memory_limit = '2GB';` followed by `SELECT pg_reload_conf();`. As a superuser, you can change the superuser-only settings with `SET` in your session. Note `memory_limit` is applied as three tiers derived from the single value: a per-index soft limit (`memory_limit / 8`) spills that index's memtable to disk, a global soft limit (`memory_limit / 2`) evicts the largest memtable across all indexes, and the hard limit (`memory_limit`) refuses further cache growth so an insert fails rather than risking the out-of-memory killer. You only set `memory_limit`; the ratios are derived automatically. Since [v1.1.0](https://github.com/timescale/pg_textsearch/releases/tag/v1.1.0) 2. **Configure language-specific text processing** You can create multiple BM25 indexes on the same column with different language configurations: ```sql -- Create an additional index with simple tokenization (no stemming) CREATE INDEX products_simple_idx ON products USING bm25(description) WITH (text_config='simple'); -- Example: French language configuration for a French products table -- CREATE INDEX products_fr_idx ON products_fr -- USING bm25(description) -- WITH (text_config='french'); ``` 3. **Tune BM25 parameters** ```sql -- Adjust term frequency saturation (k1) and length normalization (b) CREATE INDEX products_custom_idx ON products USING bm25(description) WITH (text_config='english', k1=1.5, b=0.8); ``` 4. **Monitor index usage and memory consumption** * Check index usage statistics for every BM25 index ```sql SELECT s.schemaname, s.relname, s.indexrelname, s.idx_scan, s.idx_tup_read FROM pg_stat_user_indexes s JOIN pg_class c ON c.oid = s.indexrelid JOIN pg_am a ON a.oid = c.relam WHERE a.amname = 'bm25'; ``` * Check shared memory used by memtables across all indexes ```sql SELECT * FROM bm25_memory_usage(); ``` * View index summary with corpus statistics and memory usage (**superuser**) ```sql SELECT bm25_summarize_index('products_search_idx'); ``` * View detailed index structure (output is truncated for display) (**superuser**) ```sql SELECT bm25_dump_index('products_search_idx'); ``` Note `bm25_summarize_index` and `bm25_dump_index` require superuser. On Tiger Cloud the `tsdbadmin` role is not a superuser, so these two functions are unavailable there; use `bm25_memory_usage()` and `pg_stat_user_indexes` to monitor instead. The two-argument form `bm25_dump_index('index_name', '/path/to/file.txt')` exists only in **debug builds** compiled with `-DDEBUG_DUMP_INDEX`. Standard packages expose the single-argument form only. * Force memtable spill to disk (useful for testing or memory management) ```sql SELECT bm25_spill_index('products_search_idx'); ``` You have configured `pg_textsearch` for optimal performance. For production applications, consider implementing result caching and pagination to improve user experience with large result sets. ## Current limitations * **No phrase search**: you cannot search for exact multi-word phrases (no positional index; emulate with BM25 plus a post-filter if needed). * **No compressed data support**: `pg_textsearch` does not work with **compressed** chunk or table data (for example Hypercore-compressed data where the extension applies). * **One expression per index**: each BM25 index covers a **single text expression**. To search several columns at once, index a computed expression that concatenates them (see [Index expressions, subsets of rows, and arrays](#index-expressions-subsets-of-rows-and-arrays)). * **No built-in faceted search**: there is no dedicated faceting operator; use ordinary PostgreSQL patterns (`WHERE`, `GROUP BY`, counts) alongside BM25 filters. * **Insert and update throughput**: the memtable design supports incremental writes, but **sustained write-heavy** workloads are still a focus of ongoing performance work. Prefer **bulk load, then index** (or `bm25_force_merge` after large batches) when possible. * **No background compaction**: segment compaction runs **synchronously** during memtable spill; write-heavy workloads may see latency spikes during spills. * **Partitioned table statistics**: BM25 statistics are **per partition**. Scores are accurate within a partition but are **not directly comparable** across partitions when a query spans many partitions (different IDF and length stats). Prefer queries that target a single partition when rank comparability matters. * **Word length limit**: tokenization inherits PostgreSQL’s `tsvector` **2047 character** word limit; very long tokens (for example base64 blobs) may be ignored with an `INFO` message. * **PL/pgSQL and stored procedures**: the implicit form `text <@> 'query'` relies on planner hooks that **do not run** inside PL/pgSQL `DO` blocks, functions, or procedures. Use **`to_bm25query('query', 'index_name')`** explicitly in those contexts. For more detail and workarounds, see the [limitations section](https://github.com/timescale/pg_textsearch#limitations) in the upstream README. # Encrypt data using pgcrypto > Encrypt sensitive data in your database using the pgcrypto extension The `pgcrypto` PostgreSQL extension provides cryptographic functions such as: * General hashing * Password hashing * PGP encryption * Raw encryption * Random-data For more information about these functions and the options available, see the [pgcrypto documentation](https://www.postgresql.org/docs/current/pgcrypto.html). ## Use the `pgcrypto` extension to encrypt inserted data The `pgcrypto` extension allows you to encrypt, decrypt, hash, and create digital signatures within your database. Tiger Data understands how precious your data is and safeguards sensitive information. 1. **Install the `pgcrypto` extension** ```sql CREATE EXTENSION IF NOT EXISTS pgcrypto; ``` Confirm the extension is installed using the `\dx` command: ```sql List of installed extensions Name | Version | Schema | Description ---------------------+---------+------------+--------------------------------------------------------------------------------------- pg_stat_statements | 1.10 | public | track planning and execution statistics of all SQL statements executed pgcrypto | 1.3 | public | cryptographic functions plpgsql | 1.0 | pg_catalog | PL/pgSQL procedural language timescaledb | 2.24.0 | public | Enables scalable inserts and complex queries for time-series data (Community Edition) timescaledb_toolkit | 1.22.0 | public | Library of analytical hyperfunctions, time-series pipelining, and other SQL utilities ``` 2. **Create a table named `user_passwords`** ```sql CREATE TABLE user_passwords (username varchar(100) PRIMARY KEY, crypttext text); ``` 3. **Insert values and replace `` with a password key of your choice** ```sql INSERT INTO tbl_sym_crypt (username, crypttext) VALUES ('user1', pgp_sym_encrypt('user1_password','')), ('user2', pgp_sym_encrypt('user2_password','')); ``` 4. **Confirm the password is encrypted** ```sql SELECT * FROM user_passwords; ``` The encrypted passwords are listed: ```sql username | crypttext ----------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------- user1 | \xc30d040703025caa37f9d1c731d169d240018529d6f0002b2948905a87e4787efaa0046e58fd3f04ee95594bea1803807063321f62c9651cbf0422b04508093df9644a76684b504b317cf633552fcf164f user2 | \xc30d0407030279bbcf760b81d3de73d23c01c04142632fc8527c0c1b17cc954c77f16df46022acddc565fd18f0f0f761ddb2f31b21c4ebe47a48039d685287d64506029e027cf29b5493b574df (2 rows) ``` 5. **View the decrypted passwords** Replace `` with the password key that you created: ```sql SELECT username, pgp_sym_decrypt(crypttext::bytea, '') FROM user_passwords; ``` The decrypted passwords are listed: ```sql username | pgp_sym_decrypt ----------+----------------- user1 | user1_password user2 | user2_password (2 rows) ``` # Create a chatbot using pgvector > Build a chatbot with vector similarity search using the pgvector extension The `pgvector` PostgreSQL extension helps you to store and search over machine learning-generated embeddings. It provides different capabilities that allows you to identify both exact and approximate nearest neighbors. It is designed to work seamlessly with other PostgreSQL features, including indexing and querying. For more information about these functions and the options available, see the [pgvector](https://github.com/pgvector/pgvector/blob/master/README.md) repository. ## Use the `pgvector` extension to create a `chatbot` The `pgvector` PostgreSQL extension allows you to create, store, and query OpenAI [vector embeddings](https://platform.openai.com/docs/guides/embeddings/what-are-embeddings) in a PostgreSQL database instance. This page shows you how to use [retrieval augmented generation (RAG)](https://www.promptingguide.ai/techniques/rag) to create a chatbot that combines your data with ChatGPT using OpenAI and `pgvector`. RAG provides a solution to the problem that a foundational model such as GPT-3 or GPT-4 could be missing some information needed to give a good answer, because that information was not in the dataset used to train the model. This can happen if the information is stored in private documents or only became available recently. In this example, you create embeddings, insert the embeddings into a Tiger Cloud service and query the embeddings using `pgvector`. The content for the embeddings is from the Tiger Data blog, specifically from the [Developer Q\&A](https://www.tigerdata.com/blog/tag/dev-q-a) section, which features posts by Tiger Data users talking about their real-world use cases. ## Prerequisites for this procedure To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service). - Your [connection details](/docs/integrate/find-connection-details). * Python installed. * The [pgvector repository](https://github.com/timescale/vector-cookbook/tree/main/openai_pgvector_helloworld) cloned. * An [OpenAI developer account](https://platform.openai.com/overview). * An OpenAI [API key](https://platform.openai.com/account/api-keys). Note If you are on a free plan there may be rate limiting for your API requests. 1. **Create and activate a Python virtual environment** ```bash virtualenv pgvectorenv source pgvectorenv/bin/activate ``` 2. **Set the environment variables** Set `OPENAI_API_KEY` and `TIMESCALE_CONNECTION_STRING`. In this example, to set the environment variables in macOS, open the `zshrc` profile. Replace ``, and `` with your OpenAI API key and the URL of your Tiger Cloud service: ```bash nano ~/.zshrc export OPENAI_API_KEY='' export TIMESCALE_CONNECTION_STRING='' ``` Update the shell with the new variables using `source ~/.zshrc` 3. **Confirm that you have set the environment variables** ```bash echo $OPENAI_API_KEY echo $TIMESCALE_CONNECTION_STRING ``` 4. **Install the required modules and packages** Using the `requirements.txt` file located in the `vector-cookbook\openai_pgvector_helloworld` directory: ```bash pip install -r requirements.txt ``` 5. **Create the `create_embeddings.py` file** To create embeddings for your data using the OpenAI API, open an editor of your choice and create the `create_embeddings.py` file. create\_embeddings.py ```python ############################################################################### # This script creates OpenAI embedding vectors for content in a CSV file # and saves the results to a new CSV file with the embeddings included ############################################################################### import openai import os import pandas as pd import numpy as np import json import tiktoken # Get openAI api key by reading local .env file from dotenv import load_dotenv, find_dotenv _ = load_dotenv(find_dotenv()) openai.api_key = os.environ['OPENAI_API_KEY'] # Load your CSV file into a pandas DataFrame df = pd.read_csv('blog_posts_data.csv') df.head() ############################################################################### # Helper functions to help us create the embeddings ############################################################################### # Calculate number of tokens for a string def num_tokens_from_string(string: str, encoding_name = "cl100k_base") -> int: if not string: return 0 # Returns the number of tokens in a text string encoding = tiktoken.get_encoding(encoding_name) num_tokens = len(encoding.encode(string)) return num_tokens # Calculate cost of embedding num_tokens # Assumes we're using the text-embedding-ada-002 model # See https://openai.com/pricing def get_embedding_cost(num_tokens): return num_tokens/1000*0.0001 # Calculate total cost of embedding all content in the dataframe def get_total_embeddings_cost(): total_tokens = 0 for i in range(len(df.index)): text = df['content'][i] token_len = num_tokens_from_string(text) total_tokens = total_tokens + token_len total_cost = get_embedding_cost(total_tokens) return total_cost ############################################################################### # quick check on total token amount for price estimation total_cost = get_total_embeddings_cost() print("Estimated price to embed this content = $" + str(total_cost)) ############################################################################### # Create new list with small content chunks to not hit max token limits # Note: the maximum number of tokens for a single request is 8191 # https://openai.com/docs/api-reference/requests ############################################################################### # list for chunked content and embeddings new_list = [] # Split up the text into token sizes of around 512 tokens for i in range(len(df.index)): text = df['content'][i] token_len = num_tokens_from_string(text) if token_len <= 512: new_list.append([df['title'][i], df['content'][i], df['url'][i], token_len]) else: # add content to the new list in chunks start = 0 ideal_token_size = 512 # 1 token ~ 3/4 of a word ideal_size = int(ideal_token_size // (4/3)) end = ideal_size #split text by spaces into words words = text.split() #remove empty spaces words = [x for x in words if x != ' '] total_words = len(words) #calculate iterations chunks = total_words // ideal_size if total_words % ideal_size != 0: chunks += 1 new_content = [] for j in range(chunks): if end > total_words: end = total_words new_content = words[start:end] new_content_string = ' '.join(new_content) new_content_token_len = num_tokens_from_string(new_content_string) if new_content_token_len > 0: new_list.append([df['title'][i], new_content_string, df['url'][i], new_content_token_len]) start += ideal_size end += ideal_size # Helper function: get embeddings for a text def get_embeddings(text): response = openai.Embedding.create( model="text-embedding-ada-002", input = text.replace("\n"," ") ) embedding = response['data'][0]['embedding'] return embedding # Create embeddings for each piece of content for i in range(len(new_list)): text = new_list[i][1] embedding = get_embeddings(text) new_list[i].append(embedding) # Create a new dataframe from the list df_new = pd.DataFrame(new_list, columns=['title', 'content', 'url', 'tokens', 'embeddings']) df_new.head() # Save the dataframe with embeddings as a CSV file for later use df_new.to_csv('blog_data_and_embeddings.csv', index=False) print("Done! Check the file blog_data_and_embeddings.csv for your results.") ``` 6. **Run `create_embeddings.py`** ```bash python create_embeddings.py ``` You should see an output that looks a bit like this: ```bash Estimated price to embed this content = $0.0060178 Done! Check the file blog_data_and_embeddings.csv for your results. ``` 7. **Create the `insert_embeddings.py` file** To insert these embeddings into your Tiger Cloud service using the `pgvector` extension, open an editor of your choice and create the `insert_embeddings.py` file. insert\_embeddings.py ```python ############################################################################### # This script inserts OpenAI embedding vectors into a Postgres database # using pgvector, a Postgres extension for vector similarity search ############################################################################### import openai import os import pandas as pd import numpy as np import psycopg2 import ast import pgvector import math from psycopg2.extras import execute_values from pgvector.psycopg2 import register_vector ############################################################################### # Setup your database to insert embeddings ############################################################################### # Get a Tiger Cloud / Postgres database connection string by reading local .env file connection_string = os.environ['TIMESCALE_CONNECTION_STRING'] # Connect to Postgres database in Tiger Cloud using connection string conn = psycopg2.connect(connection_string) cur = conn.cursor() #install pgvector in your database cur.execute("CREATE EXTENSION IF NOT EXISTS vector;"); conn.commit() # Register the vector type with psycopg2 register_vector(conn) # Create table to store embeddings and metadata table_create_command = """ CREATE TABLE embeddings ( id bigserial primary key, title text, url text, content text, tokens integer, embedding vector(1536) ); """ cur.execute(table_create_command) cur.close() conn.commit() ############################################################################### # Import embeddings into data frame # Note: Embeddings were created in create_embeddings.py df = pd.read_csv('blog_data_and_embeddings.csv') titles = df['title'] urls = df['url'] contents = df['content'] tokens = df['tokens'] embeds = [list(map(float, ast.literal_eval(embed_str))) for embed_str in df['embeddings']] df_new = pd.DataFrame({ 'title': titles, 'url': urls, 'content': contents, 'tokens': tokens, 'embeddings': embeds }) print(df_new.head()) ############################################################################### # Batch insert embeddings and metadata into database with psycopg2 ############################################################################### register_vector(conn) cur = conn.cursor() # Prepare the list of tuples to insert data_list = [(row['title'], row['url'], row['content'], int(row['tokens']), np.array(row['embeddings'])) for index, row in df_new.iterrows()] # Use execute_values to perform batch insertion execute_values(cur, "INSERT INTO embeddings (title, url, content, tokens, embedding) VALUES %s", data_list) # Commit after we insert all embeddings conn.commit() cur.execute("SELECT COUNT(*) as cnt FROM embeddings;") num_records = cur.fetchone()[0] print("Number of vector records in table: ", num_records,"\n") # Correct output should be 129 # print the first record in the table, for sanity-checking cur.execute("SELECT * FROM embeddings LIMIT 1;") records = cur.fetchall() print("First record in table: ", records) # Create an index on the data for faster retrieval #calculate the index parameters according to best practices num_lists = num_records / 1000 if num_lists < 10: num_lists = 10 if num_records > 1000000: num_lists = math.sqrt(num_records) #use the cosine distance measure, which is what we'll later use for querying cur.execute(f'CREATE INDEX ON embeddings USING ivfflat (embedding vector_cosine_ops) WITH (lists = {num_lists});') conn.commit() print("Index created on embeddings table") ``` 8. **Run `insert_embeddings.py`** ```bash python insert_embeddings.py ``` You should see an output that looks a bit like this: ```bash 0 How to Build a Weather Station With Elixir, Ne... ... [0.021399984136223793, 0.021850213408470154, -... 1 How to Build a Weather Station With Elixir, Ne... ... [0.01620873250067234, 0.011362895369529724, 0.... 2 How to Build a Weather Station With Elixir, Ne... ... [0.022517921403050423, -0.0019158280920237303,... 3 CloudQuery on Using Postgres for Cloud Asset... ... [0.008915113285183907, -0.004873732570558786, ... 4 CloudQuery on Using PostgreSQL for Cloud Asset... ... [0.0204352755099535, 0.010087345726788044, 0.0... [5 rows x 5 columns] Number of vector records in table: 129 First record in table: [(1, 'How to Build a Weather Station With Elixir, Nerves, and TimescaleDB', 'https://www.tigerdata.com/blog/how-to-build-a-weather-station-with-elixir-nerves-and-timescaledb', 'This is an installment of our "Community Member Spotlight" series, where we invite our customers to share their work, shining a light on their success and inspiring others with new ways to use technology to solve problems.In this edition,Alexander Koutmos, author of the Build a Weather Station with Elixir and Nerves book, joins us to share how he uses Grafana and TimescaleDB to store and visualize weather data collected from IoT sensors.About the teamThe bookBuild a Weather Station with Elixir and Nerveswas a joint effort between Bruce Tate, Frank Hunleth, and me.I have been writing software professionally for almost a decade and have been working primarily with Elixir since 2016. I currently maintain a few Elixir libraries onHexand also runStagira, a software consultancy company.Bruce Tateis a kayaker, programmer, and father of two from Chattanooga, Tennessee. He is the author of more than ten books and has been around Elixir from the beginning. He is the founder ofGroxio, a company that trains Elixir developers.Frank Hunlethis an embedded systems programmer, OSS maintainer, and Nerves core team member. When not in front of a computer, he loves running and spending time with his family.About the projectIn the Pragmatic Bookshelf book,Build a Weather Station with Elixir and Nerves, we take a project-based approach and guide the reader to create a Nerves-powered IoT weather station.For those unfamiliar with the Elixir ecosystem,Nervesis an IoT framework that allows you to build and deploy IoT applications on a wide array of embedded devices. At a high level, Nerves allows you to focus on building your project and takes care of a lot of the boilerplate associated with running Elixir on embedded devices.The goal of the book is to guide the reader through the process of building an end-to-end IoT solution for capturing, persisting, and visualizing weather data.Assembled weather station hooked up to development machine.One of the motivating factors for this book was to create a real-world project where readers could get hands-on experience with hardware without worrying too much about the nitty-gritty of soldering components together. Experimenting with hardware can often feel intimidating and confusing, but with Elixir and Nerves, we feel confident that even beginners get comfortable and productive quickly. As a result, in the book, we leverage a Raspberry Pi Zero W along with a few I2C enabled sensors to', 501, array([ 0.02139998, 0.02185021, -0.00537814, ..., -0.01257126, -0.02165324, -0.03714396], dtype=float32))] Index created on embeddings table ``` 9. **Create the `query_embeddings.py` file** To query the embeddings that you inserted into your Tiger Cloud service, open an editor of your choice and create the `query_embeddings.py` file. Here, the query is `How does Density use TimescaleDB?`. query\_embeddings.py ````python ############################################################################### # This script shows how to query embeddings stored in PostgreSQL # to find relevant documents for a given query # and use them to augment a base LLM to answer questions ############################################################################### import openai import os import pandas as pd import numpy as np import json import tiktoken import psycopg2 import ast import pgvector import math from psycopg2.extras import execute_values from pgvector.psycopg2 import register_vector # Get openAI api key by reading local .env file from dotenv import load_dotenv, find_dotenv _ = load_dotenv(find_dotenv()) openai.api_key = os.environ['OPENAI_API_KEY'] # Get Tiger Cloud / Postgres database connection string by reading local .env file connection_string = os.environ['TIMESCALE_CONNECTION_STRING'] # Connect to Postgres Tiger Cloud service using connection string conn = psycopg2.connect(connection_string) ############################################################################### # Helper functions for similarity search on documents in the database ############################################################################### # Helper function: Get top 3 most similar documents from the database def get_top3_similar_docs(query_embedding, conn): embedding_array = np.array(query_embedding) # Register pgvector extension register_vector(conn) cur = conn.cursor() # Get the top 3 most similar documents using the KNN <=> operator cur.execute("SELECT content FROM embeddings ORDER BY embedding <=> %s LIMIT 3", (embedding_array,)) top3_docs = cur.fetchall() return top3_docs # Helper function: get text completion from OpenAI API # Note we're using the latest gpt-3.5-turbo-0613 model # Change the model type to whichever model you prefer (e.g gpt-4) def get_completion_from_messages(messages, model="gpt-3.5-turbo-0613", temperature=0, max_tokens=1000): response = openai.ChatCompletion.create( model=model, messages=messages, temperature=temperature, max_tokens=max_tokens, ) return response.choices[0].message["content"] # Helper function: get embeddings for a text def get_embeddings(text): response = openai.Embedding.create( model="text-embedding-ada-002", input = text.replace("\n"," ") ) embedding = response['data'][0]['embedding'] return embedding ############################################################################### ############################################################################### # Function to process user input # with retrieval of most similar documents from the database # and completion from OpenAI API using Retrieval-Augmented Generation ############################################################################### def process_input_with_retrieval(user_input): delimiter = "```" #Step 1: Get documents related to the user input from database related_docs = get_top3_similar_docs(get_embeddings(user_input), conn) # Step 2: Get completion from OpenAI API # Set system message to help set appropriate tone and context for model system_message = f""" You are a friendly chatbot. \ You can answer questions about timescaledb, its features and its use cases. \ You respond in a concise, technically credible tone. \ """ # Prepare messages to pass to model # We use a delimiter to help the model understand the where the user_input starts and ends messages = [ {"role": "system", "content": system_message}, {"role": "user", "content": f"{delimiter}{user_input}{delimiter}"}, {"role": "assistant", "content": f"Relevant Tiger Data case studies information: \n {related_docs[0] [0]} \n {related_docs[1][0]} {related_docs[2][0]}"} ] final_response = get_completion_from_messages(messages) return final_response ############################################################################### # Question about a Tiger Data blog post we want the model to answer input = "How does Density use TimescaleDB?" # Get a response from the model using most reelvant documents from the database response = process_input_with_retrieval(input) print(input) print(response) ```` 10. **Run `query_embeddings.py`** ```bash python query_embeddings.py ``` You should see an output that looks a bit like this: ```bash How does Density use TimescaleDB? Density uses TimescaleDB as the main database in their smart city system. They store counts of people in spaces over time and derive metrics such as dwell time and space usage. TimescaleDB's flexibility and ability to handle time-series data efficiently allows Density to slice, dice, and compose queries in various ways. They also leverage TimescaleDB's continuous aggregates feature to roll up high-resolution data to lower resolutions, improving query performance. Additionally, TimescaleDB's support for percentile calculations has helped Density deliver accurate percentile values for their data. Overall, TimescaleDB has significantly improved the performance and scalability of Density's analytics workload. ``` # Analyse geospatial data with PostGIS > Store, query, and analyze geospatial data using the PostGIS extension The `postgis` PostgreSQL extension provides storing, indexing, and querying geographic data. It helps in spatial data analysis, the study of patterns, anomalies, and theories within spatial or geographical data. For more information about these functions and the options available, see the [PostGIS documentation](https://postgis.net/documentation/). ## Use the `postgis` extension to analyze geospatial data The `postgis` PostgreSQL extension allows you to conduct complex analyses of your geospatial time-series data. Tiger Data understands that you have a multitude of data challenges and helps you discover when things happened, and where they occurred. In this example you can query when the `covid` cases were reported, where they were reported, and how many were reported around a particular location. 1. **Install the `postgis` extension** ```sql CREATE EXTENSION postgis; ``` Confirm the extension is installed using the `\dx` command: ```sql List of installed extensions Name | Version | Schema | Description ---------------------+---------+------------+--------------------------------------------------------------------------------------- pg_stat_statements | 1.10 | public | track planning and execution statistics of all SQL statements executed pgcrypto | 1.3 | public | cryptographic functions plpgsql | 1.0 | pg_catalog | PL/pgSQL procedural language postgis | 3.3.3 | public | PostGIS geometry and geography spatial types and functions timescaledb | 2.24.0 | public | Enables scalable inserts and complex queries for time-series data (Community Edition) timescaledb_toolkit | 1.22.0 | public | Library of analytical hyperfunctions, time-series pipelining, and other SQL utilities (6 rows) ``` 2. **Create a hypertable named `covid_location`** `location` is a `GEOGRAPHY` type column that stores GPS coordinates using the 4326/WGS84 coordinate system, and `time` records the time the GPS coordinate was logged for a specific `state_id`. This hypertable is partitioned on the `time` column: ```sql CREATE TABLE covid_location ( time TIMESTAMPTZ NOT NULL, state_id INT NOT NULL, location GEOGRAPHY(POINT, 4326), cases INT NOT NULL, deaths INT NOT NULL ) WITH ( tsdb.hypertable ); ``` When you create a hypertable using [CREATE TABLE ... WITH ...](/docs/reference/timescaledb/hypertables/create_table), the default partitioning column is automatically the first column with a timestamp data type. Also, TimescaleDB creates a [columnstore policy](/docs/reference/timescaledb/hypercore/add_columnstore_policy) that automatically converts your data to the columnstore, after an interval equal to the value of the [chunk\_interval](/docs/reference/timescaledb/hypertables/set_chunk_time_interval), defined through `after` in the policy. This columnar format enables fast scanning and aggregation, optimizing performance for analytical workloads while also saving significant storage space. In the columnstore conversion, hypertable chunks are compressed by up to 98%, and organized for efficient, large-scale queries. You can customize this policy later using [alter\_job](/docs/reference/timescaledb/jobs-automation/alter_job). However, to change `after` or `created_before`, the compression settings, or the hypertable the policy is acting on, you must [remove the columnstore policy](/docs/reference/timescaledb/hypercore/remove_columnstore_policy) and [add a new one](/docs/reference/timescaledb/hypercore/add_columnstore_policy). You can also manually [convert chunks](/docs/reference/timescaledb/hypercore/convert_to_columnstore) in a hypertable to the columnstore. 3. **Create an index on the `state_id` column** ```sql CREATE INDEX ON covid_location (state_id, time DESC); ``` 4. **Insert sample data** The longitude and latitude coordinates of New Jersey are (-73.935242 40.730610), and New York are (-74.871826 39.833851): ```sql INSERT INTO covid_location VALUES ('2023-06-28 20:00:00',34,'POINT(-74.871826 39.833851)',5,2), ('2023-06-28 20:00:00',36,'POINT(-73.935242 40.730610)',7,1), ('2023-06-29 20:00:00',34,'POINT(-74.871826 39.833851)',14,0), ('2023-06-29 20:00:00',36,'POINT(-73.935242 40.730610)',12,1), ('2023-06-30 20:00:00',34,'POINT(-74.871826 39.833851)',10,4); ``` 5. **Fetch all cases of a specific state during a specific period** ```sql SELECT * FROM covid_location WHERE state_id = 34 AND time BETWEEN '2023-06-28 00:00:00' AND '2023-06-30 23:59:59'; ``` The data you get back looks a bit like this: ```sql time | state_id | location | cases | deaths ------------------------+----------+----------------------------------------------------+-------+-------- 2023-06-28 20:00:00+00 | 34 | 0101000020E61000005C7347FFCBB752C0535E2BA1BBEA4340 | 5 | 2 2023-06-29 20:00:00+00 | 34 | 0101000020E61000005C7347FFCBB752C0535E2BA1BBEA4340 | 14 | 0 2023-06-30 20:00:00+00 | 34 | 0101000020E61000005C7347FFCBB752C0535E2BA1BBEA4340 | 10 | 4 (3 rows) ``` 6. **Fetch the latest logged cases using SkipScan** Replace `` with the number of days between the day you are running the query and the day the last report was logged in the table, in this case 30, June, 2023: ```sql SELECT DISTINCT ON (state_id) state_id, ST_AsText(location) AS location FROM covid_location WHERE time > now() - INTERVAL '' ORDER BY state_id, time DESC; ``` The `ST_AsText(location)` function converts the binary geospatial data into human-readable format. The data you get back looks a bit like this: ```sql state_id | location ----------+----------------------------- 34 | POINT(-74.871826 39.833851) (1 row) ``` 7. **Fetch all cases within 10000 meters of Manhattan** ```sql SELECT DISTINCT cases, state_id FROM covid_location WHERE ST_DWithin( location, ST_GeogFromText('POINT(-73.9851 40.7589)'), 10000 ); ``` The data you get back looks a bit like this: ```sql cases | state_id -------+---------- 7 | 36 12 | 36 (2 rows) ``` # Maintenance and upgrades > Keep your Tiger Cloud service up to date with managed upgrades and maintenance windows Tiger Cloud offers managed database services that provide a stable and reliable environment for your applications. Each service is based on a specific version of the PostgreSQL database and the TimescaleDB extension. To ensure that you benefit from the latest features, performance and security improvements, it is important that your Tiger Cloud service is kept up to date with the latest versions of TimescaleDB and PostgreSQL. Tiger Cloud has the following upgrade policies: * **TimescaleDB upgrades**: * [Non-critical upgrades](#non-critical-upgrades) — new minor versions and patch releases, applied automatically during a [maintenance window](#define-your-maintenance-window) that you define. Or upgrade manually earlier. * [Critical security patches](#critical-security-patches) — installed outside normal maintenance windows when necessary, sometimes with a short outage. * **PostgreSQL upgrades**: * Minor version upgrades — applied automatically during your [maintenance window](#define-your-maintenance-window). * Major version upgrades — [performed by you](#manually-upgrade-postgresql), on your own schedule. Tiger Cloud only performs an [automatic upgrade](#automatic-upgrades-for-deprecated-versions) **when your current PostgreSQL version reaches end-of-life**. Note After a maintenance upgrade, the DNS name remains the same. However, the IP address often changes. ## Check your current versions In [Tiger Console](https://console.cloud.tigerdata.com/dashboard/services), select your service, then click `Operations` > `Service Upgrades`. This shows your current TimescaleDB and PostgreSQL versions, and whether a newer version is available. ![Service upgrades panel in Tiger Console showing current and latest PostgreSQL and TimescaleDB versions](/docs/_astro/tiger-console-service-upgrades-light.BE5rfg9j_1pfnWz.webp) ![Service upgrades panel in Tiger Console showing current and latest PostgreSQL and TimescaleDB versions](/docs/_astro/tiger-console-service-upgrades-dark.CX8HFv7i_Z19OqDH.webp) ## TimescaleDB upgrades Tiger Cloud keeps your services on a supported version of TimescaleDB. Non-critical upgrades — new minor versions and patch releases — are applied during your [maintenance window](#define-your-maintenance-window), or you can [trigger them manually](#non-critical-upgrades). Critical security patches are installed as soon as they are available. ### Non-critical upgrades Non-critical upgrades — new minor versions and patch releases — are applied automatically in the next available maintenance window. The schedule depends on the upgrade type and your service's environment tag: * **New minor versions on `#dev` services**: applied in the next maintenance window after the release. * **New minor versions on `#prod` services**: applied in the next maintenance window three weeks after the release. This gives you three weeks to validate the release on your `#dev` services first. * **Patch releases**: applied in the next maintenance window on both `#dev` and `#prod` services, with no waiting period. Occasionally, a new minor version is released before the previous one has completed its three-week window — for example, when it contains an important fix. In that case, the newer minor version takes priority: your `#prod` services skip the previous minor and target the new one instead, and the three-week window restarts from the new release. Your `#dev` services move to the new minor in their next maintenance window, as usual. [Subscribe](https://status.tigerdata.com/) to get an email notification before your `#dev` services are upgraded. If there are no pending upgrades during a maintenance window, no changes are performed. To apply a non-critical TimescaleDB upgrade manually before the next maintenance window: 1. **Connect to your service** In [Tiger Console](https://console.cloud.tigerdata.com/dashboard/services), select the service you want to upgrade. 2. **Upgrade TimescaleDB** Click `SQL Editor` at the bottom, then run `ALTER EXTENSION timescaledb UPDATE`. ### Critical security patches Critical security patches are installed outside your normal maintenance window as soon as they are available. ### Downtime and connection resets Most upgrades do not require any downtime. Connections and transactions in progress during the upgrade are reset, then the service connection is usually restored automatically. Some background maintenance tasks performed outside the maintenance window can also cause connection resets — when one occurs, you can immediately reconnect. Some non-critical upgrades and critical security patches do require a short outage, usually between 30 seconds and 5 minutes. Tiger Data endeavors to notify you by email ahead of an upgrade that requires downtime. In some cases — particularly for critical security patches — advance notice is not possible. To track the status of maintenance events, see the Tiger Cloud [status page](https://status.tigerdata.com/). To minimize the impact of downtime, [schedule your maintenance window](#define-your-maintenance-window) for a low-traffic period in your application, and add a [high-availability replica or read replica](high-availability/high-availability). With a replica, writes pause only briefly during maintenance and read-only queries continue to be served by the replica throughout the upgrade. During a maintenance event, each node is upgraded independently. The primary is upgraded and restarted first: * If the primary restarts within a minute, no failover occurs. The replica stays as the replica, and maintenance then proceeds on it. * If the primary takes longer than a minute to restart — and the replica has no replication lag — the replica is promoted to primary (one failover). Maintenance then proceeds on the demoted former primary. If that node is also slow to restart, the original primary is promoted back (a second failover). Each failover takes less than a few seconds. In the worst case, two failovers add up to about two minutes of write downtime. ## PostgreSQL upgrades New minor versions of PostgreSQL are applied automatically during your [maintenance window](#define-your-maintenance-window). You upgrade to a new major version of PostgreSQL manually, on your own schedule. Tiger Cloud only performs an [automatic upgrade](#automatic-upgrades-for-deprecated-versions) once your current PostgreSQL version reaches end-of-life. Upgrading to a newer version of PostgreSQL allows you to take advantage of new features, enhancements, and security fixes. It also ensures that you are using a version of PostgreSQL that's compatible with the newest version of TimescaleDB. For more information about feature changes between versions, see the [Tiger Cloud release notes](/docs/get-started/news/new), [supported systems](/docs/get-started/choose-your-path/supported-platforms), and the [PostgreSQL release notes](https://www.postgresql.org/docs/release/). The following table shows you the compatible versions of PostgreSQL and TimescaleDB: Warning TimescaleDB 2.28.x is the final minor version to support PostgreSQL 15. Support for PostgreSQL 15 is dropped in 2.29.0; patch releases in the 2.28.x cycle continue to support it. | TimescaleDB version | PostgreSQL 18 | PostgreSQL 17 | PostgreSQL 16 | PostgreSQL 15 | PostgreSQL 14 | PostgreSQL 13 | PostgreSQL 12 | PostgreSQL 11 | PostgreSQL 10 | | ------------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | | 2.28.x | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | 2.27.x | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | 2.26.x | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | 2.25.x | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | 2.24.x | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | 2.23.x | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | 2.22.x | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | 2.21.x | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | 2.20.x | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | 2.17 - 2.19 | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | | 2.16.x | ❌ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | | 2.13 - 2.15 | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | | 2.12.x | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | | 2.10.x | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | | 2.5 - 2.9 | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | | 2.4 | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | | 2.1 - 2.3 | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ❌ | | 2.0 | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | | 1.7 | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | We recommend not using TimescaleDB with PostgreSQL 17.1, 16.5, 15.9, 14.14, 13.17, 12.21.\ These minor versions [introduced a breaking binary interface change](https://www.postgresql.org/about/news/postgresql-172-166-1510-1415-1318-and-1222-released-2965/) that, once identified, was reverted in subsequent minor PostgreSQL versions 17.2, 16.6, 15.10, 14.15, 13.18, and 12.22. When you build from source, best practice is to build with PostgreSQL 17.2, 16.6, etc and higher. Users of [Tiger Cloud](https://console.cloud.tigerdata.com/) and platform packages for Linux, Windows, MacOS, Docker, and Kubernetes are unaffected. ### Manually upgrade PostgreSQL For a smooth upgrade experience, make sure you: * **Plan ahead**: upgrades cause downtime, so ideally perform an upgrade during a low traffic time. * **Run a test upgrade**: [fork your service](service-management/fork-services), then try out the upgrade on the fork before running it on your production system. This gives you a good idea of what happens during the upgrade, and how long it might take. * **Keep a copy of your service**: if you're worried about losing your data, [fork your service](service-management/fork-services) without upgrading, and keep this duplicate of your service. To reduce cost, you can immediately pause this fork and only pay for storage until you are comfortable deleting it after the upgrade is complete. Important Tiger Cloud services with replicas cannot be upgraded. To upgrade a service with a replica, you must first delete the replica and then upgrade the service. Warning Your Tiger Cloud service is unavailable until the upgrade is complete. This can take up to 20 minutes. Best practice is to test on a fork first, so you can estimate how long the upgrade will take. To upgrade your service to a newer version of PostgreSQL: 1. **Connect to your service** In [Tiger Console](https://console.cloud.tigerdata.com/dashboard/services), select the service you want to upgrade. 2. **Disable high-availability replicas** 1. Click `Operations` > `High Availability`, then click `Change configuaration`. 2. Select `Non-production (No replica)`, then click `Change configuration`. 3. **Disable read replicas** Click `Operations` > `Read scaling`, then click the trash icon next to all read replica sets. 4. **Upgrade PostgreSQL** 1. Click `Operations` > `Service Upgrades`. 2. Click `Upgrade service`, then confirm that you are ready to start the upgrade. Your Tiger Cloud service is unavailable until the upgrade is complete. This normally takes up to 20 minutes. However, it can take longer if you have a large or complex service. When the upgrade is finished, your service automatically resumes normal operations. If the upgrade is unsuccessful, the service returns to the state it was in before you started the upgrade. 5. **Enable high-availability replicas and replace your read replicas** ### Automatic upgrades for deprecated versions To ensure you benefit from the latest features, optimal performance, enhanced security, and full compatibility with TimescaleDB, Tiger Cloud supports a defined set of PostgreSQL major versions. As PostgreSQL and TimescaleDB evolve, Tiger Data periodically deprecates older PostgreSQL versions. If you do not upgrade before a deprecated version's deadline, Tiger Cloud automatically upgrades your service to keep it on a supported PostgreSQL version. The automatic upgrade is applied during your [maintenance window](#define-your-maintenance-window). Automatic upgrades can result in downtime, so best practice is to [upgrade manually](#manually-upgrade-postgresql) during a low-traffic period for your application. Tiger Data provides advance notification to allow you ample time to plan and perform your upgrade. The deprecation timeline is as follows: * **Deprecation notice**: you receive email notification of the deprecation and the timeline for the upgrade. * **Manual upgrade period**: [upgrade your service manually](#manually-upgrade-postgresql) before the deadline. * **Automatic upgrade deadline**: Tiger Cloud performs an automatic upgrade of your service. During an automatic upgrade: 1. Any configured [high-availability replicas](high-availability/high-availability) or [read replicas](high-availability/read-scaling) are temporarily removed. 2. The primary service is upgraded. 3. High-availability replicas and read replicas are added back to the service. ## Define your maintenance window When you are considering your maintenance window schedule, best practice is to choose a day and time that usually has very low activity, such as during the early hours of the morning, or over the weekend. This helps minimize the impact of a short service interruption. Alternatively, you might prefer to have your maintenance window occur during office hours, so that you can monitor your system during the upgrade. To change your maintenance window: 1. **Connect to your service** In [Tiger Console](https://console.cloud.tigerdata.com/dashboard/services), select the service you want to manage. 2. **Set your maintenance window** 1. Click `Operations` > `Environment`, then click `Change maintenance window`. ![Maintenance and upgrades](/docs/_astro/tiger-console-maintenance-upgrades-light.wucfJVqm_Z1SdmaA.webp) ![Maintenance and upgrades](/docs/_astro/tiger-console-maintenance-upgrades-dark.DNAqiJWY_Z1Jgwrf.webp) 2. Select the maintenance window start time, then click `Apply`. Maintenance windows can run for up to four hours. # About configuration in Tiger Cloud > Have your service auto-tuned or configure manually By default, Tiger Cloud uses the default PostgreSQL server configuration settings. Most configuration values for a Tiger Cloud service are initially set in accordance with best practices given the compute and storage settings of the service. Any time you increase or decrease the compute for a service, the most essential values are set to reflect the size of the new service. There are times, however, when your specific workload could require tuning some of the many available Tiger Cloud-specific and PostgreSQL parameters. By providing the ability to tune various runtime settings, Tiger Cloud provides the balance and flexibility you need when running your workloads in a hosted environment. You can use [service settings](../configuration/advanced-parameters) and [service operations](../configuration/customize-configuration) to customize Tiger Cloud configurations. # Advanced parameters > Tune advanced PostgreSQL parameters for your Tiger Cloud service in Tiger Console It is possible to configure a wide variety of Tiger Cloud service database parameters by navigating to the `Advanced parameters` tab under the `Database configuration` heading. The advanced parameters are displayed in a scrollable and searchable list. ![Database configuration advanced parameters](/docs/_astro/database-configuration-advanced-parameters.bUCg8l3U_Z1157EE.webp) As with the basic database configuration parameters, any changes are highlighted and the `Apply changes`, or `Apply changes and restart`, button is available, prompting you to confirm changes before the service is modified. ## Commonly tuned parameters These are some of the most frequently adjusted parameters available in the Advanced parameters tab: * **`max_locks_per_transaction`**: maximum number of object locks per transaction. Increase this if you see `out of shared memory` errors, especially with hypertables that have many chunks. See [Transaction lock settings](/docs/reference/timescaledb/configuration/tiger-postgres#transaction-lock-settings) for sizing guidance. * **`shared_buffers`**: amount of memory used for shared memory buffers. A good starting point is 25% of total system memory. * **`work_mem`**: amount of memory used by internal sort operations and hash tables before writing to temporary disk files. * **`maintenance_work_mem`**: maximum memory for maintenance operations such as `VACUUM` and `CREATE INDEX`. * **`max_connections`**: maximum number of concurrent connections to the database. * **`timescaledb.max_background_workers`**: maximum background workers allocated to TimescaleDB. Set to at least 1 + the number of databases in the instance. For a complete list of available parameters, see the [Grand Unified Configuration (GUC) parameters](/docs/reference/timescaledb/configuration/gucs) reference. For detailed tuning guidance, see [Configuration parameters](/docs/reference/timescaledb/configuration/tiger-postgres). ## Multiple databases To create more than one database, you need to create a new service for each database. Tiger Cloud does not support multiple databases within the same service. Having a separate service for each database affords each database its own isolated resources. You can also use [schemas](https://www.postgresql.org/docs/current/ddl-schemas.html) to organize tables into logical groups. A single database can contain multiple schemas, which in turn contain tables. The main difference between isolating with databases versus schemas is that a user can access objects in any of the schemas in the database they are connected to, so long as they have the corresponding privileges. Schemas can help isolate smaller use cases that do not warrant their own service. Please refer to the [Grand Unified Configuration (GUC) parameters](/docs/reference/timescaledb/configuration/gucs) for a complete list. ## Policies ### `timescaledb.max_background_workers (int)` Max background worker processes allocated to TimescaleDB. Set to at least 1 + the number of databases loaded with the TimescaleDB extension in a PostgreSQL instance. Default value is 16. ## Tiger Cloud service tuning ### `timescaledb.disable_load (bool)` Disable the loading of the actual extension # Configure database parameters > Customize PostgreSQL and Tiger Cloud configuration options per service Tiger Cloud allows you to customize many Tiger Cloud-specific and PostgreSQL configuration options for each service individually. Most configuration values for a service are initially set in accordance with best practices given the compute and storage settings of the service. Any time you increase or decrease the compute for a service, the most essential values are set to reflect the size of the new service. Warning You can modify most parameters without restarting the service. However, some changes do require a restart, resulting in some brief downtime that is usually about 30 seconds. An example of a change that needs a restart is modifying the compute resources of a running service. ## View service operation details To modify configuration parameters, first select the service that you want to modify. This displays the service details, with these tabs across the top: `Overview`, `Actions`, `Explorer`, `Monitoring`, `Connections`, `Operations`, and `AI`. Select `Operations`, then `Database parameters`. ![Database configuration parameters](/docs/_astro/tiger-console-service-configuration-parameters.B_9CNzSq_1YC4HB.webp) ### Modify basic parameters Under the `Common parameters` tab, you can modify a limited set of the parameters that are most often modified in a Tiger Cloud or PostgreSQL instance. To modify a configured value, hover over the value and click the revealed pencil icon. This reveals an editable field to apply your change. Clicking anywhere outside of that field saves the value to be applied. ![Change Tiger Cloud configuration parameters](/docs/_astro/tsc-settings-change.MDtFEN3R_1pf8ld.webp) ### Apply configuration changes When you have modified the configuration parameters that you would like to change, click `Apply changes`. For some changes, such as `timescaledb.max_background_workers`, the service needs to be restarted. In this case, the button reads `Apply changes and restart`. A confirmation dialog is displayed which indicates whether a restart is required. Click `Confirm` to apply the changes, and restart if necessary. # Back up and recover services > Protect your data with automatic backups Tiger Cloud provides comprehensive backup and recovery solutions to protect your data, including automatic daily backups and point-in-time recovery. ## Automatic backups [Section titled “Automatic backups”](#automatic-backups) Tiger Cloud automatically handles backup for your Tiger Cloud services using the `pgBackRest` tool. You don't need to perform backups manually. Tiger Cloud automatically creates one full backup every week, and incremental backups every day in the same region as your service. Additionally, all [Write-Ahead Log (WAL)](https://www.postgresql.org/docs/current/wal-intro.html) files are retained back to the oldest full backup. This means that you always have a full backup available for the current and previous week: ![Database backup and recovery architecture in Tiger Cloud](/docs/_astro/database-backup-recovery.DHoYmo-Y_Z1NJvh0.webp) On [Scale and Performance](/docs/deploy/tiger-cloud/tiger-cloud-azure/pricing-and-account-management) pricing plans, you can check the list of backups for the previous 14 days in Tiger Console. To do so, select your service, then click `Operations` > `Backup and restore` > `Backup history`. In the event of a storage failure, a service automatically recovers from a backup to the point of failure. If the whole availability zone goes down, your Tiger Cloud services are recovered in a different zone. In the event of a user error, you can [create a point-in-time recovery fork](../service-management/fork-services). # Manage high availability > Configure HA replicas across availability zones to minimize downtime For Tiger Cloud services where every second of uptime matters, Tiger Cloud delivers High Availability (HA) replicas. These replicas safeguard your data and keep your service running smoothly, even in the face of unexpected failures. By minimizing downtime and protecting against data loss, HA replicas ensure business continuity and give you the confidence to operate without interruption, including during routine maintenance. ![High availability replicas in Tiger Cloud across multiple AWS availability zones](/docs/_astro/tiger-cloud-ha-architecture-diagram.eNwtvpxM_x1xwG.webp) ![High availability replicas in Tiger Cloud across multiple AWS availability zones](/docs/_astro/tiger-cloud-ha-architecture-diagram-dark.DnWxlufO_18jKoM.webp) This page shows you how to choose the best high availability option for your service. ## What is HA replication? HA replicas are exact, up-to-date copies of your database hosted in multiple AWS availability zones (AZ) within the same region as your primary node. They automatically take over operations if the original primary data node becomes unavailable. The primary node streams its write-ahead log (WAL) to the replicas to minimize the chances of data loss during failover. HA replicas can be synchronous and asynchronous. * Synchronous: the primary commits its next write once the replica confirms that the previous write is complete. There is no lag between the primary and the replica. They are in the same state at all times. This is preferable if you need the highest level of data integrity. However, this affects the primary ingestion time. * Asynchronous: the primary commits its next write without the confirmation of the previous write completion. The asynchronous HA replicas often have a lag, in both time and data, compared to the primary. This is preferable if you need the shortest primary ingest time. ![Sync and async replication](/docs/_astro/sync_async_replication_draft.owC364ei_Z2rKBfJ.webp) ![Sync and async replication](/docs/_astro/sync_async_replication_draft-dark.DgNPMrCW_Z1aUROl.webp) HA replicas have separate unique addresses that you can use to serve read-only requests in parallel to your primary data node. When your primary data node fails, Tiger Cloud automatically fails over to an HA replica within 30 seconds. During failover, the read-only address is unavailable while Tiger Cloud automatically creates a new HA replica. The time to make this replica depends on several factors, including the size of your data. Operations such as upgrading your service to a new major or minor version may necessitate a service restart. Restarts are run during the [maintenance window](../upgrades). To avoid any downtime, each data node is updated in turn. That is, while the primary data node is updated, a replica is promoted to primary. After the primary is updated and online, the same maintenance is performed on the HA replicas. To ensure that all services have minimum downtime and data loss in the most common failure scenarios and during maintenance, [rapid recovery](../high-availability/overview#rapid-recovery) is enabled by default for all services. ## Choose an HA strategy The following HA configurations are available in Tiger Cloud: * **Non-production**: no replica, best for developer environments. * **High availability**: a single async replica in a different AWS availability zone from your primary. Provides high availability with cost efficiency. Best for production apps. * **Highest availability**: two replicas in different AWS availability zones from your primary. Available replication modes are: * **High performance** - two async replicas. Provides the highest level of availability with two AZs and the ability to query the HA system. Best for apps where service availability is most critical. * **High data integrity** - one sync replica and one async replica. The sync replica is identical to the primary at all times. Best for apps that can tolerate no data loss. The following table summarizes the differences between these HA configurations: | | High availability (1 async) | High performance (2 async) | High data integrity (1 sync + 1 async) | | ------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Write flow | The primary streams its WAL to the async replica, which may have a slight lag compared to the primary, providing 99.9% uptime SLA. | The primary streams its writes to both async replicas, providing 99.9+% uptime SLA. | The primary streams its writes to the sync and async replicas. The async replica is never ahead of the sync one. | | Additional read replica | Recommended. Reads from the HA replica may cause availability and lag issues. | Not needed. You can still read from the HA replica even if one of them is down. Configure an additional read replica only if your read use case is significantly different from your write use case. | Highly recommended. If you run heavy queries on a sync replica, it may fall behind the primary. Specifically, if it takes too long for the replica to confirm a transaction, the next transaction is canceled. | | Choosing the replica to read from manually | Not applicable. | Not available. Queries are load-balanced against all available HA replicas. | Not available. Queries are load-balanced against all available HA replicas. | | Sync replication | Only async replicas are supported in this configuration. | Only async replicas are supported in this configuration. | Supported. | | Failover flow | * If the primary fails, the replica becomes the primary while a new node is created, with only seconds of downtime. * If the replica fails, a new async replica is created without impacting the primary. If you read from the async HA replica, those reads fail until the new replica is available. | - If the primary fails, one of the replicas becomes the primary while a new node is created, with the other one still available for reads. - If the replica fails, a new async replica is created in another AZ, without impacting the primary. The newly created replica is behind the primary and the original replica while it catches up. | * If the primary fails, the sync replica becomes the primary while a new node is created, with the async one still available for reads. * If the async replica fails, a new async replica is created. Heavy reads on the sync replica may delay the ingest time of the primary while a new async replica is created. Data integrity remains high but primary ingest performance may degrade. * If the sync replica fails, the async replica becomes the sync one, and a new async replica is created. The primary may experience some ingest performance degradation during this time. | | Cost composition | Primary + async (2x) | Primary + 2 async (3x) | Primary + 1 async + 1 sync (3x) | | Tier | Performance, Scale, and Enterprise | Scale and Enterprise | Scale and Enterprise | The `High` and `Highest` HA strategies are available with the [Scale and the Enterprise](../pricing-and-account-management) pricing plans. To enable HA for a service: 1. **In Tiger Console, select the service to enable replication for.** 2. **Click `Operations`, then select `High availability`.** 3. **Choose your replication strategy, then click `Change configuration`.** ![High availability replica configuration in Tiger Console](/docs/_astro/tiger-console-ha-replicas.BsuEJHbW_Znnyey.webp) 4. **In `Change high availability configuration`, click `Change config`.** To change your HA replica strategy, click `Change configuration`, choose a strategy and click `Change configuration`. To download the connection information for the HA replica, either click the link next to the replica `Active configuration`, or find the information in the `Overview` tab for this service. ## Test failover for your HA replicas To test the failover mechanism, you can trigger a switchover. A switchover is a safe operation that attempts a failover, and throws an error if the replica or primary is not in a state to safely switch. 1. **Connect to your primary node as `tsdbadmin` or another user that is part of the `tsdbowner` group** Note You can also connect to the HA replica and check its node using this procedure. 2. **At the `psql` prompt, connect to the `postgres` database** ```sql \c postgres ``` You should see `postgres=>` prompt. 3. **Check if your node is currently in recovery** ```sql select pg_is_in_recovery(); ``` 4. **Check which node is currently your primary** ```sql select * from pg_stat_replication; ``` Note the `application_name`. This is your service ID followed by the node. The important part is the `-an-0` or `-an-1`. 5. **Schedule a switchover** ```sql CALL tscloud.cluster_switchover(); ``` By default, the switchover occurs in 30 secs. You can change the time by passing an interval, like this: ```sql CALL tscloud.cluster_switchover('15 seconds'::INTERVAL); ``` 6. **Wait for the switchover to occur, then check which node is your primary** ```sql SELECT * FROM pg_stat_replication; ``` You should see a notice that your connection has been reset, like this: ```sql FATAL: terminating connection due to administrator command SSL connection has been closed unexpectedly The connection to the server was lost. Attempting reset: Succeeded. ``` 7. **Check the `application_name`** If your primary was `-an-1` before, it should now be `-an-0`. If it was `-an-0`, it should now be `-an-1`. # Overview > Reduce downtime risk and scale reads with database replication In Tiger Cloud, replicas are copies of the primary data instance in a Tiger Cloud service. If your primary becomes unavailable, Tiger Cloud automatically fails over to your HA replica. The replication strategies offered by Tiger Cloud are: * [High Availability (HA) replicas](../high-availability/high-availability): significantly reduce the risk of downtime and data loss due to system failure, and enable services to avoid downtime during routine maintenance. * [Read replicas](../high-availability/read-scaling): safely scale a service to power your read-intensive apps and business intelligence tooling and remove the load from the primary data instance. For MST, see [Failover in Managed Service for TimescaleDB](/docs/deploy/mst/failover). For self-hosted TimescaleDB, see [Replication and high availability](/docs/deploy/self-hosted/replication-and-ha/about-ha). ## Rapid recovery By default, all services have rapid recovery enabled. Because compute and storage are handled separately in Tiger Cloud, services recover quickly from compute failures, but usually need a full recovery from backup for storage failures. * **Compute failure**: the most common cause of database failure. Compute failures can be caused by hardware failing, or through things like unoptimized queries, causing increased load that maxes out the CPU usage. In these cases, data on disk is unaffected and only the compute and memory needs replacing. Tiger Cloud recovery immediately provisions new compute infrastructure for the service and mounts the existing storage to the new node. PostgreSQL crash recovery then replays any unapplied WAL from disk. This process typically only takes thirty seconds. However, depending on the amount of WAL that needs replaying this may take up to twenty minutes. Even in the worst-case scenario, Tiger Cloud recovery is an order of magnitude faster than a standard recovery from backup. * **Storage failure**: in the rare occurrence of disk failure, Tiger Cloud automatically [performs a full recovery from backup](../high-availability/backup-restore). If CPU usage for a service runs high for long periods of time, issues such as WAL archiving getting queued behind other processes can occur. This can cause a failure and could result in a larger data loss. To avoid data loss, services are monitored for this kind of scenario. # Read scaling > Scale reads horizontally with read replica sets and automated load balancing When read-intensive workloads compete with high ingest rates, your primary data instance can become a bottleneck. Spiky query traffic, analytical dashboards, and business intelligence tools risk slowing down ingest performance and disrupting critical write operations. With read replica sets in Tiger Cloud, you can scale reads horizontally and keep your applications responsive. By offloading queries to replicas, your service maintains high ingest throughput while serving large or unpredictable read traffic with ease. This approach not only protects write performance but also gives you confidence that your read-heavy apps and BI workloads will run smoothly, even under pressure. ![Horizontal read scaling in Tiger Cloud: writes go to a primary, while reads are distributed across read replicas through a single load-balanced endpoint](/docs/_astro/horizontal-read-scaling-in-tiger-cloud-light.FKvfNjnh_1Ew1eU.webp) ![Horizontal read scaling in Tiger Cloud: writes go to a primary, while reads are distributed across read replicas through a single load-balanced endpoint](/docs/_astro/horizontal-read-scaling-in-tiger-cloud-dark.CujqDL0m_ZAil5b.webp) This page shows you how to create and manage read replica sets in Tiger Console. ## What is read replication? A read replica is a read-only copy of your primary database instance. Queries on read replicas have minimal impact on the performance of the primary instance. This enables you to interact with up-to-date production data for analysis, or to scale out reads beyond the limits of your primary instance. Read replicas can be short-lived and deleted when a session of data analysis is complete, or long-running to power an application or a business intelligence tool. A read replica set in Tiger Cloud is a group of one or more read replica nodes that are accessed through the same endpoint. You query each set as a single replica. Tiger Cloud balances the load between the nodes in the set for you. You can create as many read replica sets as you need. For security and resource isolation, each read replica set has unique connection details. You use read replica sets for horizontal **read** scaling. To limit data loss for your Tiger Cloud services, use [HA replicas](../high-availability/high-availability). ## Prerequisites for this procedure To follow these steps, you'll need: * A Tiger Cloud service. * A [read-only user](../security/read-only-role#create-a-read-only-user) on the primary data instance. A user with read-only permissions cannot make changes in the primary database. This user is propagated to the read replica set when you create it. ## Create a read replica set To create a secure read replica set for your read-intensive apps: 1. **In Tiger Console, select your target service** 2. **Click `Operations` > `Read scaling` > `Add a read replica set`** 3. **Configure your replica set** Configure the number of nodes, compute size, connection pooling, and the name for your replica, then click `Create read replica set`. ![Create a set in](/docs/_astro/create-read-replica-set-tiger-console.BOwX6j3I_1MzWxL.webp) 4. **Save the connection information** The username and password of a read replica set are the same as the primary service. They cannot be changed independently. The connection information for each read replica set is unique. You can add or remove nodes from an existing set and the connection information of that set will remain the same. To find the connection information for an existing read replica set: 1. Select the primary service in Tiger Console. 2. Click `Operations` > `Read scaling`. 3. Click the link icon next to the replica set in the list. ## Edit a read replica set You can edit an existing read replica set to better handle your reads. This includes changing the number of nodes, compute size, storage, and IOPS, as well as configuring VPC and other features. To change the compute and storage configuration of your read replica set: 1. **In Tiger Console, expand and click the read replica set under your primary service** ![s in](/docs/_astro/read-replica-sets-tiger-console.DRE2fSvD_1NIiPt.webp) 2. **Click `Operations` > `Compute and storage`** ![compute and storage in](/docs/_astro/read-replica-set-config-tiger-console.BZJHSkD0_Z25HFhW.webp) 3. **Change the replica configuration and click `Apply`** ## Manage data lag for your read replica sets Read replica sets use asynchronous replication. This can cause a slight lag in data to the primary database instance. The lag is measured in bytes, against the current state of the primary instance. To check the status and lag for your read replica set: 1. **In Tiger Console, select your primary service** 2. **Click `Operations` > `Read scaling`** You see a list of configured read replica sets for this service, including their status and lag: ![sets](/docs/_astro/configured-replica-set-tiger-console.3fi7MRoJ_1013A9.webp) 3. **Configure the allowable lag** 1. Select the replica set in the list. 2. Click `Operations` > `Database parameters`. 3. Adjust `max_standby_streaming_delay` and `max_standby_archive_delay`. This is not recommended for cases where changes must be immediately represented, for example, for user credentials. ## Delete a read replica set To delete a replica set: 1. **In Tiger Console, select your primary service** 2. **Click `Operations` > `Read scaling`** 3. **Click the trash icon next to a replica set** Confirm the deletion when prompted. # Monitor your services > View logs, metrics, and performance insights for your Tiger Cloud services Get complete visibility into your service performance with Tiger Cloud's powerful monitoring suite. Whether you're optimizing for peak efficiency or troubleshooting unexpected behavior, Tiger Cloud gives you the tools to quickly identify and resolve issues. When something doesn't look right, Tiger Cloud provides a complete investigation workflow: ![Monitoring suite in Tiger Cloud](/docs/_astro/tiger-monitoring-workflow-diagram.ndjrElOs_KWOK2.webp) ![Monitoring suite in Tiger Cloud](/docs/_astro/tiger-monitoring-workflow-diagram-dark.CTYAlyS5_ZUKbEt.webp) 1. **Pinpoint the bottleneck**: check [**Metrics**](#metrics) to identify exactly when CPU, memory, or storage spiked. 2. **Find the root cause**: review [**Logs**](#logs) for errors or warnings that occurred during the incident. 3. **Identify the culprit**: examine [**Insights**](#insights) to see which queries were running at that time and how they impacted resources. 4. **Check background activity**: look at [**Jobs**](#jobs) to see if scheduled tasks triggered the issue. 5. **Investigate active connections**: use [**Connections**](#connections) to see what clients were connected and what queries they were running. Want to save some time? Check out [**Recommendations**](#recommendations) for alerts that may have already flagged the problem! This page explains what specific data you get at each point. ## Metrics Tiger Cloud shows you CPU, memory, and storage metrics for up to 30 previous days and with down to 10-second granularity. To access metrics, select your service in Tiger Console, then click `Monitoring` > `Metrics`: ![Service metrics](/docs/_astro/service-metrics-tiger-console.CVDYrWWU_11SHsc.webp) The following metrics are represented by graphs: * CPU, in mCPU * Memory, in GiB * Queries per second, in q/sec * Storage used, in GiB * Read and write storage I/O, in ops/sec * Read and write storage bandwidth, in MiB/sec When you hit the limits: * **For CPU and memory**: provision more for your service in `Operations` > `Compute and storage`. * **For storage, I/O, and bandwidth**: these resources depend on your storage type and I/O Boost settings. The standard high-performance storage gives you 16TB of compressed data on a single server, regardless of the number of hypertables in your service. See [About storage tiers](/docs/learn/data-lifecycle/storage/about-storage-tiers) for how to change the available storage, I/O, and bandwidth. Hover over the graph to view metrics for a specific time point. Select an area in the graph to zoom into a specific period. Gray bars indicate that metrics have not been collected for the period shown: ![Metrics not collected](/docs/_astro/tsc-metrics_graybar.C2YZ0u1f_SSgWP.webp) ### Understand high memory usage It is normal to observe high overall memory usage for your Tiger Cloud services, especially for workloads with active read and write. Tiger Cloud service run on Linux, and high memory usage is a particularity of the Linux page cache. The Linux kernel stores file-backed data in memory to speed up read operations. PostgreSQL, and by extension, Tiger Cloud services rely heavily on disk I/O to access tables, WALs, and indexes. When your service reads these files, the kernel caches them in memory to improve performance for future access. Page cache entries are not [locked memory](https://www.gnu.org/s/libc/manual/html_node/Locked-Memory-Details.html): they are evictable and are automatically reclaimed by the kernel when actual memory pressure arises. Therefore, high memory usage shown in the monitoring dashboards is often not due to service memory allocation, but the beneficial caching behavior in the Linux kernel. The trick is to distinguish between normal memory utilization and memory pressure. High memory usage does not necessarily mean a problem, especially on read replicas or after periods of activity. For a more accurate view of database memory consumption, look at PostgreSQL-specific metrics, such as shared\_buffers or memory context breakdowns. Only [take action](https://www.postgresql.org/docs/current/runtime-config-resource.html#RUNTIME-CONFIG-RESOURCE-MEMORY) if you see signs of real memory pressure, such as OOM (Out Of Memory) events or degraded performance. ### Service states Tiger Console gives you a visual representation of the state of your service. The following states are represented with the following colors: | State | Color | | ----------- | ------ | | Configuring | Yellow | | Deleted | Yellow | | Deleting | Yellow | | Optimizing | Green | | Paused | Grey | | Pausing | Grey | | Queued | Yellow | | Ready | Green | | Resuming | Yellow | | Unstable | Yellow | | Upgrading | Yellow | | Read-only | Red | ## Logs Tiger Cloud shows you detailed logs for your service, which you can filter by type, date, and time, and then export to CSV. To access logs, select your service in Tiger Console, then click `Monitoring` > `Logs`: ![Find logs faster](/docs/_astro/tiger-console-service-logs.DLosa5jQ_Z29cxVQ.webp) ## Insights Insights help you get a comprehensive understanding of how your queries perform over time, and make the most efficient use of your resources. To view insights, select your service, then click `Monitoring` > `Insights`. Search or filter queries by type, maximum execution time, and time frame. ![Insights](/docs/_astro/insights-overview-tiger-console.i9IYWTi-_Z1cSVHC.webp) Insights include `Metrics`, `Current lock contention`, and `Queries`. `Metrics` provides a visual representation of CPU, memory, and storage input/output usage over time. It also overlays the execution times of the top three queries matching your search. This helps correlate query executions with resource utilization. Select an area of the graph to zoom into a specific time frame. `Current lock contention` shows how many queries or transactions are currently waiting for locks held by other queries or transactions. `Queries` displays the top 50 queries matching your search. This includes a whole range of metrics: ![Queries](/docs/_astro/tiger-console-query-insights.DVctYj3R_rLYWM.webp) | Column | Description | | --------------------- | --------------------------------------------------------------------------------------------------------------- | | `Executions` | Number of times the query ran during the selected period. | | `Total rows` | Total rows scanned, inserted, or updated by the query during the selected period. | | `Total time` | Combined execution time across all runs of the query. | | `Time (P95)` | Execution time in the worst typical case. 95th percentile. | | `CPU User (P95)` | CPU spent on the query's own work — calculations, sorting, filtering. 95th percentile, in millicores. | | `CPU System (P95)` | CPU spent by the OS for the query — disk reads, memory management, network I/O. 95th percentile, in millicores. | | `Memory (P95)` | Memory used during query execution. 95th percentile, in MiB. | | `Storage read (P95)` | Disk read operations caused by the query. 95th percentile, in ops/sec. | | `Storage write (P95)` | Disk write operations caused by the query. 95th percentile, in ops/sec. | | `Hypertables` | Whether the query ran on a hypertable. | | `Columnar tables` | Whether the query read data from a chunk in the columnstore. | | `User name` | The database user who ran the query. | These metrics calculations are based on the entire period you've selected. For example, if you've selected six hours, all the metrics represent an aggregation of the previous six hours of executions. Note If you have just completed a query, it can take some minutes for it to show in the table. Wait a little, then refresh the page to see your query. Check out the last update value at the top of the query table to identify the timestamp from the last processed query stat. Click a query in the list to see the drill-down view. This view not only helps you identify spikes and unexpected behaviors, but also offers information to optimize your query. ![Queries drill-down view](/docs/_astro/query-drill-down-view-tiger-console.DQQSXNNW_S2949.webp) This view includes the following graphs: * `Execution time`: the median and P95 query execution times over the selected period. This is useful to understand the consistency and efficiency of your query's execution over time. * `EXPLAIN` plan: for queries that take more than 10 seconds to execute, there is an EXPLAIN plan collected automatically. * `Rows`: the impact of your query on rows over time. If it's a `SELECT` statement, it shows the number of rows retrieved, while for an `INSERT/UPDATE` statement, it reflects the rows inserted. * `Plans and executions`: the number of query plans and executions over time. You can use this to optimize query performance, helping you assess if you can benefit from prepared statements to reduce planning overhead. * `Shared buffers hit and miss`: shared buffers play a critical role in PostgreSQL's performance by caching data in memory. A shared buffer hit occurs when the required data block is found in the shared buffer memory, while a miss indicates that PostgreSQL couldn't locate the block in memory. A miss doesn't necessarily mean a disk read, because PostgreSQL may retrieve the data from the operating system's disk pages cache. If you observe a high number of shared buffer misses, your current shared buffers setting might be insufficient. Increasing the shared buffer size can improve cache hit rates and query speed. * `Cache hit ratio`: measures how much of your query's data is read from shared buffers. A 100% value indicates that all the data required by the query was found in the shared buffer, while a 0% value means none of the necessary data blocks were in the shared buffers. This metric provides a clear understanding of how efficiently your query leverages shared buffers, helping you optimize data access and database performance. * `CPU time (user)`: CPU time the query spends on its own work — parsing, sorting, filtering, computing results. A spike here means the query is doing more computation than expected. Check for missing indexes, unnecessary joins, or large sequential scans. * `CPU time (system)`: CPU time the OS spends servicing the query — disk reads, memory management, network I/O. High values point to heavy disk access or memory pressure rather than query logic. * `Memory allocated`: memory the query uses during execution. Sudden increases often come from inefficient joins, large sorts spilling to disk, or growing result sets. If usage stays near the service's limits, optimize the query or provision more resources. * `Storage IO bytes read`: data the query reads from disk over time. High read volumes point to full table scans or queries accessing more data than necessary. Adding indexes or narrowing the time range helps reduce reads. * `Storage IO bytes written`: data the query writes to disk over time. This includes inserts, updates, and temporary files from sorts or joins. Unusually high writes often mean large intermediate results are spilling to disk. ## Jobs Tiger Cloud summarizes all [jobs](/docs/build/data-management/about-automation) set up for your service along with their details like type, target object, and status. This includes native Tiger Cloud jobs as well as custom jobs you configure based on your specific needs. 1. **View jobs** To view jobs, select your service in Tiger Console, then click `Monitoring` > `Jobs`: ![Jobs](/docs/_astro/tiger-console-jobs-table.BJLLWbId_2w8JGD.webp) Click `Timeline` at the top right to see the recent job runs with their statuses. Select the timeframe from 1 to 24 hours: ![Jobs timeline](/docs/_astro/tiger-console-jobs-timeline-view.B9LcH-xP_Z2ii5eF.webp) 2. **View job details** Click a job ID in the list to view its config and run history: ![Job details](/docs/_astro/tiger-console-job-details.M9gynlga_1olJiS.webp) 3. **Edit the job config** Click the pencil icon to edit the job config: ![Update job config](/docs/_astro/tiger-console-edit-job.o--VQZER_1Ecg7.webp) ## Connections Tiger Cloud lists current and past connections to your service. This includes details like the corresponding query, connecting application, username, connection status, start time, and duration. To view connections, select your service in Tiger Console, then click `Monitoring` > `Connections`. Expand the query underneath each connection to see the full SQL. ![Connections](/docs/_astro/tiger-console-service-connections.B7ohsR73_ZXvDaW.webp) Click the trash icon next to a connection in the list to terminate it. A lock icon means that a connection cannot be terminated; hover over the icon to see the reason. ## Recommendations Tiger Cloud offers specific tips on configuring your service. This includes a wide range of actions, from finishing account setup to tuning your service for the best performance. For example, Tiger Cloud may recommend a more suitable chunk interval or draw your attention to consistently failing jobs. To view recommendations, select your service in Tiger Console, then click `Monitoring` > `Recommendations`: ![Recommendations](/docs/_astro/recommendations-tiger-cloud.DthX8HUN_Z1CcH5B.webp) ## Query-level statistics with `pg_stat_statements` You can also get query-level statistics for your services with the `pg_stat_statements` extension. This includes the time spent planning and executing each query; the number of blocks hit, read, and written; and more. `pg_stat_statements` comes pre-installed with Tiger Cloud. Note For more information about `pg_stat_statements`, see the [PostgreSQL documentation](https://www.postgresql.org/docs/current/pgstatstatements.html). Query the `pg_stat_statements` view as you would any PostgreSQL view. The full view includes superuser queries used by Tiger Cloud to manage your service in the background. To view only your queries, filter by the current user. [Connect](/docs/get-started/quickstart/create-service#connect-to-your-service) to your service and run the following command: ```sql SELECT * FROM pg_stat_statements WHERE pg_get_userbyid(userid) = current_user; ``` For example, to identify the top five longest-running queries by their mean execution time: ```sql SELECT calls, mean_exec_time, query FROM pg_stat_statements WHERE pg_get_userbyid(userid) = current_user ORDER BY mean_exec_time DESC LIMIT 5; ``` Or the top five queries with the highest relative variability in the execution time, expressed as a percentage: ```sql SELECT calls, stddev_exec_time/mean_exec_time*100 AS rel_std_dev, query FROM pg_stat_statements WHERE pg_get_userbyid(userid) = current_user ORDER BY rel_std_dev DESC LIMIT 5; ``` For more examples and detailed explanations, see the [blog post on identifying performance bottlenecks with `pg_stat_statements`](https://www.tigerdata.com/blog/using-pg-stat-statements-to-optimize-queries). # Billing and account management > Manage billing, pricing plans, and account settings for Tiger Cloud on Azure As we enhance our offerings and align them with your evolving needs, pricing plans provide more value, flexibility, and efficiency for your business. Whether you're a growing startup or a well-established enterprise, our plans are structured to support your journey towards greater success. ![pricing plans](/docs/_astro/tiger-pricing.D4VriqHo_Z1ITAcw.webp) ![pricing plans](/docs/_astro/tiger-pricing-dark.CAPZJelR_1rI8TC.webp) This page explains pricing plans for Tiger Cloud, and how to easily manage your Tiger Cloud account. Pricing plans give you: * **Enhanced performance**: with increased CPU and storage capacities, your apps run smoother and more efficiently, even under heavy loads. * **Improved scalability**: as your business grows, so do your demands. Pricing plans scale with you, they provide the resources and support you need at each stage of your growth. Scale up or down based on your current needs, ensuring that you only pay for what you use. * **Better support**: access to enhanced support options, including production support and dedicated account management, ensures you have the help you need when you need it. * **Greater flexibility**: we know that one size doesn't fit all. Pricing plans give you the flexibility to choose the features and support levels that best match your business and engineering requirements. The ability to add features like I/O Boost and customize your pricing plan means you can tailor Tiger Cloud services to fit your specific needs. * **Cost efficiency**: by aligning our pricing with the value delivered, we ensure that you get the most out of every dollar spent. Our goal is to help you achieve more with less. It’s that simple! You don't pay for automated backups or networking costs, such as data ingest or egress. There are no per-query fees, nor additional costs to read or write data. It's all completely transparent, easily understood, and up to you. Using self-hosted TimescaleDB and our open-source products is still free. If you create a Tiger Cloud account from Azure Marketplace, the pricing options are pay-as-you-go and annual commit. See [Azure pricing](#azure-marketplace-pricing) for details. ## Disaggregated, consumption-based compute and storage [Section titled “Disaggregated, consumption-based compute and storage”](#disaggregated-consumption-based-compute-and-storage) With Tiger Cloud, you are not limited to pre-set compute and storage. Get as much as you need when provisioning your services or later, as your needs grow. * **Compute**: pay only for the compute resources you run. Compute is metered on an hourly basis, and you can [scale it up to 64 CPU](/docs/deploy/tiger-cloud/tiger-cloud-azure/service-management/change-resources) at any time. You can also [scale out using replicas](/docs/deploy/tiger-cloud/tiger-cloud-azure/high-availability/read-scaling) as your application grows. We also provide services to help you lower your compute needs while improving query performance. Tiger Cloud is very efficient and generally needs less compute than other databases to deliver the same performance. The best way to size your needs is to sign up for a $1000 credit, 30-day free trial and test with a realistic workload. * **Storage**: pay only for the storage you consume. You have high-performance storage for more frequently accessed data, and [low-cost bottomless blob storage](/docs/build/data-management/storage/manage-storage) for older or less frequently accessed data. The high-performance storage offers you up to 64 TB of compressed (typically 80-100 TB uncompressed) data and is metered on your average GB consumption per hour. We can help you compress your data by up to 98% so you pay even less. For low-cost storage, Tiger Data charges for data tiered based on its original uncompressed size in the high-performance storage tier. There are no additional expenses, such as data transfer or compute. For easy upgrades, each service stores the TimescaleDB binaries. This contributes up to 900 MB to overall storage, which amounts to less than $.80/month in additional storage costs. ## How your bill is calculated [Section titled “How your bill is calculated”](#how-your-bill-is-calculated) You are billed at the end of each month in arrears. Your monthly invoice includes an itemized cost accounting for each Tiger Cloud service and any additional charges. Tiger Cloud charges are based on consumption and your pricing plan: * **Compute**: billed and metered on an hourly basis. This means that you are billed for a full hour even if the actual consumption is less. You can scale compute up and down at any time. If the compute config changes mid-hour, you are billed for the config used at the end of that hour. * **Storage**: billed and metered on a quarter of an hour basis. Storage grows and shrinks automatically with your data. For example, over the last month your Tiger Cloud service has been running compute for 500 hours total: * 375 hours with 2 CPU * 125 hours 4 CPU and consumed high-performance storage for 720 hours total: * 200 hours with 100 GB * 520 hours with 150 GB **Compute cost** = (`375` x `hourly price for 2 CPU`) + (`125` x `hourly price for 4 CPU`) **High-performance storage cost** = (`200` x `100 GB` x `hourly price per GB`) + (`520` x `150 GB` x `hourly price per GB`) Some add-ons such as tiered storage, HA replicas, and connection pooling may incur additional charges. These charges are clearly marked in your billing snapshot in Tiger Console. Tips You are charged for all active standard services in your account, even if you are not actively using them. To reduce costs, pause or delete your unused services. ## Use Tiger Cloud for free [Section titled “Use Tiger Cloud for free”](#use-tiger-cloud-for-free) Are you just starting out with Tiger Cloud? Activate a $1000-credit, 30-day trial of our Performance or Scale plan (no credit card required). After 30 days, any remaining credits expire and we may remove your data unless you've added a payment method. After you have completed your trial, choose the [pricing plan](pricing-and-account-management#features-included-in-each-pricing-plan) that suits your business and engineering needs. Two free services are included for all plans. Note Free services are currently in beta. If you want to try out features in a higher pricing plan before upgrading, contact us. ## Upgrade or downgrade your pricing plans at any time [Section titled “Upgrade or downgrade your pricing plans at any time”](#upgrade-or-downgrade-your-pricing-plans-at-any-time) You can upgrade or downgrade between the Performance and Scale plans whenever you want using [Tiger Console](https://console.cloud.tigerdata.com/). To move to Enterprise, [get in touch with Tiger Data](https://www.tigerdata.com/contact/). If you switch your pricing plan mid-month, your prices are prorated to when you switch. Your services are not interrupted when you switch, so you can keep working without any hassle. ## Monitor usage and costs You keep track of your monthly usage in [Tiger Console](https://console.cloud.tigerdata.com/dashboard/billing/details). Console shows your resource usage and dashboards with performance insights. This allows you to closely monitor your services’ performance, and any need to scale your services or upgrade your pricing plan. Console also shows your month-to-date accrued charges, as well as a forecast of your expected month-end bill. Your previous invoices are also available as PDFs for download. Tips You are charged for all active standard services in your account, even if you are not actively using them. To reduce costs, pause or delete your unused services. ## Tiger Cloud support [Section titled “Tiger Cloud support”](#tiger-cloud-support) Tiger Data runs a global support organization with Customer Satisfaction (CSAT) scores above 99%. Support covers all timezones and is fully staffed at weekend hours. All paid pricing plans have free Developer Support through email with a target response time of 1 business day; we are often faster. If you need 24x7 responsiveness, talk to us about [Production Support](https://www.tigerdata.com/support). You can open, view, reply to, and close support tickets from the `Support` tab in Tiger Console: ![Manage support in Tiger Cloud](/docs/_astro/tiger-cloud-manage-support.oAio5bba_Z1Ot3bO.webp) ## Charging for HA and read replicas [Section titled “Charging for HA and read replicas”](#charging-for-ha-and-read-replicas) HA and read replicas are both charged at the same rate as your primary services, based on the compute and primary storage consumed by your replicas. Data tiered to our bottomless storage tier is shared by all database replicas; replicas accessing tiered storage do not add to your bill. ## Charging over regions [Section titled “Charging over regions”](#charging-over-regions) Storage is priced the same across all regions. However, compute prices vary depending on the region. This is because our cloud provider prices infrastructure differently based on region. ## Features included in each pricing plan [Section titled “Features included in each pricing plan”](#features-included-in-each-pricing-plan) The available pricing plans are: * **Performance**: for cost-focused, smaller projects. No credit card required to start. * **Scale**: for developers handling critical and demanding apps. * **Enterprise**: for enterprises with mission-critical apps. The features included in each [pricing plan](https://www.tigerdata.com/pricing) are: | Feature | Performance | Scale | Enterprise | | -------------------------------------------------------------------------- | ------------------------------------ | ------------------------------------------------ | ------------------------------------------------ | | **Compute and storage** | | | | | Number of services | Up to 2 free and 4 standard services | Up to 2 free and and unlimited standard services | Up to 2 free and and unlimited standard services | | CPU limit per service | Up to 8 CPU | Up to 32 CPU | Up to 64 CPU | | Memory limit per service | Up to 32 GB | Up to 128 GB | Up to 256 GB | | Storage limit per service | Up to 64 TB | Up to 64 TB | Up to 64 TB | | Bottomless blob storage | | Unlimited | Unlimited | | Independently scale compute and storage | Standard services only | Standard services only | Standard services only | | **Data services and workloads** | | | | | Relational | ✓ | ✓ | ✓ | | Time-series | ✓ | ✓ | ✓ | | Vector search | ✓ | ✓ | ✓ | | Tiger Cloud SQL editor | 3 seats | 10 seats | 20 seats | | Charts | ✓ | ✓ | ✓ | | Dashboards | 2 | Unlimited | Unlimited | | **Storage and performance** | | | | | IOPS (autoscale) | 3,000 - 5,000 | 5,000 - 8,000 | 5,000 - 8,000 | | Bandwidth (autoscale) | 125 - 250 MB/s | 250 - 500 MB/s | 250 - 500 MB/s | | I/O Boost | | Add-on: Up to 40,000 IOPS, 1200 MB/s BW | Add-on: Up to 80,000 IOPS, 1200 MB/s BW | | **Availability and monitoring** | | | | | High-availability replicas (Automated multi-AZ failover) | ✓ | ✓ | ✓ | | Read replicas | | ✓ | ✓ | | Backup reports | | 14 days | 14 days | | Point-in-time recovery and forking | 3 days | 14 days | 14 days | | Performance insights | ✓ | ✓ | ✓ | | Metrics and log exporters | | ✓ | ✓ | | **Security and compliance** | | | | | Role-based access | ✓ | ✓ | ✓ | | Azure Private Link | | ✓ | ✓ | | End-to-end encryption | ✓ | ✓ | ✓ | | [HIPAA compliance](https://www.hhs.gov/hipaa/for-professionals/index.html) | | | ✓ | | IP address allow list | 1 list with up to 10 IP addresses | Up to 10 lists with up to 10 IP addresses each | Up to 10 lists with up to 100 IP addresses each | | Multi-factor authentication | ✓ | ✓ | ✓ | | Federated authentication (SAML) | | | ✓ | | SOC 2 Type 2 report | | ✓ | ✓ | | Penetration testing report | | | ✓ | | Security questionnaire and review | | | ✓ | | Pay by invoice | Available at minimum spend | Available at minimum spend | ✓ | | [Uptime SLAs](https://www.tigerdata.com/legal/terms-of-service) | Standard | Standard | Enterprise | | **Support and technical services** | | | | | Community support | ✓ | ✓ | ✓ | | Email support | ✓ | ✓ | ✓ | | Production support | Add-on | Add-on | ✓ | | Named account manager | | | ✓ | | JOIN services (Jumpstart Onboarding and INtegration) | | Available at minimum spend | ✓ | For a personalized quote, [get in touch with Tiger Data](https://www.tigerdata.com/contact/). ## Manage your Tiger Cloud pricing plan [Section titled “Manage your Tiger Cloud pricing plan”](#manage-your-tiger-cloud-pricing-plan) You handle all details about your Tiger Cloud project including updates to your pricing plan, payment methods, and add-ons in the [billing section in Tiger Console](https://console.cloud.tigerdata.com/dashboard/billing/details): ![Adding a payment method in Timescale](/docs/_astro/tiger-cloud-billing.DFbo-8aP_1nHIRb.webp) * **Details**: an overview of your pricing plan, usage, and payment details. You can add up to three credit cards to your `Wallet`. If you prefer to pay by invoice, [contact Tiger Data](https://www.tigerdata.com/contact/) and ask to change to corporate billing. You can also request early access to paying with crypto. * **Emails**: the addresses Tiger Data uses to communicate with you. Payment confirmations and alerts are sent to the email address you signed up with. Add another address to send details to other departments in your organization. * **History**: the list of your downloadable Tiger Cloud invoices. * **Plans**: choose the pricing plan supplying the [features](pricing-and-account-management#features-included-in-each-pricing-plan) that suit your business and engineering needs. * **Add-ons**: add `Production support` and improved database performance for mission-critical workloads. ## Azure Marketplace pricing [Section titled “Azure Marketplace pricing”](#azure-marketplace-pricing) When you get Tiger Cloud at Azure Marketplace, the following pricing options are available: * **Pay-as-you-go**: your consumption is calculated at the end of the month and included in your Azure invoice. No upfront costs, standard Tiger Cloud rates apply. * **Annual commit**: your consumption is calculated at the end of the month ensuring predictable pricing and seamless billing through your Azure account. We confirm the contract terms with you before finalizing the commitment. Looking for a private offer? [Get in touch](mailto:sales@tigerdata.com) with our sales team. # Private endpoints > Connect your services to Azure Private Link endpoints to eliminate public internet exposure Tiger Cloud supports private connectivity through Azure Private Link. Once configured, applications in your Azure Virtual Network reach your service through a private endpoint on Microsoft's internal network, with no exposure to the public internet. This page walks you through authorizing your Azure subscription in Tiger Console, provisioning the private endpoint in Azure, and attaching it to your service. ## Prerequisites for this procedure To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service). - An [Azure account](https://azure.microsoft.com/en-gb/pricing/purchase-options/azure-account) with an active subscription. - [Permissions](https://learn.microsoft.com/en-us/azure/private-link/rbac-permissions) configured to create private endpoints. - An [Azure Virtual Network](https://learn.microsoft.com/en-us/azure/virtual-network/quickstart-create-virtual-network?tabs=portal) with a subnet for the resources you will connect to Tiger Cloud. ## Set up a private endpoint connection Take the following steps to connect your Tiger Cloud service to a Private Link endpoint. 1. **Create an Azure subscription authorization** 1. In [Tiger Console](https://console.cloud.tigerdata.com/dashboard/azure-privatelink), select `Security` > `Private Endpoints` > `Configure Private Endpoint Connection`. ![Azure Private Link authorization](/docs/_astro/configure-private-link-authorization.BVll5xeW_2pKWBB.webp) 2. In `Cloud provider`, select `Azure`. 3. In `Principal ID`, enter your [Azure subscription ID](https://learn.microsoft.com/en-us/azure/azure-portal/get-subscription-tenant-id#find-your-azure-subscription). Give your authorization a name, for convenience. Warning Click the checkmark next to `Connection name` to save your authorization. Otherwise, your input is discarded. 4. Under `Alias`, copy the alias for the region in which you need to create the connection. Choose the region closest to your Azure resources for optimal performance. 5. Click `Done`. Tiger Cloud confirms your authorization. Once it is confirmed, you can connect multiple endpoints from the same authorized subscription. 2. **Create a private endpoint in Azure** 1. In [Azure Portal](https://azure.microsoft.com/en-us/get-started/azure-portal), go to `Private endpoints` and click `Create`. ![Create Azure Private Endpoint](/docs/_astro/create-private-endpoint-azure.BEjIkaB7_Z13XlPp.webp) 2. In `Subscription`, select the subscription you have previously authorized in Tiger Cloud. 3. In `Resource group`, select an existing resource group or create a new one for your private endpoint. 4. Provide a name for your endpoint. 5. Select the region where your Virtual Network is deployed, then click `Next: Resource`. 6. In `Connection method`, select `Connect to an Azure resource by resource ID or alias`. 7. In `Resource ID or alias`, paste the alias you have copied from Tiger Console. 8. In `Request message`, enter your [Tiger Cloud project ID](/docs/integrate/find-connection-details), then click `Next: Virtual Network`. 9. Select the Virtual Network that contains the resources you want to connect to Tiger Cloud, then choose the subnet where the endpoint's private IP will be created. Optionally select an application security group, then click `Next: DNS`. 10. Optionally configure private DNS integration and tags for your endpoint, then click `Next: Review + create`. 11. Review your config and click `Create`. Azure creates your private endpoint. Wait for the deployment to succeed. 12. Go to `Private endpoints` and copy the private endpoint IP from the `Private IP` column. 3. **Sync the connection** 1. In [Tiger Console](https://console.cloud.tigerdata.com/dashboard/azure-privatelink) > `Security` > `Private Endpoints`, click `Refresh`. Tiger Cloud automatically approves connections from authorized subscriptions. Your connection appears in the list. ![Azure Private Endpoint IP](/docs/_astro/private-link-add-ip.CXR8Gl7x_Z2lS5se.webp) 2. Under `IP Address` click `Add IP` and paste the private endpoint IP address you have copied from Azure Portal. 3. Under `Services` click `Attach service`. Select your service from the dropdown and click `Attach`. You can attach a service to one private endpoint. 4. From a VM inside your Azure VNet, connect to your service using a connection string with your [connection details](/docs/integrate/find-connection-details). You should be able to connect successfully. ## Manage connections * To detach a service from a private endpoint connection, go to `Security` > `Private Endpoints`, expand the arrow in the `Services` column, and click the trash icon next to the service connection string. * To edit or remove an endpoint connection, go to `Security` > `Private Endpoints` and click the three dots next to the connection in the list. Select `Edit` or `Disconnect`, respectively. You need to detach all services from a private endpoint connection before deleting it. * To remove an authorization, click `Manage Authorizations` > trash bin icon. You need to disconnect all relevant endpoint connections before removing an authorization. # Client credentials > Create programmatic credentials to access Tiger Cloud without a password You can use client credentials to programmatically access resources instead of using your username and password. You can generate multiple client credentials for different applications or use cases rather than a single set of user credentials for everything. ## Create client credentials When you create client credentials, a public key and a private key are generated. These keys act as the username and password for programmatic client applications. It is important that you save these keys in a safe place. You can also delete these client credentials when the client applications no longer need access to Tiger Cloud resources. For more information about obtaining an access token programmatically, see the [Tiger Cloud Terraform provider documentation](https://registry.terraform.io/providers/timescale/timescale/latest/docs). 1. **Log in to Tiger Console and click `Settings`** 2. **Click `Create credentials`** 3. **In the `New client credentials` dialog, you can view the `Public key` and the `Secret Key`** Copy your secret key and store it in a secure place. You won't be able to view the `Secret Key` again in Tiger Console. 4. **Click `Done`** You can use these keys in your client applications to access Tiger Cloud resources inside the respective project. Tiger Cloud generates a default `Name` for the client credentials. 5. **Click the ⋮ menu and select `Rename credentials`** 6. **In the `Edit credential name` dialog, type the new name and click `Accept`** ## Delete client credentials 1. **Log in to Tiger Console and click `Settings`** 2. **Click the `⋮` menu of the client credential, and select `Delete`** 3. **In the `Are you sure` dialog, type the name of the client credential, and click `Delete`** # IP allow list > Restrict access to your services with IP address allow lists You can restrict access to your Tiger Cloud services to trusted IP addresses only. This prevents unauthorized connections without the need for a [Virtual Private Cloud](../security/vpc). Creating IP allow lists helps comply with security standards such as SOC 2 or HIPAA that require IP filtering. This is especially useful in regulated industries like finance, healthcare, and government. For a more fine-grained control, you create separate IP allow lists for [service and data management](/docs/get-started/quickstart/create-service). ## Create and attach an IP allow list for service management You create an IP allow list at the [project level](../security/members), then attach your service to it. Important You attach a service to either one VPC, or one IP allow list. You cannot attach a service to a VPC and an IP allow list at the same time. 1. **In Tiger Console, select `Security` > `IP Allow List`, then click `Create IP Allow List`** ![Create IP allow list](/docs/_astro/create-ip-allow-list-tiger-console.CQABvOHi_KyQo1.webp) 2. **Enter your trusted IP addresses** The number of IP addresses that you can include in one list depends on your [pricing plan](../pricing-and-account-management). ![Add IP addresses to allow list](/docs/_astro/add-ip-addresses-to-allow-list-tiger-console.Ds8zI430_217RCA.webp) 3. **Name your allow list and click `Create IP Allow List`** Click `+ Create IP Allow List` to create another list. The number of IP allow lists you can create depends on your [pricing plan](../pricing-and-account-management). 4. **Select a Tiger Cloud service, then click `Operations` > `Security` > `IP Allow List`** ![Attach IP allow list](/docs/_astro/attach-ip-allow-list-tiger-console.DAd9QaHC_CQzmd.webp) 5. **Select the list in the drop-down and click `Apply`** 6. **Type `Apply` in the confirmation popup** You have created and attached an IP allow list for service management operations. You can unattach or change the list attached to a service from the same tab. ## Create an IP allow list for querying your data You create an IP allow list in the Data view settings. 1. **In Tiger Console, toggle `Data view` at the top right** 2. **Click the cog icon to open `Preferences` > `Settings`** 3. **Log into PopSQL** 4. **Scroll down and toggle `IP Allowlist`** 5. **Add IP addresses** 1. Click `Add entry`. 2. Enter an IP address or a range of IP addresses. 3. Click `Add`. 4. When all the IP addresses have been added, click `Apply`. 5. Click `Confirm`. You have successfully added an IP allow list for querying your service in Data view. # Control user access to projects > Manage project roles, add and remove users, and transfer ownership When you sign up for a [$1000-credit, 30-day free trial](https://console.cloud.tigerdata.com/), Tiger Cloud creates a project with built-in role-based access. A single email address can create only one project. To work across multiple projects, ask an Owner to add you to their project. Role-based access includes the following roles: * **Owner**: Tiger Cloud assigns this role to you when your project is created. As the Owner, you can add and delete other users, transfer project ownership, administer services, and edit project settings. * **Admin**: the Owner assigns this role to other users in the project. A user with the Admin role has the same scope of rights as the Owner but cannot transfer project ownership. * **Developer**: the Owner and Admins assign this role to other users in the project. A Developer can build, deploy, and operate services across projects, but does not have administrative privileges over users, roles, or billing. A Developer can invite other users to the project, but only with the Viewer role. * **Viewer**: the Owner and Admins assign this role to other users in the project. A Viewer has limited, read-only access to Tiger Console. This means that a Viewer cannot modify services and their configurations in any way. A Viewer has no access to Data view and has read-queries-only access to SQL editor. ![Project users in Console](/docs/_astro/tiger-console-project-roles-overview.YQXi0kOj_2pBFfD.webp) If you have the [Enterprise pricing plan](../pricing-and-account-management), you can use your company [SAML](../security/saml) identity provider to log in to Console. Note User roles in a Tiger Cloud project do not overlap with the database-level roles for the individual services. This page describes the project roles available in Console. For the database-level user roles, see [Manage data security in your Tiger Cloud service](../security/read-only-role). ## Add a user to your project New users do not need to have a Tiger Cloud account before you add them, they are prompted to create one when they respond to the confirmation email. Existing users join a project in addition to the other projects they are already members of. To add a user to a project: 1. **In Tiger Console, click `Users`, then click `Add new user`** 2. **Type the email address of the person that you want to add, select their role, and click `Invite user`** [Enterprise pricing plan](../pricing-and-account-management) and SAML users receive a notification in Console. Users in the other pricing plans receive a confirmation email. The new user then [joins the project](#join-a-project). ## Join a project When you are asked to join a project, Tiger Console sends you an invitation email. Follow the instructions in the invitation email to join the project: * New Tiger Cloud user 1. **In the invitation email, click `Accept Invite`** Tiger Cloud opens. 2. **Follow the setup wizard and create a new account** You are added to the project you were invited to. * Existing Tiger Cloud user 1. **In the invitation email, click `Accept Invite`** Tiger Console opens, and you are added to the project. * Enterprise plan and SAML user 1. **Log in to Console using your company's identity provider** 2. **Click `Notifications`, then accept the invitation** Tiger Console opens, and you are added to the project. As you are now included in more than one project, you can easily [change projects](#change-your-current-project). ## Resend a project invitation Project invitations are valid for 7 days. To resend a project invitation: 1. **In Tiger Console, click `Users`** 2. **Next to the person you want to invite to your project, click `Resend invitation`** ## Change your current project Note A single email address can create only one project in Console. To work across multiple projects, ask an Owner to [add you to their project](#add-a-user-to-your-project). To change the project you are currently working in: 1. **In Tiger Console, click the arrows next to the project name in the top left** ![Change project in Console](/docs/_astro/tiger-console-change-project.D-LN4vwi_1pfWTk.webp) 2. **Select the project you want to use** ## Transfer project ownership Each Tiger Cloud project has one Owner. As the project Owner, you have rights to add and delete users, edit project settings, and transfer the Owner role to another user. When you transfer ownership to another user, you lose your ownership rights. To transfer project ownership: 1. **In Tiger Console, click `Users`** 2. **Next to the person you want to transfer project ownership to, click `⋮` > `Transfer project ownership`** If you are unable to transfer ownership, hover over the greyed out button to see the details. 3. **Enter your password, and click `Verify`** 4. **Complete the two-factor authentication challenge and click `Confirm`** If you have the [Enterprise pricing plan](../pricing-and-account-management), and log in to Tiger Cloud using [SAML authentication](../security/saml) or have not enabled [two-factor authentication](../security/multi-factor-authentication), [contact support](https://www.tigerdata.com/contact) to transfer project ownership. ## Leave a project To stop working in a project: 1. **In Tiger Console, click `Users`** 2. **Click `⋮` > `Leave project`, then click `Leave`** Your account is removed from the project immediately, you can no longer access this project. ## Change roles of other users in a project The Owner can change the roles of all users in the project. An Admin can change the roles of all users other than the Owner. Developer and Viewer cannot change the roles of other users. To change the role for another user: 1. **In Tiger Console, click `Users`** 2. **Next to the corresponding user, select another role in the dropdown** The user role is changed immediately. ## Remove users from a project To remove a user's access to a project: 1. **In Tiger Console, click `Users`** 2. **Next to the person you want to remove, click `⋮` > `Remove`** 3. **In `Remove user`, click `Remove`** The user is deleted immediately, they can no longer access your project. # Multi-factor authentication > Set up two-factor authentication for your Tiger Cloud account You can use two-factor authentication to log in to your Tiger Cloud account. Two-factor authentication, also known as two-step verification or 2FA, enables secure logins that require an authentication code in addition to your user password. The code is provided by an authenticator app on your mobile device. There are multiple authenticator apps available. ![2FA](/docs/_astro/tiger-console-2fa.BcA5vlKw_Mwqle.webp) This page describes how to configure two-factor authentication with Google Authenticator. ## Prerequisites for this procedure To follow these steps, you'll need: * The [Google Authenticator application](https://support.google.com/accounts/answer/1066447) installed on your mobile device. ## Configure two-factor authentication with Google Authenticator Take the following steps to configure two-factor authentication: 1. **Log in to Tiger Console with your username and password. 2FA is not available if you log in with Google SSO.** 2. **Click your user name in the top right of Tiger Console and select `Account settings`** 3. **Click `Add two-factor authentication`** 4. **On your mobile device, open Google Authenticator, tap `+`, and select `Scan a QR code`** 5. **Scan the QR code provided by Tiger Console in `Connect to an authenticator app` and click `Next`** 6. **In Tiger Console, enter the verification code provided by Google Authenticator, and click `Next`** 7. **In `Save your recovery codes`, copy, download, or print the recovery codes** These are used to recover your account if you lose your device. 8. **Verify that you have saved your recovery codes, by clicking `OK, I saved my recovery codes`** 9. **If two-factor authentication is enabled correctly, an email notification is sent to you** Warning If you lose access to the mobile device you use for multi-factor authentication, and you do not have access to your recovery codes, you cannot sign in to your Tiger Cloud account. To regain access to your account, contact . ## Regenerate recovery codes If you do not have access to your authenticator app and need to log in to Tiger Console, you can use your recovery codes. Recovery codes are single-use. If you've used all 10 recovery codes, or lost access to them, you can generate another list. Generating a new list invalidates all previously generated codes. 1. **Log in to Tiger Console with your username and password** 2. **Click your user name in the top right and select `Account settings`** 3. **In `Two-factor authentication`, click `Regenerate recovery codes`** 4. **Enter the verification code from your authenticator app** Alternatively, if you do not have access to the authenticator app, click `Use recovery code instead` to enter a recovery code. 5. **Click `Next`** 6. **In `Save your recovery codes`, copy, download, or print the recovery codes** These are used to recover your account if you lose your device. 7. **Verify that you have saved your recovery codes, by clicking `OK, I saved my recovery codes`** ## Remove two-factor authentication If you need to enroll a new device for two-factor authentication, you can remove two-factor authentication from your account and then add it again with your new device. 1. **Log in to Tiger Console with your username and password** 2. **Click your user name in the top right of Tiger Console and select `Account settings`** 3. **In `Two-factor authentication`, click `Remove two-factor authentication`** 4. **Enter the verification code from your authenticator app to confirm** Alternatively click `Use recovery code instead` to type the recovery code. 5. **Click `Remove`** # About security in Tiger Cloud > Learn how Tiger Cloud protects your PostgreSQL data with encryption, access controls, and network security Protecting data starts with secure software engineering. At Tiger Data, we embed security into every stage of development, from static code analysis and automated dependency scanning to rigorous code security reviews. To go even further, we developed [pgspot](https://github.com/timescale/pgspot), an open-source extension to identify security issues with PostgreSQL extensions, which strengthens the broader ecosystem as well as our own platform. Tiger Data products do not have any identified weaknesses. ![Tiger Cloud platform security architecture overview](/docs/_astro/tiger-platform-security-overview.DexpnyhU_Z2vdcGA.webp) ![Tiger Cloud platform security architecture overview](/docs/_astro/tiger-platform-security-overview-dark.DE5djrdr_794D7.webp) This page lists the additional things we do to ensure operational security and to lock down Tiger Cloud services. To see our security features at a glance, see [Tiger Data Security](https://www.tigerdata.com/security). ## Role-based access [Section titled “Role-based access”](#role-based-access) Tiger Cloud provides role-based access for you to: * Administer your Tiger Cloud project In Tiger Console, users with the Owner, Admin, and Viewer roles have different permissions to manage users and services in the project. * Manage data in each service To restrict access to your data on the database level, you can create other roles on top of the default tsdbadmin role. ## Data encryption [Section titled “Data encryption”](#data-encryption) Your data on Tiger Cloud is encrypted both in transit and at rest. Both active databases and backups are encrypted. Tiger Cloud uses Azure as its cloud provider, with all the security that Azure provides. Data encryption uses the industry-standard AES-256 algorithm. Cryptographic keys are managed by [Azure Key Vault](https://azure.microsoft.com/en-us/products/key-vault). Keys are never stored in plaintext. For more information about Azure security, see the Azure documentation on security in [Azure Virtual Machines](https://learn.microsoft.com/en-us/azure/virtual-machines/security-overview) and [Azure Disk Encryption](https://learn.microsoft.com/en-us/azure/virtual-machines/disk-encryption-overview). ## Networking security [Section titled “Networking security”](#networking-security) Customer access to Tiger Cloud services is only provided over TLS-encrypted connections. There is no option to use unencrypted plaintext connections. ## Private endpoints [Section titled “Private endpoints”](#private-endpoints) Tiger Cloud enables you to connect to your services through a private endpoint in Azure using Azure Private Link. Traffic stays on Microsoft's internal network and never crosses the public internet. Once a service is attached, it is reachable only through the endpoint's private IP in your Azure Virtual Network, and communication can only be initiated from your Virtual Network to Tiger Cloud. ## IP address allow lists [Section titled “IP address allow lists”](#ip-address-allow-lists) You can allow only trusted IP addresses to access your Tiger Cloud services. You do this by creating [IP address allow lists](/docs/deploy/tiger-cloud/tiger-cloud-azure/security/ip-allow-list) and attaching them to your services. ## Operator access [Section titled “Operator access”](#operator-access) Normally all the resources required for providing Tiger Cloud services are automatically created, maintained and terminated by the Tiger Cloud infrastructure. No manual operator intervention is required. However, the Tiger Data operations team has the capability to securely log in to the service virtual machines for troubleshooting purposes. These accesses are audit logged. No customer access to the virtual machine level is provided. ## GDPR compliance [Section titled “GDPR compliance”](#gdpr-compliance) Tiger Data complies with the European Union's General Data Protection Regulation (GDPR), and all practices are covered by our [Privacy Policy](https://www.tigerdata.com/legal/privacy) and the [Terms of Service](https://www.tigerdata.com/legal/terms-of-service). All customer data is processed in accordance with Tiger Data's GDPR-compliant [Data Processor Addendum](https://www.tigerdata.com/legal/timescale-cloud-data-processing-addendum), which applies to all Tiger Data customers. Tiger Data operators never access customer data, unless explicitly requested by the customer to troubleshoot a technical issue. The Tiger Data operations team has mandatory recurring training regarding the applicable policies. ## HIPAA compliance [Section titled “HIPAA compliance”](#hipaa-compliance) The Tiger Cloud [Enterprise plan](/docs/deploy/tiger-cloud/tiger-cloud-azure/pricing-and-account-management#features-included-in-each-pricing-plan) is Health Insurance Portability and Accountability Act (HIPAA) compliant. This allows organizations to securely manage and analyze sensitive healthcare data, ensuring they meet regulatory requirements while building compliant applications. ## SOC 2 compliance [Section titled “SOC 2 compliance”](#soc-2-compliance) Tiger Cloud is SOC 2 Type 2 compliant. This ensures that organizations can securely manage customer data in alignment with industry standards for security, availability, processing integrity, confidentiality, and privacy. It helps businesses meet trust requirements while confidently building applications that handle sensitive information. The annual SOC 2 report is available to customers on the Scale or Enterprise pricing plans. Open a [support ticket](https://console.cloud.tigerdata.com/dashboard/support) to get access to it. # Manage data security in your service > Restrict data access with read-only PostgreSQL roles When you create a service, Tiger Cloud assigns you the tsdmadmin role. This role has full permissions to modify data in your service. However, Tiger Cloud does not provide superuser access. tsdmadmin is not a superuser. As tsdmadmin, you can use standard PostgreSQL means to create other roles or assign individual permissions. This page shows you how to create a read-only role for your database. Adding a read-only role does not provide resource isolation. To restrict the access of a read-only user, as well as isolate resources, create a [read replica](../high-availability/read-scaling) instead. Note The database-level roles for the individual services in your project do not overlap with the Tiger Cloud project user roles. This page describes the database-level roles. For user roles available in Console, see [Control user access to Tiger Cloud projects](../security/members). ## Create a read-only user You can create a read-only user to provide limited access to your database. 1. **Connect to your service as the tsdbadmin user** 2. **Create the new role** ```sql CREATE ROLE readaccess; ``` 3. **Grant usage on the schema to allow access to objects within it** ```sql GRANT USAGE ON SCHEMA TO readaccess; ``` 4. **Grant the appropriate permissions for the role, as required** For example, to grant `SELECT` permissions to a specific table, use: ```sql GRANT SELECT ON TO readaccess; ``` To grant `SELECT` permissions to all tables in a specific schema, use: ```sql GRANT SELECT ON ALL TABLES IN SCHEMA TO readaccess; ``` 5. **Create a new user** ```sql CREATE USER read_user WITH PASSWORD 'read_password'; ``` 6. **Assign the role to the new user** ```sql GRANT readaccess TO read_user; ``` # SAML authentication > Configure SAML single sign-on for Enterprise accounts Tiger Cloud offers SAML authentication as part of its [Enterprise](https://www.tigerdata.com/enterprise) offering. SAML (Security Assertion Markup Language) is an open standard for exchanging authentication and authorization data between parties. With SAML enabled Tiger Cloud customers can log into their Tiger Cloud account using their existing SSO service provider credentials. Note Tiger Cloud supports most SAML providers that can handle IDP-initiated login ### SAML offers many benefits for the Enterprise including: * Improved security: SAML centralizes user authentication with an identity provider (IdP). This makes it more difficult for attackers to gain access to user accounts. * Reduced IT costs: SAML can help companies reduce IT costs by eliminating the need to manage multiple user accounts and passwords. * Improved user experience: SAML makes it easier for users to access multiple applications and resources. Reach out to your CSM/sales contact to get started. The connection process looks like the following: 1. **Configure the IdP to support SAML authentication** This will involve creating a new application and configuring the IdP with the settings provided by your contact. 2. **Provide your contact with the requested details about your IdP** 3. **Test the SAML authentication process to make sure that it is working correctly** # Connect with a stricter SSL mode > Use verify-ca or verify-full SSL modes for stricter connection security The default connection string for Tiger Cloud uses the Secure Sockets Layer (SSL) mode `require`. Users can choose not to use Transport Layer Security (TLS) while connecting to their databases, but connecting to production databases without encryption is strongly discouraged. To achieve even stronger security, clients may select to verify the identity of the server. If you want your connection client to verify the server's identity, you can connect with an [SSL mode](https://www.postgresql.org/docs/current/libpq-ssl.html#LIBPQ-SSL-SSLMODE-STATEMENTS) of `verify-ca` or `verify-full`. To do so, you need to store a copy of the certificate chain where your connection tool can find it. This section provides instructions for setting up a stricter SSL connection. ## SSL certificates As part of the secure connection protocol, the server proves its identity by providing clients with a certificate. This certificate should be issued and signed by a well-known and trusted Certificate Authority. Because requesting a certificate from a Certificate Authority takes some time, Tiger Cloud services are initialized with a self-signed certificate. This lets you start up a service immediately. After your service is started, a signed certificate is requested behind the scenes. The new certificate is usually received within 30 minutes. Your certificate is then replaced with almost no interruption. Connections are reset, and most clients reconnect automatically. Free services do not supply SSL certificates. With the signed certificate, you can switch your connections to a stricter SSL mode, such as `verify-ca` or `verify-full`. For more information on the different SSL modes, see the [PostgreSQL SSL mode descriptions](https://www.postgresql.org/docs/current/libpq-ssl.html#LIBPQ-SSL-SSLMODE-STATEMENTS). ## Connect to your database with a stricter SSL mode To set up a stricter SSL connection: 1. Generate a copy of your certificate chain and store it in the right location 2. Change your Tiger Cloud connection string 1) **Use the `openssl` tool to connect to your Tiger Cloud service and get the certificate bundle** Store the bundle in a file called `bundle.crt`. Replace `$SERVICE_URL_WITH_PORT` with your Tiger Cloud connection URL: ```shell openssl s_client -showcerts -partial_chain -starttls postgres \ -connect $SERVICE_URL_WITH_PORT < /dev/null 2>/dev/null | \ awk '/BEGIN CERTIFICATE/,/END CERTIFICATE/{ print }' > bundle.crt ``` 2) **Copy the bundle to your clipboard** * MacOS ```shell pbcopy < bundle.crt ``` * Linux ```shell xclip -sel clip < bundle.crt ``` * Windows ```shell clip.exe < bundle.crt ``` 3) **Navigate to https\://whatsmychaincert.com/** This online tool generates a full certificate chain, including the root Certificate Authority certificate, which is not included in the certificate bundle returned by the database. 4) **Paste your certificate bundle in the provided box** Check `Include Root Certificate`. Click `Generate Chain`. 5) **Save the downloaded certificate chain to `~/.postgresql/root.crt`** 6) **Change your Tiger Cloud connection string from `sslmode=require` to either `sslmode=verify-full` or `sslmode=verify-ca`** For example, to connect to your database with `psql`, run: ```shell psql "postgres://tsdbadmin@$SERVICE_URL_WITH_PORT/tsdb?sslmode=verify-full" ``` ## Verify the certificate type used by your database To check whether the certificate has been replaced yet, connect to your database instance and inspect the returned certificate. We are using two certificate providers - Google and ZeroSSL, that's why chances are you can have a certificate issued by either of those CAs: ```shell openssl s_client -showcerts -partial_chain -starttls postgres -connect : < /dev/null 2>/dev/null | grep "Google\|ZeroSSL" ``` # About Tiger Cloud services > Create, manage, and scale Tiger Cloud PostgreSQL service instances on Azure Tiger Cloud is the modern PostgreSQL data platform for all your applications. It enhances PostgreSQL to handle time series, events, real-time analytics, and vector search, all in a single database alongside transactional workloads. You get one system that handles live data ingestion, late and out-of-order updates, and low latency queries, with the performance, reliability, and scalability your app needs. Ideal for IoT, crypto, finance, SaaS, and a myriad other domains, Tiger Cloud allows you to build data-heavy, mission-critical apps while retaining the familiarity and reliability of PostgreSQL. A Tiger Cloud service is a single optimized PostgreSQL instance extended with innovations in the database engine and cloud infrastructure to deliver speed without sacrifice. A Tiger Cloud service is 10-1000x faster at scale! It is ideal for applications requiring strong data consistency, complex relationships, and advanced querying capabilities. Get ACID compliance, extensive SQL support, JSON handling, and extensibility through custom functions, data types, and extensions. Each service is associated with a project in Tiger Cloud. Each project can have multiple services. Each user is a [member of one or more projects](/docs/deploy/tiger-cloud/tiger-cloud-aws/security/members). You create free and standard services in Tiger Console. A standard service comes with the full range of features according to your pricing plan. A free service comes at zero cost and gives you limited features and resources. ![Tiger Cloud pricing plans: Performance, Scale, and Enterprise](/docs/_astro/tiger-pricing.D4VriqHo_Z1ITAcw.webp) ![Tiger Cloud pricing plans: Performance, Scale, and Enterprise](/docs/_astro/tiger-pricing-dark.CAPZJelR_1rI8TC.webp) To the PostgreSQL you know and love, Tiger Cloud adds the following capabilities: * **Standard services**: * *Real-time analytics*: store and query [time-series data](https://www.tigerdata.com/blog/time-series-database-an-explainer#what-is-a-time-series-database) at scale for real-time analytics and other use cases. Get faster time-based queries with hypertables, continuous aggregates, and columnar storage. Save money by compressing data into the columnstore, moving cold data to low-cost bottomless storage in Amazon S3 or Azure Blob storage, and deleting old data with automated policies. * *AI-focused*: build AI applications from start to scale. Get fast and accurate similarity search with the pgvector and pgvectorscale extensions. * *Hybrid applications*: get a full set of tools to develop applications that combine time-based data and AI. All standard Tiger Cloud services include the tooling you expect for production and developer environments: [live migration](/docs/migrate/live-migration), [automatic backups and PITR](/docs/deploy/tiger-cloud/tiger-cloud-aws/high-availability/backup-restore), [high availability](/docs/deploy/tiger-cloud/tiger-cloud-aws/high-availability/high-availability), [read replicas](/docs/deploy/tiger-cloud/tiger-cloud-aws/high-availability/read-scaling), [data forking](/docs/deploy/tiger-cloud/tiger-cloud-aws/service-management/fork-services), [connection pooling](/docs/deploy/tiger-cloud/tiger-cloud-aws/service-management/connection-pooling), [tiered storage](/docs/build/data-management/storage/manage-storage), [usage-based storage](/docs/deploy/tiger-cloud/tiger-cloud-aws/pricing-and-account-management#how-your-bill-is-calculated), secure in-Tiger Console [SQL editing](/docs/build/data-management/run-queries-from-tiger-console), service [metrics](/docs/deploy/tiger-cloud/tiger-cloud-aws/monitoring#metrics) and [insights](/docs/deploy/tiger-cloud/tiger-cloud-aws/monitoring#insights), streamlined maintenance, and much more. Tiger Cloud continuously monitors your services and prevents common PostgreSQL out-of-memory crashes. * **Free services**: *PostgreSQL with TimescaleDB and vector extensions* Free services come with pre-configured CPU and memory, are hosted in `us-east-1` region, and offer limited resources with a basic feature scope. This excludes connection pooling, replication, data tiering, exporters, metrics, VPC, and other features. Free services are perfect for schema explorations, syntax validations, and other small-scale testing. Once you reach the storage limit, a free service enters a read-only state. You can [convert your free service](/docs/deploy/tiger-cloud/tiger-cloud-aws/service-management/service-management#convert-a-free-service-to-a-standard-one) to a standard one at any time. Note Free services are currently in beta. You can [manage, pause, or delete](/docs/deploy/tiger-cloud/tiger-cloud-aws/service-management/service-management) your service at any time from Tiger Console. ## Learn more about Tiger Cloud Read about Tiger Cloud features in the documentation: * Create your first [hypertable](/docs/learn/hypertables/understand-hypertables). * Run your first query using [time\_bucket()](/docs/build/data-management/query-data/advanced-analytic-queries#group-data-into-time-buckets). * Trying more advanced time-series functions, starting with [gap filling](/docs/build/data-management/query-data/advanced-analytic-queries#fill-gaps-in-time-series-data) or [real-time aggregates](/docs/build/continuous-aggregates/create-a-continuous-aggregate). ## Keep testing during your free trial You're now on your way to a great start with Tiger Cloud. You have a $1000-credit, 30-day free trial with Tiger Cloud to continue to test your use case. Before the end of your trial, make sure you add your credit card information. This ensures a smooth transition after your trial period concludes. If you have any questions, you can [join our community Slack group](https://slack-login.timescale.com) or [contact us](https://www.tigerdata.com/contact) directly. ## Advanced configuration Tiger Cloud is a versatile hosting service that provides a growing list of advanced features for your PostgreSQL and time-series data workloads. For more information about customizing your database configuration, see the [Configuration section](../configuration/about-configuration). Note The [TimescaleDB Terraform provider](https://registry.terraform.io/providers/timescale/timescale/latest/) provides configuration management resources for Tiger Cloud. You can use it to create, rename, resize, delete, and import services. For more information about the supported service configurations and operations, see the [Terraform provider documentation](https://registry.terraform.io/providers/timescale/timescale/latest/docs). # Manually change compute resources > Resize compute resources for your Tiger Cloud service with minimal downtime Tiger Cloud charges are based on the amount of storage you use. You don't pay for fixed storage size, and you don't need to worry about scaling disk size as your data grows, we handle it all for you. To reduce your data costs further, combine [hypercore](/docs/reference/timescaledb/hypercore), a [data retention policy](/docs/learn/data-lifecycle/data-retention/about-data-retention), and [tiered storage](/docs/learn/data-lifecycle/storage/about-storage-tiers). You use [Tiger Console](https://console.cloud.tigerdata.com/) to resize the compute (CPU/RAM) resources available to your Tiger Cloud services at any time, with a short downtime. ## Update compute resources for a service You can change the CPU and memory allocation for your standard service at any time with minimal downtime, usually less than a minute. The new resources become available as soon as the service restarts. You can change the CPU and memory allocation up or down, as frequently as required. ![Change resources](/docs/_astro/tiger-console-update-cpu-manually.DRnZv6ev_Ze3Xpb.webp) To change the resources of a free service, first [convert it to a standard one](../service-management/service-management#convert-a-free-service-to-a-standard-one). Note that: * For the 48 CPU / 192 GiB option, 6 CPU / 14 GiB is reserved for platform operations. * For the 64 CPU / 256 GiB option, 6 CPU / 16 GiB is reserved for platform operations. There is momentary downtime while the new compute settings are applied. In most cases, this is less than a minute. However, before making changes to your service, best practice is to enable [HA replication](../high-availability/overview) on the service. When you resize a service with HA enabled, Tiger Cloud: 1. Resizes the replica. 2. Waits for the replica to catch up. 3. Performs a switchover to the resized replica. 4. Restarts the primary. HA reduce downtime in the case of resizes or maintenance window restarts, from a minute or so to a couple of seconds. When you change resource settings, the current and new charges are displayed immediately so that you can verify how the changes impact your costs. Warning Because compute changes require an interruption to your services, plan accordingly so that the settings are applied during an appropriate service window. 1. **In Console, choose the service to modify** 2. **Click `Operations` > `Compute and storage`** 3. **Select the new `CPU / Memory` allocation** You see the allocation and costs in the comparison chart. 4. **Click `Apply`** Your service goes down briefly while the changes are applied. ## Out of memory errors If you run intensive queries on your standard services, you might encounter out of memory (OOM) errors. This occurs if your query consumes more memory than is available. When this happens, an `OOM killer` process shuts down PostgreSQL processes using `SIGKILL` commands until the memory usage falls below the upper limit. Because this kills the entire server process, it usually requires a restart. To prevent service disruption caused by OOM errors, Tiger Cloud attempts to shut down only the query that caused the problem. This means that the problematic query does not run, but that your service continues to operate normally. * If the normal OOM killer is triggered, the error log looks like this: ```yml 2021-09-09 18:15:08 UTC [560567]:TimescaleDB: LOG: server process (PID 2351983) was terminated by signal 9: Killed ``` Wait for the service to come back online before reconnecting. * Tiger Cloud shuts the client connection only If Tiger Cloud successfully guards the service against the OOM killer, it shuts down only the client connection that was using too much memory. This prevents the entire service from shutting down, so you can reconnect immediately. The error log looks like this: ```yml 2022-02-03 17:12:04 UTC [2253150]:TimescaleDB: tsdbadmin@tsdb,app=psql [53200] ERROR: out of memory ``` If you encounter OOM errors on a free service, [convert it to a standard service](../service-management/service-management#convert-a-free-service-to-a-standard-one), then increase its CPU and memory allocation as needed to resolve the memory pressure. # Connection pooling > Create connection poolers in Tiger Console to reuse connections and boost performance You can scale your Tiger Cloud service connections and improve its performance by using connection poolers. Tiger Cloud uses `pgBouncer` for connection pooling. If your service needs a large number of short-lived connections, a connection pooler is a great way to improve performance. For example, web, serverless, and IoT applications often use an event-based architecture where data is read or written from the database for a very short amount of time. Your application rapidly opens and closes connections while the pooler maintains a set of long-running connections to the service. This improves performance because the pooler opens the connections in advance, allowing the application to open many short-lived connections, while the service opens few, long-lived connections. ## User authentication By default, the poolers have authentication to the service, so you can use any custom users you already have set up without further configuration. You can continue using the `tsdbadmin` user if that is your preferred method. However, you might need to add custom configurations for some cases such as `statement_timeout` for a pooler user. 1. **Connect to your service as the `tsdbadmin` user and create a new role** ```sql CREATE ROLE LOGIN PASSWORD ''; ``` 2. **Change the `statement_timeout` settings to 2 seconds for this user** ```sql ALTER ROLE my_app SET statement_timeout TO '2s'; ``` 3. **In a new terminal window, connect on the pooler with the new user ``** ```bash PGPASSWORD= psql 'postgres://my_app@service.project.tsdb.cloud.timescale.com:30477/tsdb?sslmode=require' ``` The output looks something like this: ```plaintext psql (15.3 (Homebrew), server 15.4 (Ubuntu 15.4-1.pgdg22.04+1)) SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, compression: off) Type "help" for help. ``` 4. **Check that the settings are correct** ```sql SELECT current_user; ┌──────────────┐ │ current_user │ ├──────────────┤ │ my_app │ └──────────────┘ (1 row) ``` Check the `statement_timeout` setting is correct for the `` user: ```plaintext tsdb=> show statement_timeout; ┌───────────────────┐ │ statement_timeout │ ├───────────────────┤ │ 2s │ └───────────────────┘ (1 row) ``` ## Pool types When you create a connection pooler, there are two pool types to choose from: session or transaction. Each pool type uses a different mode to handle connections. Session pools allocate a connection from the pool until they are closed by the application, similar to a regular PostgreSQL connection. When the application closes the connection, it is sent back to the pool. Transaction pool connections are allocated only for the duration of the transaction, releasing the connection back to the pool when the transaction ends. If your application opens and closes connections frequently, choose the transaction pool type. By default, the pooler supports both modes simultaneously. However, the connection string you use to connect your application is different, depending on whether you want a session or transaction pool type. When you create a connection pool in the Tiger Console, you are given the correct connection string for the mode you choose. For example, a connection string to connect directly to your service looks a bit like this: ```plaintext postgres://:@service.example.cloud.timescale.com:30133/tsdb?sslmode=require ``` A session pool connection string is the same, but uses a different port number, like this: ```plaintext postgres://:@service.example.cloud.timescale.com:29303/tsdb?sslmode=require ``` The transaction pool connection string uses the same port number as a session pool connection, but uses a different database name, like this: ```plaintext postgres://:@service.example.cloud.timescale.com:29303/tsdb_transaction?sslmode=require ``` Make sure you check the Tiger Console output for the correct connection string to use in your application. ## Connection pool sizes A connection pooler manages connections to both the service itself, and the client application. It keeps a fixed number of connections open with the service, while allowing clients to open and close connections. Clients can request a connection from the session pool or the transaction pool. The connection pooler will then allocate the connection if there is one free. The number of client connections allowed to each pool is proportional to the `max_connections` parameter set for the service. The session pool can have a maximum of `max_connections - 17` client connections, while the transaction pool can have a maximum of `(max_connections - 17) * 20` client connections. Of the 17 reserved connections that are not allocated to either pool, 12 are reserved for the database superuser by default, and another 5 for Tiger Cloud operations. For example, if `max_connections` is set to 500, the maximum number of client connections for your session pool is `483 (500 - 17)` and `9,660 (483 * 20)` for your transaction pool. The configurable range of `max_connections` depends on your service's compute size. See [Min and max connection ranges](#min-and-max-connection-ranges). ## Add a connection pooler When you create a new service, you can also create a connection pooler. Alternatively, you can add a connection pooler to an existing service in Console. 1. **Log in to Console and click the service you want to add a connection pooler to** 2. **In `Operations`, click `Connection pooling` > `Add pooler`** Your pooler connection details are displayed in the `Connection pooling` tab. Use this information to connect to your transaction or session pooler. For more information about the different pool types, see the [pool types](#pool-types) section. ## Remove a connection pooler If you no longer need a connection pooler, you can remove it in Console. When you have removed your connection pooler, make sure that you also update your application to adjust the port it uses to connect to your service. 1. **In Console, select the service you want to remove a connection pooler from** 2. **Select `Operations`, then `Connection pooling`** 3. **Click `Remove connection pooler`** Confirm that you want to remove the connection pooler. After you have removed a pooler, if you add it back in the future, it uses the same connection string and port that was used before. ## pgBouncer statistics commands 1. **Connect to your service** 2. **Switch to the `pgbouncer` database: `\\c pgbouncer`** 3. **Run any read-only command for the pgBouncer CLI (for example, `SHOW STATS;`)** For full options, see the pgBouncer [docs here](https://www.pgbouncer.org/usage.html). ## VPC and connection pooling VPCs are supported with connection pooling. It does not matter the order you add the pooler or connect to a VPC. Your connection strings will automatically be updated to use the VPC connection string. ## Min and max connection ranges You can set `max_connections` within the following ranges, based on your service memory and CPU configuration: | Configuration | Minimum | Maximum | | -------------------------------------- | ------- | ------- | | Shared memory and CPU | 25 | 25 | | From 0.5 CPU / 2 GiB to 4 CPU / 16 GiB | 25 | 500 | | 8 CPU / 32 GiB and more | 25 | 2000 | `max_connections` of the primary service cannot be higher than its replica's. # Fork services > Fork a Tiger Cloud service to create isolated branches for testing and development Modern development is highly iterative. Developers and AI agents need safe spaces to test changes before deploying them to production. Forkable services make this natural and easy. Spin up a branch, run your test, throw it away, or merge it back. A fork is an exact copy of a service at a specific point in time, with its own independent data and configuration, including: * The database data and schema * Configuration * An admin `tsdbadmin` user with a new password Forks are fully independent. Changes to the fork don't affect the parent service. You can query them, run migrations, add indexes, or test new features against the fork without affecting the original service. Forks are a powerful way to share production-scale data safely. Testing, BI and data science teams often need access to real datasets to build models or generate insights. With forkable services, you easily create fast, zero-copy branches of a production service that are isolated from production, but contain all the data needed for analysis. Rapid fork creation dramatically reduces friction getting insights from live data. ## Understand service forks You can use service forks for disaster recovery, CI/CD automation, and testing and development. For example, you can automatically test a major PostgreSQL upgrade on a fork before applying it to your production service. Tiger Cloud offers the following fork strategies: * `now`: create a fresh fork of your database at the current time. Use when: * You need the absolute latest data * Recent changes must be included in the fork * `last-snapshot`: fork from the most recent [automatic backup](../high-availability/backup-restore). Because Tiger Cloud takes automatic backups daily, this is usually within the previous 24 hours of your service. To see when the latest backup was taken, check [`Backup history`](../high-availability/backup-restore#automatic-backups) in Tiger Console. Use when: * You want the fastest possible fork creation * Up to 24 hours behind current data is acceptable * `timestamp`: fork from a specific point in time, up to 14 days before, depending on your [pricing plan](../pricing-and-account-management#features-included-in-each-pricing-plan). Use when: * Disaster recovery from a known-good state * Investigating issues that occurred at a specific time * Testing "what-if" scenarios from historical data ### Fork creation speed Fork creation speed depends on the type of service you want to create: * Free services: \~30-90 seconds. Use a Copy-on-Write storage architecture with zero-copy between a fork and the parent. * Standard services: varies with the size of your service, typically 5-20+ minutes. Use traditional storage architecture with backup restore + WAL replay. ### Billing You can fork a free service to a free or a standard service. However, you cannot fork a standard service to a free service. Billing on storage works in the following way: * High-performance storage: * Copy-on-Write: you are only billed for storage for the chunks that diverge from the parent service. * Traditional: you are billed for storage for the whole service. * Object storage tier: * [Tiered data](/docs/build/data-management/storage/manage-storage) is shared across forks using copy-on-write and traditional storage: * Chunks in tiered storage are only billed once, regardless of the number of forks * Only new or modified chunks in a fork incur additional costs For details, see [Replicas and forks with tiered data](/docs/build/data-management/storage/tiered-data-replicas-forks). ## Prerequisites for this procedure To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service). ## Manage forks Important You cannot fork a service while it is being provisioned or otherwise transitioning state. Wait for the service to reach `Running` or `Paused` before you start forking. * Tiger CLI To manage development forks: 1. **Install Tiger CLI** Use the terminal to install the CLI: * macOS ```shell brew install --cask timescale/tap/tiger-cli ``` * Debian ```shell curl -s https://packagecloud.io/install/repositories/timescale/tiger-cli/script.deb.sh | sudo os=any dist=any bash sudo apt-get install tiger-cli ``` * Ubuntu ```shell curl -s https://packagecloud.io/install/repositories/timescale/tiger-cli/script.deb.sh | sudo os=any dist=any bash sudo apt-get install tiger-cli ``` * Red Hat ```shell curl -s https://packagecloud.io/install/repositories/timescale/tiger-cli/script.rpm.sh | sudo os=rpm_any dist=rpm_any bash sudo yum install tiger-cli ``` * Fedora ```shell curl -s https://packagecloud.io/install/repositories/timescale/tiger-cli/script.rpm.sh | sudo os=rpm_any dist=rpm_any bash sudo yum install tiger-cli ``` * Windows PowerShell ```shell irm https://cli.tigerdata.com/install.ps1 | iex ``` * x-platform ```shell curl -fsSL https://cli.tigerdata.com | sh ``` 2. **Authenticate Tiger CLI** 1. In an interactive terminal, start the login flow: ```shell tiger auth login ``` Tiger CLI opens Console in your browser to sign in with OAuth. Log in, or sign up if you don't have a Tiger Cloud account yet, then authorize Tiger CLI. If the browser does not open automatically, copy the `Auth URL` printed in the terminal. For non-interactive sessions such as scripts and CI, you can authenticate with [client credentials](/docs/integrate/find-connection-details#create-client-credentials) instead of the browser flow. 2. Select a Tiger Cloud project: ```text Auth URL is: https://console.cloud.tigerdata.com/oauth/authorize?client_id=...&code_challenge_method=S256&redirect_uri=http://localhost:PORT/callback&response_type=code&state=... Opening browser for authentication... Select a project: > 1. Tiger Project (tgrproject) 2. YourCompany (Company wide project) (cpnproject) 3. YourCompany Department (dptproject) Successfully logged in (project: tgrproject) 🎉 Next steps: • Install MCP server for your favorite AI coding tool: tiger mcp install • List existing services: tiger service list • Create a new service: tiger service create • Enable read-only mode: tiger config set read_only true ``` If only one project is associated with your account, this step is skipped. Tiger CLI stores the OAuth session in your system keychain, or in `~/.config/tiger/credentials` with restricted file permissions (600) if the keychain is unavailable, and refreshes it automatically. Tiger CLI stores your configuration in `~/.config/tiger/config.yaml`. To sign out and revoke the session, run `tiger auth logout`. 3. **Test your authenticated connection to Tiger Cloud by listing services** ```bash tiger service list ``` This call returns something like: * No services: ```txt 🏜️ No services found! Your project is looking a bit empty. 🚀 Ready to get started? Create your first service with: tiger service create ``` * One or more services: ```txt ┌────────────┬─────────────────────┬────────┬─────────────┬──────────────┬──────────────────┐ │ SERVICE ID │ NAME │ STATUS │ TYPE │ REGION │ CREATED │ ├────────────┼─────────────────────┼────────┼─────────────┼──────────────┼──────────────────┤ │ tgrservice │ tiger-agent-service │ READY │ TIMESCALEDB │ eu-central-1 │ 2025-09-25 16:09 │ └────────────┴─────────────────────┴────────┴─────────────┴──────────────┴──────────────────┘ ``` 4. **Fork the service** ```shell tiger service fork tgrservice --now --no-wait --name bob ``` You must specify exactly one timing option: `--now` (fork at current state), `--last-snapshot` (fork at last snapshot, faster), or `--to-timestamp` (fork at specific point in time). By default a fork matches the resources of the parent Tiger Cloud service. For standard services, specify `--cpu` and/or `--memory` for dedicated resources. You see something like: ```txt 🍴 Forking service 'tgrservice' to create 'bob' at current state... ✅ Fork request accepted! 📋 New Service ID: 🔐 Password saved to system keyring for automatic authentication 🎯 Set service '' as default service. ⏳ Service is being forked. Use 'tiger service list' to check status. ┌───────────────────┬──────────────────────────────────────────────────────────────────────────────────────────────────┐ │ PROPERTY │ VALUE │ ├───────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────┤ │ Service ID │ │ │ Name │ bob │ │ Status │ │ │ Type │ TIMESCALEDB │ │ Region │ eu-central-1 │ │ CPU │ 0.5 cores (500m) │ │ Memory │ 2 GB │ │ Direct Endpoint │ ..tsdb.cloud.timescale.com: │ │ Created │ 2025-10-08 13:58:07 UTC │ │ Connection String │ postgresql://tsdbadmin@..tsdb.cloud.timescale.com:/tsdb?sslmode=require │ └───────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────┘ ``` 5. **When you are done, delete your forked service** 1. Use the CLI to request service delete: ```shell tiger service delete ``` 2. Validate the service delete: ```text Are you sure you want to delete service ''? This operation cannot be undone. Type the service ID '' to confirm: ``` You see something like: ```text 🗑️ Delete request accepted for service ''. ✅ Service '' has been successfully deleted. ``` * Tiger Console ```shell brew install --cask timescale/tap/tiger-cli ``` * macOS ```shell curl -s https://packagecloud.io/install/repositories/timescale/tiger-cli/script.deb.sh | sudo os=any dist=any bash sudo apt-get install tiger-cli ``` * Debian ```shell curl -s https://packagecloud.io/install/repositories/timescale/tiger-cli/script.deb.sh | sudo os=any dist=any bash sudo apt-get install tiger-cli ``` * Ubuntu ```shell curl -s https://packagecloud.io/install/repositories/timescale/tiger-cli/script.rpm.sh | sudo os=rpm_any dist=rpm_any bash sudo yum install tiger-cli ``` * Red Hat ```shell curl -s https://packagecloud.io/install/repositories/timescale/tiger-cli/script.rpm.sh | sudo os=rpm_any dist=rpm_any bash sudo yum install tiger-cli ``` * Fedora ```shell irm https://cli.tigerdata.com/install.ps1 | iex ``` * Windows PowerShell ```shell curl -fsSL https://cli.tigerdata.com | sh ``` * x-platform Tiger Console lets you create development forks and point-in-time recovery (PITR) forks. ### Development forks Development forks use the [`now`](#understand-service-forks) strategy. 1. **In Tiger Console, navigate to `Operations` > `Service Management` and click `Fork service`** ![Fork a Tiger Cloud service](/docs/_astro/create-fork-tiger-console-azure.BE51R9ZR_Z1gfkms.webp) 2. **Configure the fork, then click `Fork service`** A fork of the service is created. The forked service shows in `Services` with a label specifying which service it has been forked from. 3. **Update the connection strings in your app to use the fork** ### PITR forks PITR forks use the [`timestamp`](#understand-service-forks) strategy. To recover from a destructive or unwanted action, create a fork at a specific point in time. To avoid paying compute for both the recovery fork and the original service, pause the original. 1. **Navigate to `Operations` > `Service management` and click `Create recovery fork`** ![Creating a recovery fork in Tiger Console](/docs/_astro/create-recovery-fork-tiger-console-azure.BLmavzD0_Z1jyXvs.webp) 2. **Select the recovery point, ensuring the correct time zone (UTC offset)** 3. **Configure the fork to match the original service** Best practice is to match the same configuration you had at the point you want to recover to. You can also add an HA replica, tag the fork, or add a connection pooler. 4. **Confirm by clicking `Create recovery fork`** 5. **Update the connection strings in your app to use the fork** ## Integrate service forks in your CI/CD pipeline To fork your Tiger Cloud service using GitHub actions: 1. **Store your Tiger Cloud API key as a GitHub Actions secret** 1. In [Tiger Console](https://console.cloud.tigerdata.com/dashboard/settings), click `Create credentials`. 2. Save the `Public key` and `Secret key` locally, then click `Done`. 3. In your GitHub repository, click `Settings`, open `Secrets and variables`, then click `Actions`. 4. Click `New repository secret`, then set `Name` to `TIGERDATA_API_KEY`. 5. Set `Secret` to your Tiger Cloud API key in the following format `:`, then click `Add secret`. 2. **Add the GitHub Actions Marketplace to your workflow YAML files** For example, the following workflow forks a service when a pull request is opened, running tests against the fork, then automatically cleans up. ```yaml name: Test on a service fork on: pull_request jobs: test: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Fork Database id: fork uses: timescale/fork-service@v1 with: project_id: ${{ secrets.TIGERDATA_PROJECT_ID }} service_id: ${{ secrets.TIGERDATA_SERVICE_ID }} api_key: ${{ secrets.TIGERDATA_API_KEY }} fork_strategy: last-snapshot cleanup: true name: pr-${{ github.event.pull_request.number }} - name: Run Integration Tests env: DATABASE_URL: postgresql://tsdbadmin:${{ steps.fork.outputs.initial_password }}@${{ steps.fork.outputs.host }}:${{ steps.fork.outputs.port }}/tsdb?sslmode=require run: | npm install npm test - name: Run Migrations env: DATABASE_URL: postgresql://tsdbadmin:${{ steps.fork.outputs.initial_password }}@${{ steps.fork.outputs.host }}:${{ steps.fork.outputs.port }}/tsdb?sslmode=require run: npm run migrate ``` For the full list of inputs, outputs, and configuration options, see the [Tiger Data - Fork Service](https://github.com/marketplace/actions/tiger-data-fork-service) in GitHub marketplace. # Service explorer > Monitor service health and performance with the Service Explorer dashboard Service explorer in Tiger Console provides a rich administrative dashboard for understanding the state of your database instance. The explorer gives you insight into the performance of your database, giving you greater confidence and control over your data. The explorer works like an operations center as you develop and run your applications with Tiger Cloud. It gives you quick access to the key properties of your database, like table sizes, schema definitions, and foreign key references, as well as information specific to Tiger Cloud, like information on your hypertables and continuous aggregates. To see the explorer, select your service in Console and click `Explorer`. ## General information In the `General information` section, you can see a high-level summary of your service, including all your hypertables and relational tables. It summarizes your overall compression ratios, and other policy and continuous aggregate data. And, if you aren't already using key features like continuous aggregates, columnstore compression, or other automation policies and actions, it provides pointers to tutorials and documentation to help you get started. ![Service explorer](/docs/_astro/tiger-console-service-explorer.CiAngqJU_Z14JOnR.webp) ## Tables You can have a detailed look into all your tables, including information about table schemas, table indexes, and foreign keys. For your hypertables, it shows details about chunks, continuous aggregates, and policies such as data retention policies and data reordering. You can also inspect individual hypertables, including their sizes, dimension ranges, and columnstore compression status. From this section, you can also set an automated policy to compress chunks into the columnstore. For more information, see the [hypercore documentation](/docs/learn/columnar-storage/understand-hypercore). When you inspect a hypertable, a timeline chart visualizes its chunks, showing their relative sizes and whether each is in the rowstore or the columnstore. By default, the chart shows the latest 100 chunks; click and drag to zoom in on a specific time range. The chart also shows the status of the latest runs for the hypertable's columnstore job, with a link to the job's deep dive page. ![Service explorer tables](/docs/_astro/tiger-console-columstore-data-savings.DzFnEbdZ_Z2h6ct0.webp) For more information about hypertables, see the [hypertables section](/docs/learn/hypertables/understand-hypertables). ## Continuous aggregates In the `Continuous aggregate` section, you can see all your continuous aggregates, including top-level information such as their size, whether they are configured for real-time aggregation, and their refresh periods. ![Service explorer caggs](/docs/_astro/tiger-console-service-explorer-caggs.DvufWqGS_1utudM.webp) For more information about continuous aggregates, see the [continuous aggregates section](/docs/build/continuous-aggregates/create-a-continuous-aggregate). # Service management > Manage your Tiger Cloud service lifecycle in Tiger Console From the `Operations` dashboard, you can reset the password, change the environment, convert a free service to a standard one, pause, or delete the service. To fork a service, see [Fork services](../service-management/fork-services). ## Reset your service password You can reset your service password from the `Operations` dashboard. This is the password you use to connect to your service, not the password for Tiger Console. To reset your Console password, navigate to the `Account` page. When you reset your service password, you are prompted for your Console password. When you have authenticated, you can create a new service password, ask Console to auto-generate a password, or switch your authentication type between SCRAM and MD5. SCRAM (salted challenge response authentication mechanism) and MD5 (message digest algorithm 5) are cryptographic authentication mechanisms. Tiger Console uses SCRAM by default. It is more secure and strongly recommended. The MD5 option is provided for compatibility with older clients. ## Change the service environment In Tiger Cloud, you create services for the following environments: * Development services: for less demanding tasks, such as ad hoc analytics, testing, or application building. Development services have no delete protection. * Production services: for mission-critical applications like client-facing dashboards or APIs. Production services have delete protection. Tiger Cloud applies upgrades differently to development and production services. To change the environment of an existing service: 1. **In Tiger Console, select the service to update** 2. **Click `Operations` > `Environment`, then change the tag under `Change service environment`** ## Convert a free service to a standard one You can convert a free service to a standard one, to increase its resources. Note that it is only available if you haven't reached the quota on the number of standard services according to your pricing plan. 1. **In Tiger Console, select the service to update** 2. **Click `Operations` > `Environment` > `Convert service to standard` > `Yes, convert to standard`** ## Pause a service You can pause a service if you want to stop it running temporarily. When you pause a service, you are no longer billed for compute resources. However, you do need to continue paying for any storage you are using. Pausing a service ensures that it is still available, and is ready to be restarted at any time. ## Delete a service You can delete a service to remove it completely. This removes the service and its underlying data from the server. You cannot recover a deleted service. # Tiger Console overview > Manage daily operations, access control, security, and queries for Tiger Cloud services You use Tiger Console to manage your services and data in a convenient, centralized manner. When you [log into Tiger Console](https://console.cloud.tigerdata.com/), you see `Services` and other project management tabs: ![Check is running](/docs/_astro/tiger-console-services-view.F-zo-hID_1rY9dD.webp) * `Services`: create a new [service](/docs/get-started/quickstart/create-service) in your project, and view the existing services and their statuses. Each service is an optimized [PostgreSQL](/docs/learn/glossary#postgresql) instance extended with database engine innovations such as [TimescaleDB](/docs/learn/glossary#timescaledb), managed for you by Tiger Cloud. * `Security`: create [VPCs](../security/overview) and [IP allowlists](../security/overview). * `Exporters`: [export metrics and logs](../monitoring) from your services. * `Activity`: [check your services](../monitoring) and project activity. * `Settings`: [create client credentials](/docs/integrate/find-connection-details#create-client-credentials), add AI model API keys, and request a SOC 2 report. * `CLI/MCP`: install Tiger CLI and set up Tiger MCP. * `Users`: [add and remove users](../security/members) in your project. * `Billing`: [check usage](../pricing-and-account-management#monitor-usage-and-costs), [change pricing plans](../pricing-and-account-management#upgrade-or-downgrade-your-pricing-plans-at-any-time), and manage payment methods. * `Support`: open and manage [support tickets](../pricing-and-account-management#tiger-cloud-support). When you select a Tiger Cloud service in the `Services` tab, you land in the Ops view. In this view, you manage your services. You see `Overview` and other related tabs: ![Select a service to edit](/docs/_astro/ops-mode-overview-tiger-console.BaKQ_e8S_Z1vnL4H.webp) * `Overview`: get an overview of your service details, infrastructure, [connection info](/docs/get-started/quickstart/create-service#connect-to-your-service), and performance optimization tips. * `Actions`: [connect and import data](/docs/migrate) into your service. * `Explorer`: [inspect objects and storage configuration](../service-management/service-explorer) for your service. * `Monitoring`: [get visibility](../monitoring) into your service performance. * `Connectors`: [sync or stream](/docs/migrate) directly in your service, so data from another source is continuously updated. * `Operations`: fully manage your service: [configure compute and storage](../service-management/change-resources), [set up replicas](../high-availability/overview), [back up](../security/overview), upgrade, and more. * `AI`: create vector embeddings and [manage AI extensions](../tiger-cloud-extensions) for your service. Use the toggle at the top to switch to Data view. In this view, you can [write queries with autocomplete](/docs/build/data-management/run-queries-from-tiger-console#data-view) for any service, save them in folders, share them, and create charts/dashboards with the added power of AI. To query your service from any tab, click `SQL Editor` at the bottom. The floating window expands and is ready for work. Click the window icon in its top right to change the editor position on the screen. # PostgreSQL extensions > Browse and enable PostgreSQL extensions available for Tiger Cloud services The following PostgreSQL extensions are installed with each Tiger Cloud service: * [Tiger Data extensions](#tiger-data-extensions) * [PostgreSQL built-in extensions](#postgresql-built-in-extensions) * [Third-party extensions](#third-party-extensions) ## Tiger Data extensions | Extension | Description | Enabled by default | | ------------------------------------------------------------------------ | ----------------------------------------------------------------------- | ----------------------------------------------------- | | [pgai](https://github.com/timescale/pgai) | Helper functions for AI workflows | For [AI-focused](/docs/get-started) services | | [pg\_textsearch](../tiger-cloud-extensions/pg-textsearch) | [BM25](https://en.wikipedia.org/wiki/Okapi_BM25)-based full-text search | Production-ready (v1.0.0) | | [pgvector](https://github.com/pgvector/pgvector) | Vector similarity search for PostgreSQL | For [AI-focused](/docs/get-started) services | | [pgvectorscale](https://github.com/timescale/pgvectorscale) | Advanced indexing for vector data | For [AI-focused](/docs/get-started) services | | [timescaledb\_toolkit](https://github.com/timescale/timescaledb-toolkit) | TimescaleDB Toolkit | For [Real-time analytics](/docs/get-started) services | | [timescaledb](https://github.com/timescale/timescaledb) | TimescaleDB | For all services | ## PostgreSQL built-in extensions | Extension | Description | Enabled by default | | -------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------- | ------------------ | | [autoinc](https://www.postgresql.org/docs/current/contrib-spi.html#CONTRIB-SPI-AUTOINC) | Functions for autoincrementing fields | - | | [amcheck](https://www.postgresql.org/docs/current/amcheck.html) | Functions for verifying relation integrity | - | | [bloom](https://www.postgresql.org/docs/current/bloom.html) | Bloom access method - signature file-based index | - | | [bool\_plperl](https://www.postgresql.org/docs/current/plperl-funcs.html) | Transform between bool and plperl | - | | [btree\_gin](https://www.postgresql.org/docs/current/btree-gin.html) | Support for indexing common datatypes in GIN | - | | [btree\_gist](https://www.postgresql.org/docs/current/btree-gist.html) | Support for indexing common datatypes in GiST | - | | [citext](https://www.postgresql.org/docs/current/citext.html) | Data type for case-insensitive character strings | - | | [cube](https://www.postgresql.org/docs/current/cube.html) | Data type for multidimensional cubes | - | | [dict\_int](https://www.postgresql.org/docs/current/dict-int.html) | Text search dictionary template for integers | - | | [dict\_xsyn](https://www.postgresql.org/docs/current/dict-xsyn.html) | Text search dictionary template for extended synonym processing | - | | [earthdistance](https://www.postgresql.org/docs/current/earthdistance.html) | Calculate great-circle distances on the surface of the Earth | - | | [fuzzystrmatch](https://www.postgresql.org/docs/current/fuzzystrmatch.html) | Determine similarities and distance between strings | - | | [hstore](https://www.postgresql.org/docs/current/hstore.html) | Data type for storing sets of (key, value) pairs | - | | [hstore\_plperl](https://www.postgresql.org/docs/current/hstore.html) | Transform between hstore and plperl | - | | [insert\_username](https://www.postgresql.org/docs/current/contrib-spi.html#CONTRIB-SPI-INSERT-USERNAME) | Functions for tracking who changed a table | - | | [intagg](https://www.postgresql.org/docs/current/intagg.html) | Integer aggregator and enumerator (obsolete) | - | | [intarray](https://www.postgresql.org/docs/current/intarray.html) | Functions, operators, and index support for 1-D arrays of integers | - | | [isn](https://www.postgresql.org/docs/current/isn.html) | Data types for international product numbering standards | - | | [jsonb\_plperl](https://www.postgresql.org/docs/current/datatype-json.html#DATATYPE-JSON-TRANSFORMS) | Transform between jsonb and plperl | - | | [lo](https://www.postgresql.org/docs/current/lo.html) | Large object maintenance | - | | [ltree](https://www.postgresql.org/docs/current/ltree.html) | Data type for hierarchical tree-like structures | - | | [moddatetime](https://www.postgresql.org/docs/current/contrib-spi.html#CONTRIB-SPI-MODDATETIME) | Functions for tracking last modification time | - | | [old\_snapshot](https://www.postgresql.org/docs/16/oldsnapshot.html) | Utilities in support of `old_snapshot_threshold` | - | | [pgcrypto](../tiger-cloud-extensions/pgcrypto) | Cryptographic functions | - | | [pgrowlocks](https://www.postgresql.org/docs/current/pgrowlocks.html) | Show row-level locking information | - | | [pgstattuple](https://www.postgresql.org/docs/current/pgstattuple.html) | Obtain tuple-level statistics | - | | [pg\_freespacemap](https://www.postgresql.org/docs/current/pgfreespacemap.html) | Examine the free space map (FSM) | - | | [pg\_prewarm](https://www.postgresql.org/docs/current/pgprewarm.html) | Prewarm relation data | - | | [pg\_stat\_statements](https://www.postgresql.org/docs/current/pgstatstatements.html) | Track execution statistics of all SQL statements executed | For all services | | [pg\_trgm](https://www.postgresql.org/docs/current/pgtrgm.html) | Text similarity measurement and index searching based on trigrams | - | | [pg\_visibility](https://www.postgresql.org/docs/current/pgvisibility.html) | Examine the visibility map (VM) and page-level visibility info | - | | [plperl](https://www.postgresql.org/docs/current/plperl.html) | PL/Perl procedural language | - | | [plpgsql](https://www.postgresql.org/docs/current/plpgsql.html) | SQL procedural language | For all services | | [postgres\_fdw](https://www.postgresql.org/docs/current/postgres-fdw.html) | Foreign data wrappers | For all services | | [refint](https://www.postgresql.org/docs/current/contrib-spi.html) | Functions for implementing referential integrity (obsolete) | - | | [seg](https://www.postgresql.org/docs/current/seg.html) | Data type for representing line segments or floating-point intervals | - | | [sslinfo](https://www.postgresql.org/docs/current/sslinfo.html) | Information about SSL certificates | - | | [tablefunc](https://www.postgresql.org/docs/current/tablefunc.html) | Functions that manipulate whole tables, including crosstab | - | | [tcn](https://www.postgresql.org/docs/current/tcn.html) | Trigger change notifications | - | | [tsm\_system\_rows](https://www.postgresql.org/docs/current/tsm-system-rows.html) | `TABLESAMPLE` method which accepts the number of rows as a limit | - | | [tsm\_system\_time](https://www.postgresql.org/docs/current/tsm-system-time.html) | `TABLESAMPLE` method which accepts the time in milliseconds as a limit | - | | [unaccent](https://www.postgresql.org/docs/current/unaccent.html) | Text search dictionary that removes accents | - | | [uuid-ossp](https://www.postgresql.org/docs/current/uuid-ossp.html) | Generate universally unique identifiers (UUIDs) | - | ## Third-party extensions | Extension | Description | Enabled by default | | ------------------------------------------------------------------------------- | ----------------------------------------------------------------------- | ---------------------------------------------------- | | [h3](https://pgxn.org/dist/h3/) | H3 bindings for PostgreSQL | - | | [pgaudit](https://www.pgaudit.org/) | Detailed session and/or object audit logging | - | | [pgpcre](https://github.com/petere/pgpcre) | Perl-compatible RegEx | - | | [pg\_cron](https://github.com/citusdata/pg_cron) | SQL commands that you can schedule and run directly inside the database | [Contact us](mailto:support@tigerdata.com) to enable | | [pg\_repack](https://github.com/reorg/pg_repack) | Table reorganization in PostgreSQL with minimal locks | - | | [pgrouting](https://pgrouting.org/) | Geospatial routing functionality | - | | [postgis](../tiger-cloud-extensions/postgis) | PostGIS geometry and geography spatial types and functions | - | | [postgis\_raster](https://postgis.net/docs/RT_reference.html) | PostGIS raster types and functions | - | | [postgis\_sfcgal](https://postgis.net/docs/reference_sfcgal.html) | PostGIS SFCGAL functions | - | | [postgis\_tiger\_geocoder](https://postgis.net/docs/Extras.html#Tiger_Geocoder) | PostGIS Tiger Cloud geocoder and reverse geocoder | - | | [postgis\_topology](https://postgis.net/workshops/postgis-intro/topology.html) | PostGIS topology spatial types and functions | - | | [unit](https://github.com/df7cb/postgresql-unit) | SI units for PostgreSQL | - | # Optimize full text search with BM25 > Install and tune pg_textsearch (v1.1.0) for BM25 full-text search on Tiger Cloud and self-hosted deployments Note **pg\_textsearch v1.1.0** is **production-ready** on Tiger Cloud. See the [v1.1.0 release notes](https://github.com/timescale/pg_textsearch/releases/tag/v1.1.0). `pg_textsearch` supports **PostgreSQL 17 and 18**. For self-hosted installs, confirm versions against the [upstream compatibility table](https://github.com/timescale/pg_textsearch#postgresql-version-compatibility). PostgreSQL full-text search at scale consistently hits a wall where performance degrades catastrophically. Tiger Data's [pg\_textsearch](https://github.com/timescale/pg_textsearch) brings modern [BM25](https://en.wikipedia.org/wiki/Okapi_BM25)-based full-text search directly into PostgreSQL, with a memtable architecture for efficient indexing and ranking. `pg_textsearch` integrates seamlessly with SQL and provides better search quality and performance than the PostgreSQL built-in full-text search. With Block-Max WAND optimization, `pg_textsearch` delivers up to **4x faster top-k queries** compared to native BM25 implementations. Parallel index builds reduce indexing times by **4x or more** for large tables. Advanced compression using delta encoding and bitpacking reduces index sizes by **41%** while improving query performance by 10-20% for shorter queries. BM25 scores in `pg_textsearch` are returned as negative values, where lower (more negative) numbers indicate better matches. `pg_textsearch` implements the following: * **Corpus-aware ranking**: BM25 uses inverse document frequency to weight rare terms higher * **Term frequency saturation**: prevents documents with excessive term repetition from dominating results * **Length normalization**: adjusts scores based on document length relative to corpus average * **Relative ranking**: focuses on rank order rather than absolute score values This page shows you how to install `pg_textsearch`, configure BM25 indexes, and optimize your search capabilities using the following best practices: * **Parallel indexing**: enable parallel workers for faster index creation on large tables * **Language configuration**: choose appropriate text search configurations for your data language * **Hybrid search**: combine with pgvector or pgvectorscale for applications requiring both semantic and keyword search * **Query optimization**: use score thresholds to filter low-relevance results * **Index monitoring**: regularly check index usage and memory consumption ## Prerequisites for this procedure To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service). These steps use Tiger Cloud, but the same approach applies to a [self-hosted TimescaleDB](/docs/get-started/choose-your-path/install-timescaledb) instance. - Your [connection details](/docs/integrate/find-connection-details). ## Install pg\_textsearch To install this PostgreSQL extension: 1. **Connect to your Tiger Cloud service** In [Tiger Console](https://console.cloud.tigerdata.com/dashboard/services) open an [SQL editor](/docs/build/data-management/run-queries-from-tiger-console). You can also connect to your service using [psql](/docs/integrate/query-administration/psql#connect-to-your-service). 2. **Enable the extension on your Tiger Cloud service** * For new services, simply enable the extension: ```sql CREATE EXTENSION pg_textsearch; ``` * For existing services, update your instance, then enable the extension: The extension may not be available until after your next scheduled maintenance window. To pick up the update immediately, manually pause and restart your service. 3. **Verify the installation** ```sql SELECT * FROM pg_extension WHERE extname = 'pg_textsearch'; ``` You have installed `pg_textsearch` on Tiger Cloud. ## Self-hosted: load `pg_textsearch` before `CREATE EXTENSION` On self-managed PostgreSQL, the extension must be **preloaded** so the server loads it at startup: 1. Set `shared_preload_libraries` in `postgresql.conf` (append `pg_textsearch` to any existing list): ```ini shared_preload_libraries = 'pg_textsearch' ``` 2. Restart PostgreSQL. 3. In each database where you need search, run `CREATE EXTENSION pg_textsearch;`. Tiger Cloud manages preloading for you; you typically only run `CREATE EXTENSION` in the SQL editor or client. ## Create BM25 indexes on your data BM25 indexes provide modern relevance ranking that outperforms PostgreSQL's built-in ts\_rank functions by using corpus statistics and better algorithmic design. To create a BM25 index with `pg_textsearch`: 1. **Create a table with text content** ```sql CREATE TABLE products ( id serial PRIMARY KEY, name text, description text, category text, price numeric ); ``` 2. **Insert sample data** ```sql INSERT INTO products (name, description, category, price) VALUES ('Mechanical Keyboard', 'Durable mechanical switches with RGB backlighting for gaming and productivity', 'Electronics', 149.99), ('Ergonomic Mouse', 'Wireless mouse with ergonomic design to reduce wrist strain during long work sessions', 'Electronics', 79.99), ('Standing Desk', 'Adjustable height desk for better posture and productivity throughout the workday', 'Furniture', 599.99); ``` 3. **Create a BM25 index** ```sql CREATE INDEX products_search_idx ON products USING bm25(description) WITH (text_config='english'); ``` Each BM25 index covers a single text expression. That expression can be a plain column, or a computed expression that concatenates columns, extracts a JSONB field, or normalizes text. See [Index expressions, subsets of rows, and arrays](#index-expressions-subsets-of-rows-and-arrays). For optimal performance, load your data first, then create the index. You have created a BM25 index for full-text search. ## Index expressions, subsets of rows, and arrays Since [v1.1.0](https://github.com/timescale/pg_textsearch/releases/tag/v1.1.0) A BM25 index can cover more than a plain column. You can index a computed expression, restrict the index to a subset of rows, or index a `text[]` array. ### Index an expression Index a computed expression to search JSONB fields, combine multiple columns, or normalize text. The expression must return `text` and use only `IMMUTABLE` functions, and your queries must repeat the same expression in `ORDER BY`: ```sql -- Extract a JSONB field CREATE INDEX events_msg_idx ON events USING bm25 ((data->>'description')) WITH (text_config='english'); -- Combine title and body into one searchable index CREATE INDEX articles_idx ON articles USING bm25 ((coalesce(title, '') || ' ' || coalesce(body, ''))) WITH (text_config='english'); -- Normalize case before indexing CREATE INDEX docs_lower_idx ON docs USING bm25 ((lower(content))) WITH (text_config='simple'); ``` ### Index a subset of rows Add a `WHERE` clause to index only the rows you search. Partial indexes are smaller and faster when queries always target the same subset: ```sql CREATE INDEX docs_published_idx ON docs USING bm25 (content) WITH (text_config='english') WHERE status = 'published'; ``` Partial indexes require explicit index naming with `to_bm25query()`. The implicit `content <@> 'query'` syntax skips them: ```sql SELECT * FROM docs WHERE status = 'published' ORDER BY content <@> to_bm25query('search terms', 'docs_published_idx') LIMIT 10; ``` You can combine expression and partial indexes: ```sql CREATE INDEX events_error_idx ON events USING bm25 ((data->>'message')) WITH (text_config='english') WHERE (data->>'severity') = 'error'; ``` ### Search multilingual tables For tables that hold documents in several languages, create one partial index per language, each with the matching text search configuration. Each index then applies language-appropriate stemming and stop words: ```sql CREATE INDEX docs_en_idx ON docs USING bm25 (content) WITH (text_config='english') WHERE lang = 'en'; CREATE INDEX docs_de_idx ON docs USING bm25 (content) WITH (text_config='german') WHERE lang = 'de'; CREATE INDEX docs_fr_idx ON docs USING bm25 (content) WITH (text_config='french') WHERE lang = 'fr'; ``` Query with the matching predicate and index name: ```sql SELECT * FROM docs WHERE lang = 'en' ORDER BY content <@> to_bm25query('databases', 'docs_en_idx') LIMIT 10; ``` ### Index a text array `pg_textsearch` indexes `text[]` columns by scoring the array elements as a single flattened document. This is useful when you store a document as pre-split chunks: ```sql CREATE TABLE chunked_docs (id bigserial PRIMARY KEY, content text[]); CREATE INDEX chunked_docs_idx ON chunked_docs USING bm25(content) WITH (text_config='english'); ``` BM25 scores match what you would get by concatenating the elements into a single `text` value. ## Enable parallel indexing for faster index creation Since [v0.5.0](https://github.com/timescale/pg_textsearch/releases/tag/v0.5.0) `pg_textsearch` supports parallel index builds that can significantly reduce indexing times for large tables. PostgreSQL automatically uses parallel workers based on table size and available resources. 1. **Configure parallel workers (optional)** PostgreSQL uses server defaults, but you can adjust settings for your workload: ```sql -- Set number of parallel workers (uses CPU count by default) SET max_parallel_maintenance_workers = 4; -- Set memory for index builds (must be at least 64MB for parallel builds) SET maintenance_work_mem = '256MB'; ``` Note The planner requires `maintenance_work_mem >= 64MB` to enable parallel index builds. With insufficient memory, builds fall back to serial mode silently. 2. **Create index (parallel workers used automatically for large tables)** ```sql CREATE INDEX products_search_idx ON products USING bm25(description) WITH (text_config='english'); ``` When parallel build is used, you see a notice: ```plaintext NOTICE: parallel index build: launched 4 of 4 requested workers ``` 3. **Verify parallel execution in partitioned tables** For partitioned tables, each partition builds its index independently with parallel workers if the partition is large enough. This allows efficient indexing of very large partitioned datasets. You have configured parallel index builds for faster indexing. ## Merge BM25 segments after bulk loads After large batch inserts or sustained incremental writes, multiple on-disk **segments** can accumulate. Merging them into a single segment improves query speed by reducing how many segments a query must scan. This is analogous to Lucene’s `forceMerge(1)`. Use `bm25_force_merge()` **after bulk loads**, not during steady high write traffic: ```sql SELECT bm25_force_merge('products_search_idx'); ``` ## Optimize search queries for performance The `<@>` operator returns BM25-based scores as **negative** values: **lower (more negative) means a better match**. ### Ranked search with implicit query syntax (primary pattern) For typical top-k search, pass the query as a **string literal**. In `ORDER BY`, PostgreSQL can **automatically detect** the BM25 index from the indexed column: ```sql SELECT name, description, description <@> 'ergonomic work' AS score FROM products ORDER BY score LIMIT 3; ``` You see something like: ```sql name | description | score ----------------------------+-----------------------------------------------------------------------------------+--------------------- Ergonomic Mouse | Wireless mouse with ergonomic design to reduce wrist strain during long work sessions | -1.8132977485656738 Mechanical Keyboard | Durable mechanical switches with RGB backlighting for gaming and productivity | 0 Standing Desk | Adjustable height desk for better posture and productivity throughout the workday | 0 ``` ### The `bm25query` type and `to_bm25query()` Use an explicit **`bm25query`** value when you need the index name in the expression (for example in `WHERE`, or when the planner cannot infer the index from context): | Form | Use | | ------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------- | | `to_bm25query('query text')` | Query text only; use in **`ORDER BY`** with an index scan (implicit index detection from the column). | | `to_bm25query('query text', 'index_name')` | Query text **and** index name; required for **`WHERE`** filters and whenever you must name the index. | | `'index_name:query text'::bm25query` | Cast form (embedded index name). In PostgreSQL 18, the single-colon form helps the planner when expressions are evaluated early. | | `text <@> bm25query` → `double precision` | BM25 score (negative is better). | | `bm25query = bm25query` | Equality comparison. | ### Filter results by score threshold For `WHERE` clauses, use **`to_bm25query()`** with an explicit index name: ```sql SELECT name, description <@> to_bm25query('wireless', 'products_search_idx') AS score FROM products WHERE description <@> to_bm25query('wireless', 'products_search_idx') < -0.5; ``` You see something like: ```sql name | score ----------------+--------------------- Ergonomic Mouse | -0.9066488742828369 ``` ### Combine with standard SQL operations ```sql SELECT category, name, description <@> to_bm25query('ergonomic', 'products_search_idx') AS score FROM products WHERE price < 500 AND description <@> to_bm25query('ergonomic', 'products_search_idx') < -0.5 ORDER BY score LIMIT 5; ``` You see something like: ```sql category | name | score -------------+-----------------+--------------------- Electronics | Ergonomic Mouse | -0.9066488742828369 ``` ### Pre-filtering vs post-filtering How you combine BM25 with other predicates affects performance: * **Pre-filtering**: use another index (for example B-tree on `category_id`) so PostgreSQL restricts rows **before** BM25 scoring. Best when the filter is **selective** (for example matches a small fraction of rows); then `ORDER BY ... LIMIT` scores a smaller candidate set. * **Post-filtering**: the BM25 index scan produces top-k matches **first**; other `WHERE` conditions apply **after**. If the filter removes most of those rows, you can get **fewer rows than `LIMIT`**. Increase the inner `LIMIT` and re-limit in application code if needed. This mirrors tradeoffs familiar from approximate vector indexes: know whether your filter is shrinking the universe of rows before or after ranking. ### Verify plans with EXPLAIN (and sequential scans on small tables) ```sql EXPLAIN SELECT * FROM products ORDER BY description <@> 'ergonomic' LIMIT 5; ``` On **small** tables, the planner may choose a **sequential scan** instead of the BM25 index. To force index use while testing, you can run: ```sql SET enable_seqscan = off; ``` Note Even when `EXPLAIN` shows a sequential scan, `<@>` and `to_bm25query()` still use the BM25 index for **corpus statistics** (document counts, average length, and related metadata) required to score results. You have optimized your search queries for BM25 ranking. ## Build hybrid search with semantic and keyword search Combine `pg_textsearch` with `pgvector` or `pgvectorscale` to build powerful hybrid search systems that use both semantic vector search and keyword BM25 search. 1. **Enable the vectorscale extension on your Tiger Cloud service** ```sql CREATE EXTENSION IF NOT EXISTS vectorscale CASCADE; ``` 2. **Create a table with both text content and vector embeddings** ```sql CREATE TABLE articles ( id serial PRIMARY KEY, title text, content text, embedding vector(3) -- Using 3 dimensions for this example; use 1536 for OpenAI ada-002 ); ``` 3. **Insert sample data** ```sql INSERT INTO articles (title, content, embedding) VALUES ('Database Query Optimization', 'Learn how to optimize database query performance using indexes and query planning', '[0.1, 0.15, 0.2]'), ('Performance Tuning Guide', 'A comprehensive guide to performance tuning in distributed systems and databases', '[0.12, 0.18, 0.25]'), ('Introduction to Indexing', 'Understanding how database indexes improve query performance and data retrieval', '[0.09, 0.14, 0.19]'), ('Advanced SQL Techniques', 'Master advanced SQL techniques for complex data analysis and reporting', '[0.5, 0.6, 0.7]'), ('Data Warehousing Basics', 'Getting started with data warehousing and analytical query processing', '[0.8, 0.9, 0.85]'); ``` 4. **Create indexes for both search types** ```sql -- Vector index for semantic search CREATE INDEX articles_embedding_idx ON articles USING hnsw (embedding vector_cosine_ops); -- Keyword index for BM25 search CREATE INDEX articles_content_idx ON articles USING bm25(content) WITH (text_config='english'); ``` 5. **Perform hybrid search using reciprocal rank fusion** ```sql WITH vector_search AS ( SELECT id, ROW_NUMBER() OVER (ORDER BY embedding <=> '[0.1, 0.2, 0.3]'::vector) AS rank FROM articles ORDER BY embedding <=> '[0.1, 0.2, 0.3]'::vector LIMIT 20 ), keyword_search AS ( SELECT id, ROW_NUMBER() OVER (ORDER BY content <@> 'query performance') AS rank FROM articles ORDER BY content <@> 'query performance' LIMIT 20 ) SELECT a.id, a.title, COALESCE(1.0 / (60 + v.rank), 0.0) + COALESCE(1.0 / (60 + k.rank), 0.0) AS combined_score FROM articles a LEFT JOIN vector_search v ON a.id = v.id LEFT JOIN keyword_search k ON a.id = k.id WHERE v.id IS NOT NULL OR k.id IS NOT NULL ORDER BY combined_score DESC LIMIT 10; ``` You see something like: ```sql id | title | combined_score ----+----------------------------+-------------------- 3 | Introduction to Indexing | 0.0325224748810153 1 | Database Query Optimization| 0.0322664584959667 2 | Performance Tuning Guide | 0.0320020481310804 5 | Data Warehousing Basics | 0.0310096153846154 4 | Advanced SQL Techniques | 0.0310096153846154 ``` 6. **Adjust relative weights for different search types** ```sql WITH vector_search AS ( SELECT id, ROW_NUMBER() OVER (ORDER BY embedding <=> '[0.1, 0.2, 0.3]'::vector) AS rank FROM articles ORDER BY embedding <=> '[0.1, 0.2, 0.3]'::vector LIMIT 20 ), keyword_search AS ( SELECT id, ROW_NUMBER() OVER (ORDER BY content <@> 'query performance') AS rank FROM articles ORDER BY content <@> 'query performance' LIMIT 20 ) SELECT a.id, a.title, 0.7 * COALESCE(1.0 / (60 + v.rank), 0.0) + -- 70% weight to vectors 0.3 * COALESCE(1.0 / (60 + k.rank), 0.0) -- 30% weight to keywords AS combined_score FROM articles a LEFT JOIN vector_search v ON a.id = v.id LEFT JOIN keyword_search k ON a.id = k.id WHERE v.id IS NOT NULL OR k.id IS NOT NULL ORDER BY combined_score DESC LIMIT 10; ``` You see something like: ```sql id | title | combined_score ----+----------------------------+-------------------- 3 | Introduction to Indexing | 0.0163141195134849 2 | Performance Tuning Guide | 0.0160522273425499 1 | Database Query Optimization| 0.0160291438979964 4 | Advanced SQL Techniques | 0.0155528846153846 5 | Data Warehousing Basics | 0.0154567307692308 ``` You have implemented hybrid search combining semantic and keyword search. ## Configuration options Customize `pg_textsearch` behavior for your specific use case and data characteristics. 1. **Configure memory and performance settings** `pg_textsearch` keeps in-memory inverted indexes (memtables) in PostgreSQL dynamic shared memory. `memory_limit` bounds their growth: as usage approaches the cap, memtables spill to on-disk segments automatically, and if usage still exceeds the cap, inserts fail with an error rather than risking an out-of-memory kill. Tips **Crash recovery:** the memtable is rebuilt from the heap on startup, so you do **not** lose indexed data if PostgreSQL crashes before a spill to disk completes. Two settings can be changed in any session, by any role: ```sql -- Default query limit when no LIMIT clause is present (default 1000) SET pg_textsearch.default_limit = 5000; -- Segment compression with delta encoding and bitpacking (enabled by default). -- Reduces index size by ~41% with a 10-20% query performance improvement for shorter queries. SET pg_textsearch.compress_segments = on; ``` Since [v0.4.0](https://github.com/timescale/pg_textsearch/releases/tag/v0.4.0) The remaining settings are **server-level** or **superuser-only**, so a plain `SET` in a regular session does not apply them: | Setting | Default | Notes | | ----------------------------------- | -------- | ---------------------------------------------------------------------------------------- | | `pg_textsearch.memory_limit` | `2GB` | Server-level; takes effect on a configuration reload. A value of `0` disables the limit. | | `pg_textsearch.bulk_load_threshold` | `100000` | Superuser. Terms per transaction before the memtable spills to disk. | | `pg_textsearch.segments_per_level` | `8` | Superuser. Segments per level before automatic compaction (range 2–64). | | `pg_textsearch.log_bmw_stats` | `off` | Superuser. Logs Block-Max WAND block-skip statistics. | | `pg_textsearch.log_scores` | `off` | Superuser. Logs BM25 scores during index scans. | * On Tiger Cloud, change these from the Tiger Console: open your service, then `Operations` > `Database parameters`. See [Configure database parameters](/docs/deploy/tiger-cloud/tiger-cloud-aws/configuration/customize-configuration). The `tsdbadmin` role cannot apply them with `SET` or `ALTER SYSTEM`. * On self-hosted PostgreSQL, set `memory_limit` in `postgresql.conf`, or run `ALTER SYSTEM SET pg_textsearch.memory_limit = '2GB';` followed by `SELECT pg_reload_conf();`. As a superuser, you can change the superuser-only settings with `SET` in your session. Note `memory_limit` is applied as three tiers derived from the single value: a per-index soft limit (`memory_limit / 8`) spills that index's memtable to disk, a global soft limit (`memory_limit / 2`) evicts the largest memtable across all indexes, and the hard limit (`memory_limit`) refuses further cache growth so an insert fails rather than risking the out-of-memory killer. You only set `memory_limit`; the ratios are derived automatically. Since [v1.1.0](https://github.com/timescale/pg_textsearch/releases/tag/v1.1.0) 2. **Configure language-specific text processing** You can create multiple BM25 indexes on the same column with different language configurations: ```sql -- Create an additional index with simple tokenization (no stemming) CREATE INDEX products_simple_idx ON products USING bm25(description) WITH (text_config='simple'); -- Example: French language configuration for a French products table -- CREATE INDEX products_fr_idx ON products_fr -- USING bm25(description) -- WITH (text_config='french'); ``` 3. **Tune BM25 parameters** ```sql -- Adjust term frequency saturation (k1) and length normalization (b) CREATE INDEX products_custom_idx ON products USING bm25(description) WITH (text_config='english', k1=1.5, b=0.8); ``` 4. **Monitor index usage and memory consumption** * Check index usage statistics for every BM25 index ```sql SELECT s.schemaname, s.relname, s.indexrelname, s.idx_scan, s.idx_tup_read FROM pg_stat_user_indexes s JOIN pg_class c ON c.oid = s.indexrelid JOIN pg_am a ON a.oid = c.relam WHERE a.amname = 'bm25'; ``` * Check shared memory used by memtables across all indexes ```sql SELECT * FROM bm25_memory_usage(); ``` * View index summary with corpus statistics and memory usage (**superuser**) ```sql SELECT bm25_summarize_index('products_search_idx'); ``` * View detailed index structure (output is truncated for display) (**superuser**) ```sql SELECT bm25_dump_index('products_search_idx'); ``` Note `bm25_summarize_index` and `bm25_dump_index` require superuser. On Tiger Cloud the `tsdbadmin` role is not a superuser, so these two functions are unavailable there; use `bm25_memory_usage()` and `pg_stat_user_indexes` to monitor instead. The two-argument form `bm25_dump_index('index_name', '/path/to/file.txt')` exists only in **debug builds** compiled with `-DDEBUG_DUMP_INDEX`. Standard packages expose the single-argument form only. * Force memtable spill to disk (useful for testing or memory management) ```sql SELECT bm25_spill_index('products_search_idx'); ``` You have configured `pg_textsearch` for optimal performance. For production applications, consider implementing result caching and pagination to improve user experience with large result sets. ## Current limitations * **No phrase search**: you cannot search for exact multi-word phrases (no positional index; emulate with BM25 plus a post-filter if needed). * **No compressed data support**: `pg_textsearch` does not work with **compressed** chunk or table data (for example Hypercore-compressed data where the extension applies). * **One expression per index**: each BM25 index covers a **single text expression**. To search several columns at once, index a computed expression that concatenates them (see [Index expressions, subsets of rows, and arrays](#index-expressions-subsets-of-rows-and-arrays)). * **No built-in faceted search**: there is no dedicated faceting operator; use ordinary PostgreSQL patterns (`WHERE`, `GROUP BY`, counts) alongside BM25 filters. * **Insert and update throughput**: the memtable design supports incremental writes, but **sustained write-heavy** workloads are still a focus of ongoing performance work. Prefer **bulk load, then index** (or `bm25_force_merge` after large batches) when possible. * **No background compaction**: segment compaction runs **synchronously** during memtable spill; write-heavy workloads may see latency spikes during spills. * **Partitioned table statistics**: BM25 statistics are **per partition**. Scores are accurate within a partition but are **not directly comparable** across partitions when a query spans many partitions (different IDF and length stats). Prefer queries that target a single partition when rank comparability matters. * **Word length limit**: tokenization inherits PostgreSQL’s `tsvector` **2047 character** word limit; very long tokens (for example base64 blobs) may be ignored with an `INFO` message. * **PL/pgSQL and stored procedures**: the implicit form `text <@> 'query'` relies on planner hooks that **do not run** inside PL/pgSQL `DO` blocks, functions, or procedures. Use **`to_bm25query('query', 'index_name')`** explicitly in those contexts. For more detail and workarounds, see the [limitations section](https://github.com/timescale/pg_textsearch#limitations) in the upstream README. # Encrypt data using pgcrypto > Encrypt sensitive data in your PostgreSQL database using the pgcrypto extension The `pgcrypto` PostgreSQL extension provides cryptographic functions such as: * General hashing * Password hashing * PGP encryption * Raw encryption * Random-data For more information about these functions and the options available, see the [pgcrypto documentation](https://www.postgresql.org/docs/current/pgcrypto.html). ## Use the `pgcrypto` extension to encrypt inserted data The `pgcrypto` extension allows you to encrypt, decrypt, hash, and create digital signatures within your database. Tiger Data understands how precious your data is and safeguards sensitive information. 1. **Install the `pgcrypto` extension** ```sql CREATE EXTENSION IF NOT EXISTS pgcrypto; ``` Confirm the extension is installed using the `\dx` command: ```sql List of installed extensions Name | Version | Schema | Description ---------------------+---------+------------+--------------------------------------------------------------------------------------- pg_stat_statements | 1.10 | public | track planning and execution statistics of all SQL statements executed pgcrypto | 1.3 | public | cryptographic functions plpgsql | 1.0 | pg_catalog | PL/pgSQL procedural language timescaledb | 2.24.0 | public | Enables scalable inserts and complex queries for time-series data (Community Edition) timescaledb_toolkit | 1.22.0 | public | Library of analytical hyperfunctions, time-series pipelining, and other SQL utilities ``` 2. **Create a table named `user_passwords`** ```sql CREATE TABLE user_passwords (username varchar(100) PRIMARY KEY, crypttext text); ``` 3. **Insert values and replace `` with a password key of your choice** ```sql INSERT INTO tbl_sym_crypt (username, crypttext) VALUES ('user1', pgp_sym_encrypt('user1_password','')), ('user2', pgp_sym_encrypt('user2_password','')); ``` 4. **Confirm the password is encrypted** ```sql SELECT * FROM user_passwords; ``` The encrypted passwords are listed: ```sql username | crypttext ----------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------- user1 | \xc30d040703025caa37f9d1c731d169d240018529d6f0002b2948905a87e4787efaa0046e58fd3f04ee95594bea1803807063321f62c9651cbf0422b04508093df9644a76684b504b317cf633552fcf164f user2 | \xc30d0407030279bbcf760b81d3de73d23c01c04142632fc8527c0c1b17cc954c77f16df46022acddc565fd18f0f0f761ddb2f31b21c4ebe47a48039d685287d64506029e027cf29b5493b574df (2 rows) ``` 5. **View the decrypted passwords** Replace `` with the password key that you created: ```sql SELECT username, pgp_sym_decrypt(crypttext::bytea, '') FROM user_passwords; ``` The decrypted passwords are listed: ```sql username | pgp_sym_decrypt ----------+----------------- user1 | user1_password user2 | user2_password (2 rows) ``` # Create a chatbot using pgvector > Build a chatbot with vector similarity search using the pgvector extension The `pgvector` PostgreSQL extension helps you to store and search over machine learning-generated embeddings. It provides different capabilities that allows you to identify both exact and approximate nearest neighbors. It is designed to work seamlessly with other PostgreSQL features, including indexing and querying. For more information about these functions and the options available, see the [pgvector](https://github.com/pgvector/pgvector/blob/master/README.md) repository. ## Use the `pgvector` extension to create a `chatbot` The `pgvector` PostgreSQL extension allows you to create, store, and query OpenAI [vector embeddings](https://platform.openai.com/docs/guides/embeddings/what-are-embeddings) in a PostgreSQL database instance. This page shows you how to use [retrieval augmented generation (RAG)](https://www.promptingguide.ai/techniques/rag) to create a chatbot that combines your data with ChatGPT using OpenAI and `pgvector`. RAG provides a solution to the problem that a foundational model such as GPT-3 or GPT-4 could be missing some information needed to give a good answer, because that information was not in the dataset used to train the model. This can happen if the information is stored in private documents or only became available recently. In this example, you create embeddings, insert the embeddings into a Tiger Cloud service and query the embeddings using `pgvector`. The content for the embeddings is from the Tiger Data blog, specifically from the [Developer Q\&A](https://www.tigerdata.com/blog/tag/dev-q-a) section, which features posts by Tiger Data users talking about their real-world use cases. ## Prerequisites for this procedure To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service). - Your [connection details](/docs/integrate/find-connection-details). * Python installed. * The [pgvector repository](https://github.com/timescale/vector-cookbook/tree/main/openai_pgvector_helloworld) cloned. * An [OpenAI developer account](https://platform.openai.com/overview). * An OpenAI [API key](https://platform.openai.com/account/api-keys). Note If you are on a free plan there may be rate limiting for your API requests. 1. **Create and activate a Python virtual environment** ```bash virtualenv pgvectorenv source pgvectorenv/bin/activate ``` 2. **Set the environment variables** Set `OPENAI_API_KEY` and `TIMESCALE_CONNECTION_STRING`. In this example, to set the environment variables in macOS, open the `zshrc` profile. Replace ``, and `` with your OpenAI API key and the URL of your Tiger Cloud service: ```bash nano ~/.zshrc export OPENAI_API_KEY='' export TIMESCALE_CONNECTION_STRING='' ``` Update the shell with the new variables using `source ~/.zshrc` 3. **Confirm that you have set the environment variables** ```bash echo $OPENAI_API_KEY echo $TIMESCALE_CONNECTION_STRING ``` 4. **Install the required modules and packages** Using the `requirements.txt` file located in the `vector-cookbook\openai_pgvector_helloworld` directory: ```bash pip install -r requirements.txt ``` 5. **Create the `create_embeddings.py` file** To create embeddings for your data using the OpenAI API, open an editor of your choice and create the `create_embeddings.py` file. create\_embeddings.py ```python ############################################################################### # This script creates OpenAI embedding vectors for content in a CSV file # and saves the results to a new CSV file with the embeddings included ############################################################################### import openai import os import pandas as pd import numpy as np import json import tiktoken # Get openAI api key by reading local .env file from dotenv import load_dotenv, find_dotenv _ = load_dotenv(find_dotenv()) openai.api_key = os.environ['OPENAI_API_KEY'] # Load your CSV file into a pandas DataFrame df = pd.read_csv('blog_posts_data.csv') df.head() ############################################################################### # Helper functions to help us create the embeddings ############################################################################### # Calculate number of tokens for a string def num_tokens_from_string(string: str, encoding_name = "cl100k_base") -> int: if not string: return 0 # Returns the number of tokens in a text string encoding = tiktoken.get_encoding(encoding_name) num_tokens = len(encoding.encode(string)) return num_tokens # Calculate cost of embedding num_tokens # Assumes we're using the text-embedding-ada-002 model # See https://openai.com/pricing def get_embedding_cost(num_tokens): return num_tokens/1000*0.0001 # Calculate total cost of embedding all content in the dataframe def get_total_embeddings_cost(): total_tokens = 0 for i in range(len(df.index)): text = df['content'][i] token_len = num_tokens_from_string(text) total_tokens = total_tokens + token_len total_cost = get_embedding_cost(total_tokens) return total_cost ############################################################################### # quick check on total token amount for price estimation total_cost = get_total_embeddings_cost() print("Estimated price to embed this content = $" + str(total_cost)) ############################################################################### # Create new list with small content chunks to not hit max token limits # Note: the maximum number of tokens for a single request is 8191 # https://openai.com/docs/api-reference/requests ############################################################################### # list for chunked content and embeddings new_list = [] # Split up the text into token sizes of around 512 tokens for i in range(len(df.index)): text = df['content'][i] token_len = num_tokens_from_string(text) if token_len <= 512: new_list.append([df['title'][i], df['content'][i], df['url'][i], token_len]) else: # add content to the new list in chunks start = 0 ideal_token_size = 512 # 1 token ~ 3/4 of a word ideal_size = int(ideal_token_size // (4/3)) end = ideal_size #split text by spaces into words words = text.split() #remove empty spaces words = [x for x in words if x != ' '] total_words = len(words) #calculate iterations chunks = total_words // ideal_size if total_words % ideal_size != 0: chunks += 1 new_content = [] for j in range(chunks): if end > total_words: end = total_words new_content = words[start:end] new_content_string = ' '.join(new_content) new_content_token_len = num_tokens_from_string(new_content_string) if new_content_token_len > 0: new_list.append([df['title'][i], new_content_string, df['url'][i], new_content_token_len]) start += ideal_size end += ideal_size # Helper function: get embeddings for a text def get_embeddings(text): response = openai.Embedding.create( model="text-embedding-ada-002", input = text.replace("\n"," ") ) embedding = response['data'][0]['embedding'] return embedding # Create embeddings for each piece of content for i in range(len(new_list)): text = new_list[i][1] embedding = get_embeddings(text) new_list[i].append(embedding) # Create a new dataframe from the list df_new = pd.DataFrame(new_list, columns=['title', 'content', 'url', 'tokens', 'embeddings']) df_new.head() # Save the dataframe with embeddings as a CSV file for later use df_new.to_csv('blog_data_and_embeddings.csv', index=False) print("Done! Check the file blog_data_and_embeddings.csv for your results.") ``` 6. **Run `create_embeddings.py`** ```bash python create_embeddings.py ``` You should see an output that looks a bit like this: ```bash Estimated price to embed this content = $0.0060178 Done! Check the file blog_data_and_embeddings.csv for your results. ``` 7. **Create the `insert_embeddings.py` file** To insert these embeddings into your Tiger Cloud service using the `pgvector` extension, open an editor of your choice and create the `insert_embeddings.py` file. insert\_embeddings.py ```python ############################################################################### # This script inserts OpenAI embedding vectors into a Postgres database # using pgvector, a Postgres extension for vector similarity search ############################################################################### import openai import os import pandas as pd import numpy as np import psycopg2 import ast import pgvector import math from psycopg2.extras import execute_values from pgvector.psycopg2 import register_vector ############################################################################### # Setup your database to insert embeddings ############################################################################### # Get a Tiger Cloud / Postgres database connection string by reading local .env file connection_string = os.environ['TIMESCALE_CONNECTION_STRING'] # Connect to Postgres database in Tiger Cloud using connection string conn = psycopg2.connect(connection_string) cur = conn.cursor() #install pgvector in your database cur.execute("CREATE EXTENSION IF NOT EXISTS vector;"); conn.commit() # Register the vector type with psycopg2 register_vector(conn) # Create table to store embeddings and metadata table_create_command = """ CREATE TABLE embeddings ( id bigserial primary key, title text, url text, content text, tokens integer, embedding vector(1536) ); """ cur.execute(table_create_command) cur.close() conn.commit() ############################################################################### # Import embeddings into data frame # Note: Embeddings were created in create_embeddings.py df = pd.read_csv('blog_data_and_embeddings.csv') titles = df['title'] urls = df['url'] contents = df['content'] tokens = df['tokens'] embeds = [list(map(float, ast.literal_eval(embed_str))) for embed_str in df['embeddings']] df_new = pd.DataFrame({ 'title': titles, 'url': urls, 'content': contents, 'tokens': tokens, 'embeddings': embeds }) print(df_new.head()) ############################################################################### # Batch insert embeddings and metadata into database with psycopg2 ############################################################################### register_vector(conn) cur = conn.cursor() # Prepare the list of tuples to insert data_list = [(row['title'], row['url'], row['content'], int(row['tokens']), np.array(row['embeddings'])) for index, row in df_new.iterrows()] # Use execute_values to perform batch insertion execute_values(cur, "INSERT INTO embeddings (title, url, content, tokens, embedding) VALUES %s", data_list) # Commit after we insert all embeddings conn.commit() cur.execute("SELECT COUNT(*) as cnt FROM embeddings;") num_records = cur.fetchone()[0] print("Number of vector records in table: ", num_records,"\n") # Correct output should be 129 # print the first record in the table, for sanity-checking cur.execute("SELECT * FROM embeddings LIMIT 1;") records = cur.fetchall() print("First record in table: ", records) # Create an index on the data for faster retrieval #calculate the index parameters according to best practices num_lists = num_records / 1000 if num_lists < 10: num_lists = 10 if num_records > 1000000: num_lists = math.sqrt(num_records) #use the cosine distance measure, which is what we'll later use for querying cur.execute(f'CREATE INDEX ON embeddings USING ivfflat (embedding vector_cosine_ops) WITH (lists = {num_lists});') conn.commit() print("Index created on embeddings table") ``` 8. **Run `insert_embeddings.py`** ```bash python insert_embeddings.py ``` You should see an output that looks a bit like this: ```bash 0 How to Build a Weather Station With Elixir, Ne... ... [0.021399984136223793, 0.021850213408470154, -... 1 How to Build a Weather Station With Elixir, Ne... ... [0.01620873250067234, 0.011362895369529724, 0.... 2 How to Build a Weather Station With Elixir, Ne... ... [0.022517921403050423, -0.0019158280920237303,... 3 CloudQuery on Using Postgres for Cloud Asset... ... [0.008915113285183907, -0.004873732570558786, ... 4 CloudQuery on Using PostgreSQL for Cloud Asset... ... [0.0204352755099535, 0.010087345726788044, 0.0... [5 rows x 5 columns] Number of vector records in table: 129 First record in table: [(1, 'How to Build a Weather Station With Elixir, Nerves, and TimescaleDB', 'https://www.tigerdata.com/blog/how-to-build-a-weather-station-with-elixir-nerves-and-timescaledb', 'This is an installment of our "Community Member Spotlight" series, where we invite our customers to share their work, shining a light on their success and inspiring others with new ways to use technology to solve problems.In this edition,Alexander Koutmos, author of the Build a Weather Station with Elixir and Nerves book, joins us to share how he uses Grafana and TimescaleDB to store and visualize weather data collected from IoT sensors.About the teamThe bookBuild a Weather Station with Elixir and Nerveswas a joint effort between Bruce Tate, Frank Hunleth, and me.I have been writing software professionally for almost a decade and have been working primarily with Elixir since 2016. I currently maintain a few Elixir libraries onHexand also runStagira, a software consultancy company.Bruce Tateis a kayaker, programmer, and father of two from Chattanooga, Tennessee. He is the author of more than ten books and has been around Elixir from the beginning. He is the founder ofGroxio, a company that trains Elixir developers.Frank Hunlethis an embedded systems programmer, OSS maintainer, and Nerves core team member. When not in front of a computer, he loves running and spending time with his family.About the projectIn the Pragmatic Bookshelf book,Build a Weather Station with Elixir and Nerves, we take a project-based approach and guide the reader to create a Nerves-powered IoT weather station.For those unfamiliar with the Elixir ecosystem,Nervesis an IoT framework that allows you to build and deploy IoT applications on a wide array of embedded devices. At a high level, Nerves allows you to focus on building your project and takes care of a lot of the boilerplate associated with running Elixir on embedded devices.The goal of the book is to guide the reader through the process of building an end-to-end IoT solution for capturing, persisting, and visualizing weather data.Assembled weather station hooked up to development machine.One of the motivating factors for this book was to create a real-world project where readers could get hands-on experience with hardware without worrying too much about the nitty-gritty of soldering components together. Experimenting with hardware can often feel intimidating and confusing, but with Elixir and Nerves, we feel confident that even beginners get comfortable and productive quickly. As a result, in the book, we leverage a Raspberry Pi Zero W along with a few I2C enabled sensors to', 501, array([ 0.02139998, 0.02185021, -0.00537814, ..., -0.01257126, -0.02165324, -0.03714396], dtype=float32))] Index created on embeddings table ``` 9. **Create the `query_embeddings.py` file** To query the embeddings that you inserted into your Tiger Cloud service, open an editor of your choice and create the `query_embeddings.py` file. Here, the query is `How does Density use TimescaleDB?`. query\_embeddings.py ````python ############################################################################### # This script shows how to query embeddings stored in PostgreSQL # to find relevant documents for a given query # and use them to augment a base LLM to answer questions ############################################################################### import openai import os import pandas as pd import numpy as np import json import tiktoken import psycopg2 import ast import pgvector import math from psycopg2.extras import execute_values from pgvector.psycopg2 import register_vector # Get openAI api key by reading local .env file from dotenv import load_dotenv, find_dotenv _ = load_dotenv(find_dotenv()) openai.api_key = os.environ['OPENAI_API_KEY'] # Get Tiger Cloud / Postgres database connection string by reading local .env file connection_string = os.environ['TIMESCALE_CONNECTION_STRING'] # Connect to Postgres Tiger Cloud service using connection string conn = psycopg2.connect(connection_string) ############################################################################### # Helper functions for similarity search on documents in the database ############################################################################### # Helper function: Get top 3 most similar documents from the database def get_top3_similar_docs(query_embedding, conn): embedding_array = np.array(query_embedding) # Register pgvector extension register_vector(conn) cur = conn.cursor() # Get the top 3 most similar documents using the KNN <=> operator cur.execute("SELECT content FROM embeddings ORDER BY embedding <=> %s LIMIT 3", (embedding_array,)) top3_docs = cur.fetchall() return top3_docs # Helper function: get text completion from OpenAI API # Note we're using the latest gpt-3.5-turbo-0613 model # Change the model type to whichever model you prefer (e.g gpt-4) def get_completion_from_messages(messages, model="gpt-3.5-turbo-0613", temperature=0, max_tokens=1000): response = openai.ChatCompletion.create( model=model, messages=messages, temperature=temperature, max_tokens=max_tokens, ) return response.choices[0].message["content"] # Helper function: get embeddings for a text def get_embeddings(text): response = openai.Embedding.create( model="text-embedding-ada-002", input = text.replace("\n"," ") ) embedding = response['data'][0]['embedding'] return embedding ############################################################################### ############################################################################### # Function to process user input # with retrieval of most similar documents from the database # and completion from OpenAI API using Retrieval-Augmented Generation ############################################################################### def process_input_with_retrieval(user_input): delimiter = "```" #Step 1: Get documents related to the user input from database related_docs = get_top3_similar_docs(get_embeddings(user_input), conn) # Step 2: Get completion from OpenAI API # Set system message to help set appropriate tone and context for model system_message = f""" You are a friendly chatbot. \ You can answer questions about timescaledb, its features and its use cases. \ You respond in a concise, technically credible tone. \ """ # Prepare messages to pass to model # We use a delimiter to help the model understand the where the user_input starts and ends messages = [ {"role": "system", "content": system_message}, {"role": "user", "content": f"{delimiter}{user_input}{delimiter}"}, {"role": "assistant", "content": f"Relevant Tiger Data case studies information: \n {related_docs[0] [0]} \n {related_docs[1][0]} {related_docs[2][0]}"} ] final_response = get_completion_from_messages(messages) return final_response ############################################################################### # Question about a Tiger Data blog post we want the model to answer input = "How does Density use TimescaleDB?" # Get a response from the model using most reelvant documents from the database response = process_input_with_retrieval(input) print(input) print(response) ```` 10. **Run `query_embeddings.py`** ```bash python query_embeddings.py ``` You should see an output that looks a bit like this: ```bash How does Density use TimescaleDB? Density uses TimescaleDB as the main database in their smart city system. They store counts of people in spaces over time and derive metrics such as dwell time and space usage. TimescaleDB's flexibility and ability to handle time-series data efficiently allows Density to slice, dice, and compose queries in various ways. They also leverage TimescaleDB's continuous aggregates feature to roll up high-resolution data to lower resolutions, improving query performance. Additionally, TimescaleDB's support for percentile calculations has helped Density deliver accurate percentile values for their data. Overall, TimescaleDB has significantly improved the performance and scalability of Density's analytics workload. ``` # Analyse geospatial data with PostGIS > Store, query, and analyze geospatial data in PostgreSQL using the PostGIS extension The `postgis` PostgreSQL extension provides storing, indexing, and querying geographic data. It helps in spatial data analysis, the study of patterns, anomalies, and theories within spatial or geographical data. For more information about these functions and the options available, see the [PostGIS documentation](https://postgis.net/documentation/). ## Use the `postgis` extension to analyze geospatial data The `postgis` PostgreSQL extension allows you to conduct complex analyses of your geospatial time-series data. Tiger Data understands that you have a multitude of data challenges and helps you discover when things happened, and where they occurred. In this example you can query when the `covid` cases were reported, where they were reported, and how many were reported around a particular location. 1. **Install the `postgis` extension** ```sql CREATE EXTENSION postgis; ``` Confirm the extension is installed using the `\dx` command: ```sql List of installed extensions Name | Version | Schema | Description ---------------------+---------+------------+--------------------------------------------------------------------------------------- pg_stat_statements | 1.10 | public | track planning and execution statistics of all SQL statements executed pgcrypto | 1.3 | public | cryptographic functions plpgsql | 1.0 | pg_catalog | PL/pgSQL procedural language postgis | 3.3.3 | public | PostGIS geometry and geography spatial types and functions timescaledb | 2.24.0 | public | Enables scalable inserts and complex queries for time-series data (Community Edition) timescaledb_toolkit | 1.22.0 | public | Library of analytical hyperfunctions, time-series pipelining, and other SQL utilities (6 rows) ``` 2. **Create a hypertable named `covid_location`** `location` is a `GEOGRAPHY` type column that stores GPS coordinates using the 4326/WGS84 coordinate system, and `time` records the time the GPS coordinate was logged for a specific `state_id`. This hypertable is partitioned on the `time` column: ```sql CREATE TABLE covid_location ( time TIMESTAMPTZ NOT NULL, state_id INT NOT NULL, location GEOGRAPHY(POINT, 4326), cases INT NOT NULL, deaths INT NOT NULL ) WITH ( tsdb.hypertable ); ``` When you create a hypertable using [CREATE TABLE ... WITH ...](/docs/reference/timescaledb/hypertables/create_table), the default partitioning column is automatically the first column with a timestamp data type. Also, TimescaleDB creates a [columnstore policy](/docs/reference/timescaledb/hypercore/add_columnstore_policy) that automatically converts your data to the columnstore, after an interval equal to the value of the [chunk\_interval](/docs/reference/timescaledb/hypertables/set_chunk_time_interval), defined through `after` in the policy. This columnar format enables fast scanning and aggregation, optimizing performance for analytical workloads while also saving significant storage space. In the columnstore conversion, hypertable chunks are compressed by up to 98%, and organized for efficient, large-scale queries. You can customize this policy later using [alter\_job](/docs/reference/timescaledb/jobs-automation/alter_job). However, to change `after` or `created_before`, the compression settings, or the hypertable the policy is acting on, you must [remove the columnstore policy](/docs/reference/timescaledb/hypercore/remove_columnstore_policy) and [add a new one](/docs/reference/timescaledb/hypercore/add_columnstore_policy). You can also manually [convert chunks](/docs/reference/timescaledb/hypercore/convert_to_columnstore) in a hypertable to the columnstore. 3. **Create an index on the `state_id` column** ```sql CREATE INDEX ON covid_location (state_id, time DESC); ``` 4. **Insert sample data** The longitude and latitude coordinates of New Jersey are (-73.935242 40.730610), and New York are (-74.871826 39.833851): ```sql INSERT INTO covid_location VALUES ('2023-06-28 20:00:00',34,'POINT(-74.871826 39.833851)',5,2), ('2023-06-28 20:00:00',36,'POINT(-73.935242 40.730610)',7,1), ('2023-06-29 20:00:00',34,'POINT(-74.871826 39.833851)',14,0), ('2023-06-29 20:00:00',36,'POINT(-73.935242 40.730610)',12,1), ('2023-06-30 20:00:00',34,'POINT(-74.871826 39.833851)',10,4); ``` 5. **Fetch all cases of a specific state during a specific period** ```sql SELECT * FROM covid_location WHERE state_id = 34 AND time BETWEEN '2023-06-28 00:00:00' AND '2023-06-30 23:59:59'; ``` The data you get back looks a bit like this: ```sql time | state_id | location | cases | deaths ------------------------+----------+----------------------------------------------------+-------+-------- 2023-06-28 20:00:00+00 | 34 | 0101000020E61000005C7347FFCBB752C0535E2BA1BBEA4340 | 5 | 2 2023-06-29 20:00:00+00 | 34 | 0101000020E61000005C7347FFCBB752C0535E2BA1BBEA4340 | 14 | 0 2023-06-30 20:00:00+00 | 34 | 0101000020E61000005C7347FFCBB752C0535E2BA1BBEA4340 | 10 | 4 (3 rows) ``` 6. **Fetch the latest logged cases using SkipScan** Replace `` with the number of days between the day you are running the query and the day the last report was logged in the table, in this case 30, June, 2023: ```sql SELECT DISTINCT ON (state_id) state_id, ST_AsText(location) AS location FROM covid_location WHERE time > now() - INTERVAL '' ORDER BY state_id, time DESC; ``` The `ST_AsText(location)` function converts the binary geospatial data into human-readable format. The data you get back looks a bit like this: ```sql state_id | location ----------+----------------------------- 34 | POINT(-74.871826 39.833851) (1 row) ``` 7. **Fetch all cases within 10000 meters of Manhattan** ```sql SELECT DISTINCT cases, state_id FROM covid_location WHERE ST_DWithin( location, ST_GeogFromText('POINT(-73.9851 40.7589)'), 10000 ); ``` The data you get back looks a bit like this: ```sql cases | state_id -------+---------- 7 | 36 12 | 36 (2 rows) ``` # Maintenance and upgrades > Keep your Tiger Cloud service up to date with managed upgrades and maintenance windows Tiger Cloud offers managed database services that provide a stable and reliable environment for your applications. Each service is based on a specific version of the PostgreSQL database and the TimescaleDB extension. To ensure that you benefit from the latest features, performance and security improvements, it is important that your Tiger Cloud service is kept up to date with the latest versions of TimescaleDB and PostgreSQL. Tiger Cloud has the following upgrade policies: * **TimescaleDB upgrades**: * [Non-critical upgrades](#non-critical-upgrades) — new minor versions and patch releases, applied automatically during a [maintenance window](#define-your-maintenance-window) that you define. Or upgrade manually earlier. * [Critical security patches](#critical-security-patches) — installed outside normal maintenance windows when necessary, sometimes with a short outage. * **PostgreSQL upgrades**: * Minor version upgrades — applied automatically during your [maintenance window](#define-your-maintenance-window). * Major version upgrades — [performed by you](#manually-upgrade-postgresql), on your own schedule. Tiger Cloud only performs an [automatic upgrade](#automatic-upgrades-for-deprecated-versions) **when your current PostgreSQL version reaches end-of-life**. Note After a maintenance upgrade, the DNS name remains the same. However, the IP address often changes. ## Check your current versions In [Tiger Console](https://console.cloud.tigerdata.com/dashboard/services), select your service, then click `Operations` > `Service Upgrades`. This shows your current TimescaleDB and PostgreSQL versions, and whether a newer version is available. ![Service upgrades panel in Tiger Console showing current and latest PostgreSQL and TimescaleDB versions](/docs/_astro/tiger-console-service-upgrades-light.BE5rfg9j_1pfnWz.webp) ![Service upgrades panel in Tiger Console showing current and latest PostgreSQL and TimescaleDB versions](/docs/_astro/tiger-console-service-upgrades-dark.CX8HFv7i_Z19OqDH.webp) ## TimescaleDB upgrades Tiger Cloud keeps your services on a supported version of TimescaleDB. Non-critical upgrades — new minor versions and patch releases — are applied during your [maintenance window](#define-your-maintenance-window), or you can [trigger them manually](#non-critical-upgrades). Critical security patches are installed as soon as they are available. ### Non-critical upgrades Non-critical upgrades — new minor versions and patch releases — are applied automatically in the next available maintenance window. The schedule depends on the upgrade type and your service's environment tag: * **New minor versions on `#dev` services**: applied in the next maintenance window after the release. * **New minor versions on `#prod` services**: applied in the next maintenance window three weeks after the release. This gives you three weeks to validate the release on your `#dev` services first. * **Patch releases**: applied in the next maintenance window on both `#dev` and `#prod` services, with no waiting period. Occasionally, a new minor version is released before the previous one has completed its three-week window — for example, when it contains an important fix. In that case, the newer minor version takes priority: your `#prod` services skip the previous minor and target the new one instead, and the three-week window restarts from the new release. Your `#dev` services move to the new minor in their next maintenance window, as usual. [Subscribe](https://status.tigerdata.com/) to get an email notification before your `#dev` services are upgraded. If there are no pending upgrades during a maintenance window, no changes are performed. To apply a non-critical TimescaleDB upgrade manually before the next maintenance window: 1. **Connect to your service** In [Tiger Console](https://console.cloud.tigerdata.com/dashboard/services), select the service you want to upgrade. 2. **Upgrade TimescaleDB** Click `SQL Editor` at the bottom, then run `ALTER EXTENSION timescaledb UPDATE`. ### Critical security patches Critical security patches are installed outside your normal maintenance window as soon as they are available. ### Downtime and connection resets Most upgrades do not require any downtime. Connections and transactions in progress during the upgrade are reset, then the service connection is usually restored automatically. Some background maintenance tasks performed outside the maintenance window can also cause connection resets — when one occurs, you can immediately reconnect. Some non-critical upgrades and critical security patches do require a short outage, usually between 30 seconds and 5 minutes. Tiger Data endeavors to notify you by email ahead of an upgrade that requires downtime. In some cases — particularly for critical security patches — advance notice is not possible. To track the status of maintenance events, see the Tiger Cloud [status page](https://status.tigerdata.com/). To minimize the impact of downtime, [schedule your maintenance window](#define-your-maintenance-window) for a low-traffic period in your application, and add a [high-availability replica or read replica](high-availability/high-availability). With a replica, writes pause only briefly during maintenance and read-only queries continue to be served by the replica throughout the upgrade. During a maintenance event, each node is upgraded independently. The primary is upgraded and restarted first: * If the primary restarts within a minute, no failover occurs. The replica stays as the replica, and maintenance then proceeds on it. * If the primary takes longer than a minute to restart — and the replica has no replication lag — the replica is promoted to primary (one failover). Maintenance then proceeds on the demoted former primary. If that node is also slow to restart, the original primary is promoted back (a second failover). Each failover takes less than a few seconds. In the worst case, two failovers add up to about two minutes of write downtime. ## PostgreSQL upgrades New minor versions of PostgreSQL are applied automatically during your [maintenance window](#define-your-maintenance-window). You upgrade to a new major version of PostgreSQL manually, on your own schedule. Tiger Cloud only performs an [automatic upgrade](#automatic-upgrades-for-deprecated-versions) once your current PostgreSQL version reaches end-of-life. Upgrading to a newer version of PostgreSQL allows you to take advantage of new features, enhancements, and security fixes. It also ensures that you are using a version of PostgreSQL that's compatible with the newest version of TimescaleDB. For more information about feature changes between versions, see the [Tiger Cloud release notes](/docs/get-started/news/new), [supported systems](/docs/get-started/choose-your-path/supported-platforms), and the [PostgreSQL release notes](https://www.postgresql.org/docs/release/). The following table shows you the compatible versions of PostgreSQL and TimescaleDB: Warning TimescaleDB 2.28.x is the final minor version to support PostgreSQL 15. Support for PostgreSQL 15 is dropped in 2.29.0; patch releases in the 2.28.x cycle continue to support it. | TimescaleDB version | PostgreSQL 18 | PostgreSQL 17 | PostgreSQL 16 | PostgreSQL 15 | PostgreSQL 14 | PostgreSQL 13 | PostgreSQL 12 | PostgreSQL 11 | PostgreSQL 10 | | ------------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | | 2.28.x | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | 2.27.x | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | 2.26.x | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | 2.25.x | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | 2.24.x | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | 2.23.x | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | 2.22.x | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | 2.21.x | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | 2.20.x | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | 2.17 - 2.19 | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | | 2.16.x | ❌ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | | 2.13 - 2.15 | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | | 2.12.x | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | | 2.10.x | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | | 2.5 - 2.9 | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | | 2.4 | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | | 2.1 - 2.3 | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ❌ | | 2.0 | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | | 1.7 | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | We recommend not using TimescaleDB with PostgreSQL 17.1, 16.5, 15.9, 14.14, 13.17, 12.21.\ These minor versions [introduced a breaking binary interface change](https://www.postgresql.org/about/news/postgresql-172-166-1510-1415-1318-and-1222-released-2965/) that, once identified, was reverted in subsequent minor PostgreSQL versions 17.2, 16.6, 15.10, 14.15, 13.18, and 12.22. When you build from source, best practice is to build with PostgreSQL 17.2, 16.6, etc and higher. Users of [Tiger Cloud](https://console.cloud.tigerdata.com/) and platform packages for Linux, Windows, MacOS, Docker, and Kubernetes are unaffected. ### Manually upgrade PostgreSQL For a smooth upgrade experience, make sure you: * **Plan ahead**: upgrades cause downtime, so ideally perform an upgrade during a low traffic time. * **Run a test upgrade**: [fork your service](service-management/fork-services), then try out the upgrade on the fork before running it on your production system. This gives you a good idea of what happens during the upgrade, and how long it might take. * **Keep a copy of your service**: if you're worried about losing your data, [fork your service](service-management/fork-services) without upgrading, and keep this duplicate of your service. To reduce cost, you can immediately pause this fork and only pay for storage until you are comfortable deleting it after the upgrade is complete. Important Tiger Cloud services with replicas cannot be upgraded. To upgrade a service with a replica, you must first delete the replica and then upgrade the service. Warning Your Tiger Cloud service is unavailable until the upgrade is complete. This can take up to 20 minutes. Best practice is to test on a fork first, so you can estimate how long the upgrade will take. To upgrade your service to a newer version of PostgreSQL: 1. **Connect to your service** In [Tiger Console](https://console.cloud.tigerdata.com/dashboard/services), select the service you want to upgrade. 2. **Disable high-availability replicas** 1. Click `Operations` > `High Availability`, then click `Change configuaration`. 2. Select `Non-production (No replica)`, then click `Change configuration`. 3. **Disable read replicas** Click `Operations` > `Read scaling`, then click the trash icon next to all read replica sets. 4. **Upgrade PostgreSQL** 1. Click `Operations` > `Service Upgrades`. 2. Click `Upgrade service`, then confirm that you are ready to start the upgrade. Your Tiger Cloud service is unavailable until the upgrade is complete. This normally takes up to 20 minutes. However, it can take longer if you have a large or complex service. When the upgrade is finished, your service automatically resumes normal operations. If the upgrade is unsuccessful, the service returns to the state it was in before you started the upgrade. 5. **Enable high-availability replicas and replace your read replicas** ### Automatic upgrades for deprecated versions To ensure you benefit from the latest features, optimal performance, enhanced security, and full compatibility with TimescaleDB, Tiger Cloud supports a defined set of PostgreSQL major versions. As PostgreSQL and TimescaleDB evolve, Tiger Data periodically deprecates older PostgreSQL versions. If you do not upgrade before a deprecated version's deadline, Tiger Cloud automatically upgrades your service to keep it on a supported PostgreSQL version. The automatic upgrade is applied during your [maintenance window](#define-your-maintenance-window). Automatic upgrades can result in downtime, so best practice is to [upgrade manually](#manually-upgrade-postgresql) during a low-traffic period for your application. Tiger Data provides advance notification to allow you ample time to plan and perform your upgrade. The deprecation timeline is as follows: * **Deprecation notice**: you receive email notification of the deprecation and the timeline for the upgrade. * **Manual upgrade period**: [upgrade your service manually](#manually-upgrade-postgresql) before the deadline. * **Automatic upgrade deadline**: Tiger Cloud performs an automatic upgrade of your service. During an automatic upgrade: 1. Any configured [high-availability replicas](high-availability/high-availability) or [read replicas](high-availability/read-scaling) are temporarily removed. 2. The primary service is upgraded. 3. High-availability replicas and read replicas are added back to the service. ## Define your maintenance window When you are considering your maintenance window schedule, best practice is to choose a day and time that usually has very low activity, such as during the early hours of the morning, or over the weekend. This helps minimize the impact of a short service interruption. Alternatively, you might prefer to have your maintenance window occur during office hours, so that you can monitor your system during the upgrade. To change your maintenance window: 1. **Connect to your service** In [Tiger Console](https://console.cloud.tigerdata.com/dashboard/services), select the service you want to manage. 2. **Set your maintenance window** 1. Click `Operations` > `Environment`, then click `Change maintenance window`. ![Maintenance and upgrades](/docs/_astro/tiger-console-maintenance-upgrades-light.wucfJVqm_Z1SdmaA.webp) ![Maintenance and upgrades](/docs/_astro/tiger-console-maintenance-upgrades-dark.DNAqiJWY_Z1Jgwrf.webp) 2. Select the maintenance window start time, then click `Apply`. Maintenance windows can run for up to four hours. # Troubleshoot Tiger Cloud > Diagnose and resolve common issues with Tiger Cloud services ## Cannot create another database ```text ERROR: tsdb_admin: database is not an allowed database name HINT: Contact your administrator to configure the "tsdb_admin.allowed_databases" ``` Each Tiger Cloud service hosts a single PostgreSQL instance called `tsdb`. You see this error when you try to create an additional database in a service. If you need another database, [create a new service](/docs/get-started/quickstart/create-service). ## User permissions do not allow chunks to be converted to columnstore or rowstore ```text ERROR: must be owner of hypertable "HYPERTABLE_NAME" ``` You might get this error if you attempt to compress a chunk into the columnstore, or decompress it back into rowstore with a non-privileged user account. To compress or decompress a chunk, your user account must have permissions that allow it to perform `CREATE INDEX` on the chunk. You can check the permissions of the current user with this command at the `psql` command prompt: ```sql \dn+ ``` To resolve this problem, grant your user account the appropriate privileges with this command: ```sql GRANT PRIVILEGES ON TABLE TO ; ``` For more information about the `GRANT` command, see the [PostgreSQL documentation](https://www.postgresql.org/docs/current/sql-grant.html). ## Background jobs failing with "out of shared memory" You might see this error when running compression, continuous aggregate refresh, or other background jobs that touch many chunks: ```text FATAL: out of shared memory HINT: You might need to increase max_locks_per_transaction. ``` Despite the wording, this error is **not caused by insufficient RAM**. It means the PostgreSQL lock table is full. TimescaleDB acquires a lock on every chunk involved in a query or background job. When a hypertable has many chunks, these locks can exceed the default `max_locks_per_transaction` limit (usually 64). ### Diagnose the issue 1. **Check the current setting:** ```sql SELECT name, setting FROM pg_settings WHERE name = 'max_locks_per_transaction'; ``` 2. **Count chunks per hypertable:** ```sql SELECT hypertable_name, num_chunks FROM timescaledb_information.hypertables ORDER BY num_chunks DESC; ``` ### Calculate the right value For most workloads, use this formula: ```text max_locks_per_transaction = (2 × max_chunks_in_any_hypertable) / max_connections ``` The factor of 2 accounts for index locks. Round up and add headroom for future growth, because changing this parameter requires a database restart. ### Apply the fix Tips **Tiger Cloud:** Adjust `max_locks_per_transaction` from Tiger Console under **Database configuration → Advanced parameters**. Search for the parameter, edit the value, and click **Apply changes and restart**. See [Advanced parameters](/docs/deploy/tiger-cloud/tiger-cloud-aws/configuration/advanced-parameters) for details. **Self-hosted:** Set `max_locks_per_transaction` in the `postgresql.conf` configuration file, then restart PostgreSQL. For more information, see [Transaction lock settings](/docs/reference/timescaledb/configuration/tiger-postgres#transaction-lock-settings) and the [PostgreSQL lock management documentation](https://www.postgresql.org/docs/current/runtime-config-locks.html). # Vectorizer and in-database LLM calls migration guide > Plan your move off Tiger Cloud-managed vectorizer workers and in-database LLM SQL helpers before June 30, 2026. Tables, embeddings, and vectorizer definitions stay in the database. This guide is for **Tiger Cloud** (AWS and Azure). It does not apply to self-hosted deployments that never used Tiger Cloud-managed vectorizer scheduling. The following AI capabilities are deprecated on Tiger Cloud and **will be removed on June 30, 2026**: * **Managed vectorizer**: the Tiger Cloud-managed service that ran vectorizer workers for you. Your vectorizer definitions and embedding tables stay in place; you run the **vectorizer worker** yourself (see [Migrate the vectorizer](#migrate-the-vectorizer)). * **In-database LLM calls**: SQL-callable helpers such as `ai.openai_embed`, `ai.openai_chat_complete`, `ai.anthropic_generate`, `ai.ollama_embed`, `ai.cohere_embed`, and related `ai.*` entry points. These will no longer be available from SQL. **Your data is not affected.** Tables, embeddings, and vectorizer configuration remain in the database. Only Tiger Cloud-managed execution of the worker and the in-database LLM call surface are going away. **What is not changing:** Semantic search with [pgvector](https://github.com/pgvector/pgvector) and [pgvectorscale](https://github.com/timescale/pgvectorscale), and keyword search with BM25 via [pg\_textsearch](https://github.com/timescale/pg_textsearch), stay available on Tiger Cloud. Those are still the building blocks for hybrid search. ## Upgrade the extension [Section titled “Upgrade the extension”](#upgrade-the-extension) In pgai **v0.10.0**, vectorizer code moved out of the `ai` extension into the standalone **pgai** Python package. On extension versions **0.4.0 through 0.9.x**, vectorizer SQL objects still belong to the extension. **Upgrade the extension** before you follow the worker migration below. ### 1. check your current pgai version [Section titled “1. check your current pgai version”](#1-check-your-current-pgai-version) ```sql SELECT extversion FROM pg_extension WHERE extname = 'ai'; ``` If the result is `0.9.x` or earlier, complete the upgrade steps below. If it is **0.10.0** or later, skip to [Migrate the vectorizer](#migrate-the-vectorizer). ### 2. upgrade pgai to the latest version [Section titled “2. upgrade pgai to the latest version”](#2-upgrade-pgai-to-the-latest-version) ```sql ALTER EXTENSION ai UPDATE; ``` This migration detaches vectorizer objects from the extension **without** dropping them. Definitions and data stay intact. ### 3. install the pgai library for ongoing vectorizer management [Section titled “3. install the pgai library for ongoing vectorizer management”](#3-install-the-pgai-library-for-ongoing-vectorizer-management) **pip:** ```bash pip install "pgai[vectorizer-worker]" pgai install -d "postgres://tsdbadmin:@:/tsdb?sslmode=require" ``` **Docker:** ```bash docker run --pull always --rm --entrypoint python \ timescale/pgai-vectorizer-worker:latest \ -m pgai install -d "postgres://tsdbadmin:@:/tsdb?sslmode=require" ``` After these steps, continue with [Migrate the vectorizer](#migrate-the-vectorizer). ## Migrate the vectorizer [Section titled “Migrate the vectorizer”](#migrate-the-vectorizer) ### 1. disable cloud scheduling [Section titled “1. disable cloud scheduling”](#1-disable-cloud-scheduling) Connect to your database and remove the cloud scheduler from your vectorizers. This deletes the TimescaleDB background jobs and sets scheduling to `none`, while leaving vectorizers enabled so your self-hosted worker can process them: ```sql -- Delete the TimescaleDB scheduled jobs SELECT public.delete_job((config->'scheduling'->>'job_id')::int) FROM ai.vectorizer WHERE config->'scheduling'->>'implementation' = 'timescaledb'; -- Switch scheduling to none UPDATE ai.vectorizer SET config = jsonb_set(config, '{scheduling}', '{"config_type": "scheduling", "implementation": "none"}'::jsonb) WHERE config->'scheduling'->>'implementation' = 'timescaledb'; ``` ### 2. get your connection string [Section titled “2. get your connection string”](#2-get-your-connection-string) Copy your [Tiger Cloud connection string](/docs/integrate/find-connection-details) from Tiger Console. It looks like: ```plaintext postgres://tsdbadmin:@:/tsdb?sslmode=require ``` ### 3. run the vectorizer worker [Section titled “3. run the vectorizer worker”](#3-run-the-vectorizer-worker) Pick one of the following. #### Option A: Docker (recommended) [Section titled “Option A: Docker (recommended)”](#option-a-docker-recommended) Create a `.env` file with your API keys: ```plaintext OPENAI_API_KEY=sk-your-openai-api-key ``` Run the worker: ```bash docker run \ --env-file .env \ timescale/pgai-vectorizer-worker:latest \ --db-url "postgres://tsdbadmin:@:/tsdb?sslmode=require" \ --poll-interval 5m \ -c 4 ``` #### Option B: Docker Compose [Section titled “Option B: Docker Compose”](#option-b-docker-compose) ```yaml name: pgai-vectorizer services: vectorizer-worker: image: timescale/pgai-vectorizer-worker:latest environment: PGAI_VECTORIZER_WORKER_DB_URL: "postgres://tsdbadmin:@:/tsdb?sslmode=require" OPENAI_API_KEY: "sk-your-openai-api-key" command: ["--poll-interval", "5m", "-c", "4"] restart: unless-stopped ``` Start it: ```bash docker compose up -d ``` #### Option C: CLI [Section titled “Option C: CLI”](#option-c-cli) ```bash pip install pgai[vectorizer-worker] ``` ```bash export OPENAI_API_KEY=sk-your-openai-api-key pgai vectorizer worker -d "postgres://tsdbadmin:@:/tsdb?sslmode=require" --poll-interval 5m -c 4 ``` #### Option D: Python [Section titled “Option D: Python”](#option-d-python) ```python import asyncio from datetime import timedelta from pgai import Worker worker = Worker( db_url="postgres://tsdbadmin:@:/tsdb?sslmode=require", poll_interval=timedelta(minutes=5), concurrency=4, ) asyncio.run(worker.run()) ``` Full worker options: [pgai vectorizer worker documentation](https://github.com/timescale/pgai/blob/main/docs/vectorizer/worker.md). ## Migrate away from in-database LLM calls [Section titled “Migrate away from in-database LLM calls”](#migrate-away-from-in-database-llm-calls) Move provider calls into **application code**. Keep using PostgreSQL for storage and vector search; generate embeddings and completions **outside** the database, then pass vectors or text into SQL as parameters. ### Migrate embedding calls [Section titled “Migrate embedding calls”](#migrate-embedding-calls) **Before** (embedding inside the database): ```sql SELECT id, content FROM documents ORDER BY embedding <=> ai.openai_embed('text-embedding-3-small', 'search query') LIMIT 5; ``` **After** (embedding in Python, vector as a query parameter): ```python import openai import psycopg2 client = openai.OpenAI() # uses OPENAI_API_KEY env var def semantic_search(query: str, limit: int = 5): response = client.embeddings.create( model="text-embedding-3-small", input=query, ) embedding = response.data[0].embedding conn = psycopg2.connect("postgres://tsdbadmin:@:/tsdb?sslmode=require") cur = conn.cursor() cur.execute( """ SELECT id, content FROM documents ORDER BY embedding <=> %s::vector LIMIT %s """, (embedding, limit), ) return cur.fetchall() ``` With **asyncpg**, bind the embedding the way your driver and pgvector build expect (for example a string literal for `vector` or a supported sequence type): ```python import openai import asyncpg client = openai.OpenAI() async def semantic_search(query: str, limit: int = 5): response = client.embeddings.create( model="text-embedding-3-small", input=query, ) embedding = response.data[0].embedding conn = await asyncpg.connect("postgres://tsdbadmin:@:/tsdb?sslmode=require") rows = await conn.fetch( """ SELECT id, content FROM documents ORDER BY embedding <=> $1::vector LIMIT $2 """, str(embedding), limit, ) return rows ``` ### Migrate chat completion calls [Section titled “Migrate chat completion calls”](#migrate-chat-completion-calls) **Before:** ```sql SELECT ai.openai_chat_complete( 'gpt-4o', jsonb_build_array( jsonb_build_object('role', 'user', 'content', 'Summarize this: ' || doc.content) ) )->'choices'->0->'message'->>'content' AS summary FROM documents doc WHERE doc.id = 1; ``` **After:** ```python import openai client = openai.OpenAI() def summarize(content: str) -> str: response = client.chat.completions.create( model="gpt-4o", messages=[{"role": "user", "content": f"Summarize this: {content}"}], ) return response.choices[0].message.content ``` ### Migrate Anthropic calls [Section titled “Migrate Anthropic calls”](#migrate-anthropic-calls) **Before:** ```sql SELECT ai.anthropic_generate( 'claude-sonnet-4-20250514', jsonb_build_array( jsonb_build_object('role', 'user', 'content', 'Explain this concept') ) ); ``` **After:** ```python import anthropic client = anthropic.Anthropic() def generate(prompt: str) -> str: message = client.messages.create( model="claude-sonnet-4-20250514", max_tokens=1024, messages=[{"role": "user", "content": prompt}], ) return message.content[0].text ``` ### Migrate Cohere reranking [Section titled “Migrate Cohere reranking”](#migrate-cohere-reranking) **Before:** ```sql SELECT ai.cohere_rerank( 'rerank-english-v3.0', 'search query', jsonb_agg(content) ) FROM documents LIMIT 100; ``` **After:** ```python import cohere client = cohere.Client() def rerank(query: str, documents: list[str]) -> list: response = client.rerank( model="rerank-english-v3.0", query=query, documents=documents, ) return response.results ``` ### Summary of function replacements [Section titled “Summary of function replacements”](#summary-of-function-replacements) | Deprecated function | Replacement | | ----------------------------------------------- | ------------------------------------------------------------------------- | | `ai.openai_embed(model, text)` | `openai.OpenAI().embeddings.create(model=model, input=text)` | | `ai.openai_chat_complete(model, messages)` | `openai.OpenAI().chat.completions.create(model=model, messages=messages)` | | `ai.openai_chat_complete_simple(model, prompt)` | `openai.OpenAI().chat.completions.create(model=model, messages=[...])` | | `ai.openai_moderate(model, input)` | `openai.OpenAI().moderations.create(model=model, input=input)` | | `ai.anthropic_generate(model, messages)` | `anthropic.Anthropic().messages.create(model=model, messages=messages)` | | `ai.ollama_embed(model, text)` | `ollama.embed(model=model, input=text)` | | `ai.ollama_generate(model, prompt)` | `ollama.generate(model=model, prompt=prompt)` | | `ai.ollama_chat_complete(model, messages)` | `ollama.chat(model=model, messages=messages)` | | `ai.cohere_embed(model, text)` | `cohere.Client().embed(model=model, texts=[text])` | | `ai.cohere_rerank(model, query, docs)` | `cohere.Client().rerank(model=model, query=query, documents=docs)` | | `ai.cohere_chat_complete(model, messages)` | `cohere.Client().chat(model=model, messages=messages)` | | `ai.voyageai_embed(model, text)` | `voyageai.Client().embed(texts=[text], model=model)` | | `ai.voyageai_rerank(model, query, docs)` | `voyageai.Client().rerank(query=query, documents=docs, model=model)` | ### General migration pattern [Section titled “General migration pattern”](#general-migration-pattern) For any `ai.*` SQL function: 1. **Identify the provider** from the name (`openai_`, `anthropic_`, `ollama_`, `cohere_`, `voyageai_`). 2. **Install that provider’s SDK** (`pip install openai`, `pip install anthropic`, and so on). 3. **Call the SDK from your application** before or after your SQL. 4. **Pass structured results as parameters** (for embeddings, pass the vector into `ORDER BY ... <=> $1` style queries). ## Next steps [Section titled “Next steps”](#next-steps) | Topic | Where to read | | --------------------------------- | ------------------------------------------------------------------------------------------------------------------ | | pgai SQL and vectorizer reference | **[pgai on GitHub](https://github.com/timescale/pgai)** | | Vector search concepts | **[Key vector concepts for pgvector](/docs/learn/search/key-vector-database-concepts-for-understanding-pgvector)** | | Tiger Cloud + pgvector overview | **[Understand pgvector and pgvectorscale](/docs/learn/search/pgvector-pgvectorsearch)** | # Get started with Tiger Data > Welcome to Tiger Data, choose your path to get started Tiger Data delivers PostgreSQL optimized for time-series, analytics, and AI workloads. Ingest millions of rows per second, query terabytes in milliseconds, and reduce storage costs by up to 98%, all with the PostgreSQL you already know. ## Choose your setup [Section titled “Choose your setup”](#choose-your-setup) [Tiger Cloud](/docs/get-started/quickstart/quickstart-5-minutes) [Fully managed on AWS or Azure. Built-in high availability, tiered storage, observability, and automated operations.](/docs/get-started/quickstart/quickstart-5-minutes) [Self-hosted TimescaleDB](/docs/get-started/choose-your-path/install-timescaledb) [Install the open-source engine on your own infrastructure. Full control over deployment and configuration.](/docs/get-started/choose-your-path/install-timescaledb) Looking to host on Google Cloud? [Explore MST →](/docs/deploy/mst) ## Why Tiger Data? [Section titled “Why Tiger Data?”](#why-tiger-data) **Hypertables** automatically partition your data by time, making inserts and queries dramatically faster without changing your SQL. **Hypercore storage engine** combines row and columnar storage, queries run up to 350x faster, ingestion is 44% faster, and storage shrinks by up to 98%. **Continuous aggregates** pre-compute and auto-refresh your analytics, so dashboards and reports respond instantly. **Tiered storage** seamlessly moves cold data to low-cost object storage while keeping it fully queryable. **100% PostgreSQL compatible**, use your existing tools, drivers, and SQL skills. ## Keep on going! [Section titled “Keep on going!”](#keep-on-going) Once you're up and running, learn how things work, build your first queries, migrate your data, or configure your deployment. [Learn](/docs/learn) [Understand the concepts behind hypertables, hypercore, continuous aggregates, and more](/docs/learn) [Build](/docs/build) [How-to guides for writing queries, managing data, integrating tools, and optimizing performance](/docs/build) [Migrate](/docs/migrate) [Move your data from PostgreSQL, S3, Kafka, and other sources](/docs/migrate) [Deploy](/docs/deploy) [Configure, secure, scale, and monitor your deployment](/docs/deploy) ## What's new [Section titled “What's new”](#whats-new) Check out [News and updates](/docs/get-started/news) for the latest features and improvements. # Install self-hosted TimescaleDB > Install TimescaleDB on Linux, macOS, Windows, Docker, Kubernetes, or from source TimescaleDB is an open-source PostgreSQL extension that powers Tiger Cloud. Designed for running real-time analytics on time-series data, it supercharges ingest, query, storage, and analytics performance. Tips The following instructions are for development and testing installations. For a production environment, we strongly recommend that you implement the following, many of which you can achieve using PostgreSQL tooling: * Incremental backup and database snapshots, with efficient point-in-time recovery. * High availability replication, ideally with nodes across multiple availability zones. * Automatic failure detection with fast restarts, for both non-replicated and replicated deployments. * Asynchronous replicas for scaling reads when needed. * Connection poolers for scaling client connections. * Zero-down-time minor version and extension upgrades. * Forking workflows for major version upgrades and other feature testing. * Monitoring and observability. - Linux Install TimescaleDB based on your distribution. Tiger Data recommends Ubuntu or Debian for the smoothest installation and support experience. Warning If you have previously installed PostgreSQL, you may encounter errors following these install instructions. Best practice is to fully remove any existing PostgreSQL installations before you begin. To keep your current PostgreSQL installation, install from source. * Debian 1. **Install the latest PostgreSQL packages** ```bash sudo apt install gnupg postgresql-common apt-transport-https lsb-release wget ``` 2. **Run the PostgreSQL package setup script** ```bash sudo /usr/share/postgresql-common/pgdg/apt.postgresql.org.sh ``` 3. **Add the TimescaleDB package** ```bash echo "deb https://packagecloud.io/timescale/timescaledb/debian/ $(lsb_release -c -s) main" | sudo tee /etc/apt/sources.list.d/timescaledb.list ``` 4. **Install the TimescaleDB GPG key** ```bash wget --quiet -O - https://packagecloud.io/timescale/timescaledb/gpgkey | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/timescaledb.gpg ``` 5. **Update your local repository list** ```bash sudo apt update ``` 6. **Install TimescaleDB** ```bash sudo apt install timescaledb-2-postgresql-18 postgresql-client-18 ``` To install a specific TimescaleDB [release](https://packagecloud.io/timescale/timescaledb), set the version. For example: `sudo apt-get install timescaledb-2-postgresql-14='2.6.0*' timescaledb-2-loader-postgresql-14='2.6.0*'` Older versions of TimescaleDB may not support all the OS versions listed on this page. 7. **Tune your PostgreSQL instance for TimescaleDB** ```bash sudo timescaledb-tune ``` By default, this script is included with the `timescaledb-tools` package when you install TimescaleDB. Use the prompts to tune your development or production environment. For more information on manual configuration, see [Configuration](/docs/reference/configuration). If you have an issue, run `sudo apt install timescaledb-tools`. 8. **Restart PostgreSQL** ```bash sudo systemctl restart postgresql ``` 9. **Log in to PostgreSQL as `postgres`** ```bash sudo -u postgres psql ``` You are in the psql shell. 10. **Set the password for `postgres`** ```bash \password postgres ``` When you have set the password, type `\q` to exit psql. * Ubuntu 1. **Install the latest PostgreSQL packages** ```bash sudo apt install gnupg postgresql-common apt-transport-https lsb-release wget ``` 2. **Run the PostgreSQL package setup script** ```bash sudo /usr/share/postgresql-common/pgdg/apt.postgresql.org.sh ``` 3. **Add the TimescaleDB package** ```bash echo "deb https://packagecloud.io/timescale/timescaledb/ubuntu/ $(lsb_release -c -s) main" | sudo tee /etc/apt/sources.list.d/timescaledb.list ``` 4. **Install the TimescaleDB GPG key** ```bash wget --quiet -O - https://packagecloud.io/timescale/timescaledb/gpgkey | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/timescaledb.gpg ``` For Ubuntu 21.10 and earlier use the following command: ```bash wget --quiet -O - https://packagecloud.io/timescale/timescaledb/gpgkey | sudo apt-key add - ``` 5. **Update your local repository list** ```bash sudo apt update ``` 6. **Install TimescaleDB** ```bash sudo apt install timescaledb-2-postgresql-18 postgresql-client-18 ``` To install a specific TimescaleDB [release](https://packagecloud.io/timescale/timescaledb), set the version. For example: `sudo apt-get install timescaledb-2-postgresql-14='2.6.0*' timescaledb-2-loader-postgresql-14='2.6.0*'` Older versions of TimescaleDB may not support all the OS versions listed on this page. 7. **Tune your PostgreSQL instance for TimescaleDB** ```bash sudo timescaledb-tune ``` By default, this script is included with the `timescaledb-tools` package when you install TimescaleDB. Use the prompts to tune your development or production environment. For more information on manual configuration, see [Configuration](/docs/reference/configuration). If you have an issue, run `sudo apt install timescaledb-tools`. 8. **Restart PostgreSQL** ```bash sudo systemctl restart postgresql ``` 9. **Log in to PostgreSQL as `postgres`** ```bash sudo -u postgres psql ``` You are in the psql shell. 10. **Set the password for `postgres`** ```bash \password postgres ``` When you have set the password, type `\q` to exit psql. * Red Hat 1. **Install the latest PostgreSQL packages** ```bash sudo yum install https://download.postgresql.org/pub/repos/yum/reporpms/EL-$(rpm -E %{rhel})-x86_64/pgdg-redhat-repo-latest.noarch.rpm ``` 2. **Add the TimescaleDB repository** ```bash sudo tee /etc/yum.repos.d/timescale_timescaledb.repo < Note On Red Hat Enterprise Linux 8 and later, disable the built-in PostgreSQL module: `sudo dnf -qy module disable postgresql` 5. **Initialize the PostgreSQL instance** ```bash sudo /usr/pgsql-18/bin/postgresql-18-setup initdb ``` 6. **Tune your PostgreSQL instance for TimescaleDB** ```bash sudo timescaledb-tune --pg-config=/usr/pgsql-18/bin/pg_config ``` This script is included with the `timescaledb-tools` package when you install TimescaleDB. For more information, see [configuration](/docs/reference/configuration). 7. **Enable and start PostgreSQL** ```bash sudo systemctl enable postgresql-18 sudo systemctl start postgresql-18 ``` 8. **Log in to PostgreSQL as `postgres`** ```bash sudo -u postgres psql ``` You are now in the psql shell. 9. **Set the password for `postgres`** ```bash \password postgres ``` When you have set the password, type `\q` to exit psql. * Fedora 1. **Install the latest PostgreSQL packages** ```bash sudo yum install https://download.postgresql.org/pub/repos/yum/reporpms/F-$(rpm -E %{fedora})-x86_64/pgdg-fedora-repo-latest.noarch.rpm ``` 2. **Add the TimescaleDB repository** ```bash sudo tee /etc/yum.repos.d/timescale_timescaledb.repo < Note On Red Hat Enterprise Linux 8 and later, disable the built-in PostgreSQL module: `sudo dnf -qy module disable postgresql` 5. **Initialize the PostgreSQL instance** ```bash sudo /usr/pgsql-18/bin/postgresql-18-setup initdb ``` 6. **Tune your PostgreSQL instance for TimescaleDB** ```bash sudo timescaledb-tune --pg-config=/usr/pgsql-18/bin/pg_config ``` This script is included with the `timescaledb-tools` package when you install TimescaleDB. For more information, see [configuration](/docs/reference/configuration). 7. **Enable and start PostgreSQL** ```bash sudo systemctl enable postgresql-18 sudo systemctl start postgresql-18 ``` 8. **Log in to PostgreSQL as `postgres`** ```bash sudo -u postgres psql ``` You are now in the psql shell. 9. **Set the password for `postgres`** ```bash \password postgres ``` When you have set the password, type `\q` to exit psql. * RockyLinux 1. **Update your local repository list** ```bash sudo dnf update -y sudo dnf install -y epel-release ``` 2. **Install the latest PostgreSQL packages** ```bash sudo dnf install -y https://download.postgresql.org/pub/repos/yum/reporpms/EL-9-x86_64/pgdg-redhat-repo-latest.noarch.rpm ``` 3. **Add the TimescaleDB repository** ```bash sudo tee /etc/yum.repos.d/timescale_timescaledb.repo <` to the name of that database: ```bash psql -d "postgres://:@:/" ``` 2. **Add TimescaleDB to the database** ```sql CREATE EXTENSION IF NOT EXISTS timescaledb; ``` 3. **Check that TimescaleDB is installed** ```sql \dx ``` You see the list of installed extensions: ```sql List of installed extensions Name | Version | Default version | Schema | Description -------------+---------+-----------------+------------+--------------------------------------------------------------------------------------- plpgsql | 1.0 | 1.0 | pg_catalog | PL/pgSQL procedural language timescaledb | 2.24.0 | 2.24.0 | public | Enables scalable inserts and complex queries for time-series data (Community Edition) (2 rows) ``` Press `\q` to exit the list of extensions. - macOS 1. **Install the latest PostgreSQL packages** ```bash sudo apt install gnupg postgresql-common apt-transport-https lsb-release wget ``` 2. **Run the PostgreSQL package setup script** ```bash sudo /usr/share/postgresql-common/pgdg/apt.postgresql.org.sh ``` 3. **Add the TimescaleDB package** ```bash echo "deb https://packagecloud.io/timescale/timescaledb/debian/ $(lsb_release -c -s) main" | sudo tee /etc/apt/sources.list.d/timescaledb.list ``` 4. **Install the TimescaleDB GPG key** ```bash wget --quiet -O - https://packagecloud.io/timescale/timescaledb/gpgkey | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/timescaledb.gpg ``` 5. **Update your local repository list** ```bash sudo apt update ``` 6. **Install TimescaleDB** ```bash sudo apt install timescaledb-2-postgresql-18 postgresql-client-18 ``` To install a specific TimescaleDB [release](https://packagecloud.io/timescale/timescaledb), set the version. For example: `sudo apt-get install timescaledb-2-postgresql-14='2.6.0*' timescaledb-2-loader-postgresql-14='2.6.0*'` Older versions of TimescaleDB may not support all the OS versions listed on this page. 7. **Tune your PostgreSQL instance for TimescaleDB** ```bash sudo timescaledb-tune ``` By default, this script is included with the `timescaledb-tools` package when you install TimescaleDB. Use the prompts to tune your development or production environment. For more information on manual configuration, see [Configuration](/docs/reference/configuration). If you have an issue, run `sudo apt install timescaledb-tools`. 8. **Restart PostgreSQL** ```bash sudo systemctl restart postgresql ``` 9. **Log in to PostgreSQL as `postgres`** ```bash sudo -u postgres psql ``` You are in the psql shell. 10. **Set the password for `postgres`** ```bash \password postgres ``` When you have set the password, type `\q` to exit psql. - Windows 1. **Install the latest PostgreSQL packages** ```bash sudo apt install gnupg postgresql-common apt-transport-https lsb-release wget ``` 2. **Run the PostgreSQL package setup script** ```bash sudo /usr/share/postgresql-common/pgdg/apt.postgresql.org.sh ``` 3. **Add the TimescaleDB package** ```bash echo "deb https://packagecloud.io/timescale/timescaledb/ubuntu/ $(lsb_release -c -s) main" | sudo tee /etc/apt/sources.list.d/timescaledb.list ``` 4. **Install the TimescaleDB GPG key** ```bash wget --quiet -O - https://packagecloud.io/timescale/timescaledb/gpgkey | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/timescaledb.gpg ``` For Ubuntu 21.10 and earlier use the following command: ```bash wget --quiet -O - https://packagecloud.io/timescale/timescaledb/gpgkey | sudo apt-key add - ``` 5. **Update your local repository list** ```bash sudo apt update ``` 6. **Install TimescaleDB** ```bash sudo apt install timescaledb-2-postgresql-18 postgresql-client-18 ``` To install a specific TimescaleDB [release](https://packagecloud.io/timescale/timescaledb), set the version. For example: `sudo apt-get install timescaledb-2-postgresql-14='2.6.0*' timescaledb-2-loader-postgresql-14='2.6.0*'` Older versions of TimescaleDB may not support all the OS versions listed on this page. 7. **Tune your PostgreSQL instance for TimescaleDB** ```bash sudo timescaledb-tune ``` By default, this script is included with the `timescaledb-tools` package when you install TimescaleDB. Use the prompts to tune your development or production environment. For more information on manual configuration, see [Configuration](/docs/reference/configuration). If you have an issue, run `sudo apt install timescaledb-tools`. 8. **Restart PostgreSQL** ```bash sudo systemctl restart postgresql ``` 9. **Log in to PostgreSQL as `postgres`** ```bash sudo -u postgres psql ``` You are in the psql shell. 10. **Set the password for `postgres`** ```bash \password postgres ``` When you have set the password, type `\q` to exit psql. - Docker 1. **Install the latest PostgreSQL packages** ```bash sudo yum install https://download.postgresql.org/pub/repos/yum/reporpms/EL-$(rpm -E %{rhel})-x86_64/pgdg-redhat-repo-latest.noarch.rpm ``` 2. **Add the TimescaleDB repository** ```bash sudo tee /etc/yum.repos.d/timescale_timescaledb.repo < Note On Red Hat Enterprise Linux 8 and later, disable the built-in PostgreSQL module: `sudo dnf -qy module disable postgresql` 5. **Initialize the PostgreSQL instance** ```bash sudo /usr/pgsql-18/bin/postgresql-18-setup initdb ``` 6. **Tune your PostgreSQL instance for TimescaleDB** ```bash sudo timescaledb-tune --pg-config=/usr/pgsql-18/bin/pg_config ``` This script is included with the `timescaledb-tools` package when you install TimescaleDB. For more information, see [configuration](/docs/reference/configuration). 7. **Enable and start PostgreSQL** ```bash sudo systemctl enable postgresql-18 sudo systemctl start postgresql-18 ``` 8. **Log in to PostgreSQL as `postgres`** ```bash sudo -u postgres psql ``` You are now in the psql shell. 9. **Set the password for `postgres`** ```bash \password postgres ``` When you have set the password, type `\q` to exit psql. - Kubernetes 1. **Install the latest PostgreSQL packages** ```bash sudo yum install https://download.postgresql.org/pub/repos/yum/reporpms/F-$(rpm -E %{fedora})-x86_64/pgdg-fedora-repo-latest.noarch.rpm ``` 2. **Add the TimescaleDB repository** ```bash sudo tee /etc/yum.repos.d/timescale_timescaledb.repo < Note On Red Hat Enterprise Linux 8 and later, disable the built-in PostgreSQL module: `sudo dnf -qy module disable postgresql` 5. **Initialize the PostgreSQL instance** ```bash sudo /usr/pgsql-18/bin/postgresql-18-setup initdb ``` 6. **Tune your PostgreSQL instance for TimescaleDB** ```bash sudo timescaledb-tune --pg-config=/usr/pgsql-18/bin/pg_config ``` This script is included with the `timescaledb-tools` package when you install TimescaleDB. For more information, see [configuration](/docs/reference/configuration). 7. **Enable and start PostgreSQL** ```bash sudo systemctl enable postgresql-18 sudo systemctl start postgresql-18 ``` 8. **Log in to PostgreSQL as `postgres`** ```bash sudo -u postgres psql ``` You are now in the psql shell. 9. **Set the password for `postgres`** ```bash \password postgres ``` When you have set the password, type `\q` to exit psql. - Source 1. **Update your local repository list** ```bash sudo dnf update -y sudo dnf install -y epel-release ``` 2. **Install the latest PostgreSQL packages** ```bash sudo dnf install -y https://download.postgresql.org/pub/repos/yum/reporpms/EL-9-x86_64/pgdg-redhat-repo-latest.noarch.rpm ``` 3. **Add the TimescaleDB repository** ```bash sudo tee /etc/yum.repos.d/timescale_timescaledb.repo < Warning If you have already installed PostgreSQL using a method other than Homebrew or MacPorts, you may encounter errors following these install instructions. Best practice is to fully remove any existing PostgreSQL installations before you begin. To keep your current PostgreSQL installation, install from source. * Homebrew 1. **Install Homebrew** If homebrew is not already installed on your machine, run the following command: ```bash /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" ``` For more information about Homebrew, including installation instructions, see the [Homebrew documentation](https://docs.brew.sh/Installation). 2. **Install and start PostgreSQL** ```bash brew install postgresql@17 brew cleanup --prune-prefix brew services start postgresql@17 ``` 3. **Add PostgreSQL to your path** ```bash echo 'export PATH="/opt/homebrew/opt/postgresql@17/bin:$PATH"' >> ~/.zshrc source ~/.zshrc ``` 4. **Add the TimescaleDB Homebrew tap** ```bash brew tap timescale/tap ``` 5. **Install TimescaleDB** ```bash brew install timescaledb ``` 6. **Configure your database** ```bash timescaledb-tune --quiet --yes ``` 7. **Run the setup script to complete installation** ```bash $(brew --prefix)/Cellar/timescaledb/$(brew list --versions timescaledb | awk '{print $2}')/bin/timescaledb_move.sh ``` This command automatically locates the correct version and runs the setup script. 8. **Start the PostgreSQL service** ```bash brew services restart postgresql@17 ``` 9. **Verify PostgreSQL is running** ```bash brew services list | grep postgresql ``` You should see `postgresql@17` with status `started`. 10. **Connect to PostgreSQL** ```bash psql postgres ``` You are now in the psql shell. 11. **Set the password for your database user** (optional but recommended) ```bash \password ``` When you have set the password, type `\q` to exit psql. * MacPorts 1. **Install MacPorts by downloading and running the package installer.** For more information about MacPorts, including installation instructions, see the [MacPorts documentation](https://guide.macports.org/#installing.macports). 2. **Install TimescaleDB and psql:** ```bash sudo port install timescaledb libpqxx ``` To view the files installed, run: ```bash port contents timescaledb libpqxx ``` Tips MacPorts does not install the `timescaledb-tools` package or run the `timescaledb-tune` script. For more information about tuning your database, see the [TimescaleDB tuning tool](/docs/reference/configuration). 3. **Log in to PostgreSQL as `postgres`** ```bash sudo -u postgres psql ``` You are in the psql shell. 4. **Set the password for `postgres`** ```bash \password postgres ``` When you have set the password, type `\q` to exit psql. ## Add the TimescaleDB extension to your database [Section titled “Add the TimescaleDB extension to your database”](#add-the-timescaledb-extension-to-your-database-1) For improved performance, you enable TimescaleDB on each database on your self-hosted PostgreSQL instance. This section shows you how to enable TimescaleDB for a new database in PostgreSQL using `psql` from the command line. 1. **Connect to a database on your PostgreSQL instance** In PostgreSQL, the default user and database are both `postgres`. To use a different database, set `` to the name of that database: ```bash psql -d "postgres://:@:/" ``` 2. **Add TimescaleDB to the database** ```sql CREATE EXTENSION IF NOT EXISTS timescaledb; ``` 3. **Check that TimescaleDB is installed** ```sql \dx ``` You see the list of installed extensions: ```sql List of installed extensions Name | Version | Default version | Schema | Description -------------+---------+-----------------+------------+--------------------------------------------------------------------------------------- plpgsql | 1.0 | 1.0 | pg_catalog | PL/pgSQL procedural language timescaledb | 2.24.0 | 2.24.0 | public | Enables scalable inserts and complex queries for time-series data (Community Edition) (2 rows) ``` Press `\q` to exit the list of extensions. - Red Hat 1. **Install Homebrew** If homebrew is not already installed on your machine, run the following command: ```bash /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" ``` For more information about Homebrew, including installation instructions, see the [Homebrew documentation](https://docs.brew.sh/Installation). 2. **Install and start PostgreSQL** ```bash brew install postgresql@17 brew cleanup --prune-prefix brew services start postgresql@17 ``` 3. **Add PostgreSQL to your path** ```bash echo 'export PATH="/opt/homebrew/opt/postgresql@17/bin:$PATH"' >> ~/.zshrc source ~/.zshrc ``` 4. **Add the TimescaleDB Homebrew tap** ```bash brew tap timescale/tap ``` 5. **Install TimescaleDB** ```bash brew install timescaledb ``` 6. **Configure your database** ```bash timescaledb-tune --quiet --yes ``` 7. **Run the setup script to complete installation** ```bash $(brew --prefix)/Cellar/timescaledb/$(brew list --versions timescaledb | awk '{print $2}')/bin/timescaledb_move.sh ``` This command automatically locates the correct version and runs the setup script. 8. **Start the PostgreSQL service** ```bash brew services restart postgresql@17 ``` 9. **Verify PostgreSQL is running** ```bash brew services list | grep postgresql ``` You should see `postgresql@17` with status `started`. 10. **Connect to PostgreSQL** ```bash psql postgres ``` You are now in the psql shell. 11. **Set the password for your database user** (optional but recommended) ```bash \password ``` When you have set the password, type `\q` to exit psql. - Fedora 1. **Install MacPorts by downloading and running the package installer.** For more information about MacPorts, including installation instructions, see the [MacPorts documentation](https://guide.macports.org/#installing.macports). 2. **Install TimescaleDB and psql:** ```bash sudo port install timescaledb libpqxx ``` To view the files installed, run: ```bash port contents timescaledb libpqxx ``` Tips MacPorts does not install the `timescaledb-tools` package or run the `timescaledb-tune` script. For more information about tuning your database, see the [TimescaleDB tuning tool](/docs/reference/configuration). 3. **Log in to PostgreSQL as `postgres`** ```bash sudo -u postgres psql ``` You are in the psql shell. 4. **Set the password for `postgres`** ```bash \password postgres ``` When you have set the password, type `\q` to exit psql. - RockyLinux To install TimescaleDB on Windows, you need: * **OpenSSL v3.x**. For TimescaleDB v2.14.1 only, install OpenSSL v1.1.1. * **[Visual C++ Redistributable for Visual Studio 2015](https://www.microsoft.com/en-us/download/details.aspx?id=48145)** Warning If you have previously installed PostgreSQL, you may encounter errors following these install instructions. Best practice is to fully remove any existing PostgreSQL installations before you begin. To keep your current PostgreSQL installation, install from source. 1. **Install the latest version of PostgreSQL and psql** 1. Download [PostgreSQL](https://www.postgresql.org/download/windows/), then run the installer. 1. In the `Select Components` dialog, check `Command Line Tools`, along with any other components you want to install, and click `Next`. 2. Complete the installation wizard. 2. Check that you can run `pg_config`. If you cannot run `pg_config` from the command line, in the Windows Search tool, enter `system environment variables`. The path should be `C:\Program Files\PostgreSQL\\bin`. 2. **Install TimescaleDB** 1. Unzip the [installer](/docs/get-started/choose-your-path/install-timescaledb#windows-downloads) to ``, that is, your selected directory. Best practice is to use the latest version. 2. In `\timescaledb`, right-click `setup.exe`, then choose `Run as Administrator`. 3. Complete the installation wizard. During installation, the wizard will prompt you to run the `timescaledb-tune` script to optimize your PostgreSQL configuration. It is recommended to accept this option. For more information, see [configuration](/docs/reference/configuration). If you see an error like `could not load library "C:/Program Files/PostgreSQL/17/lib/timescaledb-2.17.2.dll": The specified module could not be found.`, use [Dependencies](https://github.com/lucasg/Dependencies) to ensure that your system can find the compatible DLLs for this release of TimescaleDB. 3. **Restart the PostgreSQL service** ```powershell Restart-Service postgresql-x64-18 ``` Replace `18` with your PostgreSQL version if different. ## Add the TimescaleDB extension to your database [Section titled “Add the TimescaleDB extension to your database”](#add-the-timescaledb-extension-to-your-database-2) For improved performance, you enable TimescaleDB on each database on your self-hosted PostgreSQL instance. This section shows you how to enable TimescaleDB for a new database in PostgreSQL using `psql` from the command line. 1. **Connect to a database on your PostgreSQL instance** In PostgreSQL, the default user and database are both `postgres`. To use a different database, set `` to the name of that database: ```bash psql -d "postgres://:@:/" ``` 2. **Add TimescaleDB to the database** ```sql CREATE EXTENSION IF NOT EXISTS timescaledb; ``` 3. **Check that TimescaleDB is installed** ```sql \dx ``` You see the list of installed extensions: ```sql List of installed extensions Name | Version | Default version | Schema | Description -------------+---------+-----------------+------------+--------------------------------------------------------------------------------------- plpgsql | 1.0 | 1.0 | pg_catalog | PL/pgSQL procedural language timescaledb | 2.24.0 | 2.24.0 | public | Enables scalable inserts and complex queries for time-series data (Community Edition) (2 rows) ``` Press `\q` to exit the list of extensions. ## Windows downloads [Section titled “Windows downloads”](#windows-downloads) The latest TimescaleDB releases for PostgreSQL: * [PostgreSQL 18](https://github.com/timescale/timescaledb/releases/download/2.28.1/timescaledb-postgresql-18-windows-amd64.zip) * [PostgreSQL 17](https://github.com/timescale/timescaledb/releases/download/2.28.1/timescaledb-postgresql-17-windows-amd64.zip) * [PostgreSQL 16](https://github.com/timescale/timescaledb/releases/download/2.28.1/timescaledb-postgresql-16-windows-amd64.zip) * [PostgreSQL 15](https://github.com/timescale/timescaledb/releases/download/2.28.1/timescaledb-postgresql-15-windows-amd64.zip) For release information, see the [GitHub releases page](https://github.com/timescale/timescaledb/releases). - ArchLinux You can install a TimescaleDB instance on any local system from a pre-built Docker container. To install TimescaleDB from Docker, you need: * **[Docker](https://docs.docker.com/get-started/get-docker/)** * **[psql](/docs/integrate/query-administration/psql)** - TimescaleDB-HA 1. **Run the TimescaleDB Docker image** The [TimescaleDB HA](https://hub.docker.com/r/timescale/timescaledb-ha) Docker image offers the most complete TimescaleDB experience. It uses [Ubuntu](https://ubuntu.com), includes [TimescaleDB Toolkit](https://github.com/timescale/timescaledb-toolkit), and support for PostGIS and Patroni. To install the latest release based on PostgreSQL 18: ```plaintext docker pull timescale/timescaledb-ha:pg18 ``` TimescaleDB is pre-created in the default PostgreSQL database and is added by default to any new database you create in this image. 2. **Run the container** Replace `` with the path to the folder you want to keep your data in the following command. ```plaintext docker run -d --name timescaledb -p 5432:5432 -v :/pgdata -e PGDATA=/pgdata -e POSTGRES_PASSWORD=password timescale/timescaledb-ha:pg18 ``` If you are running multiple container instances, change the port each Docker instance runs on. Note If you have a local PostgreSQL instance already running on port 5432, change the Docker port mapping to avoid conflicts. For example, use `-p 5433:5432` and adjust the connection string accordingly. On UNIX-based systems, Docker modifies Linux IP tables to bind the container. If your system uses Linux Uncomplicated Firewall (UFW), Docker may [override your UFW port binding settings](https://www.techrepublic.com/article/how-to-fix-the-docker-and-ufw-security-flaw/). To prevent this, add `DOCKER_OPTS="--iptables=false"` to `/etc/default/docker`. 3. **Connect to a database on your PostgreSQL instance** The default user and database are both `postgres`. You set the password in `POSTGRES_PASSWORD` in the previous step. The default command to connect to PostgreSQL is: ```bash psql -d "postgres://postgres:password@127.0.0.1:5432/postgres" ``` If you changed the port mapping in step 2 (e.g., to 5433), update the port in the connection string accordingly. 4. **Check that TimescaleDB is installed** ```sql \dx ``` You see the list of installed extensions: ```sql Name | Version | Default version | Schema | Description ---------------------+---------+-----------------+------------+--------------------------------------------------------------------------------------- plpgsql | 1.0 | 1.0 | pg_catalog | PL/pgSQL procedural language timescaledb | 2.24.0 | 2.24.0 | public | Enables scalable inserts and complex queries for time-series data (Community Edition) timescaledb_toolkit | 1.22.0 | 1.22.0 | public | Library of analytical hyperfunctions, time-series pipelining, and other SQL utilities (3 rows) ``` Press `q` to exit the list of extensions. ## More Docker options If you want to access the container from the host but avoid exposing it to the outside world, you can bind to `127.0.0.1` instead of the public interface, using this command: ```bash docker run -d --name timescaledb -p 127.0.0.1:5432:5432 \ -v :/pgdata -e PGDATA=/pgdata -e POSTGRES_PASSWORD=password timescale/timescaledb-ha:pg18 ``` If you don't want to install `psql` and other PostgreSQL client tools locally, or if you are using a Microsoft Windows host system, you can connect using the version of `psql` that is bundled within the container with this command: ```bash docker exec -it timescaledb psql -U postgres ``` When you install TimescaleDB using a Docker container, the PostgreSQL settings are inherited from the container. In most cases, you do not need to adjust them. However, if you need to change a setting, you can add `-c setting=value` to your Docker `run` command. For more information, see the [Docker documentation](https://hub.docker.com/_/postgres). The link provided in these instructions is for the latest version of TimescaleDB on PostgreSQL 18. To find other Docker tags you can use, see the [Dockerhub repository](https://hub.docker.com/r/timescale/timescaledb/tags?page=1\&ordering=last_updated). ## View logs in Docker If you have TimescaleDB installed in a Docker container, you can view your logs using Docker, instead of looking in `/var/lib/logs` or `/var/logs`. For more information, see the [Docker documentation on logs](https://docs.docker.com/engine/logging/). - TimescaleDB light 1. **Run the TimescaleDB Docker image** The light-weight [TimescaleDB](https://hub.docker.com/r/timescale/timescaledb) Docker image uses [Alpine](https://alpinelinux.org/) and does not contain [TimescaleDB Toolkit](https://github.com/timescale/timescaledb-toolkit) or support for PostGIS and Patroni. To install the latest release based on PostgreSQL 18: ```plaintext docker pull timescale/timescaledb:latest-pg18 ``` TimescaleDB is pre-created in the default PostgreSQL database and added by default to any new database you create in this image. 2. **Run the container** ```plaintext docker run -v :/pgdata -e PGDATA=/pgdata \ -d --name timescaledb -p 5432:5432 -e POSTGRES_PASSWORD=password timescale/timescaledb:latest-pg18 ``` If you are running multiple container instances, change the port each Docker instance runs on. Note If you have a local PostgreSQL instance already running on port 5432, change the Docker port mapping to avoid conflicts. For example, use `-p 5433:5432` and adjust the connection string accordingly. On UNIX-based systems, Docker modifies Linux IP tables to bind the container. If your system uses Linux Uncomplicated Firewall (UFW), Docker may [override your UFW port binding settings](https://www.techrepublic.com/article/how-to-fix-the-docker-and-ufw-security-flaw/). To prevent this, add `DOCKER_OPTS="--iptables=false"` to `/etc/default/docker`. 3. **Connect to a database on your PostgreSQL instance** The default user and database are both `postgres`. You set the password in `POSTGRES_PASSWORD` in the previous step. The default command to connect to PostgreSQL in this image is: ```bash psql -d "postgres://postgres:password@127.0.0.1:5432/postgres" ``` If you changed the port mapping in step 2 (e.g., to 5433), update the port in the connection string accordingly. 4. **Check that TimescaleDB is installed** ```sql \dx ``` You see the list of installed extensions: ```sql List of installed extensionsh Name | Version | Default version | Schema | Description -------------+---------+-----------------+------------+--------------------------------------------------------------------------------------- plpgsql | 1.0 | 1.0 | pg_catalog | PL/pgSQL procedural language timescaledb | 2.24.0 | 2.24.0 | public | Enables scalable inserts and complex queries for time-series data (Community Edition) ``` Press `q` to exit the list of extensions. ## More Docker options If you want to access the container from the host but avoid exposing it to the outside world, you can bind to `127.0.0.1` instead of the public interface, using this command: ```bash docker run -v :/pgdata -e PGDATA=/pgdata \ -d --name timescaledb -p 127.0.0.1:5432:5432 \ -e POSTGRES_PASSWORD=password timescale/timescaledb:latest-pg18 ``` If you don't want to install `psql` and other PostgreSQL client tools locally, or if you are using a Microsoft Windows host system, you can connect using the version of `psql` that is bundled within the container with this command: ```bash docker exec -it timescaledb psql -U postgres ``` Existing containers can be stopped using `docker stop` and started again with `docker start` while retaining their volumes and data. When you create a new container using the `docker run` command, by default you also create a new data volume. When you remove a Docker container with `docker rm`, the data volume persists on disk until you explicitly delete it. You can use the `docker volume ls` command to list existing docker volumes. If you want to store the data from your Docker container in a host directory, or you want to run the Docker image on top of an existing data directory, you can specify the directory to mount a data volume using the `-v` flag: ```bash docker run -d --name timescaledb -p 5432:5432 \ -v :/pgdata -e PGDATA=/pgdata \ -e POSTGRES_PASSWORD=password timescale/timescaledb:latest-pg18 ``` When you install TimescaleDB using a Docker container, the PostgreSQL settings are inherited from the container. In most cases, you do not need to adjust them. However, if you need to change a setting, you can add `-c setting=value` to your Docker `run` command. For more information, see the [Docker documentation](https://hub.docker.com/_/postgres). The link provided in these instructions is for the latest version of TimescaleDB on PostgreSQL 18. To find other Docker tags you can use, see the [Dockerhub repository](https://hub.docker.com/r/timescale/timescaledb/tags?page=1\&ordering=last_updated). ## View logs in Docker If you have TimescaleDB installed in a Docker container, you can view your logs using Docker, instead of looking in `/var/log`. For more information, see the [Docker documentation on logs](https://docs.docker.com/engine/logging/). - Chainguard Commercial Builds Chainguard Commercial Builds provide secure, production-ready TimescaleDB container images, packaged inside the Chainguard Factory, a SLSA Level 3-compliant system that delivers. Adopting the commercial Chainguard image provides several critical benefits for your production environments: * Zero known vulnerabilities: Images are continuously rebuilt and updated to ensure there are no known CVEs (Common Vulnerabilities and Exposures) at the time of deployment. * Minimal attack surface: The images are built distroless, meaning they include only the exact components required to run TimescaleDB. Unnecessary operating system packages, package managers, and shells are strictly omitted. * Supply chain security: Fully compliant with stringent enterprise requirements, featuring verifiable cryptographic signatures and out-of-the-box Software Bill of Materials (SBOMs). * Drop-in replacement: You get an enterprise-grade security posture while maintaining complete compatibility with standard TimescaleDB and PostgreSQL configurations. Because accessing commercial Chainguard images requires proper authentication with the Chainguard registry, deployment steps differ slightly from our standard community images. For comprehensive instructions on how to authenticate, pull, and run the secure images in your environment, please refer to the official Chainguard documentation: [Commercial TimescaleDB Image Overview](https://images.chainguard.dev/directory/image/commercial-timescaledb/overview) The overview page provides all the necessary details regarding available tags, registry endpoints, and container configuration to get your hardened TimescaleDB instance up and running. - Homebrew 1. **Run the TimescaleDB Docker image** The [TimescaleDB HA](https://hub.docker.com/r/timescale/timescaledb-ha) Docker image offers the most complete TimescaleDB experience. It uses [Ubuntu](https://ubuntu.com), includes [TimescaleDB Toolkit](https://github.com/timescale/timescaledb-toolkit), and support for PostGIS and Patroni. To install the latest release based on PostgreSQL 18: ```plaintext docker pull timescale/timescaledb-ha:pg18 ``` TimescaleDB is pre-created in the default PostgreSQL database and is added by default to any new database you create in this image. 2. **Run the container** Replace `` with the path to the folder you want to keep your data in the following command. ```plaintext docker run -d --name timescaledb -p 5432:5432 -v :/pgdata -e PGDATA=/pgdata -e POSTGRES_PASSWORD=password timescale/timescaledb-ha:pg18 ``` If you are running multiple container instances, change the port each Docker instance runs on. Note If you have a local PostgreSQL instance already running on port 5432, change the Docker port mapping to avoid conflicts. For example, use `-p 5433:5432` and adjust the connection string accordingly. On UNIX-based systems, Docker modifies Linux IP tables to bind the container. If your system uses Linux Uncomplicated Firewall (UFW), Docker may [override your UFW port binding settings](https://www.techrepublic.com/article/how-to-fix-the-docker-and-ufw-security-flaw/). To prevent this, add `DOCKER_OPTS="--iptables=false"` to `/etc/default/docker`. 3. **Connect to a database on your PostgreSQL instance** The default user and database are both `postgres`. You set the password in `POSTGRES_PASSWORD` in the previous step. The default command to connect to PostgreSQL is: ```bash psql -d "postgres://postgres:password@127.0.0.1:5432/postgres" ``` If you changed the port mapping in step 2 (e.g., to 5433), update the port in the connection string accordingly. 4. **Check that TimescaleDB is installed** ```sql \dx ``` You see the list of installed extensions: ```sql Name | Version | Default version | Schema | Description ---------------------+---------+-----------------+------------+--------------------------------------------------------------------------------------- plpgsql | 1.0 | 1.0 | pg_catalog | PL/pgSQL procedural language timescaledb | 2.24.0 | 2.24.0 | public | Enables scalable inserts and complex queries for time-series data (Community Edition) timescaledb_toolkit | 1.22.0 | 1.22.0 | public | Library of analytical hyperfunctions, time-series pipelining, and other SQL utilities (3 rows) ``` Press `q` to exit the list of extensions. ## More Docker options If you want to access the container from the host but avoid exposing it to the outside world, you can bind to `127.0.0.1` instead of the public interface, using this command: ```bash docker run -d --name timescaledb -p 127.0.0.1:5432:5432 \ -v :/pgdata -e PGDATA=/pgdata -e POSTGRES_PASSWORD=password timescale/timescaledb-ha:pg18 ``` If you don't want to install `psql` and other PostgreSQL client tools locally, or if you are using a Microsoft Windows host system, you can connect using the version of `psql` that is bundled within the container with this command: ```bash docker exec -it timescaledb psql -U postgres ``` When you install TimescaleDB using a Docker container, the PostgreSQL settings are inherited from the container. In most cases, you do not need to adjust them. However, if you need to change a setting, you can add `-c setting=value` to your Docker `run` command. For more information, see the [Docker documentation](https://hub.docker.com/_/postgres). The link provided in these instructions is for the latest version of TimescaleDB on PostgreSQL 18. To find other Docker tags you can use, see the [Dockerhub repository](https://hub.docker.com/r/timescale/timescaledb/tags?page=1\&ordering=last_updated). ## View logs in Docker If you have TimescaleDB installed in a Docker container, you can view your logs using Docker, instead of looking in `/var/lib/logs` or `/var/logs`. For more information, see the [Docker documentation on logs](https://docs.docker.com/engine/logging/). - MacPorts 1. **Run the TimescaleDB Docker image** The light-weight [TimescaleDB](https://hub.docker.com/r/timescale/timescaledb) Docker image uses [Alpine](https://alpinelinux.org/) and does not contain [TimescaleDB Toolkit](https://github.com/timescale/timescaledb-toolkit) or support for PostGIS and Patroni. To install the latest release based on PostgreSQL 18: ```plaintext docker pull timescale/timescaledb:latest-pg18 ``` TimescaleDB is pre-created in the default PostgreSQL database and added by default to any new database you create in this image. 2. **Run the container** ```plaintext docker run -v :/pgdata -e PGDATA=/pgdata \ -d --name timescaledb -p 5432:5432 -e POSTGRES_PASSWORD=password timescale/timescaledb:latest-pg18 ``` If you are running multiple container instances, change the port each Docker instance runs on. Note If you have a local PostgreSQL instance already running on port 5432, change the Docker port mapping to avoid conflicts. For example, use `-p 5433:5432` and adjust the connection string accordingly. On UNIX-based systems, Docker modifies Linux IP tables to bind the container. If your system uses Linux Uncomplicated Firewall (UFW), Docker may [override your UFW port binding settings](https://www.techrepublic.com/article/how-to-fix-the-docker-and-ufw-security-flaw/). To prevent this, add `DOCKER_OPTS="--iptables=false"` to `/etc/default/docker`. 3. **Connect to a database on your PostgreSQL instance** The default user and database are both `postgres`. You set the password in `POSTGRES_PASSWORD` in the previous step. The default command to connect to PostgreSQL in this image is: ```bash psql -d "postgres://postgres:password@127.0.0.1:5432/postgres" ``` If you changed the port mapping in step 2 (e.g., to 5433), update the port in the connection string accordingly. 4. **Check that TimescaleDB is installed** ```sql \dx ``` You see the list of installed extensions: ```sql List of installed extensionsh Name | Version | Default version | Schema | Description -------------+---------+-----------------+------------+--------------------------------------------------------------------------------------- plpgsql | 1.0 | 1.0 | pg_catalog | PL/pgSQL procedural language timescaledb | 2.24.0 | 2.24.0 | public | Enables scalable inserts and complex queries for time-series data (Community Edition) ``` Press `q` to exit the list of extensions. ## More Docker options If you want to access the container from the host but avoid exposing it to the outside world, you can bind to `127.0.0.1` instead of the public interface, using this command: ```bash docker run -v :/pgdata -e PGDATA=/pgdata \ -d --name timescaledb -p 127.0.0.1:5432:5432 \ -e POSTGRES_PASSWORD=password timescale/timescaledb:latest-pg18 ``` If you don't want to install `psql` and other PostgreSQL client tools locally, or if you are using a Microsoft Windows host system, you can connect using the version of `psql` that is bundled within the container with this command: ```bash docker exec -it timescaledb psql -U postgres ``` Existing containers can be stopped using `docker stop` and started again with `docker start` while retaining their volumes and data. When you create a new container using the `docker run` command, by default you also create a new data volume. When you remove a Docker container with `docker rm`, the data volume persists on disk until you explicitly delete it. You can use the `docker volume ls` command to list existing docker volumes. If you want to store the data from your Docker container in a host directory, or you want to run the Docker image on top of an existing data directory, you can specify the directory to mount a data volume using the `-v` flag: ```bash docker run -d --name timescaledb -p 5432:5432 \ -v :/pgdata -e PGDATA=/pgdata \ -e POSTGRES_PASSWORD=password timescale/timescaledb:latest-pg18 ``` When you install TimescaleDB using a Docker container, the PostgreSQL settings are inherited from the container. In most cases, you do not need to adjust them. However, if you need to change a setting, you can add `-c setting=value` to your Docker `run` command. For more information, see the [Docker documentation](https://hub.docker.com/_/postgres). The link provided in these instructions is for the latest version of TimescaleDB on PostgreSQL 18. To find other Docker tags you can use, see the [Dockerhub repository](https://hub.docker.com/r/timescale/timescaledb/tags?page=1\&ordering=last_updated). ## View logs in Docker If you have TimescaleDB installed in a Docker container, you can view your logs using Docker, instead of looking in `/var/log`. For more information, see the [Docker documentation on logs](https://docs.docker.com/engine/logging/). - TimescaleDB-HA Chainguard Commercial Builds provide secure, production-ready TimescaleDB container images, packaged inside the Chainguard Factory, a SLSA Level 3-compliant system that delivers. Adopting the commercial Chainguard image provides several critical benefits for your production environments: * Zero known vulnerabilities: Images are continuously rebuilt and updated to ensure there are no known CVEs (Common Vulnerabilities and Exposures) at the time of deployment. * Minimal attack surface: The images are built distroless, meaning they include only the exact components required to run TimescaleDB. Unnecessary operating system packages, package managers, and shells are strictly omitted. * Supply chain security: Fully compliant with stringent enterprise requirements, featuring verifiable cryptographic signatures and out-of-the-box Software Bill of Materials (SBOMs). * Drop-in replacement: You get an enterprise-grade security posture while maintaining complete compatibility with standard TimescaleDB and PostgreSQL configurations. Because accessing commercial Chainguard images requires proper authentication with the Chainguard registry, deployment steps differ slightly from our standard community images. For comprehensive instructions on how to authenticate, pull, and run the secure images in your environment, please refer to the official Chainguard documentation: [Commercial TimescaleDB Image Overview](https://images.chainguard.dev/directory/image/commercial-timescaledb/overview) The overview page provides all the necessary details regarding available tags, registry endpoints, and container configuration to get your hardened TimescaleDB instance up and running. - TimescaleDB light You can run TimescaleDB inside Kubernetes using the TimescaleDB Docker container images. To run TimescaleDB inside Kubernetes, you need: * **[Self-managed Kubernetes](https://kubernetes.io/docs/setup/) or a Kubernetes [Turnkey Cloud Solution](https://kubernetes.io/docs/setup/production-environment/turnkey-solutions/)**. * **[kubectl](https://kubernetes.io/docs/tasks/tools/)** for command-line interaction with your cluster. ## Integrate TimescaleDB in a Kubernetes cluster [Section titled “Integrate TimescaleDB in a Kubernetes cluster”](#integrate-timescaledb-in-a-kubernetes-cluster) Running TimescaleDB on Kubernetes is similar to running PostgreSQL. This procedure outlines the steps for a non-distributed system. To connect your Kubernetes cluster to self-hosted TimescaleDB running in the cluster: 1. **Create a default namespace for Tiger Data components** 1. Create the Tiger Data namespace: ```shell kubectl create namespace tigerdata ``` 2. Set this namespace as the default for your session: ```shell kubectl config set-context --current --namespace=tigerdata ``` For more information, see [Kubernetes Namespaces](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/). 2. **Set up a persistent volume claim (PVC) storage** To manually set up a persistent volume and claim for self-hosted Kubernetes, run the following command: ```yaml kubectl apply -f - < ``` This command produces an error that you are now in `detached head` state. It is expected behavior, and it occurs because you have checked out a tag, and not a branch. Continue with the steps in this procedure as normal. 2. **Build the source** 1. Bootstrap the build system: * Linux ```bash ./bootstrap ``` * Windows ```powershell bootstrap.bat ``` For installation on Microsoft Windows, you might need to add the `pg_config` and `cmake` file locations to your path. In the Windows Search tool, search for `system environment variables`. The path for `pg_config` should be `C:\Program Files\PostgreSQL\\bin`. The path for `cmake` is within the Visual Studio directory. 2. Build the extension: * Linux ```bash cd build && make ``` * Windows ```powershell cmake --build ./build --config Release ``` 3. **Install TimescaleDB** * Linux ```bash make install ``` * Windows ```powershell cmake --build ./build --config Release --target install ``` 4. **Configure PostgreSQL** If you have more than one version of PostgreSQL installed, TimescaleDB can only be associated with one of them. The TimescaleDB build scripts use `pg_config` to find out where PostgreSQL stores its extension files, so you can use `pg_config` to find out which PostgreSQL installation TimescaleDB is using. 1. Locate the `postgresql.conf` configuration file: ```bash psql -d postgres -c "SHOW config_file;" ``` 2. Open the `postgresql.conf` file and update `shared_preload_libraries` to: ```bash shared_preload_libraries = 'timescaledb' ``` If you use other preloaded libraries, make sure they are comma separated. 3. Tune your PostgreSQL instance for TimescaleDB ```bash sudo timescaledb-tune ``` This script is included with the `timescaledb-tools` package when you install TimescaleDB. For more information, see [configuration](/docs/reference/configuration). 4. Restart the PostgreSQL instance: * Linux ```bash service postgresql restart ``` * Windows ```powershell pg_ctl restart ``` 5. **Set the user password** 1. Log in to PostgreSQL as `postgres` ```bash sudo -u postgres psql ``` You are in the psql shell. 2. Set the password for `postgres` ```bash \password postgres ``` When you have set the password, type `\q` to exit psql. ## Add the TimescaleDB extension to your database [Section titled “Add the TimescaleDB extension to your database”](#add-the-timescaledb-extension-to-your-database-3) For improved performance, you enable TimescaleDB on each database on your self-hosted PostgreSQL instance. This section shows you how to enable TimescaleDB for a new database in PostgreSQL using `psql` from the command line. 1. **Connect to a database on your PostgreSQL instance** In PostgreSQL, the default user and database are both `postgres`. To use a different database, set `` to the name of that database: ```bash psql -d "postgres://:@:/" ``` 2. **Add TimescaleDB to the database** ```sql CREATE EXTENSION IF NOT EXISTS timescaledb; ``` 3. **Check that TimescaleDB is installed** ```sql \dx ``` You see the list of installed extensions: ```sql List of installed extensions Name | Version | Default version | Schema | Description -------------+---------+-----------------+------------+--------------------------------------------------------------------------------------- plpgsql | 1.0 | 1.0 | pg_catalog | PL/pgSQL procedural language timescaledb | 2.24.0 | 2.24.0 | public | Enables scalable inserts and complex queries for time-series data (Community Edition) (2 rows) ``` Press `\q` to exit the list of extensions. - Linux ```bash ./bootstrap ``` - Windows ```powershell bootstrap.bat ``` - Linux ```bash cd build && make ``` - Windows ```powershell cmake --build ./build --config Release ``` - Linux ```bash make install ``` - Windows ```powershell cmake --build ./build --config Release --target install ``` - Linux ```bash service postgresql restart ``` - Windows ```powershell pg_ctl restart ``` ## Supported platforms [Section titled “Supported platforms”](#supported-platforms) * Linux Tiger Data recommends Ubuntu or Debian for the smoothest installation and support experience. | Operation system | Version | | ------------------------------- | --------------------------------------------------------------------------------------------------------- | | Debian | 13 Trixe, 12 Bookworm, 11 Bullseye | | Ubuntu | 26.04 LTS Resolute Raccoon, 24.04 LTS Noble Numbat | | Red Hat Enterprise | Linux 10, Linux 9. | | Fedora | Fedora 35, Fedora 34, Fedora 33 | | Rocky Linux | Rocky Linux 9 (x86\_64), Rocky Linux 8 | | ArchLinux (community-supported) | Check the [available packages](https://archlinux.org/packages/?sort=\&q=timescale\&maintainer=\&flagged=) | * macOS | Operation system | Version | | ---------------- | -------------------------------- | | macOS | From 10.15 Catalina to 14 Sonoma | * Windows You can deploy TimescaleDB on the following systems: | Operation system | Version | | ------------------------ | ---------------- | | Microsoft Windows | 10, 11 | | Microsoft Windows Server | 2019, 2022, 2025 | # Supported platforms > The platforms and systems supported by Tiger Data products This page lists the platforms and systems that Tiger Data products have been tested on for the following options: * **Tiger Cloud**: all the latest features that just work. A reliable and worry-free PostgreSQL cloud for all your workloads. * **Self-hosted products**: create your best app from the comfort of your own developer environment. ## Tiger Cloud [Section titled “Tiger Cloud”](#tiger-cloud) Tiger Cloud always runs the latest version of all Tiger Data products. With Tiger Cloud you: * Build everything on one service, and each service hosts one database * Get faster queries using less compute * Compress data without sacrificing performance * View insights on performance, queries, and more * Reduce storage with automated retention policies See the available [service capabilities](#available-service-capabilities) and [regions](#available-regions). ### Available service capabilities [Section titled “Available service capabilities”](#available-service-capabilities) Tiger Cloud services run optimized Tiger Data extensions on latest PostgreSQL, in a highly secure cloud environment. Each service is a specialized database instance tuned for your workload. Available capabilities are: | Capability | Extensions | | ------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Real-time analytics** - Lightning-fast ingest and querying of time-based and event data. | * TimescaleDB * TimescaleDB Toolkit | | **AI and vector** - Seamlessly build RAG, search, and AI agents. | - TimescaleDB - pgvector - pgvectorscale - pgai | | **Hybrid** - Everything for real-time analytics and AI workloads, combined. | * TimescaleDB * TimescaleDB Toolkit * pgvector * pgvectorscale * pgai | | **Support** | - 24/7 support no matter where you are. - Continuous incremental backup/recovery. - Point-in-time forking/branching. - Zero-downtime upgrades. - Multi-AZ high availability. - An experienced global ops and support team that can build and manage PostgreSQL at scale. | ### Available regions [Section titled “Available regions”](#available-regions) * Tiger Cloud on AWS Tiger Cloud services run in the following Amazon Web Services (AWS) regions: | Region | Zone | Location | | ---------------- | ------------- | -------------- | | `ap-south-1` | Asia Pacific | Mumbai | | `ap-southeast-1` | Asia Pacific | Singapore | | `ap-southeast-2` | Asia Pacific | Sydney | | `ap-northeast-1` | Asia Pacific | Tokyo | | `ca-central-1` | Canada | Central | | `eu-central-1` | Europe | Frankfurt | | `eu-central-2` | Europe | Zurich | | `eu-west-1` | Europe | Ireland | | `eu-west-2` | Europe | London | | `sa-east-1` | South America | São Paulo | | `us-east-1` | United States | North Virginia | | `us-east-2` | United States | Ohio | | `us-west-2` | United States | Oregon | * Tiger Cloud on Azure Tiger Cloud services run in the following Microsoft Azure regions: | Region | Location | | -------------------- | --------- | | `eastus2` | Virginia | | `westeurope` | Amsterdam | | `germanywestcentral` | Frankfurt | | `southeastasia` | Singapore | ## Self-hosted products [Section titled “Self-hosted products”](#self-hosted-products) You use Tiger Data's open-source products to create your best app from the comfort of your own developer environment. See the [available services](#available-services) and [supported systems](#supported-operating-system). ### Available services Tiger Data offers the following services for your self-hosted installations: | Service type | Description | | ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Self-hosted support** | * 24/7 support no matter where you are. * An experienced global ops and support team that can build and manage PostgreSQL at scale.Want to try it out? [See how we can help](https://www.tigerdata.com/self-managed-support). | ### PostgreSQL, TimescaleDB support matrix TimescaleDB and TimescaleDB Toolkit run on PostgreSQL v10, v11, v12, v13, v14, v15, v16, v17, and v18. The latest versions support PostgreSQL 15 and higher. Warning TimescaleDB 2.28.x is the final minor version to support PostgreSQL 15. Support for PostgreSQL 15 is dropped in 2.29.0; patch releases in the 2.28.x cycle continue to support it. | TimescaleDB version | PostgreSQL 18 | PostgreSQL 17 | PostgreSQL 16 | PostgreSQL 15 | PostgreSQL 14 | PostgreSQL 13 | PostgreSQL 12 | PostgreSQL 11 | PostgreSQL 10 | | ------------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | | 2.28.x | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | 2.27.x | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | 2.26.x | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | 2.25.x | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | 2.24.x | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | 2.23.x | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | 2.22.x | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | 2.21.x | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | 2.20.x | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | 2.17 - 2.19 | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | | 2.16.x | ❌ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | | 2.13 - 2.15 | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | | 2.12.x | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | | 2.10.x | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | | 2.5 - 2.9 | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | | 2.4 | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | | 2.1 - 2.3 | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ❌ | | 2.0 | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | | 1.7 | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | We recommend not using TimescaleDB with PostgreSQL 17.1, 16.5, 15.9, 14.14, 13.17, 12.21.\ These minor versions [introduced a breaking binary interface change](https://www.postgresql.org/about/news/postgresql-172-166-1510-1415-1318-and-1222-released-2965/) that, once identified, was reverted in subsequent minor PostgreSQL versions 17.2, 16.6, 15.10, 14.15, 13.18, and 12.22. When you build from source, best practice is to build with PostgreSQL 17.2, 16.6, etc and higher. Users of [Tiger Cloud](https://console.cloud.tigerdata.com/) and platform packages for Linux, Windows, MacOS, Docker, and Kubernetes are unaffected. ### Supported operating system You can deploy TimescaleDB and TimescaleDB Toolkit on the following systems: * Linux Tiger Data recommends Ubuntu or Debian for the smoothest installation and support experience. | Operation system | Version | | ------------------------------- | --------------------------------------------------------------------------------------------------------- | | Debian | 13 Trixe, 12 Bookworm, 11 Bullseye | | Ubuntu | 26.04 LTS Resolute Raccoon, 24.04 LTS Noble Numbat | | Red Hat Enterprise | Linux 10, Linux 9. | | Fedora | Fedora 35, Fedora 34, Fedora 33 | | Rocky Linux | Rocky Linux 9 (x86\_64), Rocky Linux 8 | | ArchLinux (community-supported) | Check the [available packages](https://archlinux.org/packages/?sort=\&q=timescale\&maintainer=\&flagged=) | * Windows You can deploy TimescaleDB on the following systems: | Operation system | Version | | ------------------------ | ---------------- | | Microsoft Windows | 10, 11 | | Microsoft Windows Server | 2019, 2022, 2025 | * MacOS | Operation system | Version | | ---------------- | -------------------------------- | | macOS | From 10.15 Catalina to 14 Sonoma | # Compare TimescaleDB editions > See the difference between the TimescaleDB Community and TimescaleDB Apache 2 editions The following versions of TimescaleDB are available: * TimescaleDB Apache 2 Edition * TimescaleDB Community Edition Your use of either TimescaleDB Apache 2 Edition or TimescaleDB Community Edition is governed by our [Privacy Policy](https://www.tigerdata.com/legal/privacy). ## Check your installed edition [Section titled “Check your installed edition”](#check-your-installed-edition) If you are already running TimescaleDB, check your edition with this command: ```sql SHOW timescaledb.license; ``` This returns either `timescale` (TimescaleDB Community Edition features enabled, the default) or `apache` (Apache 2-only mode). For more detail, query `pg_settings`: ```sql SELECT name, setting, source, sourcefile, sourceline FROM pg_settings WHERE name = 'timescaledb.license'; ``` This is more informative because it tells you: * `setting`: the current value, `timescale` or `apache`. * `source`: where it was set, such as the default, a configuration file, or the session. * `sourcefile` and `sourceline`: which configuration file and line it came from, if applicable. ## TimescaleDB Apache 2 Edition [Section titled “TimescaleDB Apache 2 Edition”](#timescaledb-apache-2-edition) TimescaleDB Apache 2 Edition is available under the [Apache 2.0 license](https://github.com/timescale/timescaledb/blob/master/LICENSE-APACHE). This is a classic open source license, meaning that it is completely unrestricted - anyone can take this code and offer it as a service. You can install TimescaleDB Apache 2 Edition on your own on-premises or cloud infrastructure and run it for free. You can sell TimescaleDB Apache 2 Edition as a service, even if you're not the main contributor. You can modify the TimescaleDB Apache 2 Edition source code and run it for production use. ## TimescaleDB Community Edition [Section titled “TimescaleDB Community Edition”](#timescaledb-community-edition) TimescaleDB Community Edition is the advanced, best, and most feature complete version of TimescaleDB, available under the terms of the [Tiger Data License (TSL)](https://github.com/timescale/timescaledb/blob/master/tsl/LICENSE-TIMESCALE). For more information about the Tiger Data license, see [this blog post](https://www.tigerdata.com/blog/how-we-are-building-a-self-sustaining-open-source-business-in-the-cloud-era). Many of the most recent features of TimescaleDB are only available in TimescaleDB Community Edition. You can install TimescaleDB Community Edition in your own on-premises or cloud infrastructure and run it for free. TimescaleDB Community Edition is completely free if you manage your own service. You cannot sell TimescaleDB Community Edition as a service, even if you are the main contributor. You can modify the TimescaleDB Community Edition source code and run it for production use. Developers using TimescaleDB Community Edition have the "right to repair" and make modifications to the source code and run it in their own on-premises or cloud infrastructure. However, you cannot make modifications to the TimescaleDB Community Edition source code and offer it as a service. You can access a hosted version of TimescaleDB Community Edition through [Tiger Cloud](/docs/get-started/quickstart/create-service), a cloud-native platform for time-series and real-time analytics. ## Feature comparison [Section titled “Feature comparison”](#feature-comparison) | Features | TimescaleDB Apache 2 Edition | TimescaleDB Community Edition | | ---------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------- | ----------------------------- | | **Hypertables and chunks** | | | | [CREATE TABLE](https://www.tigerdata.com/docs/api/latest/hypertable/create_table/) | ✅ | ✅ | | [create\_hypertable](https://www.tigerdata.com/docs/api/latest/hypertable/create_hypertable/) | ✅ | ✅ | | [show\_chunks](https://www.tigerdata.com/docs/api/latest/hypertable/show_chunks/) | ✅ | ✅ | | [drop\_chunks](https://www.tigerdata.com/docs/api/latest/hypertable/drop_chunks/) | ✅ | ✅ | | [split\_chunk](https://www.tigerdata.com/docs/api/latest/hypertable/split_chunk/) | ❌ | ✅ | | [reorder\_chunk](https://www.tigerdata.com/docs/api/latest/hypertable/reorder_chunk/) | ❌ | ✅ | | [move\_chunk](https://www.tigerdata.com/docs/api/latest/hypertable/move_chunk/) | ❌ | ✅ | | [add\_reorder\_policy](https://www.tigerdata.com/docs/api/latest/hypertable/add_reorder_policy/) | ❌ | ✅ | | [attach\_tablespace](https://www.tigerdata.com/docs/api/latest/hypertable/attach_tablespace/) | ✅ | ✅ | | [detach\_tablespace()](https://www.tigerdata.com/docs/api/latest/hypertable/detach_tablespace/) | ✅ | ✅ | | [detach\_tablespaces()](https://www.tigerdata.com/docs/api/latest/hypertable/detach_tablespaces/) | ✅ | ✅ | | [show\_tablespaces](https://www.tigerdata.com/docs/api/latest/hypertable/show_tablespaces/) | ✅ | ✅ | | [set\_chunk\_time\_interval](https://www.tigerdata.com/docs/api/latest/hypertable/set_chunk_time_interval/) | ✅ | ✅ | | [set\_integer\_now\_func](https://www.tigerdata.com/docs/api/latest/hypertable/set_integer_now_func/) | ✅ | ✅ | | [add\_dimension()](https://www.tigerdata.com/docs/api/latest/hypertable/add_dimension/) | ✅ | ✅ | | [create\_index (Transaction Per Chunk)](https://www.tigerdata.com/docs/api/latest/hypertable/create_index/) | ✅ | ✅ | | [hypertable\_size](https://www.tigerdata.com/docs/api/latest/hypertable/hypertable_size/) | ✅ | ✅ | | [hypertable\_detailed\_size](https://www.tigerdata.com/docs/api/latest/hypertable/hypertable_detailed_size/) | ✅ | ✅ | | [hypertable\_index\_size](https://www.tigerdata.com/docs/api/latest/hypertable/hypertable_index_size/) | ✅ | ✅ | | [chunks\_detailed\_size](https://www.tigerdata.com/docs/api/latest/hypertable/chunks_detailed_size/) | ✅ | ✅ | | [SkipScan](https://www.tigerdata.com/docs/use-timescale/latest/query-data/skipscan/) | ❌ | ✅ | | **Hypercore** | | | | [ALTER TABLE (Hypercore)](https://www.tigerdata.com/docs/api/latest/hypercore/alter_table/) | ❌ | ✅ | | [add\_columnstore\_policy](https://www.tigerdata.com/docs/api/latest/hypercore/add_columnstore_policy/) | ❌ | ✅ | | [remove\_columnstore\_policy](https://www.tigerdata.com/docs/api/latest/hypercore/remove_columnstore_policy/) | ❌ | ✅ | | [convert\_to\_columnstore](https://www.tigerdata.com/docs/api/latest/hypercore/convert_to_columnstore/) | ❌ | ✅ | | [convert\_to\_rowstore](https://www.tigerdata.com/docs/api/latest/hypercore/convert_to_rowstore/) | ❌ | ✅ | | [hypertable\_columnstore\_settings](https://www.tigerdata.com/docs/api/latest/hypercore/hypertable_columnstore_settings/) | ❌ | ✅ | | [hypertable\_columnstore\_stats](https://www.tigerdata.com/docs/api/latest/hypercore/hypertable_columnstore_stats/) | ❌ | ✅ | | [chunk\_columnstore\_settings](https://www.tigerdata.com/docs/api/latest/hypercore/chunk_columnstore_settings/) | ❌ | ✅ | | [chunk\_columnstore\_stats](https://www.tigerdata.com/docs/api/latest/hypercore/chunk_columnstore_stats/) | ❌ | ✅ | | **Continuous aggregates** | | | | [CREATE MATERIALIZED VIEW (Continuous Aggregate)](https://www.tigerdata.com/docs/api/latest/continuous-aggregates/create_materialized_view/) | ❌ | ✅ | | [ALTER MATERIALIZED VIEW (Continuous Aggregate)](https://www.tigerdata.com/docs/api/latest/continuous-aggregates/alter_materialized_view/) | ❌ | ✅ | | [DROP MATERIALIZED VIEW (Continuous Aggregate)](https://www.tigerdata.com/docs/api/latest/continuous-aggregates/drop_materialized_view/) | ❌ | ✅ | | [add\_continuous\_aggregate\_policy()](https://www.tigerdata.com/docs/api/latest/continuous-aggregates/add_continuous_aggregate_policy/) | ❌ | ✅ | | [refresh\_continuous\_aggregate](https://www.tigerdata.com/docs/api/latest/continuous-aggregates/refresh_continuous_aggregate/) | ❌ | ✅ | | [remove\_continuous\_aggregate\_policy()](https://www.tigerdata.com/docs/api/latest/continuous-aggregates/remove_continuous_aggregate_policy/) | ❌ | ✅ | | **Data retention** | | | | [add\_retention\_policy](https://www.tigerdata.com/docs/api/latest/data-retention/add_retention_policy/) | ❌ | ✅ | | [remove\_retention\_policy](https://www.tigerdata.com/docs/api/latest/data-retention/remove_retention_policy/) | ❌ | ✅ | | **Jobs and automation** | | | | [add\_job](https://www.tigerdata.com/docs/api/latest/jobs-automation/add_job/) | ❌ | ✅ | | [alter\_job](https://www.tigerdata.com/docs/api/latest/jobs-automation/alter_job/) | ❌ | ✅ | | [delete\_job](https://www.tigerdata.com/docs/api/latest/jobs-automation/delete_job/) | ❌ | ✅ | | [run\_job](https://www.tigerdata.com/docs/api/latest/jobs-automation/run_job/) | ❌ | ✅ | | **Hyperfunctions** | | | | [approximate\_row\_count](https://www.tigerdata.com/docs/api/latest/hyperfunctions/approximate_row_count/) | ✅ | ✅ | | [first](https://www.tigerdata.com/docs/api/latest/hyperfunctions/first/) | ✅ | ✅ | | [last](https://www.tigerdata.com/docs/api/latest/hyperfunctions/last/) | ✅ | ✅ | | [histogram](https://www.tigerdata.com/docs/api/latest/hyperfunctions/histogram/) | ✅ | ✅ | | [time\_bucket](https://www.tigerdata.com/docs/api/latest/hyperfunctions/time_bucket/) | ✅ | ✅ | | [time\_bucket\_gapfill](https://www.tigerdata.com/docs/api/latest/hyperfunctions/gapfilling/time_bucket_gapfill/) | ❌ | ✅ | | [locf](https://www.tigerdata.com/docs/api/latest/hyperfunctions/gapfilling/time_bucket_gapfill#locf) | ❌ | ✅ | | [interpolate](https://www.tigerdata.com/docs/api/latest/hyperfunctions/gapfilling/time_bucket_gapfill#interpolate) | ❌ | ✅ | | [percentile\_agg](https://www.tigerdata.com/docs/api/latest/hyperfunctions/percentile-approximation/uddsketch/#percentile-agg) | ❌ | ✅ | | [approx\_percentile](https://www.tigerdata.com/docs/api/latest/hyperfunctions/percentile-approximation/uddsketch/#approx_percentile) | ❌ | ✅ | | [approx\_percentile\_rank](https://www.tigerdata.com/docs/api/latest/hyperfunctions/percentile-approximation/uddsketch/#approx_percentile_rank) | ❌ | ✅ | | [rollup](https://www.tigerdata.com/docs/api/latest/hyperfunctions/percentile-approximation/uddsketch/#rollup) | ❌ | ✅ | | [max\_val](https://www.tigerdata.com/docs/api/latest/hyperfunctions/percentile-approximation/tdigest/#max_val) | ❌ | ✅ | | [mean](https://www.tigerdata.com/docs/api/latest/hyperfunctions/percentile-approximation/uddsketch/#mean) | ❌ | ✅ | | [error](https://www.tigerdata.com/docs/api/latest/hyperfunctions/percentile-approximation/uddsketch/#error) | ❌ | ✅ | | [min\_val](https://www.tigerdata.com/docs/api/latest/hyperfunctions/percentile-approximation/tdigest/#min_val) | ❌ | ✅ | | [num\_vals](https://www.tigerdata.com/docs/api/latest/hyperfunctions/percentile-approximation/uddsketch/#num_vals) | ❌ | ✅ | | [uddsketch](https://www.tigerdata.com/docs/api/latest/hyperfunctions/percentile-approximation/uddsketch/#uddsketch) | ❌ | ✅ | | [tdigest](https://www.tigerdata.com/docs/api/latest/hyperfunctions/percentile-approximation/tdigest/#tdigest) | ❌ | ✅ | | [time\_weight](https://www.tigerdata.com/docs/api/latest/hyperfunctions/time-weighted-calculations/time_weight/) | ❌ | ✅ | | [rollup](https://www.tigerdata.com/docs/api/latest/hyperfunctions/time-weighted-calculations/time_weight#rollup) | ❌ | ✅ | | [average](https://www.tigerdata.com/docs/api/latest/hyperfunctions/time-weighted-calculations/time_weight#average) | ❌ | ✅ | | **Informational Views** | | | | [timescaledb\_information.chunks](https://www.tigerdata.com/docs/api/latest/informational-views/chunks/#available-columns) | ✅ | ✅ | | [timescaledb\_information.continuous\_aggregates](https://www.tigerdata.com/docs/api/latest/informational-views/continuous_aggregates/#sample-usage) | ✅ | ✅ | | [timescaledb\_information.dimension](https://www.tigerdata.com/docs/api/latest/informational-views/dimensions/#timescaledb-information-dimensions) | ✅ | ✅ | | [timescaledb\_information.hypertables](https://www.tigerdata.com/docs/api/latest/informational-views/hypertables/) | ✅ | ✅ | | [timescaledb\_information.jobs](https://www.tigerdata.com/docs/api/latest/informational-views/jobs/#available-columns) | ✅ | ✅ | | [timescaledb\_information.job\_stats](https://www.tigerdata.com/docs/api/latest/informational-views/job_stats/#available-columns) | ✅ | ✅ | | **Administration functions** | | | | [timescaledb\_pre\_restore](https://www.tigerdata.com/docs/api/latest/administration/#timescaledb_pre_restore) | ✅ | ✅ | | [timescaledb\_post\_restore](https://www.tigerdata.com/docs/api/latest/administration/#timescaledb_post_restore) | ✅ | ✅ | | [get\_telemetry\_report](https://www.tigerdata.com/docs/api/latest/administration/#get_telemetry_report) | ✅ | ✅ | | [dump\_meta\_data](https://www.tigerdata.com/docs/api/latest/administration/#dump-timescaledb-meta-data) | ✅ | ✅ | # Cloud-exclusive features > Learn about the features and capabilities available exclusively in Tiger Cloud ## Tiered storage Tiger Cloud provides a [hierarchical storage architecture](/docs/learn/data-lifecycle/storage/about-storage-tiers) with two tiers: * **High-performance tier**: block storage optimized for fast reads and writes, providing up to 64 TB and 80,000 IOPS. * **Object storage tier**: low-cost, virtually unlimited storage backed by AWS S3 or Azure Blob, using Apache Parquet format for efficient analytical queries. Tiering policies automate data movement between tiers based on age, so recent data stays fast and older data stays affordable. Tiered data remains fully queryable with standard SQL. Learn how to [manage storage](/docs/build/data-management/storage/manage-storage) and [query tiered data](/docs/build/data-management/storage/query-tiered-data). ## High-availability replicas Tiger Cloud supports automatic high-availability replicas that maintain a synchronized copy of your service. If the primary instance becomes unavailable, the replica is promoted automatically, minimizing downtime. HA replicas are available on Scale and Enterprise pricing plans. Learn more about high availability on [AWS](/docs/deploy/tiger-cloud/tiger-cloud-aws/high-availability/high-availability) or [Azure](/docs/deploy/tiger-cloud/tiger-cloud-azure/high-availability/high-availability). ## Read replicas Offload read-heavy workloads — dashboards, reporting queries, and analytics — to dedicated read replicas. Read replicas receive streaming replication from the primary and serve read-only queries independently, which helps avoid resource contention on the primary. Set up read replicas on [AWS](/docs/deploy/tiger-cloud/tiger-cloud-aws/high-availability/read-scaling) or [Azure](/docs/deploy/tiger-cloud/tiger-cloud-azure/high-availability/read-scaling). ## Built-in observability Tiger Console provides observability tools without requiring additional monitoring infrastructure: * **Insights**: analyze query performance, identify slow queries, and understand resource utilization patterns directly from Tiger Console. * **Service metrics**: monitor CPU, memory, storage, and connection usage with built-in dashboards. * **External integrations**: export metrics to CloudWatch, Datadog, Prometheus, or Azure Monitor to integrate with your existing monitoring stack. Learn more about monitoring on [AWS](/docs/deploy/tiger-cloud/tiger-cloud-aws/monitoring) or [Azure](/docs/deploy/tiger-cloud/tiger-cloud-azure/monitoring). ## Data connectors Tiger Cloud provides built-in connectors for ingesting and exporting data: * **[Source PostgreSQL connector](/docs/integrate/connectors/source/sync-from-postgres)**: replicate data from an external PostgreSQL database into your service. * **[Source S3 connector](/docs/integrate/connectors/source/sync-from-s3)**: import data from CSV and Parquet files stored in Amazon S3. * **[Kafka connector](/docs/integrate/connectors/source/sync-from-kafka)**: stream data from Apache Kafka topics directly into hypertables. * **[Iceberg connector](/docs/integrate/connectors/destination/tigerlake)**: export data from Tiger Cloud to Apache Iceberg tables for lakehouse analytics workflows. Note Connector availability varies by cloud provider and pricing plan. See the [feature comparison](/docs/get-started/feature-comparison) for details. ## Managed security and compliance Tiger Cloud includes security features managed at the platform level: * **Encryption**: data is encrypted at rest and in transit by default. * **Network isolation**: [VPC peering](/docs/deploy/tiger-cloud/tiger-cloud-aws/security/vpc) (AWS), [AWS Transit Gateway](/docs/deploy/tiger-cloud/tiger-cloud-aws/security/transit-gateway), and [Azure Private Link](/docs/deploy/tiger-cloud/tiger-cloud-azure/security/azure-privatelink) provide private connectivity between your application and service. * **SOC 2 compliance**: Tiger Cloud is SOC 2 Type II certified. Learn more about security on [AWS](/docs/deploy/tiger-cloud/tiger-cloud-aws/security/overview) or [Azure](/docs/deploy/tiger-cloud/tiger-cloud-azure/security/overview). ## Automated operations Tiger Cloud manages the operational overhead of running a production database: * **Automated backups**: continuous backups with point-in-time recovery (PITR). See backup and restore on [AWS](/docs/deploy/tiger-cloud/tiger-cloud-aws/high-availability/backup-restore) or [Azure](/docs/deploy/tiger-cloud/tiger-cloud-azure/high-availability/backup-restore). * **Upgrades**: PostgreSQL major version upgrades, TimescaleDB updates, and security patches are applied automatically with minimal disruption. See upgrades on [AWS](/docs/deploy/tiger-cloud/tiger-cloud-aws/upgrades) or [Azure](/docs/deploy/tiger-cloud/tiger-cloud-azure/upgrades). * **Scaling**: resize compute and storage independently through Tiger Console or the Tiger REST API. See scaling on [AWS](/docs/deploy/tiger-cloud/tiger-cloud-aws/service-management/change-resources) or [Azure](/docs/deploy/tiger-cloud/tiger-cloud-azure/service-management/change-resources). ## Compare features across products For a detailed side-by-side comparison of Tiger Cloud on AWS, Tiger Cloud on Azure, and self-hosted TimescaleDB, see the [feature comparison](/docs/get-started/feature-comparison). # Contribute to the docs > How to contribute to Tiger Data documentation, open issues, submit changes via PR, or request changelog entries We welcome contributions to the Tiger Data documentation. Whether you're reporting a typo, suggesting clearer wording, or adding a changelog entry, your feedback helps everyone. This page explains the main ways to contribute and what good looks like for each. ## Ways to contribute [Section titled “Ways to contribute”](#ways-to-contribute) You can help improve the docs in three main ways: * **[Open an issue](#open-an-issue-for-feedback)**: Share feedback, report problems, or suggest improvements without changing code. * **[Submit a change via pull request (PR)](#submit-a-change-via-pull-request-pr)**: Fix errors, improve clarity, or add content; your change goes through review and our PR checklist. * **[Submit a changelog request](#submit-a-changelog-request)**: Propose a new entry for the [changelog](/docs/get-started/news/new) (for example, a new feature or release). Each path has a template and expectations so maintainers can review quickly and keep the docs consistent. Below we walk through each option and give examples of strong contributions. ## Open an issue for feedback [Section titled “Open an issue for feedback”](#open-an-issue-for-feedback) Use a **GitHub issue** when you want to: * Report unclear, outdated, or incorrect documentation * Suggest new topics or improvements without writing the change yourself * Ask questions about how something is documented * Flag broken links, missing steps, or confusing examples **Why we use issues:** Issues create a visible record of feedback, allow discussion, and can be linked from PRs when someone implements a fix. They also help us prioritize and triage. ### Choose the right issue template [Section titled “Choose the right issue template”](#choose-the-right-issue-template) In the [Tiger Data Docs repository](https://github.com/timescale/Tiger-Data-Docs), we provide issue templates so your report has the right structure: * **[Doc feedback/bug](https://github.com/timescale/Tiger-Data-Docs/issues/new?template=doc-feedback.md)**: For general documentation feedback, errors, or "this doesn’t match what I see" reports. * **[Doc correction or change](https://github.com/timescale/Tiger-Data-Docs/issues/new?template=doc-correction-or-change.md)**: When you’re planning to (or already) submit a PR and want to link an issue for context. * **[Changelog request](https://github.com/timescale/Tiger-Data-Docs/issues/new?template=changelog-request.md)**: When you want to propose a new changelog entry (for example, for a release or feature); the template reminds you of the required fields. Using the template ensures we get the necessary context (page, section, and description) without back-and-forth. ### Example of a good issue [Section titled “Example of a good issue”](#example-of-a-good-issue) **Title:** `[Get started] Quickstart step 3 is missing the connection port` **Body (using the doc feedback template):** * **Page/section:** Get started → Quickstart (5 minutes) → Step 3 "Connect to your service" * **What’s wrong:** The example connection string doesn’t include the port. New users copy-paste and get a connection error. * **What you expected:** Either include the port in the example or add a note that the port is required and where to find it in the console. * **Optional:** "I can submit a PR with the fix if that helps." **Why this is good:** The maintainer knows exactly where to look, what’s wrong, and what would fix it. Offering a PR is optional but speeds things up. ### Example of a weak issue [Section titled “Example of a weak issue”](#example-of-a-weak-issue) **Title:** `Docs are wrong` **Body:** "The quickstart doesn’t work." **Why it’s weak:** We don’t know which page, which step, or what "doesn’t work" means (connection? copy-paste? wrong product?). That leads to follow-up questions and slower resolution. ## Submit a change via pull request (PR) [Section titled “Submit a change via pull request (PR)”](#submit-a-change-via-pull-request-pr) When you’re ready to change the docs (fix a typo, clarify a sentence, add a step, or add new content), you do that via a **pull request**. All PRs must use the repository’s **pull request template** and pass the **checklist** before merge. This keeps quality consistent and makes review faster. ### What the PR template requires [Section titled “What the PR template requires”](#what-the-pr-template-requires) Every PR in this repo uses the template in [`.github/pull_request_template.md`](https://github.com/timescale/Tiger-Data-Docs/blob/main/.github/pull_request_template.md). You’ll see something like: 1. **Describe your changes**: what did you change and why? 2. **Affected pages**: leave this empty. Once you open the PR and the build runs, links to your changed pages appear here automatically; review them to confirm the rendered output, links, code blocks, and images look right. 3. **Related Issues**: if there’s a related issue, add its number after the `#` so GitHub links it and closes it when the PR merges. 4. **Checklist before requesting a review**: a list of boxes you must check before asking for review (see below). Filling this out is **required**. Our CI checks that the template sections are present and that the checklist is complete; incomplete PRs get a reminder comment. ### Checklist before requesting a review [Section titled “Checklist before requesting a review”](#checklist-before-requesting-a-review) The template includes a **checklist** that you must complete before requesting review. In short: * Confirm the PR is ready for review (or open it as a draft). * Confirm you’ve reviewed your own changes. * Confirm the content is technically accurate. * If you added or changed code, confirm it was tested on the latest available version. * Confirm the content is free of typos and grammar errors. * Confirm images and videos are clear and match production (or dev for unreleased features). * If the change references a feature that isn’t public yet, add a note so we can schedule the merge for after release. **Why we have this:** Docs are read in production. Checking accuracy, staging, and visuals in one place reduces the chance of shipping mistakes and keeps the bar high. ### Helpful resources before you edit [Section titled “Helpful resources before you edit”](#helpful-resources-before-you-edit) * **[README](https://github.com/timescale/Tiger-Data-Docs/blob/main/README.md)**: development quickstart, site structure, and how content is organized (for example, `get-started/`, `learn/`, `build/`). Use this to find where your change belongs and how to run the site locally. * **[Component usage (README-component.md)](https://github.com/timescale/Tiger-Data-Docs/blob/main/README-component.md)**: how to use callouts (Tip, Note, Important, Warning, CTA), buttons, and other MDX components so your new or edited content matches existing style. * **Doc constants**: the repo uses shared constants (for product and database names, for example) in `src/constants.ts`. Use them in prose and headings (such as `{C.PG}`) instead of hardcoding names; the README explains how. Following these keeps your PR consistent with the rest of the docs and avoids rework in review. ### Example of a good PR [Section titled “Example of a good PR”](#example-of-a-good-pr) **Title:** `Fix missing port in quickstart connection string` **Describe your changes:** * Updated the connection string in "Connect to your service" to include the port (for example, `:5432`). * Added a one-line note that the port is shown in the Tiger Console connection details. * Reason: Users were copy-pasting the string and hitting connection failures; including the port and a pointer to the console reduces support load and improves first-run success. **Related Issues:** Issue #42 **Checklist:** All boxes checked. Author notes: "Verified on staging; connection string matches console. No new code, only prose and one example." **Why this is good:** Clear scope, linked issue, and checklist done. The description explains the "why," which helps future readers and maintainers. ### Example of a weak PR [Section titled “Example of a weak PR”](#example-of-a-weak-pr) * **No description**: "Fixed the doc." Reviewers don’t know what was fixed or where. * **Checklist not completed**: CI will comment and block merge until the template is filled and boxes are checked. * **New content that ignores components**: for example, using raw HTML or markdown that doesn’t use the project’s Callout/Button patterns; that leads to style inconsistencies and rework. ## Submit a changelog request [Section titled “Submit a changelog request”](#submit-a-changelog-request) The [changelog](/docs/get-started/news/new) is the place for product and doc updates (new features, improvements, releases). If you have information about a change that should be announced there but you’re not editing the repo yourself, you can **submit a changelog request** via an issue. ### How changelog entries are added [Section titled “How changelog entries are added”](#how-changelog-entries-are-added) Changelog entries live in a single file: **`src/content/docs/get-started/news/new.mdx`**. Each entry is structured with: * A `##` heading (for the right-hand "On this page" nav) * A `` block with `title`, `date` (YYYY-MM-DD), and `tags` * A heading slot (the visible card title) and body content (with at least one `###` subsection) The full format, allowed tags, and examples are documented in **[README-changelog.md](https://github.com/timescale/Tiger-Data-Docs/blob/main/README-changelog.md)**. Maintainers use that guide when adding or editing entries. ### Using the changelog request issue template [Section titled “Using the changelog request issue template”](#using-the-changelog-request-issue-template) [Open a changelog request](https://github.com/timescale/Tiger-Data-Docs/issues/new?template=changelog-request.md) (or when opening a new issue, choose `Changelog request`). The template will ask for: * **Title of the entry**: Short, clear title for the new announcement. * **Date**: When the change was or will be released (YYYY-MM-DD). * **Suggested tags**: for example, `new-feature`, `improvement`, `bug-fix`, `breaking-change` (from the allowed list in README-changelog.md). * **Summary / body**: what’s new, for whom, and any links (docs, blog, and so on). You don’t need to write MDX; just provide the information. A maintainer can turn it into a proper `` using README-changelog.md. ### Example of a good changelog request [Section titled “Example of a good changelog request”](#example-of-a-good-changelog-request) **Title:** Changelog request: Automated chunk tuning (beta) **Body:** * **Entry title:** Automated chunk tuning (beta) * **Date:** 2026-03-06 * **Tags:** new-feature, beta * **Summary:** Tiger Cloud now offers automated chunk tuning in beta. Users can enable it per hypertable in the Explorer. The tuner recommends chunk intervals even if they don’t enable auto-tuning. Changes are incremental and logged in the Activity log. Link to docs: [improve-hypertable-performance](/docs/build/performance-optimization/improve-hypertable-performance#optimize-hypertable-chunk-intervals). **Why this is good:** We have a clear title, date, tags, and a concise summary that a maintainer can drop into the changelog file and format according to README-changelog.md. ### Example of a weak changelog request [Section titled “Example of a weak changelog request”](#example-of-a-weak-changelog-request) * **Vague title**: "New feature" with no product or feature name. * **Missing date or tags**: We need both to add the entry correctly. * **No summary**: "We released something" doesn’t give enough to write a useful card for readers. ## Writing style [Section titled “Writing style”](#writing-style) Prose follows the [Google developer documentation style guide](https://developers.google.com/style), enforced automatically by [Vale](https://vale.sh) in CI (only the lines your PR changes are checked). Run it locally on your changed files with `pnpm lint:prose`. Beyond Google's defaults, a few house rules matter most: no em dashes (see below), no Latinisms (write "for example" instead of the Latin abbreviation), use the `src/constants.ts` product-name constants instead of hardcoding names, and put UI element names in `code font` rather than **bold**. ### Punctuation [Section titled “Punctuation”](#punctuation) * **Do not use the em dash (Unicode U+2014)** in prose, frontmatter descriptions, tables, or comments. It is easy to confuse with hyphens or en dashes and it is not consistent across keyboards. Use **commas**, **semicolons**, **colons**, or **parentheses** instead, or split the thought into two sentences. * For **label + gloss** patterns (for example in lists of links or options), use a **colon** after the label: `**From CSV**: For comma-separated files…` rather than an em dash between the label and the rest. ### SEO summary in frontmatter [Section titled “SEO summary in frontmatter”](#seo-summary-in-frontmatter) * **`description`**: Used by Starlight for ``, Open Graph, and (in this site) subtitles in the sidebar and some nav UI. Use for pages where that short line should appear next to the title in navigation. * **`seoDescription`**: Use when you want the same kind of one-line summary for **search and social meta tags only**. It is injected in `src/components/Head.astro` and does **not** populate the sidebar. Prefer `description` for normal docs pages; use `seoDescription` when the menu should show titles only. * **If both are set**: **`description` wins.** Starlight keeps using `description` for meta and nav; `seoDescription` is ignored for meta injection so you do not get duplicate tags. Remove the extra field when you notice both, or leave it as a harmless no-op until cleanup. ### Product metadata in frontmatter [Section titled “Product metadata in frontmatter”](#product-metadata-in-frontmatter) Add a `products` field to every content page: `cloud`, `self_hosted`, or both. Use `mst` only on `deploy/mst` pages. ## Quick reference [Section titled “Quick reference”](#quick-reference) | Contribution type | Where to start | Key reference | | --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Feedback/report a problem** | [Open doc feedback issue](https://github.com/timescale/Tiger-Data-Docs/issues/new?template=doc-feedback.md) | [doc-feedback.md](https://github.com/timescale/Tiger-Data-Docs/blob/main/.github/ISSUE_TEMPLATE/doc-feedback.md) | | **Doc correction (with or without PR)** | [Open doc correction issue](https://github.com/timescale/Tiger-Data-Docs/issues/new?template=doc-correction-or-change.md) | [doc-correction-or-change.md](https://github.com/timescale/Tiger-Data-Docs/blob/main/.github/ISSUE_TEMPLATE/doc-correction-or-change.md) | | **Fix or improve the docs** | Fork, edit, then [open a PR](https://github.com/timescale/Tiger-Data-Docs/compare) | [PR template](https://github.com/timescale/Tiger-Data-Docs/blob/main/.github/pull_request_template.md), [README](https://github.com/timescale/Tiger-Data-Docs/blob/main/README.md), [README-component.md](https://github.com/timescale/Tiger-Data-Docs/blob/main/README-component.md) | | **Propose a changelog entry** | [Open changelog request](https://github.com/timescale/Tiger-Data-Docs/issues/new?template=changelog-request.md) | [README-changelog.md](https://github.com/timescale/Tiger-Data-Docs/blob/main/README-changelog.md), [changelog-request.md](https://github.com/timescale/Tiger-Data-Docs/blob/main/.github/ISSUE_TEMPLATE/changelog-request.md) | Want to add the changelog entry yourself? If you're comfortable editing MDX, you can add the entry directly in `src/content/docs/get-started/news/new.mdx` and open a PR. Follow the structure and checklist in [README-changelog.md](https://github.com/timescale/Tiger-Data-Docs/blob/main/README-changelog.md) so your entry matches existing ones and passes review. Thank you for helping improve Tiger Data documentation. # Feature comparison > Compare features across Tiger Cloud on AWS, Tiger Cloud on Azure, and self-hosted TimescaleDB This page compares the features available in **Tiger Cloud on AWS**, **Tiger Cloud on Azure**, and **self-hosted TimescaleDB Community Edition**. **Best-in-class PostgreSQL performance** | Feature | Tiger Cloud on AWS | Tiger Cloud on Azure | Self-hosted TimescaleDB | | --------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :-----------------------------------------------------------------------------------------------------------------------------------: | :---------------------: | | Automatic partitioning via hypertables for efficient indexes and faster ingest | ✅ | ✅ | ✅ | | Continuous aggregates | ✅ | ✅ | ✅ | | Time/partition-oriented constraint exclusion for faster queries | ✅ | ✅ | ✅ | | Skip scans, ordered appends, custom optimizations for faster LIMIT and DISTINCT queries | ✅ | ✅ | ✅ | | Columnar storage for accelerated scans | ✅ | ✅ | ✅ | | Vectorized query execution (SIMD) | ✅ | ✅ | ✅ | | Specialized vector indexes for AI applications | ✅ | ✅ | ✅ | | Source connectors | [PostgreSQL](/docs/integrate/connectors/source/sync-from-postgres), [Amazon S3](/docs/integrate/connectors/source/sync-from-s3), [Apache Kafka](/docs/integrate/connectors/source/sync-from-kafka) | [PostgreSQL](/docs/integrate/connectors/source/sync-from-postgres), [Apache Kafka](/docs/integrate/connectors/source/sync-from-kafka) | Manual | | Destination connectors | [Iceberg on S3 Tables](/docs/integrate/connectors/destination/tigerlake) | ❌ | ❌ | | In-Console CSV, Parquet, and text file imports | ✅ | ❌ | ❌ | *** **Flexible analysis with full SQL** | Feature | Tiger Cloud on AWS | Tiger Cloud on Azure | Self-hosted TimescaleDB | | ---------------------------------------------------------------------------------------- | :----------------: | :------------------: | :---------------------: | | Complete PostgreSQL ecosystem (all PostgreSQL features, connectors, third-party drivers) | ✅ | ✅ | ✅ | | Cross-table JOINs for time-series and events with relational tables | ✅ | ✅ | ✅ | | Rich timestamp and timezone support | ✅ | ✅ | ✅ | | Flexible time-bucketing for time-oriented analysis | ✅ | ✅ | ✅ | | Advanced hyperfunctions (interpolation, approximation, visualization) | ✅ | ✅ | ✅ | | Geospatial and vector data types | ✅ | ✅ | ✅ | *** **Automated data management** | Feature | Tiger Cloud on AWS | Tiger Cloud on Azure | Self-hosted TimescaleDB | | --------------------------------------------------- | :----------------: | :------------------: | :---------------------: | | Columnstore compression (up to 98% storage savings) | ✅ | ✅ | ✅ | | Data retention policies | ✅ | ✅ | ✅ | | Data tiering with automated policies | ✅ | ✅ | ❌ | | Data reordering for efficient disk scans | ✅ | ✅ | ✅ | | Data downsampling for efficient historical analysis | ✅ | ✅ | ✅ | | Background job scheduler and user-defined jobs | ✅ | ✅ | ✅ | | Automated chunk interval tuning | ✅ | ✅ | ❌ | *** **Enterprise scalability** | Feature | Tiger Cloud on AWS | Tiger Cloud on Azure | Self-hosted TimescaleDB | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :----------------: | :------------------: | :---------------------: | | Disaggregated compute and storage | ✅ | ✅ | Manual | | Dynamic compute resizing | ✅ | ✅ | ❌ | | Dynamic disk storage with usage-based pricing | ✅ | ✅ | ❌ | | Dynamic I/O provisioning for high-read/high-write performance | ✅ | ✅ | Manual | | Low-cost storage with infinite capacity | ✅ | ✅ | ❌ | | Transparent queries across high-performance and low-cost tiers | ✅ | ✅ | ❌ | | Read replicas with load balancing for seamless read scaling | ✅ | ✅ | Manual | | Connection pooling for connection scaling | ✅ | ✅ | Manual | | Automated resource-aware parameter tuning | ✅ | ✅ | ❌ | | Terraform for infrastructure-as-code control | ✅ | ✅ | ❌ | | Management interfaces ([Console](/docs/deploy/tiger-cloud/tiger-cloud-aws/service-management/service-overview), [REST API](/docs/reference/tiger-cloud-rest), [CLI](/docs/reference/tiger-cloud/tiger-cli), [Tiger MCP](/docs/get-started/quickstart/mcp-cli)) | ✅ | ✅ | ❌ | *** **High availability and reliability** | Feature | Tiger Cloud on AWS | Tiger Cloud on Azure | Self-hosted TimescaleDB | | ----------------------------------------------------------- | :----------------: | :------------------: | :---------------------: | | Multi-AZ deployments for high availability | ✅ | ✅ | Manual | | Continuous incremental backup and automated restore | ✅ | ✅ | Manual | | Cross-region backup and restore | ✅ | ❌ | ❌ | | Point-in-time recovery and forking | ✅ | ✅ | Manual | | Regular database and disk snapshots for fast restore | ✅ | ✅ | ❌ | | Rapid recovery (fast database restart, remote disk remount) | ✅ | ✅ | ❌ | | Memory guard protections to avoid out-of-memory crashes | ✅ | ✅ | ❌ | | Decoupled control/data planes for greater resilience | ✅ | ✅ | ❌ | | Commercial SLAs | ✅ | ✅ | ❌ | *** **Automated upgrades and software patching** | Feature | Tiger Cloud on AWS | Tiger Cloud on Azure | Self-hosted TimescaleDB | | -------------------------------------------------------------------------- | :----------------: | :------------------: | :---------------------: | | Automated upgrades during maintenance windows | ✅ | ✅ | ❌ | | Phased, zero-downtime TimescaleDB and PostgreSQL minor upgrades | ✅ | ✅ | ❌ | | PostgreSQL major version upgrades with forking workflow and disk snapshots | ✅ | ✅ | ❌ | | HA-replica-aware coordinated upgrades | ✅ | ✅ | ❌ | | Fleet-wide version and stability monitoring with staged roll-out/roll-back | ✅ | ✅ | ❌ | *** **Security and compliance** | Feature | Tiger Cloud on AWS | Tiger Cloud on Azure | Self-hosted TimescaleDB | | ------------------------------------------------------------ | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :-----------------------------------------------------------------------------------------: | :---------------------: | | SOC 2 Type 2, GDPR, HIPAA certified compliance | ✅ | ✅ | ❌ | | Data encryption at rest (disk and backup) | ✅ | ✅ | Manual | | Data encryption in transit | ✅ | ✅ | Manual | | Database SSL with fully verifiable certificate chains | ✅ | ✅ | ❌ | | Control plane role-based access control | ✅ | ✅ | ❌ | | Database role-based access control | ✅ | ✅ | ✅ | | Multi-factor authentication | ✅ | ✅ | ❌ | | Corporate SSO and SAML | ✅ | ✅ | ❌ | | IP allow lists | ✅ | ✅ | Manual | | Private connectivity to your cloud network | [AWS PrivateLink](/docs/deploy/tiger-cloud/tiger-cloud-aws/security/aws-privatelink), [Transit Gateway](/docs/deploy/tiger-cloud/tiger-cloud-aws/security/transit-gateway), [VPC peering](/docs/deploy/tiger-cloud/tiger-cloud-aws/security/vpc) | [Azure Private Link](/docs/deploy/tiger-cloud/tiger-cloud-azure/security/azure-privatelink) | ❌ | | Layered database privilege escalation protections | ✅ | ✅ | ❌ | | Secure SDLC, vulnerability scanning, third-party pen testing | ✅ | ✅ | ❌ | *** **Deep observability** | Feature | Tiger Cloud on AWS | Tiger Cloud on Azure | Self-hosted TimescaleDB | | -------------------------------------------------------------------------- | :-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :---------------------: | | Operational database visibility (performance, regressions, optimization) | ✅ | ✅ | ❌ | | Automated query analysis and statistics | ✅ | ✅ | ❌ | | Per-query drill-downs (execution times, row results, plans, memory, cache) | ✅ | ✅ | ❌ | | In-Console metric visualization and system logs | ✅ | ✅ | ❌ | | Metric exporters | [CloudWatch](/docs/integrate/observability-alerting/cloudwatch), [Prometheus](/docs/integrate/observability-alerting/prometheus), [Datadog](/docs/integrate/observability-alerting/datadog) | [Azure Monitor](/docs/integrate/observability-alerting/azure-monitor), [Prometheus](/docs/integrate/observability-alerting/prometheus), [Datadog](/docs/integrate/observability-alerting/datadog) | ❌ | | Connection monitoring and management | ✅ | ✅ | Manual | *** **Production-grade support and operations** | Feature | Tiger Cloud on AWS | Tiger Cloud on Azure | Self-hosted TimescaleDB | | --------------------------------------------------------------------------- | :----------------: | :------------------: | :-----------------------------------------: | | 24/7 follow-the-sun support (APAC, EMEA, Americas) | ✅ | ✅ | [Contact sales](mailto:sales@tigerdata.com) | | Production support (severity 1) | ✅ | ✅ | [Contact sales](mailto:sales@tigerdata.com) | | Architectural reviews, data modeling, query optimization, migration support | ✅ | ✅ | [Contact sales](mailto:sales@tigerdata.com) | | 24/7 operational monitoring and control | ✅ | ✅ | [Contact sales](mailto:sales@tigerdata.com) | | 98%+ customer satisfaction (CSAT scores) | ✅ | ✅ | [Contact sales](mailto:sales@tigerdata.com) | *** Choosing a deployment * **Tiger Cloud on AWS**: Full feature set, including AWS-specific integrations. Private connectivity through AWS PrivateLink, Transit Gateway, and VPC peering, cross-region backup, an Amazon S3 source connector, an Iceberg destination connector for S3 Tables, and metric export to CloudWatch (as well as Prometheus and Datadog). * **Tiger Cloud on Azure**: The same core database and management platform. Private connectivity through Azure Private Link and metric export to Azure Monitor (as well as Prometheus and Datadog). Some AWS-specific integrations, such as the S3 and Iceberg connectors, VPC peering, cross-region backup, and CloudWatch, are not yet available. * **Self-hosted TimescaleDB**: Full SQL and time-series capabilities. You manage backups, high availability, scaling, and observability yourself. Managed operations and enterprise support are available by [contacting sales](mailto:sales@tigerdata.com). # News and updates > Changelog, release notes, and product updates for Tiger Data [Changelog](/docs/get-started/news/new) [Latest updates and changes to Tiger Cloud with links to detailed documentation](/docs/get-started/news/new) [Release notes](/docs/get-started/news/release-notes) [New features and fixes across TimescaleDB, Toolkit, pgai, and related products](/docs/get-started/news/release-notes) # Changelog > Get the latest updates to Tiger Cloud with links to detailed documentation Search updates... Filter by: All ✨ New Feature 💡 Improvement 🏎️ Performance 🔐 Security 🔌 Integration 🌍 Availability ⚠️ Deprecation 🐛 Bug Fix 🚨 Breaking 🧪 Beta 🎉 GA ## TimescaleDB v2.28 now on Tiger Cloud [Section titled “TimescaleDB v2.28 now on Tiger Cloud”](#timescaledb-v228-now-on-tiger-cloud) July 2, 2026 ✨ New Feature 🏎️ Performance ⚠️ Deprecation [#](#timescaledb-v2-28-now-on-tiger-cloud) ## TimescaleDB v2.28 now on Tiger Cloud TimescaleDB 2.28.2 is the first release of the 2.28 line on Tiger Cloud, and it rolls up three upstream releases into a single upgrade: the 2.28.0 feature release, the 2.28.1 correctness patch, and the 2.28.2 upgrade-path patch. 2.28.0 closes several long-requested developer experience asks on continuous aggregates and continues maturing the columnstore, while 2.28.1 and 2.28.2 harden correctness on compressed data and stabilize the upgrade path. TimescaleDB 2.28.0 was released on June 16, 2026, 2.28.1 on June 23, and 2.28.2 on June 30; all three are delivered to Tiger Cloud as 2.28.2 starting July 2, 2026. ### Highlighted features in TimescaleDB v2.28 [Section titled “Highlighted features in TimescaleDB v2.28”](#highlighted-features-in-timescaledb-v228) **Developer experience improvements on continuous aggregates** Continuous aggregates now support in-place schema evolution, direct statistics maintenance, and incremental manual refresh, so you can build and maintain materialized data with the same familiar SQL you already use for ordinary tables. * **`ALTER MATERIALIZED VIEW ... ADD COLUMN`:** Continuous aggregates now support adding new aggregate columns in place, when the new column is a `GENERATED ALWAYS AS () STORED` expression. Previously, evolving a CAgg's schema meant creating a versioned copy, waiting for it to materialize, and switching application reads over. Now the column is added directly — new refreshes populate it going forward, and you backfill historical rows. ```sql ALTER MATERIALIZED VIEW conditions_summary_hourly ADD COLUMN max_temp double precision GENERATED ALWAYS AS (max(temperature)) STORED; ``` * **`ANALYZE` and `VACUUM` work directly on continuous aggregates:** Both maintenance commands are now transparently redirected to the underlying materialization hypertable and all associated chunks, including columnstore chunks, so you no longer invoke them on internal table names. * **Incremental manual refresh:** `refresh_continuous_aggregate()` now supports batched, incremental refresh through a new JSONB options argument, bringing manual refresh in line with policy-driven refresh and reducing memory pressure and lock duration on large refreshes. ```sql CALL refresh_continuous_aggregate( 'metrics_hourly', now() - INTERVAL '30 days', now(), options => '{"buckets_per_batch": 24}'::jsonb ); ``` * **`timescaledb.skip_cagg_invalidation` GUC for bulk loads:** A new session- or transaction-scoped GUC suppresses continuous aggregate invalidation tracking during DML and DDL on hypertables, for trusted bulk-load workflows where the caller refreshes affected CAggs afterward. This eliminates per-row invalidation overhead during large ingests. * **Lighter locking during invalidation processing:** Refresh operations now use a targeted row-level lock on the `continuous_agg` catalog row instead of a broader `ShareUpdateExclusiveLock`, so unrelated work on the same CAgg can proceed concurrently with refresh. **Faster recency queries on the columnstore** Sparse indexes for compressed chunks now record `first` and `last` values alongside the existing `min`/`max` metadata for each `orderby` column, and `ColumnarIndexScan` (default-on since 2.26) picks them up. "Latest reading per series" queries using two-argument `first(value, orderby)` or `last(value, orderby)` can now read straight from sparse-index metadata without touching batch payloads. Existing compressed chunks must be recompressed or retrofitted to pick up the new metadata. ```sql SELECT device_id, last(value, time) FROM metrics GROUP BY device_id; ``` **Broader vectorized execution** * **Vectorized `CASE` in the columnar executor:** `CASE ... WHEN ... THEN ... ELSE ... END` expressions inside vectorized aggregation over compressed hypertables no longer force a row-by-row fallback — a meaningful speedup for conditional aggregation. * **Higher parallelism for `SELECT` on compressed hypertables:** Parallelism is now sized more closely to the uncompressed data size, speeding up large analytical scans that were previously under-parallelized. **Sparse-index maintenance** * **Retrofit sparse indexes on existing chunks:** A new internal utility `_timescaledb_functions.rebuild_sparse_index()` updates sparse-index metadata on already-compressed chunks without a full decompress and recompress cycle. It diffs current metadata against the active compression settings and applies only what changed — useful after changing `segmentby`, `orderby`, or index settings on a hypertable that already has compressed data. * **Automatic cleanup when dropping columns:** When a column is dropped from a hypertable on the columnstore, TimescaleDB now removes all associated sparse-index metadata columns from compressed chunks. Previously those columns were left behind. **In-memory observability for compressed chunks** A new in-memory observability layer captures per-chunk statistics during compression events, queries, and DML on compressed data. Inspect compressed-batch counts, row averages, and activity counters through the `stat_chunk_activity` view or the `chunk_statistics()` functions. A new GUC `timescaledb.stats_max_chunks` controls how many chunks are tracked in memory. ```sql SELECT hypertable_name, chunk_name, compressed_batch_rows_avg FROM timescaledb_information.stat_chunk_activity; ``` **Gap-fill planning improvements** * **Better row-count estimates for `time_bucket_gapfill()`:** The planner now accounts for rows synthesized by gap-filling rather than relying solely on the subplan's row count. When `start`, `finish`, and `period` are known at plan time, estimates are materially closer to actual emitted rows, leading to better join and aggregation plans downstream. * **Subquery execution-parameter arguments to `time_bucket_gapfill()`:** Gap-fill arguments can now resolve from subquery execution parameters, removing a previous limitation for some dynamic-argument query shapes. ### Correctness and upgrade hardening [Section titled “Correctness and upgrade hardening”](#correctness-and-upgrade-hardening) The 2.28.1 and 2.28.2 patches folded into this rollout focus on compressed-data correctness and a clean upgrade path. * **Compressed-data correctness (2.28.1):** Fixes a data-loss bug where a `DELETE` with a subquery resolving to constant false could remove every row; fixes a server crash during DML when the planner chose a Bitmap Heap Scan; enforces `CHECK`, `NOT NULL`, and view `WITH CHECK OPTION` constraints on direct-compress inserts; corrects `ALTER TABLE ... RENAME COLUMN` on compressed chunks; and resolves several `first`/`last` aggregate edge cases. * **Upgrade path (2.28.1 and 2.28.2):** The 2.28.0 upgrade blocker that prevented older instances (created before 2.15.0 without a prior PostgreSQL version upgrade) from upgrading is resolved. 2.28.2 stabilizes the migration further by correctly migrating internal catalog tables, automatically dropping incompatible bloom filters during upgrade instead of halting for manual intervention, and fixing sparse-index column ordering for `first`/`last`. ### Deprecations and backward-incompatible changes [Section titled “Deprecations and backward-incompatible changes”](#deprecations-and-backward-incompatible-changes) * **PostgreSQL 15 support removed:** TimescaleDB 2.28.0 removes support for PostgreSQL 15, ending the deprecation window announced in 2.23. This is a TimescaleDB requirement, not a Tiger Cloud platform deprecation: PostgreSQL 15 remains available on Tiger Cloud, but a PostgreSQL 15 service must upgrade to PostgreSQL 16 or higher before it can adopt TimescaleDB 2.28.2. * **`_timescaledb_catalog.chunk_constraint` removed:** The internal `chunk_constraint` catalog table has been removed. Customers who query this catalog table directly will need to migrate. There are no SQL-surface behavior changes for standard TimescaleDB users. For complete details, refer to the [TimescaleDB 2.28.0 release notes](https://github.com/timescale/timescaledb/releases/tag/2.28.0), [2.28.1 release notes](https://github.com/timescale/timescaledb/releases/tag/2.28.1), and [2.28.2 release notes](https://github.com/timescale/timescaledb/releases/tag/2.28.2). ## Chunk timeline visualization [Section titled “Chunk timeline visualization”](#chunk-timeline-visualization) July 1, 2026 💡 Improvement [#](#chunk-timeline-visualization) ## Chunk timeline visualization You can now see your chunks visualized in the Explorer for each hypertable. The chart shows a timeline view of the hypertable's chunks, their relative sizes, and their composition (rowstore/columnstore). By default, a max of 100 of the latest chunks are shown. Click and drag on the chart to zoom in on a specific time range. It also shows the status of the latest runs for the associated columnstore job and lets you jump directly to the job deep dive page for that hypertable. ![Chunk timeline panel in Tiger Cloud showing compression stats and rowstore versus columnstore chunk sizes over time](/docs/_astro/chunk-timeline-tiger-cloud.BgIPUdKi_Z1cHvAA.webp) ## Updated service creation flow on Tiger Cloud [Section titled “Updated service creation flow on Tiger Cloud”](#updated-service-creation-flow-on-tiger-cloud) July 1, 2026 💡 Improvement [#](#updated-service-creation-flow-on-tiger-cloud) ## Updated service creation flow on Tiger Cloud We have updated the service creation flow in the Console to make it clearer and easier to use: * Fewer steps: We have reduced the number of steps down to three. * Persistent price display: You can see the price as you change various configurations, to better explore your options. * Updated GUI: The overall design is easier to navigate and parse. It is now the default in Tiger Cloud, so check it out. ![Step 1 of the updated Tiger Cloud service creation flow, showing compute and region selection with a live cost estimate](/docs/_astro/service-creation-flow-tiger-cloud.DKfl1gLq_1l1VaP.webp) ## TimescaleDB v2.27.2 now on Tiger Cloud [Section titled “TimescaleDB v2.27.2 now on Tiger Cloud”](#timescaledb-v2272-now-on-tiger-cloud) June 22, 2026 🐛 Bug Fix ✨ New Feature [#](#timescaledb-v2-27-2-now-on-tiger-cloud) ## TimescaleDB v2.27.2 now on Tiger Cloud TimescaleDB v2.27.2 is a stability and security patch that supersedes 2.27.1 on Tiger Cloud. Together, 2.27.1 and 2.27.2 close three access-control gaps in the jobs and policy subsystems, fix a correctness bug in grouped aggregation on the columnstore, refine the columnstore query planner, and ship upgrade-compatibility scripts for composite bloom filter metadata. TimescaleDB 2.27.1 was released on May 19, 2026 and 2.27.2 on June 2, 2026; both are delivered to Tiger Cloud as 2.27.2 starting June 22, 2026. ### Highlighted fixes in TimescaleDB v2.27.2 [Section titled “Highlighted fixes in TimescaleDB v2.27.2”](#highlighted-fixes-in-timescaledb-v2272) **Tightened access controls** Three independent fixes in the jobs and policy subsystems close cases where users could observe or act on objects they did not own. * **Hypertable ownership enforced before recompression:** Recompression operations now verify ownership before proceeding. Previously, users could trigger recompression on hypertables they did not own. * **`job_errors` view restricted to owners:** Failed-job visibility is now scoped — only owners see failure rows for their own jobs. Previously, non-owners could read failure information they should not have had access to. * **`policy_reorder_remove` ownership check added:** The policy-management operation now enforces ownership before allowing changes, closing a path that previously leaked policy-existence information to non-owners. **Correctness and planner refinements on the columnstore** * **Correct results for `GROUP BY` columns not in the `SELECT` list:** Queries grouping by a column not included in the `SELECT` list could return incorrect rows or crash when running through vectorized aggregation or `ColumnarIndexScan`. This is now fixed — no SQL change required. * **Safer `ColumnarIndexScan` planning:** The path now correctly bypasses query shapes it can't safely handle — including `GROUPING SETS` / `ROLLUP` / `CUBE`, grouping by expressions, and `WHERE` clauses containing subplans — avoiding inefficient or incorrect plan choices. **Operational fixes** * **Columnstore policy creation no longer blocked by stale refresh-policy check:** Adding a columnstore policy was failing in some workflows due to an unnecessary check on existing refresh policies. The check has been removed. * **Upgrade compatibility:** The upgrade path now cleans up orphaned `compression_settings` rows before migrating the catalog, and ships migration scripts for installations carrying composite bloom filter metadata (complementing the format change in 2.27.0). For complete details, refer to the [TimescaleDB 2.27.1 release notes](https://github.com/timescale/timescaledb/releases/tag/2.27.1) and the [TimescaleDB 2.27.2 release notes](https://github.com/timescale/timescaledb/releases/tag/2.27.2). ## Automated chunk tuning is now GA [Section titled “Automated chunk tuning is now GA”](#automated-chunk-tuning-is-now-ga) June 26, 2026 🎉 GA [#](#automated-chunk-tuning-is-now-ga) ## Automated chunk tuning is now GA Automated chunk tuning is now GA for all services. Enable the feature for each hypertable through the Explorer without having to request access. When enabled, Tiger Cloud continuously evaluates your hypertable and adjusts the interval on your behalf when it detects it is no longer optimal. Note that updated chunk intervals apply to new chunks only. ![Chunk auto-tuning in Tiger Console](/docs/_astro/automated-chunk-tuning.fHKpGxep_bJbUP.webp) See the [docs](/docs/build/performance-optimization/improve-hypertable-performance#automated-tuning) for details. ## New Azure regions for Tiger Cloud services [Section titled “New Azure regions for Tiger Cloud services”](#new-azure-regions-for-tiger-cloud-services) June 24, 2026 ✨ New Feature [#](#new-azure-regions-for-tiger-cloud-services) ## New Azure regions for Tiger Cloud services You can now create Tiger Cloud services in two more Microsoft Azure regions: * `germanywestcentral` (Frankfurt) * `southeastasia` (Singapore) See all available regions in [Supported regions](/docs/learn/tiger-cloud/regions). ## Updates to the Goals flow in Tiger Console [Section titled “Updates to the Goals flow in Tiger Console”](#updates-to-the-goals-flow-in-tiger-console) June 9, 2026 💡 Improvement [#](#updates-to-the-goals-flow-in-tiger-console) ## Updates to the Goals flow in Tiger Console We have updated the Goals flow Actions in Tiger Cloud to make them more effective and easier to use. * **Add columns from the UI:** The Action for creating a hypertable now lets you define columns directly in the Console instead of writing SQL. * **Set up columnstore at creation time:** You can configure a columnstore for your hypertable as part of the same creation flow, without a separate step. ![Create a hypertable wizard](/docs/_astro/create-hypertable-1.3lCLJvsJ_ZknbAV.webp)![Completed Create a hypertable Action](/docs/_astro/create-hypertable-2.jUFJsSW__MC22u.webp) ## TimescaleDB v2.27 now on Tiger Cloud [Section titled “TimescaleDB v2.27 now on Tiger Cloud”](#timescaledb-v227-now-on-tiger-cloud) May 19, 2026 ✨ New Feature [#](#timescaledb-v2-27-now-on-tiger-cloud) ## TimescaleDB v2.27 now on Tiger Cloud TimescaleDB 2.27.0 is a performance-focused release that extends bloom filter pruning into the write path so `UPDATE`, `DELETE`, and `UPSERT` can now skip decompressing batches that can't match, broadens vectorized execution on the columnstore to cover more analytical query patterns, and introduces an opt-in query rewriter that transparently routes matching queries to existing continuous aggregates. TimescaleDB 2.27.0 was released on May 12, 2026, and is available on GitHub and on Tiger Cloud as of May 19, 2026. ### Highlighted features in TimescaleDB v2.27 [Section titled “Highlighted features in TimescaleDB v2.27”](#highlighted-features-in-timescaledb-v227) * **Bloom filter pruning now applies to writes** This release extends bloom filter pruning — previously only used by `SELECT` — to `UPDATE`, `DELETE`, and `UPSERT` on compressed chunks. * **Bloom filter pruning for `UPDATE` and `DELETE`:** Statements with equality predicates now use bloom filters to skip decompressing batches whose compressed rows can't possibly match. When multiple bloom filters apply, the most selective one (by column count) is tried first. Benchmark examples show up to **160x faster** execution on selective workloads. `EXPLAIN` now reports new "Compressed batches filtered" and "Batches filtered after decompression" counters. * **Bloom filter pruning for `UPSERT`:** `UPSERT` queries now leverage bloom filters — including the composite bloom filters introduced in 2.26 — to skip decompressing batches when the arbiter values are guaranteed not to be present. The most selective filter is chosen automatically when multiple apply. `EXPLAIN` adds new statistics for visibility into pruning effectiveness (batches checked, batches pruned, batches without bloom, false positives). * **More queries on the vectorized columnstore path** This release expands the set of queries that stay on the high-performance columnar execution engine instead of falling back to row-based processing. * **Vectorized filters via the standard PostgreSQL function path:** The columnstore engine now supports vectorized evaluation of filters inline through the standard PostgreSQL function path. This broadens the class of queries that can take the fast columnar path — including continuous aggregate refreshes — with benchmarks showing speedups from **30% up to 2x** on affected workloads. * **Vectorized aggregation with more `WHERE` filters:** Vectorized aggregation now applies in cases where the `WHERE` clause contains filters not handled through the Vectorized Filters facility, including filters on `time_bucket()`. * **Scalar array operation pushdown:** Predicates of the form `col IN (…)` and `col = ANY(…)` are now pushed down into the columnar metadata scan as `OR`/`AND` clauses, so each value can be evaluated against compressed-chunk metadata before any rows are expanded. * **Continuous aggregates** * **[Automatic query rewriting with continuous aggregates](/docs/build/performance-optimization/cagg-query-rewrites) (experimental, opt-in):** A new planner optimization transparently rewrites user queries to use eligible continuous aggregates when the query's aggregation exactly matches a CAgg's definition — so applications get pre-materialized performance without referencing the CAgg directly. This feature is experimental in 2.27 and off by default. Only real-time CAggs are eligible, and CAggs with active invalidations or pending materialization ranges are excluded. PostgreSQL 16+ only. Enable with the `timescaledb.enable_cagg_rewrites` GUC; `timescaledb.cagg_rewrites_debug_info` prints eligibility diagnostics. * **[Compress as part of the continuous aggregate refresh policy](/docs/build/continuous-aggregates/compression-on-continuous-aggregates#convert-refreshed-chunks-to-the-columnstore-during-refresh):** The CAgg refresh policy can now compress chunks of the materialization hypertable that fall within the refresh window as part of the same job, consolidating what used to require a separate columnstore policy. Example: ```sql SELECT add_continuous_aggregate_policy( continuous_aggregate => 'metrics_hourly', start_offset => INTERVAL '30 days', end_offset => INTERVAL '1 hour', schedule_interval => INTERVAL '1 hour' ); add_continuous_aggregate_policy --------------------------------- 1001 (1 row) SELECT alter_job(1001, config => jsonb_set( (SELECT config FROM _timescaledb_catalog.bgw_job WHERE id = 1001), '{compress_after_refresh}', 'true' )); ``` * **Smarter chunk exclusion** * **Chunk exclusion for `IN` / `ANY` on open time dimensions:** Queries that filter the time dimension with `IN` or `= ANY(...)` clauses now benefit from chunk pruning — the planner derives a bounding min/max range from the values so irrelevant chunks can be excluded earlier. * **Faster `ORDER BY … LIMIT` on more compressed data:** Batch Sorted Merge — which returns ordered results from compressed chunks without fully decompressing and sorting them — now applies to chunks with no `segmentby`, and to chunks where the query pins every `segmentby` column to a constant. * **Quality of life and operability** * **Index creation progress reporting:** Hypertable index builds now populate PostgreSQL's built-in `pg_stat_progress_create_index` view, so long-running builds can be monitored the same way as on plain PostgreSQL. * **Automatic `segmentby` for direct compress:** Direct compress now analyzes buffered tuples at flush time to pick an appropriate `segmentby` when one isn't explicitly configured, replacing the previous static heuristic. * **`ALTER TABLE … RESET` on materialization hypertables:** Materialization hypertables now accept `ALTER TABLE … RESET (...)`, in line with regular hypertables. * **`ENABLE` / `DISABLE TRIGGER` on hypertables:** Standard PostgreSQL trigger-state commands now work directly on hypertables and propagate to every underlying chunk (including `ENABLE ALWAYS`, `ENABLE REPLICA`, and `DISABLE TRIGGER USER`/`ALL`). * **Notice when compression settings change:** A SQL-level notice is now emitted when compression settings change so behavior shifts are visible at run time. ### Backward-incompatible changes worth flagging [Section titled “Backward-incompatible changes worth flagging”](#backward-incompatible-changes-worth-flagging) * A bug in bloom filter sparse indexes on compressed `int2` columns could cause `SELECT` to miss matching rows. Upgrades are blocked for affected databases until the incorrect indexes are dropped manually. * 2.27 uses a new naming convention for composite bloom filter metadata. Queries continue to work, but composite bloom filters created in 2.26 won't be used until a lightweight, catalog-only [migration script](https://github.com/timescale/timescaledb-extras/blob/main/utils/2.27.x-fix-composite-bloom-columns.sql) renames the legacy columns (no data recompression required). ### Upcoming PostgreSQL 15 EOL [Section titled “Upcoming PostgreSQL 15 EOL”](#upcoming-postgresql-15-eol) As announced in the [v2.23.0 changelog](https://github.com/timescale/timescaledb/releases/tag/2.23.0) on October 29, 2025, the upcoming TimescaleDB release in June 2026 will be the last version supporting PostgreSQL 15. If you are still on PostgreSQL 15, plan an upgrade to PostgreSQL 16 or higher to maintain access to new features, bug fixes, and performance improvements. For complete details, refer to the [TimescaleDB 2.27.0 release notes](https://github.com/timescale/timescaledb/releases/tag/2.27.0). ## New I/O Boost options and extended storage capacity [Section titled “New I/O Boost options and extended storage capacity”](#new-io-boost-options-and-extended-storage-capacity) May 11, 2026 ✨ New Feature 🏎️ Performance [#](#new-i-o-boost-options-and-extended-storage-capacity) ## New I/O Boost options and extended storage capacity Standard high-performance storage now supports up to 80K IOPS and 4x more storage capacity (up to 64 TB) for services on the Scale and Enterprise pricing plans. * **Scale:** Choose between 16,000, 24,000, 32,000, and 40,000 IOPS and up to 1,500 MB/s of throughput. * **Enterprise:** Additional options of 60,000 and 80,000 and up to 2,000 MB/s of throughput. IOPS changes are applied without downtime and billed $0.41 per hour per 16K IOPS, prorated to the selected option. Minimum CPU limits apply. For full details, see [Manage storage and tiering](/docs/build/data-management/storage/manage-storage). ![High-performance storage options with I/O Boost in Tiger Console](/docs/_astro/high-performance-storage-tiger-console.N5Mk8-Vu_1uj4MP.webp) ## Support for private endpoints is now generally available on AWS and Azure [Section titled “Support for private endpoints is now generally available on AWS and Azure”](#support-for-private-endpoints-is-now-generally-available-on-aws-and-azure) May 8, 2026 ✨ New Feature 🔐 Security [#](#support-for-private-endpoints-is-now-generally-available-on-aws-and-azure) ## Support for private endpoints is now generally available on AWS and Azure Support for private endpoints is now generally available on AWS and Azure across all Tiger Cloud supported regions. Connect your applications to Tiger Cloud over private network links to keep database traffic off the public internet and inside the trust boundary your security team already relies on. * **AWS PrivateLink:** Reach Tiger Cloud services from your AWS VPC over the AWS private network, with no exposure to the public internet. * **Azure Private Link:** Connect from your Azure VNet to Tiger Cloud over Microsoft's private backbone with the same private-by-default model. * **All supported regions:** Available everywhere Tiger Cloud runs on AWS and Azure, with no regional gaps at GA. * **Console-based setup:** Provision and manage private endpoints directly from Tiger Console — no manual networking configuration required. * **Scale and Enterprise plans:** Included on the Scale and Enterprise pricing tiers. For information on pricing, see . ## Cross-region restore is now available on Enterprise [Section titled “Cross-region restore is now available on Enterprise”](#cross-region-restore-is-now-available-on-enterprise) May 6, 2026 ✨ New Feature [#](#cross-region-restore-is-now-available-on-enterprise) ## Cross-region restore is now available on Enterprise Cross-region backup protects data by copying backups to a separate, geographically distant region. Users subscribed to the Enterprise plan can now restore from a cross-region backup using the Tiger Console, ensuring business continuity in the event of a regional outage. To learn more see [cross-region backup and restore](/docs/deploy/tiger-cloud/tiger-cloud-aws/high-availability/backup-restore#enable-cross-region-backup). ![Enabling cross-region backup in Tiger Console](/docs/_astro/create-cross-region-backup-in-tiger-console.CGDp6LDi_2tHbvp.webp) ## Logs page revamped with search, filters, histogram view, and more [Section titled “Logs page revamped with search, filters, histogram view, and more”](#logs-page-revamped-with-search-filters-histogram-view-and-more) April 29, 2026 ✨ New Feature 💡 Improvement [#](#logs-page-revamped-with-search-filters-histogram-view-and-more) ## Logs page revamped with search, filters, histogram view, and more The Logs page in Tiger Console has been redesigned to make it easier to investigate and debug database issues. This release adds several new capabilities and improves the overall experience based on customer feedback. * **Search:** Search log output directly in Tiger Console to find specific phrases or messages * **Severity filters:** Filter by log severity level to focus on the entries relevant to your investigation * **Histogram view:** A log volume histogram shows how activity has changed over time, making it easier to correlate spikes with database or application changes * **Improved time selectors:** Updated date and time pickers let you scope the log view to a specific window more precisely * **CSV export:** Download logs as a CSV for analysis in external tools * **Scroll improvements:** Fixes glitchy scroll behavior when browsing large log volumes * **Copy:** Copy individual log lines directly from the UI ![Redesigned Logs page in Tiger Console with search, severity filters, and histogram view](/docs/_astro/tiger-console-service-logs.DLosa5jQ_Z29cxVQ.webp)![Download logs as CSV from the Tiger Console Logs page](/docs/_astro/tiger-cloud-download-logs.CuJEqaSp_2lPKoT.webp) ## PostgreSQL Source Connector is now GA (production-ready) [Section titled “PostgreSQL Source Connector is now GA (production-ready)”](#postgresql-source-connector-is-now-ga-production-ready) April 24, 2026 🎉 GA [#](#postgresql-source-connector-is-now-ga) ## PostgreSQL Source Connector is now GA (production-ready) The PostgreSQL Source Connector is now generally available (GA), completing its migration to the Tiger Connect architecture. This improves reliability, scalability, and consistency with other connectors. **What's new** * Aligned with Tiger Connect architecture, replacing legacy components * Configurable worker count for initial data copy during setup for improved performance * Flexible table selection: * Select via publication * Select tables directly * Redesigned UI with separate `Overview` and `Settings` views * SSH tunneling support for secure connections * Bulk updates for table and schema mappings * Improved hypertable configuration experience * General UX and workflow improvements **Notes**: This release marks the connector as stable and ready for production use. ## Dark mode now available on Tiger Cloud [Section titled “Dark mode now available on Tiger Cloud”](#dark-mode-now-available-on-tiger-cloud) April 21, 2026 ✨ New Feature [#](#dark-mode-now-available-on-tiger-cloud) ## Dark mode now available on Tiger Cloud Dark mode is now live in Tiger Console for all customers. Switch themes from the theme toggle in the top navigation — choose `Light`, `Dark`, or `System` to match your OS preference. ![Theme toggle in Tiger Cloud Console showing Light, Dark, and System options](/docs/_astro/tiger-cloud-dark-mode-1.CRExmgLM_eU2U2.webp) The default remains light mode for now, with an upcoming update planned to follow the user's system preference by default. ![Tiger Cloud Console project services page in dark mode](/docs/_astro/tiger-cloud-dark-mode-2.CwKgWDVO_Zopx1H.webp)![Tiger Cloud Console service overview page in dark mode](/docs/_astro/tiger-cloud-dark-mode-3.DpeuokFp_ZHzhi.webp) ## TimescaleDB v2.26.3 now on Tiger Cloud [Section titled “TimescaleDB v2.26.3 now on Tiger Cloud”](#timescaledb-v2263-now-on-tiger-cloud) April 14, 2026 🐛 Bug Fix [#](#timescaledb-v2-26-3-now-on-tiger-cloud) ## TimescaleDB v2.26.3 now on Tiger Cloud TimescaleDB v2.26.3 is a bug-fix release focused on operational reliability, with the most important improvement addressing concurrent refreshes of continuous aggregates. It was released on April 14, 2026, and was made available on Tiger Cloud right after release. ### Highlighted features in TimescaleDB v2.26.3 [Section titled “Highlighted features in TimescaleDB v2.26.3”](#highlighted-features-in-timescaledb-v2263) **More reliable continuous aggregate refreshes** This release fixes concurrent refresh issues for continuous aggregates, an important improvement for production workloads that run overlapping or adjacent refreshes. * **Fix concurrent refreshes of continuous aggregates:** Resolves issues seen when multiple CAgg refreshes run at the same time. The underlying fix replaces the earlier intermediate queue model with a new registration-based approach for refresh windows, adds better overlap detection, uses lock-skipping to avoid deadlocks, and ensures cleanup happens correctly even when refreshes are cancelled or interrupted. * **Why it matters:** The previous design could lead to deadlocks between overlapping refreshes and leave stale ranges behind when refreshes were interrupted. The new model makes concurrent refresh behavior safer and more predictable. **Additional bug fixes** * **Fix `alter_job` for retention policies with `drop_created_before`:** Resolves a failure case when updating retention policies with this argument. * **Fix cleanup of orphaned `compression_chunk_size` entries during extension upgrade:** Improves upgrade reliability. * **Fix resource leaks during continuous aggregate refresh:** Addresses error-path leaks during CAgg refresh processing. * **Fix gapfill DST bucket ordering issue:** Resolves out-of-order bucket creation during DST shifts in gapfill queries. For complete details, refer to the [TimescaleDB 2.26.3 release notes](https://github.com/timescale/timescaledb/releases/tag/2.26.3). ## TimescaleDB v2.26.2 now on Tiger Cloud [Section titled “TimescaleDB v2.26.2 now on Tiger Cloud”](#timescaledb-v2262-now-on-tiger-cloud) April 7, 2026 🐛 Bug Fix [#](#timescaledb-v2-26-2-now-on-tiger-cloud) ## TimescaleDB v2.26.2 now on Tiger Cloud TimescaleDB v2.26.2 continues the 2.26 stabilization work with a broader set of fixes across query correctness, memory safety, chunk pruning, and continuous aggregates. It was released on April 7, 2026, and was made available on Tiger Cloud right after release. ### Highlighted features in TimescaleDB v2.26.2 [Section titled “Highlighted features in TimescaleDB v2.26.2”](#highlighted-features-in-timescaledb-v2262) **Query correctness and planner fixes** This release improves correctness and stability for a number of query patterns introduced or expanded in recent releases. * **Fix chunk exclusion with mutable expressions:** Resolves a wrong-result issue when performing chunk exclusion using a mutable expression. * **Fix chunk skipping with dropped columns:** Improves correctness for chunk pruning in schemas that include dropped columns. * **Fix `GROUP BY ROLLUP` on compressed continuous aggregates:** Addresses an issue affecting this query pattern on compressed CAggs. **Reliability and memory-safety fixes** * **Fix WAL record tracking in `EXPLAIN` for direct compress:** Corrects WAL tracking shown in `EXPLAIN` for direct compress workflows. * **Fix several use-after-free issues:** Includes fixes for invalidation handling, job owner validation, and `reorder_chunk`, improving overall runtime safety. For complete details, refer to the [TimescaleDB 2.26.2 release notes](https://github.com/timescale/timescaledb/releases/tag/2.26.2). ## New cloud platform status page [Section titled “New cloud platform status page”](#new-cloud-platform-status-page) April 6, 2026 ✨ New Feature [#](#new-cloud-platform-status-page) ## New cloud platform status page The Tiger Cloud platform status page has been migrated to [status.tigerdata.com](https://status.tigerdata.com/). The new status page is tightly integrated with our incident response workflows to provide better visibility into service health, including historical uptime. You can subscribe to be notified whenever an incident is created, updated, and resolved. ## TimescaleDB v2.26.1 now on Tiger Cloud [Section titled “TimescaleDB v2.26.1 now on Tiger Cloud”](#timescaledb-v2261-now-on-tiger-cloud) March 30, 2026 🐛 Bug Fix [#](#timescaledb-v2-26-1-now-on-tiger-cloud) ## TimescaleDB v2.26.1 now on Tiger Cloud TimescaleDB v2.26.1 is a small follow-up release focused on stability. It was released on March 30, 2026, and was made available on Tiger Cloud right after release. ### Highlighted features in TimescaleDB v2.26.1 [Section titled “Highlighted features in TimescaleDB v2.26.1”](#highlighted-features-in-timescaledb-v2261) **Stability improvements** This release is focused on a targeted fix to improve reliability for workloads using the new columnstore query paths introduced in 2.26. * **Fix memory leak in `ColumnarScan`:** Resolves a memory leak in `ColumnarScan`, improving stability for affected query workloads. For complete details, refer to the [TimescaleDB 2.26.1 release notes](https://github.com/timescale/timescaledb/releases/tag/2.26.1). ## TimescaleDB v2.26 now on Tiger Cloud [Section titled “TimescaleDB v2.26 now on Tiger Cloud”](#timescaledb-v226-now-on-tiger-cloud) March 30, 2026 ✨ New Feature [#](#timescaledb-v2-26-now-on-tiger-cloud) ## TimescaleDB v2.26 now on Tiger Cloud TimescaleDB v2.26 continues improving how TimescaleDB scales for real-world analytical workloads, with major gains in columnstore query performance, better filtering efficiency on compressed data, and practical query-planning improvements that reduce unnecessary work. TimescaleDB 2.26.0 was released on March 24, 2026, and is available on GitHub and on Tiger Cloud as of March 30, 2026. ### Highlighted features in TimescaleDB v2.26 [Section titled “Highlighted features in TimescaleDB v2.26”](#highlighted-features-in-timescaledb-v226) * **Faster analytical queries on the columnstore** This release expands the vectorized query path so more analytical queries can stay on the high-performance columnar execution engine instead of falling back to slower row-based processing. * **Vectorized PostgreSQL functions in aggregation paths:** Queries that use functions like `time_bucket()` in grouping or aggregation expressions can now continue running in the columnar pipeline. This delivers significantly faster performance for common analytical patterns, with benchmark examples showing roughly **3.5x faster** execution on affected workloads. * **Faster `MIN()`/`MAX()` on text columns:** `MIN()` and `MAX()` on text columns using C collation now run natively in the vectorized aggregation path, avoiding row-based fallback and improving performance for workloads that use text values as tie-breakers or grouping outputs. * **Faster summary-style queries on compressed data** TimescaleDB 2.26 enables a new fast path for common aggregate and "latest/earliest value" queries on compressed chunks. * **`ColumnarIndexScan` enabled by default:** First introduced in 2.25.0 and now enabled by default in 2.26, this execution path allows queries using `COUNT`, `MIN`, `MAX`, and partial `FIRST`/`LAST` to read directly from sparse min/max metadata instead of decompressing full batches. This substantially improves performance for common summary queries on the columnstore, with benchmark examples showing speedups of up to **70x+**. * **Better filtering and UPSERT performance with composite bloom filters** This release improves how TimescaleDB prunes compressed batches when queries or conflict checks involve multiple columns. * Composite bloom filters: Multi-column predicates can now be pushed down directly into compressed scans for both `SELECT` and `UPSERT` operations. * This helps avoid unnecessary decompression, improving efficiency for workloads with selective multi-column lookups and conflict-heavy writes. * Benchmark examples show **2x+ faster** performance when composite bloom filters apply. * `EXPLAIN` plans now surface more details to help you understand pruning effectiveness. * **Smarter chunk exclusion** This release improves query planning so TimescaleDB can skip more irrelevant chunks earlier. * **Runtime chunk exclusion on nested loop joins:** Chunk exclusion now applies on the inner side of nested loop joins, reducing unnecessary scans for join-heavy workloads. * **Chunk exclusion for `IN`/`ANY` on open time dimensions:** Queries using these predicate patterns can now benefit from pruning as well, reducing wasted work on large datasets. * **Quality of life and operability** * **Safer extension re-creation in-session:** `CREATE EXTENSION timescaledb` now works correctly after `DROP EXTENSION` in the same session. * **More reliable chunk creation in replication edge cases:** Fixes a failure mode tied to replica identity invalidation. * **Improved background worker reliability:** Advisory locks were removed from background worker job coordination and replaced with graceful cancellation logic, reducing contention and deadlock risk under concurrency. * **Internal catalog cleanup:** Removes a dropped column from `_timescaledb_catalog.chunk`; if you query internal catalog tables directly, update any dependencies. For complete details, refer to the [TimescaleDB 2.26.0 release notes](https://github.com/timescale/timescaledb/releases/tag/2.26.0). ## New performance graphs in Metrics and Insights [Section titled “New performance graphs in Metrics and Insights”](#new-performance-graphs-in-metrics-and-insights) March 31, 2026 ✨ New Feature [#](#new-performance-graphs-in-metrics-and-insights) ## New performance graphs in Metrics and Insights Several new graphs are now available to help you monitor database performance and resource usage over time. In `Metrics`, a new `Queries per Second` graph gives you a real-time view of your database's throughput, making it easier to spot unexpected spikes or drops in query volume. ![Queries per second in Tiger Cloud](/docs/_astro/queries-per-second-tiger-cloud.BdpulxpZ_ZCpKXJ.webp) In `Insights`, the query deep dive page now includes resource consumption metrics — CPU, memory, and storage IO (read and write) — tracked over time. Use this to identify whether a query's performance is improving or degrading, and to understand the downstream impact of your queries on overall system health. ![Resource consumption metrics in Tiger Cloud](/docs/_astro/resource-consumption-metrics-tiger-cloud.BfLtWHmY_1kagqF.webp) See [Metrics](/docs/deploy/tiger-cloud/tiger-cloud-aws/monitoring#metrics) and [Insights](/docs/deploy/tiger-cloud/tiger-cloud-aws/monitoring#insights) for more information. ## pg\_textsearch v1.0.0 (GA) [Section titled “pg\_textsearch v1.0.0 (GA)”](#pg_textsearch-v100-ga) March 31, 2026 ✨ New Feature 🏎️ Performance [#](#pg-textsearch-v1-0-0-ga) ## pg\_textsearch v1.0.0 (GA) **pg\_textsearch v1.0.0** is **generally available** and **production-ready** on Tiger Cloud. This release marks BM25 full-text search as supported for production workloads, with documentation and operations guidance aligned to GA. This includes implicit `<@>` query syntax, `bm25_force_merge()` for segment consolidation, expanded configuration and limitations, and PostgreSQL 17–18 compatibility. **Learn more:** * [Optimize full text search with BM25](/docs/deploy/tiger-cloud/tiger-cloud-aws/tiger-cloud-extensions/pg-textsearch) * [pg\_textsearch v1.0.0 release notes](https://github.com/timescale/pg_textsearch/releases/tag/v1.0.0) ## Additional metrics available for export [Section titled “Additional metrics available for export”](#additional-metrics-available-for-export) March 24, 2026 ✨ New Feature 🔌 Integration [#](#additional-metrics-available-for-export) ## Additional metrics available for export You can now enable the export of database metrics such as replication, cache usage, and background activity to Amazon CloudWatch, Datadog, and Prometheus by selecting `PostgreSQL metrics` when creating or modifying an exporter in Tiger Console. Additionally, system-level disk metrics such as IO and throughput are now being exported by default. See [Exported metrics](/docs/integrate/observability-alerting/exported-metrics) for the full list of default and additional metrics, and [Observability & Alerting](/docs/integrate/observability-alerting) for how to create exporters. ## Support cases in Tiger Console [Section titled “Support cases in Tiger Console”](#support-cases-in-tiger-console) March 23, 2026 ✨ New Feature [#](#support-cases-in-tiger-console) ## Support cases in Tiger Console You can now view and manage your support cases directly in Tiger Console. Open the `Support` tab at the project level to: * View all support cases for your project from the last 90 days * Read the full email conversation thread * Reply to open cases * Close open cases ![Manage support in Tiger Cloud](/docs/_astro/tiger-cloud-manage-support.oAio5bba_Z1Ot3bO.webp) ## Support for Azure Monitor [Section titled “Support for Azure Monitor”](#support-for-azure-monitor) March 17, 2026 ✨ New Feature 🔌 Integration [#](#support-for-azure-monitor) ## Support for Azure Monitor Tiger Cloud now offers support for exporting telemetry data from your Tiger Cloud services with the time-series and analytics capability enabled to Azure Monitor. To learn more, see [Integrate Azure Monitor with Tiger Cloud](/docs/integrate/observability-alerting/azure-monitor). ## Automated chunk tuning (beta) [Section titled “Automated chunk tuning (beta)”](#automated-chunk-tuning-beta) March 6, 2026 ✨ New Feature 🧪 Beta [#](#automated-chunk-tuning-beta) ## Automated chunk tuning (beta) Tiger Cloud now offers automated chunk tuning in beta, taking the guesswork out of setting and maintaining chunk intervals for your hypertables. When enabled, the tuner monitors your workload and adjusts chunk intervals automatically, no manual intervention required. * **Per-hypertable opt-in:** disabled by default, automated chunk tuning can be enabled for individual hypertables in the Explorer. * **Recommendations for all users:** even without enabling automated tuning, you can view the recommended chunk interval for any hypertable in the Explorer. * **Gradual, safe adjustments:** the tuner moves chunk intervals incrementally to avoid disruptive jumps, and will never increase an interval beyond your configured compression lookback. * **Activity log integration:** chunk interval changes are recorded in the Activity log for a full audit trail. * **No instance overhead:** all calculations run outside your service via Schemata. ![Chunk auto-tuning in Tiger Cloud](/docs/_astro/automated-chunk-tuning.fHKpGxep_bJbUP.webp) If you experience any negative impact, you can opt out at any time and manually reset the interval via the UI or SQL. [Learn more](/docs/build/performance-optimization/improve-hypertable-performance#optimize-hypertable-chunk-intervals) about automated chunk tuning. ## TimescaleDB v2.25 now on Tiger Cloud [Section titled “TimescaleDB v2.25 now on Tiger Cloud”](#timescaledb-v225-now-on-tiger-cloud) March 5, 2026 ✨ New Feature 🏎️ Performance [#](#timescaledb-v2-25-now-on-tiger-cloud) ## TimescaleDB v2.25 now on Tiger Cloud TimescaleDB v2.25 continues improving how PostgreSQL scales for time-series workloads, with major wins in query performance on compressed data, smoother continuous aggregate refresh behavior, and practical operational improvements as datasets and chunk counts grow. TimescaleDB 2.25 was released on January 29, 2026, and is now available to all users on Tiger Cloud. ### Release highlights [Section titled “Release highlights”](#release-highlights) This release includes the following highlights: * **Faster queries on compressed data** A new `ColumnarIndexScan` execution path uses columnar metadata to accelerate common aggregates and `FIRST`/`LAST` queries on compressed chunks. The feature is currently disabled by default due to a known issue with partial aggregations, but can be enabled by setting the GUC `enable_columnarindexscan` to `true`. It will be enabled by default in 2.26. * **`MIN`/`MAX`/`FIRST`/`LAST` on compressed data** can take fast paths and avoid scanning full chunks (reported up to **289× faster** in the example). * **`COUNT(*)` with time filters** can often skip reading the time column entirely (reported up to **50× faster** in the example), reducing CPU and memory pressure. * **Continuous aggregates** This release reduces contention and refresh overhead for continuous aggregates, especially when using the columnstore. * **Direct compress on continuous aggregate refresh (GUC, off by default)** writes refresh results directly to the columnstore, bypassing the rowstore step that can cause policy contention and temporary storage overhead. This also reduces WAL activity, and will be enabled by default in an upcoming release. * **Direct batch delete** restores a key columnstore optimization for `DELETE`s on compressed hypertables: when conditions allow, it can delete whole batches without decompressing and deleting row-by-row. Previously, this path was disabled for tables with continuous aggregates because it didn't emit the invalidation details needed to trigger the right refresh ranges. In 2.25, that support is added, so hypertables using the columnstore and continuous aggregates can benefit from faster, lower-I/O deletes. * **Smaller, steadier refresh transactions by default**: continuous aggregate refreshes can be broken into smaller "batches," where each batch refreshes a portion of the overall time range. This keeps materialization results (and transactions) smaller, reducing stress on the service and improving predictability under heavy refresh/backfill patterns. The default `buckets_per_batch` is now **10** (previously **1**, effectively no batching). * **Safer defaults for continuous aggregates using the columnstore:** we've seen cases where columnstore-backed CAggs can time out during refresh due to suboptimal defaults. This release removes the automatic assignment of `segmentby` for columnstore CAggs (effectively using no `segmentby`) and sets the `orderby` column to the bucketing timestamp to improve refresh behavior. * **Quality of life/operability** * **Estimate original size for columnstore chunks** to support workflows that depend on pre-compression sizing (for example, tiering/billing and Console reporting), especially as direct compress becomes more common and raw-size stats may not exist. This also improves accuracy versus older "frozen" compression stats that didn't reflect subsequent DML (like deletes). Example: ```sql SELECT _timescaledb_functions.estimate_uncompressed_size('.'); ``` * **Option to auto-add newly created chunks to a publication** (GUC, off by default) to simplify logical replication workflows. * **Configurable `work_mem` for background worker jobs** when jobs need more memory to avoid spooling to disk. You can set it per job via the job `config` JSON. Example: ```sql SELECT add_job('my_proc', '1 hour', config => '{"work_mem": "256MB"}'::jsonb); ``` Update an existing job: ```sql SELECT alter_job(id, config := jsonb_set(config,'{work_mem}','"256MB"')) FROM _timescaledb_catalog.bgw_job WHERE id = ; ``` For complete details, refer to the [TimescaleDB 2.25 release notes](https://github.com/timescale/timescaledb/releases/tag/2.25.0). ## CLI and MCP walkthrough in Tiger Console [Section titled “CLI and MCP walkthrough in Tiger Console”](#cli-and-mcp-walkthrough-in-tiger-console) March 4, 2026 ✨ New Feature [#](#cli-and-mcp-walkthrough-in-tiger-console) ## CLI and MCP walkthrough in Tiger Console We have added a new panel in Tiger Console that walks you through setting up: * the Tiger Cloud command line interface (CLI) for easily controlling your project from your terminal * the Tiger Data MCP server for adding functionality to your AI building tools There are walkthroughs for MacOS, Linux, and Windows. Find the actions in the project view under the `CLI/MCP` tab. ![Tiger Cloud CLI/MCP walkthrough](/docs/_astro/cli-mcp-walkthrough-tiger-console.BnV9UKe2_2oBHfe.webp) ## Jobs timeline view [Section titled “Jobs timeline view”](#jobs-timeline-view) March 2, 2026 ✨ New Feature [#](#jobs-timeline-view) ## Jobs timeline view We have added a timeline view to the `Jobs` page so you can easily see the status of all of your recent job runs at a glance. Hover over each run for more detail, or click on a specific job to go into the deep dive jobs view. See [Monitor your Tiger Cloud services](/docs/deploy/tiger-cloud/tiger-cloud-aws/monitoring#jobs). ![Tiger Cloud jobs timeline](/docs/_astro/tiger-console-jobs-timeline-view.B9LcH-xP_Z2ii5eF.webp) ## Azure Marketplace automation and UI updates [Section titled “Azure Marketplace automation and UI updates”](#azure-marketplace-automation-and-ui-updates) February 18, 2026 ✨ New Feature 💡 Improvement [#](#azure-marketplace-automation-and-ui-updates) ## Azure Marketplace automation and UI updates ### Azure Marketplace automation [Section titled “Azure Marketplace automation”](#azure-marketplace-automation) Microsoft customers can quickly sign up for Tiger Cloud through Azure Marketplace, enabling streamlined procurement and consolidated billing. Previously the signup process was manual, requiring help from us at Tiger Data. Now it's completely self-serve! You can find the product under the annual commit and pay as you go pricing options at the marketplace. ![Azure marketplace Tiger Cloud entry](/docs/_astro/azure-marketplace-automation-1.DBewpA2P_WpPp4.webp)![Azure marketplace Tiger Cloud integration](/docs/_astro/azure-marketplace-automation-3.egoI1tPC_giPcR.webp) ### Floating SQL editor [Section titled “Floating SQL editor”](#floating-sql-editor) We have changed the interface for the SQL editor to be floating, as opposed to a dedicated tab. Now SQL editor follows you as you navigate around the UI within an individual service. This change improves the usability and allows you to easily reference your schema (through Explorer) when writing SQL queries. ![Floating SQL editor](/docs/_astro/floating-sql-editor-tiger-cloud.BY0zI1Ac_OPpBS.webp) ### Data/Ops view toggle added at the service level [Section titled “Data/Ops view toggle added at the service level”](#dataops-view-toggle-added-at-the-service-level) We brought back the toggle between the Ops view and Data view within the context of a service. You can now easily switch to Data view from inside an individual service. The toggle is available above the top navigation in the UI. ![View toggle](/docs/_astro/tiger-cloud-view-toggle.VAwzX9p3_2fbxO.webp) ## pg\_textsearch v0.5.0 [Section titled “pg\_textsearch v0.5.0”](#pg_textsearch-v050) February 13, 2026 ✨ New Feature [#](#pg-textsearch-v0-5-0) ## pg\_textsearch v0.5.0 pg\_textsearch v0.5.0 is now available on Tiger Cloud. This release includes the following highlights: * Parallel index builds: `CREATE INDEX` now uses multiple workers for faster indexing of large tables. PostgreSQL automatically allocates workers based on the table size and the `max_parallel_maintenance_workers` setting. * Improvements for BM25 indexes on hypertables. * Stability fixes. See [Optimize full text search with BM25](/docs/deploy/tiger-cloud/tiger-cloud-aws/tiger-cloud-extensions/pg-textsearch) for how to use it. ## Tiered storage on Microsoft Azure [Section titled “Tiered storage on Microsoft Azure”](#tiered-storage-on-microsoft-azure) February 11, 2026 ✨ New Feature [#](#tiered-storage-on-microsoft-azure) ## Tiered Storage on Microsoft Azure Tiered Storage is now available for Tiger Cloud services running on Microsoft Azure, bringing cost-effective data management to our Azure customers. This feature enables you to automatically move rarely accessed data to low-cost storage on Azure Blob Storage while maintaining the ability to query it seamlessly with standard SQL. Customers typically see a reduction of 2-5x in storage costs depending on data compression rates. Azure Tiered Storage works just like the AWS version: enable it in Tiger Console, set tiering policies on your hypertables, and Tiger Cloud handles the rest. With this release, Tiger Cloud Services on Azure offer the same powerful data lifecycle management capabilities as those running on AWS. ## Europe (Zurich) is now available [Section titled “Europe (Zurich) is now available”](#europe-zurich-is-now-available) February 11, 2026 🌍 Availability [#](#europe-zurich-is-now-available) ## Europe (Zurich) is now available Starting today, you can use Tiger Cloud in the AWS Europe (Zurich) Region. This enables applications to have low-latency access to Tiger Cloud services while meeting data residency requirements. To create your first service, see [Get started with Tiger Data](https://www.tigerdata.com/docs/getting-started/latest). For a complete list of regional availability, see [available regions](https://www.tigerdata.com/docs/about/latest/supported-platforms#available-regions). ## pg\_textsearch improvements (v0.3.0 and v0.4.0) [Section titled “pg\_textsearch improvements (v0.3.0 and v0.4.0)”](#pg_textsearch-improvements-v030-and-v040) January 16, 2026 🏎️ Performance [#](#pg-textsearch-improvements-v0-3-0-and-v0-4-0) ## pg\_textsearch improvements (v0.3.0 and v0.4.0) Tiger Cloud now includes significant improvements to `pg_textsearch`, bringing major gains in query performance, index size, and scalability. **What's new:** * **Block MAX-WAND ranked search (v0.3.0):** Introduces the Block MAX-WAND algorithm for ranked keyword search, delivering substantial performance improvements. Query performance is now competitive with the fastest PostgreSQL-based search solutions. * **Posting-list compression (v0.4.0):** Reduces index sizes by 40% or more, making `pg_textsearch` indexes smaller and more efficient. * **Improved partition handling (v0.4.0):** Fixes and stability improvements for indexes on tables with large numbers of partitions. Later releases added **parallel index builds** (v0.5.0) and further refinements; see the [pg\_textsearch changelog](https://github.com/timescale/pg_textsearch/blob/main/CHANGELOG.md) and [v1.0.0 GA](/docs/deploy/tiger-cloud/tiger-cloud-aws/tiger-cloud-extensions/pg-textsearch) for current status. **Learn more:** * [pg\_textsearch v0.3.0 release notes](https://github.com/timescale/pg_textsearch/releases/tag/v0.3.0) * [pg\_textsearch v0.4.0 release notes](https://github.com/timescale/pg_textsearch/releases/tag/v0.4.0) ## PostgreSQL 18 support [Section titled “PostgreSQL 18 support”](#postgresql-18-support) January 13, 2026 ✨ New Feature [#](#postgresql-18-support) ## PostgreSQL 18 support Tiger Cloud now supports **PostgreSQL 18**. All new services are created with PostgreSQL 18 by default, and existing services can upgrade to PostgreSQL 18. **PostgreSQL 18 highlights include:** * **Asynchronous I/O (AIO)** for significantly faster read-heavy workloads * **Faster, less disruptive major upgrades** with improved `pg_upgrade` capabilities * **Virtual generated columns** and the **`uuidv7()`** function for better UUID handling * **Query performance improvements**, including expanded index usage and parallel GIN index builds * **Security enhancements** including OAuth 2.0 authentication support For more details about PostgreSQL 18, see the [official announcement](https://www.postgresql.org/about/news/postgresql-18-released-3142/). ## Terraform support for S3 source connectors and pg\_textsearch update [Section titled “Terraform support for S3 source connectors and pg\_textsearch update”](#terraform-support-for-s3-source-connectors-and-pg_textsearch-update) January 9, 2026 🔌 Integration [#](#terraform-support-for-s3-source-connectors-and-pg-textsearch-update) ## Terraform Support for S3 Source Connectors and pg\_textsearch update ### Terraform support for S3 source connectors [Section titled “Terraform support for S3 source connectors”](#terraform-support-for-s3-source-connectors) We've added a new `timescale_connector_s3` resource to the Tiger Data Terraform provider, enabling full Infrastructure as Code management of S3 source connectors. Teams can now declaratively create, update, and manage S3 connectors supporting CSV and Parquet files, multiple auth methods, and configurable sync options. Available in [Terraform provider v2.7.0+](https://releases.hashicorp.com/terraform-provider-archive). ### pg\_textsearch v0.2.0 released and on Tiger Cloud [Section titled “pg\_textsearch v0.2.0 released and on Tiger Cloud”](#pg_textsearch-v020-released-and-on-tiger-cloud) The newest version of pg\_textsearch has automated benchmark infrastructure, groundwork for storage and query optimizations in upcoming releases, and numerous bugfixes. See more details in the [pg\_textsearch GitHub release notes](https://github.com/timescale/pg_textsearch/releases/tag/v0.2.0). ## Activity log and TimescaleDB v2.24 [Section titled “Activity log and TimescaleDB v2.24”](#activity-log-and-timescaledb-v224) December 12, 2025 ✨ New Feature 💡 Improvement [#](#activity-log-and-timescaledb-v2-24) ## Activity log and TimescaleDB v2.24 ### Activity log [Section titled “Activity log”](#activity-log) Tiger Cloud Console now offers the Activity tab that displays the activity log for all your services. This serves as a record of actions that have happened to your services and Tiger Cloud account, such as service resizes and project invitations. The activity log includes the corresponding service, the user who performed the action, and a description of the action itself. ![Activity log on Tiger Cloud](/docs/_astro/activity-log.BjZOKP7B_17Id74.webp) ### TimescaleDB 2.24 [Section titled “TimescaleDB 2.24”](#timescaledb-224) TimescaleDB 2.24 delivers more efficient recompression operations, expanded use of continuous aggregates, and better invalidation behavior. **Highlighted features:** * **In-memory recompression:** A new `recompress := true` option for `convert_to_columnstore()` performs batch compaction entirely in memory, offering 4–5x faster performance compared to the previous spill-to-disk method. * **Bloom filters on ARM-based Tiger Cloud services:** Bloom filters return with corrected hashing support for the ARM architecture. * **Continuous aggregate updates:** Smarter invalidation range capping, direct compress invalidation support, and UUIDv7 support for continuous aggregates. For complete details, see the [TimescaleDB 2.24 release notes](https://github.com/timescale/timescaledb/releases/tag/2.24.0). ## New navigation in Console [Section titled “New navigation in Console”](#new-navigation-in-console) December 5, 2025 💡 Improvement [#](#new-navigation-in-console) ## New navigation in Console We have updated the design of the navigation in Console to improve consistency, reduce distractions, and make it easier and faster to navigate through the different menus. ![New console navigation](/docs/_astro/console-new-navigation.C3bb9RpJ_Z1yU9RU.webp) ## S3 connector GA and TigerLake public beta [Section titled “S3 connector GA and TigerLake public beta”](#s3-connector-ga-and-tigerlake-public-beta) November 28, 2025 🎉 GA 🧪 Beta [#](#s3-connector-ga-and-tigerlake-public-beta) ## S3 connector GA and TigerLake public beta ### S3 source connector general availability [Section titled “S3 source connector general availability”](#s3-source-connector-general-availability) The S3 source connector is now production-ready, delivering major improvements in reliability, performance, and correctness across the entire ingestion pipeline. ### TigerLake public beta [Section titled “TigerLake public beta”](#tigerlake-public-beta) TigerLake is now available in public beta with full DML support, high-performance ingestion, enhanced resilience and self-healing, and improved deployment experience. Read the [documentation](https://www.tigerdata.com/docs/use-timescale/latest/tigerlake) to get started. ## TimescaleDB v2.23 – improved getting started, automatic columnstore, and faster continuous aggregate invalidation tracking [Section titled “TimescaleDB v2.23 – improved getting started, automatic columnstore, and faster continuous aggregate invalidation tracking”](#timescaledb-v223--improved-getting-started-automatic-columnstore-and-faster-continuous-aggregate-invalidation-tracking) November 14, 2025 ✨ New Feature ⚠️ Deprecation [#](#timescaledb-v2-23-improved-getting-started-automatic-columnstore-and-faster-continuous-aggregate-invalidation-tracking) ## TimescaleDB v2.23 – improved getting started, automatic columnstore, and faster continuous aggregate invalidation tracking ### Highlighted features in TimescaleDB v2.23.0 [Section titled “Highlighted features in TimescaleDB v2.23.0”](#highlighted-features-in-timescaledb-v2230) * Simplified hypertable creation with automatic selection of all parameters * UUIDv7 compression enabled by default (\~30% storage savings and up to 2x faster queries) * Direct-to-Columnstore (tech preview) with INSERT support * Relaxed locking for chunk merging * Unlogged hypertables for improved performance when durability is not required * Continuous aggregates improvements with 10–20% faster DML performance ### Deprecations [Section titled “Deprecations”](#deprecations) * PostgreSQL 15 deprecation: Support will be removed after June 2026 * WAL-based invalidation will be sunset in upcoming releases For more details on simplified hypertable creation, see [Create a hypertable](https://www.tigerdata.com/docs/api/latest/hypertable/create_table/#samples). To learn more about Direct-to-Columnstore, see the [documentation](https://www.tigerdata.com/docs/use-timescale/latest/write-data/insert/#direct-compress-on-insert) and this [blog post](https://www.tigerdata.com/blog/introducing-direct-compress-up-to-40x-faster-leaner-data-ingestion-for-developers-tech-preview). For currently supported TimescaleDB versions, see [Supported platforms](https://www.tigerdata.com/docs/about/latest/supported-platforms/#postgres-timescaledb-support-matrix). For a comprehensive list of changes, see the [TimescaleDB 2.23 release notes](https://github.com/timescale/timescaledb/blob/main/CHANGELOG.md#2230-2025-10-29). ## S3 source connector and crypto payments [Section titled “S3 source connector and crypto payments”](#s3-source-connector-and-crypto-payments) October 31, 2025 ✨ New Feature 🔐 Security [#](#s3-source-connector-and-crypto-payments) ## S3 source connector and crypto payments ### Crypto payments: early access! [Section titled “Crypto payments: early access!”](#crypto-payments-early-access) You can now pay Tiger Cloud invoices with stablecoins (USDC, USDP, USDG) through Stripe's crypto payments. ![Crypto payments](/docs/_astro/tiger-cloud-crypto-payment.DKnIzw9Y_25w5sk.webp) ### Detailed S3 source connector progress screen [Section titled “Detailed S3 source connector progress screen”](#detailed-s3-source-connector-progress-screen) Major improvements to the S3 source connector with better observability including cumulative summary, search, detailed file statuses, filtering, bulk retry, lifecycle history, and auto-refresh. ![S3 connector stats](/docs/_astro/tiger-console-s3-connector-import-details.DbkXHMp-_1BMtgk.webp) ## AI and Tiger Cloud major changes! [Section titled “AI and Tiger Cloud major changes!”](#ai-and-tiger-cloud-major-changes) October 24, 2025 ✨ New Feature [#](#ai-and-tiger-cloud-major-changes) ## AI and Tiger Cloud major changes! ### Free pricing plan and free services [Section titled “Free pricing plan and free services”](#free-pricing-plan-and-free-services) New Free plan with up to two free services for prototyping. ### Tiger CLI [Section titled “Tiger CLI”](#tiger-cli) New CLI tool to control Tiger Cloud from the terminal for signing up, spinning up services, and executing SQL. ### Tiger MCP for PostgreSQL and TimescaleDB [Section titled “Tiger MCP for PostgreSQL and TimescaleDB”](#tiger-mcp-for-postgresql-and-timescaledb) New MCP server enabling AI agents to interact with the database with built-in master prompts and high-level tools. ### Fluid storage [Section titled “Fluid storage”](#fluid-storage) New distributed storage layer for instant forks, snapshots, and automatic scaling with over 100,000 IOPS throughput. ### pg\_textsearch extension preview [Section titled “pg\_textsearch extension preview”](#pg_textsearch-extension-preview) New extension enabling BM25 on PostgreSQL for significant improvements in text search. ## TimescaleDB 2.22.1 – configurable indexing, enhanced partitioning, and faster queries [Section titled “TimescaleDB 2.22.1 – configurable indexing, enhanced partitioning, and faster queries”](#timescaledb-2221--configurable-indexing-enhanced-partitioning-and-faster-queries) October 10, 2025 🏎️ Performance ✨ New Feature [#](#timescaledb-2-22-1-configurable-indexing-enhanced-partitioning-and-faster-queries) ## TimescaleDB 2.22.1 – configurable indexing, enhanced partitioning, and faster queries ### Highlighted features [Section titled “Highlighted features”](#highlighted-features) * **Configurable sparse indexes:** Manually configure sparse indexes on compressed hypertables * **UUIDv7 support:** Native support for UUIDv7 for compression and partitioning * **Multi-column SkipScan:** Support for multiple distinct keys with millisecond-fast deduplication and `DISTINCT ON` queries across billions of rows. Learn more in our [blog post](https://www.tigerdata.com/blog/skipscan-in-timescaledb-why-distinct-was-slow-how-we-built-it-and-how-you-can-use-it) and [documentation](https://www.tigerdata.com/docs/use-timescale/latest/query-data/skipscan/). * **Compression improvements:** Default `segmentby` and `orderby` settings are now applied at compression time for each chunk, automatically adapting to evolving data patterns for better performance. This was previously set at the hypertable level and fixed across all chunks. For a comprehensive list of changes, refer to the TimescaleDB [2.22](https://github.com/timescale/timescaledb/releases/tag/2.22.0) and [2.22.1](https://github.com/timescale/timescaledb/releases/tag/2.22.1) release notes, as well as [TimescaleDB 2.22.1](https://github.com/timescale/timescaledb/releases). ## Kafka source connector (beta) [Section titled “Kafka source connector (beta)”](#kafka-source-connector-beta) September 19, 2025 🧪 Beta 🔌 Integration [#](#kafka-source-connector-beta) ## Kafka Source Connector (beta) New Kafka Source Connector enables you to connect existing Kafka clusters directly to Tiger Cloud and ingest data from Kafka topics into hypertables. Supports AVRO format and Confluent Cloud and Amazon Managed Streaming for Apache Kafka. Learn more in the [Kafka Source Connector docs](https://www.tigerdata.com/docs/migrate/latest/livesync-for-kafka/). ![Kafka source connector in Tiger Cloud](/docs/_astro/kafka-source-connector-tiger-data.0RSrebOp_ZxV71X.webp)![Kafka source connector streaming in Tiger Cloud](/docs/_astro/kafka-source-connector-streaming.VTY9OdlM_ZteHBr.webp) ## Phased update rollouts, pg\_cron, larger compute options, and backup reports [Section titled “Phased update rollouts, pg\_cron, larger compute options, and backup reports”](#phased-update-rollouts-pg_cron-larger-compute-options-and-backup-reports) September 12, 2025 ✨ New Feature 💡 Improvement [#](#phased-update-rollouts-pg-cron-larger-compute-options-and-backup-reports) ## Phased update rollouts, pg\_cron, larger compute options, and backup reports ### Phased rollouts for TimescaleDB minor releases [Section titled “Phased rollouts for TimescaleDB minor releases”](#phased-rollouts-for-timescaledb-minor-releases) Starting with TimescaleDB 2.22.0, minor releases roll out in phases, dev services first, followed by prod after 21 days. See [Maintenance and upgrades](https://www.tigerdata.com/docs/use-timescale/latest/upgrades/) for details. ### pg\_cron extension [Section titled “pg\_cron extension”](#pg_cron-extension) `pg_cron` is now available on Tiger Cloud for scheduling SQL commands automatically. ### Larger compute options: 48 and 64 CPU [Section titled “Larger compute options: 48 and 64 CPU”](#larger-compute-options-48-and-64-cpu) New compute options with 48 and 64 CPUs available on Enterprise plan. ![CPU options in Tiger Cloud](/docs/_astro/tiger-cloud-cpu-options.BRmYoK88_ZexpNd.webp) ### Backup report for compliance [Section titled “Backup report for compliance”](#backup-report-for-compliance) Scale and Enterprise customers can now view backup lists for compliance needs. ![Backup reports in Tiger Cloud](/docs/_astro/backup-history-tiger-cloud.ks7kQEWt_1TeMS9.webp) ## New data import wizard [Section titled “New data import wizard”](#new-data-import-wizard) September 5, 2025 💡 Improvement [#](#new-data-import-wizard) ## New data import wizard ### Improved import experience [Section titled “Improved import experience”](#improved-import-experience) Cleaner, more intuitive UI for data import with recommended options and clear categories. ![Data import in Tiger Cloud](/docs/_astro/data-import-wizard-in-tiger-cloud.DkQN8EFR_pbL7A.webp) A new data import component has been added to the overview dashboard: ![Overview dashboard in Tiger Cloud](/docs/_astro/service-dashboard-tiger-cloud.qsMYDbTm_ZTG9WY.webp) ## Enhancements to the PostgreSQL source connector [Section titled “Enhancements to the PostgreSQL source connector”](#enhancements-to-the-postgresql-source-connector) August 28, 2025 💡 Improvement [#](#enhancements-to-the-postgresql-source-connector) ## Enhancements to the PostgreSQL source connector ### What's new [Section titled “What's new”](#whats-new) * Easy table selection with dropdown menus * Enhanced metadata sync showing initial data copy progress and change data capture replication lag * Improved UX design with separate sections for initial data copy and change data capture ![Connectors UX](/docs/_astro/connectors-new-ui.CoW2nvhj_Q032p.webp) ## Developer role GA and hypertable transformation in Console [Section titled “Developer role GA and hypertable transformation in Console”](#developer-role-ga-and-hypertable-transformation-in-console) August 21, 2025 🎉 GA ✨ New Feature [#](#developer-role-ga-and-hypertable-transformation-in-console) ## Developer role GA and hypertable transformation in Console ### Developer role (GA) [Section titled “Developer role (GA)”](#developer-role-ga) Project-scoped permission set for technical users to build, operate services, create or modify resources, and use observability. ### Transform a table to a hypertable from the Explorer [Section titled “Transform a table to a hypertable from the Explorer”](#transform-a-table-to-a-hypertable-from-the-explorer) Easy hypertable creation directly from the Explorer for regular PostgreSQL tables. ![Transform a table to a hypertable](/docs/_astro/table_to_hypertable_1.Brdn2xRH_ZVLz2j.webp)![Transform a table to a hypertable](/docs/_astro/table_to_hypertable_2.BZo-eWo3_Z1atsK5.webp) ## Cross-region backups, PostgreSQL options, and onboarding [Section titled “Cross-region backups, PostgreSQL options, and onboarding”](#cross-region-backups-postgresql-options-and-onboarding) August 14, 2025 ✨ New Feature 🔐 Security [#](#cross-region-backups-postgresql-options-and-onboarding) ## Cross-region backups, PostgreSQL options, and onboarding ### Cross-region backups [Section titled “Cross-region backups”](#cross-region-backups) Store backups in a different region than your service for improved resilience. ### Standard PostgreSQL instructions for onboarding [Section titled “Standard PostgreSQL instructions for onboarding”](#standard-postgresql-instructions-for-onboarding) Basic instructions for INSERT, UPDATE, DELETE commands in the Import Data page. ### PostgreSQL-only service type [Section titled “PostgreSQL-only service type”](#postgresql-only-service-type) Option to choose PostgreSQL-only in the service creation flow. ## Viewer role GA, EXPLAIN plans, and chunk index sizes in Explorer [Section titled “Viewer role GA, EXPLAIN plans, and chunk index sizes in Explorer”](#viewer-role-ga-explain-plans-and-chunk-index-sizes-in-explorer) July 31, 2025 🎉 GA 💡 Improvement [#](#viewer-role-ga-explain-plans-and-chunk-index-sizes-in-explorer) ## Viewer role GA, EXPLAIN plans, and chunk index sizes in Explorer ### GA release of the viewer role [Section titled “GA release of the viewer role”](#ga-release-of-the-viewer-role) Read-only access to services, metrics, and logs without modify permissions. ### EXPLAIN plans in Insights [Section titled “EXPLAIN plans in Insights”](#explain-plans-in-insights) Automatically generated EXPLAIN plans for queries taking longer than 10 seconds. ### Chunk index size in Explorer [Section titled “Chunk index size in Explorer”](#chunk-index-size-in-explorer) Find the index size of hypertable chunks in the Explorer. ## TimescaleDB v2.21 and catalog objects in the Console Explorer [Section titled “TimescaleDB v2.21 and catalog objects in the Console Explorer”](#timescaledb-v221-and-catalog-objects-in-the-console-explorer) July 25, 2025 🏎️ Performance ✨ New Feature [#](#timescaledb-v2-21-and-catalog-objects-in-the-console-explorer) ## TimescaleDB v2.21 and catalog objects in the Console Explorer ### TimescaleDB v2.21 [Section titled “TimescaleDB v2.21”](#timescaledb-v221) * **High-scale ingestion performance:** Compress data directly into columnstore during ingestion (1.2M rows/second in tests) * **Faster UPSERTs:** 2.5x faster for heavily constrained tables * **Faster deletion:** 42x faster DELETE operations on non-segmentby columns * **Enhanced continuous aggregates:** Concurrent refresh policies and batched refreshes enabled by default For a comprehensive list of changes, see the [TimescaleDB v2.21 release notes](https://github.com/timescale/timescaledb/releases/tag/2.21.0). ### Catalog objects in the Console Explorer [Section titled “Catalog objects in the Console Explorer”](#catalog-objects-in-the-console-explorer) View catalog objects in the Console Explorer to understand database inner workings. ![Explore catalog objects](/docs/_astro/tiger-cloud-explorer-catalog-objects.B5WkWcUv_1Hhz92.webp) ## Iceberg destination connector (Tiger lake) [Section titled “Iceberg destination connector (Tiger lake)”](#iceberg-destination-connector-tiger-lake) July 18, 2025 🧪 Beta 🔌 Integration [#](#iceberg-destination-connector-tigerlake) ## Iceberg Destination Connector (TigerLake) ### Beta availability [Section titled “Beta availability”](#beta-availability) Beta Iceberg destination connector enables Scale and Enterprise users to integrate Tiger Cloud with Amazon S3 tables. For setup details, see the [Iceberg destination connector documentation](https://www.tigerdata.com/docs/use-timescale/latest/tigerlake/). ## Console just got better [Section titled “Console just got better”](#console-just-got-better) July 11, 2025 💡 Improvement [#](#console-just-got-better) ## Console just got better ### Editable jobs in Console [Section titled “Editable jobs in Console”](#editable-jobs-in-console) Edit jobs directly in Console with custom wizards for Tiger Cloud job types. ![Edit jobs in console](/docs/_astro/console-jobs-edit.Cy5fuIMM_Z18m3p6.webp) ### Connection history [Section titled “Connection history”](#connection-history) View historical connection counts in the Connections tab. ![View connection history in console](/docs/_astro/console-connection-history.BU7uzpN7_1lPHs6.webp) ### Read-Only access through RBAC (Public beta) [Section titled “Read-Only access through RBAC (Public beta)”](#read-only-access-through-rbac-public-beta) New viewer-only permissions for users who need visibility without modify access. ## Super useful doc updates [Section titled “Super useful doc updates”](#super-useful-doc-updates) July 4, 2025 💡 Improvement [#](#super-useful-doc-updates) ## Super useful doc updates ### Updates to livesync instructions [Section titled “Updates to livesync instructions”](#updates-to-livesync-instructions) Clarified step-by-step procedures for setting up livesync with more detailed examples. ### New optional argument for add\_continuous\_aggregate\_policy API [Section titled “New optional argument for add\_continuous\_aggregate\_policy API”](#new-optional-argument-for-add_continuous_aggregate_policy-api) Added the `refresh_newest_first` argument to control incremental refresh order. ## Multi-command queries in SQL editor, improved job page experience, multiple AWS Transit gateways, and a new service creation flow [Section titled “Multi-command queries in SQL editor, improved job page experience, multiple AWS Transit gateways, and a new service creation flow”](#multi-command-queries-in-sql-editor-improved-job-page-experience-multiple-aws-transit-gateways-and-a-new-service-creation-flow) June 20, 2025 ✨ New Feature 💡 Improvement [#](#multi-command-queries-in-sql-editor-improved-job-page-experience-multiple-aws-transit-gateways-and-a-new-service-creation-flow) ## Multi-command queries in SQL editor, improved job page experience, multiple AWS Transit Gateways, and a new service creation flow ### Run multiple statements in SQL editor [Section titled “Run multiple statements in SQL editor”](#run-multiple-statements-in-sql-editor) Execute complex queries with multiple commands in a single run. ### Branch conversations in SQL assistant [Section titled “Branch conversations in SQL assistant”](#branch-conversations-in-sql-assistant) Start new discussion threads from any point in SQL assistant chat. ### Smarter results table [Section titled “Smarter results table”](#smarter-results-table) * Expand JSON data instantly into readable columns * Smart filters with dropdown options ### Jobs page improvements [Section titled “Jobs page improvements”](#jobs-page-improvements) Individual job pages now display their configuration for TimescaleDB job types. ### Multiple AWS Transit gateways [Section titled “Multiple AWS Transit gateways”](#multiple-aws-transit-gateways) Connect multiple AWS Transit Gateways with overlapping CIDRs. ### Updated service creation flow [Section titled “Updated service creation flow”](#updated-service-creation-flow) Clearer service type options for PostgreSQL extensions. ![Create a new Tiger Cloud service in Tiger Console](/docs/_astro/create-tiger-cloud-service.DR9ESqG0_1WwCkT.webp) ## Improved Terraform support and TimescaleDB v2.20.3 [Section titled “Improved Terraform support and TimescaleDB v2.20.3”](#improved-terraform-support-and-timescaledb-v2203) June 13, 2025 🔌 Integration 💡 Improvement [#](#improved-terraform-support-and-timescaledb-v2-20-3) ## Improved Terraform support and TimescaleDB v2.20.3 ### Terraform support [Section titled “Terraform support”](#terraform-support) Latest version adds support for creating and attaching observability exporters and AWS Transit Gateway configuration. Check the [Timescale Terraform provider documentation](https://registry.terraform.io/providers/timescale/timescale/latest/docs) for more details. ### TimescaleDB v2.20.3 [Section titled “TimescaleDB v2.20.3”](#timescaledb-v2203) Patch release with bug fixes including SkipScan costing adjustments and dump/restore operation fixes. For a comprehensive list of changes, see the [TimescaleDB 2.20.3 release notes](https://github.com/timescale/timescaledb/releases/tag/2.20.3). ## Read replica sets, faster tables, new anthropic models, and VPC support in data mode [Section titled “Read replica sets, faster tables, new anthropic models, and VPC support in data mode”](#read-replica-sets-faster-tables-new-anthropic-models-and-vpc-support-in-data-mode) June 6, 2025 ✨ New Feature 🏎️ Performance [#](#read-replica-sets-faster-tables-new-anthropic-models-and-vpc-support-in-data-mode) ## Read replica sets, faster tables, new anthropic models, and VPC support in data mode ### Horizontal read scaling with read replica sets [Section titled “Horizontal read scaling with read replica sets”](#horizontal-read-scaling-with-read-replica-sets) [Read replica sets](https://docs.timescale.com/use-timescale/latest/ha-replicas/read-scaling/) let you scale reads horizontally by creating up to 10 replica nodes behind a single read endpoint. ![Create a read replica set in Timescale Console](/docs/_astro/create-read-replica-set-timescale-console.B66ju_Nt_ZlGNTh.webp) ### Faster, smarter results tables in data mode [Section titled “Faster, smarter results tables in data mode”](#faster-smarter-results-tables-in-data-mode) Completely rebuilt query results display with smooth scrolling, custom settings preservation, and better filtering. ### Latest anthropic models added to SQL assistant [Section titled “Latest anthropic models added to SQL assistant”](#latest-anthropic-models-added-to-sql-assistant) Support for Sonnet 4, Sonnet 4 (extended thinking), Opus 4, and Opus 4 (extended thinking). ### VPC support for passwordless data mode connections [Section titled “VPC support for passwordless data mode connections”](#vpc-support-for-passwordless-data-mode-connections) Expanded passwordless data mode functionality to services using a VPC. ## Enhanced service monitoring, TimescaleDB v2.20, and livesync for PostgreSQL [Section titled “Enhanced service monitoring, TimescaleDB v2.20, and livesync for PostgreSQL”](#enhanced-service-monitoring-timescaledb-v220-and-livesync-for-postgresql) May 30, 2025 🏎️ Performance ✨ New Feature ⚠️ Deprecation [#](#enhanced-service-monitoring-timescaledb-v2-20-and-livesync-for-postgresql) ## Enhanced service monitoring, TimescaleDB v2.20, and livesync for PostgreSQL ### Updated top-level navigation - monitoring tab [Section titled “Updated top-level navigation - monitoring tab”](#updated-top-level-navigation---monitoring-tab) Consolidated service information tabs into a single Monitoring tab. ![Insights](/docs/_astro/insights_overview_timescale.BhuLGBUI_1YYAD2.webp) ### Monitor active connections [Section titled “Monitor active connections”](#monitor-active-connections) View information about active connections including queries, application names, and duration. ![Connections](/docs/_astro/console-monitoring-connections.DHoCoklS_ZlvtQh.webp) ### TimescaleDB v2.20 [Section titled “TimescaleDB v2.20”](#timescaledb-v220) All new services use TimescaleDB v2.20 with significant performance improvements: * Handle data updates and upserts up to 10x faster * Up to 6x faster point queries with new bloom filters * Up to 2500x faster DISTINCT operations with SkipScan * 8x more efficient Boolean column storage For details, see the [TimescaleDB v2.20 release notes](https://github.com/timescale/timescaledb/releases/tag/2.20.0). ### PostgreSQL 13 and 14 deprecated on Tiger Cloud [Section titled “PostgreSQL 13 and 14 deprecated on Tiger Cloud”](#postgresql-13-and-14-deprecated-on-tiger-cloud) TimescaleDB 2.20 requires PostgreSQL 15+. Upgrade required by September 15, 2025. ### Enhancements to livesync for PostgreSQL [Section titled “Enhancements to livesync for PostgreSQL”](#enhancements-to-livesync-for-postgresql) * Edit running livesync to add and drop tables * Create multiple livesync instances per service * Dropdown menu for schema and table name selection For setup steps, see [Livesync for PostgreSQL](https://docs.timescale.com/migrate/latest/livesync-for-postgresql/). ## More storage types and IOPS [Section titled “More storage types and IOPS”](#more-storage-types-and-iops) May 22, 2025 ✨ New Feature [#](#more-storage-types-and-iops) ## More storage types and IOPS ### Enhanced storage: scale to 64 TB and 32,000 IOPS [Section titled “Enhanced storage: scale to 64 TB and 32,000 IOPS”](#enhanced-storage-scale-to-64-tb-and-32000-iops) New storage type supporting up to 64 TB capacity and 32,000 IOPS per service, powered by AWS io2 volumes. ![I/O boost in Timescale Cloud](/docs/_astro/io-boost-timescale-cloud.BVtNAdBZ_16DwU.webp) Learn more in the [data tiering documentation](https://docs.timescale.com/use-timescale/latest/data-tiering/enabling-data-tiering/). ## New export and import options [Section titled “New export and import options”](#new-export-and-import-options) May 15, 2025 🔌 Integration ✨ New Feature [#](#new-export-and-import-options) ## New export and import options ### Ship TimescaleDB metrics to Prometheus [Section titled “Ship TimescaleDB metrics to Prometheus”](#ship-timescaledb-metrics-to-prometheus) Prometheus Exporter for Tiger Cloud enables shipping of TimescaleDB metrics with custom Grafana dashboards and alerting. For configuration details, see [metrics to Prometheus](https://docs.timescale.com/use-timescale/latest/metrics-logging/metrics-to-prometheus/). ![Prometheus export user interface](/docs/_astro/timescale-create-prometheus-exporter.BHEEPZEg_ZxycP6.webp) ### Import text files into PostgreSQL tables [Section titled “Import text files into PostgreSQL tables”](#import-text-files-into-postgresql-tables) New import option for local text files for use with Vectorizers, available from **Service → Actions → Import Data** in Timescale Console. ## Automatic document embeddings from S3 and a sample dataset for AI testing [Section titled “Automatic document embeddings from S3 and a sample dataset for AI testing”](#automatic-document-embeddings-from-s3-and-a-sample-dataset-for-ai-testing) May 9, 2025 ✨ New Feature 🔌 Integration [#](#automatic-document-embeddings-from-s3-and-a-sample-dataset-for-ai-testing) ## Automatic document embeddings from S3 and a sample dataset for AI testing ### Automatic document embeddings from S3 [Section titled “Automatic document embeddings from S3”](#automatic-document-embeddings-from-s3) pgai vectorizer now supports automatic document vectorization from S3 with support for PDFs, DOCX, XLSX, HTML, and more. ![create a vectorizer](/docs/_astro/console-create-a-vectorizer.Cs7zTRP2_12gHpR.webp) ### Sample dataset for AI testing [Section titled “Sample dataset for AI testing”](#sample-dataset-for-ai-testing) Import a dataset directly from Hugging Face using Tiger Console. ![hugging face sample data](/docs/_astro/console-import-huggingface-data.DtglKMky_Z1TBsE6.webp) ## Livesync for S3 and passwordless connections for data mode [Section titled “Livesync for S3 and passwordless connections for data mode”](#livesync-for-s3-and-passwordless-connections-for-data-mode) April 25, 2025 🧪 Beta ✨ New Feature [#](#livesync-for-s3-and-passwordless-connections-for-data-mode) ## Livesync for S3 and passwordless connections for data mode ### Livesync for S3 (beta) [Section titled “Livesync for S3 (beta)”](#livesync-for-s3-beta) [Livesync for S3](https://docs.timescale.com/migrate/latest/livesync-for-s3/) syncs data in S3 buckets to a Tiger Cloud service, handling both existing and new data in real time. ![Timescale Console new livesync](/docs/_astro/livesync-s3-start-new-livesync.CQL6Yfwa_bxTQJ.webp)![Timescale Console livesync view status](/docs/_astro/livesync-s3-view-status.BiwCeKxV_2nHCPO.webp) ### UX improvements to livesync for PostgreSQL [Section titled “UX improvements to livesync for PostgreSQL”](#ux-improvements-to-livesync-for-postgresql) Added a detailed two-step checklist with configuration instructions to Tiger Console, building on [livesync for PostgreSQL](https://docs.timescale.com/migrate/latest/livesync-for-postgresql/). ![Livesync PostgreSQL configuration instructions in Tiger Console](/docs/_astro/livesync-postgres-console-config-instuctions.BU_CuHCl_oFUWt.webp) ### Passwordless data mode connections [Section titled “Passwordless data mode connections”](#passwordless-data-mode-connections) All new services created in Tiger Cloud are now automatically accessible from [data mode](https://docs.timescale.com/getting-started/latest/run-queries-from-console/#connect-to-your-timescale-cloud-service-in-the-data-mode) without credentials. ![Timescale Console passwordless data mode](/docs/_astro/data-mode-connections.CPJY0TVX_Zn1Wdq.webp) ## Embeddings spot checks, TimescaleDB v2.19.3, and new models in SQL assistant [Section titled “Embeddings spot checks, TimescaleDB v2.19.3, and new models in SQL assistant”](#embeddings-spot-checks-timescaledb-v2193-and-new-models-in-sql-assistant) April 18, 2025 💡 Improvement ✨ New Feature [#](#embeddings-spot-checks-timescaledb-v2-19-3-and-new-models-in-sql-assistant) ## Embeddings spot checks, TimescaleDB v2.19.3, and new models in SQL Assistant ### Embeddings spot checks [Section titled “Embeddings spot checks”](#embeddings-spot-checks) Quickly check embedding quality from vectorizers' outputs using a simple UI. ![Embedding Quality Inspection](/docs/_astro/ai-spot-checks.DhtQ_Bb8_1e5dIj.webp) ### TimescaleDB v2.19.3 [Section titled “TimescaleDB v2.19.3”](#timescaledb-v2193) New services use TimescaleDB v2.19.3 with bug fixes for columnstore queries and hypercore table access. ### New OpenAI, Llama, and Gemini models [Section titled “New OpenAI, Llama, and Gemini models”](#new-openai-llama-and-gemini-models) SQL Assistant now supports GPT-4.1, Llama 4, and Gemini models. ![SQL Assistant - New Models](/docs/_astro/sql-assistant-new-models.C23zc_TE_Z1S9HTl.webp) ## TimescaleDB v2.19, new service overview page, and log improvements [Section titled “TimescaleDB v2.19, new service overview page, and log improvements”](#timescaledb-v219-new-service-overview-page-and-log-improvements) April 11, 2025 🏎️ Performance 💡 Improvement [#](#timescaledb-v2-19-new-service-overview-page-and-log-improvements) ## TimescaleDB v2.19, new service overview page, and log improvements ### TimescaleDB v2.19 [Section titled “TimescaleDB v2.19”](#timescaledb-v219) * Improved concurrency of INSERT, UPDATE, and DELETE on columnstore * Faster continuous aggregate refreshes with smaller batches * Faster analytical queries with SIMD vectorization * Enable chunk size optimization with merge\_chunk See the [TimescaleDB v2.19 release notes](https://github.com/timescale/timescaledb/releases/tag/2.19.0) for more details. ### New service overview page [Section titled “New service overview page”](#new-service-overview-page) Overhauled service overview with architecture diagram and general information with recommendations. ![New Service Overview page](/docs/_astro/new-timescale-service-overview.BNwv99Oz_Z2grpnI.webp) ### Find logs faster [Section titled “Find logs faster”](#find-logs-faster) Added date, time, and timezone picker for quick log navigation. ![Find logs faster](/docs/_astro/find-logs-faster-timescale-console.D-SL2XSC_1mCJwA.webp) ## Faster vector search and improved job information [Section titled “Faster vector search and improved job information”](#faster-vector-search-and-improved-job-information) April 4, 2025 🏎️ Performance 💡 Improvement [#](#faster-vector-search-and-improved-job-information) ## Faster vector search and improved job information ### pgvectorscale 0.7.0 [Section titled “pgvectorscale 0.7.0”](#pgvectorscale-070) Label-based filtered vector search with filtered indexes for StreamingDiskANN. For details, see the [pgvectorscale 0.7.0 release notes](https://github.com/timescale/pgvectorscale/releases/tag/0.7.0) and the [usage example](https://github.com/timescale/pgvectorscale?tab=readme-ov-file#label-based-filtering-with-diskann). ### Job errors and individual job pages [Section titled “Job errors and individual job pages”](#job-errors-and-individual-job-pages) Each job has individual page with detailed error information for debugging. * Successful jobs: ![Log success in Timescale Console](/docs/_astro/changelog-job-success-page.BkxJXCH6_6pw3H.webp) * Unsuccessful jobs with errors: ![Log errors in Timescale Console](/docs/_astro/changelog-job-error-page.BZvvzk7t_Z2bPEiE.webp) ## In-Console Livesync for PostgreSQL [Section titled “In-Console Livesync for PostgreSQL”](#in-console-livesync-for-postgresql) March 21, 2025 ✨ New Feature [#](#in-console-livesync-for-postgresql) ## In-Console Livesync for PostgreSQL ### Set up in Console [Section titled “Set up in Console”](#set-up-in-console) Set up active data ingestion with [livesync for PostgreSQL](https://docs.timescale.com/migrate/latest/livesync-for-postgresql/) in Tiger Console. ![Livesync in Timescale Console](/docs/_astro/timescale-cloud-livesync-tile.Cgbq_FBz_Z1WsRo7.webp)![Set up Timescale Livesync](/docs/_astro/set-up-timescale-cloud-livesync.DHsrZI57_1Xrx3q.webp)![Select tables for Livesync](/docs/_astro/select-tables-for-timescale-cloud-livesync.CIqfcfCY_9ftJL.webp)![Timescale Livesync running](/docs/_astro/livesync-view-status.DsDu_Sad_ZLmAqn.webp) ## 16K dimensions on pgvectorscale plus new pgai Vectorizer support [Section titled “16K dimensions on pgvectorscale plus new pgai Vectorizer support”](#16k-dimensions-on-pgvectorscale-plus-new-pgai-vectorizer-support) March 14, 2025 ✨ New Feature 🔌 Integration [#](#16k-dimensions-on-pgvectorscale-plus-new-pgai-vectorizer-support) ## 16K dimensions on pgvectorscale plus new pgai Vectorizer support ### pgvectorscale 0.6 [Section titled “pgvectorscale 0.6”](#pgvectorscale-06) Support for vectors with up to 16,000 dimensions. NEON support for SIMD distance calculations on aarch64 processors. See the [pgvectorscale 0.6.0 release notes](https://github.com/timescale/pgvectorscale/releases/tag/0.6.0) for details. ### pgai Vectorizer supports models from AWS Bedrock, Azure AI, Google Vertex via LiteLLM [Section titled “pgai Vectorizer supports models from AWS Bedrock, Azure AI, Google Vertex via LiteLLM”](#pgai-vectorizer-supports-models-from-aws-bedrock-azure-ai-google-vertex-via-litellm) Access embedding models from popular cloud model hubs. ## Agent mode for PopSQL and more [Section titled “Agent mode for PopSQL and more”](#agent-mode-for-popsql-and-more) March 7, 2025 ✨ New Feature 💡 Improvement [#](#agent-mode-for-popsql-and-more) ## Agent Mode for PopSQL and more ### Agent mode for PopSQL [Section titled “Agent mode for PopSQL”](#agent-mode-for-popsql) SQL Assistant automatically adjusts and executes queries without intervention. ![Timescale SQL Assistant Agent Mode](/docs/_astro/timescale-sql-assistant-agent-mode.D0Pu8NGz.gif) ### Improved AWS marketplace integration [Section titled “Improved AWS marketplace integration”](#improved-aws-marketplace-integration) Enhanced AWS Marketplace workflow with full automation and seamless billing integration. ### Tiger Console recommendations [Section titled “Tiger Console recommendations”](#tiger-console-recommendations) Recommendations based on service context for onboarding and service health. ![Timescale Console recommendations](/docs/_astro/timescale-console-recommendations.DsWDxz1v_qhBzJ.webp) ## Configuration options for secure connections and more [Section titled “Configuration options for secure connections and more”](#configuration-options-for-secure-connections-and-more) February 28, 2025 🔐 Security 💡 Improvement [#](#configuration-options-for-secure-connections-and-more) ## Configuration Options for Secure Connections and More ### Edit VPC and AWS Transit Gateway CIDRs [Section titled “Edit VPC and AWS Transit Gateway CIDRs”](#edit-vpc-and-aws-transit-gateway-cidrs) Modify CIDR blocks for VPC or Transit Gateway directly from Tiger Console. ![VPC connection wizard](/docs/_astro/2025-02-27changelog_VPC_transit_gateway.CKv0Ux5r_1MpFy3.webp) ### Improved log filtering [Section titled “Improved log filtering”](#improved-log-filtering) Enhanced Logs screen with Warning and Log filters. ![Logs with filters](/docs/_astro/2025-02-27changelog_log_filtering.DtG4O0-P_1xRbXt.webp) ### TimescaleDB v2.18.2 [Section titled “TimescaleDB v2.18.2”](#timescaledb-v2182) New services use TimescaleDB v2.18.2 with bug fixes. See the [TimescaleDB v2.18.2 release notes](https://github.com/timescale/timescaledb/releases/tag/2.18.2) for details. ### SQL assistant improvements [Section titled “SQL assistant improvements”](#sql-assistant-improvements) Support for Claude 3.7 Sonnet with extended thinking and ability to abort requests. ## SQL assistant improvements and pgai docs reorganization [Section titled “SQL assistant improvements and pgai docs reorganization”](#sql-assistant-improvements-and-pgai-docs-reorganization) February 21, 2025 💡 Improvement 🔌 Integration [#](#sql-assistant-improvements-and-pgai-docs-reorganization) ## SQL Assistant Improvements and Pgai Docs Reorganization ### New models and improved UX for SQL assistant [Section titled “New models and improved UX for SQL assistant”](#new-models-and-improved-ux-for-sql-assistant) Added fireworks.ai and Groq as providers with new LLM options and improved model picker with descriptions. ![Timescale Cloud SQL Assistant AI model picker](/docs/_astro/sql-assistant-ai-models.BaBBXzWO_Z2lfPS6.webp) ### Updated and reorganized docs for pgai [Section titled “Updated and reorganized docs for pgai”](#updated-and-reorganized-docs-for-pgai) Improved GitHub docs with grouped sections and comprehensive summary. See the updated docs on [GitHub](https://github.com/timescale/pgai/tree/main/docs). ## TimescaleDB v2.18.1 and AWS Transit Gateway support generally available [Section titled “TimescaleDB v2.18.1 and AWS Transit Gateway support generally available”](#timescaledb-v2181-and-aws-transit-gateway-support-generally-available) February 14, 2025 🎉 GA 🏎️ Performance [#](#timescaledb-v2-18-1-and-aws-transit-gateway-support-generally-available) ## TimescaleDB v2.18.1 and AWS Transit Gateway Support Generally Available ### TimescaleDB v2.18.1 [Section titled “TimescaleDB v2.18.1”](#timescaledb-v2181) New services use [TimescaleDB v2.18.1](https://github.com/timescale/timescaledb/releases/tag/2.18.1) with faster columnar scans and improved constraint handling. ### AWS Transit Gateway support is now generally available! [Section titled “AWS Transit Gateway support is now generally available!”](#aws-transit-gateway-support-is-now-generally-available) Securely connect Tiger Cloud to multiple VPCs across different environments. See the [AWS Transit Gateway documentation](https://docs.timescale.com/use-timescale/latest/security/transit-gateway/) for configuration details. ## TimescaleDB v2.18 and SQL assistant improvements in data mode and PopSQL [Section titled “TimescaleDB v2.18 and SQL assistant improvements in data mode and PopSQL”](#timescaledb-v218-and-sql-assistant-improvements-in-data-mode-and-popsql) February 6, 2025 🏎️ Performance 💡 Improvement [#](#timescaledb-v2-18-and-sql-assistant-improvements-in-data-mode-and-popsql) ## TimescaleDB v2.18 and SQL Assistant Improvements in Data Mode and PopSQL ### TimescaleDB v2.18 [Section titled “TimescaleDB v2.18”](#timescaledb-v218) * Dense indexes in columnstore through new hypercore table access method * SIMD vectorization improvements for aggregations * Hypertables support triggers for transition tables * Updated methods to manage hybrid row-columnar store For more information, see the [TimescaleDB v2.18 release notes](https://github.com/timescale/timescaledb/releases/tag/2.18.0). ### SQL assistant improvements [Section titled “SQL assistant improvements”](#sql-assistant-improvements-1) * Dedicated conversation threads for queries * Ability to delete messages from threads * Support for OpenAI o3-mini ![Dedicated threads](/docs/_astro/timescale-cloud-sql-assistant-threads.CBjxZ3w9.gif)![Delete messages in SQL Assistant threads](/docs/_astro/timescale-cloud-sql-assistant-delete-messages.BhbqUED1_Z1fUI2T.webp)![SQL Assistant o3 mini](/docs/_astro/timescale-cloud-sql-assistant-o3-mini.BSPDngYm_mfyxb.webp) ## IP allowlists in data mode and PopSQL [Section titled “IP allowlists in data mode and PopSQL”](#ip-allowlists-in-data-mode-and-popsql) January 31, 2025 🔐 Security [#](#ip-allowlists-in-data-mode-and-popsql) ## IP Allowlists in Data Mode and PopSQL ### Now available [Section titled “Now available”](#now-available) IP allowlists now available in Tiger Console data mode and PopSQL for enhanced network security. For background and configuration examples, see the [IP allow list documentation](https://docs.timescale.com/use-timescale/latest/security/ip-allow-list/). ![Timescale Console data mode IP allowlist](/docs/_astro/timescale-data-mode-ip-allowlist.CqqN3cJD_Z12MDTd.webp) ## pgai extension and Python library updates [Section titled “pgai extension and Python library updates”](#pgai-extension-and-python-library-updates) January 24, 2025 🔌 Integration ✨ New Feature [#](#pgai-extension-and-python-library-updates) ## pgai Extension and Python Library Updates ### pgai PostgreSQL extension 0.7.0 [Section titled “pgai PostgreSQL extension 0.7.0”](#pgai-postgresql-extension-070) Added configurable base\_url support for OpenAI API, public granting of vectorizers, and improved Ollama client. See all changes in the [pgai extension 0.7.0 release](https://github.com/timescale/pgai/releases/tag/extension-0.7.0). ### pgai Python library 0.5.0 [Section titled “pgai Python library 0.5.0”](#pgai-python-library-050) Comprehensive SQLAlchemy and Alembic support for vector embeddings. See all changes in the [pgai Python library v0.5.0 release](https://github.com/timescale/pgai/releases/tag/pgai-v0.5.0). ## AWS Transit Gateway support [Section titled “AWS Transit Gateway support”](#aws-transit-gateway-support) January 17, 2025 🧪 Beta 🔐 Security [#](#aws-transit-gateway-support) ## AWS Transit Gateway Support ### AWS Transit Gateway support (Early access) [Section titled “AWS Transit Gateway support (Early access)”](#aws-transit-gateway-support-early-access) Connect to Tiger Cloud services through AWS Transit Gateway for Scale and Enterprise customers. ## New region in India, PostgreSQL 17 upgrades, and TimescaleDB on AWS marketplace [Section titled “New region in India, PostgreSQL 17 upgrades, and TimescaleDB on AWS marketplace”](#new-region-in-india-postgresql-17-upgrades-and-timescaledb-on-aws-marketplace) January 10, 2025 🌍 Availability ✨ New Feature [#](#new-region-in-india-postgresql-17-upgrades-and-timescaledb-on-aws-marketplace) ## New region in India, PostgreSQL 17 upgrades, and TimescaleDB on AWS Marketplace ### Welcome India! (Mumbai region) [Section titled “Welcome India! (Mumbai region)”](#welcome-india-mumbai-region) Tiger Cloud now supports the Mumbai region. ### PostgreSQL major version upgrades to PG 17 [Section titled “PostgreSQL major version upgrades to PG 17”](#postgresql-major-version-upgrades-to-pg-17) Services can now upgrade directly to PostgreSQL 17 from versions 14, 15, or 16. ### Tiger Cloud available on AWS marketplace [Section titled “Tiger Cloud available on AWS marketplace”](#tiger-cloud-available-on-aws-marketplace) Centralize billing on AWS account with committed spend via [AWS Marketplace](https://aws.amazon.com/marketplace/seller-profile?id=seller-wbtecrjp3kxpm). ## PostgreSQL 17, feature requests, and PostgreSQL Livesync [Section titled “PostgreSQL 17, feature requests, and PostgreSQL Livesync”](#postgresql-17-feature-requests-and-postgresql-livesync) December 20, 2024 ✨ New Feature 🧪 Beta [#](#postgresql-17-feature-requests-and-postgresql-livesync) ## PostgreSQL 17, feature requests, and PostgreSQL Livesync ### PostgreSQL 17 [Section titled “PostgreSQL 17”](#postgresql-17) All new Tiger Cloud services come with PostgreSQL 17.2 with significant performance improvements and enhanced JSON support. ### Submit feature requests from Tiger Console [Section titled “Submit feature requests from Tiger Console”](#submit-feature-requests-from-tiger-console) Submit feature requests directly from Console with visibility into status. ![Submit a feature request in Timescale Console](/docs/_astro/submit-feature-request.Bzdxh2O8_Z6MM2x.webp) ### PostgreSQL Livesync (Alpha release) [Section titled “PostgreSQL Livesync (Alpha release)”](#postgresql-livesync-alpha-release) Continuously replicate PostgreSQL tables directly into Tiger Cloud. Learn more in the [Livesync for PostgreSQL documentation](https://docs.timescale.com/migrate/latest/livesync-for-postgresql/). ## In-Console import from S3, I/O boost, and jobs Explorer [Section titled “In-Console import from S3, I/O boost, and jobs Explorer”](#in-console-import-from-s3-io-boost-and-jobs-explorer) December 13, 2024 ✨ New Feature 💡 Improvement [#](#in-console-import-from-s3-i-o-boost-and-jobs-explorer) ## In-Console import from S3, I/O Boost, and Jobs Explorer ### In-Console import from S3 (CSV and Parquet files) [Section titled “In-Console import from S3 (CSV and Parquet files)”](#in-console-import-from-s3-csv-and-parquet-files) Connect S3 buckets to import CSV and Parquet files with 10 GB size limit. ![Import data into Timescale with S3](/docs/_astro/import-your-data-s3.CDllLX3x_Z2SzjU.webp)![Import data into Timescale with S3 details](/docs/_astro/import-data-s3-details.C0BubIic_Z1fur3G.webp) ### Self-Serve I/O boost [Section titled “Self-Serve I/O boost”](#self-serve-io-boost) Add-on for Scale or Enterprise tiers maximizing I/O to 16,000 IOPS with $0.41/hour per node. ![Timescale I/O Boost](/docs/_astro/timescale-i-o-boost.DtXyGO_l_Z1lSnnQ.webp) ### Jobs Explorer [Section titled “Jobs Explorer”](#jobs-explorer) New Jobs tab showing all associated jobs with status and detailed history. ![Timescale Console Jobs tab](/docs/_astro/timescale-console-jobs-tab.CHd-dTvi_ZEG7QF.webp)![Timescale Console Jobs tab expanded](/docs/_astro/timescale-console-jobs-expanded.BggmAE-q_2iw98b.webp) ## New service creation flow [Section titled “New service creation flow”](#new-service-creation-flow) December 6, 2024 💡 Improvement [#](#new-service-creation-flow) ## New service creation flow ### What's new [Section titled “What's new”](#whats-new-1) * **AI and Vector options:** Choose AI and Vector-ready services with pgai, pgvector, and pgvectorscale ![Create Timescale Cloud service](/docs/_astro/create-timescale-service.5lJYjTPf_ZkWlsU.webp) * **Compute size recommendations:** Recommendations based on planned data volume ![Service compute recommendation](/docs/_astro/timescale-service-compute-size.BzqV3evZ_Z1Ufx26.webp) * **More information about configuration options:** Clearer descriptions for informed choices ## IP allow lists! [Section titled “IP allow lists!”](#ip-allow-lists) November 21, 2024 🔐 Security [#](#ip-allow-lists) ## IP Allow Lists! ### IP allow lists now available [Section titled “IP allow lists now available”](#ip-allow-lists-now-available) IP Allow Lists let you specify IP addresses that have access to services and block others. They provide a lightweight alternative to a full [Virtual Private Cloud (VPC)](https://docs.timescale.com/use-timescale/latest/security/vpc/). To get started, open [Timescale Console](https://console.cloud.tigerdata.com/), then go to **Operations → Security → IP Allow List**. For more details, see the [IP allow list documentation](https://docs.timescale.com/use-timescale/latest/security/ip-allow-list/). ![IP Allow lists](/docs/_astro/IP-Allow-lists.CT150lU1_5YEb5.webp) ## SQL assistant, TimescaleDB v2.17, HIPAA compliance, and better logging [Section titled “SQL assistant, TimescaleDB v2.17, HIPAA compliance, and better logging”](#sql-assistant-timescaledb-v217-hipaa-compliance-and-better-logging) November 14, 2024 ✨ New Feature 🔐 Security 💡 Improvement [#](#sql-assistant-timescaledb-v2-17-hipaa-compliance-and-better-logging) ## SQL Assistant, TimescaleDB v2.17, HIPAA compliance, and better logging ### New AI companion: SQL assistant [Section titled “New AI companion: SQL assistant”](#new-ai-companion-sql-assistant) Chat with AI models to get help writing SQL, propose fixes for errors, and generate titles/descriptions. ![AI error fix](/docs/_astro/ai-error-fix.CtpBoXeU_Z1KbVJa.webp)![AI generated query title](/docs/_astro/ai-generate-title.W6ui3GGn_Z3CQ9E.webp) See the [SQL Assistant blog post](https://www.tigerdata.com/blog/postgres-gui-sql-assistant/) and [SQL Assistant docs](https://www.tigerdata.com/docs/getting-started/latest/run-queries-from-console/#sql-assistant) for full details. ### TimescaleDB v2.17 [Section titled “TimescaleDB v2.17”](#timescaledb-v217) Significant performance improvements for continuous aggregate refreshes and analytical queries. For details, see the [TimescaleDB v2.17 release notes](https://github.com/timescale/timescaledb/releases/tag/2.17.0). ### HIPAA compliance [Section titled “HIPAA compliance”](#hipaa-compliance) Tiger Cloud's Enterprise plan is now HIPAA compliant. ### Expanded logging within Tiger Console [Section titled “Expanded logging within Tiger Console”](#expanded-logging-within-tiger-console) Access more than 500 logs with infinite scrolling capabilities. ![Expanded console logs](/docs/_astro/console-expanded-logs.K0pBlxbN.gif) ## Connect to Tiger Cloud from .NET stack and check status of recent jobs [Section titled “Connect to Tiger Cloud from .NET stack and check status of recent jobs”](#connect-to-tiger-cloud-from-net-stack-and-check-status-of-recent-jobs) November 7, 2024 🔌 Integration 💡 Improvement [#](#connect-to-tiger-cloud-from-net-stack-and-check-status-of-recent-jobs) ## Connect to Tiger Cloud from .NET Stack and check status of recent jobs ### Connect with your .NET stack [Section titled “Connect with your .NET stack”](#connect-with-your-net-stack) Added instructions for connecting using Npgsql for .NET workflows. ![.NET instructions](/docs/_astro/connect-via-net.CgKjpO8m_ZqMyyL.webp) ### Last 5 jobs status [Section titled “Last 5 jobs status”](#last-5-jobs-status) See status of last 5 runs of each job in the Jobs section. ![job status](/docs/_astro/explorer-job-list.B5vckOX0_Z1mVg3I.webp) ## New AI, data integration, and performance enhancements [Section titled “New AI, data integration, and performance enhancements”](#new-ai-data-integration-and-performance-enhancements) October 31, 2024 ✨ New Feature 🏎️ Performance [#](#new-ai-data-integration-and-performance-enhancements) ## New AI, data integration, and performance enhancements ### Pgai Vectorizer (early access) [Section titled “Pgai Vectorizer (early access)”](#pgai-vectorizer-early-access) Automatically create, update, and maintain embeddings as data changes. For more details, see the [pgai Vectorizer guide](https://github.com/timescale/pgai/blob/main/docs/vectorizer/overview.md). ![Vectorizer setup](/docs/_astro/vectorizer-setup.BWMriB7K_Z1NxG2C.webp) ### PostgreSQL-to-PostgreSQL foreign data wrappers [Section titled “PostgreSQL-to-PostgreSQL foreign data wrappers”](#postgresql-to-postgresql-foreign-data-wrappers) Fetch and query data from multiple PostgreSQL databases directly using [foreign data wrappers (FDW)](https://docs.timescale.com/use-timescale/latest/schema-management/foreign-data-wrappers/). ### Faster queries over tiered data [Section titled “Faster queries over tiered data”](#faster-queries-over-tiered-data) Support for runtime chunk exclusion for [tiered storage](https://docs.timescale.com/use-timescale/latest/data-tiering/). For background, see the blog post on [implementing constraint exclusion for faster query performance](https://www.tigerdata.com/blog/implementing-constraint-exclusion-for-faster-query-performance), and the [`disable_tiering` function docs](https://docs.timescale.com/use-timescale/latest/data-tiering/enabling-data-tiering/#disable-tiering). ### Chunk interval recommendations [Section titled “Chunk interval recommendations”](#chunk-interval-recommendations) Console shows recommendations for services with too many small chunks. ![Chunk interval recommendation](/docs/_astro/chunk-interval-recommendation.Dekuccb3_Z25mYFM.webp) ## Help with hypertables and faster notebooks [Section titled “Help with hypertables and faster notebooks”](#help-with-hypertables-and-faster-notebooks) October 18, 2024 💡 Improvement [#](#help-with-hypertables-and-faster-notebooks) ## Help with hypertables and faster notebooks ### Hypertable creation wizard [Section titled “Hypertable creation wizard”](#hypertable-creation-wizard) Create hypertable directly in Tiger Console after service creation. ![Hypertable creation wizard: image 1](/docs/_astro/hypertable-creation-wizard-1.D7xMmkCw_diqNv.webp) ### PopSQL Notebooks [Section titled “PopSQL Notebooks”](#popsql-notebooks) Newly developed v3 query engine provides significantly faster query response times. ## Production-Ready low-Downtime migrations, MySQL import, actions tab, and current lock contention visibility in SQL editor [Section titled “Production-Ready low-Downtime migrations, MySQL import, actions tab, and current lock contention visibility in SQL editor”](#production-ready-low-downtime-migrations-mysql-import-actions-tab-and-current-lock-contention-visibility-in-sql-editor) October 10, 2024 🎉 GA ✨ New Feature [#](#production-ready-low-downtime-migrations-mysql-import-actions-tab-and-current-lock-contention-visibility-in-sql-editor) ## Production-Ready Low-Downtime Migrations, MySQL Import, Actions Tab, and Current Lock Contention Visibility in SQL Editor ### Live migrations v1.0 release [Section titled “Live migrations v1.0 release”](#live-migrations-v10-release) Live migration is production-ready with enhanced functionality and performance optimizations. ### Actions tab [Section titled “Actions tab”](#actions-tab) Actions persisted within service for on-demand access to connect, import, and migrate. ![Timescale Console Actions tab](/docs/_astro/timescale-console-actions-tab.CmtRJ_mG_Z1KvVkC.webp) ### Import data from MySQL [Section titled “Import data from MySQL”](#import-data-from-mysql) Easy-to-follow instructions for converting MySQL schema and data. ![MySQL import instructions](/docs/_astro/mysql-import-instructions.DP8VxQwC_Z1s04Bo.webp) ### Current lock contention [Section titled “Current lock contention”](#current-lock-contention) Console displays current lock contention when queries wait on locks. ![View console services](/docs/_astro/current-lock-contention.DDhJVEA8_ZJwJ0w.webp) ## CIDR & VPC updates [Section titled “CIDR & VPC updates”](#cidr--vpc-updates) October 3, 2024 🔐 Security [#](#cidr-vpc-updates) ## CIDR & VPC Updates ### Multiple CIDRs supported [Section titled “Multiple CIDRs supported”](#multiple-cidrs-supported) Timescale now supports multiple CIDRs on customer VPC. ## New modes in Tiger Console: ops and data mode, and Console based Parquet file import [Section titled “New modes in Tiger Console: ops and data mode, and Console based Parquet file import”](#new-modes-in-tiger-console-ops-and-data-mode-and-console-based-parquet-file-import) September 19, 2024 ✨ New Feature [#](#new-modes-in-tiger-console-ops-and-data-mode-and-console-based-parquet-file-import) ## New modes in Tiger Console: Ops and Data mode, and Console based Parquet File Import ### Ops and data mode [Section titled “Ops and data mode”](#ops-and-data-mode) **Ops mode:** Manage services, add replicas, configure compression, change parameters. **Data mode:** Full PopSQL experience with queries, visualizations, dashboards, and sharing. ![Timescale Console Ops and Data mode](/docs/_astro/ops-data-mode.Dy4Pr2x_.gif) ### Console based Parquet file import [Section titled “Console based Parquet file import”](#console-based-parquet-file-import) Upload Parquet files from local file system. ![Upload from Parquet to Timescale Cloud](/docs/_astro/upload_parquet.DNlrGNcR.gif) ## High availability, usability, and migrations improvements [Section titled “High availability, usability, and migrations improvements”](#high-availability-usability-and-migrations-improvements) September 12, 2024 ✨ New Feature 💡 Improvement [#](#high-availability-usability-and-migrations-improvements) ## High availability, usability, and migrations improvements ### Multiple HA replicas [Section titled “Multiple HA replicas”](#multiple-ha-replicas) Scale and Enterprise customers can configure two new HA replica options. For more information, see [Manage high availability](https://docs.timescale.com/use-timescale/latest/ha-replicas/high-availability/). ![Change Replica Configuration](/docs/_astro/change-replica-configuration.DPnJ1ewF_ZSBWGt.webp)![High Availability](/docs/_astro/high-availability.CYb7tdCG_EL7KS.webp) ### Other improvements [Section titled “Other improvements”](#other-improvements) Session health indicator in Console SQL editor shows connection status. ![Session Status Indicator](/docs/_astro/session-status-indicator.D1S-s987.gif) ## One-click SQL statement execution from Tiger Console, and session support in the SQL editor [Section titled “One-click SQL statement execution from Tiger Console, and session support in the SQL editor”](#one-click-sql-statement-execution-from-tiger-console-and-session-support-in-the-sql-editor) September 5, 2024 ✨ New Feature [#](#one-click-sql-statement-execution-from-tiger-console-and-session-support-in-the-sql-editor) ## One-click SQL statement execution from Tiger Console, and session support in the SQL editor ### One-click SQL execution [Section titled “One-click SQL execution”](#one-click-sql-execution) Execute SQL statements with one click throughout Console for hypertables, extensions, and CSV imports. This requires that the [SQL editor](https://docs.timescale.com/getting-started/latest/run-queries-from-console/#sql-editor) is enabled for the service. ![Enable Continuous Aggregates](/docs/_astro/enable-continuous-aggregates.BhHoS7mw.gif)![Enable extensions from Console](/docs/_astro/enable-extensions-from-console.CnSK6N20.gif)![Query data after CSV import](/docs/_astro/query-data-after-csv-import.BfFPh6HW.gif) ### Session support in the SQL editor [Section titled “Session support in the SQL editor”](#session-support-in-the-sql-editor) Database session persistence enables transactions, temporary tables, and set commands. ## Query your database directly from the Console and enhanced data import and migration options [Section titled “Query your database directly from the Console and enhanced data import and migration options”](#query-your-database-directly-from-the-console-and-enhanced-data-import-and-migration-options) August 30, 2024 ✨ New Feature 💡 Improvement [#](#query-your-database-directly-from-the-console-and-enhanced-data-import-and-migration-options) ## Query your database directly from the Console and enhanced data import and migration options ### SQL editor in Tiger Console [Section titled “SQL editor in Tiger Console”](#sql-editor-in-tiger-console) New tab allowing direct database queries without leaving console. ![SQL Editor](/docs/_astro/sql-editor-query.BjuXrhAw_12iuNl.webp) ### Enhanced data import options [Section titled “Enhanced data import options”](#enhanced-data-import-options) Dedicated section with multiple import options and enhanced PostgreSQL import instructions. ![Data import screen](/docs/_astro/data-import-screen.DBNiUJ5Q_Z1FAcWU.webp) ### Improvements to live migration [Section titled “Improvements to live migration”](#improvements-to-live-migration) Support for timescaledb on non-public schema and pre-migration compatibility checks. ## Improved tooling in Tiger Cloud and new AI and vector extension releases [Section titled “Improved tooling in Tiger Cloud and new AI and vector extension releases”](#improved-tooling-in-tiger-cloud-and-new-ai-and-vector-extension-releases) August 22, 2024 ✨ New Feature 🔌 Integration [#](#improved-tooling-in-tiger-cloud-and-new-ai-and-vector-extension-releases) ## Improved tooling in Tiger Cloud and new AI and Vector extension releases ### CSV import [Section titled “CSV import”](#csv-import) Added CSV import tool with data type selection and hypertable creation. ![CSV import](/docs/_astro/csv-import.CZA8AOX1_x3qWm.webp) ### Replica lag [Section titled “Replica lag”](#replica-lag) New Replica Lag parameter shows bytes behind primary for replicas. ![Replica lag indicator](/docs/_astro/replica-lag-indicator.DF2ZWmx3_ZUcdD1.webp) ### Adjust chunk interval [Section titled “Adjust chunk interval”](#adjust-chunk-interval) Change chunk interval for hypertables and continuous aggregates through UI. ![Edit chunk interval](/docs/_astro/edit-chunk-interval.X8SnpGcq_Zj7Dmy.webp) ### CloudWatch permissions via role assumption [Section titled “CloudWatch permissions via role assumption”](#cloudwatch-permissions-via-role-assumption) Permission granting via role assumption for CloudWatch integration. For more details, see the CloudWatch integration documentation at . ### Two-factor authentication (2FA) indicator [Section titled “Two-factor authentication (2FA) indicator”](#two-factor-authentication-2fa-indicator) 2FA status column on Members page. ![2FA status](/docs/_astro/2FA-status-indicator.CnSypf3U_2hHQt1.webp) ### Anthropic and Cohere integrations in pgai [Section titled “Anthropic and Cohere integrations in pgai”](#anthropic-and-cohere-integrations-in-pgai) pgai extension v0.3.0 supports Anthropic and Cohere models. For details and examples, see the blog posts on pgai with Cohere at and pgai with Anthropic at . ### pgvectorscale extension [Section titled “pgvectorscale extension”](#pgvectorscale-extension) V0.3.0 adds ARM processor support and improved recall for low dimensional vectors. See the pgvectorscale v0.3.0 release notes at . ## Optimizations for compressed data and extended join support in continuous aggregates [Section titled “Optimizations for compressed data and extended join support in continuous aggregates”](#optimizations-for-compressed-data-and-extended-join-support-in-continuous-aggregates) August 15, 2024 🏎️ Performance [#](#optimizations-for-compressed-data-and-extended-join-support-in-continuous-aggregates) ## Optimizations for compressed data and extended join support in continuous aggregates ### TimescaleDB v2.16.0 [Section titled “TimescaleDB v2.16.0”](#timescaledb-v2160) Significant performance improvements for DML over compressed chunks, chunk skipping on non-partitioning columns, foreign key support, and extended continuous aggregate join support. ## Performance, packaging and stability improvements for Tiger Cloud [Section titled “Performance, packaging and stability improvements for Tiger Cloud”](#performance-packaging-and-stability-improvements-for-tiger-cloud) August 8, 2024 💡 Improvement [#](#performance-packaging-and-stability-improvements-for-tiger-cloud) ## Performance, packaging and stability improvements for Tiger Cloud ### New plans [Section titled “New plans”](#new-plans) Three new plans: Performance (cost-focused), Scale (critical apps), and Enterprise (mission-critical). For details on features and differences, see the pricing and account management docs at . ### Improvements to the Tiger Console [Section titled “Improvements to the Tiger Console”](#improvements-to-the-tiger-console) Enhanced service tiles with HA status information. ### Live migration release v0.0.24 [Section titled “Live migration release v0.0.24”](#live-migration-release-v0024) Automatic retries, pgcopydb support for initial copy, and TimescaleDB v2.13.x support. ## Performance and stability improvements for Tiger Cloud and TimescaleDB [Section titled “Performance and stability improvements for Tiger Cloud and TimescaleDB”](#performance-and-stability-improvements-for-tiger-cloud-and-timescaledb) July 12, 2024 🏎️ Performance 🐛 Bug Fix [#](#performance-and-stability-improvements-for-tiger-cloud-and-timescaledb) ## Performance and stability improvements for Tiger Cloud and TimescaleDB ### Improvements in this release [Section titled “Improvements in this release”](#improvements-in-this-release) * Connection pooler improvements * tsdbadmin user role enhancements * TimescaleDB v2.15.3 and v2.15.2 patch releases with multiple improvements. For more information, see the [2.15.3 release note](https://github.com/timescale/timescaledb/releases/tag/2.15.3) and the [2.15.2 release note](https://github.com/timescale/timescaledb/releases/tag/2.15.2). ## Performance improvements for live migration to Tiger Cloud [Section titled “Performance improvements for live migration to Tiger Cloud”](#performance-improvements-for-live-migration-to-tiger-cloud) June 27, 2024 🏎️ Performance [#](#performance-improvements-for-live-migration-to-tiger-cloud) ## Performance improvements for live migration to Tiger Cloud ### Live migration updates [Section titled “Live migration updates”](#live-migration-updates) The Timescale live-migration Docker image now supports table-based filtering during live migration, improvements to `pbcopydb` (including better performance and removal of unhelpful warnings), and a user notification log to help you always select the most recent release for a migration run. For improved stability and new features, update to the latest `timescale/live-migration` image on Docker Hub: . To learn more, see the live migration docs at . ## Ollama integration in pgai [Section titled “Ollama integration in pgai”](#ollama-integration-in-pgai) June 21, 2024 🔌 Integration [#](#ollama-integration-in-pgai) ## Ollama integration in pgai ### Ollama now available [Section titled “Ollama now available”](#ollama-now-available) Ollama is now integrated with the pgai extension, making it easy to work with open-source language models such as Llama 3, Mistral, Phi 3, and Gemma. Ollama acts like “Docker for LLMs,” providing a simple way to run and manage models locally. With pgai installed in your database, you can embed Ollama-powered AI directly into your app using SQL, for example, by generating completions or working with images in a single query. To learn more and see a quick-start example, see the pgai Ollama documentation at . ## Compression wizard [Section titled “Compression wizard”](#compression-wizard) June 13, 2024 ✨ New Feature [#](#compression-wizard) ## Compression Wizard ### Now available in Console [Section titled “Now available in Console”](#now-available-in-console) The compression wizard is now available in Timescale Console. From the Explorer, select a hypertable and, in the top-right corner, hover where it says "Compression off" to open the wizard. You’ll be guided through configuring compression for the hypertable and can enable it directly from the UI. ![Run the compression wizard in Timescale Console](/docs/_astro/compress-data-in-console.cGL8pcZZ_Z1nXNy3.webp) ## High performance AI apps with pgvectorscale [Section titled “High performance AI apps with pgvectorscale”](#high-performance-ai-apps-with-pgvectorscale) June 11, 2024 ✨ New Feature 🏎️ Performance [#](#high-performance-ai-apps-with-pgvectorscale) ## High Performance AI Apps With pgvectorscale ### pgvectorscale on Tiger Cloud [Section titled “pgvectorscale on Tiger Cloud”](#pgvectorscale-on-tiger-cloud) The pgvectorscale extension is now available on Timescale Cloud. It complements pgvector and introduces: * A new index type, StreamingDiskANN, inspired by Microsoft’s DiskANN algorithm. * Statistical Binary Quantization, a compression method developed by Timescale researchers that improves on standard binary quantization. On a benchmark dataset of 50 million Cohere embeddings (768 dimensions), PostgreSQL with pgvector and pgvectorscale achieves dramatically lower p95 latency and higher query throughput than Pinecone’s storage-optimized index at comparable recall, while running at significantly lower cost on self-hosted AWS EC2. For more, see the pgvectorscale documentation at . ## Integrate AI into your database using pgai [Section titled “Integrate AI into your database using pgai”](#integrate-ai-into-your-database-using-pgai) June 11, 2024 ✨ New Feature 🔌 Integration [#](#integrate-ai-into-your-database-using-pgai) ## Integrate AI Into Your Database Using pgai ### pgai on Tiger Cloud [Section titled “pgai on Tiger Cloud”](#pgai-on-tiger-cloud) The pgai extension is now available on Timescale Cloud, bringing embedding and generative AI models closer to your database. With pgai you can: * Create embeddings for your data directly from SQL. * Retrieve LLM chat completions from models like OpenAI GPT‑4o. * Run classification, summarization, enrichment, and other AI-powered workloads over existing relational data in PostgreSQL. For an overview, see the pgai documentation at . ## Continuous aggregate and hypertable improvements for TimescaleDB [Section titled “Continuous aggregate and hypertable improvements for TimescaleDB”](#continuous-aggregate-and-hypertable-improvements-for-timescaledb) June 7, 2024 🏎️ Performance 💡 Improvement [#](#continuous-aggregate-and-hypertable-improvements-for-timescaledb) ## Continuous Aggregate and Hypertable Improvements for TimescaleDB ### TimescaleDB v2.15.x [Section titled “TimescaleDB v2.15.x”](#timescaledb-v215x) * Continuous Aggregate supports time\_bucket with origin and/or offset * Compression improvements with optimized defaults and planner enhancements These releases also add planner improvements to inspect more WHERE conditions before decompression (reducing the number of rows that must be decompressed), allow minmax sparse indexes to be used when compressing columns with btree indexes, and introduce vectorized filters in WHERE clauses containing text equality operators and LIKE expressions. For more details, see the TimescaleDB release notes at . ## Database audit logging with pgaudit [Section titled “Database audit logging with pgaudit”](#database-audit-logging-with-pgaudit) May 31, 2024 🔐 Security [#](#database-audit-logging-with-pgaudit) ## Database Audit Logging with pgaudit ### pgaudit now available [Section titled “pgaudit now available”](#pgaudit-now-available) PostgreSQL Audit extension (pgaudit) now available on Tiger Cloud for detailed session and object audit logging. See the [PG Audit extension (pgaudit)](https://github.com/pgaudit/pgaudit/) and [pgaudit documentation](https://github.com/pgaudit/pgaudit/) for more information. ## International system of unit support with postgresql-unit [Section titled “International system of unit support with postgresql-unit”](#international-system-of-unit-support-with-postgresql-unit) May 31, 2024 ✨ New Feature [#](#international-system-of-unit-support-with-postgresql-unit) ## International System of Unit Support with postgresql-unit ### postgresql-unit extension [Section titled “postgresql-unit extension”](#postgresql-unit-extension) The [SI Units for PG extension(unit)](https://github.com/df7cb/postgresql-unit) provides support for the [ISU](https://en.wikipedia.org/wiki/International_System_of_Units) in Timescale Cloud. You can use Timescale Cloud to solve day-to-day questions. For example, to see what 50°C is in °F, run the following query in your Timescale Cloud service: ```sql SELECT '50°C'::unit @ '°F' as temp; temp -------- 122 °F (1 row) ``` To learn more, see the [postgresql-unit documentation](https://github.com/df7cb/postgresql-unit). # Release notes > Get an overview of the new features, improvements, and fixes released in every version of TimescaleDB and other downloadable products For information about new features and improvements to Tiger Cloud, see the [Changelog](/docs/get-started/news/new). For other products, see: * [TimescaleDB](https://github.com/timescale/timescaledb/releases) - an open-source database that makes SQL scalable for time-series data, packaged as a PostgreSQL extension. * [TimescaleDB Toolkit](https://github.com/timescale/timescaledb-toolkit/releases) - additional functions to ease all things analytics when using TimescaleDB. * [pgai](https://github.com/timescale/pgai/releases) - brings AI workflows to your PostgreSQL database. * [pgvectorscale](https://github.com/timescale/pgvectorscale/releases/tag/0.2.0) - higher performance embedding search and cost-efficient storage for AI applications on PostgreSQL. * [pgspot](https://github.com/timescale/pgspot/releases) - spot vulnerabilities in PostgreSQL extension scripts. * [live-migration](https://hub.docker.com/r/timescale/live-migration/tags) - a Docker image to migrate data to a Tiger Cloud service. # Connect your app > Connect your existing application to TimescaleDB Easily integrate your app with self-hosted TimescaleDB. Use your favorite programming language to connect to your database, create and manage hypertables, then ingest and query data. * Ruby ## Prerequisites for this tutorial To follow these steps, you'll need: * A [self-hosted TimescaleDB](/docs/get-started/choose-your-path/install-timescaledb) instance. - Your [connection details](/docs/integrate/find-connection-details). * [Rails](https://guides.rubyonrails.org/install_ruby_on_rails.html#installing-rails) installed. ## Connect a Rails app to TimescaleDB TimescaleDB extends PostgreSQL with time-series superpowers. You connect to your TimescaleDB database from a standard Rails app configured for PostgreSQL. 1. **Create a new Rails app configured for PostgreSQL** Rails creates and bundles your app, then installs the standard PostgreSQL Gems. ```bash rails new my_app -d=postgresql cd my_app ``` 2. **Install the TimescaleDB gem** 1. Open `Gemfile`, add the following line, then save your changes: ```ruby gem 'timescaledb' ``` 2. In the terminal, run the following command: ```bash bundle install ``` 3. **Connect your app to TimescaleDB** 1. In `/config/database.yml` update the configuration to securely connect to your TimescaleDB database by adding `url: <%= ENV['DATABASE_URL'] %>` to the default configuration: ```yaml default: &default adapter: postgresql encoding: unicode pool: <%= ENV.fetch("RAILS_MAX_THREADS") { 5 } %> url: <%= ENV['DATABASE_URL'] %> ``` 2. Set the environment variable for `DATABASE_URL` to your connection string: ```bash export DATABASE_URL="postgres://username:password@host:port/dbname" ``` 3. Create the database for the project: ```bash rails db:create ``` 4. Run migrations: ```bash rails db:migrate ``` 5. Verify the connection from your app to your TimescaleDB database: ```bash echo "\dx" | rails dbconsole ``` The result shows the list of extensions in your database | Name | Version | Schema | Description | | -------------------- | ------- | ----------- | ------------------------------------------------------------------------------------- | | pg\_buffercache | 1.5 | public | examine the shared buffer cache | | pg\_stat\_statements | 1.11 | public | track planning and execution statistics of all SQL statements executed | | plpgsql | 1.0 | pg\_catalog | PL/pgSQL procedural language | | postgres\_fdw | 1.1 | public | foreign-data wrapper for remote PostgreSQL servers | | timescaledb | 2.24.0 | public | Enables scalable inserts and complex queries for time-series data (Community Edition) | | timescaledb\_toolkit | 1.22.0 | public | Library of analytical hyperfunctions, time-series pipelining, and other SQL utilities | ## Optimize time-series data in hypertables Hypertables are PostgreSQL tables designed to simplify and accelerate data analysis. Anything you can do with regular PostgreSQL tables, you can do with hypertables - but much faster and more conveniently. In this section, you use the helpers in the TimescaleDB gem to create and manage a [hypertable](/docs/build/how-to/your-first-hypertable). 1. **Generate a migration to create the page loads table** ```bash rails generate migration create_page_loads ``` This creates the `/db/migrate/_create_page_loads.rb` migration file. 2. **Add hypertable options** Replace the contents of `/db/migrate/_create_page_loads.rb` with the following: ```ruby class CreatePageLoads < ActiveRecord::Migration[8.0] def change hypertable_options = { time_column: 'created_at', chunk_time_interval: '1 day', compress_segmentby: 'path', compress_orderby: 'created_at', compress_after: '7 days', drop_after: '30 days' } create_table :page_loads, id: false, primary_key: [:created_at, :user_agent, :path], hypertable: hypertable_options do |t| t.timestamptz :created_at, null: false t.string :user_agent t.string :path t.float :performance end end end ``` The `id` column is not included in the table. This is because TimescaleDB requires that any `UNIQUE` or `PRIMARY KEY` indexes on the table include all partitioning columns. In this case, this is the time column. A new Rails model includes a `PRIMARY KEY` index for id by default: either remove the column or make sure that the index includes time as part of a "composite key." For more information, check the Roby docs around [composite primary keys](https://guides.rubyonrails.org/active_record_composite_primary_keys.html). 3. **Create a `PageLoad` model** Create a new file called `/app/models/page_load.rb` and add the following code: ```ruby class PageLoad < ApplicationRecord extend Timescaledb::ActsAsHypertable include Timescaledb::ContinuousAggregatesHelper acts_as_hypertable time_column: "created_at", segment_by: "path", value_column: "performance" # Basic scopes for filtering by browser scope :chrome_users, -> { where("user_agent LIKE ?", "%Chrome%") } scope :firefox_users, -> { where("user_agent LIKE ?", "%Firefox%") } scope :safari_users, -> { where("user_agent LIKE ?", "%Safari%") } # Performance analysis scopes scope :performance_stats, -> { select("stats_agg(#{value_column}) as stats_agg") } scope :slow_requests, -> { where("performance > ?", 1.0) } scope :fast_requests, -> { where("performance < ?", 0.1) } # Set up continuous aggregates for different timeframes continuous_aggregates scopes: [:performance_stats], timeframes: [:minute, :hour, :day], refresh_policy: { minute: { start_offset: '3 minute', end_offset: '1 minute', schedule_interval: '1 minute' }, hour: { start_offset: '3 hours', end_offset: '1 hour', schedule_interval: '1 minute' }, day: { start_offset: '3 day', end_offset: '1 day', schedule_interval: '1 minute' } } end ``` 4. **Run the migration** ```bash rails db:migrate ``` ## Insert data into your database The TimescaleDB gem provides efficient ways to insert data into hypertables. This section shows you how to ingest test data into your hypertable. 1. **Create a controller to handle page loads** Create a new file called `/app/controllers/application_controller.rb` and add the following code: ```ruby class ApplicationController < ActionController::Base around_action :track_page_load private def track_page_load start_time = Time.current yield end_time = Time.current PageLoad.create( path: request.path, user_agent: request.user_agent, performance: (end_time - start_time) ) end end ``` 2. **Generate some test data** Use `bin/console` to join a Rails console session and run the following code to define some random page load access data: ```ruby def generate_sample_page_loads(total: 1000) time = 1.month.ago paths = %w[/ /about /contact /products /blog] browsers = [ "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:89.0) Gecko/20100101 Firefox/89.0", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.1 Safari/605.1.15" ] total.times.map do time = time + rand(60).seconds { path: paths.sample, user_agent: browsers.sample, performance: rand(0.1..2.0), created_at: time, updated_at: time } end end ``` 3. **Insert the generated data into your TimescaleDB database** ```bash # Insert the data in batches PageLoad.insert_all(generate_sample_page_loads, returning: false) ``` 4. **Validate the test data in your TimescaleDB database** ```bash PageLoad.count PageLoad.first ``` ## Reference This section lists the most common tasks you might perform with the TimescaleDB gem. ### Query scopes The TimescaleDB gem provides several convenient scopes for querying your time-series data. * Built-in time-based scopes: ```ruby PageLoad.last_hour.count PageLoad.today.count PageLoad.this_week.count PageLoad.this_month.count ``` * Browser-specific scopes: ```ruby # Count requests by browser PageLoad.chrome_users.last_hour.count PageLoad.firefox_users.last_hour.count PageLoad.safari_users.last_hour.count # Performance analysis PageLoad.slow_requests.last_hour.count PageLoad.fast_requests.last_hour.count ``` * Query continuous aggregates: This query fetches the average and standard deviation from the performance stats for the `/products` path over the last day. ```ruby # Access aggregated performance stats through generated classes PageLoad::PerformanceStatsPerMinute.last_hour PageLoad::PerformanceStatsPerHour.last_day PageLoad::PerformanceStatsPerDay.last_month # Get statistics for a specific path stats = PageLoad::PerformanceStatsPerHour.last_day.where(path: '/products').select("average(stats_agg) as average, stddev(stats_agg) as stddev").first puts "Average: #{stats.average}" puts "Standard Deviation: #{stats.stddev}" ``` ### TimescaleDB features The TimescaleDB gem provides utility methods to access hypertable and chunk information. Every model that uses the `acts_as_hypertable` method has access to these methods. #### Access hypertable and chunk information * View chunk or hypertable information: ```ruby PageLoad.chunks.count PageLoad.hypertable.detailed_size ``` * Compress/Decompress chunks: ```ruby PageLoad.chunks.uncompressed.first.compress! # Compress the first uncompressed chunk PageLoad.chunks.compressed.first.decompress! # Decompress the oldest chunk PageLoad.hypertable.compression_stats # View compression stats ``` #### Access hypertable stats You collect hypertable stats using methods that provide insights into your hypertable's structure, size, and compression status: * Get basic hypertable information: ```ruby hypertable = PageLoad.hypertable hypertable.hypertable_name # The name of your hypertable hypertable.schema_name # The schema where the hypertable is located ``` * Get detailed size information: ```ruby hypertable.detailed_size # Get detailed size information for the hypertable hypertable.compression_stats # Get compression statistics hypertable.chunks_detailed_size # Get chunk information hypertable.approximate_row_count # Get approximate row count hypertable.dimensions.map(&:column_name) # Get dimension information hypertable.continuous_aggregates.map(&:view_name) # Get continuous aggregate view names ``` #### Continuous aggregates The `continuous_aggregates` method generates a class for each continuous aggregate. * Get all the continuous aggregate classes: ```ruby PageLoad.descendants # Get all continuous aggregate classes ``` * Manually refresh a continuous aggregate: ```ruby PageLoad.refresh_aggregates ``` * Create or drop a continuous aggregate: Create or drop all the continuous aggregates in the proper order to build them hierarchically. See more about how it works in this [blog post](https://www.tigerdata.com/blog/building-a-better-ruby-orm-for-time-series-and-analytics). ```ruby PageLoad.create_continuous_aggregates PageLoad.drop_continuous_aggregates ``` ## Next steps Now that you have integrated the ruby gem into your app: * Learn more about the [TimescaleDB gem](https://github.com/timescale/timescaledb-ruby). * Check out the [official docs](https://timescale.github.io/timescaledb-ruby/). * Follow the [LTTB](https://timescale.github.io/timescaledb-ruby/toolkit_lttb_tutorial/), [Open AI long-term storage](https://timescale.github.io/timescaledb-ruby/chat_gpt_tutorial/), and [candlesticks](https://timescale.github.io/timescaledb-ruby/toolkit_candlestick/) tutorials. * Python ## Prerequisites for this tutorial To follow these steps, you'll need: * A [self-hosted TimescaleDB](/docs/get-started/choose-your-path/install-timescaledb) instance. - Your [connection details](/docs/integrate/find-connection-details). * The `psycopg2` library installed. For more information, see the [psycopg2 documentation](https://pypi.org/project/psycopg2/). * (Optional) A [Python virtual environment](https://docs.python.org/3/library/venv.html). ## Connect to TimescaleDB In this section, you create a connection to TimescaleDB using the `psycopg2` library. This library is one of the most popular PostgreSQL libraries for Python. It allows you to execute raw SQL queries efficiently and safely, and prevents common attacks such as SQL injection. 1. **Import the psycogpg2 library:** ```python import psycopg2 ``` 2. **Locate your TimescaleDB credentials** Use them to compose a connection string for `psycopg2`. You'll need: * password * username * host URL * port * database name 3. **Compose your connection string variable** Use a [libpq connection string](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING) format: ```python CONNECTION = "postgres://username:password@host:port/dbname" ``` If you require an SSL connection, use this version instead: ```python CONNECTION = "postgres://username:password@host:port/dbname?sslmode=require" ``` Alternatively you can specify each parameter in the connection string as follows ```python CONNECTION = "dbname=tsdb user=tsdbadmin password=secret host=host.com port=5432 sslmode=require" ``` Warning This method of composing a connection string is for test or development purposes only. For production, use environment variables for sensitive details like your password, hostname, and port number. 4. **Create a database session with psycopg2** Use the `psycopg2` connect function to create a new database session and a new [cursor object](https://www.psycopg.org/docs/connection.html?highlight=cursor#connection.cursor) to interact with the database. In your `main` function, add these lines: ```python CONNECTION = "postgres://username:password@host:port/dbname" with psycopg2.connect(CONNECTION) as conn: cursor = conn.cursor() # use the cursor to interact with your database # cursor.execute("SELECT * FROM table") ``` Alternatively, you can create a connection object and pass the object around as needed, like opening a cursor to perform database operations: ```python CONNECTION = "postgres://username:password@host:port/dbname" conn = psycopg2.connect(CONNECTION) cursor = conn.cursor() # use the cursor to interact with your database cursor.execute("SELECT 'hello world'") print(cursor.fetchone()) ``` ## Create a relational table In this section, you create a table called `sensors` which holds the ID, type, and location of your fictional sensors. Additionally, you create a hypertable called `sensor_data` which holds the measurements of those sensors. The measurements contain the time, sensor\_id, temperature reading, and CPU percentage of the sensors. 1. **Compose the SQL statement to create a relational table** This example creates a table called `sensors`, with columns `id`, `type` and `location`: ```python query_create_sensors_table = """CREATE TABLE sensors ( id SERIAL PRIMARY KEY, type VARCHAR(50), location VARCHAR(50) ); """ ``` 2. **Execute the query and commit the changes** Open a cursor, execute the query you created in the previous step, and commit the query to make the changes persistent. Afterward, close the cursor to clean up: ```python cursor = conn.cursor() # see definition in Step 1 cursor.execute(query_create_sensors_table) conn.commit() cursor.close() ``` ## Create a hypertable When you have created the relational table, you can create a hypertable. Creating tables and indexes, altering tables, inserting data, selecting data, and most other tasks are executed on the hypertable. 1. **Create the `CREATE TABLE` SQL statement for your hypertable** Notice how the hypertable has the compulsory time column: ```python # create sensor data hypertable query_create_sensordata_table = """CREATE TABLE sensor_data ( time TIMESTAMPTZ NOT NULL, sensor_id INTEGER, temperature DOUBLE PRECISION, cpu DOUBLE PRECISION, FOREIGN KEY (sensor_id) REFERENCES sensors (id) ); """ ``` 2. **Convert the table to a hypertable** Formulate a `SELECT` statement that converts the `sensor_data` table to a hypertable. You must specify the table name to convert to a hypertable, and the name of the time column as the two arguments. For more information, see the [`create_hypertable` docs](/docs/reference/timescaledb/hypertables/create_hypertable): ```python query_create_sensordata_hypertable = "SELECT create_hypertable('sensor_data', by_range('time'));" ``` Note The `by_range` dimension builder is an addition to TimescaleDB 2.13. 3. **Execute the statements and commit your changes** Open a cursor with the connection, execute the statements from the previous steps, commit your changes, and close the cursor: ```python cursor = conn.cursor() cursor.execute(query_create_sensordata_table) cursor.execute(query_create_sensordata_hypertable) # commit changes to the database to make changes persistent conn.commit() cursor.close() ``` ## Insert rows of data You can insert data into your hypertables in several different ways. In this section, you can use `psycopg2` with prepared statements, or you can use `pgcopy` for a faster insert. 1. **Insert relational data into the `sensors` table** Open a cursor with a connection to the database, use prepared statements to formulate the `INSERT` SQL statement, and then execute that statement: ```python sensors = [('a', 'floor'), ('a', 'ceiling'), ('b', 'floor'), ('b', 'ceiling')] cursor = conn.cursor() for sensor in sensors: try: cursor.execute("INSERT INTO sensors (type, location) VALUES (%s, %s);", (sensor[0], sensor[1])) except (Exception, psycopg2.Error) as error: print(error.pgerror) conn.commit() ``` 2. **(Optional) Use separate SQL and data variables** Alternatively, you can pass variables to the `cursor.execute` function and separate the formulation of the SQL statement, `SQL`, from the data being passed with it into the prepared statement, `data`: ```python SQL = "INSERT INTO sensors (type, location) VALUES (%s, %s);" sensors = [('a', 'floor'), ('a', 'ceiling'), ('b', 'floor'), ('b', 'ceiling')] cursor = conn.cursor() for sensor in sensors: try: data = (sensor[0], sensor[1]) cursor.execute(SQL, data) except (Exception, psycopg2.Error) as error: print(error.pgerror) conn.commit() ``` If you choose to use `pgcopy` instead, install the `pgcopy` package [using pip](https://pypi.org/project/pgcopy/), and then add this line to your list of `import` statements: ```python from pgcopy import CopyManager ``` 1. **Generate random sensor data** Use the `generate_series` function provided by PostgreSQL. This example inserts a total of 480 rows of data (4 readings, every 5 minutes, for 24 hours). In your application, this would be the query that saves your time-series data into the hypertable: ```python # for sensors with ids 1-4 for id in range(1, 4, 1): data = (id,) # create random data simulate_query = """SELECT generate_series(now() - interval '24 hour', now(), interval '5 minute') AS time, %s as sensor_id, random()*100 AS temperature, random() AS cpu; """ cursor.execute(simulate_query, data) values = cursor.fetchall() ``` 2. **Define the column names for the target table** This example uses the `sensor_data` hypertable created earlier. This hypertable consists of columns named `time`, `sensor_id`, `temperature` and `cpu`. The column names are defined in a list of strings called `cols`: ```python cols = ['time', 'sensor_id', 'temperature', 'cpu'] ``` 3. **Insert data using pgcopy CopyManager** Create an instance of the `pgcopy` CopyManager, `mgr`, and pass the connection variable, hypertable name, and list of column names. Then use the `copy` function to insert the data quickly. ```python mgr = CopyManager(conn, 'sensor_data', cols) mgr.copy(values) ``` 4. **Commit to persist changes:** ```python conn.commit() ``` 5. **(Optional) Full sample code for pgcopy insert** The full sample code to insert data into TimescaleDB using `pgcopy`, using the example of sensor data from four sensors: ```python # insert using pgcopy def fast_insert(conn): cursor = conn.cursor() # for sensors with ids 1-4 for id in range(1, 4, 1): data = (id,) # create random data simulate_query = """SELECT generate_series(now() - interval '24 hour', now(), interval '5 minute') AS time, %s as sensor_id, random()*100 AS temperature, random() AS cpu; """ cursor.execute(simulate_query, data) values = cursor.fetchall() # column names of the table you're inserting into cols = ['time', 'sensor_id', 'temperature', 'cpu'] # create copy manager with the target table and insert mgr = CopyManager(conn, 'sensor_data', cols) mgr.copy(values) # commit after all sensor data is inserted # could also commit after each sensor insert is done conn.commit() ``` 6. **(Optional) You can also check if the insertion worked:** ```python cursor.execute("SELECT * FROM sensor_data LIMIT 5;") print(cursor.fetchall()) ``` ## Execute a query This section covers how to execute queries against your database. The first procedure shows a simple `SELECT *` query. For more complex queries, you can use prepared statements to ensure queries are executed safely against the database. For more information about properly using placeholders in `psycopg2`, see the [basic module usage document](https://www.psycopg.org/docs/usage.html). For more information about how to execute more complex queries in `psycopg2`, see the [psycopg2 documentation](https://www.psycopg.org/docs/usage.html). ### Execute a query 1. **Define the SQL query** This example is a simple `SELECT` statement querying each row from the previously created `sensor_data` table. ```python query = "SELECT * FROM sensor_data;" ``` 2. **Execute the query** Open a cursor from the existing database connection, `conn`, and then execute the query you defined: ```python cursor = conn.cursor() query = "SELECT * FROM sensor_data;" cursor.execute(query) ``` 3. **Access the query results** Use one of `psycopg2`'s [results retrieval methods](https://www.psycopg.org/docs/cursor.html), such as `fetchall()` or `fetchmany()`. This example prints the results of the query, row by row. Note that the result of `fetchall()` is a list of tuples, so you can handle them accordingly: ```python cursor = conn.cursor() query = "SELECT * FROM sensor_data;" cursor.execute(query) for row in cursor.fetchall(): print(row) cursor.close() ``` 4. **(Optional) Use DictCursor for dictionary results** If you want a list of dictionaries instead, define the cursor using [`DictCursor`](https://www.psycopg.org/docs/extras.html#dictionary-like-cursor): ```python cursor = conn.cursor(cursor_factory=psycopg2.extras.DictCursor) ``` Using this cursor, `cursor.fetchall()` returns a list of dictionary-like objects. For more complex queries, you can use prepared statements to ensure queries are executed safely against the database. ### Execute queries using prepared statements 1. **Write the query using prepared statements:** ```python # query with placeholders cursor = conn.cursor() query = """ SELECT time_bucket('5 minutes', time) AS five_min, avg(cpu) FROM sensor_data JOIN sensors ON sensors.id = sensor_data.sensor_id WHERE sensors.location = %s AND sensors.type = %s GROUP BY five_min ORDER BY five_min DESC; """ location = "floor" sensor_type = "a" data = (location, sensor_type) cursor.execute(query, data) results = cursor.fetchall() ``` * Node.js ## Prerequisites for this tutorial To follow these steps, you'll need: * A [self-hosted TimescaleDB](/docs/get-started/choose-your-path/install-timescaledb) instance. - Your [connection details](/docs/integrate/find-connection-details). * [Node.js](https://nodejs.org) installed. * The Node.js package manager [npm](https://docs.npmjs.com/getting-started) installed. ## Connect to TimescaleDB In this section, you create a connection to TimescaleDB with a common Node.js ORM (object relational mapper) called [Sequelize](https://sequelize.org). 1. **Initialize a new Node.js app** At the command prompt, initialize a new Node.js app: ```bash npm init -y ``` This creates a `package.json` file in your directory, which contains all of the dependencies for your project. It looks something like this: ```json { "name": "node-sample", "version": "1.0.0", "description": "", "main": "index.js", "scripts": { "test": "echo \"Error: no test specified\" && exit 1" }, "keywords": [], "author": "", "license": "ISC" } ``` 2. **Install Express.js** Install Express.js: ```bash npm install express ``` 3. **Create a simple web page** Create a simple web page to check the connection. Create a new file called `index.js`, with this content: ```js const express = require('express') const app = express() const port = 3000; app.use(express.json()); app.get('/', (req, res) => res.send('Hello World!')) app.listen(port, () => console.log(`Example app listening at http://localhost:${port}`)) ``` 4. **Test your connection** Test your connection by starting the application: ```bash node index.js ``` In your web browser, navigate to `http://localhost:3000`. If the connection is successful, it shows "Hello World!" 5. **Add Sequelize to your project** Add Sequelize to your project: ```bash npm install sequelize sequelize-cli pg pg-hstore ``` 6. **Locate your credentials** Locate your TimescaleDB credentials and use them to compose a connection string for Sequelize. You'll need: * password * username * host URL * port * database name 7. **Compose the connection string** Compose your connection string variable, using this format: ```js 'postgres://:@:/' ``` 8. **Configure Sequelize in index.js** Open the `index.js` file you created. Require Sequelize in the application, and declare the connection string: ```js const Sequelize = require('sequelize') const sequelize = new Sequelize('postgres://:@:/', { dialect: 'postgres', protocol: 'postgres', dialectOptions: { ssl: { require: true, rejectUnauthorized: false } } }) ``` Make sure you add the SSL settings in the `dialectOptions` section if your TimescaleDB instance requires SSL connections. 9. **Test the database connection** You can test the connection by adding these lines to `index.js` after the `app.get` statement: ```js sequelize.authenticate().then(() => { console.log('Connection has been established successfully.'); }).catch(err => { console.error('Unable to connect to the database:', err); }); ``` Start the application on the command line: ```bash node index.js ``` If the connection is successful, you'll get output like this: ```bash Example app listening at http://localhost:3000 Executing (default): SELECT 1+1 AS result Connection has been established successfully. ``` ## Create a relational table In this section, you create a relational table called `page_loads`. 1. **Generate the model and migration** Use the Sequelize command line tool to create a table and model called `page_loads`: ```bash npx sequelize model:generate --name page_loads \ --attributes userAgent:string,time:date ``` The output looks similar to this: ```bash Sequelize CLI [Node: 12.16.2, CLI: 5.5.1, ORM: 5.21.11] New model was created at . New migration was created at . ``` 2. **Edit the migration file** Edit the migration file so that it sets up a migration key: ```js 'use strict'; module.exports = { up: async (queryInterface, Sequelize) => { await queryInterface.createTable('page_loads', { userAgent: { primaryKey: true, type: Sequelize.STRING }, time: { primaryKey: true, type: Sequelize.DATE } }); }, down: async (queryInterface, Sequelize) => { await queryInterface.dropTable('page_loads'); } }; ``` 3. **Run the migration** Migrate the change and make sure that it is reflected in the database: ```bash npx sequelize db:migrate ``` The output looks similar to this: ```bash Sequelize CLI [Node: 12.16.2, CLI: 5.5.1, ORM: 5.21.11] Loaded configuration file "config/config.json". Using environment "development". == 20200528195725-create-page-loads: migrating ======= == 20200528195725-create-page-loads: migrated (0.443s) ``` 4. **Create the PageLoads model** Create the `PageLoads` model in your code. In the `index.js` file, above the `app.use` statement, add these lines: ```js let PageLoads = sequelize.define('page_loads', { userAgent: {type: Sequelize.STRING, primaryKey: true }, time: {type: Sequelize.DATE, primaryKey: true } }, { timestamps: false }); ``` 5. **Save data to the database** Instantiate a `PageLoads` object and save it to the database. ## Create a hypertable When you have created the relational table, you can create a hypertable. Creating tables and indexes, altering tables, inserting data, selecting data, and most other tasks are executed on the hypertable. 1. **Generate the hypertable migration** Create a migration to modify the `page_loads` relational table, and change it to a hypertable by first running the following command: ```bash npx sequelize migration:generate --name add_hypertable ``` The output looks similar to this: ```bash Sequelize CLI [Node: 12.16.2, CLI: 5.5.1, ORM: 5.21.11] migrations folder at already exists. New migration was created at /20200601202912-add_hypertable.js . ``` 2. **Add the hypertable creation query** In the `migrations` folder, there is now a new file. Open the file, and add this content: ```js 'use strict'; module.exports = { up: (queryInterface, Sequelize) => { return queryInterface.sequelize.query("SELECT create_hypertable('page_loads', by_range('time'));"); }, down: (queryInterface, Sequelize) => { } }; ``` Note The `by_range` dimension builder is an addition to TimescaleDB 2.13. 3. **Run the hypertable migration** At the command prompt, run the migration command: ```bash npx sequelize db:migrate ``` The output looks similar to this: ```bash Sequelize CLI [Node: 12.16.2, CLI: 5.5.1, ORM: 5.21.11] Loaded configuration file "config/config.json". Using environment "development". == 20200601202912-add_hypertable: migrating ======= == 20200601202912-add_hypertable: migrated (0.426s) ``` ## Insert rows of data This section covers how to insert data into your hypertables. 1. **Modify the route to insert data** In the `index.js` file, modify the `/` route to get the `user-agent` from the request object (`req`) and the current timestamp. Then, call the `create` method on `PageLoads` model, supplying the user agent and timestamp parameters. The `create` call executes an `INSERT` on the database: ```js app.get('/', async (req, res) => { // get the user agent and current time const userAgent = req.get('user-agent'); const time = new Date().getTime(); try { // insert the record await PageLoads.create({ userAgent, time }); // send response res.send('Inserted!'); } catch (e) { console.log('Error inserting data', e) } }) ``` ## Execute a query This section covers how to execute queries against your database. In this example, every time the page is reloaded, all information currently in the table is displayed. 1. **Retrieve and display all data** Modify the `/` route in the `index.js` file to call the Sequelize `findAll` function and retrieve all data from the `page_loads` table using the `PageLoads` model: ```js app.get('/', async (req, res) => { // get the user agent and current time const userAgent = req.get('user-agent'); const time = new Date().getTime(); try { // insert the record await PageLoads.create({ userAgent, time }); // now display everything in the table const messages = await PageLoads.findAll(); res.send(messages); } catch (e) { console.log('Error inserting data', e) } }) ``` Now, when you reload the page, you should see all of the rows currently in the `page_loads` table. * Go ## Prerequisites for this tutorial To follow these steps, you'll need: * A [self-hosted TimescaleDB](/docs/get-started/choose-your-path/install-timescaledb) instance. - Your [connection details](/docs/integrate/find-connection-details). * [Go](https://golang.org/doc/install) installed. * The [PGX driver for Go](https://github.com/jackc/pgx) installed. ## Connect to TimescaleDB In this section, you create a connection to TimescaleDB using the PGX driver. PGX is a toolkit designed to help Go developers work directly with PostgreSQL. You can use it to help your Go application interact directly with TimescaleDB. 1. **Locate your TimescaleDB credentials** Use them to compose a connection string for PGX. You'll need: * password * username * host URL * port number * database name 2. **Compose your connection string variable** Use a [libpq connection string](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING) format: ```go connStr := "postgres://username:password@host:port/dbname" ``` If you require an SSL connection, use this format instead: ```go connStr := "postgres://username:password@host:port/dbname?sslmode=require" ``` 3. **(Optional) Verify your database connection** You can check that you're connected to your database with this hello world program: ```go package main import ( "context" "fmt" "os" "github.com/jackc/pgx/v5" ) //connect to database using a single connection func main() { /***********************************************/ /* Single Connection to TimescaleDB/ PostgreSQL */ /***********************************************/ ctx := context.Background() connStr := "yourConnectionStringHere" conn, err := pgx.Connect(ctx, connStr) if err != nil { fmt.Fprintf(os.Stderr, "Unable to connect to database: %v\n", err) os.Exit(1) } defer conn.Close(ctx) //run a simple query to check our connection var greeting string err = conn.QueryRow(ctx, "select 'Hello, Timescale!'").Scan(&greeting) if err != nil { fmt.Fprintf(os.Stderr, "QueryRow failed: %v\n", err) os.Exit(1) } fmt.Println(greeting) } ``` If you'd like to specify your connection string as an environment variable, you can use this syntax to access it in place of the `connStr` variable: ```go os.Getenv("DATABASE_CONNECTION_STRING") ``` Alternatively, you can connect to TimescaleDB using a connection pool. Connection pooling is useful to conserve computing resources, and can also result in faster database queries: 1. **Create a connection pool** To create a connection pool for concurrent connections to your database, use the `pgxpool.New()` function instead of `pgx.Connect()`. Also note that this script imports `github.com/jackc/pgx/v5/pgxpool`, instead of `pgx/v5` which was used to create a single connection: ```go package main import ( "context" "fmt" "os" "github.com/jackc/pgx/v5/pgxpool" ) func main() { ctx := context.Background() connStr := "yourConnectionStringHere" dbpool, err := pgxpool.New(ctx, connStr) if err != nil { fmt.Fprintf(os.Stderr, "Unable to connect to database: %v\n", err) os.Exit(1) } defer dbpool.Close() //run a simple query to check our connection var greeting string err = dbpool.QueryRow(ctx, "select 'Hello, Tiger Data (but concurrently)'").Scan(&greeting) if err != nil { fmt.Fprintf(os.Stderr, "QueryRow failed: %v\n", err) os.Exit(1) } fmt.Println(greeting) } ``` ## Create a relational table In this section, you create a table called `sensors` which holds the ID, type, and location of your fictional sensors. Additionally, you create a hypertable called `sensor_data` which holds the measurements of those sensors. The measurements contain the time, sensor\_id, temperature reading, and CPU percentage of the sensors. 1. **Compose the SQL statement to create a relational table** This example creates a table called `sensors`, with columns for ID, type, and location: ```go queryCreateTable := `CREATE TABLE sensors (id SERIAL PRIMARY KEY, type VARCHAR(50), location VARCHAR(50));` ``` 2. **Execute the `CREATE TABLE` statement** Use the `Exec()` function on the `dbpool` object, using the arguments of the current context and the statement string you created: ```go package main import ( "context" "fmt" "os" "github.com/jackc/pgx/v5/pgxpool" ) func main() { ctx := context.Background() connStr := "yourConnectionStringHere" dbpool, err := pgxpool.New(ctx, connStr) if err != nil { fmt.Fprintf(os.Stderr, "Unable to connect to database: %v\n", err) os.Exit(1) } defer dbpool.Close() /********************************************/ /* Create relational table */ /********************************************/ //Create relational table called sensors queryCreateTable := `CREATE TABLE sensors (id SERIAL PRIMARY KEY, type VARCHAR(50), location VARCHAR(50));` _, err = dbpool.Exec(ctx, queryCreateTable) if err != nil { fmt.Fprintf(os.Stderr, "Unable to create SENSORS table: %v\n", err) os.Exit(1) } fmt.Println("Successfully created relational table SENSORS") } ``` ## Generate a hypertable When you have created the relational table, you can create a hypertable. Creating tables and indexes, altering tables, inserting data, selecting data, and most other tasks are executed on the hypertable. 1. **Create a variable for the `CREATE TABLE SQL` statement for your hypertable.** Notice how the hypertable has the compulsory time column: ```go queryCreateTable := `CREATE TABLE sensor_data ( time TIMESTAMPTZ NOT NULL, sensor_id INTEGER, temperature DOUBLE PRECISION, cpu DOUBLE PRECISION, FOREIGN KEY (sensor_id) REFERENCES sensors (id)); ` ``` 2. **Formulate the `SELECT` statement to convert the table into a hypertable** You must specify the table name to convert to a hypertable, and its time column name as the second argument. For more information, see the [`create_hypertable` docs](/docs/reference/timescaledb/hypertables/create_hypertable): ```go queryCreateHypertable := `SELECT create_hypertable('sensor_data', by_range('time'));` ``` Note The `by_range` dimension builder is an addition to TimescaleDB 2.13. 3. **Execute the statements to create the hypertable** Execute the `CREATE TABLE` statement and `SELECT` statement which converts the table into a hypertable. You can do this by calling the `Exec()` function on the `dbpool` object, using the arguments of the current context, and the `queryCreateTable` and `queryCreateHypertable` statement strings: ```go package main import ( "context" "fmt" "os" "github.com/jackc/pgx/v5/pgxpool" ) func main() { ctx := context.Background() connStr := "yourConnectionStringHere" dbpool, err := pgxpool.New(ctx, connStr) if err != nil { fmt.Fprintf(os.Stderr, "Unable to connect to database: %v\n", err) os.Exit(1) } defer dbpool.Close() /********************************************/ /* Create Hypertable */ /********************************************/ // Create hypertable of time-series data called sensor_data queryCreateTable := `CREATE TABLE sensor_data ( time TIMESTAMPTZ NOT NULL, sensor_id INTEGER, temperature DOUBLE PRECISION, cpu DOUBLE PRECISION, FOREIGN KEY (sensor_id) REFERENCES sensors (id)); ` queryCreateHypertable := `SELECT create_hypertable('sensor_data', by_range('time'));` //execute statement _, err = dbpool.Exec(ctx, queryCreateTable+queryCreateHypertable) if err != nil { fmt.Fprintf(os.Stderr, "Unable to create the `sensor_data` hypertable: %v\n", err) os.Exit(1) } fmt.Println("Successfully created hypertable `sensor_data`") } ``` ## Insert rows of data You can insert rows into your database in a couple of different ways. Each of these example inserts the data from the two arrays, `sensorTypes` and `sensorLocations`, into the relational table named `sensors`. The first example inserts a single row of data at a time. The second example inserts multiple rows of data. The third example uses batch inserts to speed up the process. 1. **Insert data using prepared statements** Open a connection pool to the database, then use prepared statements to formulate an `INSERT` SQL statement, and execute it: ```go package main import ( "context" "fmt" "os" "github.com/jackc/pgx/v5/pgxpool" ) func main() { ctx := context.Background() connStr := "yourConnectionStringHere" dbpool, err := pgxpool.New(ctx, connStr) if err != nil { fmt.Fprintf(os.Stderr, "Unable to connect to database: %v\n", err) os.Exit(1) } defer dbpool.Close() /********************************************/ /* INSERT into relational table */ /********************************************/ //Insert data into relational table // Slices of sample data to insert // observation i has type sensorTypes[i] and location sensorLocations[i] sensorTypes := []string{"a", "a", "b", "b"} sensorLocations := []string{"floor", "ceiling", "floor", "ceiling"} for i := range sensorTypes { //INSERT statement in SQL queryInsertMetadata := `INSERT INTO sensors (type, location) VALUES ($1, $2);` //Execute INSERT command _, err := dbpool.Exec(ctx, queryInsertMetadata, sensorTypes[i], sensorLocations[i]) if err != nil { fmt.Fprintf(os.Stderr, "Unable to insert data into database: %v\n", err) os.Exit(1) } fmt.Printf("Inserted sensor (%s, %s) into database \n", sensorTypes[i], sensorLocations[i]) } fmt.Println("Successfully inserted all sensors into database") } ``` Instead of inserting a single row of data at a time, you can use this procedure to insert multiple rows of data, instead: 1. **Generate sample time-series data** This example uses PostgreSQL to generate some sample time-series data to insert into the `sensor_data` hypertable. Define the SQL statement to generate the data, called `queryDataGeneration`. Then use the `.Query()` function to execute the statement and return the sample data. The data returned by the query is stored in `results`, a slice of structs, which is then used as a source to insert data into the hypertable: ```go package main import ( "context" "fmt" "os" "time" "github.com/jackc/pgx/v5/pgxpool" ) func main() { ctx := context.Background() connStr := "yourConnectionStringHere" dbpool, err := pgxpool.New(ctx, connStr) if err != nil { fmt.Fprintf(os.Stderr, "Unable to connect to database: %v\n", err) os.Exit(1) } defer dbpool.Close() // Generate data to insert //SQL query to generate sample data queryDataGeneration := ` SELECT generate_series(now() - interval '24 hour', now(), interval '5 minute') AS time, floor(random() * (3) + 1)::int as sensor_id, random()*100 AS temperature, random() AS cpu ` //Execute query to generate samples for sensor_data hypertable rows, err := dbpool.Query(ctx, queryDataGeneration) if err != nil { fmt.Fprintf(os.Stderr, "Unable to generate sensor data: %v\n", err) os.Exit(1) } defer rows.Close() fmt.Println("Successfully generated sensor data") //Store data generated in slice results type result struct { Time time.Time SensorId int Temperature float64 CPU float64 } var results []result for rows.Next() { var r result err = rows.Scan(&r.Time, &r.SensorId, &r.Temperature, &r.CPU) if err != nil { fmt.Fprintf(os.Stderr, "Unable to scan %v\n", err) os.Exit(1) } results = append(results, r) } // Any errors encountered by rows.Next or rows.Scan are returned here if rows.Err() != nil { fmt.Fprintf(os.Stderr, "rows Error: %v\n", rows.Err()) os.Exit(1) } // Check contents of results slice fmt.Println("Contents of RESULTS slice") for i := range results { var r result r = results[i] fmt.Printf("Time: %s | ID: %d | Temperature: %f | CPU: %f |\n", &r.Time, r.SensorId, r.Temperature, r.CPU) } } ``` 2. **Formulate an SQL insert statement for the `sensor_data` hypertable:** ```go //SQL query to generate sample data queryInsertTimeseriesData := ` INSERT INTO sensor_data (time, sensor_id, temperature, cpu) VALUES ($1, $2, $3, $4); ` ``` 3. **Execute the SQL statement for each sample in the results slice:** ```go //Insert contents of results slice into TimescaleDB for i := range results { var r result r = results[i] _, err := dbpool.Exec(ctx, queryInsertTimeseriesData, r.Time, r.SensorId, r.Temperature, r.CPU) if err != nil { fmt.Fprintf(os.Stderr, "Unable to insert sample into TimescaleDB %v\n", err) os.Exit(1) } defer rows.Close() } fmt.Println("Successfully inserted samples into sensor_data hypertable") ``` 4. **(Optional) Full sample code for data generation and insert** This example `main.go` generates sample data and inserts it into the `sensor_data` hypertable: ```go package main import ( "context" "fmt" "os" "time" "github.com/jackc/pgx/v5/pgxpool" ) func main() { /********************************************/ /* Connect using Connection Pool */ /********************************************/ ctx := context.Background() connStr := "yourConnectionStringHere" dbpool, err := pgxpool.New(ctx, connStr) if err != nil { fmt.Fprintf(os.Stderr, "Unable to connect to database: %v\n", err) os.Exit(1) } defer dbpool.Close() /********************************************/ /* Insert data into hypertable */ /********************************************/ // Generate data to insert //SQL query to generate sample data queryDataGeneration := ` SELECT generate_series(now() - interval '24 hour', now(), interval '5 minute') AS time, floor(random() * (3) + 1)::int as sensor_id, random()*100 AS temperature, random() AS cpu ` //Execute query to generate samples for sensor_data hypertable rows, err := dbpool.Query(ctx, queryDataGeneration) if err != nil { fmt.Fprintf(os.Stderr, "Unable to generate sensor data: %v\n", err) os.Exit(1) } defer rows.Close() fmt.Println("Successfully generated sensor data") //Store data generated in slice results type result struct { Time time.Time SensorId int Temperature float64 CPU float64 } var results []result for rows.Next() { var r result err = rows.Scan(&r.Time, &r.SensorId, &r.Temperature, &r.CPU) if err != nil { fmt.Fprintf(os.Stderr, "Unable to scan %v\n", err) os.Exit(1) } results = append(results, r) } // Any errors encountered by rows.Next or rows.Scan are returned here if rows.Err() != nil { fmt.Fprintf(os.Stderr, "rows Error: %v\n", rows.Err()) os.Exit(1) } // Check contents of results slice fmt.Println("Contents of RESULTS slice") for i := range results { var r result r = results[i] fmt.Printf("Time: %s | ID: %d | Temperature: %f | CPU: %f |\n", &r.Time, r.SensorId, r.Temperature, r.CPU) } //Insert contents of results slice into TimescaleDB //SQL query to generate sample data queryInsertTimeseriesData := ` INSERT INTO sensor_data (time, sensor_id, temperature, cpu) VALUES ($1, $2, $3, $4); ` //Insert contents of results slice into TimescaleDB for i := range results { var r result r = results[i] _, err := dbpool.Exec(ctx, queryInsertTimeseriesData, r.Time, r.SensorId, r.Temperature, r.CPU) if err != nil { fmt.Fprintf(os.Stderr, "Unable to insert sample into TimescaleDB %v\n", err) os.Exit(1) } defer rows.Close() } fmt.Println("Successfully inserted samples into sensor_data hypertable") } ``` Inserting multiple rows of data using this method executes as many `insert` statements as there are samples to be inserted. This can make ingestion of data slow. To speed up ingestion, you can batch insert data instead. Here's a sample pattern for how to do so, using the sample data you generated in the previous procedure. It uses the pgx `Batch` object: 1. **This example batch inserts data into the database:** ```go package main import ( "context" "fmt" "os" "time" "github.com/jackc/pgx/v5" "github.com/jackc/pgx/v5/pgxpool" ) func main() { /********************************************/ /* Connect using Connection Pool */ /********************************************/ ctx := context.Background() connStr := "yourConnectionStringHere" dbpool, err := pgxpool.New(ctx, connStr) if err != nil { fmt.Fprintf(os.Stderr, "Unable to connect to database: %v\n", err) os.Exit(1) } defer dbpool.Close() // Generate data to insert //SQL query to generate sample data queryDataGeneration := ` SELECT generate_series(now() - interval '24 hour', now(), interval '5 minute') AS time, floor(random() * (3) + 1)::int as sensor_id, random()*100 AS temperature, random() AS cpu ` //Execute query to generate samples for sensor_data hypertable rows, err := dbpool.Query(ctx, queryDataGeneration) if err != nil { fmt.Fprintf(os.Stderr, "Unable to generate sensor data: %v\n", err) os.Exit(1) } defer rows.Close() fmt.Println("Successfully generated sensor data") //Store data generated in slice results type result struct { Time time.Time SensorId int Temperature float64 CPU float64 } var results []result for rows.Next() { var r result err = rows.Scan(&r.Time, &r.SensorId, &r.Temperature, &r.CPU) if err != nil { fmt.Fprintf(os.Stderr, "Unable to scan %v\n", err) os.Exit(1) } results = append(results, r) } // Any errors encountered by rows.Next or rows.Scan are returned here if rows.Err() != nil { fmt.Fprintf(os.Stderr, "rows Error: %v\n", rows.Err()) os.Exit(1) } // Check contents of results slice /*fmt.Println("Contents of RESULTS slice") for i := range results { var r result r = results[i] fmt.Printf("Time: %s | ID: %d | Temperature: %f | CPU: %f |\n", &r.Time, r.SensorId, r.Temperature, r.CPU) }*/ //Insert contents of results slice into TimescaleDB //SQL query to generate sample data queryInsertTimeseriesData := ` INSERT INTO sensor_data (time, sensor_id, temperature, cpu) VALUES ($1, $2, $3, $4); ` /********************************************/ /* Batch Insert into TimescaleDB */ /********************************************/ //create batch batch := &pgx.Batch{} //load insert statements into batch queue for i := range results { var r result r = results[i] batch.Queue(queryInsertTimeseriesData, r.Time, r.SensorId, r.Temperature, r.CPU) } batch.Queue("select count(*) from sensor_data") //send batch to connection pool br := dbpool.SendBatch(ctx, batch) //execute statements in batch queue _, err = br.Exec() if err != nil { fmt.Fprintf(os.Stderr, "Unable to execute statement in batch queue %v\n", err) os.Exit(1) } fmt.Println("Successfully batch inserted data") //Compare length of results slice to size of table fmt.Printf("size of results: %d\n", len(results)) //check size of table for number of rows inserted // result of last SELECT statement var rowsInserted int err = br.QueryRow().Scan(&rowsInserted) fmt.Printf("size of table: %d\n", rowsInserted) err = br.Close() if err != nil { fmt.Fprintf(os.Stderr, "Unable to closer batch %v\n", err) os.Exit(1) } } ``` ## Execute a query This section covers how to execute queries against your database. 1. **Define the SQL query** This example uses a SQL query that combines time-series and relational data. It returns the average CPU values for every 5 minute interval, for sensors located on location `ceiling` and of type `a`: ```go // Formulate query in SQL // Note the use of prepared statement placeholders $1 and $2 queryTimebucketFiveMin := ` SELECT time_bucket('5 minutes', time) AS five_min, avg(cpu) FROM sensor_data JOIN sensors ON sensors.id = sensor_data.sensor_id WHERE sensors.location = $1 AND sensors.type = $2 GROUP BY five_min ORDER BY five_min DESC; ` ``` 2. **Execute the query** Use the `.Query()` function to execute the query string. Make sure you specify the relevant placeholders: ```go //Execute query on TimescaleDB rows, err := dbpool.Query(ctx, queryTimebucketFiveMin, "ceiling", "a") if err != nil { fmt.Fprintf(os.Stderr, "Unable to execute query %v\n", err) os.Exit(1) } defer rows.Close() fmt.Println("Successfully executed query") ``` 3. **Access the returned rows** Create a struct with fields representing the columns that you expect to be returned, then use the `rows.Next()` function to iterate through the rows returned and fill `results` with the array of structs. This uses the `rows.Scan()` function, passing in pointers to the fields that you want to scan for results. This example prints out the results returned from the query, but you might want to use those results for some other purpose. Once you've scanned through all the rows returned you can then use the results array however you like. ```go //Do something with the results of query // Struct for results type result2 struct { Bucket time.Time Avg float64 } // Print rows returned and fill up results slice for later use var results []result2 for rows.Next() { var r result2 err = rows.Scan(&r.Bucket, &r.Avg) if err != nil { fmt.Fprintf(os.Stderr, "Unable to scan %v\n", err) os.Exit(1) } results = append(results, r) fmt.Printf("Time bucket: %s | Avg: %f\n", &r.Bucket, r.Avg) } // Any errors encountered by rows.Next or rows.Scan are returned here if rows.Err() != nil { fmt.Fprintf(os.Stderr, "rows Error: %v\n", rows.Err()) os.Exit(1) } // use results here… ``` 4. **(Optional) Full sample code for querying** This example program runs a query, and accesses the results of that query: ```go package main import ( "context" "fmt" "os" "time" "github.com/jackc/pgx/v5/pgxpool" ) func main() { ctx := context.Background() connStr := "yourConnectionStringHere" dbpool, err := pgxpool.New(ctx, connStr) if err != nil { fmt.Fprintf(os.Stderr, "Unable to connect to database: %v\n", err) os.Exit(1) } defer dbpool.Close() /********************************************/ /* Execute a query */ /********************************************/ // Formulate query in SQL // Note the use of prepared statement placeholders $1 and $2 queryTimebucketFiveMin := ` SELECT time_bucket('5 minutes', time) AS five_min, avg(cpu) FROM sensor_data JOIN sensors ON sensors.id = sensor_data.sensor_id WHERE sensors.location = $1 AND sensors.type = $2 GROUP BY five_min ORDER BY five_min DESC; ` //Execute query on TimescaleDB rows, err := dbpool.Query(ctx, queryTimebucketFiveMin, "ceiling", "a") if err != nil { fmt.Fprintf(os.Stderr, "Unable to execute query %v\n", err) os.Exit(1) } defer rows.Close() fmt.Println("Successfully executed query") //Do something with the results of query // Struct for results type result2 struct { Bucket time.Time Avg float64 } // Print rows returned and fill up results slice for later use var results []result2 for rows.Next() { var r result2 err = rows.Scan(&r.Bucket, &r.Avg) if err != nil { fmt.Fprintf(os.Stderr, "Unable to scan %v\n", err) os.Exit(1) } results = append(results, r) fmt.Printf("Time bucket: %s | Avg: %f\n", &r.Bucket, r.Avg) } // Any errors encountered by rows.Next or rows.Scan are returned here if rows.Err() != nil { fmt.Fprintf(os.Stderr, "rows Error: %v\n", rows.Err()) os.Exit(1) } } ``` ## Next steps Now that you're able to connect, read, and write to a TimescaleDB instance from your Go application, check out these resources: * Refer to the [pgx documentation](https://pkg.go.dev/github.com/jackc/pgx) for more information about pgx. * Want fast inserts on CSV data? Check out [TimescaleDB parallel copy](https://github.com/timescale/timescaledb-parallel-copy), a tool for fast inserts, written in Go. * Java ## Prerequisites for this tutorial To follow these steps, you'll need: * A [self-hosted TimescaleDB](/docs/get-started/choose-your-path/install-timescaledb) instance. - Your [connection details](/docs/integrate/find-connection-details). * The [Java Development Kit (JDK)](https://openjdk.java.net) installed. * The [PostgreSQL JDBC driver](https://jdbc.postgresql.org) installed. All code in this quick start is for Java 16 and later. If you are working with older JDK versions, use legacy coding techniques. ## Connect to TimescaleDB In this section, you create a connection to your TimescaleDB database using an application in a single file. You can use any of your favorite build tools, including `gradle` or `maven`. 1. **Create the Main.java file** Create a directory containing a text file called `Main.java`, with this content: ```java package com.timescale.java; public class Main { public static void main(String... args) { System.out.println("Hello, World!"); } } ``` 2. **Run the application** From the command line in the current directory, run the application: ```bash java Main.java ``` If the command is successful, `Hello, World!` line output is printed to your console. 3. **Import the PostgreSQL JDBC driver** If you are using a dependency manager, include the [PostgreSQL JDBC Driver](https://mvnrepository.com/artifact/org.postgresql/postgresql) as a dependency. 4. **Download the JDBC Driver JAR** Download the [JAR artifact of the JDBC Driver](https://jdbc.postgresql.org/download/) and save it with the `Main.java` file. 5. **Import the JDBC Driver into the application** Import the `JDBC Driver` into the Java application and display a list of available drivers for the check: ```java package com.timescale.java; import java.sql.DriverManager; public class Main { public static void main(String... args) { DriverManager.drivers().forEach(System.out::println); } } ``` 6. **Run the examples** Run all the examples: ```bash java -cp *.jar Main.java ``` If the command is successful, a string similar to `org.postgresql.Driver@7f77e91b` is printed to your console. This means that you are ready to connect to TimescaleDB from Java. 7. **Locate your TimescaleDB credentials** Locate your TimescaleDB credentials and use them to compose a connection string for JDBC. You'll need: * password * username * host URL * port * database name 8. **Compose the connection string** Compose your connection string variable, using this format: ```java var connUrl = "jdbc:postgresql://:/?user=&password="; ``` For more information about creating connection strings, see the [JDBC documentation](https://jdbc.postgresql.org/documentation/datasource/). Warning This method of composing a connection string is for test or development purposes only. For production, use environment variables for sensitive details like your password, hostname, and port number. ```java package com.timescale.java; import java.sql.DriverManager; import java.sql.SQLException; public class Main { public static void main(String... args) throws SQLException { var connUrl = "jdbc:postgresql://:/?user=&password="; var conn = DriverManager.getConnection(connUrl); System.out.println(conn.getClientInfo()); } } ``` 9. **Run the code** Run the code: ```bash java -cp *.jar Main.java ``` If the command is successful, a string similar to `{ApplicationName=PostgreSQL JDBC Driver}` is printed to your console. ## Create a relational table In this section, you create a table called `sensors` which holds the ID, type, and location of your fictional sensors. Additionally, you create a hypertable called `sensor_data` which holds the measurements of those sensors. The measurements contain the time, sensor\_id, temperature reading, and CPU percentage of the sensors. 1. **Compose the `CREATE TABLE` statement** Compose a string which contains the SQL statement to create a relational table. This example creates a table called `sensors`, with columns `id`, `type` and `location`: ```sql CREATE TABLE sensors ( id SERIAL PRIMARY KEY, type TEXT NOT NULL, location TEXT NOT NULL ); ``` 2. **Execute the query and verify** Create a statement, execute the query you created in the previous step, and check that the table was created successfully: ```java package com.timescale.java; import java.sql.DriverManager; import java.sql.SQLException; public class Main { public static void main(String... args) throws SQLException { var connUrl = "jdbc:postgresql://:/?user=&password="; var conn = DriverManager.getConnection(connUrl); var createSensorTableQuery = """ CREATE TABLE sensors ( id SERIAL PRIMARY KEY, type TEXT NOT NULL, location TEXT NOT NULL ) """; try (var stmt = conn.createStatement()) { stmt.execute(createSensorTableQuery); } var showAllTablesQuery = "SELECT tablename FROM pg_catalog.pg_tables WHERE schemaname = 'public'"; try (var stmt = conn.createStatement(); var rs = stmt.executeQuery(showAllTablesQuery)) { System.out.println("Tables in the current database: "); while (rs.next()) { System.out.println(rs.getString("tablename")); } } } } ``` ## Create a hypertable When you have created the relational table, you can create a hypertable. Creating tables and indexes, altering tables, inserting data, selecting data, and most other tasks are executed on the hypertable. 1. **Write the `CREATE TABLE` statement for the hypertable** Create a `CREATE TABLE` SQL statement for your hypertable. Notice how the hypertable has the compulsory time column: ```sql CREATE TABLE sensor_data ( time TIMESTAMPTZ NOT NULL, sensor_id INTEGER REFERENCES sensors (id), value DOUBLE PRECISION ); ``` 2. **Convert the table to a hypertable** Create a statement, execute the query you created in the previous step: ```sql SELECT create_hypertable('sensor_data', by_range('time')); ``` Note The `by_range` and `by_hash` dimension builder is an addition to TimescaleDB 2.13. 3. **Execute and commit the schema changes** Execute the two statements you created, and commit your changes to the database: ```java package com.timescale.java; import java.sql.Connection; import java.sql.DriverManager; import java.sql.SQLException; import java.util.List; public class Main { public static void main(String... args) { final var connUrl = "jdbc:postgresql://:/?user=&password="; try (var conn = DriverManager.getConnection(connUrl)) { createSchema(conn); insertData(conn); } catch (SQLException ex) { System.err.println(ex.getMessage()); } } private static void createSchema(final Connection conn) throws SQLException { try (var stmt = conn.createStatement()) { stmt.execute(""" CREATE TABLE sensors ( id SERIAL PRIMARY KEY, type TEXT NOT NULL, location TEXT NOT NULL ) """); } try (var stmt = conn.createStatement()) { stmt.execute(""" CREATE TABLE sensor_data ( time TIMESTAMPTZ NOT NULL, sensor_id INTEGER REFERENCES sensors (id), value DOUBLE PRECISION ) """); } try (var stmt = conn.createStatement()) { stmt.execute("SELECT create_hypertable('sensor_data', by_range('time'))"); } } } ``` ## Insert data You can insert data into your hypertables in several different ways. In this section, you can insert single rows, or insert by batches of rows. 1. **Insert rows with prepared statements** Open a connection to the database, use prepared statements to formulate the `INSERT` SQL statement, then execute the statement: ```java final List sensors = List.of( new Sensor("temperature", "bedroom"), new Sensor("temperature", "living room"), new Sensor("temperature", "outside"), new Sensor("humidity", "kitchen"), new Sensor("humidity", "outside")); for (final var sensor : sensors) { try (var stmt = conn.prepareStatement("INSERT INTO sensors (type, location) VALUES (?, ?)")) { stmt.setString(1, sensor.type()); stmt.setString(2, sensor.location()); stmt.executeUpdate(); } } ``` If you want to insert a batch of rows by using a batching mechanism. In this example, you generate some sample time-series data to insert into the `sensor_data` hypertable: 1. **Insert batches of rows** Insert batches of rows: ```java final var sensorDataCount = 100; final var insertBatchSize = 10; try (var stmt = conn.prepareStatement(""" INSERT INTO sensor_data (time, sensor_id, value) VALUES ( generate_series(now() - INTERVAL '24 hours', now(), INTERVAL '5 minutes'), floor(random() * 4 + 1)::INTEGER, random() ) """)) { for (int i = 0; i < sensorDataCount; i++) { stmt.addBatch(); if ((i > 0 && i % insertBatchSize == 0) || i == sensorDataCount - 1) { stmt.executeBatch(); } } } ``` ## Execute a query This section covers how to execute queries against your database. 1. **Define the SQL query** Define the SQL query you'd like to run on the database. This example combines time-series and relational data. It returns the average values for every 15 minute interval for sensors with specific type and location. ```sql SELECT time_bucket('15 minutes', time) AS bucket, avg(value) FROM sensor_data JOIN sensors ON sensors.id = sensor_data.sensor_id WHERE sensors.type = ? AND sensors.location = ? GROUP BY bucket ORDER BY bucket DESC; ``` 2. **Execute the query and read results** Execute the query with the prepared statement and read out the result set for all `a`-type sensors located on the `floor`: ```java try (var stmt = conn.prepareStatement(""" SELECT time_bucket('15 minutes', time) AS bucket, avg(value) FROM sensor_data JOIN sensors ON sensors.id = sensor_data.sensor_id WHERE sensors.type = ? AND sensors.location = ? GROUP BY bucket ORDER BY bucket DESC """)) { stmt.setString(1, "temperature"); stmt.setString(2, "living room"); try (var rs = stmt.executeQuery()) { while (rs.next()) { System.out.printf("%s: %f%n", rs.getTimestamp(1), rs.getDouble(2)); } } } ``` If the command is successful, you'll see output like this: ```bash 2021-05-12 23:30:00.0: 0,508649 2021-05-12 23:15:00.0: 0,477852 2021-05-12 23:00:00.0: 0,462298 2021-05-12 22:45:00.0: 0,457006 2021-05-12 22:30:00.0: 0,568744 ... ``` ## Complete code samples This section contains complete code samples. ### Complete code sample ```java package com.timescale.java; import java.sql.Connection; import java.sql.DriverManager; import java.sql.SQLException; import java.util.List; public class Main { public static void main(String... args) { final var connUrl = "jdbc:postgresql://:/?user=&password="; try (var conn = DriverManager.getConnection(connUrl)) { createSchema(conn); insertData(conn); } catch (SQLException ex) { System.err.println(ex.getMessage()); } } private static void createSchema(final Connection conn) throws SQLException { try (var stmt = conn.createStatement()) { stmt.execute(""" CREATE TABLE sensors ( id SERIAL PRIMARY KEY, type TEXT NOT NULL, location TEXT NOT NULL ) """); } try (var stmt = conn.createStatement()) { stmt.execute(""" CREATE TABLE sensor_data ( time TIMESTAMPTZ NOT NULL, sensor_id INTEGER REFERENCES sensors (id), value DOUBLE PRECISION ) """); } try (var stmt = conn.createStatement()) { stmt.execute("SELECT create_hypertable('sensor_data', by_range('time'))"); } } private static void insertData(final Connection conn) throws SQLException { final List sensors = List.of( new Sensor("temperature", "bedroom"), new Sensor("temperature", "living room"), new Sensor("temperature", "outside"), new Sensor("humidity", "kitchen"), new Sensor("humidity", "outside")); for (final var sensor : sensors) { try (var stmt = conn.prepareStatement("INSERT INTO sensors (type, location) VALUES (?, ?)")) { stmt.setString(1, sensor.type()); stmt.setString(2, sensor.location()); stmt.executeUpdate(); } } final var sensorDataCount = 100; final var insertBatchSize = 10; try (var stmt = conn.prepareStatement(""" INSERT INTO sensor_data (time, sensor_id, value) VALUES ( generate_series(now() - INTERVAL '24 hours', now(), INTERVAL '5 minutes'), floor(random() * 4 + 1)::INTEGER, random() ) """)) { for (int i = 0; i < sensorDataCount; i++) { stmt.addBatch(); if ((i > 0 && i % insertBatchSize == 0) || i == sensorDataCount - 1) { stmt.executeBatch(); } } } } private record Sensor(String type, String location) { } } ``` ### Execute more complex queries ```java package com.timescale.java; import java.sql.Connection; import java.sql.DriverManager; import java.sql.SQLException; import java.util.List; public class Main { public static void main(String... args) { final var connUrl = "jdbc:postgresql://:/?user=&password="; try (var conn = DriverManager.getConnection(connUrl)) { createSchema(conn); insertData(conn); executeQueries(conn); } catch (SQLException ex) { System.err.println(ex.getMessage()); } } private static void createSchema(final Connection conn) throws SQLException { try (var stmt = conn.createStatement()) { stmt.execute(""" CREATE TABLE sensors ( id SERIAL PRIMARY KEY, type TEXT NOT NULL, location TEXT NOT NULL ) """); } try (var stmt = conn.createStatement()) { stmt.execute(""" CREATE TABLE sensor_data ( time TIMESTAMPTZ NOT NULL, sensor_id INTEGER REFERENCES sensors (id), value DOUBLE PRECISION ) """); } try (var stmt = conn.createStatement()) { stmt.execute("SELECT create_hypertable('sensor_data', by_range('time'))"); } } private static void insertData(final Connection conn) throws SQLException { final List sensors = List.of( new Sensor("temperature", "bedroom"), new Sensor("temperature", "living room"), new Sensor("temperature", "outside"), new Sensor("humidity", "kitchen"), new Sensor("humidity", "outside")); for (final var sensor : sensors) { try (var stmt = conn.prepareStatement("INSERT INTO sensors (type, location) VALUES (?, ?)")) { stmt.setString(1, sensor.type()); stmt.setString(2, sensor.location()); stmt.executeUpdate(); } } final var sensorDataCount = 100; final var insertBatchSize = 10; try (var stmt = conn.prepareStatement(""" INSERT INTO sensor_data (time, sensor_id, value) VALUES ( generate_series(now() - INTERVAL '24 hours', now(), INTERVAL '5 minutes'), floor(random() * 4 + 1)::INTEGER, random() ) """)) { for (int i = 0; i < sensorDataCount; i++) { stmt.addBatch(); if ((i > 0 && i % insertBatchSize == 0) || i == sensorDataCount - 1) { stmt.executeBatch(); } } } } private static void executeQueries(final Connection conn) throws SQLException { try (var stmt = conn.prepareStatement(""" SELECT time_bucket('15 minutes', time) AS bucket, avg(value) FROM sensor_data JOIN sensors ON sensors.id = sensor_data.sensor_id WHERE sensors.type = ? AND sensors.location = ? GROUP BY bucket ORDER BY bucket DESC """)) { stmt.setString(1, "temperature"); stmt.setString(2, "living room"); try (var rs = stmt.executeQuery()) { while (rs.next()) { System.out.printf("%s: %f%n", rs.getTimestamp(1), rs.getDouble(2)); } } } } private record Sensor(String type, String location) { } } ``` ## Next steps Now that you're able to connect, read, and write to a TimescaleDB instance from your Java application, and generate the scaffolding necessary to build a new application from an existing TimescaleDB instance, be sure to check out these advanced TimescaleDB tutorials: * [Continuous aggregates](/docs/build/continuous-aggregates/create-a-continuous-aggregate) * [Migrate your data](/docs/migrate) You are not limited to these languages! You can interface with TimescaleDB using any [PostgreSQL client driver](https://wiki.postgresql.org/wiki/List_of_drivers). # Create a Tiger Cloud service > Create a Tiger Cloud account and your first service in Tiger Console Tips You can also create a service from the command line with Tiger CLI. See [Get started with Tiger CLI](/docs/get-started/quickstart/tiger-cli). * Tiger Cloud on AWS Tiger Cloud is the modern PostgreSQL data platform for all your applications. It enhances PostgreSQL to handle time series, events, real-time analytics, and vector search, all in a single database alongside transactional workloads. You get one system that handles live data ingestion, late and out-of-order updates, and low latency queries, with the performance, reliability, and scalability your app needs. Ideal for IoT, crypto, finance, SaaS, and a myriad other domains, Tiger Cloud allows you to build data-heavy, mission-critical apps while retaining the familiarity and reliability of PostgreSQL. ## What is a Tiger Cloud service? [Section titled “What is a Tiger Cloud service?”](#what-is-a-tiger-cloud-service) A Tiger Cloud service is a single optimized PostgreSQL instance extended with innovations in the database engine and cloud infrastructure to deliver speed without sacrifice. A Tiger Cloud service is 10-1000x faster at scale! It is ideal for applications requiring strong data consistency, complex relationships, and advanced querying capabilities. Get ACID compliance, extensive SQL support, JSON handling, and extensibility through custom functions, data types, and extensions. Each service is associated with a project in Tiger Cloud. Each project can have multiple services. Each user is a [member of one or more projects](/docs/deploy/tiger-cloud/tiger-cloud-aws/security/members). You create free and standard services in Tiger Console. A standard service comes with the full range of features according to your pricing plan. A free service comes at zero cost and gives you limited features and resources. ![Tiger Cloud pricing plans: Performance, Scale, and Enterprise](/docs/_astro/tiger-pricing.D4VriqHo_Z1ITAcw.webp) ![Tiger Cloud pricing plans: Performance, Scale, and Enterprise](/docs/_astro/tiger-pricing-dark.CAPZJelR_1rI8TC.webp) To the PostgreSQL you know and love, Tiger Cloud adds the following capabilities: * **Standard services**: * *Real-time analytics*: store and query [time-series data](https://www.tigerdata.com/blog/time-series-database-an-explainer#what-is-a-time-series-database) at scale for real-time analytics and other use cases. Get faster time-based queries with hypertables, continuous aggregates, and columnar storage. Save money by compressing data into the columnstore, moving cold data to low-cost bottomless storage in Amazon S3 or Azure Blob storage, and deleting old data with automated policies. * *AI-focused*: build AI applications from start to scale. Get fast and accurate similarity search with the pgvector and pgvectorscale extensions. * *Hybrid applications*: get a full set of tools to develop applications that combine time-based data and AI. All standard Tiger Cloud services include the tooling you expect for production and developer environments: [live migration](/docs/migrate/live-migration), [automatic backups and PITR](/docs/deploy/tiger-cloud/tiger-cloud-aws/high-availability/backup-restore), [high availability](/docs/deploy/tiger-cloud/tiger-cloud-aws/high-availability/high-availability), [read replicas](/docs/deploy/tiger-cloud/tiger-cloud-aws/high-availability/read-scaling), [data forking](/docs/deploy/tiger-cloud/tiger-cloud-aws/service-management/fork-services), [connection pooling](/docs/deploy/tiger-cloud/tiger-cloud-aws/service-management/connection-pooling), [tiered storage](/docs/build/data-management/storage/manage-storage), [usage-based storage](/docs/deploy/tiger-cloud/tiger-cloud-aws/pricing-and-account-management#how-your-bill-is-calculated), secure in-Tiger Console [SQL editing](/docs/build/data-management/run-queries-from-tiger-console), service [metrics](/docs/deploy/tiger-cloud/tiger-cloud-aws/monitoring#metrics) and [insights](/docs/deploy/tiger-cloud/tiger-cloud-aws/monitoring#insights), streamlined maintenance, and much more. Tiger Cloud continuously monitors your services and prevents common PostgreSQL out-of-memory crashes. * **Free services**: *PostgreSQL with TimescaleDB and vector extensions* Free services come with pre-configured CPU and memory, are hosted in `us-east-1` region, and offer limited resources with a basic feature scope. This excludes connection pooling, replication, data tiering, exporters, metrics, VPC, and other features. Free services are perfect for schema explorations, syntax validations, and other small-scale testing. Once you reach the storage limit, a free service enters a read-only state. You can [convert your free service](/docs/deploy/tiger-cloud/tiger-cloud-aws/service-management/service-management#convert-a-free-service-to-a-standard-one) to a standard one at any time. Note Free services are currently in beta. You can [manage, pause, or delete](/docs/deploy/tiger-cloud/tiger-cloud-aws/service-management/service-management) your service at any time from Tiger Console. To start using Tiger Cloud for your data: 1. [Create a Tiger Cloud account](/docs/get-started/quickstart/create-service#create-a-tiger-cloud-account): register to get access to Tiger Console as a centralized point to administer and interact with your data. 2. [Create a Tiger Cloud service](/docs/get-started/quickstart/create-service#create-a-service): that is, a PostgreSQL database instance, powered by [TimescaleDB](https://www.tigerdata.com/docs/#TimescaleDB), built for production, and extended with cloud features like transparent data tiering to object storage. 3. [Connect to your Tiger Cloud service](/docs/get-started/quickstart/create-service#connect-to-your-service): to run queries, add and migrate your data from other sources. ## Create a Tiger Cloud account [Section titled “Create a Tiger Cloud account”](#create-a-tiger-cloud-account) You create a Tiger Cloud account to manage your services and data in a centralized and efficient manner in Tiger Console. From there, you can create and delete services, run queries, manage access and billing, integrate other services, contact support, and more. * Tiger Cloud You create a standalone account to manage Tiger Cloud as a separate unit in your infrastructure, which includes separate billing and invoicing. To set up Tiger Cloud: 1. **Sign up for a $1000-credit, 30-day free trial** Open [Sign up for Tiger Cloud](https://console.cloud.tigerdata.com/signup) and add your details, then click `Start your free trial`. You receive a confirmation email in your inbox. 2. **Confirm your email address** In the confirmation email, click the link supplied. 3. **Select the pricing plan** You are now logged into Tiger Console. You can change the pricing plan to better accommodate your growing needs on the [`Billing` page](https://console.cloud.tigerdata.com/dashboard/billing/plans). * AWS Marketplace To have Tiger Cloud as a part of your AWS infrastructure, you create or link your Tiger Cloud account through AWS Marketplace. In this case, Tiger Cloud is a line item in your AWS invoice. To set up Tiger Cloud via AWS: 1. **Open AWS Marketplace and search for `Tiger Cloud`** You see two pricing options, [pay-as-you-go](https://aws.amazon.com/marketplace/pp/prodview-iestawpo5ihca?applicationId=AWSMPContessa\&ref_=beagle\&sr=0-1) and [annual commit](https://aws.amazon.com/marketplace/pp/prodview-ezxwlmjyr6x4u?applicationId=AWSMPContessa\&ref_=beagle\&sr=0-2). 2. **Select the pricing option that suits you and click `View purchase options`** 3. **Review and configure the purchase details, then click `Subscribe`** 4. **Click `Set up your account` at the top of the page** You are redirected to Tiger Console. 5. **Sign up for a $1000-credit, 30-day free trial** Add your details, then click `Start your free trial`. If you want to link an existing Tiger Cloud account to AWS, log in with your existing credentials. 6. **Select the pricing plan** You are now logged into Tiger Console. You can change the pricing plan later to better accommodate your growing needs on the [`Billing` page](https://console.cloud.tigerdata.com/dashboard/billing/plans). 7. **In `Confirm AWS Marketplace connection`, click `Connect`** Your Tiger Cloud and AWS accounts are now connected. ## Create a service [Section titled “Create a service”](#create-a-service) Now that you have an active Tiger Cloud account, you create and manage your services in Tiger Console. When you create a service, you effectively create a blank PostgreSQL database with additional Tiger Cloud features available under your pricing plan. You then add or migrate your data into this database. 1. **In the service creation page, click `+ New service`.** Follow the wizard to configure your service depending on its type. 2. **Click `Create service`.** Your service is constructed and ready to use in a few seconds. 3. **Click `Download the config` and store the configuration information you need to connect to this service in a secure location.** This file contains the passwords and configuration information you need to connect to your service using the Tiger Console Data view, from the command line, or using third-party database administration tools. You can [manage, pause, or delete](/docs/deploy/tiger-cloud/tiger-cloud-aws/service-management/service-management) your service at any time from Tiger Console. ## Connect to your service [Section titled “Connect to your service”](#connect-to-your-service) To run queries and perform other operations, connect to your service: 1. **Check your service is running correctly** In [Tiger Console](https://console.cloud.tigerdata.com/dashboard/services), check that your service is marked as `Running`. ![Tiger Console services list showing a running service](/docs/_astro/tiger-console-services-view.F-zo-hID_1rY9dD.webp) 2. **Connect to your service** Connect using Data view or SQL editor in Tiger Console, or psql in the command line: * Data view in Console 1. In Tiger Console, toggle `Data view`. 2. In the connection drop-down in the top right, select your service. ![Select a connection in Tiger Cloud data mode](/docs/_astro/tiger-console-data-mode-connection-dropdown.eS_sjlnj_1J2XiX.webp) 3. Run a test query: ```sql SELECT CURRENT_DATE; ``` This query gives you the current date, you have successfully connected to your service. And that is it, you are up and running. Enjoy developing with Tiger Data. * SQL editor in Console 1. In Tiger Console, select your service. 2. Click `SQL Editor` at the bottom. ![SQL editor in Tiger Console Ops mode](/docs/_astro/tiger-console-ops-mode-sql-editor-empty.D6fPmf48_LfDRl.webp) 3. Run a test query: ```sql SELECT CURRENT_DATE; ``` This query gives you the current date, you have successfully connected to your service. And that is it, you are up and running. Enjoy developing with Tiger Data. * psql on the command line 1. Install psql. 2. Run the following command in the terminal using the service URL from the config file you have saved during service creation: ```plaintext psql "" ``` 3. Run a test query: ```sql SELECT CURRENT_DATE; ``` This query returns the current date. You have successfully connected to your service. And that is it, you are up and running. Enjoy developing with Tiger Data. Quick recap. You: * Manage your services in the Ops view in Tiger Console: add read replicas and enable high availability, compress data into the columnstore, change parameters, and so on. * Analyze your data in the [Data view](https://console.cloud.tigerdata.com/dashboard/data-view?popsql) in Tiger Console: write queries with autocomplete, save them in folders, share them, create charts/dashboards, and much more. * Store configuration and security information in your config file. * Tiger Cloud on Azure You create a standalone account to manage Tiger Cloud as a separate unit in your infrastructure, which includes separate billing and invoicing. To set up Tiger Cloud: 1. **Sign up for a $1000-credit, 30-day free trial** Open [Sign up for Tiger Cloud](https://console.cloud.tigerdata.com/signup) and add your details, then click `Start your free trial`. You receive a confirmation email in your inbox. 2. **Confirm your email address** In the confirmation email, click the link supplied. 3. **Select the pricing plan** You are now logged into Tiger Console. You can change the pricing plan to better accommodate your growing needs on the [`Billing` page](https://console.cloud.tigerdata.com/dashboard/billing/plans). * Tiger Cloud To have Tiger Cloud as a part of your AWS infrastructure, you create or link your Tiger Cloud account through AWS Marketplace. In this case, Tiger Cloud is a line item in your AWS invoice. To set up Tiger Cloud via AWS: 1. **Open AWS Marketplace and search for `Tiger Cloud`** You see two pricing options, [pay-as-you-go](https://aws.amazon.com/marketplace/pp/prodview-iestawpo5ihca?applicationId=AWSMPContessa\&ref_=beagle\&sr=0-1) and [annual commit](https://aws.amazon.com/marketplace/pp/prodview-ezxwlmjyr6x4u?applicationId=AWSMPContessa\&ref_=beagle\&sr=0-2). 2. **Select the pricing option that suits you and click `View purchase options`** 3. **Review and configure the purchase details, then click `Subscribe`** 4. **Click `Set up your account` at the top of the page** You are redirected to Tiger Console. 5. **Sign up for a $1000-credit, 30-day free trial** Add your details, then click `Start your free trial`. If you want to link an existing Tiger Cloud account to AWS, log in with your existing credentials. 6. **Select the pricing plan** You are now logged into Tiger Console. You can change the pricing plan later to better accommodate your growing needs on the [`Billing` page](https://console.cloud.tigerdata.com/dashboard/billing/plans). 7. **In `Confirm AWS Marketplace connection`, click `Connect`** Your Tiger Cloud and AWS accounts are now connected. * AWS Marketplace 1. In Tiger Console, toggle `Data view`. 2. In the connection drop-down in the top right, select your service. ![Select a connection in Tiger Cloud data mode](/docs/_astro/tiger-console-data-mode-connection-dropdown.eS_sjlnj_1J2XiX.webp) 3. Run a test query: ```sql SELECT CURRENT_DATE; ``` This query gives you the current date, you have successfully connected to your service. And that is it, you are up and running. Enjoy developing with Tiger Data. * Data view in Console 1. In Tiger Console, select your service. 2. Click `SQL Editor` at the bottom. ![SQL editor in Tiger Console Ops mode](/docs/_astro/tiger-console-ops-mode-sql-editor-empty.D6fPmf48_LfDRl.webp) 3. Run a test query: ```sql SELECT CURRENT_DATE; ``` This query gives you the current date, you have successfully connected to your service. And that is it, you are up and running. Enjoy developing with Tiger Data. * SQL editor in Console 1. Install psql. 2. Run the following command in the terminal using the service URL from the config file you have saved during service creation: ```plaintext psql "" ``` 3. Run a test query: ```sql SELECT CURRENT_DATE; ``` This query returns the current date. You have successfully connected to your service. And that is it, you are up and running. Enjoy developing with Tiger Data. * psql on the command line Tiger Cloud is the modern PostgreSQL data platform for all your applications. It enhances PostgreSQL to handle time series, events, real-time analytics, and vector search, all in a single database alongside transactional workloads. You get one system that handles live data ingestion, late and out-of-order updates, and low latency queries, with the performance, reliability, and scalability your app needs. Ideal for IoT, crypto, finance, SaaS, and a myriad other domains, Tiger Cloud allows you to build data-heavy, mission-critical apps while retaining the familiarity and reliability of PostgreSQL. ## What is a Tiger Cloud service? [Section titled “What is a Tiger Cloud service?”](#what-is-a-tiger-cloud-service-1) A Tiger Cloud service is a single optimized PostgreSQL instance extended with innovations in the database engine and cloud infrastructure to deliver speed without sacrifice. A Tiger Cloud service is 10-1000x faster at scale! It is ideal for applications requiring strong data consistency, complex relationships, and advanced querying capabilities. Get ACID compliance, extensive SQL support, JSON handling, and extensibility through custom functions, data types, and extensions. Each service is associated with a project in Tiger Cloud. Each project can have multiple services. Each user is a [member of one or more projects](/docs/deploy/tiger-cloud/tiger-cloud-aws/security/members). You create free and standard services in Tiger Console. A standard service comes with the full range of features according to your pricing plan. A free service comes at zero cost and gives you limited features and resources. ![Tiger Cloud pricing plans: Performance, Scale, and Enterprise](/docs/_astro/tiger-pricing.D4VriqHo_Z1ITAcw.webp) ![Tiger Cloud pricing plans: Performance, Scale, and Enterprise](/docs/_astro/tiger-pricing-dark.CAPZJelR_1rI8TC.webp) To the PostgreSQL you know and love, Tiger Cloud adds the following capabilities: * **Standard services**: * *Real-time analytics*: store and query [time-series data](https://www.tigerdata.com/blog/time-series-database-an-explainer#what-is-a-time-series-database) at scale for real-time analytics and other use cases. Get faster time-based queries with hypertables, continuous aggregates, and columnar storage. Save money by compressing data into the columnstore, moving cold data to low-cost bottomless storage in Amazon S3 or Azure Blob storage, and deleting old data with automated policies. * *AI-focused*: build AI applications from start to scale. Get fast and accurate similarity search with the pgvector and pgvectorscale extensions. * *Hybrid applications*: get a full set of tools to develop applications that combine time-based data and AI. All standard Tiger Cloud services include the tooling you expect for production and developer environments: [live migration](/docs/migrate/live-migration), [automatic backups and PITR](/docs/deploy/tiger-cloud/tiger-cloud-aws/high-availability/backup-restore), [high availability](/docs/deploy/tiger-cloud/tiger-cloud-aws/high-availability/high-availability), [read replicas](/docs/deploy/tiger-cloud/tiger-cloud-aws/high-availability/read-scaling), [data forking](/docs/deploy/tiger-cloud/tiger-cloud-aws/service-management/fork-services), [connection pooling](/docs/deploy/tiger-cloud/tiger-cloud-aws/service-management/connection-pooling), [tiered storage](/docs/build/data-management/storage/manage-storage), [usage-based storage](/docs/deploy/tiger-cloud/tiger-cloud-aws/pricing-and-account-management#how-your-bill-is-calculated), secure in-Tiger Console [SQL editing](/docs/build/data-management/run-queries-from-tiger-console), service [metrics](/docs/deploy/tiger-cloud/tiger-cloud-aws/monitoring#metrics) and [insights](/docs/deploy/tiger-cloud/tiger-cloud-aws/monitoring#insights), streamlined maintenance, and much more. Tiger Cloud continuously monitors your services and prevents common PostgreSQL out-of-memory crashes. * **Free services**: *PostgreSQL with TimescaleDB and vector extensions* Free services come with pre-configured CPU and memory, are hosted in `us-east-1` region, and offer limited resources with a basic feature scope. This excludes connection pooling, replication, data tiering, exporters, metrics, VPC, and other features. Free services are perfect for schema explorations, syntax validations, and other small-scale testing. Once you reach the storage limit, a free service enters a read-only state. You can [convert your free service](/docs/deploy/tiger-cloud/tiger-cloud-aws/service-management/service-management#convert-a-free-service-to-a-standard-one) to a standard one at any time. Note Free services are currently in beta. You can [manage, pause, or delete](/docs/deploy/tiger-cloud/tiger-cloud-aws/service-management/service-management) your service at any time from Tiger Console. To start using Tiger Cloud for your data: 1. [Create a Tiger Cloud account](/docs/get-started/quickstart/create-service#create-a-tiger-cloud-account): register to get access to Tiger Console as a centralized point to administer and interact with your data. 2. [Create a Tiger Cloud service](/docs/get-started/quickstart/create-service#create-a-service): that is, a PostgreSQL database instance, powered by [TimescaleDB](https://www.tigerdata.com/docs/#TimescaleDB), built for production, and extended with cloud features like high-availability replicas. 3. [Connect to your Tiger Cloud service](/docs/get-started/quickstart/create-service#connect-to-your-service): to run queries, add and migrate your data from other sources. ## Create a Tiger Cloud account [Section titled “Create a Tiger Cloud account”](#create-a-tiger-cloud-account-1) You create a Tiger Cloud account to manage your services and data in a centralized and efficient manner in Tiger Console. From there, you can create and delete services, run queries, manage access and billing, integrate other services, contact support, and more. * Tiger Cloud You create a standalone account to manage Tiger Cloud as a separate unit in your infrastructure, which includes separate billing and invoicing. To set up Tiger Cloud: 1. **Sign up for a $1000-credit, 30-day free trial** Open [Sign up for Tiger Cloud](https://console.cloud.tigerdata.com/signup) and add your details, then click `Start your free trial`. You receive a confirmation email in your inbox. 2. **Confirm your email address** In the confirmation email, click the link supplied. 3. **Select the pricing plan** You are now logged into Tiger Console. You can change the pricing plan to better accommodate your growing needs on the [`Billing` page](https://console.cloud.tigerdata.com/dashboard/billing/plans). * Azure Marketplace To have Tiger Cloud as a part of your Azure infrastructure, you create or link your Tiger Cloud account through Azure Marketplace. In this case, Tiger Cloud is a line item in your Azure invoice. 1. **Open Azure Marketplace and search for `Tiger Data`** You see two pricing options, [pay-as-you-go](https://portal.azure.com/#view/Microsoft_Azure_Marketplace/GalleryItemDetailsBladeNopdl/id/timescale1759504210261.tigerdata-payg) and [annual commit](https://portal.azure.com/#view/Microsoft_Azure_Marketplace/GalleryItemDetailsBladeNopdl/id/timescale1759504210261.tigerdata-annualcommit). Select and click the right pricing option. 2. **Review product details** Review the details for the pricing option you have selected, then click `Subscribe`. 3. **Configure project and SaaS details** Select the Azure subscription and resource group, provide a name for your subscription to Tiger Cloud, configure auto-renew settings, then click `Review & subscribe`. 4. **Review the legal terms** Review the Terms of use, Privacy policy, and your subscription basics, then click `Subscribe`. Your SaaS resource is being created. 5. **Click `Configure account`** You are redirected to Tiger Console. 6. **Sign up for a $1000-credit, 30-day free trial** Add your details, then click `Start your free trial`. If you want to link an existing Tiger Cloud account to Azure, log in with your existing credentials. 7. **Select the pricing plan** You are now logged into Tiger Console. You can change the pricing plan later to better accommodate your growing needs on the [`Billing` page](https://console.cloud.tigerdata.com/dashboard/billing/plans). 8. **In `Confirm Azure Marketplace connection`, click `Connect`** Your Tiger Cloud and Azure accounts are now connected. ## Create a service [Section titled “Create a service”](#create-a-service-1) Now that you have an active Tiger Cloud account, you create and manage your services in Tiger Console. When you create a service, you effectively create a blank PostgreSQL database with additional Tiger Cloud features available under your pricing plan. You then add or migrate your data into this database. 1. **In the service creation page, click `+ New service`.** Follow the wizard to configure your service depending on its type. 2. **Click `Create service`.** Your service is constructed and ready to use in a few seconds. 3. **Click `Download the config` and store the configuration information you need to connect to this service in a secure location.** This file contains the passwords and configuration information you need to connect to your service using the Tiger Console Data view, from the command line, or using third-party database administration tools. You can [manage, pause, or delete](/docs/deploy/tiger-cloud/tiger-cloud-aws/service-management/service-management) your service at any time from Tiger Console. ## Connect to your service [Section titled “Connect to your service”](#connect-to-your-service-1) To run queries and perform other operations, connect to your service: 1. **Check your service is running correctly** In [Tiger Console](https://console.cloud.tigerdata.com/dashboard/services), check that your service is marked as `Running`. ![Tiger Console services list showing a running service](/docs/_astro/tiger-console-services-view.F-zo-hID_1rY9dD.webp) 2. **Connect to your service** Connect using Data view or SQL editor in Tiger Console, or psql in the command line: * Data view in Console 1. In Tiger Console, toggle `Data view`. 2. In the connection drop-down in the top right, select your service. ![Select a connection in Tiger Cloud data mode](/docs/_astro/tiger-console-data-mode-connection-dropdown.eS_sjlnj_1J2XiX.webp) 3. Run a test query: ```sql SELECT CURRENT_DATE; ``` This query gives you the current date, you have successfully connected to your service. And that is it, you are up and running. Enjoy developing with Tiger Data. * SQL editor in Console 1. In Tiger Console, select your service. 2. Click `SQL Editor` at the bottom. ![SQL editor in Tiger Console Ops mode](/docs/_astro/tiger-console-ops-mode-sql-editor-empty.D6fPmf48_LfDRl.webp) 3. Run a test query: ```sql SELECT CURRENT_DATE; ``` This query gives you the current date, you have successfully connected to your service. And that is it, you are up and running. Enjoy developing with Tiger Data. * psql on the command line 1. Install psql. 2. Run the following command in the terminal using the service URL from the config file you have saved during service creation: ```plaintext psql "" ``` 3. Run a test query: ```sql SELECT CURRENT_DATE; ``` This query returns the current date. You have successfully connected to your service. And that is it, you are up and running. Enjoy developing with Tiger Data. Quick recap. You: * Manage your services in the Ops view in Tiger Console: add read replicas and enable high availability, compress data into the columnstore, change parameters, and so on. * Analyze your data in the [Data view](https://console.cloud.tigerdata.com/dashboard/data-view?popsql) in Tiger Console: write queries with autocomplete, save them in folders, share them, create charts/dashboards, and much more. * Store configuration and security information in your config file. * Tiger Cloud You create a standalone account to manage Tiger Cloud as a separate unit in your infrastructure, which includes separate billing and invoicing. To set up Tiger Cloud: 1. **Sign up for a $1000-credit, 30-day free trial** Open [Sign up for Tiger Cloud](https://console.cloud.tigerdata.com/signup) and add your details, then click `Start your free trial`. You receive a confirmation email in your inbox. 2. **Confirm your email address** In the confirmation email, click the link supplied. 3. **Select the pricing plan** You are now logged into Tiger Console. You can change the pricing plan to better accommodate your growing needs on the [`Billing` page](https://console.cloud.tigerdata.com/dashboard/billing/plans). * Azure Marketplace To have Tiger Cloud as a part of your Azure infrastructure, you create or link your Tiger Cloud account through Azure Marketplace. In this case, Tiger Cloud is a line item in your Azure invoice. 1. **Open Azure Marketplace and search for `Tiger Data`** You see two pricing options, [pay-as-you-go](https://portal.azure.com/#view/Microsoft_Azure_Marketplace/GalleryItemDetailsBladeNopdl/id/timescale1759504210261.tigerdata-payg) and [annual commit](https://portal.azure.com/#view/Microsoft_Azure_Marketplace/GalleryItemDetailsBladeNopdl/id/timescale1759504210261.tigerdata-annualcommit). Select and click the right pricing option. 2. **Review product details** Review the details for the pricing option you have selected, then click `Subscribe`. 3. **Configure project and SaaS details** Select the Azure subscription and resource group, provide a name for your subscription to Tiger Cloud, configure auto-renew settings, then click `Review & subscribe`. 4. **Review the legal terms** Review the Terms of use, Privacy policy, and your subscription basics, then click `Subscribe`. Your SaaS resource is being created. 5. **Click `Configure account`** You are redirected to Tiger Console. 6. **Sign up for a $1000-credit, 30-day free trial** Add your details, then click `Start your free trial`. If you want to link an existing Tiger Cloud account to Azure, log in with your existing credentials. 7. **Select the pricing plan** You are now logged into Tiger Console. You can change the pricing plan later to better accommodate your growing needs on the [`Billing` page](https://console.cloud.tigerdata.com/dashboard/billing/plans). 8. **In `Confirm Azure Marketplace connection`, click `Connect`** Your Tiger Cloud and Azure accounts are now connected. * Data view in Console 1. In Tiger Console, toggle `Data view`. 2. In the connection drop-down in the top right, select your service. ![Select a connection in Tiger Cloud data mode](/docs/_astro/tiger-console-data-mode-connection-dropdown.eS_sjlnj_1J2XiX.webp) 3. Run a test query: ```sql SELECT CURRENT_DATE; ``` This query gives you the current date, you have successfully connected to your service. And that is it, you are up and running. Enjoy developing with Tiger Data. * SQL editor in Console 1. In Tiger Console, select your service. 2. Click `SQL Editor` at the bottom. ![SQL editor in Tiger Console Ops mode](/docs/_astro/tiger-console-ops-mode-sql-editor-empty.D6fPmf48_LfDRl.webp) 3. Run a test query: ```sql SELECT CURRENT_DATE; ``` This query gives you the current date, you have successfully connected to your service. And that is it, you are up and running. Enjoy developing with Tiger Data. * psql on the command line 1. Install psql. 2. Run the following command in the terminal using the service URL from the config file you have saved during service creation: ```plaintext psql "" ``` 3. Run a test query: ```sql SELECT CURRENT_DATE; ``` This query returns the current date. You have successfully connected to your service. And that is it, you are up and running. Enjoy developing with Tiger Data. # Integrate Tiger Cloud with your AI Agent (MCP and CLI) > Install Tiger CLI, set up Tiger MCP, and manage Tiger Cloud resources from Claude, Cursor, and other AI agents Tiger MCP gives your AI agent access to Tiger Cloud so you can manage services and query your data using natural language. Tiger MCP is built into the Tiger CLI binary. Alongside tools to manage services and run SQL, it includes built-in skills (for example, schema design, hypertable setup, and migration planning) and is wired to Tiger Data documentation, so your AI agent can design, analyze, and recommend improvements with up-to-date guidance. This page walks you through installing Tiger CLI, configuring authentication for Tiger MCP, and managing Tiger Cloud resources from your AI agent. ## Prerequisites for this tutorial To follow these steps, you'll need: * A [Tiger Cloud account](/docs/get-started/quickstart/create-service#create-a-tiger-cloud-account). - An AI agent installed on your machine with an active API key. The following AI agents are **automatically** configured by Tiger MCP: * [`claude-code`](https://docs.anthropic.com/en/docs/claude-code) * [`cursor`](https://cursor.com) * [`windsurf`](https://windsurf.com) * [`codex`](https://openai.com/codex) * [`gemini`](https://github.com/google-gemini/gemini-cli) * [`vscode`](https://code.visualstudio.com) * [`antigravity`](https://antigravity.google) * [`kiro-cli`](https://kiro.dev/cli) Support for additional clients is added over time; if yours isn't listed, use [manual configuration](#manually-configure-tiger-mcp) below. ## Install and configure Tiger MCP [Section titled “Install and configure Tiger MCP”](#install-and-configure-tiger-mcp) Tiger MCP is bundled with Tiger CLI. 1. **Install the Tiger CLI** In a terminal, install the CLI using the method for your platform (see [Get started with Tiger CLI](/docs/get-started/quickstart/tiger-cli) for the latest install instructions). 2. **Authenticate Tiger CLI** 1. **Log in to Tiger Cloud** In an interactive terminal, run: ```bash tiger auth login ``` Tiger CLI opens Console in your browser to sign in with OAuth. Log in, or sign up if you don't have a Tiger Cloud account yet, then authorize Tiger CLI. 2. **Select a Tiger Cloud project** If you have multiple projects, the CLI will prompt you to choose one (for example, Tiger Project, company-wide project, or department project). If you have only one project, this step is skipped. Credentials are stored in the system keychain (or credential manager) when possible. If that fails, they are stored in `~/.config/tiger/credentials` with restricted permissions (600). Configuration is stored by default in `~/.config/tiger/config.yaml`. 3. **Confirm the connection** List services to verify authentication: ```bash tiger service list ``` You'll see either an empty list with a prompt to create a service, or a table of your services (service ID, name, status, type, region, created). 3. **Install Tiger MCP** Run: ```bash tiger mcp install ``` Choose the MCP client to integrate with (for example, `claude-code`, `cursor`, `windsurf`, `codex`, `gemini-cli`, `vscode`) and press `Enter`. CLI and Tiger MCP commands The exact list of clients and subcommands (for example, `tiger mcp install`, `tiger mcp list`) can change with new CLI releases. If a command fails, run `tiger mcp --help`. After this, you can use Tiger MCP from your AI agent to manage Tiger Cloud. ## Manage Tiger Cloud from your AI agent [Section titled “Manage Tiger Cloud from your AI agent”](#manage-tiger-cloud-from-your-ai-agent) Once connected, you can manage services and learn best practices through your agent. 1. **Start your AI agent** (For example, `claude`). It will start the Tiger MCP server and connect to Tiger Cloud. 2. **Confirm Tiger MCP is active** Ask: *"Is the Tiger MCP server active?"* You should see a summary of available tools (service management, database operations, documentation search, skills for hypertables, and others). 3. **List services** Ask: *"Can you list my active services?"* to see your Tiger Cloud services. 4. **Manage services in plain language** For example: *"Create a new AI service called bob with an HA replica."* The agent will use Tiger MCP tools to create the service. 5. **Get best practices** For example: *"I need to migrate a database with 25 million rows to Tiger Cloud; what service optimizations should I do before migration?"* The agent can use docs and tools to give tailored advice. ## Restrict Tiger MCP to read-only [Section titled “Restrict Tiger MCP to read-only”](#restrict-tiger-mcp-to-read-only) If you want your AI agent to explore and query services without changing anything, enable read-only mode: ```bash tiger config set read_only true ``` When read-only mode is on: * Mutating CLI commands (`service create`, `fork`, `start`, `stop`, `resize`, `update-password`, and `delete`) are refused with an error, and the matching mutating Tiger MCP tools are not registered, so an agent cannot call them. * The database sessions opened by the `db_execute_query` tool, `tiger db connect`, and `tiger db connection-string` run in Tiger Cloud's immutable read-only mode, so writes and schema changes (DDL) are rejected by the server. * Read tools and commands, such as listing services or running `SELECT` queries, are unaffected. To set read-only mode for a single session without changing your saved configuration, set the `TIGER_READ_ONLY` environment variable instead: ```bash TIGER_READ_ONLY=true tiger mcp start ``` To turn read-only mode back off, run `tiger config unset read_only`. ## Manually configure Tiger MCP [Section titled “Manually configure Tiger MCP”](#manually-configure-tiger-mcp) If your MCP client isn't supported by `tiger mcp install`, add Tiger MCP manually. Many clients use a JSON config that starts the server with `tiger mcp start`: ```json { "mcpServers": { "tiger": { "command": "tiger", "args": ["mcp", "start"] } } } ``` Config format may vary Config key names (for example, `mcpServers`) and structure depend on your client. See your AI agent's MCP documentation for the exact format and where to place the file. ## Reference [Section titled “Reference”](#reference) For the full list of Tiger MCP tools and `tiger mcp` commands, see the [Tiger MCP reference](/docs/reference/tiger-cloud/tiger-mcp). For every CLI command, configuration parameter, and global flag, see the [Tiger CLI reference](/docs/reference/tiger-cloud/tiger-cli). ## Next steps [Section titled “Next steps”](#next-steps) * [File an issue in the Tiger CLI repo](https://github.com/timescale/tiger-cli/issues/new): Report bugs or request features to help shape the open-source Tiger CLI and Tiger MCP. # 5-minute quickstart > Get up and running with Tiger Cloud in 5 minutes On this page, you will create a Tiger Cloud service, connect to it, create your first hypertable, and run a query. By the end you'll have a working time-series table and see query results, all in under 5 minutes. For a more extensive lesson on what hypertables are and why they matter, see [Your first hypertable](/docs/build/how-to/your-first-hypertable). ## Prerequisites for this tutorial To follow these steps, you'll need: * A [Tiger Cloud account](https://console.cloud.tigerdata.com/) Tips Prefer the terminal? You can also create and manage services with [Tiger CLI](/docs/get-started/quickstart/tiger-cli). ## Step 1: Create a service [Section titled “Step 1: Create a service”](#step-1-create-a-service) 1. **Create a service** In [Tiger Console](https://console.cloud.tigerdata.com/), click `New service` and follow the wizard to configure your service. 2. **Wait for the service to be ready** Click `Create service`. The service is ready in a few seconds. 3. **Download the database config** Download the database config and store it somewhere safe, you'll need the connection string to connect to your service. For more detail, see [Create a Tiger Cloud service](/docs/get-started/quickstart/create-service). ## Step 2: Connect to your service [Section titled “Step 2: Connect to your service”](#step-2-connect-to-your-service) Pick one way to run SQL: * **Tiger Console (easiest):** In Tiger Console, open your service, then use `Data view` at the top or `SQL Editor` at the bottom. You're connected as soon as you see the query panel. * **psql:** From your terminal, run `psql ""` using the connection string from the config file you downloaded. Run this to confirm you're connected: ```sql SELECT CURRENT_DATE; ``` You should see today's date. If you do, you're ready for the next step. For connection options, including other clients, see [Connect your app](/docs/get-started/quickstart/connect-your-app). ## Step 3: Create your first hypertable and run a query [Section titled “Step 3: Create your first hypertable and run a query”](#step-3-create-your-first-hypertable-and-run-a-query) We'll create a small **hypertable** (a table optimized for time-series data), insert a few rows, and query them. You'll use standard SQL; the database handles partitioning automatically. 1. **Create the hypertable** Run this in Tiger Console or your SQL client: ```sql CREATE TABLE conditions ( time TIMESTAMPTZ NOT NULL, location TEXT NOT NULL, device TEXT NOT NULL, temperature DOUBLE PRECISION NULL, humidity DOUBLE PRECISION NULL ) WITH ( tsdb.hypertable, tsdb.segmentby = 'device', tsdb.orderby = 'time DESC' ); ``` You should see a confirmation that the table was created. The `tsdb.hypertable` option makes this a hypertable partitioned by time. 2. **Insert sample data** ```sql INSERT INTO conditions (time, location, device, temperature, humidity) VALUES (NOW() - INTERVAL '2 hours', 'office', 'sensor-1', 22.1, 45.0), (NOW() - INTERVAL '1 hour', 'office', 'sensor-1', 22.3, 44.8), (NOW(), 'office', 'sensor-1', 22.2, 45.1); ``` 3. **Query the data** ```sql SELECT * FROM conditions ORDER BY time DESC; ``` You should see your three rows with the most recent first. You've just created a hypertable, inserted time-series data, and queried it, all with standard SQL. Tips **What just happened?** The table is partitioned by time into **chunks** behind the scenes. You didn't have to manage chunks; you just used the table. For the full picture, see [Your first hypertable](/docs/build/how-to/your-first-hypertable). ## Next steps [Section titled “Next steps”](#next-steps) * **[Integrate Tiger Cloud with your AI agent](/docs/get-started/quickstart/mcp-cli)**: Set up Tiger MCP and manage your service from Claude, Cursor, and other AI agents. * **[Your first hypertable](/docs/build/how-to/your-first-hypertable)**: Go deeper with hypertables, chunks, and time-series queries. * **[Write and query data](/docs/build/data-management)**: Learn how to write, query, and manage your time-series data. * **[Deploy](/docs/deploy)**: Configure, secure, and manage your Tiger Cloud deployment. # Get started with the REST API > Manage the resources in your Tiger Cloud project programmatically with the Tiger Cloud REST API Use Tiger REST API to build your own integrations or when you need language-native access from an application. The [Tiger CLI](/docs/get-started/quickstart/tiger-cli) is a convenience wrapper over a subset of this API; the REST API is the complete surface. [Tiger REST API](/docs/reference/tiger-cloud-rest) is a comprehensive RESTful API you use to manage Tiger Cloud resources including VPCs, services, and read replicas. This page shows you how to set up secure authentication for the Tiger REST API and create your first service. ## Prerequisites for this tutorial To follow these steps, you'll need: * A [Tiger Cloud account](/docs/get-started/quickstart/create-service#create-a-tiger-cloud-account). - [curl](https://curl.se/) installed. ## Configure secure authentication Tiger REST API uses HTTP Basic Authentication with access keys and secret keys. All API requests must include proper authentication headers. 1. **Set up API credentials** 1. In Tiger Console [copy your project ID](/docs/integrate/find-connection-details#find-your-project-and-service-id) and store it securely using an environment variable: ```bash export TIGERDATA_PROJECT_ID="your-project-id" ``` 2. In Tiger Console [create your client credentials](/docs/integrate/find-connection-details#create-client-credentials) and store them securely using environment variables: ```bash export TIGERDATA_ACCESS_KEY="Public key" export TIGERDATA_SECRET_KEY="Secret key" ``` 2. **Configure the API endpoint** Set the base URL in your environment: ```bash export API_BASE_URL="https://console.cloud.tigerdata.com/public/api/v1" ``` 3. **Test your authenticated connection to Tiger REST API by listing the services in the current Tiger Cloud project** ```bash curl -X GET "${API_BASE_URL}/projects/${TIGERDATA_PROJECT_ID}/services" \ -u "${TIGERDATA_ACCESS_KEY}:${TIGERDATA_SECRET_KEY}" \ -H "Content-Type: application/json" ``` This call returns something like: * No services: ```txt [] ``` * One or more services: ```txt [{"service_id":"tgrservice","project_id":"tgrproject","name":"tiger-eon", "region_code":"us-east-1","service_type":"TIMESCALEDB", "created":"2025-10-20T12:21:28.216172Z","status":"READY", "resources":[{"id":"104977","spec":{"cpu_millis":500,"memory_gbs":2}}], "metrics":{"memory_mb":166,"storage_mb":892,"milli_cpu":6}, "metadata":{"environment":"DEV"}, "endpoint":{"host":"tgrservice.tgrproject.tsdb.cloud.timescale.com","port":11111}}] ``` ## Create your first Tiger Cloud service Create a new service using the Tiger REST API: 1. **Create a service using the POST endpoint** ```bash curl -X POST "${API_BASE_URL}/projects/${TIGERDATA_PROJECT_ID}/services" \ -u "${TIGERDATA_ACCESS_KEY}:${TIGERDATA_SECRET_KEY}" \ -H "Content-Type: application/json" \ -d '{ "name": "my-first-service", "addons": ["time-series"], "region_code": "us-east-1", "replica_count": 1, "cpu_millis": "1000", "memory_gbs": "4" }' ``` Tiger Cloud creates a Development environment for you. That is, no delete protection, high-availability, spooling or read replication. You see something like: ```txt {"service_id":"tgrservice","project_id":"tgrproject","name":"my-first-service", "region_code":"us-east-1","service_type":"TIMESCALEDB", "created":"2025-10-20T22:29:33.052075713Z","status":"QUEUED", "resources":[{"id":"105120","spec":{"cpu_millis":1000,"memory_gbs":4}}], "metadata":{"environment":"DEV"}, "endpoint":{"host":"tgrservice.tgrproject.tsdb.cloud.timescale.com","port":11111}, "initial_password":"notTellingYou", "ha_replicas":{"sync_replica_count":0,"replica_count":1}} ``` 2. **Save `service_id` from the response to a variable:** ```bash # Extract service_id from the JSON response export SERVICE_ID="service_id-from-response" ``` 3. **Check the configuration for the service** ```bash curl -X GET "${API_BASE_URL}/projects/${TIGERDATA_PROJECT_ID}/services/${SERVICE_ID}" \ -u "${TIGERDATA_ACCESS_KEY}:${TIGERDATA_SECRET_KEY}" \ -H "Content-Type: application/json" ``` You see something like: ```txt {"service_id":"tgrservice","project_id":"tgrproject","name":"my-first-service", "region_code":"us-east-1","service_type":"TIMESCALEDB", "created":"2025-10-20T22:29:33.052075Z","status":"READY", "resources":[{"id":"105120","spec":{"cpu_millis":1000,"memory_gbs":4}}], "metrics":{"memory_mb":166,"storage_mb":892,"milli_cpu":6}, "metadata":{"environment":"DEV"}, "endpoint":{"host":"tgrservice.tgrproject.tsdb.cloud.timescale.com","port":11111}, "ha_replicas":{"sync_replica_count":0,"replica_count":1}} ``` And that is it, you are ready to use the [Tiger REST API](/docs/reference/tiger-cloud-rest) to manage your services in Tiger Cloud. ## Security best practices Follow these security guidelines when working with the Tiger REST API: * **Credential management** * Store API credentials as environment variables, not in code * Use credential rotation policies for production environments * Never commit credentials to version control systems * **Network security** * Use HTTPS endpoints exclusively for API communication * Implement proper certificate validation in your HTTP clients * **Data protection** * Use secure storage for service connection strings and passwords * Implement proper backup and recovery procedures for created services * Follow data residency requirements for your region For every endpoint, parameter, and response, see the [Tiger Cloud REST API reference](/docs/reference/tiger-cloud-rest). # Get started with Tiger CLI > Install Tiger CLI and manage the resources in your Tiger Cloud project from the command line Tiger CLI is a command-line interface for managing Tiger Cloud programmatically. It lets you, your scripts, and AI agents provision, configure, and manage Tiger Cloud services. Tiger CLI calls Tiger REST API under the hood and bundles Tiger MCP for AI agents. This page shows you how to install and set up secure authentication for Tiger CLI, then create your first service. Along the way you manage Tiger Cloud resources such as VPCs, services, and related infrastructure. ## Install and configure Tiger CLI 1. **Install Tiger CLI** Use the terminal to install the CLI: * macOS ```shell brew install --cask timescale/tap/tiger-cli ``` * Debian ```shell curl -s https://packagecloud.io/install/repositories/timescale/tiger-cli/script.deb.sh | sudo os=any dist=any bash sudo apt-get install tiger-cli ``` * Ubuntu ```shell curl -s https://packagecloud.io/install/repositories/timescale/tiger-cli/script.deb.sh | sudo os=any dist=any bash sudo apt-get install tiger-cli ``` * Red Hat ```shell curl -s https://packagecloud.io/install/repositories/timescale/tiger-cli/script.rpm.sh | sudo os=rpm_any dist=rpm_any bash sudo yum install tiger-cli ``` * Fedora ```shell curl -s https://packagecloud.io/install/repositories/timescale/tiger-cli/script.rpm.sh | sudo os=rpm_any dist=rpm_any bash sudo yum install tiger-cli ``` * Windows PowerShell ```shell irm https://cli.tigerdata.com/install.ps1 | iex ``` * x-platform ```shell curl -fsSL https://cli.tigerdata.com | sh ``` 2. **Authenticate Tiger CLI** 1. In an interactive terminal, start the login flow: ```shell tiger auth login ``` Tiger CLI opens Console in your browser to sign in with OAuth. Log in, or sign up if you don't have a Tiger Cloud account yet, then authorize Tiger CLI. If the browser does not open automatically, copy the `Auth URL` printed in the terminal. For non-interactive sessions such as scripts and CI, you can authenticate with [client credentials](/docs/integrate/find-connection-details#create-client-credentials) instead of the browser flow. 2. Select a Tiger Cloud project: ```text Auth URL is: https://console.cloud.tigerdata.com/oauth/authorize?client_id=...&code_challenge_method=S256&redirect_uri=http://localhost:PORT/callback&response_type=code&state=... Opening browser for authentication... Select a project: > 1. Tiger Project (tgrproject) 2. YourCompany (Company wide project) (cpnproject) 3. YourCompany Department (dptproject) Successfully logged in (project: tgrproject) 🎉 Next steps: • Install MCP server for your favorite AI coding tool: tiger mcp install • List existing services: tiger service list • Create a new service: tiger service create • Enable read-only mode: tiger config set read_only true ``` If only one project is associated with your account, this step is skipped. Tiger CLI stores the OAuth session in your system keychain, or in `~/.config/tiger/credentials` with restricted file permissions (600) if the keychain is unavailable, and refreshes it automatically. Tiger CLI stores your configuration in `~/.config/tiger/config.yaml`. To sign out and revoke the session, run `tiger auth logout`. 3. **Test your authenticated connection to Tiger Cloud by listing services** ```bash tiger service list ``` This call returns something like: * No services: ```txt 🏜️ No services found! Your project is looking a bit empty. 🚀 Ready to get started? Create your first service with: tiger service create ``` * One or more services: ```txt ┌────────────┬─────────────────────┬────────┬─────────────┬──────────────┬──────────────────┐ │ SERVICE ID │ NAME │ STATUS │ TYPE │ REGION │ CREATED │ ├────────────┼─────────────────────┼────────┼─────────────┼──────────────┼──────────────────┤ │ tgrservice │ tiger-agent-service │ READY │ TIMESCALEDB │ eu-central-1 │ 2025-09-25 16:09 │ └────────────┴─────────────────────┴────────┴─────────────┴──────────────┴──────────────────┘ ``` ## Create your first Tiger Cloud service Create a new Tiger Cloud service using Tiger CLI: 1. **Submit a service creation request** Run the service creation command: ```shell tiger service create ``` By default, this creates a service with 0.5 CPU and 2 GB memory. To customize the configuration, use the [`service create` command flags](/docs/reference/tiger-cloud/tiger-cli)—for example, `tiger service create --cpu shared --memory shared` for a free service. Tiger Cloud services come with several extensions pre-installed, including `pgvector`, `pgvectorscale`, and `pg_textsearch`. To use any of these extensions in your database, you must first create them with SQL: ```sql CREATE EXTENSION pgvector; ``` Note Free services are currently in beta. Free services with shared CPU/memory are only available in the `us-east-1` region. Standard services can be created in any available AWS or Azure region using the `--region` flag. Tiger Cloud creates a Development environment for you. That is, no delete protection, high-availability, spooling or read replication. You see something like: ```txt 🚀 Creating service 'db-11111' (auto-generated name)... ✅ Service creation request accepted! 📋 Service ID: 🔐 Password saved to system keyring for automatic authentication 🎯 Set service '' as default service. ⏳ Waiting for service to be ready (wait timeout: 30m0s)... 🎉 Service is ready and running! 🔌 Run 'tiger db connect' to connect to your new service ┌───────────────────┬───────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ PROPERTY │ VALUE │ ├───────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ Service ID │ │ │ Name │ db-11111 │ │ Status │ READY │ │ Type │ TIMESCALEDB │ │ Region │ us-east-1 │ │ Environment │ DEV │ │ CPU │ 0.5 cores (500m) │ │ Memory │ 2 GB │ │ Direct Endpoint │ ..tsdb.cloud.timescale.com: │ │ Created │ 2026-07-07 13:36:16 UTC │ │ Connection String │ postgresql://tsdbadmin@..tsdb.cloud.timescale.com:/tsdb?sslmode=require │ │ Console URL │ https://console.cloud.tigerdata.com/dashboard/services/ │ └───────────────────┴───────────────────────────────────────────────────────────────────────────────────────────────────────┘ ``` This service is set as default by the CLI. 2. **Check the CLI configuration** ```shell tiger config show ``` You see something like: ```txt ┌──────────────────┬───────────────────────────────────────────────────────────────┐ │ PROPERTY │ VALUE │ ├──────────────────┼───────────────────────────────────────────────────────────────┤ │ api_url │ https://console.cloud.tigerdata.com/public/api/v1 │ │ analytics │ true │ │ console_url │ https://console.cloud.tigerdata.com │ │ debug │ false │ │ docs_mcp │ true │ │ docs_mcp_url │ https://mcp.tigerdata.com/docs?disabled_skills=ghost-database │ │ gateway_url │ https://console.cloud.tigerdata.com/api │ │ mcp_max_rows │ 100 │ │ color │ true │ │ output │ table │ │ password_storage │ keyring │ │ read_only │ false │ │ releases_url │ https://cli.tigerdata.com │ │ service_id │ │ │ version_check │ true │ └──────────────────┴───────────────────────────────────────────────────────────────┘ ``` And that is it, you are ready to use Tiger CLI to manage your services in Tiger Cloud. For the full list of commands, configuration parameters, and global flags, see the [Tiger CLI reference](/docs/reference/tiger-cloud/tiger-cli). ## Next steps [Section titled “Next steps”](#next-steps) * [Integrate Tiger Cloud with your AI agent](/docs/get-started/quickstart/mcp-cli): Set up Tiger MCP so your AI agent can manage services and run queries. * [Get started with the REST API](/docs/get-started/quickstart/rest-api): Manage the same resources over raw HTTP for custom integrations. * [Tiger CLI reference](/docs/reference/tiger-cloud/tiger-cli): Every command, flag, and configuration parameter. * [File an issue in the Tiger CLI repo](https://github.com/timescale/tiger-cli/issues/new): Report bugs or request features to help shape the open-source Tiger CLI and Tiger MCP. # 10-minute quickstart > Get started with self-hosted TimescaleDB in about 10 minutes This guide helps you run TimescaleDB locally, create your first hypertable with the columnstore enabled, write data to the columnstore, and see instant analytical query performance. ## Prerequisites for this procedure To follow these steps, you'll need: * Docker installed on your machine * 8GB RAM recommended * The `psql` client (included with PostgreSQL) or any PostgreSQL client like [pgAdmin](https://www.pgadmin.org/download/) ## Step 1: Start TimescaleDB [Section titled “Step 1: Start TimescaleDB”](#step-1-start-timescaledb) Start TimescaleDB using one of these options. * One-line install (recommended) The easiest way to get started: Important This script is intended for local development and testing only. Do not use it for production deployments. For production-ready installation options, see the [TimescaleDB installation guide](/docs/get-started/choose-your-path/install-timescaledb). On Linux or Mac: ```sh curl -sL https://tsdb.co/start-local | sh ``` This command: * Downloads and starts TimescaleDB, if not already downloaded. * Exposes PostgreSQL on port `6543`, a non-standard port that avoids conflicts with other PostgreSQL instances on port 5432. * Automatically tunes settings for your environment using timescaledb-tune. * Sets up a persistent data volume. * Manual Docker command Run TimescaleDB directly with Docker. This option also works on Windows: ```bash docker run -d --name timescaledb \ -p 6543:5432 \ -e POSTGRES_PASSWORD=password \ timescale/timescaledb-ha:pg18 ``` This example uses port `6543`, mapped to container port 5432, to avoid conflicts if you have other PostgreSQL instances running on the standard port 5432. Wait about 1 to 2 minutes for TimescaleDB to download and initialize. ## Step 2: Connect to TimescaleDB [Section titled “Step 2: Connect to TimescaleDB”](#step-2-connect-to-timescaledb) Connect using `psql`: ```bash psql -h localhost -p 6543 -U postgres # When prompted, enter password: password ``` You should see the PostgreSQL prompt. Verify TimescaleDB is installed: ```sql SELECT extname, extversion FROM pg_extension WHERE extname = 'timescaledb'; ``` Expected output: ```plaintext extname | extversion -------------+------------ timescaledb | 2.x.x ``` Tips Prefer a GUI? If you would rather use a graphical tool instead of the command line, you can download [pgAdmin](https://www.pgadmin.org/download/) and connect to TimescaleDB using the same connection details (host: `localhost`, port: `6543`, user: `postgres`, password: `password`). ## Step 3: Create your first hypertable [Section titled “Step 3: Create your first hypertable”](#step-3-create-your-first-hypertable) Create a hypertable for IoT sensor data with the columnstore enabled: ```sql -- Create a hypertable with automatic columnstore CREATE TABLE sensor_data ( time TIMESTAMPTZ NOT NULL, sensor_id TEXT NOT NULL, temperature DOUBLE PRECISION, humidity DOUBLE PRECISION, pressure DOUBLE PRECISION ) WITH ( tsdb.hypertable ); ``` `tsdb.hypertable` converts this into a TimescaleDB hypertable. See more: * [Create a hypertable](/docs/learn/hypertables/creating-and-configuring-hypertables) * [Set up hypercore](/docs/build/columnar-storage/setup-hypercore) ## Step 4: Insert sample data [Section titled “Step 4: Insert sample data”](#step-4-insert-sample-data) Add some sample sensor readings: ```sql -- Enable timing to see time to execute queries \timing on -- Insert sample data for multiple sensors -- SET timescaledb.enable_direct_compress_insert = on to insert data directly to the columnstore (columnnar format for performance) SET timescaledb.enable_direct_compress_insert = on; INSERT INTO sensor_data (time, sensor_id, temperature, humidity, pressure) SELECT time, 'sensor_' || ((random() * 9)::int + 1), 20 + (random() * 15), 40 + (random() * 30), 1000 + (random() * 50) FROM generate_series( NOW() - INTERVAL '90 days', NOW(), INTERVAL '1 seconds' ) AS time; -- Once data is inserted into the columnstore we optimize the order and structure -- this compacts and orders the data in the chunks for optimal query performance and compression DO $$ DECLARE ch TEXT; BEGIN FOR ch IN SELECT show_chunks('sensor_data') LOOP CALL convert_to_columnstore(ch, recompress := true); END LOOP; END $$; ``` This generates \~7,776,001 readings across 10 sensors over the past 90 days. Verify the data was inserted: ```sql SELECT COUNT(*) FROM sensor_data; ``` ## Step 5: Run your first analytical queries [Section titled “Step 5: Run your first analytical queries”](#step-5-run-your-first-analytical-queries) Now run some analytical queries that showcase TimescaleDB's performance: ```sql -- Enable query timing to see performance \timing on -- Query 1: Average readings per sensor over the last 7 days SELECT sensor_id, COUNT(*) as readings, ROUND(AVG(temperature)::numeric, 2) as avg_temp, ROUND(AVG(humidity)::numeric, 2) as avg_humidity, ROUND(AVG(pressure)::numeric, 2) as avg_pressure FROM sensor_data WHERE time > NOW() - INTERVAL '7 days' GROUP BY sensor_id ORDER BY sensor_id; -- Query 2: Hourly averages using time_bucket -- Time buckets enable you to aggregate data in hypertables by time interval and calculate summary values. SELECT time_bucket('1 hour', time) AS hour, sensor_id, ROUND(AVG(temperature)::numeric, 2) as avg_temp, ROUND(AVG(humidity)::numeric, 2) as avg_humidity FROM sensor_data WHERE time > NOW() - INTERVAL '24 hours' GROUP BY hour, sensor_id ORDER BY hour DESC, sensor_id LIMIT 20; -- Query 3: Daily statistics across all sensors SELECT time_bucket('1 day', time) AS day, COUNT(*) as total_readings, ROUND(AVG(temperature)::numeric, 2) as avg_temp, ROUND(MIN(temperature)::numeric, 2) as min_temp, ROUND(MAX(temperature)::numeric, 2) as max_temp FROM sensor_data GROUP BY day ORDER BY day DESC LIMIT 10; -- Query 4: Latest reading for each sensor -- Highlights the value of Skipscan executing in under 100ms without skipscan it takes over 5sec SELECT DISTINCT ON (sensor_id) sensor_id, time, ROUND(temperature::numeric, 2) as temperature, ROUND(humidity::numeric, 2) as humidity, ROUND(pressure::numeric, 2) as pressure FROM sensor_data ORDER BY sensor_id, time DESC; ``` Notice how fast these analytical queries run, even with aggregations across millions of rows. This is the power of TimescaleDB's columnstore. ## Next steps [Section titled “Next steps”](#next-steps) Now that you have TimescaleDB running locally, explore more: * [Create continuous aggregates](/docs/build/continuous-aggregates/create-a-continuous-aggregate) for faster real-time analytics. * [Connect your app](/docs/get-started/quickstart/connect-your-app) to TimescaleDB. * [Try a guided project](/docs/build/examples) with a real-world dataset. # Integrate with Tiger Data > Connect Tiger Data to your stack; ETL, BI, streaming, connectors, and observability # Integrate with Tiger Data Connect Tiger Data to your stack; ETL, BI, streaming, connectors, and observability Search by tool or keyword... Type of tool \[ ] Data engineering & ETL \[ ] BI & visualization \[ ] Data ingestion & streaming \[ ] Connectors \[ ] Code & libraries \[ ] Query & administration \[ ] Secure connectivity \[ ] Observability & alerting \[ ] Configuration & deployment Industry \[ ] Oil and gas \[ ] IoT \[ ] Energy \[ ] Crypto \[ ] Healthcare \[ ] Manufacturing Platform \[ ] Tiger Cloud on AWS \[ ] Tiger Cloud on Azure \[ ] Self-hosted First-party/third-party \[ ] First-party \[ ] Third-party Technology \[ ] PostgreSQL \[ ] Python \[ ] Ruby \[ ] Node.js \[ ] Go \[ ] Java \[ ] SQL \[ ] Kafka \[ ] MQTT \[ ] OPC UA \[ ] Apache Iceberg \[ ] AWS \[ ] Azure \[ ] GCP \[ ] Terraform \[ ] Kubernetes \[ ] Grafana \[ ] Prometheus Clear filters No integrations match your current filters. Try broadening your search. [![](/docs/_astro/aws.DojyZ-Di.png) ![](/docs/_astro/cloudwatch-dark.DNiASpcm.png)](/integrate/observability-alerting/cloudwatch) [Observability & alerting](/integrate/observability-alerting/cloudwatch) ### [Amazon CloudWatch](/integrate/observability-alerting/cloudwatch) [Export telemetry data to monitor and analyze your service performance and health](/integrate/observability-alerting/cloudwatch) [![](/docs/_astro/aws-lambda.DG6s7052.png)](/integrate/connectors/source/sync-from-s3) [Connectors](/integrate/connectors/source/sync-from-s3) ### [Amazon S3 source connector](/integrate/connectors/source/sync-from-s3) [Sync data from S3 to your Tiger Cloud service in real time](/integrate/connectors/source/sync-from-s3) [![](/docs/_astro/aws.DojyZ-Di.png) ![](/docs/_astro/cloudwatch-dark.DNiASpcm.png)](/integrate/data-engineering-etl/amazon-sagemaker) [Data engineering & ETL](/integrate/data-engineering-etl/amazon-sagemaker) ### [Amazon SageMaker](/integrate/data-engineering-etl/amazon-sagemaker) [Build, train, and deploy ML models with time-series data storage and analysis](/integrate/data-engineering-etl/amazon-sagemaker) [![](/docs/_astro/aws.DojyZ-Di.png) ![](/docs/_astro/aws-dark.Bg3PLj71.png)](/integrate/secure-connectivity/aws) [Secure connectivity](/integrate/secure-connectivity/aws) ### [Amazon Web Services](/integrate/secure-connectivity/aws) [Connect your AWS infrastructure to Tiger Cloud using AWS Transit Gateway](/integrate/secure-connectivity/aws) [![](/docs/_astro/apache-airflow.D5EHMtIZ.png)](/integrate/data-engineering-etl/apache-airflow) [Data engineering & ETL](/integrate/data-engineering-etl/apache-airflow) ### [Apache Airflow](/integrate/data-engineering-etl/apache-airflow) [Author, schedule, and monitor workflows to orchestrate your data pipelines](/integrate/data-engineering-etl/apache-airflow) [![](/docs/_astro/tigerlake-destination.BL5yF7HZ.png) ![](/docs/_astro/tigerlake-destination-dark.BGE8rCpw.png)](/integrate/connectors/destination/tigerlake) [Connectors](/integrate/connectors/destination/tigerlake) ### [Apache Iceberg destination connector](/integrate/connectors/destination/tigerlake) [Unify the Tiger Cloud operational architecture with data lake architectures](/integrate/connectors/destination/tigerlake) [![](/docs/_astro/apache-kafka.D5LvM1Jo.png) ![](/docs/_astro/kafka-dark.BW9mhyzR.png)](/integrate/data-engineering-etl/apache-kafka) [Data engineering & ETL](/integrate/data-engineering-etl/apache-kafka) ### [Apache Kafka](/integrate/data-engineering-etl/apache-kafka) [Stream, process, and analyze real-time event data from Apache Kafka topics](/integrate/data-engineering-etl/apache-kafka) [![](/docs/_astro/apache-kafka.D5LvM1Jo.png) ![](/docs/_astro/kafka-dark.BW9mhyzR.png)](/integrate/connectors/source/sync-from-kafka) [Connectors](/integrate/connectors/source/sync-from-kafka) ### [Apache Kafka source connector](/integrate/connectors/source/sync-from-kafka) [Store, query, and analyze your Apache Kafka events in Tiger Cloud](/integrate/connectors/source/sync-from-kafka) [![](/docs/_astro/aws-lambda.DG6s7052.png) ![](/docs/_astro/aws-lambda-dark.CL_o7L21.png)](/integrate/data-engineering-etl/aws-lambda) [Data engineering & ETL](/integrate/data-engineering-etl/aws-lambda) ### [AWS Lambda](/integrate/data-engineering-etl/aws-lambda) [Run serverless code to process and store your data without managing infrastructure](/integrate/data-engineering-etl/aws-lambda) [![](/docs/_astro/azure-data-studio.3aaqduEY.png)](/integrate/query-administration/azure-data-studio) [Query & administration](/integrate/query-administration/azure-data-studio) ### [Azure Data Studio](/integrate/query-administration/azure-data-studio) [Query and manage your database with a cross-platform data analytics tool](/integrate/query-administration/azure-data-studio) [![](/docs/_astro/azure-monitor.D_o5qtne.png)](/integrate/observability-alerting/azure-monitor) [Observability & alerting](/integrate/observability-alerting/azure-monitor) ### [Azure Monitor](/integrate/observability-alerting/azure-monitor) [Export telemetry data to monitor and analyze your service performance and health with Azure Monitor](/integrate/observability-alerting/azure-monitor) [![](/docs/_astro/tigerlake-destination.BL5yF7HZ.png) ![](/docs/_astro/tigerlake-destination-dark.BGE8rCpw.png)](/integrate/code/connect-your-app) [Code & libraries](/integrate/code/connect-your-app) ### [Connect your app](/integrate/code/connect-your-app) [Integrate your app with self-hosted TimescaleDB using your preferred programming language](/integrate/code/connect-your-app) [![](/docs/_astro/datadog.Bsgp6D09.png)](/integrate/observability-alerting/datadog) [Observability & alerting](/integrate/observability-alerting/datadog) ### [Datadog](/integrate/observability-alerting/datadog) [Export telemetry data and collect metrics to monitor your service performance](/integrate/observability-alerting/datadog) [![](/docs/_astro/dbeaver.CFz2mnCH.png)](/integrate/query-administration/dbeaver) [Query & administration](/integrate/query-administration/dbeaver) ### [DBeaver](/integrate/query-administration/dbeaver) [Manage and query your data with a cross-platform SQL editor and administration tool](/integrate/query-administration/dbeaver) [![](/docs/_astro/decodable.DZ8NAHuw.png)](/integrate/data-engineering-etl/decodable) [Data engineering & ETL](/integrate/data-engineering-etl/decodable) ### [Decodable](/integrate/data-engineering-etl/decodable) [Build and manage real-time data pipelines to stream and process your data effortlessly](/integrate/data-engineering-etl/decodable) [![](/docs/_astro/emqx.ChTj3TXl.png)](/integrate/data-ingestion-streaming/emqx) [Data ingestion & streaming](/integrate/data-ingestion-streaming/emqx) ### [EMQX](/integrate/data-ingestion-streaming/emqx) [Connect EMQX to Tiger Cloud and stream MQTT messages into a hypertable](/integrate/data-ingestion-streaming/emqx) [![](/docs/_astro/fivetran.DPKOUKs7.png)](/integrate/data-ingestion-streaming/fivetran) [Data ingestion & streaming](/integrate/data-ingestion-streaming/fivetran) ### [Fivetran](/integrate/data-ingestion-streaming/fivetran) [Automatically sync data from multiple sources to simplify ETL processes](/integrate/data-ingestion-streaming/fivetran) [![](/docs/_astro/google-cloud.CynVrHy8.png)](/integrate/secure-connectivity/google-cloud) [Secure connectivity](/integrate/secure-connectivity/google-cloud) ### [Google Cloud](/integrate/secure-connectivity/google-cloud) [Connect your Google Cloud infrastructure to Tiger Cloud using AWS Transit Gateway](/integrate/secure-connectivity/google-cloud) [![](/docs/_astro/grafana.C_V2qUhy.png)](/integrate/observability-alerting/grafana) [Observability & alerting](/integrate/observability-alerting/grafana) ### [Grafana](/integrate/observability-alerting/grafana) [Query, visualize, and explore your data with interactive dashboards](/integrate/observability-alerting/grafana) [![](/docs/_astro/highbyte.CckL7hx3.png)](/integrate/data-ingestion-streaming/highbyte) [Data ingestion & streaming](/integrate/data-ingestion-streaming/highbyte) ### [HighByte](/integrate/data-ingestion-streaming/highbyte) [Connect operational technology sources, model the data, and stream it to your service](/integrate/data-ingestion-streaming/highbyte) [![](/docs/_astro/hivemq.DFHYitQJ.png) ![](/docs/_astro/hivemq-dark.CHhKOOBf.png)](/integrate/data-ingestion-streaming/hivemq) [Data ingestion & streaming](/integrate/data-ingestion-streaming/hivemq) ### [HiveMQ](/integrate/data-ingestion-streaming/hivemq) [Connect HiveMQ to Tiger Cloud and stream MQTT messages into a hypertable](/integrate/data-ingestion-streaming/hivemq) [![](/docs/_astro/ignition.DVDINpsG.png)](/integrate/data-ingestion-streaming/ignition) [Data ingestion & streaming](/integrate/data-ingestion-streaming/ignition) ### [Ignition](/integrate/data-ingestion-streaming/ignition) [Connect Ignition SCADA to Tiger Cloud and store tag history](/integrate/data-ingestion-streaming/ignition) [![](/docs/_astro/debezium.BRfICIJ5.png)](/integrate/data-engineering-etl/debezium) [Data engineering & ETL](/integrate/data-engineering-etl/debezium) ### [Integrate Debezium with TimescaleDB](/integrate/data-engineering-etl/debezium) [Capture and stream database changes in real time for event-driven architecture](/integrate/data-engineering-etl/debezium) [![](/docs/_astro/kubernetes.C5k7E6O3.png)](/integrate/configuration-deployment/cloudnativepg) [Configuration & deployment](/integrate/configuration-deployment/cloudnativepg) ### [Integrate TimescaleDB with CloudNativePG on Kubernetes](/integrate/configuration-deployment/cloudnativepg) [Deploy and operate TimescaleDB on Kubernetes with CloudNativePG for production-ready time-series workloads](/integrate/configuration-deployment/cloudnativepg) [![](/docs/_astro/kepware-kepserverex.BCL5I7w6.png)](/integrate/data-ingestion-streaming/kepware-kepserverex) [Data ingestion & streaming](/integrate/data-ingestion-streaming/kepware-kepserverex) ### [Kepware KEPServerEX](/integrate/data-ingestion-streaming/kepware-kepserverex) [Stream OPC tag data from Kepware DataLogger into Tiger Cloud and store it in a hypertable](/integrate/data-ingestion-streaming/kepware-kepserverex) [![](/docs/_astro/kubernetes.C5k7E6O3.png)](/integrate/configuration-deployment/kubernetes) [Configuration & deployment](/integrate/configuration-deployment/kubernetes) ### [Kubernetes](/integrate/configuration-deployment/kubernetes) [Automate deployment, scaling, and management of your containerized workloads](/integrate/configuration-deployment/kubernetes) [![](/docs/_astro/litmus-edge.Dw40ohCb.png) ![](/docs/_astro/litmus-edge-dark.BTBdtIiX.png)](/integrate/data-ingestion-streaming/litmus-edge) [Data ingestion & streaming](/integrate/data-ingestion-streaming/litmus-edge) ### [Litmus Edge](/integrate/data-ingestion-streaming/litmus-edge) [Stream industrial device data from Litmus Edge into your service using the PostgreSQL integration connector](/integrate/data-ingestion-streaming/litmus-edge) [![](/docs/_astro/exported-metrics.C79Q0cGO.png) ![](/docs/_astro/exported-metrics-dark.D0rJL-2Z.png)](/integrate/observability-alerting/exported-metrics) ### [Metrics exported by Tiger Data exporters](/integrate/observability-alerting/exported-metrics) [Reference for system and PostgreSQL metrics exported by Tiger Cloud](/integrate/observability-alerting/exported-metrics) [![](/docs/_astro/microsoft-azure.8bTSmnWF.png)](/integrate/secure-connectivity/microsoft-azure) [Secure connectivity](/integrate/secure-connectivity/microsoft-azure) ### [Microsoft Azure](/integrate/secure-connectivity/microsoft-azure) [Connect your Azure infrastructure to Tiger Cloud on Azure using Azure Private Link](/integrate/secure-connectivity/microsoft-azure) [![](/docs/_astro/node-red.DToBbBjK.png)](/integrate/data-ingestion-streaming/node-red) [Data ingestion & streaming](/integrate/data-ingestion-streaming/node-red) ### [Node-RED](/integrate/data-ingestion-streaming/node-red) [Use Node-RED to stream events into Tiger Data and persist them in a hypertable.](/integrate/data-ingestion-streaming/node-red) [![](/docs/_astro/psql.kQHaU3Jr.png)](/integrate/query-administration/pgadmin) [Query & administration](/integrate/query-administration/pgadmin) ### [pgAdmin](/integrate/query-administration/pgadmin) [Query, visualize, and manage your database with an open-source administration tool](/integrate/query-administration/pgadmin) [![](/docs/_astro/psql.kQHaU3Jr.png)](/integrate/query-administration/postgresql) [Query & administration](/integrate/query-administration/postgresql) ### [Postgres](/integrate/query-administration/postgresql) [Query other Postgres data bases using foreign data wrappers](/integrate/query-administration/postgresql) [![](/docs/_astro/psql.kQHaU3Jr.png)](/integrate/connectors/source/sync-from-postgres) [Connectors](/integrate/connectors/source/sync-from-postgres) ### [PostgreSQL source connector](/integrate/connectors/source/sync-from-postgres) [Sync updates to your primary Postgres database with Tiger Cloud in real time](/integrate/connectors/source/sync-from-postgres) [![](/docs/_astro/power-bi.8VDXuUZ7.png)](/integrate/bi-vizualization/power-bi) [BI & visualization](/integrate/bi-vizualization/power-bi) ### [Power BI](/integrate/bi-vizualization/power-bi) [Create interactive reports and dashboards to analyze and share insights from your data](/integrate/bi-vizualization/power-bi) [![](/docs/_astro/prometheus.Bg2u0eSS.png)](/integrate/observability-alerting/prometheus) [Observability & alerting](/integrate/observability-alerting/prometheus) ### [Prometheus](/integrate/observability-alerting/prometheus) [Export telemetry metrics from your service to monitor system performance and health](/integrate/observability-alerting/prometheus) [![](/docs/_astro/psql.kQHaU3Jr.png)](/integrate/query-administration/psql) [Query & administration](/integrate/query-administration/psql) ### [psql](/integrate/query-administration/psql) [Connect to your Tiger Cloud service and run interactive queries with psql](/integrate/query-administration/psql) [![](/docs/_astro/qstudio.ByVaaBQ4.png)](/integrate/query-administration/qstudio) [Query & administration](/integrate/query-administration/qstudio) ### [qStudio](/integrate/query-administration/qstudio) [Connect to your Tiger Cloud service and visualize your data with qStudio](/integrate/query-administration/qstudio) [![](/docs/_astro/snowflake-logo.DKj2GNfx.png)](/integrate/connectors/destination/snowflake) [Connectors](/integrate/connectors/destination/snowflake) ### [Snowflake](/integrate/connectors/destination/snowflake) [Query Tiger Cloud time-series data from Snowflake using Apache Iceberg and Amazon S3 Tables, without ETL pipelines or data duplication.](/integrate/connectors/destination/snowflake) [![](/docs/_astro/supabase.DrHzjiH8.png)](/integrate/data-engineering-etl/supabase) [Data engineering & ETL](/integrate/data-engineering-etl/supabase) ### [Supabase](/integrate/data-engineering-etl/supabase) [Run real-time analytical queries across databases using foreign data wrappers](/integrate/data-engineering-etl/supabase) [![](/docs/_astro/tableau.DrDeSIv1.png)](/integrate/bi-vizualization/tableau) [BI & visualization](/integrate/bi-vizualization/tableau) ### [Tableau](/integrate/bi-vizualization/tableau) [Visualize and analyze your data to gain business intelligence and share insights](/integrate/bi-vizualization/tableau) [![](/docs/_astro/telegraf.Cek8pwKe.png) ![](/docs/_astro/telegraf-dark.BLThQpAr.png)](/integrate/observability-alerting/telegraf) [Observability & alerting](/integrate/observability-alerting/telegraf) ### [Telegraf](/integrate/observability-alerting/telegraf) [Ingest metrics and events from databases, systems, and IoT sensors](/integrate/observability-alerting/telegraf) [![](/docs/_astro/terraform.Djl_w5v9.png)](/integrate/configuration-deployment/terraform) [Configuration & deployment](/integrate/configuration-deployment/terraform) ### [Terraform](/integrate/configuration-deployment/terraform) [Provision and manage your infrastructure as code with predictable deployments](/integrate/configuration-deployment/terraform) [![](/docs/_astro/corporate-data-center.Dd56KRrg.png) ![](/docs/_astro/corporate-data-center-dark.DPOq3H0l.png)](/integrate/secure-connectivity/corporate-data-center) [Secure connectivity](/integrate/secure-connectivity/corporate-data-center) ### [Your data center](/integrate/secure-connectivity/corporate-data-center) [Connect your on-premise infrastructure to Tiger Cloud using AWS Transit Gateway](/integrate/secure-connectivity/corporate-data-center) ## PostgreSQL compatibility [Section titled “PostgreSQL compatibility”](#postgresql-compatibility) Tiger Data is compatible with PostgreSQL, so any tool that works with PostgreSQL should work with Tiger Data. # BI & visualization > Use your BI and visualization tools with Tiger Data The guides below show how to connect BI and visualization tools to Tiger Data, enabling one to build dashboards, reports, and interactive analytics on your time-series and relational data and share insights across your team. ## Integrations [Section titled “Integrations”](#integrations) [Power BI](/docs/integrate/bi-vizualization/power-bi) [Create interactive reports and dashboards to analyze and share insights from your data](/docs/integrate/bi-vizualization/power-bi) [Tableau](/docs/integrate/bi-vizualization/tableau) [Visualize and analyze your data to gain business intelligence and share insights](/docs/integrate/bi-vizualization/tableau) # Integrate Power BI with Tiger Cloud > Create interactive reports and dashboards to analyze and share insights from your data [Power BI](https://www.microsoft.com/en-us/power-platform/products/power-bi/) is a business analytics tool for visualizing data, creating interactive reports, and sharing insights across an organization. This page explains how to integrate Power BI with Tiger Cloud using the PostgreSQL ODBC driver, so that you can build interactive reports based on the data in your Tiger Cloud service. ## Prerequisites for this integration guide To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service). These steps use Tiger Cloud, but the same approach applies to a [self-hosted TimescaleDB](/docs/get-started/choose-your-path/install-timescaledb) instance. - Your [connection details](/docs/integrate/find-connection-details). * [Power BI Desktop](https://www.microsoft.com/en-us/power-platform/products/power-bi/downloads) installed on your Microsoft Windows machine. * The [PostgreSQL ODBC driver](https://www.postgresql.org/ftp/odbc/releases/) installed. ## Add your service as an ODBC data source [Section titled “Add your service as an ODBC data source”](#add-your-service-as-an-odbc-data-source) Use the PostgreSQL ODBC driver to connect Power BI to Tiger Cloud. 1. **Open the ODBC data sources** On your Windows machine, search for and select `ODBC Data Sources`. 2. **Connect to your Tiger Cloud service** 1. Under `User DSN`, click `Add`. 2. Choose `PostgreSQL Unicode` and click `Finish`. 3. Use your [connection details](/docs/integrate/find-connection-details) to configure the data source. 4. Click `Test` to ensure the connection works, then click `Save`. ## Import the data from your service into Power BI [Section titled “Import the data from your service into Power BI”](#import-the-data-from-your-service-into-power-bi) Establish a connection and import data from your Tiger Cloud service into Power BI: 1. **Connect Power BI to your Tiger Cloud service** 1. Open Power BI, then click `Get data from other sources`. 2. Search for and select `ODBC`, then click `Connect`. 3. In `Data source name (DSN)`, select the Tiger Cloud data source and click `OK`. 4. Use your [connection details](/docs/integrate/find-connection-details) to enter your `User Name` and `Password`, then click `Connect`. After connecting, `Navigator` displays the available tables and schemas. 2. **Import your data into Power BI** 1. Select the tables to import and click `Load`. The `Data` pane shows your imported tables. 2. To visualize your data and build reports, drag fields from the tables onto the canvas. You have successfully integrated Power BI with Tiger Cloud. # Integrate Tableau and Tiger Cloud > Visualize and analyze your data to gain business intelligence and share insights [Tableau](https://www.tableau.com/) is a popular analytics platform that helps you gain greater intelligence about your business. You can use it to visualize data stored in Tiger Cloud. ## Prerequisites for this integration guide To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service). These steps use Tiger Cloud, but the same approach applies to a [self-hosted TimescaleDB](/docs/get-started/choose-your-path/install-timescaledb) instance. - Your [connection details](/docs/integrate/find-connection-details). * A [Tableau Server](https://www.tableau.com/support/releases/server/2024.2.6#esdalt) installation or a [Tableau Cloud](https://www.tableau.com/products/trial) account. ## Add your service as a virtual connection [Section titled “Add your service as a virtual connection”](#add-your-service-as-a-virtual-connection) To connect the data in your Tiger Cloud service to Tableau: 1. **Log in to Tableau** * Tableau Cloud: [sign in](http://online.tableau.com/), then click `Explore` and select a project. * Tableau Desktop: sign in, then open a workbook. 2. **Configure Tableau to connect to your Tiger Cloud service** 1. Add a new data source: * Tableau Cloud: click `New` > `Virtual Connection`. * Tableau Desktop: click `Data` > `New Data Source`. 2. Search for and select `PostgreSQL`. For Tableau Desktop, download the driver and restart Tableau. 3. Configure the connection: * `Server`, `Port`, `Database`, `Username`, `Password`: configure using your [connection details](/docs/integrate/find-connection-details). * `Require SSL`: tick the checkbox. 3. **Click `Sign In` and connect Tableau to your service** You have successfully integrated Tableau with Tiger Cloud. # Code and libraries > Connect applications to Tiger Data using your preferred programming language and tools ## Integrations [Section titled “Integrations”](#integrations) [Connect your app](/docs/integrate/code/connect-your-app) [Language-specific examples for connecting your app to self-hosted TimescaleDB](/docs/integrate/code/connect-your-app) # Connect your app > Integrate your app with self-hosted TimescaleDB using your preferred programming language Easily integrate your app with self-hosted TimescaleDB. Use your favorite programming language to connect to your database, create and manage hypertables, then ingest and query data. * Ruby ## Prerequisites for this tutorial To follow these steps, you'll need: * A [self-hosted TimescaleDB](/docs/get-started/choose-your-path/install-timescaledb) instance. - Your [connection details](/docs/integrate/find-connection-details). * [Rails](https://guides.rubyonrails.org/install_ruby_on_rails.html#installing-rails) installed. ## Connect a Rails app to TimescaleDB TimescaleDB extends PostgreSQL with time-series superpowers. You connect to your TimescaleDB database from a standard Rails app configured for PostgreSQL. 1. **Create a new Rails app configured for PostgreSQL** Rails creates and bundles your app, then installs the standard PostgreSQL Gems. ```bash rails new my_app -d=postgresql cd my_app ``` 2. **Install the TimescaleDB gem** 1. Open `Gemfile`, add the following line, then save your changes: ```ruby gem 'timescaledb' ``` 2. In the terminal, run the following command: ```bash bundle install ``` 3. **Connect your app to TimescaleDB** 1. In `/config/database.yml` update the configuration to securely connect to your TimescaleDB database by adding `url: <%= ENV['DATABASE_URL'] %>` to the default configuration: ```yaml default: &default adapter: postgresql encoding: unicode pool: <%= ENV.fetch("RAILS_MAX_THREADS") { 5 } %> url: <%= ENV['DATABASE_URL'] %> ``` 2. Set the environment variable for `DATABASE_URL` to your connection string: ```bash export DATABASE_URL="postgres://username:password@host:port/dbname" ``` 3. Create the database for the project: ```bash rails db:create ``` 4. Run migrations: ```bash rails db:migrate ``` 5. Verify the connection from your app to your TimescaleDB database: ```bash echo "\dx" | rails dbconsole ``` The result shows the list of extensions in your database | Name | Version | Schema | Description | | -------------------- | ------- | ----------- | ------------------------------------------------------------------------------------- | | pg\_buffercache | 1.5 | public | examine the shared buffer cache | | pg\_stat\_statements | 1.11 | public | track planning and execution statistics of all SQL statements executed | | plpgsql | 1.0 | pg\_catalog | PL/pgSQL procedural language | | postgres\_fdw | 1.1 | public | foreign-data wrapper for remote PostgreSQL servers | | timescaledb | 2.24.0 | public | Enables scalable inserts and complex queries for time-series data (Community Edition) | | timescaledb\_toolkit | 1.22.0 | public | Library of analytical hyperfunctions, time-series pipelining, and other SQL utilities | ## Optimize time-series data in hypertables Hypertables are PostgreSQL tables designed to simplify and accelerate data analysis. Anything you can do with regular PostgreSQL tables, you can do with hypertables - but much faster and more conveniently. In this section, you use the helpers in the TimescaleDB gem to create and manage a [hypertable](/docs/build/how-to/your-first-hypertable). 1. **Generate a migration to create the page loads table** ```bash rails generate migration create_page_loads ``` This creates the `/db/migrate/_create_page_loads.rb` migration file. 2. **Add hypertable options** Replace the contents of `/db/migrate/_create_page_loads.rb` with the following: ```ruby class CreatePageLoads < ActiveRecord::Migration[8.0] def change hypertable_options = { time_column: 'created_at', chunk_time_interval: '1 day', compress_segmentby: 'path', compress_orderby: 'created_at', compress_after: '7 days', drop_after: '30 days' } create_table :page_loads, id: false, primary_key: [:created_at, :user_agent, :path], hypertable: hypertable_options do |t| t.timestamptz :created_at, null: false t.string :user_agent t.string :path t.float :performance end end end ``` The `id` column is not included in the table. This is because TimescaleDB requires that any `UNIQUE` or `PRIMARY KEY` indexes on the table include all partitioning columns. In this case, this is the time column. A new Rails model includes a `PRIMARY KEY` index for id by default: either remove the column or make sure that the index includes time as part of a "composite key." For more information, check the Roby docs around [composite primary keys](https://guides.rubyonrails.org/active_record_composite_primary_keys.html). 3. **Create a `PageLoad` model** Create a new file called `/app/models/page_load.rb` and add the following code: ```ruby class PageLoad < ApplicationRecord extend Timescaledb::ActsAsHypertable include Timescaledb::ContinuousAggregatesHelper acts_as_hypertable time_column: "created_at", segment_by: "path", value_column: "performance" # Basic scopes for filtering by browser scope :chrome_users, -> { where("user_agent LIKE ?", "%Chrome%") } scope :firefox_users, -> { where("user_agent LIKE ?", "%Firefox%") } scope :safari_users, -> { where("user_agent LIKE ?", "%Safari%") } # Performance analysis scopes scope :performance_stats, -> { select("stats_agg(#{value_column}) as stats_agg") } scope :slow_requests, -> { where("performance > ?", 1.0) } scope :fast_requests, -> { where("performance < ?", 0.1) } # Set up continuous aggregates for different timeframes continuous_aggregates scopes: [:performance_stats], timeframes: [:minute, :hour, :day], refresh_policy: { minute: { start_offset: '3 minute', end_offset: '1 minute', schedule_interval: '1 minute' }, hour: { start_offset: '3 hours', end_offset: '1 hour', schedule_interval: '1 minute' }, day: { start_offset: '3 day', end_offset: '1 day', schedule_interval: '1 minute' } } end ``` 4. **Run the migration** ```bash rails db:migrate ``` ## Insert data into your database The TimescaleDB gem provides efficient ways to insert data into hypertables. This section shows you how to ingest test data into your hypertable. 1. **Create a controller to handle page loads** Create a new file called `/app/controllers/application_controller.rb` and add the following code: ```ruby class ApplicationController < ActionController::Base around_action :track_page_load private def track_page_load start_time = Time.current yield end_time = Time.current PageLoad.create( path: request.path, user_agent: request.user_agent, performance: (end_time - start_time) ) end end ``` 2. **Generate some test data** Use `bin/console` to join a Rails console session and run the following code to define some random page load access data: ```ruby def generate_sample_page_loads(total: 1000) time = 1.month.ago paths = %w[/ /about /contact /products /blog] browsers = [ "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:89.0) Gecko/20100101 Firefox/89.0", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.1 Safari/605.1.15" ] total.times.map do time = time + rand(60).seconds { path: paths.sample, user_agent: browsers.sample, performance: rand(0.1..2.0), created_at: time, updated_at: time } end end ``` 3. **Insert the generated data into your TimescaleDB database** ```bash # Insert the data in batches PageLoad.insert_all(generate_sample_page_loads, returning: false) ``` 4. **Validate the test data in your TimescaleDB database** ```bash PageLoad.count PageLoad.first ``` ## Reference This section lists the most common tasks you might perform with the TimescaleDB gem. ### Query scopes The TimescaleDB gem provides several convenient scopes for querying your time-series data. * Built-in time-based scopes: ```ruby PageLoad.last_hour.count PageLoad.today.count PageLoad.this_week.count PageLoad.this_month.count ``` * Browser-specific scopes: ```ruby # Count requests by browser PageLoad.chrome_users.last_hour.count PageLoad.firefox_users.last_hour.count PageLoad.safari_users.last_hour.count # Performance analysis PageLoad.slow_requests.last_hour.count PageLoad.fast_requests.last_hour.count ``` * Query continuous aggregates: This query fetches the average and standard deviation from the performance stats for the `/products` path over the last day. ```ruby # Access aggregated performance stats through generated classes PageLoad::PerformanceStatsPerMinute.last_hour PageLoad::PerformanceStatsPerHour.last_day PageLoad::PerformanceStatsPerDay.last_month # Get statistics for a specific path stats = PageLoad::PerformanceStatsPerHour.last_day.where(path: '/products').select("average(stats_agg) as average, stddev(stats_agg) as stddev").first puts "Average: #{stats.average}" puts "Standard Deviation: #{stats.stddev}" ``` ### TimescaleDB features The TimescaleDB gem provides utility methods to access hypertable and chunk information. Every model that uses the `acts_as_hypertable` method has access to these methods. #### Access hypertable and chunk information * View chunk or hypertable information: ```ruby PageLoad.chunks.count PageLoad.hypertable.detailed_size ``` * Compress/Decompress chunks: ```ruby PageLoad.chunks.uncompressed.first.compress! # Compress the first uncompressed chunk PageLoad.chunks.compressed.first.decompress! # Decompress the oldest chunk PageLoad.hypertable.compression_stats # View compression stats ``` #### Access hypertable stats You collect hypertable stats using methods that provide insights into your hypertable's structure, size, and compression status: * Get basic hypertable information: ```ruby hypertable = PageLoad.hypertable hypertable.hypertable_name # The name of your hypertable hypertable.schema_name # The schema where the hypertable is located ``` * Get detailed size information: ```ruby hypertable.detailed_size # Get detailed size information for the hypertable hypertable.compression_stats # Get compression statistics hypertable.chunks_detailed_size # Get chunk information hypertable.approximate_row_count # Get approximate row count hypertable.dimensions.map(&:column_name) # Get dimension information hypertable.continuous_aggregates.map(&:view_name) # Get continuous aggregate view names ``` #### Continuous aggregates The `continuous_aggregates` method generates a class for each continuous aggregate. * Get all the continuous aggregate classes: ```ruby PageLoad.descendants # Get all continuous aggregate classes ``` * Manually refresh a continuous aggregate: ```ruby PageLoad.refresh_aggregates ``` * Create or drop a continuous aggregate: Create or drop all the continuous aggregates in the proper order to build them hierarchically. See more about how it works in this [blog post](https://www.tigerdata.com/blog/building-a-better-ruby-orm-for-time-series-and-analytics). ```ruby PageLoad.create_continuous_aggregates PageLoad.drop_continuous_aggregates ``` ## Next steps Now that you have integrated the ruby gem into your app: * Learn more about the [TimescaleDB gem](https://github.com/timescale/timescaledb-ruby). * Check out the [official docs](https://timescale.github.io/timescaledb-ruby/). * Follow the [LTTB](https://timescale.github.io/timescaledb-ruby/toolkit_lttb_tutorial/), [Open AI long-term storage](https://timescale.github.io/timescaledb-ruby/chat_gpt_tutorial/), and [candlesticks](https://timescale.github.io/timescaledb-ruby/toolkit_candlestick/) tutorials. * Python ## Prerequisites for this tutorial To follow these steps, you'll need: * A [self-hosted TimescaleDB](/docs/get-started/choose-your-path/install-timescaledb) instance. - Your [connection details](/docs/integrate/find-connection-details). * The `psycopg2` library installed. For more information, see the [psycopg2 documentation](https://pypi.org/project/psycopg2/). * (Optional) A [Python virtual environment](https://docs.python.org/3/library/venv.html). ## Connect to TimescaleDB In this section, you create a connection to TimescaleDB using the `psycopg2` library. This library is one of the most popular PostgreSQL libraries for Python. It allows you to execute raw SQL queries efficiently and safely, and prevents common attacks such as SQL injection. 1. **Import the psycogpg2 library:** ```python import psycopg2 ``` 2. **Locate your TimescaleDB credentials** Use them to compose a connection string for `psycopg2`. You'll need: * password * username * host URL * port * database name 3. **Compose your connection string variable** Use a [libpq connection string](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING) format: ```python CONNECTION = "postgres://username:password@host:port/dbname" ``` If you require an SSL connection, use this version instead: ```python CONNECTION = "postgres://username:password@host:port/dbname?sslmode=require" ``` Alternatively you can specify each parameter in the connection string as follows ```python CONNECTION = "dbname=tsdb user=tsdbadmin password=secret host=host.com port=5432 sslmode=require" ``` Warning This method of composing a connection string is for test or development purposes only. For production, use environment variables for sensitive details like your password, hostname, and port number. 4. **Create a database session with psycopg2** Use the `psycopg2` connect function to create a new database session and a new [cursor object](https://www.psycopg.org/docs/connection.html?highlight=cursor#connection.cursor) to interact with the database. In your `main` function, add these lines: ```python CONNECTION = "postgres://username:password@host:port/dbname" with psycopg2.connect(CONNECTION) as conn: cursor = conn.cursor() # use the cursor to interact with your database # cursor.execute("SELECT * FROM table") ``` Alternatively, you can create a connection object and pass the object around as needed, like opening a cursor to perform database operations: ```python CONNECTION = "postgres://username:password@host:port/dbname" conn = psycopg2.connect(CONNECTION) cursor = conn.cursor() # use the cursor to interact with your database cursor.execute("SELECT 'hello world'") print(cursor.fetchone()) ``` ## Create a relational table In this section, you create a table called `sensors` which holds the ID, type, and location of your fictional sensors. Additionally, you create a hypertable called `sensor_data` which holds the measurements of those sensors. The measurements contain the time, sensor\_id, temperature reading, and CPU percentage of the sensors. 1. **Compose the SQL statement to create a relational table** This example creates a table called `sensors`, with columns `id`, `type` and `location`: ```python query_create_sensors_table = """CREATE TABLE sensors ( id SERIAL PRIMARY KEY, type VARCHAR(50), location VARCHAR(50) ); """ ``` 2. **Execute the query and commit the changes** Open a cursor, execute the query you created in the previous step, and commit the query to make the changes persistent. Afterward, close the cursor to clean up: ```python cursor = conn.cursor() # see definition in Step 1 cursor.execute(query_create_sensors_table) conn.commit() cursor.close() ``` ## Create a hypertable When you have created the relational table, you can create a hypertable. Creating tables and indexes, altering tables, inserting data, selecting data, and most other tasks are executed on the hypertable. 1. **Create the `CREATE TABLE` SQL statement for your hypertable** Notice how the hypertable has the compulsory time column: ```python # create sensor data hypertable query_create_sensordata_table = """CREATE TABLE sensor_data ( time TIMESTAMPTZ NOT NULL, sensor_id INTEGER, temperature DOUBLE PRECISION, cpu DOUBLE PRECISION, FOREIGN KEY (sensor_id) REFERENCES sensors (id) ); """ ``` 2. **Convert the table to a hypertable** Formulate a `SELECT` statement that converts the `sensor_data` table to a hypertable. You must specify the table name to convert to a hypertable, and the name of the time column as the two arguments. For more information, see the [`create_hypertable` docs](/docs/reference/timescaledb/hypertables/create_hypertable): ```python query_create_sensordata_hypertable = "SELECT create_hypertable('sensor_data', by_range('time'));" ``` Note The `by_range` dimension builder is an addition to TimescaleDB 2.13. 3. **Execute the statements and commit your changes** Open a cursor with the connection, execute the statements from the previous steps, commit your changes, and close the cursor: ```python cursor = conn.cursor() cursor.execute(query_create_sensordata_table) cursor.execute(query_create_sensordata_hypertable) # commit changes to the database to make changes persistent conn.commit() cursor.close() ``` ## Insert rows of data You can insert data into your hypertables in several different ways. In this section, you can use `psycopg2` with prepared statements, or you can use `pgcopy` for a faster insert. 1. **Insert relational data into the `sensors` table** Open a cursor with a connection to the database, use prepared statements to formulate the `INSERT` SQL statement, and then execute that statement: ```python sensors = [('a', 'floor'), ('a', 'ceiling'), ('b', 'floor'), ('b', 'ceiling')] cursor = conn.cursor() for sensor in sensors: try: cursor.execute("INSERT INTO sensors (type, location) VALUES (%s, %s);", (sensor[0], sensor[1])) except (Exception, psycopg2.Error) as error: print(error.pgerror) conn.commit() ``` 2. **(Optional) Use separate SQL and data variables** Alternatively, you can pass variables to the `cursor.execute` function and separate the formulation of the SQL statement, `SQL`, from the data being passed with it into the prepared statement, `data`: ```python SQL = "INSERT INTO sensors (type, location) VALUES (%s, %s);" sensors = [('a', 'floor'), ('a', 'ceiling'), ('b', 'floor'), ('b', 'ceiling')] cursor = conn.cursor() for sensor in sensors: try: data = (sensor[0], sensor[1]) cursor.execute(SQL, data) except (Exception, psycopg2.Error) as error: print(error.pgerror) conn.commit() ``` If you choose to use `pgcopy` instead, install the `pgcopy` package [using pip](https://pypi.org/project/pgcopy/), and then add this line to your list of `import` statements: ```python from pgcopy import CopyManager ``` 1. **Generate random sensor data** Use the `generate_series` function provided by PostgreSQL. This example inserts a total of 480 rows of data (4 readings, every 5 minutes, for 24 hours). In your application, this would be the query that saves your time-series data into the hypertable: ```python # for sensors with ids 1-4 for id in range(1, 4, 1): data = (id,) # create random data simulate_query = """SELECT generate_series(now() - interval '24 hour', now(), interval '5 minute') AS time, %s as sensor_id, random()*100 AS temperature, random() AS cpu; """ cursor.execute(simulate_query, data) values = cursor.fetchall() ``` 2. **Define the column names for the target table** This example uses the `sensor_data` hypertable created earlier. This hypertable consists of columns named `time`, `sensor_id`, `temperature` and `cpu`. The column names are defined in a list of strings called `cols`: ```python cols = ['time', 'sensor_id', 'temperature', 'cpu'] ``` 3. **Insert data using pgcopy CopyManager** Create an instance of the `pgcopy` CopyManager, `mgr`, and pass the connection variable, hypertable name, and list of column names. Then use the `copy` function to insert the data quickly. ```python mgr = CopyManager(conn, 'sensor_data', cols) mgr.copy(values) ``` 4. **Commit to persist changes:** ```python conn.commit() ``` 5. **(Optional) Full sample code for pgcopy insert** The full sample code to insert data into TimescaleDB using `pgcopy`, using the example of sensor data from four sensors: ```python # insert using pgcopy def fast_insert(conn): cursor = conn.cursor() # for sensors with ids 1-4 for id in range(1, 4, 1): data = (id,) # create random data simulate_query = """SELECT generate_series(now() - interval '24 hour', now(), interval '5 minute') AS time, %s as sensor_id, random()*100 AS temperature, random() AS cpu; """ cursor.execute(simulate_query, data) values = cursor.fetchall() # column names of the table you're inserting into cols = ['time', 'sensor_id', 'temperature', 'cpu'] # create copy manager with the target table and insert mgr = CopyManager(conn, 'sensor_data', cols) mgr.copy(values) # commit after all sensor data is inserted # could also commit after each sensor insert is done conn.commit() ``` 6. **(Optional) You can also check if the insertion worked:** ```python cursor.execute("SELECT * FROM sensor_data LIMIT 5;") print(cursor.fetchall()) ``` ## Execute a query This section covers how to execute queries against your database. The first procedure shows a simple `SELECT *` query. For more complex queries, you can use prepared statements to ensure queries are executed safely against the database. For more information about properly using placeholders in `psycopg2`, see the [basic module usage document](https://www.psycopg.org/docs/usage.html). For more information about how to execute more complex queries in `psycopg2`, see the [psycopg2 documentation](https://www.psycopg.org/docs/usage.html). ### Execute a query 1. **Define the SQL query** This example is a simple `SELECT` statement querying each row from the previously created `sensor_data` table. ```python query = "SELECT * FROM sensor_data;" ``` 2. **Execute the query** Open a cursor from the existing database connection, `conn`, and then execute the query you defined: ```python cursor = conn.cursor() query = "SELECT * FROM sensor_data;" cursor.execute(query) ``` 3. **Access the query results** Use one of `psycopg2`'s [results retrieval methods](https://www.psycopg.org/docs/cursor.html), such as `fetchall()` or `fetchmany()`. This example prints the results of the query, row by row. Note that the result of `fetchall()` is a list of tuples, so you can handle them accordingly: ```python cursor = conn.cursor() query = "SELECT * FROM sensor_data;" cursor.execute(query) for row in cursor.fetchall(): print(row) cursor.close() ``` 4. **(Optional) Use DictCursor for dictionary results** If you want a list of dictionaries instead, define the cursor using [`DictCursor`](https://www.psycopg.org/docs/extras.html#dictionary-like-cursor): ```python cursor = conn.cursor(cursor_factory=psycopg2.extras.DictCursor) ``` Using this cursor, `cursor.fetchall()` returns a list of dictionary-like objects. For more complex queries, you can use prepared statements to ensure queries are executed safely against the database. ### Execute queries using prepared statements 1. **Write the query using prepared statements:** ```python # query with placeholders cursor = conn.cursor() query = """ SELECT time_bucket('5 minutes', time) AS five_min, avg(cpu) FROM sensor_data JOIN sensors ON sensors.id = sensor_data.sensor_id WHERE sensors.location = %s AND sensors.type = %s GROUP BY five_min ORDER BY five_min DESC; """ location = "floor" sensor_type = "a" data = (location, sensor_type) cursor.execute(query, data) results = cursor.fetchall() ``` * Node.js ## Prerequisites for this tutorial To follow these steps, you'll need: * A [self-hosted TimescaleDB](/docs/get-started/choose-your-path/install-timescaledb) instance. - Your [connection details](/docs/integrate/find-connection-details). * [Node.js](https://nodejs.org) installed. * The Node.js package manager [npm](https://docs.npmjs.com/getting-started) installed. ## Connect to TimescaleDB In this section, you create a connection to TimescaleDB with a common Node.js ORM (object relational mapper) called [Sequelize](https://sequelize.org). 1. **Initialize a new Node.js app** At the command prompt, initialize a new Node.js app: ```bash npm init -y ``` This creates a `package.json` file in your directory, which contains all of the dependencies for your project. It looks something like this: ```json { "name": "node-sample", "version": "1.0.0", "description": "", "main": "index.js", "scripts": { "test": "echo \"Error: no test specified\" && exit 1" }, "keywords": [], "author": "", "license": "ISC" } ``` 2. **Install Express.js** Install Express.js: ```bash npm install express ``` 3. **Create a simple web page** Create a simple web page to check the connection. Create a new file called `index.js`, with this content: ```js const express = require('express') const app = express() const port = 3000; app.use(express.json()); app.get('/', (req, res) => res.send('Hello World!')) app.listen(port, () => console.log(`Example app listening at http://localhost:${port}`)) ``` 4. **Test your connection** Test your connection by starting the application: ```bash node index.js ``` In your web browser, navigate to `http://localhost:3000`. If the connection is successful, it shows "Hello World!" 5. **Add Sequelize to your project** Add Sequelize to your project: ```bash npm install sequelize sequelize-cli pg pg-hstore ``` 6. **Locate your credentials** Locate your TimescaleDB credentials and use them to compose a connection string for Sequelize. You'll need: * password * username * host URL * port * database name 7. **Compose the connection string** Compose your connection string variable, using this format: ```js 'postgres://:@:/' ``` 8. **Configure Sequelize in index.js** Open the `index.js` file you created. Require Sequelize in the application, and declare the connection string: ```js const Sequelize = require('sequelize') const sequelize = new Sequelize('postgres://:@:/', { dialect: 'postgres', protocol: 'postgres', dialectOptions: { ssl: { require: true, rejectUnauthorized: false } } }) ``` Make sure you add the SSL settings in the `dialectOptions` section if your TimescaleDB instance requires SSL connections. 9. **Test the database connection** You can test the connection by adding these lines to `index.js` after the `app.get` statement: ```js sequelize.authenticate().then(() => { console.log('Connection has been established successfully.'); }).catch(err => { console.error('Unable to connect to the database:', err); }); ``` Start the application on the command line: ```bash node index.js ``` If the connection is successful, you'll get output like this: ```bash Example app listening at http://localhost:3000 Executing (default): SELECT 1+1 AS result Connection has been established successfully. ``` ## Create a relational table In this section, you create a relational table called `page_loads`. 1. **Generate the model and migration** Use the Sequelize command line tool to create a table and model called `page_loads`: ```bash npx sequelize model:generate --name page_loads \ --attributes userAgent:string,time:date ``` The output looks similar to this: ```bash Sequelize CLI [Node: 12.16.2, CLI: 5.5.1, ORM: 5.21.11] New model was created at . New migration was created at . ``` 2. **Edit the migration file** Edit the migration file so that it sets up a migration key: ```js 'use strict'; module.exports = { up: async (queryInterface, Sequelize) => { await queryInterface.createTable('page_loads', { userAgent: { primaryKey: true, type: Sequelize.STRING }, time: { primaryKey: true, type: Sequelize.DATE } }); }, down: async (queryInterface, Sequelize) => { await queryInterface.dropTable('page_loads'); } }; ``` 3. **Run the migration** Migrate the change and make sure that it is reflected in the database: ```bash npx sequelize db:migrate ``` The output looks similar to this: ```bash Sequelize CLI [Node: 12.16.2, CLI: 5.5.1, ORM: 5.21.11] Loaded configuration file "config/config.json". Using environment "development". == 20200528195725-create-page-loads: migrating ======= == 20200528195725-create-page-loads: migrated (0.443s) ``` 4. **Create the PageLoads model** Create the `PageLoads` model in your code. In the `index.js` file, above the `app.use` statement, add these lines: ```js let PageLoads = sequelize.define('page_loads', { userAgent: {type: Sequelize.STRING, primaryKey: true }, time: {type: Sequelize.DATE, primaryKey: true } }, { timestamps: false }); ``` 5. **Save data to the database** Instantiate a `PageLoads` object and save it to the database. ## Create a hypertable When you have created the relational table, you can create a hypertable. Creating tables and indexes, altering tables, inserting data, selecting data, and most other tasks are executed on the hypertable. 1. **Generate the hypertable migration** Create a migration to modify the `page_loads` relational table, and change it to a hypertable by first running the following command: ```bash npx sequelize migration:generate --name add_hypertable ``` The output looks similar to this: ```bash Sequelize CLI [Node: 12.16.2, CLI: 5.5.1, ORM: 5.21.11] migrations folder at already exists. New migration was created at /20200601202912-add_hypertable.js . ``` 2. **Add the hypertable creation query** In the `migrations` folder, there is now a new file. Open the file, and add this content: ```js 'use strict'; module.exports = { up: (queryInterface, Sequelize) => { return queryInterface.sequelize.query("SELECT create_hypertable('page_loads', by_range('time'));"); }, down: (queryInterface, Sequelize) => { } }; ``` Note The `by_range` dimension builder is an addition to TimescaleDB 2.13. 3. **Run the hypertable migration** At the command prompt, run the migration command: ```bash npx sequelize db:migrate ``` The output looks similar to this: ```bash Sequelize CLI [Node: 12.16.2, CLI: 5.5.1, ORM: 5.21.11] Loaded configuration file "config/config.json". Using environment "development". == 20200601202912-add_hypertable: migrating ======= == 20200601202912-add_hypertable: migrated (0.426s) ``` ## Insert rows of data This section covers how to insert data into your hypertables. 1. **Modify the route to insert data** In the `index.js` file, modify the `/` route to get the `user-agent` from the request object (`req`) and the current timestamp. Then, call the `create` method on `PageLoads` model, supplying the user agent and timestamp parameters. The `create` call executes an `INSERT` on the database: ```js app.get('/', async (req, res) => { // get the user agent and current time const userAgent = req.get('user-agent'); const time = new Date().getTime(); try { // insert the record await PageLoads.create({ userAgent, time }); // send response res.send('Inserted!'); } catch (e) { console.log('Error inserting data', e) } }) ``` ## Execute a query This section covers how to execute queries against your database. In this example, every time the page is reloaded, all information currently in the table is displayed. 1. **Retrieve and display all data** Modify the `/` route in the `index.js` file to call the Sequelize `findAll` function and retrieve all data from the `page_loads` table using the `PageLoads` model: ```js app.get('/', async (req, res) => { // get the user agent and current time const userAgent = req.get('user-agent'); const time = new Date().getTime(); try { // insert the record await PageLoads.create({ userAgent, time }); // now display everything in the table const messages = await PageLoads.findAll(); res.send(messages); } catch (e) { console.log('Error inserting data', e) } }) ``` Now, when you reload the page, you should see all of the rows currently in the `page_loads` table. * Go ## Prerequisites for this tutorial To follow these steps, you'll need: * A [self-hosted TimescaleDB](/docs/get-started/choose-your-path/install-timescaledb) instance. - Your [connection details](/docs/integrate/find-connection-details). * [Go](https://golang.org/doc/install) installed. * The [PGX driver for Go](https://github.com/jackc/pgx) installed. ## Connect to TimescaleDB In this section, you create a connection to TimescaleDB using the PGX driver. PGX is a toolkit designed to help Go developers work directly with PostgreSQL. You can use it to help your Go application interact directly with TimescaleDB. 1. **Locate your TimescaleDB credentials** Use them to compose a connection string for PGX. You'll need: * password * username * host URL * port number * database name 2. **Compose your connection string variable** Use a [libpq connection string](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING) format: ```go connStr := "postgres://username:password@host:port/dbname" ``` If you require an SSL connection, use this format instead: ```go connStr := "postgres://username:password@host:port/dbname?sslmode=require" ``` 3. **(Optional) Verify your database connection** You can check that you're connected to your database with this hello world program: ```go package main import ( "context" "fmt" "os" "github.com/jackc/pgx/v5" ) //connect to database using a single connection func main() { /***********************************************/ /* Single Connection to TimescaleDB/ PostgreSQL */ /***********************************************/ ctx := context.Background() connStr := "yourConnectionStringHere" conn, err := pgx.Connect(ctx, connStr) if err != nil { fmt.Fprintf(os.Stderr, "Unable to connect to database: %v\n", err) os.Exit(1) } defer conn.Close(ctx) //run a simple query to check our connection var greeting string err = conn.QueryRow(ctx, "select 'Hello, Timescale!'").Scan(&greeting) if err != nil { fmt.Fprintf(os.Stderr, "QueryRow failed: %v\n", err) os.Exit(1) } fmt.Println(greeting) } ``` If you'd like to specify your connection string as an environment variable, you can use this syntax to access it in place of the `connStr` variable: ```go os.Getenv("DATABASE_CONNECTION_STRING") ``` Alternatively, you can connect to TimescaleDB using a connection pool. Connection pooling is useful to conserve computing resources, and can also result in faster database queries: 1. **Create a connection pool** To create a connection pool for concurrent connections to your database, use the `pgxpool.New()` function instead of `pgx.Connect()`. Also note that this script imports `github.com/jackc/pgx/v5/pgxpool`, instead of `pgx/v5` which was used to create a single connection: ```go package main import ( "context" "fmt" "os" "github.com/jackc/pgx/v5/pgxpool" ) func main() { ctx := context.Background() connStr := "yourConnectionStringHere" dbpool, err := pgxpool.New(ctx, connStr) if err != nil { fmt.Fprintf(os.Stderr, "Unable to connect to database: %v\n", err) os.Exit(1) } defer dbpool.Close() //run a simple query to check our connection var greeting string err = dbpool.QueryRow(ctx, "select 'Hello, Tiger Data (but concurrently)'").Scan(&greeting) if err != nil { fmt.Fprintf(os.Stderr, "QueryRow failed: %v\n", err) os.Exit(1) } fmt.Println(greeting) } ``` ## Create a relational table In this section, you create a table called `sensors` which holds the ID, type, and location of your fictional sensors. Additionally, you create a hypertable called `sensor_data` which holds the measurements of those sensors. The measurements contain the time, sensor\_id, temperature reading, and CPU percentage of the sensors. 1. **Compose the SQL statement to create a relational table** This example creates a table called `sensors`, with columns for ID, type, and location: ```go queryCreateTable := `CREATE TABLE sensors (id SERIAL PRIMARY KEY, type VARCHAR(50), location VARCHAR(50));` ``` 2. **Execute the `CREATE TABLE` statement** Use the `Exec()` function on the `dbpool` object, using the arguments of the current context and the statement string you created: ```go package main import ( "context" "fmt" "os" "github.com/jackc/pgx/v5/pgxpool" ) func main() { ctx := context.Background() connStr := "yourConnectionStringHere" dbpool, err := pgxpool.New(ctx, connStr) if err != nil { fmt.Fprintf(os.Stderr, "Unable to connect to database: %v\n", err) os.Exit(1) } defer dbpool.Close() /********************************************/ /* Create relational table */ /********************************************/ //Create relational table called sensors queryCreateTable := `CREATE TABLE sensors (id SERIAL PRIMARY KEY, type VARCHAR(50), location VARCHAR(50));` _, err = dbpool.Exec(ctx, queryCreateTable) if err != nil { fmt.Fprintf(os.Stderr, "Unable to create SENSORS table: %v\n", err) os.Exit(1) } fmt.Println("Successfully created relational table SENSORS") } ``` ## Generate a hypertable When you have created the relational table, you can create a hypertable. Creating tables and indexes, altering tables, inserting data, selecting data, and most other tasks are executed on the hypertable. 1. **Create a variable for the `CREATE TABLE SQL` statement for your hypertable.** Notice how the hypertable has the compulsory time column: ```go queryCreateTable := `CREATE TABLE sensor_data ( time TIMESTAMPTZ NOT NULL, sensor_id INTEGER, temperature DOUBLE PRECISION, cpu DOUBLE PRECISION, FOREIGN KEY (sensor_id) REFERENCES sensors (id)); ` ``` 2. **Formulate the `SELECT` statement to convert the table into a hypertable** You must specify the table name to convert to a hypertable, and its time column name as the second argument. For more information, see the [`create_hypertable` docs](/docs/reference/timescaledb/hypertables/create_hypertable): ```go queryCreateHypertable := `SELECT create_hypertable('sensor_data', by_range('time'));` ``` Note The `by_range` dimension builder is an addition to TimescaleDB 2.13. 3. **Execute the statements to create the hypertable** Execute the `CREATE TABLE` statement and `SELECT` statement which converts the table into a hypertable. You can do this by calling the `Exec()` function on the `dbpool` object, using the arguments of the current context, and the `queryCreateTable` and `queryCreateHypertable` statement strings: ```go package main import ( "context" "fmt" "os" "github.com/jackc/pgx/v5/pgxpool" ) func main() { ctx := context.Background() connStr := "yourConnectionStringHere" dbpool, err := pgxpool.New(ctx, connStr) if err != nil { fmt.Fprintf(os.Stderr, "Unable to connect to database: %v\n", err) os.Exit(1) } defer dbpool.Close() /********************************************/ /* Create Hypertable */ /********************************************/ // Create hypertable of time-series data called sensor_data queryCreateTable := `CREATE TABLE sensor_data ( time TIMESTAMPTZ NOT NULL, sensor_id INTEGER, temperature DOUBLE PRECISION, cpu DOUBLE PRECISION, FOREIGN KEY (sensor_id) REFERENCES sensors (id)); ` queryCreateHypertable := `SELECT create_hypertable('sensor_data', by_range('time'));` //execute statement _, err = dbpool.Exec(ctx, queryCreateTable+queryCreateHypertable) if err != nil { fmt.Fprintf(os.Stderr, "Unable to create the `sensor_data` hypertable: %v\n", err) os.Exit(1) } fmt.Println("Successfully created hypertable `sensor_data`") } ``` ## Insert rows of data You can insert rows into your database in a couple of different ways. Each of these example inserts the data from the two arrays, `sensorTypes` and `sensorLocations`, into the relational table named `sensors`. The first example inserts a single row of data at a time. The second example inserts multiple rows of data. The third example uses batch inserts to speed up the process. 1. **Insert data using prepared statements** Open a connection pool to the database, then use prepared statements to formulate an `INSERT` SQL statement, and execute it: ```go package main import ( "context" "fmt" "os" "github.com/jackc/pgx/v5/pgxpool" ) func main() { ctx := context.Background() connStr := "yourConnectionStringHere" dbpool, err := pgxpool.New(ctx, connStr) if err != nil { fmt.Fprintf(os.Stderr, "Unable to connect to database: %v\n", err) os.Exit(1) } defer dbpool.Close() /********************************************/ /* INSERT into relational table */ /********************************************/ //Insert data into relational table // Slices of sample data to insert // observation i has type sensorTypes[i] and location sensorLocations[i] sensorTypes := []string{"a", "a", "b", "b"} sensorLocations := []string{"floor", "ceiling", "floor", "ceiling"} for i := range sensorTypes { //INSERT statement in SQL queryInsertMetadata := `INSERT INTO sensors (type, location) VALUES ($1, $2);` //Execute INSERT command _, err := dbpool.Exec(ctx, queryInsertMetadata, sensorTypes[i], sensorLocations[i]) if err != nil { fmt.Fprintf(os.Stderr, "Unable to insert data into database: %v\n", err) os.Exit(1) } fmt.Printf("Inserted sensor (%s, %s) into database \n", sensorTypes[i], sensorLocations[i]) } fmt.Println("Successfully inserted all sensors into database") } ``` Instead of inserting a single row of data at a time, you can use this procedure to insert multiple rows of data, instead: 1. **Generate sample time-series data** This example uses PostgreSQL to generate some sample time-series data to insert into the `sensor_data` hypertable. Define the SQL statement to generate the data, called `queryDataGeneration`. Then use the `.Query()` function to execute the statement and return the sample data. The data returned by the query is stored in `results`, a slice of structs, which is then used as a source to insert data into the hypertable: ```go package main import ( "context" "fmt" "os" "time" "github.com/jackc/pgx/v5/pgxpool" ) func main() { ctx := context.Background() connStr := "yourConnectionStringHere" dbpool, err := pgxpool.New(ctx, connStr) if err != nil { fmt.Fprintf(os.Stderr, "Unable to connect to database: %v\n", err) os.Exit(1) } defer dbpool.Close() // Generate data to insert //SQL query to generate sample data queryDataGeneration := ` SELECT generate_series(now() - interval '24 hour', now(), interval '5 minute') AS time, floor(random() * (3) + 1)::int as sensor_id, random()*100 AS temperature, random() AS cpu ` //Execute query to generate samples for sensor_data hypertable rows, err := dbpool.Query(ctx, queryDataGeneration) if err != nil { fmt.Fprintf(os.Stderr, "Unable to generate sensor data: %v\n", err) os.Exit(1) } defer rows.Close() fmt.Println("Successfully generated sensor data") //Store data generated in slice results type result struct { Time time.Time SensorId int Temperature float64 CPU float64 } var results []result for rows.Next() { var r result err = rows.Scan(&r.Time, &r.SensorId, &r.Temperature, &r.CPU) if err != nil { fmt.Fprintf(os.Stderr, "Unable to scan %v\n", err) os.Exit(1) } results = append(results, r) } // Any errors encountered by rows.Next or rows.Scan are returned here if rows.Err() != nil { fmt.Fprintf(os.Stderr, "rows Error: %v\n", rows.Err()) os.Exit(1) } // Check contents of results slice fmt.Println("Contents of RESULTS slice") for i := range results { var r result r = results[i] fmt.Printf("Time: %s | ID: %d | Temperature: %f | CPU: %f |\n", &r.Time, r.SensorId, r.Temperature, r.CPU) } } ``` 2. **Formulate an SQL insert statement for the `sensor_data` hypertable:** ```go //SQL query to generate sample data queryInsertTimeseriesData := ` INSERT INTO sensor_data (time, sensor_id, temperature, cpu) VALUES ($1, $2, $3, $4); ` ``` 3. **Execute the SQL statement for each sample in the results slice:** ```go //Insert contents of results slice into TimescaleDB for i := range results { var r result r = results[i] _, err := dbpool.Exec(ctx, queryInsertTimeseriesData, r.Time, r.SensorId, r.Temperature, r.CPU) if err != nil { fmt.Fprintf(os.Stderr, "Unable to insert sample into TimescaleDB %v\n", err) os.Exit(1) } defer rows.Close() } fmt.Println("Successfully inserted samples into sensor_data hypertable") ``` 4. **(Optional) Full sample code for data generation and insert** This example `main.go` generates sample data and inserts it into the `sensor_data` hypertable: ```go package main import ( "context" "fmt" "os" "time" "github.com/jackc/pgx/v5/pgxpool" ) func main() { /********************************************/ /* Connect using Connection Pool */ /********************************************/ ctx := context.Background() connStr := "yourConnectionStringHere" dbpool, err := pgxpool.New(ctx, connStr) if err != nil { fmt.Fprintf(os.Stderr, "Unable to connect to database: %v\n", err) os.Exit(1) } defer dbpool.Close() /********************************************/ /* Insert data into hypertable */ /********************************************/ // Generate data to insert //SQL query to generate sample data queryDataGeneration := ` SELECT generate_series(now() - interval '24 hour', now(), interval '5 minute') AS time, floor(random() * (3) + 1)::int as sensor_id, random()*100 AS temperature, random() AS cpu ` //Execute query to generate samples for sensor_data hypertable rows, err := dbpool.Query(ctx, queryDataGeneration) if err != nil { fmt.Fprintf(os.Stderr, "Unable to generate sensor data: %v\n", err) os.Exit(1) } defer rows.Close() fmt.Println("Successfully generated sensor data") //Store data generated in slice results type result struct { Time time.Time SensorId int Temperature float64 CPU float64 } var results []result for rows.Next() { var r result err = rows.Scan(&r.Time, &r.SensorId, &r.Temperature, &r.CPU) if err != nil { fmt.Fprintf(os.Stderr, "Unable to scan %v\n", err) os.Exit(1) } results = append(results, r) } // Any errors encountered by rows.Next or rows.Scan are returned here if rows.Err() != nil { fmt.Fprintf(os.Stderr, "rows Error: %v\n", rows.Err()) os.Exit(1) } // Check contents of results slice fmt.Println("Contents of RESULTS slice") for i := range results { var r result r = results[i] fmt.Printf("Time: %s | ID: %d | Temperature: %f | CPU: %f |\n", &r.Time, r.SensorId, r.Temperature, r.CPU) } //Insert contents of results slice into TimescaleDB //SQL query to generate sample data queryInsertTimeseriesData := ` INSERT INTO sensor_data (time, sensor_id, temperature, cpu) VALUES ($1, $2, $3, $4); ` //Insert contents of results slice into TimescaleDB for i := range results { var r result r = results[i] _, err := dbpool.Exec(ctx, queryInsertTimeseriesData, r.Time, r.SensorId, r.Temperature, r.CPU) if err != nil { fmt.Fprintf(os.Stderr, "Unable to insert sample into TimescaleDB %v\n", err) os.Exit(1) } defer rows.Close() } fmt.Println("Successfully inserted samples into sensor_data hypertable") } ``` Inserting multiple rows of data using this method executes as many `insert` statements as there are samples to be inserted. This can make ingestion of data slow. To speed up ingestion, you can batch insert data instead. Here's a sample pattern for how to do so, using the sample data you generated in the previous procedure. It uses the pgx `Batch` object: 1. **This example batch inserts data into the database:** ```go package main import ( "context" "fmt" "os" "time" "github.com/jackc/pgx/v5" "github.com/jackc/pgx/v5/pgxpool" ) func main() { /********************************************/ /* Connect using Connection Pool */ /********************************************/ ctx := context.Background() connStr := "yourConnectionStringHere" dbpool, err := pgxpool.New(ctx, connStr) if err != nil { fmt.Fprintf(os.Stderr, "Unable to connect to database: %v\n", err) os.Exit(1) } defer dbpool.Close() // Generate data to insert //SQL query to generate sample data queryDataGeneration := ` SELECT generate_series(now() - interval '24 hour', now(), interval '5 minute') AS time, floor(random() * (3) + 1)::int as sensor_id, random()*100 AS temperature, random() AS cpu ` //Execute query to generate samples for sensor_data hypertable rows, err := dbpool.Query(ctx, queryDataGeneration) if err != nil { fmt.Fprintf(os.Stderr, "Unable to generate sensor data: %v\n", err) os.Exit(1) } defer rows.Close() fmt.Println("Successfully generated sensor data") //Store data generated in slice results type result struct { Time time.Time SensorId int Temperature float64 CPU float64 } var results []result for rows.Next() { var r result err = rows.Scan(&r.Time, &r.SensorId, &r.Temperature, &r.CPU) if err != nil { fmt.Fprintf(os.Stderr, "Unable to scan %v\n", err) os.Exit(1) } results = append(results, r) } // Any errors encountered by rows.Next or rows.Scan are returned here if rows.Err() != nil { fmt.Fprintf(os.Stderr, "rows Error: %v\n", rows.Err()) os.Exit(1) } // Check contents of results slice /*fmt.Println("Contents of RESULTS slice") for i := range results { var r result r = results[i] fmt.Printf("Time: %s | ID: %d | Temperature: %f | CPU: %f |\n", &r.Time, r.SensorId, r.Temperature, r.CPU) }*/ //Insert contents of results slice into TimescaleDB //SQL query to generate sample data queryInsertTimeseriesData := ` INSERT INTO sensor_data (time, sensor_id, temperature, cpu) VALUES ($1, $2, $3, $4); ` /********************************************/ /* Batch Insert into TimescaleDB */ /********************************************/ //create batch batch := &pgx.Batch{} //load insert statements into batch queue for i := range results { var r result r = results[i] batch.Queue(queryInsertTimeseriesData, r.Time, r.SensorId, r.Temperature, r.CPU) } batch.Queue("select count(*) from sensor_data") //send batch to connection pool br := dbpool.SendBatch(ctx, batch) //execute statements in batch queue _, err = br.Exec() if err != nil { fmt.Fprintf(os.Stderr, "Unable to execute statement in batch queue %v\n", err) os.Exit(1) } fmt.Println("Successfully batch inserted data") //Compare length of results slice to size of table fmt.Printf("size of results: %d\n", len(results)) //check size of table for number of rows inserted // result of last SELECT statement var rowsInserted int err = br.QueryRow().Scan(&rowsInserted) fmt.Printf("size of table: %d\n", rowsInserted) err = br.Close() if err != nil { fmt.Fprintf(os.Stderr, "Unable to closer batch %v\n", err) os.Exit(1) } } ``` ## Execute a query This section covers how to execute queries against your database. 1. **Define the SQL query** This example uses a SQL query that combines time-series and relational data. It returns the average CPU values for every 5 minute interval, for sensors located on location `ceiling` and of type `a`: ```go // Formulate query in SQL // Note the use of prepared statement placeholders $1 and $2 queryTimebucketFiveMin := ` SELECT time_bucket('5 minutes', time) AS five_min, avg(cpu) FROM sensor_data JOIN sensors ON sensors.id = sensor_data.sensor_id WHERE sensors.location = $1 AND sensors.type = $2 GROUP BY five_min ORDER BY five_min DESC; ` ``` 2. **Execute the query** Use the `.Query()` function to execute the query string. Make sure you specify the relevant placeholders: ```go //Execute query on TimescaleDB rows, err := dbpool.Query(ctx, queryTimebucketFiveMin, "ceiling", "a") if err != nil { fmt.Fprintf(os.Stderr, "Unable to execute query %v\n", err) os.Exit(1) } defer rows.Close() fmt.Println("Successfully executed query") ``` 3. **Access the returned rows** Create a struct with fields representing the columns that you expect to be returned, then use the `rows.Next()` function to iterate through the rows returned and fill `results` with the array of structs. This uses the `rows.Scan()` function, passing in pointers to the fields that you want to scan for results. This example prints out the results returned from the query, but you might want to use those results for some other purpose. Once you've scanned through all the rows returned you can then use the results array however you like. ```go //Do something with the results of query // Struct for results type result2 struct { Bucket time.Time Avg float64 } // Print rows returned and fill up results slice for later use var results []result2 for rows.Next() { var r result2 err = rows.Scan(&r.Bucket, &r.Avg) if err != nil { fmt.Fprintf(os.Stderr, "Unable to scan %v\n", err) os.Exit(1) } results = append(results, r) fmt.Printf("Time bucket: %s | Avg: %f\n", &r.Bucket, r.Avg) } // Any errors encountered by rows.Next or rows.Scan are returned here if rows.Err() != nil { fmt.Fprintf(os.Stderr, "rows Error: %v\n", rows.Err()) os.Exit(1) } // use results here… ``` 4. **(Optional) Full sample code for querying** This example program runs a query, and accesses the results of that query: ```go package main import ( "context" "fmt" "os" "time" "github.com/jackc/pgx/v5/pgxpool" ) func main() { ctx := context.Background() connStr := "yourConnectionStringHere" dbpool, err := pgxpool.New(ctx, connStr) if err != nil { fmt.Fprintf(os.Stderr, "Unable to connect to database: %v\n", err) os.Exit(1) } defer dbpool.Close() /********************************************/ /* Execute a query */ /********************************************/ // Formulate query in SQL // Note the use of prepared statement placeholders $1 and $2 queryTimebucketFiveMin := ` SELECT time_bucket('5 minutes', time) AS five_min, avg(cpu) FROM sensor_data JOIN sensors ON sensors.id = sensor_data.sensor_id WHERE sensors.location = $1 AND sensors.type = $2 GROUP BY five_min ORDER BY five_min DESC; ` //Execute query on TimescaleDB rows, err := dbpool.Query(ctx, queryTimebucketFiveMin, "ceiling", "a") if err != nil { fmt.Fprintf(os.Stderr, "Unable to execute query %v\n", err) os.Exit(1) } defer rows.Close() fmt.Println("Successfully executed query") //Do something with the results of query // Struct for results type result2 struct { Bucket time.Time Avg float64 } // Print rows returned and fill up results slice for later use var results []result2 for rows.Next() { var r result2 err = rows.Scan(&r.Bucket, &r.Avg) if err != nil { fmt.Fprintf(os.Stderr, "Unable to scan %v\n", err) os.Exit(1) } results = append(results, r) fmt.Printf("Time bucket: %s | Avg: %f\n", &r.Bucket, r.Avg) } // Any errors encountered by rows.Next or rows.Scan are returned here if rows.Err() != nil { fmt.Fprintf(os.Stderr, "rows Error: %v\n", rows.Err()) os.Exit(1) } } ``` ## Next steps Now that you're able to connect, read, and write to a TimescaleDB instance from your Go application, check out these resources: * Refer to the [pgx documentation](https://pkg.go.dev/github.com/jackc/pgx) for more information about pgx. * Want fast inserts on CSV data? Check out [TimescaleDB parallel copy](https://github.com/timescale/timescaledb-parallel-copy), a tool for fast inserts, written in Go. * Java ## Prerequisites for this tutorial To follow these steps, you'll need: * A [self-hosted TimescaleDB](/docs/get-started/choose-your-path/install-timescaledb) instance. - Your [connection details](/docs/integrate/find-connection-details). * The [Java Development Kit (JDK)](https://openjdk.java.net) installed. * The [PostgreSQL JDBC driver](https://jdbc.postgresql.org) installed. All code in this quick start is for Java 16 and later. If you are working with older JDK versions, use legacy coding techniques. ## Connect to TimescaleDB In this section, you create a connection to your TimescaleDB database using an application in a single file. You can use any of your favorite build tools, including `gradle` or `maven`. 1. **Create the Main.java file** Create a directory containing a text file called `Main.java`, with this content: ```java package com.timescale.java; public class Main { public static void main(String... args) { System.out.println("Hello, World!"); } } ``` 2. **Run the application** From the command line in the current directory, run the application: ```bash java Main.java ``` If the command is successful, `Hello, World!` line output is printed to your console. 3. **Import the PostgreSQL JDBC driver** If you are using a dependency manager, include the [PostgreSQL JDBC Driver](https://mvnrepository.com/artifact/org.postgresql/postgresql) as a dependency. 4. **Download the JDBC Driver JAR** Download the [JAR artifact of the JDBC Driver](https://jdbc.postgresql.org/download/) and save it with the `Main.java` file. 5. **Import the JDBC Driver into the application** Import the `JDBC Driver` into the Java application and display a list of available drivers for the check: ```java package com.timescale.java; import java.sql.DriverManager; public class Main { public static void main(String... args) { DriverManager.drivers().forEach(System.out::println); } } ``` 6. **Run the examples** Run all the examples: ```bash java -cp *.jar Main.java ``` If the command is successful, a string similar to `org.postgresql.Driver@7f77e91b` is printed to your console. This means that you are ready to connect to TimescaleDB from Java. 7. **Locate your TimescaleDB credentials** Locate your TimescaleDB credentials and use them to compose a connection string for JDBC. You'll need: * password * username * host URL * port * database name 8. **Compose the connection string** Compose your connection string variable, using this format: ```java var connUrl = "jdbc:postgresql://:/?user=&password="; ``` For more information about creating connection strings, see the [JDBC documentation](https://jdbc.postgresql.org/documentation/datasource/). Warning This method of composing a connection string is for test or development purposes only. For production, use environment variables for sensitive details like your password, hostname, and port number. ```java package com.timescale.java; import java.sql.DriverManager; import java.sql.SQLException; public class Main { public static void main(String... args) throws SQLException { var connUrl = "jdbc:postgresql://:/?user=&password="; var conn = DriverManager.getConnection(connUrl); System.out.println(conn.getClientInfo()); } } ``` 9. **Run the code** Run the code: ```bash java -cp *.jar Main.java ``` If the command is successful, a string similar to `{ApplicationName=PostgreSQL JDBC Driver}` is printed to your console. ## Create a relational table In this section, you create a table called `sensors` which holds the ID, type, and location of your fictional sensors. Additionally, you create a hypertable called `sensor_data` which holds the measurements of those sensors. The measurements contain the time, sensor\_id, temperature reading, and CPU percentage of the sensors. 1. **Compose the `CREATE TABLE` statement** Compose a string which contains the SQL statement to create a relational table. This example creates a table called `sensors`, with columns `id`, `type` and `location`: ```sql CREATE TABLE sensors ( id SERIAL PRIMARY KEY, type TEXT NOT NULL, location TEXT NOT NULL ); ``` 2. **Execute the query and verify** Create a statement, execute the query you created in the previous step, and check that the table was created successfully: ```java package com.timescale.java; import java.sql.DriverManager; import java.sql.SQLException; public class Main { public static void main(String... args) throws SQLException { var connUrl = "jdbc:postgresql://:/?user=&password="; var conn = DriverManager.getConnection(connUrl); var createSensorTableQuery = """ CREATE TABLE sensors ( id SERIAL PRIMARY KEY, type TEXT NOT NULL, location TEXT NOT NULL ) """; try (var stmt = conn.createStatement()) { stmt.execute(createSensorTableQuery); } var showAllTablesQuery = "SELECT tablename FROM pg_catalog.pg_tables WHERE schemaname = 'public'"; try (var stmt = conn.createStatement(); var rs = stmt.executeQuery(showAllTablesQuery)) { System.out.println("Tables in the current database: "); while (rs.next()) { System.out.println(rs.getString("tablename")); } } } } ``` ## Create a hypertable When you have created the relational table, you can create a hypertable. Creating tables and indexes, altering tables, inserting data, selecting data, and most other tasks are executed on the hypertable. 1. **Write the `CREATE TABLE` statement for the hypertable** Create a `CREATE TABLE` SQL statement for your hypertable. Notice how the hypertable has the compulsory time column: ```sql CREATE TABLE sensor_data ( time TIMESTAMPTZ NOT NULL, sensor_id INTEGER REFERENCES sensors (id), value DOUBLE PRECISION ); ``` 2. **Convert the table to a hypertable** Create a statement, execute the query you created in the previous step: ```sql SELECT create_hypertable('sensor_data', by_range('time')); ``` Note The `by_range` and `by_hash` dimension builder is an addition to TimescaleDB 2.13. 3. **Execute and commit the schema changes** Execute the two statements you created, and commit your changes to the database: ```java package com.timescale.java; import java.sql.Connection; import java.sql.DriverManager; import java.sql.SQLException; import java.util.List; public class Main { public static void main(String... args) { final var connUrl = "jdbc:postgresql://:/?user=&password="; try (var conn = DriverManager.getConnection(connUrl)) { createSchema(conn); insertData(conn); } catch (SQLException ex) { System.err.println(ex.getMessage()); } } private static void createSchema(final Connection conn) throws SQLException { try (var stmt = conn.createStatement()) { stmt.execute(""" CREATE TABLE sensors ( id SERIAL PRIMARY KEY, type TEXT NOT NULL, location TEXT NOT NULL ) """); } try (var stmt = conn.createStatement()) { stmt.execute(""" CREATE TABLE sensor_data ( time TIMESTAMPTZ NOT NULL, sensor_id INTEGER REFERENCES sensors (id), value DOUBLE PRECISION ) """); } try (var stmt = conn.createStatement()) { stmt.execute("SELECT create_hypertable('sensor_data', by_range('time'))"); } } } ``` ## Insert data You can insert data into your hypertables in several different ways. In this section, you can insert single rows, or insert by batches of rows. 1. **Insert rows with prepared statements** Open a connection to the database, use prepared statements to formulate the `INSERT` SQL statement, then execute the statement: ```java final List sensors = List.of( new Sensor("temperature", "bedroom"), new Sensor("temperature", "living room"), new Sensor("temperature", "outside"), new Sensor("humidity", "kitchen"), new Sensor("humidity", "outside")); for (final var sensor : sensors) { try (var stmt = conn.prepareStatement("INSERT INTO sensors (type, location) VALUES (?, ?)")) { stmt.setString(1, sensor.type()); stmt.setString(2, sensor.location()); stmt.executeUpdate(); } } ``` If you want to insert a batch of rows by using a batching mechanism. In this example, you generate some sample time-series data to insert into the `sensor_data` hypertable: 1. **Insert batches of rows** Insert batches of rows: ```java final var sensorDataCount = 100; final var insertBatchSize = 10; try (var stmt = conn.prepareStatement(""" INSERT INTO sensor_data (time, sensor_id, value) VALUES ( generate_series(now() - INTERVAL '24 hours', now(), INTERVAL '5 minutes'), floor(random() * 4 + 1)::INTEGER, random() ) """)) { for (int i = 0; i < sensorDataCount; i++) { stmt.addBatch(); if ((i > 0 && i % insertBatchSize == 0) || i == sensorDataCount - 1) { stmt.executeBatch(); } } } ``` ## Execute a query This section covers how to execute queries against your database. 1. **Define the SQL query** Define the SQL query you'd like to run on the database. This example combines time-series and relational data. It returns the average values for every 15 minute interval for sensors with specific type and location. ```sql SELECT time_bucket('15 minutes', time) AS bucket, avg(value) FROM sensor_data JOIN sensors ON sensors.id = sensor_data.sensor_id WHERE sensors.type = ? AND sensors.location = ? GROUP BY bucket ORDER BY bucket DESC; ``` 2. **Execute the query and read results** Execute the query with the prepared statement and read out the result set for all `a`-type sensors located on the `floor`: ```java try (var stmt = conn.prepareStatement(""" SELECT time_bucket('15 minutes', time) AS bucket, avg(value) FROM sensor_data JOIN sensors ON sensors.id = sensor_data.sensor_id WHERE sensors.type = ? AND sensors.location = ? GROUP BY bucket ORDER BY bucket DESC """)) { stmt.setString(1, "temperature"); stmt.setString(2, "living room"); try (var rs = stmt.executeQuery()) { while (rs.next()) { System.out.printf("%s: %f%n", rs.getTimestamp(1), rs.getDouble(2)); } } } ``` If the command is successful, you'll see output like this: ```bash 2021-05-12 23:30:00.0: 0,508649 2021-05-12 23:15:00.0: 0,477852 2021-05-12 23:00:00.0: 0,462298 2021-05-12 22:45:00.0: 0,457006 2021-05-12 22:30:00.0: 0,568744 ... ``` ## Complete code samples This section contains complete code samples. ### Complete code sample ```java package com.timescale.java; import java.sql.Connection; import java.sql.DriverManager; import java.sql.SQLException; import java.util.List; public class Main { public static void main(String... args) { final var connUrl = "jdbc:postgresql://:/?user=&password="; try (var conn = DriverManager.getConnection(connUrl)) { createSchema(conn); insertData(conn); } catch (SQLException ex) { System.err.println(ex.getMessage()); } } private static void createSchema(final Connection conn) throws SQLException { try (var stmt = conn.createStatement()) { stmt.execute(""" CREATE TABLE sensors ( id SERIAL PRIMARY KEY, type TEXT NOT NULL, location TEXT NOT NULL ) """); } try (var stmt = conn.createStatement()) { stmt.execute(""" CREATE TABLE sensor_data ( time TIMESTAMPTZ NOT NULL, sensor_id INTEGER REFERENCES sensors (id), value DOUBLE PRECISION ) """); } try (var stmt = conn.createStatement()) { stmt.execute("SELECT create_hypertable('sensor_data', by_range('time'))"); } } private static void insertData(final Connection conn) throws SQLException { final List sensors = List.of( new Sensor("temperature", "bedroom"), new Sensor("temperature", "living room"), new Sensor("temperature", "outside"), new Sensor("humidity", "kitchen"), new Sensor("humidity", "outside")); for (final var sensor : sensors) { try (var stmt = conn.prepareStatement("INSERT INTO sensors (type, location) VALUES (?, ?)")) { stmt.setString(1, sensor.type()); stmt.setString(2, sensor.location()); stmt.executeUpdate(); } } final var sensorDataCount = 100; final var insertBatchSize = 10; try (var stmt = conn.prepareStatement(""" INSERT INTO sensor_data (time, sensor_id, value) VALUES ( generate_series(now() - INTERVAL '24 hours', now(), INTERVAL '5 minutes'), floor(random() * 4 + 1)::INTEGER, random() ) """)) { for (int i = 0; i < sensorDataCount; i++) { stmt.addBatch(); if ((i > 0 && i % insertBatchSize == 0) || i == sensorDataCount - 1) { stmt.executeBatch(); } } } } private record Sensor(String type, String location) { } } ``` ### Execute more complex queries ```java package com.timescale.java; import java.sql.Connection; import java.sql.DriverManager; import java.sql.SQLException; import java.util.List; public class Main { public static void main(String... args) { final var connUrl = "jdbc:postgresql://:/?user=&password="; try (var conn = DriverManager.getConnection(connUrl)) { createSchema(conn); insertData(conn); executeQueries(conn); } catch (SQLException ex) { System.err.println(ex.getMessage()); } } private static void createSchema(final Connection conn) throws SQLException { try (var stmt = conn.createStatement()) { stmt.execute(""" CREATE TABLE sensors ( id SERIAL PRIMARY KEY, type TEXT NOT NULL, location TEXT NOT NULL ) """); } try (var stmt = conn.createStatement()) { stmt.execute(""" CREATE TABLE sensor_data ( time TIMESTAMPTZ NOT NULL, sensor_id INTEGER REFERENCES sensors (id), value DOUBLE PRECISION ) """); } try (var stmt = conn.createStatement()) { stmt.execute("SELECT create_hypertable('sensor_data', by_range('time'))"); } } private static void insertData(final Connection conn) throws SQLException { final List sensors = List.of( new Sensor("temperature", "bedroom"), new Sensor("temperature", "living room"), new Sensor("temperature", "outside"), new Sensor("humidity", "kitchen"), new Sensor("humidity", "outside")); for (final var sensor : sensors) { try (var stmt = conn.prepareStatement("INSERT INTO sensors (type, location) VALUES (?, ?)")) { stmt.setString(1, sensor.type()); stmt.setString(2, sensor.location()); stmt.executeUpdate(); } } final var sensorDataCount = 100; final var insertBatchSize = 10; try (var stmt = conn.prepareStatement(""" INSERT INTO sensor_data (time, sensor_id, value) VALUES ( generate_series(now() - INTERVAL '24 hours', now(), INTERVAL '5 minutes'), floor(random() * 4 + 1)::INTEGER, random() ) """)) { for (int i = 0; i < sensorDataCount; i++) { stmt.addBatch(); if ((i > 0 && i % insertBatchSize == 0) || i == sensorDataCount - 1) { stmt.executeBatch(); } } } } private static void executeQueries(final Connection conn) throws SQLException { try (var stmt = conn.prepareStatement(""" SELECT time_bucket('15 minutes', time) AS bucket, avg(value) FROM sensor_data JOIN sensors ON sensors.id = sensor_data.sensor_id WHERE sensors.type = ? AND sensors.location = ? GROUP BY bucket ORDER BY bucket DESC """)) { stmt.setString(1, "temperature"); stmt.setString(2, "living room"); try (var rs = stmt.executeQuery()) { while (rs.next()) { System.out.printf("%s: %f%n", rs.getTimestamp(1), rs.getDouble(2)); } } } } private record Sensor(String type, String location) { } } ``` ## Next steps Now that you're able to connect, read, and write to a TimescaleDB instance from your Java application, and generate the scaffolding necessary to build a new application from an existing TimescaleDB instance, be sure to check out these advanced TimescaleDB tutorials: * [Continuous aggregates](/docs/build/continuous-aggregates/create-a-continuous-aggregate) * [Migrate your data](/docs/migrate) You are not limited to these languages. Tiger Cloud is based on PostgreSQL, you can interface with TimescaleDB and Tiger Cloud using any [PostgreSQL client driver](https://wiki.postgresql.org/wiki/List_of_drivers). # Configuration & deployment > Use your IaC and deployment tools with Tiger Data The guides below show how to connect infrastructure-as-code and deployment tools to Tiger Data products to provision, deploy, and manage your database in a repeatable, version-controlled way alongside the rest of your stack. ## Integrations [Section titled “Integrations”](#integrations) [Kubernetes](/docs/integrate/configuration-deployment/kubernetes) [Automate deployment, scaling, and management of your containerized workloads](/docs/integrate/configuration-deployment/kubernetes) [Terraform](/docs/integrate/configuration-deployment/terraform) [Provision and manage your infrastructure as code with predictable deployments](/docs/integrate/configuration-deployment/terraform) # Integrate TimescaleDB with CloudNativePG on Kubernetes > Deploy and operate TimescaleDB on Kubernetes with CloudNativePG for production-ready time-series workloads ![](https://avatars.githubusercontent.com/shusaan?s=128) Husn E Rabbi [GitHub](https://github.com/shusaan) · [LinkedIn](http://www.linkedin.com/in/husn-e-rabbi) Managing PostgreSQL on Kubernetes has come a long way. While deploying a database is relatively straightforward, operating it reliably in production is where the real challenge begins. Backups, failover, upgrades, monitoring, and disaster recovery all require careful planning. Traditionally, this meant combining Helm charts with custom scripts or manually managing PostgreSQL images as requirements evolved. [CloudNativePG](https://cloudnative-pg.io/) takes a different approach. Rather than treating PostgreSQL as just another container, CloudNativePG manages it as a Kubernetes-native resource. You declare the desired state of your PostgreSQL cluster, and the operator continuously reconciles the cluster to match it. Tasks such as provisioning, failover, backups, rolling updates, and lifecycle management become automated. With CloudNativePG's PostgreSQL extension containers, you can also deploy TimescaleDB without maintaining custom PostgreSQL images. PostgreSQL and its extensions are managed independently, making upgrades significantly easier. This integration guide walks you through deploying TimescaleDB on CloudNativePG, enabling the extension, and creating your first hypertable. ## Prerequisites for this integration guide To follow these steps, you'll need: * Kubernetes cluster (v1.31 or later recommended) * `kubectl` configured * A default StorageClass * Cluster-admin permissions * A running Kubernetes node with persistent storage ## Architecture Overview [Section titled “Architecture Overview”](#architecture-overview) The deployment architecture is intentionally simple. ```text +------------------------------------------------+ | Kubernetes Cluster | +------------------------------------------------+ | ▼ CloudNativePG Operator | ▼ PostgreSQL Cluster (CNPG) | ▼ TimescaleDB OSS Extension Container | ▼ Time-Series Applications & Services ``` CloudNativePG manages the PostgreSQL cluster lifecycle, while the TimescaleDB extension container provides the required extension binaries. This separation allows PostgreSQL and extensions to evolve independently. ## Why Extension Containers? [Section titled “Why Extension Containers?”](#why-extension-containers) Historically, installing PostgreSQL extensions in Kubernetes often required building and maintaining custom PostgreSQL images. That approach introduces several operational challenges: * Custom image maintenance * Rebuilding images for PostgreSQL upgrades * Managing extension compatibility * Larger CI/CD pipelines * More complex image security scanning CloudNativePG extension containers eliminate these problems. Instead of embedding every extension into the PostgreSQL image, extensions are packaged separately. During cluster initialization, CloudNativePG mounts the extension container into PostgreSQL, making the extension available without modifying the base database image. This keeps PostgreSQL images small, simplifies upgrades, and makes extension lifecycle management significantly cleaner. ## Verify Kubernetes Feature Gates [Section titled “Verify Kubernetes Feature Gates”](#verify-kubernetes-feature-gates) CloudNativePG extension containers rely on the Kubernetes `ImageVolume` feature. Ensure your Kubernetes cluster supports this feature before continuing. If you're using a managed Kubernetes service, consult your provider's documentation to verify support. To confirm the API server has the feature gate enabled, run: ```bash kubectl get pods -n kube-system \ -l component=kube-apiserver \ -o yaml | grep -A 20 "containers:" | grep -E "args:|feature-gates" ``` You should see output similar to: ```text --feature-gates=ImageVolume=true ``` If the feature gate is not enabled, extension containers will not function correctly. ## Creating a Local Kubernetes Cluster with Kind [Section titled “Creating a Local Kubernetes Cluster with Kind”](#creating-a-local-kubernetes-cluster-with-kind) If you don't already have a Kubernetes cluster available, Kind (Kubernetes in Docker) is one of the easiest ways to create a local development environment. Create a configuration file named `kind-config.yaml`. ```yaml kind: Cluster apiVersion: kind.x-k8s.io/v1alpha4 name: ops featureGates: ImageVolume: true ``` Create the cluster: ```bash kind create cluster --config kind-config.yaml ``` Check the nodes: ```bash kubectl get nodes ``` You should see something similar to: ```text NAME STATUS ROLES AGE VERSION ops-control-plane Ready control-plane 1m v1.34.x ``` ## Installing CloudNativePG [Section titled “Installing CloudNativePG”](#installing-cloudnativepg) At the time of writing, CloudNativePG 1.30 is the latest stable release in the 1.30 series. 1. **Install the operator** Install the operator: ```bash kubectl apply --server-side -f \ https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/release-1.30/releases/cnpg-1.30.0.yaml ``` 2. **Verify that the operator is running** Verify that the operator is running: ```bash kubectl rollout status deployment -n cnpg-system cnpg-controller-manager ``` Once the deployment is successfully rolled out, your Kubernetes cluster is ready to manage PostgreSQL resources. ## Deploying TimescaleDB [Section titled “Deploying TimescaleDB”](#deploying-timescaledb) ### Step 1 – Create the PostgreSQL Cluster [Section titled “Step 1 – Create the PostgreSQL Cluster”](#step-1--create-the-postgresql-cluster) Create a file named `cluster-timescaledb.yaml`. ```yaml apiVersion: postgresql.cnpg.io/v1 kind: Cluster metadata: name: cluster-timescaledb spec: imageName: ghcr.io/cloudnative-pg/postgresql:18-system-trixie instances: 1 storage: size: 1Gi postgresql: shared_preload_libraries: - "timescaledb" parameters: timescaledb.telemetry_level: 'off' max_locks_per_transaction: '128' extensions: - name: timescaledb-oss image: reference: ghcr.io/cloudnative-pg/timescaledb-oss:2.27.2-18-trixie ``` A few things are worth noting: * `2.27.2` is the TimescaleDB version. * `18` represents the PostgreSQL major version. * `trixie` indicates the Debian base image. Deploy the cluster: ```bash kubectl apply -f cluster-timescaledb.yaml ``` CloudNativePG provisions the PostgreSQL instance and automatically mounts the TimescaleDB extension container. ### Step 2 – Create a Database [Section titled “Step 2 – Create a Database”](#step-2--create-a-database) Create `database-timescaledb.yaml`. ```yaml apiVersion: postgresql.cnpg.io/v1 kind: Database metadata: name: cluster-timescaledb-app spec: name: app owner: app cluster: name: cluster-timescaledb extensions: - name: timescaledb version: "2.27.2" ``` Apply it: ```bash kubectl apply -f database-timescaledb.yaml ``` CloudNativePG creates the database and enables the TimescaleDB extension automatically. ### Step 3 – Verify Installation [Section titled “Step 3 – Verify Installation”](#step-3--verify-installation) After deploying the cluster and database resources, connect to PostgreSQL and verify that the TimescaleDB extension has been installed successfully. 1. **Open a shell inside the PostgreSQL pod** Open a shell inside the PostgreSQL pod: ```bash kubectl exec -it cluster-timescaledb-1 -- sh ``` 2. **Launch the PostgreSQL client** Launch the PostgreSQL client: ```bash psql ``` 3. **Connect to the application database** Connect to the application database: ```sql \c app ``` 4. **List installed extensions** List installed extensions: ```sql \dx ``` You should see output similar to: ```text List of installed extensions Name | Version | Schema | Description -------------+---------+------------+------------------------------ plpgsql | 1.0 | pg_catalog | PL/pgSQL procedural language timescaledb | 2.27.2 | public | TimescaleDB OSS ``` The TimescaleDB extension is now active and ready for use. ## Creating Your First Hypertable [Section titled “Creating Your First Hypertable”](#creating-your-first-hypertable) Create a hypertable for sensor readings. ```sql CREATE TABLE sensor_data ( time TIMESTAMPTZ NOT NULL, sensor_id INTEGER, temperature DOUBLE PRECISION, humidity DOUBLE PRECISION ) WITH (tsdb.hypertable, tsdb.partition_column = 'time'); ``` Unlike regular PostgreSQL tables, hypertables automatically partition data by time while presenting a single logical table to applications. ### Insert Sample Data [Section titled “Insert Sample Data”](#insert-sample-data) Populate the table with some test data. ```sql INSERT INTO sensor_data SELECT NOW() - (g || ' minutes')::interval, (random()*10)::int, random()*30, random()*100 FROM generate_series(1,10000) g; ``` ### Query Time-Series Data [Section titled “Query Time-Series Data”](#query-time-series-data) Now let's use one of TimescaleDB's most popular functions. Calculate the average temperature per hour. ```sql SELECT time_bucket('1 hour', time) AS bucket, avg(temperature) AS avg_temperature FROM sensor_data GROUP BY bucket ORDER BY bucket; ``` The `time_bucket()` function groups timestamps into fixed intervals, making time-series aggregations both intuitive and efficient. This is one of the key capabilities that distinguishes TimescaleDB from standard PostgreSQL. ## Production Considerations [Section titled “Production Considerations”](#production-considerations) This guide intentionally keeps the deployment simple. For production environments, you'll typically want to enable additional capabilities. ### High Availability [Section titled “High Availability”](#high-availability) Deploy multiple PostgreSQL instances. ```yaml instances: 3 ``` CloudNativePG automatically manages leader election and failover. ### Automated Backups [Section titled “Automated Backups”](#automated-backups) CloudNativePG supports several backup strategies: * S3-compatible object storage * Google Cloud Storage * Azure Blob Storage * CSI VolumeSnapshots Choose the approach that aligns with your storage platform and recovery objectives. ### Point-in-Time Recovery [Section titled “Point-in-Time Recovery”](#point-in-time-recovery) CloudNativePG supports restoring your database to an exact point in time using archived WAL files. This capability is essential for recovering from accidental deletes or application errors. ### Monitoring [Section titled “Monitoring”](#monitoring) Production databases should expose metrics to Prometheus and visualize them using Grafana dashboards. Typical metrics include: * Query latency * Replication lag * Storage usage * WAL generation * Connection count ### Storage [Section titled “Storage”](#storage) Database performance depends heavily on storage. Use SSD-backed persistent volumes with an appropriate StorageClass, and size volumes based on expected data growth and retention requirements. ### Upgrading PostgreSQL and TimescaleDB [Section titled “Upgrading PostgreSQL and TimescaleDB”](#upgrading-postgresql-and-timescaledb) One advantage of CloudNativePG extension containers is that PostgreSQL and extension upgrades become independent operations. Instead of rebuilding custom PostgreSQL images every time a new TimescaleDB release becomes available, you can update the extension container version while allowing CloudNativePG to manage PostgreSQL upgrades according to its lifecycle. This significantly reduces operational overhead and helps keep both PostgreSQL and TimescaleDB current with minimal effort. ## Conclusion [Section titled “Conclusion”](#conclusion) CloudNativePG and TimescaleDB complement each other exceptionally well. CloudNativePG simplifies PostgreSQL operations through Kubernetes-native automation, while TimescaleDB extends PostgreSQL with powerful time-series capabilities such as hypertables and time-based analytics. Together they provide a clean, declarative, and production-ready platform for building time-series applications on Kubernetes. By leveraging CloudNativePG's extension container framework, you can avoid maintaining custom PostgreSQL images, simplify upgrades, and keep your deployment aligned with cloud-native best practices. Whether you're collecting IoT telemetry, monitoring infrastructure, processing financial events, or analyzing application metrics, this architecture provides a scalable foundation with minimal operational complexity. # Integrate Kubernetes with Tiger Cloud > Automate deployment, scaling, and management of your containerized workloads [Kubernetes](https://kubernetes.io/) is an open-source container orchestration system that automates the deployment, scaling, and management of containerized applications. You can connect Kubernetes to Tiger Cloud, and deploy TimescaleDB within your Kubernetes clusters. This guide explains how to connect a Kubernetes cluster to Tiger Cloud, configure persistent storage, and deploy TimescaleDB in your kubernetes cluster. ## Prerequisites for this integration guide To follow these steps, you'll need: * **[Self-managed Kubernetes](https://kubernetes.io/docs/setup/) or a Kubernetes [Turnkey Cloud Solution](https://kubernetes.io/docs/setup/production-environment/turnkey-solutions/)**. * **[kubectl](https://kubernetes.io/docs/tasks/tools/)** for command-line interaction with your cluster. ## Integrate TimescaleDB in a Kubernetes cluster [Section titled “Integrate TimescaleDB in a Kubernetes cluster”](#integrate-timescaledb-in-a-kubernetes-cluster) * Tiger Cloud To connect your Kubernetes cluster to your Tiger Cloud service: 1. **Create a default namespace for your Tiger Cloud components** 1. Create a namespace: ```shell kubectl create namespace timescale ``` 2. Set this namespace as the default for your session: ```shell kubectl config set-context --current --namespace=timescale ``` For more information, see [Kubernetes Namespaces](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/). 2. **Create a Kubernetes secret that stores your Tiger Cloud service credentials** Update the following command with your [connection details](/docs/integrate/find-connection-details), then run it: ```shell kubectl create secret generic timescale-secret \ --from-literal=PGHOST= \ --from-literal=PGPORT= \ --from-literal=PGDATABASE= \ --from-literal=PGUSER= \ --from-literal=PGPASSWORD= ``` 3. **Configure network access to Tiger Cloud** * **Managed Kubernetes**: outbound connections to external databases like Tiger Cloud work by default. Make sure your cluster's security group or firewall rules allow outbound traffic to Tiger Cloud IP. * **Self-hosted Kubernetes**: if your cluster is behind a firewall or running on-premise, you may need to allow egress traffic to Tiger Cloud. Test connectivity using your [connection details](/docs/integrate/find-connection-details): ```shell nc -zv ``` If the connection fails, check your firewall rules. 4. **Create a Kubernetes deployment that can access your Tiger Cloud** Run the following command to apply the deployment: ```shell kubectl apply -f - < Provision and manage your infrastructure as code with predictable deployments [Terraform](https://developer.hashicorp.com/terraform) is an infrastructure-as-code tool that enables you to safely and predictably provision and manage infrastructure. This page explains how to configure Terraform to manage your Tiger Cloud service or self-hosted TimescaleDB. ## Prerequisites for this integration guide To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service), or a running instance of [self-hosted TimescaleDB](/docs/get-started/choose-your-path/install-timescaledb). - Your [connection details](/docs/integrate/find-connection-details). * [Terraform](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli) downloaded and installed. ## Configure Terraform [Section titled “Configure Terraform”](#configure-terraform) Configure Terraform based on your deployment type: * Tiger Cloud You use the [Tiger Data Terraform provider](https://registry.terraform.io/providers/timescale/timescale/latest/docs) to manage Tiger Cloud services: 1. **Generate client credentials** [Generate client credentials](/docs/integrate/find-connection-details#create-client-credentials) for the Tiger Data Terraform provider. 2. **Configure Tiger Data Terraform provider** 1. Create a `main.tf` configuration file with at least the following content. Change `x.y.z` to the [latest version](https://registry.terraform.io/providers/timescale/timescale/latest/docs) of the provider. ```hcl terraform { required_providers { timescale = { source = "timescale/timescale" version = "x.y.z" } } } # Authenticate using client credentials generated in Tiger Cloud Console. # When required, these credentials will change to a short-lived JWT to do the calls. provider "timescale" { project_id = var.ts_project_id access_key = var.ts_access_key secret_key = var.ts_secret_key } variable "ts_project_id" { type = string } variable "ts_access_key" { type = string } variable "ts_secret_key" { type = string } ``` 2. Create a `terraform.tfvars` file in the same directory as your `main.tf` to pass in the variable values: ```hcl export TF_VAR_ts_project_id="" export TF_VAR_ts_access_key="" export TF_VAR_ts_secret_key="" ``` 3. **Add your resources** Add your Tiger Cloud services or VPC connections to the `main.tf` configuration file. For example: ```hcl resource "timescale_service" "test" { name = "test-service" milli_cpu = 500 memory_gb = 2 region_code = "us-east-1" enable_ha_replica = false timeouts = { create = "30m" } } resource "timescale_vpc" "vpc" { cidr = "10.10.0.0/16" name = "test-vpc" region_code = "us-east-1" } ``` You can now manage your resources with Terraform. See more about [available resources](https://registry.terraform.io/providers/timescale/timescale/latest/docs/resources/peering_connection) and [data sources](https://registry.terraform.io/providers/timescale/timescale/latest/docs/data-sources/products). * Self-hosted TimescaleDB You use the [`cyrilgdn/postgresql`](https://registry.terraform.io/providers/cyrilgdn/postgresql/latest) PostgreSQL provider to connect to your self-hosted TimescaleDB instance. Create a `main.tf` configuration file with the following content, using your [connection details](/docs/integrate/find-connection-details): ```hcl terraform { required_providers { postgresql = { source = "cyrilgdn/postgresql" version = ">= 1.15.0" } } } provider "postgresql" { host = "your-timescaledb-host" port = "your-timescaledb-port" database = "your-database-name" username = "your-username" password = "your-password" sslmode = "require" # Or "disable" if SSL isn't enabled } ``` You can now manage your database with Terraform. # Connectors > Ingest and sync data into Tiger Data using source and destination connectors Set up Tiger Cloud–managed connectors to stream data into your service from external sources like Apache Kafka, PostgreSQL, or Amazon S3, or to export data to Apache Iceberg tables in S3. ## Integrations [Section titled “Integrations”](#integrations) [Sync from Apache Kafka](/docs/integrate/connectors/source/sync-from-kafka) [Stream events from Kafka topics into your Tiger Data service](/docs/integrate/connectors/source/sync-from-kafka) [Sync from Postgres](/docs/integrate/connectors/source/sync-from-postgres) [Replicate data from a PostgreSQL database into Tiger Data](/docs/integrate/connectors/source/sync-from-postgres) [Sync from S3](/docs/integrate/connectors/source/sync-from-s3) [Load object data from Amazon S3 into Tiger Data](/docs/integrate/connectors/source/sync-from-s3) [Sync to Apache Iceberg](/docs/integrate/connectors/destination/tigerlake) [Export data from your service to Apache Iceberg tables in Amazon S3](/docs/integrate/connectors/destination/tigerlake) # Integrate Snowflake with Tiger Cloud > Query Tiger Cloud time-series data from Snowflake using Apache Iceberg and Amazon S3 Tables, without ETL pipelines or data duplication. [Snowflake](https://www.snowflake.com/) is a cloud data platform for data warehousing, analytics, and data sharing. This page shows you how to configure Snowflake to query data from your Tiger Cloud service using Tiger Cloud Iceberg connector, with no ETL pipelines or data duplication required. Because Snowflake reads directly from Amazon S3 Tables in your AWS account, your data stays in one place and is always current. In this integration guide, you: * Configure a Snowflake Catalog Integration to connect to your S3 Tables namespace * Create a dedicated AWS IAM role that grants Snowflake read-only access to your data * Register your synced tables in Snowflake and run queries ## Prerequisites for this integration guide To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service). - Tiger Cloud Iceberg connector set up and active in your Tiger Cloud service. Follow the [Tiger Cloud Iceberg connector guide](/docs/integrate/connectors/destination/tigerlake) to create one. Note your `S3TableBucketArn` from the CloudFormation stack outputs. You need it throughout this guide. - A [Snowflake](https://docs.snowflake.com/en/user-guide/getting-started-tutorial) account with `ACCOUNTADMIN` privileges. `CREATE CATALOG INTEGRATION` requires `ACCOUNTADMIN` specifically, not just `SYSADMIN`. - [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) installed and authenticated to the same AWS account as your S3 Table Bucket. Warning Your Tiger Cloud service and your S3 Table Bucket must be in the same AWS region. Cross-region traffic incurs per-GB transfer fees. ## Configure Snowflake [Section titled “Configure Snowflake”](#configure-snowflake) Set up the Catalog Integration that tells Snowflake where your tables are and how to authenticate to your S3 Table Bucket. 1. **Find your namespace** Run the following command, replacing `` and `` with the values from your Tiger Cloud Iceberg connector setup: ```shell aws s3tables list-namespaces \ --table-bucket-arn \ --region ``` The output includes two values you use in every step of this guide: * `namespace`: your `` * `ownerAccountId`: your `` This guide uses the following placeholders throughout. Replace them with your actual values before running each command: | Placeholder | What it is | Where to find it | | ------------------------ | -------------------------------------------------- | -------------------------------------------------------------------------------------------------------- | | `` | AWS region where your S3 Table Bucket was created | The segment of your `S3TableBucketArn` after `s3tables:`, for example `us-east-1` | | `` | Your 12-digit AWS account number | Your `S3TableBucketArn`, or the `ownerAccountId` field above | | `` | Name of your S3 Table Bucket | The portion of your `S3TableBucketArn` after `bucket/` | | `` | Logical grouping of tables inside the bucket | The `namespace` field from this step | | `` | Name of a specific synced table | Output of `aws s3tables list-tables` (see [Register tables in Snowflake](#register-tables-in-snowflake)) | | `` | Snowflake's AWS identity | Output of `DESC INTEGRATION` (see the next step) | | `` | Snowflake's external ID for secure role assumption | Output of `DESC INTEGRATION` (see the next step) | For example, for the ARN `arn:aws:s3tables:us-east-1:111122223333:bucket/my-iceberg-bucket`, the placeholders are: ```plaintext → us-east-1 → 111122223333 → my-iceberg-bucket ``` 2. **Create the Catalog Integration** Open a Snowflake worksheet with the `ACCOUNTADMIN` role. Replace all placeholders using the reference table above, then run: ```sql USE ROLE ACCOUNTADMIN; CREATE OR REPLACE CATALOG INTEGRATION tiger_s3tables_catalog CATALOG_SOURCE = ICEBERG_REST TABLE_FORMAT = ICEBERG CATALOG_NAMESPACE = '' REST_CONFIG = ( CATALOG_URI = 'https://glue..amazonaws.com/iceberg' CATALOG_API_TYPE = AWS_GLUE WAREHOUSE = ':s3tablescatalog/' ACCESS_DELEGATION_MODE = VENDED_CREDENTIALS ) REST_AUTHENTICATION = ( TYPE = SIGV4 SIGV4_IAM_ROLE = 'arn:aws:iam:::role/snowflake-s3tables-reader' SIGV4_SIGNING_REGION = '' ) REFRESH_INTERVAL_SECONDS = 120 ENABLED = TRUE; ``` The role name `snowflake-s3tables-reader` is the IAM role you create in the next section. You reference it here in advance so Snowflake knows which role to assume. 3. **Get Snowflake's AWS identity** Run the following in the same Snowflake worksheet: ```sql DESC INTEGRATION tiger_s3tables_catalog; ``` From the output, save these two values, which you need in the next section: | Field | Example value | | ---------------------- | --------------------------------------- | | `API_AWS_IAM_USER_ARN` | `arn:aws:iam::111122223333:user/abc123` | | `API_AWS_EXTERNAL_ID` | `ABC12345_SFCRole=2_xxxx=` | ## Grant AWS access to Snowflake [Section titled “Grant AWS access to Snowflake”](#grant-aws-access-to-snowflake) Create a dedicated IAM role so Snowflake can authenticate to your S3 Table Bucket. A separate role keeps the Tiger Cloud Iceberg connector write path isolated. Snowflake configuration changes can never affect the sync, and access can be revoked independently. 1. **Create the IAM role** Use the `API_AWS_IAM_USER_ARN` and `API_AWS_EXTERNAL_ID` values from the previous section: ```shell aws iam create-role \ --role-name snowflake-s3tables-reader \ --assume-role-policy-document '{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "" }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "sts:ExternalId": "" } } } ] }' ``` 2. **Attach read permissions to the role** ```shell aws iam put-role-policy \ --role-name snowflake-s3tables-reader \ --policy-name snowflake-s3tables-access \ --policy-document '{ "Version": "2012-10-17", "Statement": [ { "Sid": "GlueAccess", "Effect": "Allow", "Action": [ "glue:GetCatalog", "glue:GetDatabase", "glue:GetDatabases", "glue:GetTable", "glue:GetTables" ], "Resource": "*" }, { "Sid": "LakeFormationAccess", "Effect": "Allow", "Action": [ "lakeformation:GetDataAccess" ], "Resource": "*" }, { "Sid": "S3TablesReadAccess", "Effect": "Allow", "Action": [ "s3tables:GetTableBucket", "s3tables:GetNamespace", "s3tables:ListNamespaces", "s3tables:GetTable", "s3tables:ListTables", "s3tables:GetTableData", "s3tables:GetTableMetadataLocation" ], "Resource": [ "arn:aws:s3tables:::bucket/", "arn:aws:s3tables:::bucket//table/*" ] } ] }' ``` 3. **Grant Lake Formation permissions** Grant the Snowflake role `SELECT` and `DESCRIBE` access to all tables in your namespace: ```shell aws lakeformation grant-permissions \ --region \ --principal DataLakePrincipalIdentifier=arn:aws:iam:::role/snowflake-s3tables-reader \ --resource '{ "Table": { "CatalogId": ":s3tablescatalog/", "DatabaseName": "", "TableWildcard": {} } }' \ --permissions "SELECT" "DESCRIBE" ``` ## Register tables in Snowflake [Section titled “Register tables in Snowflake”](#register-tables-in-snowflake) Each table that Tiger Cloud Iceberg connector has synced must be registered once in Snowflake before it can be queried. 1. **Create a database and schema in Snowflake** ```sql CREATE DATABASE IF NOT EXISTS tiger_data; CREATE SCHEMA IF NOT EXISTS tiger_data.; ``` 2. **List the synced tables** Run the following command to see which tables Tiger Cloud Iceberg connector has written to your bucket: ```shell aws s3tables list-tables \ --table-bucket-arn arn:aws:s3tables:::bucket/ \ --namespace \ --region ``` The output lists your table names. These are your `` values. 3. **Register each table** Run the following once per table name from the previous step: ```sql CREATE ICEBERG TABLE tiger_data.. CATALOG = 'tiger_s3tables_catalog' CATALOG_TABLE_NAME = '' AUTO_REFRESH = TRUE; ``` Each time Tiger Cloud Iceberg connector syncs a new table, run this statement to make it queryable in Snowflake. ## Verify the integration [Section titled “Verify the integration”](#verify-the-integration) To confirm Snowflake can access your Tiger Cloud service data: 1. **Verify catalog connectivity** Run the following in a Snowflake worksheet: ```sql SELECT SYSTEM$VERIFY_CATALOG_INTEGRATION('tiger_s3tables_catalog'); ``` A successful response looks like: ```json { "success": true, "errorCode": "", "errorMessage": "" } ``` 2. **Query a registered table** Run a query against one of the tables you registered: ```sql SELECT * FROM tiger_data.. LIMIT 10; ``` You see the first ten rows of data synced from your Tiger Cloud service. The real value is joining time-series data from Tiger Cloud with reference data that lives natively in Snowflake, with no pipeline required. For example, given a `customers` table already in Snowflake and a `sensor_readings` hypertable synced from Tiger Cloud: ```sql SELECT c.customer_name, DATE_TRUNC('hour', s.time) AS hour, AVG(s.temperature) AS avg_temp FROM tiger_data..sensor_readings AS s JOIN snowflake_warehouse.public.customers AS c ON s.customer_id = c.id WHERE s.time >= DATEADD(day, -7, CURRENT_TIMESTAMP()) GROUP BY 1, 2 ORDER BY 2 DESC; ``` You have successfully integrated Snowflake with Tiger Cloud. ## Refresh metadata on demand [Section titled “Refresh metadata on demand”](#refresh-metadata-on-demand) New rows inserted in Tiger Cloud are visible in Snowflake within the `REFRESH_INTERVAL_SECONDS` window (120 seconds by default). To see changes immediately without waiting: ```sql ALTER CATALOG INTEGRATION tiger_s3tables_catalog REFRESH; ``` ## Troubleshooting [Section titled “Troubleshooting”](#troubleshooting) * **`sts:AssumeRole not authorized`:** The `API_AWS_IAM_USER_ARN` or `API_AWS_EXTERNAL_ID` in the IAM role trust policy is incorrect or stale. Re-run the `DESC INTEGRATION` step and recreate the role with fresh values. * **`glue:GetCatalog not authorized`:** The Snowflake IAM role is missing Glue permissions. Re-run the `aws iam put-role-policy` step. * **`Unable to retrieve credentials from Lake Formation`:** The Snowflake IAM role is missing `lakeformation:GetDataAccess`. Re-run the `aws iam put-role-policy` step. * **`Insufficient Lake Formation permission on table`:** The Lake Formation grant is missing. Re-run the `aws lakeformation grant-permissions` step. * **`Unmatched catalog api type PUBLIC and authentication type SIGV4`:** `CATALOG_API_TYPE = AWS_GLUE` is missing from `REST_CONFIG`. Re-run the `CREATE OR REPLACE CATALOG INTEGRATION` step. * **Table not visible after `list-tables`:** Tiger Cloud Iceberg connector has not completed a full sync cycle yet. Check connector status in Tiger Console. * **Stale data in Snowflake:** Run `ALTER CATALOG INTEGRATION tiger_s3tables_catalog REFRESH` to force a metadata refresh. * **Tiger Cloud Iceberg connector write path failing:** The Tiger Cloud Iceberg connector IAM role and the `snowflake-s3tables-reader` role are completely separate. Check that you have not modified the Tiger Cloud Iceberg connector IAM role. ## Limitations [Section titled “Limitations”](#limitations) * Snowflake has read-only access to your S3 Tables data. Writing back to S3 via Snowflake is not supported. * Tables added by Tiger Cloud Iceberg connector after initial setup are not automatically visible in Snowflake. Run `CREATE ICEBERG TABLE` for each new table. * This integration is not available for Azure deployments of Tiger Cloud. ## Next steps [Section titled “Next steps”](#next-steps) [Iceberg destination connector](/docs/integrate/connectors/destination/tigerlake) [Set up or manage the Apache Iceberg destination connector that syncs your service data to Amazon S3 Tables.](/docs/integrate/connectors/destination/tigerlake) [Snowflake integration cookbook](https://github.com/timescale/cookbook-integrations/tree/start/integrations-cookbook/snowflake) [End-to-end example using a real dataset with an analytical join between Iceberg and Snowflake-native tables.](https://github.com/timescale/cookbook-integrations/tree/start/integrations-cookbook/snowflake) # Integrate data lakes with Tiger Cloud > Unify the Tiger Cloud operational architecture with data lake architectures Tiger Cloud Iceberg connector enables you to build real-time applications alongside efficient data pipeline management within a single system. ![TigerLake architecture: Iceberg connector in Tiger Cloud](/docs/_astro/iceberg-connector-tiger-cloud.CdQvMEuj_ZQC2Hs.webp) ![TigerLake architecture: Iceberg connector in Tiger Cloud](/docs/_astro/iceberg-connector-tiger-cloud-dark.DnJNpBlG_Z1yxyia.webp) Tiger Cloud Iceberg connector is a native integration enabling synchronization between hypertables and relational tables running in Tiger Cloud services to Iceberg tables running in [Amazon S3 Tables](https://aws.amazon.com/s3/features/tables/) in your AWS account. ## Prerequisites for this integration guide To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service). Note This feature is currently not supported for Tiger Cloud on Microsoft Azure. ## Integrate a data lake with your service [Section titled “Integrate a data lake with your service”](#integrate-a-data-lake-with-your-service) To connect a Tiger Cloud service to your data lake: * AWS Management Console 1. **Set the AWS region to host your table bucket** 1. In [AWS CloudFormation](https://console.aws.amazon.com/cloudformation/), select the current AWS region at the top-right of the page. 2. Set it to the Region you want to create your table bucket in. **This must match the region your Tiger Cloud service is running in**: if the regions do not match AWS charges you for cross-region data transfer. 2. **Create your CloudFormation stack** 1. Click `Create stack`, then select `With new resources (standard)`. 2. In `Amazon S3 URL`, paste the following URL, then click `Next`. ```http https://tigerlake.s3.us-east-1.amazonaws.com/tigerlake-connect-cloudformation.yaml ``` 3. In `Specify stack details`, enter the following details, then click `Next`: * `Stack Name`: a name for this CloudFormation stack * `BucketName`: a name for this S3 table bucket * `ProjectID` and `ServiceID`: enter the [connection details](/docs/integrate/find-connection-details#find-your-project-and-service-id) for your Tiger Cloud Iceberg connector service 4. In `Configure stack options` check `I acknowledge that AWS CloudFormation might create IAM resources`, then click `Next`. 5. In `Review and create`, click `Submit`, then wait for the deployment to complete. AWS deploys your stack and creates the S3 table bucket and IAM role. 6. Click `Outputs`, then copy all four outputs. 3. **Connect your service to the data lake** 1. In [Tiger Console](https://console.cloud.tigerdata.com/dashboard/services), select the service you want to integrate with AWS S3 Tables, then click `Connectors`. 2. Select the Apache Iceberg connector and supply the: * ARN of the S3Table bucket * ARN of a role with permissions to write to the table bucket Provisioning takes a couple of minutes. * AWS CloudFormation CLI 1. **Create your CloudFormation stack** Replace the following values in the command, then run it from the terminal: * `Region`: region of the S3 table bucket * `StackName`: the name for this CloudFormation stack * `BucketName`: the name of the S3 table bucket to create * `ProjectID`: enter your Tiger Cloud service [connection details](/docs/integrate/find-connection-details#find-your-project-and-service-id) * `ServiceID`: enter your Tiger Cloud service [connection details](/docs/integrate/find-connection-details#find-your-project-and-service-id) ```shell aws cloudformation create-stack \ --capabilities CAPABILITY_IAM \ --template-url https://tigerlake.s3.us-east-1.amazonaws.com/tigerlake-connect-cloudformation.yaml \ --region \ --stack-name \ --parameters \ ParameterKey=BucketName,ParameterValue="" \ ParameterKey=ProjectID,ParameterValue="" \ ParameterKey=ServiceID,ParameterValue="" ``` Setting up the integration through Tiger Console in Tiger Cloud, provides a convenient copy-paste option with the placeholders populated. 2. **Connect your service to the data lake** 1. In [Tiger Console](https://console.cloud.tigerdata.com/dashboard/services), select the service you want to integrate with AWS S3 Tables, then click `Connectors`. 2. Select the Apache Iceberg connector and supply the: * ARN of the S3Table bucket * ARN of a role with permissions to write to the table bucket Provisioning takes a couple of minutes. * Manual configuration 1. **Create a S3 Bucket** 1. Set the AWS region to host your table bucket: 1. In [Amazon S3 console](https://console.aws.amazon.com/s3/), select the current AWS region at the top-right of the page. 2. Set it to the Region your you want to create your table bucket in. **This must match the region your Tiger Cloud service is running in**: if the regions do not match AWS charges you for cross-region data transfer. 2. In the left navigation pane, click `Table buckets`, then click `Create table bucket`. 3. Enter `Table bucket name`, then click `Create table bucket`. 4. Copy the `Amazon Resource Name (ARN)` for your table bucket. 2. **Create an ARN role** 1. In [IAM Dashboard](https://console.aws.amazon.com/iamv2/home), click `Roles` then click `Create role`. 2. In `Select trusted entity`, click `Custom trust policy`, replace the `Custom trust policy` code block with the following: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::142548018081:root" }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "sts:ExternalId": "/" } } } ] } ``` `"Principal": { "AWS": "arn:aws:iam::123456789012:root" }` does not mean `root` access. This delegates permissions to the entire AWS account, not just the root user. 3. Replace `` and `` with the [connection details](/docs/integrate/find-connection-details#find-your-project-and-service-id) for your Tiger Cloud Iceberg connector service, then click `Next`. 4. In `Permissions policies`, click `Next`. 5. In `Role details`, enter `Role name`, then click `Create role`. 6. In `Roles`, select the role you just created, then click `Add Permissions` > `Create inline policy`. 7. Select `JSON` then replace the `Policy editor` code block with the following: ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "BucketOps", "Effect": "Allow", "Action": [ "s3tables:*" ], "Resource": "" }, { "Sid": "BucketTableOps", "Effect": "Allow", "Action": [ "s3tables:*" ], "Resource": "/table/*" } ] } ``` 8. Replace `` with the `Amazon Resource Name (ARN)` for the table bucket you just created. 9. Click `Next`, then give the inline policy a name and click `Create policy`. 3. **Connect your service to the data lake** 1. In [Tiger Console](https://console.cloud.tigerdata.com/dashboard/services), select the service you want to integrate with AWS S3 Tables, then click `Connectors`. 2. Select the Apache Iceberg connector and supply the: * ARN of the S3Table bucket * ARN of a role with permissions to write to the table bucket Provisioning takes a couple of minutes. Important If your table bucket is encrypted with a customer managed AWS KMS key ([SSE-KMS](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-kms-encryption.html)) instead of the default AWS managed encryption, the role you supply to Tiger Cloud Iceberg connector also needs `kms:GenerateDataKey` and `kms:Decrypt` permissions on that key. Without them, Iceberg connector cannot write to the bucket and syncs fail with a `not authorized to perform: kms:GenerateDataKey` error. Add the following statement to the role's inline policy, replacing `` with the ARN of your KMS key: ```json { "Sid": "KMSAccess", "Effect": "Allow", "Action": [ "kms:GenerateDataKey", "kms:Decrypt" ], "Resource": "" } ``` ## Stream data from your service to your data lake [Section titled “Stream data from your service to your data lake”](#stream-data-from-your-service-to-your-data-lake) Records are imported in time order, from oldest to newest. Your hypertable or relational table must have a primary key, or composite primary keys as a prerequisite to sync to Iceberg. When you start syncing, all data in the table is streamed to Iceberg in the following processes: * Table snapshot: stream data from a snapshot of the source table to the destination Iceberg table at approximately 300.000 records a second. For larger tables, import speeds are approximately 1 billion records or 100 GB of data an hour. However, these numbers vary on table width and the complexity of the schema. * Table changes: stream changes made to the source table (CDC) after the snapshot is taken to a branch of the destination Iceberg table. This happens at approximately 30.000 events a second. Ingest bursts exceeding this can be handled for a certain amount of time and feathered out over time. This depends on duration of the ingestion burst, and the amount of extra events to be handled. Once the snapshot is fully imported, the snapshot and CDC Iceberg table branches are merged. Merging takes from a couple of seconds, to ten minutes for larger tables of 5TB or more. During this time, new events are held on the WAL. Once the merge is completed, events in the WAL are CDC'd to Iceberg. This implies eventual consistency of the Iceberg table after you started the the sync. To stream data from a PostgreSQL relational table, or a hypertable in your Tiger Cloud service to your data lake, run the following statement: ```sql ALTER TABLE SET ( tigerlake.iceberg_sync = true | false, tigerlake.iceberg_partitionby = '', tigerlake.iceberg_namespace = '', tigerlake.iceberg_table = '
' ) ``` * `tigerlake.iceberg_sync`: `boolean`, set to `true` to start streaming, or `false` to stop the stream. A stream **cannot** resume after being stopped. * `tigerlake.iceberg_partitionby`: optional property to define a partition specification in Iceberg. By default the Iceberg table is partitioned as `day()`. This default behavior is only applicable to hypertables. For more information, see [partitioning](/docs/integrate/connectors/destination/tigerlake#partitioning-intervals). * `tigerlake.iceberg_namespace`: optional property to set a namespace, the default is `timescaledb`. * `tigerlake.iceberg_table`: optional property to specify a different table name. If no name is specified the PostgreSQL table name is used. ### Partitioning intervals [Section titled “Partitioning intervals”](#partitioning-intervals) By default, the partition interval for an Iceberg table is one day(time-column) for a Hypertable. PostgreSQL table sync does not enable any partitioning in Iceberg for non-hypertables. You can set it using [tigerlake.iceberg\_partitionby](/docs/integrate/connectors/destination/tigerlake#sample-code). The following partition intervals and specifications are supported: | Interval | Description | Source types | | ------------- | ------------------------------------------------------------------------------------------------------ | ---------------------------------- | | `hour` | Extract a date or timestamp day, as days from epoch. Epoch is 1970-01-01. | `date`, `timestamp`, `timestamptz` | | `day` | Extract a date or timestamp day, as days from epoch. | `date`, `timestamp`, `timestamptz` | | `month` | Extract a date or timestamp day, as days from epoch. | `date`, `timestamp`, `timestamptz` | | `year` | Extract a date or timestamp day, as days from epoch. | `date`, `timestamp`, `timestamptz` | | `truncate[W]` | Value truncated to width W, see [options](https://iceberg.apache.org/spec/#truncate-transform-details) | | These partitions define the behavior using the [Iceberg partition specification](https://iceberg.apache.org/spec/#partition-transforms). ### Sample code [Section titled “Sample code”](#sample-code) The following samples show you how to tune data sync from a hypertable or a PostgreSQL relational table to your data lake: * **Sync a hypertable with the default one-day partitioning interval on the `ts_column` column** To start syncing data from a hypertable to your data lake using the default one-day chunk interval as the partitioning scheme to the Iceberg table, run the following statement: ```sql ALTER TABLE my_hypertable SET (tigerlake.iceberg_sync = true); ``` This is equivalent to `day(ts_column)`. * **Specify a custom partitioning scheme for a hypertable** You use the `tigerlake.iceberg_partitionby` property to specify a different partitioning scheme for the Iceberg table at sync start. For example, to enforce an hourly partition scheme from the chunks on `ts_column` on a hypertable, run the following statement: ```sql ALTER TABLE my_hypertable SET ( tigerlake.iceberg_sync = true, tigerlake.iceberg_partitionby = 'hour(ts_column)' ); ``` * **Set the partition to sync relational tables** PostgreSQL relational tables do not forward a partitioning scheme to Iceberg, you must specify the partitioning scheme using `tigerlake.iceberg_partitionby` when you start the sync. For example, for a standard PostgreSQL table to sync to the Iceberg table with daily partitioning , run the following statement: ```sql ALTER TABLE my_postgres_table SET ( tigerlake.iceberg_sync = true, tigerlake.iceberg_partitionby = 'day(timestamp_col)' ); ``` * **Stop sync to an Iceberg table for a hypertable or a PostgreSQL relational table** ```sql ALTER TABLE my_hypertable SET (tigerlake.iceberg_sync = false); ``` * **Update or add the partitioning scheme of an Iceberg table** To change the partitioning scheme of an Iceberg table, you specify the desired partitioning scheme using the `tigerlake.iceberg_partitionby` property. For example. if the `samples` table has an hourly (`hour(ts)`) partition on the `ts` timestamp column, to change to daily partitioning, call the following statement: ```sql ALTER TABLE samples SET (tigerlake.iceberg_partitionby = 'day(ts)'); ``` This statement is also correct for Iceberg tables without a partitioning scheme. When you change the partition, you **do not** have to pause the sync to Iceberg. Apache Iceberg handles the partitioning operation in function of the internal implementation. **Specify a different namespace** By default, tables are created in the the `timescaledb` namespace. To specify a different namespace when you start the sync, use the `tigerlake.iceberg_namespace` property. For example: ```sql ALTER TABLE my_hypertable SET ( tigerlake.iceberg_sync = true, tigerlake.iceberg_namespace = 'my_namespace' ); ``` **Specify a different Iceberg table name** The table name in Iceberg is the same as the source table in Tiger Cloud. Some services do not allow mixed case, or have other constraints for table names. To define a different table name for the Iceberg table at sync start, use the `tigerlake.iceberg_table` property. For example: ```sql ALTER TABLE Mixed_CASE_TableNAME SET ( tigerlake.iceberg_sync = true, tigerlake.iceberg_table = 'my_table_name' ); ``` ## Limitations [Section titled “Limitations”](#limitations) * Service requires PostgreSQL 17.6 and above is supported. * [Amazon S3 Tables Iceberg REST](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-integrating-open-source.html) catalog only is supported. * In order to collect deletes made to data in the columstore, certain columnstore optimizations are disabled for hypertables, this includes [Direct Compress](/docs/learn/hypertables/optimize-data-in-hypertables#speed-up-data-ingestion). * The `TRUNCATE` statement is not supported, and does not truncate data in the corresponding Iceberg table. * Data in a hypertable that has been moved to the [low-cost object storage tier](/docs/learn/data-lifecycle/storage/about-storage-tiers) is not synced. * Writing to the same S3 table bucket from multiple services is not supported, bucket-to-service mapping is one-to-one. * Iceberg snapshots are pruned automatically if the amount exceeds 2500. * A hypertable with long running continuous aggregates refresh transactions, plus 30 minutes, can cause issues with holding the replication slot too long. Please consider batching in these cases. # Stream data from Kafka > Store, query, and analyze your Apache Kafka events in Tiger Cloud Early access Note The Kafka source connector is not yet supported for production use. You use the Kafka source connector in Tiger Cloud to stream events from Kafka into your service. Tiger Cloud connects to your Confluent Cloud Kafka cluster and Schema Registry using SASL/SCRAM authentication and service account–based API keys. Only the Avro format is currently supported [with some limitations](#known-limitations-and-unsupported-types). This page explains how to connect Tiger Cloud to your Confluent Cloud Kafka cluster. Note This page covers the fully managed Kafka source connector, which streams from Confluent Cloud clusters into your service. To stream from a self-managed Apache Kafka cluster using your own Kafka Connect deployment, see [Integrate Apache Kafka with Tiger Cloud](/docs/integrate/data-engineering-etl/apache-kafka). ## Prerequisites for this procedure To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service). - A [Confluent Cloud](https://www.confluent.io/get-started/) account. - A [Kafka cluster](https://docs.confluent.io/cloud/current/clusters/create-cluster.html) created in Confluent Cloud. ## Access your Kafka cluster in Confluent Cloud Take the following steps to prepare your Kafka cluster for connection to Tiger Cloud: 1. **Create a service account** If you already have a service account for Tiger Cloud, you can reuse it. To create a new service account: 1. Log in to [Confluent Cloud](https://confluent.cloud/). 2. Click the burger menu at the top-right of the pane, then press `Access control` > `Service accounts` > `Add service account`. 3. Enter the following details: * Name: `tigerdata-access` * Description: `Service account for the Tiger Cloud source connector` 4. Add the service account owner role, then click `Next`. 5. Select a role assignment, then click `Add`. 6. Click `Next`, then click `Create service account`. 2. **Create API keys** 1. In Confluent Cloud, click `Home` > `Environments` > Select your environment > Select your cluster. 2. Under `Cluster overview` in the left sidebar, select `API Keys`. 3. Click `Add key`, choose `Service Account` and click `Next`. 4. Select `tigerdata-access`, then click `Next`. 5. For your cluster, choose the `Operation` and select the following `Permission`s, then click `Next`: * `Resource type`: `Cluster` * `Operation`: `DESCRIBE` * `Permission`: `ALLOW` 6. Click `Download and continue`, then securely store the ACL. 7. Use the same procedure to add the following keys: * ACL 2: Topic access * `Resource type`: `Topic` * `Topic name`: Select the topics that Tiger Cloud should read * `Pattern type`: `LITERAL` * `Operation`: `READ` * `Permission`: `ALLOW` * ACL 3: Consumer group access * `Resource type`: `Consumer group` * `Consumer group ID`: `tigerdata-kafka/`. See [Connect to Tiger Data](/docs/integrate/find-connection-details) for where to find your project ID * `Pattern type`: `PREFIXED` * `Operation`: `READ` * `Permission`: `ALLOW` You need these to configure your Kafka source connector in Tiger Cloud. ## Configure Confluent Cloud Schema Registry Tiger Cloud requires access to the Schema Registry to fetch schemas for Kafka topics. To configure the Schema Registry: 1. **Navigate to `Schema Registry`** In Confluent Cloud, click `Environments` and select your environment, then click `Stream Governance`. 2. **Create a Schema Registry API key** 1. Click `API Keys`, then click `Add API Key`. 2. Choose `Service Account`, select `tigerdata-access`, then click `Next`. 3. Under `Resource scope`, choose `Schema Registry`, select the `default` environment, then click `Next`. 4. In `Create API Key`, add the following, then click `Create API Key`: * `Name`: `tigerdata-schema-registry-access` * `Description`: `API key for Tiger Cloud schema registry access` 5. Click `Download API Key` and securely store the API key and secret, then click `Complete`. 3. **Assign roles for Schema Registry** 1. Click the burger menu at the top-right of the pane, then press `Access control` > `Accounts & access` > `Service accounts`. 2. Select the `tigerdata-access` service account. 3. In the `Access` tab, add the following role assignments for `All schema subjects`: * `ResourceOwner` on the service account. * `DeveloperRead` on schema subjects. Choose `All schema subjects` or restrict to specific subjects as required. 4. Save the role assignments. Your Confluent Cloud Schema Registry is now accessible to Tiger Cloud using the API key and secret. ## Add Kafka source connector in Tiger Cloud Take the following steps to create a Kafka source connector in Tiger Console. 1. **In Console, select your service** Open [Console](https://console.cloud.tigerdata.com/) and select your service. 2. **Go to Connectors > Source connectors** Click `New Connector`, then select `Kafka`. 3. **Set the connector name** Click the pencil icon, then set the connector name. 4. **Set up Kafka authentication** Enter the name of your cluster in Confluent Cloud and the information from the first `api-key-*.txt` that you downloaded, then click `Authenticate`. 5. **Set up the Schema Registry** Enter the service account ID and the information from the second `api-key-*.txt` that you downloaded, then click `Authenticate`. 6. **Select topics to sync** Add the schema and table, map the columns in the table, and click `Create connector`. Your Kafka connector is configured and ready to stream events. ## Known limitations and unsupported types The following Avro schema types are not supported: ### Union types Multi-type non-nullable unions are blocked. Examples: * Multiple type union: ```json { "type": "record", "name": "Message", "fields": [ {"name": "content", "type": ["string", "bytes", "null"]} ] } ``` * Union as root schema: ```json ["null", "string"] ``` ### Reference types (named type references) Referencing a previously defined named type by name, instead of inline, is not supported. Examples: * Named type definition: ```json { "type": "record", "name": "Address", "fields": [ {"name": "street", "type": "string"}, {"name": "city", "type": "string"} ] } ``` * Failing reference: ```json { "type": "record", "name": "Person", "fields": [ {"name": "name", "type": "string"}, {"name": "address", "type": "Address"} ] } ``` ### Unsupported logical types Only the logical types in the hardcoded supported list are supported. This includes: * decimal, date, time-millis, time-micros * timestamp-millis, timestamp-micros, timestamp-nanos * local-timestamp-millis, local-timestamp-micros, local-timestamp-nanos * uuid, duration Unsupported examples: ```json { "type": "int", "logicalType": "date-time" } { "type": "string", "logicalType": "json" } { "type": "bytes", "logicalType": "custom-type" } ``` # Sync data from Postgres > Sync updates to your primary Postgres database with Tiger Cloud in real time Tips **Source PostgreSQL connector vs. Livesync replication:** The source PostgreSQL connector provides **continuous ongoing replication** where PostgreSQL stays the primary and Tiger Cloud acts as a logical replica. For a **one-time migration** to Tiger Cloud with a cutover at the end, see [Livesync replication](/docs/migrate/livesync-replication) instead. Both features share the same underlying technology, but the workflows and end states differ. You use the source PostgreSQL connector in Tiger Cloud to synchronize all data or specific tables from a PostgreSQL database instance to your service, in real time. You run the connector continuously, turning PostgreSQL into a primary database with your service as a logical replica. This enables you to leverage Tiger Cloud's real-time analytics capabilities on your replica data. ![Connectors overview in Tiger Console](/docs/_astro/tiger-console-connector-overview.C3brL-kO_Z2bCaMM.webp) The source PostgreSQL connector in Tiger Cloud leverages the well-established PostgreSQL logical replication protocol. By relying on this protocol, Tiger Cloud ensures compatibility, familiarity, and a broader knowledge base, making it easier for you to adopt the connector and integrate your data. You use the source PostgreSQL connector for data synchronization, rather than migration. This includes: * Copy existing data from a PostgreSQL instance to a Tiger Cloud service: * Copy data at up to 150 GB/hr. You need at least a 4 CPU/16 GB source database, and a 4 CPU/16 GB target service. * Copy the publication tables in parallel. Large individual tables still use a single connection, except PostgreSQL declarative partitioned tables published with `publish_via_partition_root = false` (the PostgreSQL default), which are copied leaf partition by leaf partition in parallel since v0.23.0. * Forget foreign key relationships. The connector disables foreign key validation during the sync. For example, if a `metrics` table refers to the `id` column on the `tags` table, you can still sync only the `metrics` table without worrying about their foreign key relationships. * Track progress. PostgreSQL exposes `COPY` progress under `pg_stat_progress_copy`. * Synchronize real-time changes from a PostgreSQL instance to a Tiger Cloud service. * Add and remove tables on demand using the [PostgreSQL PUBLICATION interface](https://www.postgresql.org/docs/current/sql-createpublication.html). * Enable features such as [hypertables](/docs/learn/hypertables/understand-hypertables), [columnstore](/docs/learn/columnar-storage/understand-hypercore), and [continuous aggregates](/docs/learn/continuous-aggregates) on your logical replica. * Indexes, primary key, unique constraints, and sequences are **not** migrated. Create needed indexes on the target for your queries. * TimescaleDB as source has limited support (for example, no continuous aggregates). * Schema changes must be coordinated: apply compatible changes on the target first, then on the source. * WAL volume on the source increases during large table copy. * **Continuous aggregates:** The connector uses `session_replication_role=replica` during copy, so triggers (including continuous aggregate invalidation) do not run. Data synced during initial load below a continuous aggregate's materialization watermark may not appear in the aggregate until you manually refresh. If the aggregate exists on the source, include it in the connector's publication; if only on the target, use the `force` option of [refresh\_continuous\_aggregate](/docs/reference/timescaledb/continuous-aggregates/refresh_continuous_aggregate) to refresh affected ranges. - Tiger Console ## Prerequisites for this integration guide To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service). - Your [connection details](/docs/integrate/find-connection-details). * The [PostgreSQL client tools](/docs/integrate/query-administration/psql) installed on your sync machine. * The source PostgreSQL instance and the target Tiger Cloud service must have the same extensions installed. The source PostgreSQL connector does not create extensions on the target. If the table uses column types from an extension, first create the extension on the target Tiger Cloud service before syncing the table. ## Limitations * Indexes, including the primary key, unique constraints, and sequences are not migrated to the target Tiger Cloud service. We recommend that, depending on your query patterns, you create only the necessary indexes on the target Tiger Cloud service. - Using TimescaleDB as the source has limited support (no CAGGs). - The source must be running PostgreSQL 13 or later. - Schema changes must be co-ordinated. Make compatible changes to the schema in your Tiger Cloud service first, then make the same changes to the source PostgreSQL instance. - Ensure that the source PostgreSQL instance and the target Tiger Cloud service have the same extensions installed. The source PostgreSQL connector does not create extensions on the target. If the table uses column types from an extension, first create the extension on the target Tiger Cloud service before syncing the table. - There is WAL volume growth on the source PostgreSQL instance during large table copy. - Continuous aggregate invalidation The connector uses `session_replication_role=replica` during data replication, which prevents table triggers from firing. This includes the internal triggers that mark continuous aggregates as invalid when underlying data changes. If you have continuous aggregates on your target database, they do not automatically refresh for data inserted during the migration. This limitation only applies to data below the continuous aggregate's materialization watermark. For example, backfilled data. New rows synced above the continuous aggregate watermark are used correctly when refreshing. This can lead to: * Missing data in continuous aggregates for the migration period. * Stale aggregate data. * Queries returning incomplete results. If the continuous aggregate exists in the source database, best practice is to add it to the PostgreSQL connector publication. If it only exists on the target database, manually refresh the continuous aggregate using the `force` option of [refresh\_continuous\_aggregate](/docs/reference/timescaledb/continuous-aggregates/refresh_continuous_aggregate#samples). ## Set your connection string This variable holds the connection information for the source database. In the terminal on your migration machine, set the following: ```bash export SOURCE="postgres://:@:/" ``` Important Avoid using connection strings that route through connection poolers like PgBouncer or similar tools. This tool requires a direct connection to the database to function properly. ## Tune your source database * From AWS RDS/Aurora Updating parameters on a PostgreSQL instance will cause an outage. Choose a time that will cause the least issues to tune this database. 1. **Tune the Write Ahead Log (WAL) on the RDS/Aurora PostgreSQL source database** 1. In [RDS console](https://console.aws.amazon.com/rds/home#databases:), select the RDS instance to migrate. 2. Click `Configuration`, scroll down and note the `DB instance parameter group`, then click `Parameter Groups`. ![RDS parameter groups in the AWS console](/docs/_astro/awsrds-parameter-groups.CiXrFBVV_1Diea.webp) 3. Click `Create parameter group`, fill in the form with the following values, then click `Create`: * `Parameter group name`, whatever suits your fancy. * `Description`, knock yourself out with this one. * `Engine type`, `PostgreSQL` * `Parameter group family`, the same as `DB instance parameter group` in your `Configuration`. 4. In `Parameter groups`, select the parameter group you created, then click `Edit`. 5. Update the following parameters, then click `Save changes`: * `rds.logical_replication` set to `1`: record the information needed for logical decoding. * `wal_sender_timeout` set to `0`: disable the timeout for the sender process. 6. In RDS, navigate back to your [databases](https://console.aws.amazon.com/rds/home#databases:), select the RDS instance to migrate, and click `Modify`. 7. Scroll down to `Database options`, select your new parameter group, and click `Continue`. 8. Click `Apply immediately` or choose a maintenance window, then click `Modify DB instance`. Changing parameters will cause an outage. Wait for the database instance to reboot before continuing. After it comes back up, verify that the new settings are in effect on your database. 2. **Create a user for the source PostgreSQL connector and assign permissions** 1. Create ``: ```bash psql $SOURCE -c "CREATE USER PASSWORD ''" ``` You can use an existing user. However, you must ensure that the user has the following permissions. 2. Grant permissions to create a replication slot: ```bash psql $SOURCE -c "GRANT rds_replication TO " ``` 3. Grant permissions to create a publication: ```bash psql $SOURCE -c "GRANT CREATE ON DATABASE TO " ``` 4. Assign the user permissions on the source database: ```bash psql $SOURCE <; GRANT SELECT ON ALL TABLES IN SCHEMA "public" TO ; ALTER DEFAULT PRIVILEGES IN SCHEMA "public" GRANT SELECT ON TABLES TO ; EOF ``` If the tables you are syncing are not in the `public` schema, grant the user permissions for each schema you are syncing: ```bash psql $SOURCE < TO ; GRANT SELECT ON ALL TABLES IN SCHEMA TO ; ALTER DEFAULT PRIVILEGES IN SCHEMA GRANT SELECT ON TABLES TO ; EOF ``` 5. On each table you want to sync, make `` the owner: ```bash psql $SOURCE -c 'ALTER TABLE
OWNER TO ;' ``` You can skip this step if the replicating user is already the owner of the tables. 3. **Enable replication DELETE and UPDATE operations** Replica identity assists data replication by identifying the rows being modified. Your options are that each table and hypertable in the source database should either have: * **A primary key**: data replication defaults to the primary key of the table being replicated. Nothing to do. * **A viable unique index**: each table has a unique, non-partial, non-deferrable index that includes only columns marked as `NOT NULL`. If a UNIQUE index does not exist, create one to assist the migration. You can delete it after migration. For each table, set `REPLICA IDENTITY` to the viable unique index: ``` psql -X -d $SOURCE -c 'ALTER TABLE
REPLICA IDENTITY USING INDEX <_index_name>' ``` * **No primary key or viable unique index**: use brute force. For each table, set `REPLICA IDENTITY` to `FULL`: ``` psql -X -d $SOURCE -c 'ALTER TABLE REPLICA IDENTITY FULL' ``` For each `UPDATE` or `DELETE` statement, PostgreSQL reads the whole table to find all matching rows. This results in significantly slower replication. If you are expecting a large number of `UPDATE` or `DELETE` operations on the table, best practice is to not use `FULL`. * From PostgreSQL 1. **Tune the Write Ahead Log (WAL) on the PostgreSQL source database** ```sql psql $SOURCE <`: ```sql psql $SOURCE -c "CREATE USER PASSWORD ''" ``` You can use an existing user. However, you must ensure that the user has the following permissions. 2. Grant permissions to create a replication slot: ```sql psql $SOURCE -c "ALTER ROLE REPLICATION" ``` 3. Grant permissions to create a publication: ```sql psql $SOURCE -c "GRANT CREATE ON DATABASE TO " ``` 4. Assign the user permissions on the source database: ```sql psql $SOURCE <; GRANT SELECT ON ALL TABLES IN SCHEMA "public" TO ; ALTER DEFAULT PRIVILEGES IN SCHEMA "public" GRANT SELECT ON TABLES TO ; EOF ``` If the tables you are syncing are not in the `public` schema, grant the user permissions for each schema you are syncing: ```sql psql $SOURCE < TO ; GRANT SELECT ON ALL TABLES IN SCHEMA TO ; ALTER DEFAULT PRIVILEGES IN SCHEMA GRANT SELECT ON TABLES TO ; EOF ``` 5. On each table you want to sync, make `` the owner: ```sql psql $SOURCE -c 'ALTER TABLE
OWNER TO ;' ``` You can skip this step if the replicating user is already the owner of the tables. 3. **Enable replication DELETE and UPDATE operations** Replica identity assists data replication by identifying the rows being modified. Your options are that each table and hypertable in the source database should either have: * **A primary key**: data replication defaults to the primary key of the table being replicated. Nothing to do. * **A viable unique index**: each table has a unique, non-partial, non-deferrable index that includes only columns marked as `NOT NULL`. If a UNIQUE index does not exist, create one to assist the migration. You can delete it after migration. For each table, set `REPLICA IDENTITY` to the viable unique index: ``` psql -X -d $SOURCE -c 'ALTER TABLE
REPLICA IDENTITY USING INDEX <_index_name>' ``` * **No primary key or viable unique index**: use brute force. For each table, set `REPLICA IDENTITY` to `FULL`: ``` psql -X -d $SOURCE -c 'ALTER TABLE REPLICA IDENTITY FULL' ``` For each `UPDATE` or `DELETE` statement, PostgreSQL reads the whole table to find all matching rows. This results in significantly slower replication. If you are expecting a large number of `UPDATE` or `DELETE` operations on the table, best practice is to not use `FULL`. ## Synchronize data to your Tiger Cloud service To sync data from your PostgreSQL database to your Tiger Cloud service using Tiger Console: 1. **Connect to your Tiger Cloud service** In [Tiger Console](https://console.cloud.tigerdata.com/dashboard/services), select the service to sync live data to. 2. **Prepare the source database** ![PostgreSQL connector wizard in Tiger Console](/docs/_astro/pg-connector-wizard-tiger-console.BPVoP0DK_Z1kktnR.webp) 1. Click `Connectors` > `Postgres`. 2. Set the name for the new connector by clicking the pencil icon. 3. Check the boxes for `Set wal_level to logical` and `Update your credentials`, then click `Continue`. 3. **Connect the source database and the target service** ![PostgreSQL connector connection string in Tiger Console](/docs/_astro/pg-connector-connection-string.DUeVjYuI_1Kk98E.webp) 1. Enter your database credentials or a PostgreSQL connection string. This is the connection string for [``](#tune-your-source-database). 2. **(Recommended for private databases)** If the source database is not reachable from the public Internet, toggle `Enable SSH tunneling` and route the connector through a bastion host: 1. Copy the public key shown in the wizard and append it to the `authorized_keys` file of the user the connector will log in as on the bastion host: ```bash echo "" >> ~/.ssh/authorized_keys # Verify cat ~/.ssh/authorized_keys ``` 2. Enter the bastion `Hostname`, `User`, and `Port` in the wizard. 3. Click `Connect to database`. Tiger Console connects to the source database and retrieves the schema information. 4. **Select the data to synchronize** ![Starting the PostgreSQL connector in Tiger Console](/docs/_astro/pg-connector-start-tiger-console.DPSXCOf0_AQ7bm.webp) Choose where the connector picks tables from: * **From an existing publication**: select a PostgreSQL `PUBLICATION` name, then pick schemas and tables from those it includes. Use this when you cannot grant the connector user `CREATE` privilege on the source — your DBA can pre-create the publication — or when you want to apply [row or column filters](https://www.postgresql.org/docs/current/logical-replication-row-filter.html) on published tables. * **Directly from the source**: select schemas, then pick the tables to sync from them. Tiger Console creates the publication for you. Click `Select tables +`. For each selected table, Tiger Console checks the schema and, if possible, suggests the column to use as the time dimension in a hypertable. 5. **Configure Initial Data Copy workers** ![Configure Initial Data Copy worker count in Tiger Console](/docs/_astro/pg-connector-configure-idc-worker-count.D6aj0UEK_Z12BfdB.webp) Specify the number of parallel connections — Initial Data Copy (IDC) workers — used to process the initial data copy. Higher values can speed up the initial copy but may increase load on the source database. Defaults to `4`. Tune this based on the resources available on the source database and the number of tables being synced. Large individual tables still copy through a single connection, so higher worker counts help most when you have many small to medium tables. Click `Create Connector`. Tiger Console starts the source PostgreSQL connector between the source database and the target service and displays the progress. 6. **Monitor and manage the connector** ![Editing a PostgreSQL connector in Tiger Console](/docs/_astro/edit-pg-connector-tiger-console.CAiEG08p_ZedUDY.webp) 1. To review the syncing progress for each table, click `Connectors` > `Source connectors`, then select the name of your connector in the table. 2. To edit the connector, click `Connectors` > `Source connectors`, then select the name of your connector in the table. You can rename the connector, delete or add new tables for syncing. 3. To pause a connector, click `Connectors` > `Source connectors`, then open the three-dot menu on the right and select `Pause`. 4. To delete a connector, click `Connectors` > `Source connectors`, then open the three-dot menu on the right and select `Delete`. You must pause the connector before deleting it. And that is it, you are using the source PostgreSQL connector to synchronize all the data, or specific tables, from a PostgreSQL database instance to your Tiger Cloud service, in real time. - Self-hosted PostgreSQL connector Updating parameters on a PostgreSQL instance will cause an outage. Choose a time that will cause the least issues to tune this database. 1. **Tune the Write Ahead Log (WAL) on the RDS/Aurora PostgreSQL source database** 1. In [RDS console](https://console.aws.amazon.com/rds/home#databases:), select the RDS instance to migrate. 2. Click `Configuration`, scroll down and note the `DB instance parameter group`, then click `Parameter Groups`. ![RDS parameter groups in the AWS console](/docs/_astro/awsrds-parameter-groups.CiXrFBVV_1Diea.webp) 3. Click `Create parameter group`, fill in the form with the following values, then click `Create`: * `Parameter group name`, whatever suits your fancy. * `Description`, knock yourself out with this one. * `Engine type`, `PostgreSQL` * `Parameter group family`, the same as `DB instance parameter group` in your `Configuration`. 4. In `Parameter groups`, select the parameter group you created, then click `Edit`. 5. Update the following parameters, then click `Save changes`: * `rds.logical_replication` set to `1`: record the information needed for logical decoding. * `wal_sender_timeout` set to `0`: disable the timeout for the sender process. 6. In RDS, navigate back to your [databases](https://console.aws.amazon.com/rds/home#databases:), select the RDS instance to migrate, and click `Modify`. 7. Scroll down to `Database options`, select your new parameter group, and click `Continue`. 8. Click `Apply immediately` or choose a maintenance window, then click `Modify DB instance`. Changing parameters will cause an outage. Wait for the database instance to reboot before continuing. After it comes back up, verify that the new settings are in effect on your database. 2. **Create a user for the source PostgreSQL connector and assign permissions** 1. Create ``: ```bash psql $SOURCE -c "CREATE USER PASSWORD ''" ``` You can use an existing user. However, you must ensure that the user has the following permissions. 2. Grant permissions to create a replication slot: ```bash psql $SOURCE -c "GRANT rds_replication TO " ``` 3. Grant permissions to create a publication: ```bash psql $SOURCE -c "GRANT CREATE ON DATABASE TO " ``` 4. Assign the user permissions on the source database: ```bash psql $SOURCE <; GRANT SELECT ON ALL TABLES IN SCHEMA "public" TO ; ALTER DEFAULT PRIVILEGES IN SCHEMA "public" GRANT SELECT ON TABLES TO ; EOF ``` If the tables you are syncing are not in the `public` schema, grant the user permissions for each schema you are syncing: ```bash psql $SOURCE < TO ; GRANT SELECT ON ALL TABLES IN SCHEMA TO ; ALTER DEFAULT PRIVILEGES IN SCHEMA GRANT SELECT ON TABLES TO ; EOF ``` 5. On each table you want to sync, make `` the owner: ```bash psql $SOURCE -c 'ALTER TABLE
OWNER TO ;' ``` You can skip this step if the replicating user is already the owner of the tables. 3. **Enable replication DELETE and UPDATE operations** Replica identity assists data replication by identifying the rows being modified. Your options are that each table and hypertable in the source database should either have: * **A primary key**: data replication defaults to the primary key of the table being replicated. Nothing to do. * **A viable unique index**: each table has a unique, non-partial, non-deferrable index that includes only columns marked as `NOT NULL`. If a UNIQUE index does not exist, create one to assist the migration. You can delete it after migration. For each table, set `REPLICA IDENTITY` to the viable unique index: ``` psql -X -d $SOURCE -c 'ALTER TABLE
REPLICA IDENTITY USING INDEX <_index_name>' ``` * **No primary key or viable unique index**: use brute force. For each table, set `REPLICA IDENTITY` to `FULL`: ``` psql -X -d $SOURCE -c 'ALTER TABLE REPLICA IDENTITY FULL' ``` For each `UPDATE` or `DELETE` statement, PostgreSQL reads the whole table to find all matching rows. This results in significantly slower replication. If you are expecting a large number of `UPDATE` or `DELETE` operations on the table, best practice is to not use `FULL`. - From AWS RDS/Aurora 1. **Tune the Write Ahead Log (WAL) on the PostgreSQL source database** ```sql psql $SOURCE <`: ```sql psql $SOURCE -c "CREATE USER PASSWORD ''" ``` You can use an existing user. However, you must ensure that the user has the following permissions. 2. Grant permissions to create a replication slot: ```sql psql $SOURCE -c "ALTER ROLE REPLICATION" ``` 3. Grant permissions to create a publication: ```sql psql $SOURCE -c "GRANT CREATE ON DATABASE TO " ``` 4. Assign the user permissions on the source database: ```sql psql $SOURCE <; GRANT SELECT ON ALL TABLES IN SCHEMA "public" TO ; ALTER DEFAULT PRIVILEGES IN SCHEMA "public" GRANT SELECT ON TABLES TO ; EOF ``` If the tables you are syncing are not in the `public` schema, grant the user permissions for each schema you are syncing: ```sql psql $SOURCE < TO ; GRANT SELECT ON ALL TABLES IN SCHEMA TO ; ALTER DEFAULT PRIVILEGES IN SCHEMA GRANT SELECT ON TABLES TO ; EOF ``` 5. On each table you want to sync, make `` the owner: ```sql psql $SOURCE -c 'ALTER TABLE
OWNER TO ;' ``` You can skip this step if the replicating user is already the owner of the tables. 3. **Enable replication DELETE and UPDATE operations** Replica identity assists data replication by identifying the rows being modified. Your options are that each table and hypertable in the source database should either have: * **A primary key**: data replication defaults to the primary key of the table being replicated. Nothing to do. * **A viable unique index**: each table has a unique, non-partial, non-deferrable index that includes only columns marked as `NOT NULL`. If a UNIQUE index does not exist, create one to assist the migration. You can delete it after migration. For each table, set `REPLICA IDENTITY` to the viable unique index: ``` psql -X -d $SOURCE -c 'ALTER TABLE
REPLICA IDENTITY USING INDEX <_index_name>' ``` * **No primary key or viable unique index**: use brute force. For each table, set `REPLICA IDENTITY` to `FULL`: ``` psql -X -d $SOURCE -c 'ALTER TABLE REPLICA IDENTITY FULL' ``` For each `UPDATE` or `DELETE` statement, PostgreSQL reads the whole table to find all matching rows. This results in significantly slower replication. If you are expecting a large number of `UPDATE` or `DELETE` operations on the table, best practice is to not use `FULL`. - From PostgreSQL ## Prerequisites for this integration guide To follow these steps, you'll need: * A target [Tiger Cloud service](/docs/get-started/quickstart/create-service). Best practice is to create a Tiger Cloud service with at least 8 CPUs for a smoother experience. A higher-spec instance can significantly reduce the overall migration window. * A migration machine to run the commands that move data from your source database to your target Tiger Cloud service. Best practice: use an [Ubuntu EC2 instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EC2_GetStarted.html#ec2-launch-instance) hosted in the same region as your Tiger Cloud service. * An [adjusted maintenance window](/docs/deploy/tiger-cloud/tiger-cloud-aws/upgrades#define-your-maintenance-window) to prevent maintenance from running during migration. - The source PostgreSQL instance and the target Tiger Cloud service must have the same extensions installed. The source PostgreSQL connector does not create extensions on the target. If the table uses column types from an extension, first create the extension on the target Tiger Cloud service before syncing the table. - [Docker](https://docs.docker.com/engine/install/) installed on your sync machine. For a better experience, use a 4 CPU/16GB EC2 instance or greater to run the source PostgreSQL connector. - The [PostgreSQL client tools](/docs/integrate/query-administration/psql) installed on your sync machine. This includes `psql`, `pg_dump`, `pg_dumpall`, and `vacuumdb` commands. ## Limitations * The schema is not migrated by the source PostgreSQL connector, you use `pg_dump`/`pg_restore` to migrate it. - Using TimescaleDB as the source has limited support (no CAGGs). - The source must be running PostgreSQL 13 or later. - Schema changes must be co-ordinated. Make compatible changes to the schema in your Tiger Cloud service first, then make the same changes to the source PostgreSQL instance. - Ensure that the source PostgreSQL instance and the target Tiger Cloud service have the same extensions installed. The source PostgreSQL connector does not create extensions on the target. If the table uses column types from an extension, first create the extension on the target Tiger Cloud service before syncing the table. - There is WAL volume growth on the source PostgreSQL instance during large table copy. - Continuous aggregate invalidation The connector uses `session_replication_role=replica` during data replication, which prevents table triggers from firing. This includes the internal triggers that mark continuous aggregates as invalid when underlying data changes. If you have continuous aggregates on your target database, they do not automatically refresh for data inserted during the migration. This limitation only applies to data below the continuous aggregate's materialization watermark. For example, backfilled data. New rows synced above the continuous aggregate watermark are used correctly when refreshing. This can lead to: * Missing data in continuous aggregates for the migration period. * Stale aggregate data. * Queries returning incomplete results. If the continuous aggregate exists in the source database, best practice is to add it to the PostgreSQL connector publication. If it only exists on the target database, manually refresh the continuous aggregate using the `force` option of [refresh\_continuous\_aggregate](/docs/reference/timescaledb/continuous-aggregates/refresh_continuous_aggregate#samples). ## Set your connection strings The `` in the `SOURCE` connection must have the replication role granted in order to create a replication slot. These variables hold the connection information for the source database and target Tiger Cloud service. In Terminal on your migration machine, set the following: ```bash export SOURCE="postgres://:@:/" export TARGET="postgres://tsdbadmin:@:/tsdb?sslmode=require" ``` You find the connection information for your Tiger Cloud service in the configuration file you downloaded when you created the service. Important Avoid using connection strings that route through connection poolers like PgBouncer or similar tools. This tool requires a direct connection to the database to function properly. ## Tune your source database * From AWS RDS/Aurora Updating parameters on a PostgreSQL instance will cause an outage. Choose a time that will cause the least issues to tune this database. 1. **Update the DB instance parameter group for your source database** 1. In , select the RDS instance to migrate. 2. Click `Configuration`, scroll down and note the `DB instance parameter group`, then click `Parameter groups` ![RDS parameter groups in the AWS console](/docs/_astro/awsrds-parameter-groups.CiXrFBVV_1Diea.webp) 3. Click `Create parameter group`, fill in the form with the following values, then click `Create`. * **Parameter group name** - whatever suits your fancy. * **Description** - knock yourself out with this one. * **Engine type** - `PostgreSQL` * **Parameter group family** - the same as `DB instance parameter group` in your `Configuration`. 4. In `Parameter groups`, select the parameter group you created, then click `Edit`. 5. Update the following parameters, then click `Save changes`. * `rds.logical_replication` set to `1`: record the information needed for logical decoding. * `wal_sender_timeout` set to `0`: disable the timeout for the sender process. 6. In RDS, navigate back to your [databases](https://console.aws.amazon.com/rds/home#databases:), select the RDS instance to migrate, and click `Modify`. 7. Scroll down to `Database options`, select your new parameter group, and click `Continue`. 8. Click `Apply immediately` or choose a maintenance window, then click `Modify DB instance`. Changing parameters will cause an outage. Wait for the database instance to reboot before continuing. 9. Verify that the settings are live in your database. 2. **Enable replication `DELETE` and `UPDATE` operations** Replica identity assists data replication by identifying the rows being modified. Your options are that each table and hypertable in the source database should either have: * **A primary key**: data replication defaults to the primary key of the table being replicated. Nothing to do. * **A viable unique index**: each table has a unique, non-partial, non-deferrable index that includes only columns marked as `NOT NULL`. If a UNIQUE index does not exist, create one to assist the migration. You can delete it after migration. For each table, set `REPLICA IDENTITY` to the viable unique index: ``` psql -X -d $SOURCE -c 'ALTER TABLE
REPLICA IDENTITY USING INDEX <_index_name>' ``` * **No primary key or viable unique index**: use brute force. For each table, set `REPLICA IDENTITY` to `FULL`: ``` psql -X -d $SOURCE -c 'ALTER TABLE REPLICA IDENTITY FULL' ``` For each `UPDATE` or `DELETE` statement, PostgreSQL reads the whole table to find all matching rows. This results in significantly slower replication. If you are expecting a large number of `UPDATE` or `DELETE` operations on the table, best practice is to not use `FULL`. * From PostgreSQL 1. **Tune the Write Ahead Log (WAL) on the PostgreSQL source database** ```sql psql $SOURCE <`: ```sql psql $SOURCE -c "CREATE USER PASSWORD ''" ``` You can use an existing user. However, you must ensure that the user has the following permissions. 2. Grant permissions to create a replication slot: ```sql psql $SOURCE -c "ALTER ROLE REPLICATION" ``` 3. Grant permissions to create a publication: ```sql psql $SOURCE -c "GRANT CREATE ON DATABASE TO " ``` 4. Assign the user permissions on the source database: ```sql psql $SOURCE <; GRANT SELECT ON ALL TABLES IN SCHEMA "public" TO ; ALTER DEFAULT PRIVILEGES IN SCHEMA "public" GRANT SELECT ON TABLES TO ; EOF ``` If the tables you are syncing are not in the `public` schema, grant the user permissions for each schema you are syncing: ```sql psql $SOURCE < TO ; GRANT SELECT ON ALL TABLES IN SCHEMA TO ; ALTER DEFAULT PRIVILEGES IN SCHEMA GRANT SELECT ON TABLES TO ; EOF ``` 5. On each table you want to sync, make `` the owner: ```sql psql $SOURCE -c 'ALTER TABLE
OWNER TO ;' ``` You can skip this step if the replicating user is already the owner of the tables. 3. **Enable replication DELETE and UPDATE operations** Replica identity assists data replication by identifying the rows being modified. Your options are that each table and hypertable in the source database should either have: * **A primary key**: data replication defaults to the primary key of the table being replicated. Nothing to do. * **A viable unique index**: each table has a unique, non-partial, non-deferrable index that includes only columns marked as `NOT NULL`. If a UNIQUE index does not exist, create one to assist the migration. You can delete it after migration. For each table, set `REPLICA IDENTITY` to the viable unique index: ``` psql -X -d $SOURCE -c 'ALTER TABLE
REPLICA IDENTITY USING INDEX <_index_name>' ``` * **No primary key or viable unique index**: use brute force. For each table, set `REPLICA IDENTITY` to `FULL`: ``` psql -X -d $SOURCE -c 'ALTER TABLE REPLICA IDENTITY FULL' ``` For each `UPDATE` or `DELETE` statement, PostgreSQL reads the whole table to find all matching rows. This results in significantly slower replication. If you are expecting a large number of `UPDATE` or `DELETE` operations on the table, best practice is to not use `FULL`. ## Migrate the table schema to the Tiger Cloud service Use `pg_dump` to: 1. **Download the schema from the source database** ```shell pg_dump $SOURCE \ --no-privileges \ --no-owner \ --no-publications \ --no-subscriptions \ --no-table-access-method \ --no-tablespaces \ --schema-only \ --file=schema.sql ``` 2. **Apply the schema on the target service** ```shell psql $TARGET -f schema.sql ``` ## Convert partitions and tables with time-series data into hypertables For efficient querying and analysis, you can convert tables which contain time-series or events data, and tables that are already partitioned using PostgreSQL declarative partition into [hypertables](/docs/learn/hypertables/understand-hypertables). 1. **Convert tables to hypertables** Run the following on each table in the target Tiger Cloud service to convert it to a hypertable: ```shell psql -X -d $TARGET -c "SELECT public.create_hypertable('
', by_range('', ''::interval));" ``` For example, to convert the *metrics* table into a hypertable with *time* as a partition column and *1 day* as a partition interval: ```shell psql -X -d $TARGET -c "SELECT public.create_hypertable('public.metrics', by_range('time', '1 day'::interval));" ``` 2. **Convert PostgreSQL partitions to hypertables** Rename the partition and create a new regular table with the same name as the partitioned table, then convert to a hypertable: ```shell psql $TARGET -f - <<'EOF' BEGIN; ALTER TABLE public.events RENAME TO events_part; CREATE TABLE public.events(LIKE public.events_part INCLUDING ALL); SELECT create_hypertable('public.events', by_range('time', '1 day'::interval)); COMMIT; EOF ``` ## Specify the tables to synchronize After the schema is migrated, you [`CREATE PUBLICATION`](https://www.postgresql.org/docs/current/sql-createpublication.html) on the source database that specifies the tables to synchronize. 1. **Create a publication that specifies the table to synchronize** A `PUBLICATION` enables you to synchronize some or all the tables in the schema or database. ```sql CREATE PUBLICATION FOR TABLE , ; ``` To add tables after to an existing publication, use [ALTER PUBLICATION](https://www.postgresql.org/docs/current/sql-alterpublication.html)\*\* ```sql ALTER PUBLICATION ADD TABLE ; ``` 2. **Publish the PostgreSQL declarative partitioned table** Leave `publish_via_partition_root` at its default (`false`). Since live-sync v0.23.0, this copies the partitioned table's leaf partitions in parallel and makes the initial copy resumable, instead of streaming the whole table through a single connection: ```sql ALTER PUBLICATION SET(publish_via_partition_root=false); ``` Only set it to `true` if you need the older serial, whole-table copy: ```sql ALTER PUBLICATION SET(publish_via_partition_root=true); ``` To convert a partitioned table to a hypertable, follow [Convert partitions and tables with time-series data into hypertables](#convert-partitions-and-tables-with-time-series-data-into-hypertables). 3. **Stop syncing a table in the `PUBLICATION`, use `DROP TABLE`** ```sql ALTER PUBLICATION DROP TABLE ; ``` ## Synchronize data to your Tiger Cloud service You use the source PostgreSQL connector docker image to synchronize changes in real time from a PostgreSQL database instance to a Tiger Cloud service: 1. **Start the source PostgreSQL connector** As you run the source PostgreSQL connector continuously, best practice is to run it as a Docker daemon. ```shell docker run -d --rm --name livesync timescale/live-sync:latest run \ --publication --subscription \ --source $SOURCE --target $TARGET --table-map ``` * `version-tag`: The latest available version tag of the live-sync image. See [Docker Hub](https://hub.docker.com/r/timescale/live-sync). * `--publication`: The name of the publication as you created in the previous step. To use multiple publications, repeat the `--publication` flag. * `--subscription`: The name that identifies the subscription on the target Tiger Cloud service. * `--source`: The connection string to the source PostgreSQL database. * `--target`: The connection string to the target Tiger Cloud service. * `--table-map`: (Optional) A JSON string that maps source tables to target tables. If not provided, the source and target table names are assumed to be the same. For example, to map the source table `metrics` to the target table `metrics_data`: `--table-map '{"source": {"schema": "public", "table": "metrics"}, "target": {"schema": "public", "table": "metrics_data"}}'` To map only the schema, use: `--table-map '{"source": {"schema": "public"}, "target": {"schema": "analytics"}}'` This flag can be repeated for multiple table mappings. * `--table-sync-workers`: (Optional) The number of parallel workers to use for initial table sync. Default is 4. * `--copy-data`: (Optional) By default, the initial table data is copied from source to target before starting logical replication. Set to `false` so only changes made after replication slot creation are replicated. Best practice is to set to `false` during dry-run livesync so you do not copy table data. 2. **Capture logs** Once the source PostgreSQL connector is running as a docker daemon, you can also capture the logs: ```shell docker logs -f livesync ``` 3. **View the progress of tables being synchronized** List the tables being synchronized by the source PostgreSQL connector using the `_ts_live_sync.subscription_rel` table in the target Tiger Cloud service: ```bash psql $TARGET -c "SELECT * FROM _ts_live_sync.subscription_rel" ``` You see something like the following: ```txt subname | pubname | schemaname | tablename | rrelid | state | lsn | updated_at | last_error | created_at | rows_copied | approximate_rows | bytes_copied | approximate_size | target_schema | target_table ----------+-----------+------------+-----------+--------+-------+------------+-------------------------------+------------+-------------------------------+-------------+------------------+--------------+------------------+---------------+------------- livesync | analytics | public | metrics | 20856 | r | 6/1A8CBA48 | 2025-06-24 06:16:21.434898+00 | | 2025-06-24 06:03:58.172946+00 | 18225440 | 18225440 | 1387359359 | 1387359359 | public | metrics ``` The `state` column indicates the current state of the table synchronization. Possible values for `state` are: | state | description | | ----- | ----------------------------------------------------------------------- | | i | initial state, table data sync not started | | d | initial table data sync is in progress | | f | initial table data sync completed, catching up with incremental changes | | s | synchronized, waiting for the main apply worker to take over | | r | table is ready, applying changes in real-time | To see the replication lag, run the following against the SOURCE database: ```bash psql $SOURCE -f - <<'EOF' SELECT slot_name, pg_size_pretty(pg_current_wal_flush_lsn() - confirmed_flush_lsn) AS lag FROM pg_replication_slots WHERE slot_name LIKE 'live_sync_%' AND slot_type = 'logical' EOF ``` 4. **Add or remove tables from the publication** To add tables, use [ALTER PUBLICATION .. ADD TABLE](https://www.postgresql.org/docs/current/sql-alterpublication.html)\*\* ```sql ALTER PUBLICATION ADD TABLE ; ``` To remove tables, use [ALTER PUBLICATION .. DROP TABLE](https://www.postgresql.org/docs/current/sql-alterpublication.html)\*\* ```sql ALTER PUBLICATION DROP TABLE ; ``` 5. **Update table statistics** If you have a large table, you can run `ANALYZE` on the target Tiger Cloud service to update the table statistics after the initial sync is complete. This helps the query planner make better decisions for query execution plans. ```bash vacuumdb --analyze --verbose --dbname=$TARGET ``` 6. **Stop the source PostgreSQL connector** ```shell docker stop live-sync ``` 7. **(Optional) Reset sequence nextval on the target Tiger Cloud service** The source PostgreSQL connector does not automatically reset the sequence nextval on the target Tiger Cloud service. Run the following script to reset the sequence for all tables that have a serial or identity column in the target Tiger Cloud service: ```bash psql $TARGET -f - <<'EOF' DO $$ DECLARE rec RECORD; BEGIN FOR rec IN ( SELECT sr.target_schema AS table_schema, sr.target_table AS table_name, col.column_name, pg_get_serial_sequence( sr.target_schema || '.' || sr.target_table, col.column_name ) AS seqname FROM _ts_live_sync.subscription_rel AS sr JOIN information_schema.columns AS col ON col.table_schema = sr.target_schema AND col.table_name = sr.target_table WHERE col.column_default LIKE 'nextval(%' -- only serial/identity columns ) LOOP EXECUTE format( 'SELECT setval(%L, COALESCE((SELECT MAX(%I) FROM %I.%I), 0) + 1, false );', rec.seqname, -- the sequence identifier rec.column_name, -- the column to MAX() rec.table_schema, -- schema for MAX() rec.table_name -- table for MAX() ); END LOOP; END; $$ LANGUAGE plpgsql; EOF ``` 8. **Clean up** When you are permanently done syncing (for example, you have fully cut over to Tiger Cloud, or you want to restart the sync from scratch), use the `drop` sub-command to remove the replication slot from the source and all subscription state from the target: ```shell docker run -it --rm --name livesync timescale/live-sync:latest drop \ --subscription --source $SOURCE --target $TARGET ``` `drop` removes: * The replication slot (`live_sync_`) on the **source** database. * The subscription and internal tracking tables (`_ts_live_sync.*`) on the **target** Tiger Cloud service. Warning Run `drop` only when you intend to permanently stop or fully restart the sync. Dropping the replication slot while the source is still active causes WAL to be retained until the slot is recreated, which can grow disk usage on the source. - From AWS RDS/Aurora Updating parameters on a PostgreSQL instance will cause an outage. Choose a time that will cause the least issues to tune this database. 1. **Update the DB instance parameter group for your source database** 1. In , select the RDS instance to migrate. 2. Click `Configuration`, scroll down and note the `DB instance parameter group`, then click `Parameter groups` ![RDS parameter groups in the AWS console](/docs/_astro/awsrds-parameter-groups.CiXrFBVV_1Diea.webp) 3. Click `Create parameter group`, fill in the form with the following values, then click `Create`. * **Parameter group name** - whatever suits your fancy. * **Description** - knock yourself out with this one. * **Engine type** - `PostgreSQL` * **Parameter group family** - the same as `DB instance parameter group` in your `Configuration`. 4. In `Parameter groups`, select the parameter group you created, then click `Edit`. 5. Update the following parameters, then click `Save changes`. * `rds.logical_replication` set to `1`: record the information needed for logical decoding. * `wal_sender_timeout` set to `0`: disable the timeout for the sender process. 6. In RDS, navigate back to your [databases](https://console.aws.amazon.com/rds/home#databases:), select the RDS instance to migrate, and click `Modify`. 7. Scroll down to `Database options`, select your new parameter group, and click `Continue`. 8. Click `Apply immediately` or choose a maintenance window, then click `Modify DB instance`. Changing parameters will cause an outage. Wait for the database instance to reboot before continuing. 9. Verify that the settings are live in your database. 2. **Enable replication `DELETE` and `UPDATE` operations** Replica identity assists data replication by identifying the rows being modified. Your options are that each table and hypertable in the source database should either have: * **A primary key**: data replication defaults to the primary key of the table being replicated. Nothing to do. * **A viable unique index**: each table has a unique, non-partial, non-deferrable index that includes only columns marked as `NOT NULL`. If a UNIQUE index does not exist, create one to assist the migration. You can delete it after migration. For each table, set `REPLICA IDENTITY` to the viable unique index: ``` psql -X -d $SOURCE -c 'ALTER TABLE
REPLICA IDENTITY USING INDEX <_index_name>' ``` * **No primary key or viable unique index**: use brute force. For each table, set `REPLICA IDENTITY` to `FULL`: ``` psql -X -d $SOURCE -c 'ALTER TABLE REPLICA IDENTITY FULL' ``` For each `UPDATE` or `DELETE` statement, PostgreSQL reads the whole table to find all matching rows. This results in significantly slower replication. If you are expecting a large number of `UPDATE` or `DELETE` operations on the table, best practice is to not use `FULL`. - From PostgreSQL 1. **Tune the Write Ahead Log (WAL) on the PostgreSQL source database** ```sql psql $SOURCE <`: ```sql psql $SOURCE -c "CREATE USER PASSWORD ''" ``` You can use an existing user. However, you must ensure that the user has the following permissions. 2. Grant permissions to create a replication slot: ```sql psql $SOURCE -c "ALTER ROLE REPLICATION" ``` 3. Grant permissions to create a publication: ```sql psql $SOURCE -c "GRANT CREATE ON DATABASE TO " ``` 4. Assign the user permissions on the source database: ```sql psql $SOURCE <; GRANT SELECT ON ALL TABLES IN SCHEMA "public" TO ; ALTER DEFAULT PRIVILEGES IN SCHEMA "public" GRANT SELECT ON TABLES TO ; EOF ``` If the tables you are syncing are not in the `public` schema, grant the user permissions for each schema you are syncing: ```sql psql $SOURCE < TO ; GRANT SELECT ON ALL TABLES IN SCHEMA TO ; ALTER DEFAULT PRIVILEGES IN SCHEMA GRANT SELECT ON TABLES TO ; EOF ``` 5. On each table you want to sync, make `` the owner: ```sql psql $SOURCE -c 'ALTER TABLE
OWNER TO ;' ``` You can skip this step if the replicating user is already the owner of the tables. 3. **Enable replication DELETE and UPDATE operations** Replica identity assists data replication by identifying the rows being modified. Your options are that each table and hypertable in the source database should either have: * **A primary key**: data replication defaults to the primary key of the table being replicated. Nothing to do. * **A viable unique index**: each table has a unique, non-partial, non-deferrable index that includes only columns marked as `NOT NULL`. If a UNIQUE index does not exist, create one to assist the migration. You can delete it after migration. For each table, set `REPLICA IDENTITY` to the viable unique index: ``` psql -X -d $SOURCE -c 'ALTER TABLE
REPLICA IDENTITY USING INDEX <_index_name>' ``` * **No primary key or viable unique index**: use brute force. For each table, set `REPLICA IDENTITY` to `FULL`: ``` psql -X -d $SOURCE -c 'ALTER TABLE REPLICA IDENTITY FULL' ``` For each `UPDATE` or `DELETE` statement, PostgreSQL reads the whole table to find all matching rows. This results in significantly slower replication. If you are expecting a large number of `UPDATE` or `DELETE` operations on the table, best practice is to not use `FULL`. # Sync data from S3 > Sync data from S3 to your Tiger Cloud service in real time You use the source S3 connector in Tiger Cloud to synchronize CSV and Parquet files from an S3 bucket to your Tiger Cloud service in real time. The connector runs continuously, enabling you to leverage Tiger Cloud as your analytics database with data constantly synced from S3. This lets you take full advantage of Tiger Cloud's real-time analytics capabilities without having to develop or manage custom ETL solutions between S3 and Tiger Cloud. ![Connectors overview in Tiger Console](/docs/_astro/tiger-console-connector-overview.C3brL-kO_Z2bCaMM.webp) You can use the source S3 connector to synchronize your existing and new data. Here's what the connector can do: * Sync data from an S3 bucket instance to a Tiger Cloud service: * Use glob patterns to identify the objects to sync. * Watch an S3 bucket for new files and import them automatically. It runs on a configurable schedule and tracks processed files. * **Important**: the connector processes files in [lexicographical order](https://en.wikipedia.org/wiki/Lexicographic_order). It uses the name of the last file processed as a marker and fetches only files later in the alphabet in subsequent queries. Files added with names earlier in the alphabet than the marker are skipped and never synced. For example, if you add the file Bob when the marker is at Elephant, Bob is never processed. * For large backlogs, check every minute until caught up. * Sync data from multiple file formats: * CSV: check for compression in GZ and ZIP format, then process using [timescaledb-parallel-copy](https://github.com/timescale/timescaledb-parallel-copy). * Parquet: convert to CSV, then process using [timescaledb-parallel-copy](https://github.com/timescale/timescaledb-parallel-copy). * The source S3 connector offers an option to enable a [hypertable](/docs/learn/hypertables/understand-hypertables) during the file-to-table schema mapping setup. You can enable [columnstore](/docs/learn/columnar-storage/understand-hypercore) and [continuous aggregates](/docs/learn/continuous-aggregates) through the SQL editor once the connector has started running. * The connector offers a default 1-minute polling interval. This means that Tiger Cloud checks the S3 source every minute for new data. You can customize this interval by setting up a cron expression. The Source S3 connector continuously imports data from an Amazon S3 bucket into your database. It monitors your S3 bucket for new files matching a specified pattern and automatically imports them into your designated database table. Note The connector currently only syncs existing and new files. It does not support updating or deleting records based on updates and deletes from S3 to tables in a Tiger Cloud service. If you have any questions or feedback about the source S3 connector, join us in the [Tiger Data community](https://app.slack.com/client/T4GT3N2JK/C086NU9EZ88). ## Prerequisites for this integration guide To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service). - Your [connection details](/docs/integrate/find-connection-details). * Access to a standard Amazon S3 bucket containing your data files. Directory buckets are not supported. * Access credentials for the S3 bucket configured. The following credentials are supported: * [IAM Role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user.html#roles-creatingrole-user-console). * Configure the trust policy. Set the: * `Principal`: `arn:aws:iam::142548018081:role/timescale-s3-connections`. * `ExternalID`: set to the [Tiger Cloud project and Tiger Cloud service ID](/docs/integrate/find-connection-details#find-your-project-and-service-id) of the service you are syncing to in the format `/`. This is to avoid the [confused deputy problem](https://docs.aws.amazon.com/IAM/latest/UserGuide/confused-deputy.html). * Give the following access permissions: * `s3:GetObject`. * `s3:ListBucket`. * [Public anonymous user](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html#example-bucket-policies-anonymous-user). Note This feature is currently not supported for Tiger Cloud on Microsoft Azure. ## Limitations [Section titled “Limitations”](#limitations) * **File naming**: Files must follow lexicographical ordering conventions. Files with names that sort earlier than already-processed files are permanently skipped. Example: if `file_2024_01_15.csv` has been processed, a file named `file_2024_01_10.csv` added later will never be synced. Recommended naming patterns: timestamps (for example, `YYYY-MM-DD-HHMMSS`), sequential numbers with fixed padding (for example, `file_00001`, `file_00002`). * **CSV**: * Maximum file size: 1 GB To increase this limit, contact sales\@tigerdata.com * Maximum row size: 2 MB * Supported compressed formats: * GZ * ZIP * Advanced settings: * Delimiter: the default character is `,`, you can choose a different delimiter * Skip header: skip the first row if your file has headers * **Parquet**: * Maximum file size: 1 GB * Maximum row size: 2 MB * **Sync iteration**: To prevent system overload, the connector tracks up to 100 files for each sync iteration. Additional checks only fill empty queue slots. ## Synchronize data to your service [Section titled “Synchronize data to your service”](#synchronize-data-to-your-service) To sync data from your S3 bucket to your Tiger Cloud service using Tiger Console: 1. **Connect to your Tiger Cloud service** In [Tiger Console](https://console.cloud.tigerdata.com/dashboard/services), select the service to sync live data to. 2. **Connect the source S3 bucket to the target service** ![Connecting Tiger Cloud to an S3 bucket](/docs/_astro/s3-connector-tiger-console.DDuCo9oV_ZBF0v1.webp) 1. Click `Connectors` > `Amazon S3`. 2. Click the pencil icon, then set the name for the new connector. 3. Set the `Bucket name` and `Authentication method`, then click `Continue`. For instructions on creating the IAM role to connect your S3 bucket, click `Learn how`. Tiger Console connects to the source bucket. 4. In `Define files to sync`, choose the `File type` and set the `Glob pattern`. Use the following patterns: * `/*`: match all files in a folder. Also, any pattern ending with `/` is treated as `/*`. * `/**`: match all recursively. * `/**/*.csv`: match a specific file type. The source S3 connector uses prefix filters where possible, place patterns carefully at the end of your glob expression. AWS S3 doesn't support complex filtering. If your expression filters too many files, the list operation may time out. 5. Click the search icon. You see the files to sync. Click `Continue`. 3. **Optimize the data to synchronize in hypertables** ![S3 connector table selection in Tiger Console](/docs/_astro/tiger-console-s3-connector-create-tables.CWZ2J2w5_Z1XzU94.webp) Tiger Console checks the file schema and, if possible, suggests the column to use as the time dimension in a [hypertable](/docs/learn/hypertables/understand-hypertables). 1. Choose `Create a new table for your data` or `Ingest data to an existing table`. 2. Choose the `Data type` for each column, then click `Continue`. 3. Configure the insert behavior when there is a conflict, then click `Continue`. 4. Choose the polling interval. This can be a minute, an hour, or a [cron expression](https://en.wikipedia.org/wiki/Cron#Cron_expression). 5. Click `Start Connector`. Tiger Console starts the connection between the source database and the target service and displays the progress. 4. **Monitor synchronization** The Source S3 connector provides comprehensive observability that gives you maximum visibility on connector performance. This includes summarized insights into connector state, quick actions, filtering and search to easily find specific files, and detailed lifecycle tracking as each file is imported. 1. To view the amount of data replicated, click `Connectors`. The diagram in `CONNECTOR DATA FLOW` shows the connectors you have created, their status, and how much data has been replicated. ![Connectors overview in Tiger Console](/docs/_astro/tiger-console-connector-overview.C3brL-kO_Z2bCaMM.webp) 2. To view file import statistics and logs, click `Connectors` > `Source connectors`, then select the name of your connector in the table. ![S3 connector import details and statistics](/docs/_astro/tiger-console-s3-connector-import-details.DbkXHMp-_1BMtgk.webp) The connector dashboard displays all imports at a glance. Use this page to: * `Search by file name`: find specific files from the list of imports * `Filter by status`: filter files based on their current status: * `All statuses`: all files * `Cancelled`: files where import is aborted * `Failure`: files where an error occurred during import * `In Queue`: files that are awaiting processing * `Paused`: files where processing is on hold * `Pending Retry`: files that are re-queued for processing * `Running`: files currently being imported * `Success`: files that have been imported * `Bulk retry`: retry importing all files with the `Error` status * `Lifecycle history`: view detailed information for all imports, and time spent in each status * `Refresh every minute`: enable auto-refresh 5. **Manage the connector** 1. To pause the connector, click `Connectors` > `Source connectors`. Open the three-dot menu next to your connector in the table, then click `Pause`. ![Pausing an S3 connector in Tiger Console](/docs/_astro/tiger-console-s3-connector-pause.C5OYyjFX_1JBHaD.webp) 2. To edit the connector, click `Connectors` > `Source connectors`. Open the three-dot menu next to your connector in the table, then click `Edit`. Select `Connector settings`. You must pause the connector before editing it. ![Editing S3 connector settings in Tiger Console](/docs/_astro/tiger-console-s3-connector-edit.Dwm-LblS_XvOVP.webp) 3. To pause or delete the connector, click `Connectors` > `Source connectors`, then open the three-dot menu on the right and select an option. You must pause the connector before deleting it. And that is it, you are using the source S3 connector to synchronize all the data, or specific files, from an S3 bucket to your Tiger Cloud service in real time. # Data engineering & ETL > Use your ETL tools and data pipelines with Tiger Data The guides below show how to connect ETL (extract, transform, load) and data pipeline tools to Tiger Data products, enabling you to move, transform, and load data from sources like Kafka, Airflow, or serverless functions into Tiger Data for analytics and time-series workloads. ## Integrations [Section titled “Integrations”](#integrations) [Amazon SageMaker](/docs/integrate/data-engineering-etl/amazon-sagemaker) [Build, train, and deploy ML models with time-series data storage and analysis](/docs/integrate/data-engineering-etl/amazon-sagemaker) [Apache Airflow](/docs/integrate/data-engineering-etl/apache-airflow) [Author, schedule, and monitor workflows to orchestrate your data pipelines](/docs/integrate/data-engineering-etl/apache-airflow) [Apache Kafka](/docs/integrate/data-engineering-etl/apache-kafka) [Stream, process, and analyze real-time event data from Apache Kafka topics](/docs/integrate/data-engineering-etl/apache-kafka) [AWS Lambda](/docs/integrate/data-engineering-etl/aws-lambda) [Run serverless code to process and store your data without managing infrastructure](/docs/integrate/data-engineering-etl/aws-lambda) [Debezium](/docs/integrate/data-engineering-etl/debezium) [Capture and stream database changes in real time for event-driven architecture](/docs/integrate/data-engineering-etl/debezium) [Decodable](/docs/integrate/data-engineering-etl/decodable) [Build and manage real-time data pipelines to stream and process your data effortlessly](/docs/integrate/data-engineering-etl/decodable) [Supabase](/docs/integrate/data-engineering-etl/supabase) [Run real-time analytical queries across databases using foreign data wrappers](/docs/integrate/data-engineering-etl/supabase) # Integrate Amazon SageMaker with Tiger Cloud > Build, train, and deploy ML models with time-series data storage and analysis [Amazon SageMaker AI](https://docs.aws.amazon.com/sagemaker/latest/dg/whatis.html) is a fully managed machine learning (ML) service. With SageMaker AI, data scientists and developers can quickly and confidently build, train, and deploy ML models into a production-ready hosted environment. This page shows you how to integrate Amazon SageMaker with a Tiger Cloud service. ## Prerequisites for this integration guide To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service). These steps use Tiger Cloud, but the same approach applies to a [self-hosted TimescaleDB](/docs/get-started/choose-your-path/install-timescaledb) instance. - Your [connection details](/docs/integrate/find-connection-details). * An [AWS Account](https://signin.aws.amazon.com/signup?request_type=register) ## Prepare your service to ingest data from SageMaker [Section titled “Prepare your service to ingest data from SageMaker”](#prepare-your-service-to-ingest-data-from-sagemaker) Create a table in Tiger Cloud service to store model predictions generated by SageMaker. 1. **Connect to your Tiger Cloud service** For Tiger Cloud, open an [SQL editor](/docs/build/data-management/run-queries-from-tiger-console) in [Tiger Console](https://console.cloud.tigerdata.com/dashboard/services). For self-hosted TimescaleDB, use [`psql`](/docs/integrate/query-administration/psql). 2. **For better performance and easier real-time analytics, create a hypertable** [Hypertables](/docs/learn/hypertables/understand-hypertables) are PostgreSQL tables that automatically partition your data by time. You interact with hypertables in the same way as regular PostgreSQL tables, but with extra features that makes managing your time-series data much easier. ```sql CREATE TABLE model_predictions ( time TIMESTAMPTZ NOT NULL, model_name TEXT NOT NULL, prediction DOUBLE PRECISION NOT NULL ) WITH ( tsdb.hypertable ); ``` When you create a hypertable using [CREATE TABLE ... WITH ...](/docs/reference/timescaledb/hypertables/create_table), the default partitioning column is automatically the first column with a timestamp data type. Also, TimescaleDB creates a [columnstore policy](/docs/reference/timescaledb/hypercore/add_columnstore_policy) that automatically converts your data to the columnstore, after an interval equal to the value of the [chunk\_interval](/docs/reference/timescaledb/hypertables/set_chunk_time_interval), defined through `after` in the policy. This columnar format enables fast scanning and aggregation, optimizing performance for analytical workloads while also saving significant storage space. In the columnstore conversion, hypertable chunks are compressed by up to 98%, and organized for efficient, large-scale queries. You can customize this policy later using [alter\_job](/docs/reference/timescaledb/jobs-automation/alter_job). However, to change `after` or `created_before`, the compression settings, or the hypertable the policy is acting on, you must [remove the columnstore policy](/docs/reference/timescaledb/hypercore/remove_columnstore_policy) and [add a new one](/docs/reference/timescaledb/hypercore/add_columnstore_policy). You can also manually [convert chunks](/docs/reference/timescaledb/hypercore/convert_to_columnstore) in a hypertable to the columnstore. ## Create the code to inject data into a service [Section titled “Create the code to inject data into a service”](#create-the-code-to-inject-data-into-a-service) 1. **Create a SageMaker Notebook instance** 1. In [Amazon SageMaker > Notebooks and Git repos](https://console.aws.amazon.com/sagemaker/home#/notebooks-and-git-repos), click `Create Notebook instance`. 2. Follow the wizard to create a default Notebook instance. 2. **Write a Notebook script that inserts data into your Tiger Cloud service** 1. When your Notebook instance is `inService,` click `Open JupyterLab` and click `conda_python3`. 2. Update the following script with your [connection details](/docs/integrate/find-connection-details), then paste it in the Notebook. ```python import psycopg2 from datetime import datetime def insert_prediction(model_name, prediction, host, port, user, password, dbname): conn = psycopg2.connect( host=host, port=port, user=user, password=password, dbname=dbname ) cursor = conn.cursor() query = """ INSERT INTO model_predictions (time, model_name, prediction) VALUES (%s, %s, %s); """ values = (datetime.utcnow(), model_name, prediction) cursor.execute(query, values) conn.commit() cursor.close() conn.close() # Example usage insert_prediction( model_name="example_model", prediction=0.95, host="", port="", user="", password="", dbname="" ) ``` 3. **Test your SageMaker script** 1. Run the script in your SageMaker notebook. 2. Verify that the data is in your service. Open an [SQL editor](/docs/build/data-management/run-queries-from-tiger-console) and check the `sensor_data` table: ```sql SELECT * FROM model_predictions; ``` You see something like: | time | model\_name | prediction | | ----------------------------- | --------------------- | ---------- | | 2025-02-06 16:56:34.370316+00 | timescale-cloud-model | 0.95 | Now you can seamlessly integrate Amazon SageMaker with Tiger Cloud to store and analyze time-series data generated by machine learning models. You can also integrate visualization tools like [Grafana](/docs/integrate/observability-alerting/grafana) or [Tableau](/docs/integrate/bi-vizualization/tableau) with Tiger Cloud to create real-time dashboards of your model predictions. # Integrate Apache Airflow with Tiger Cloud > Author, schedule, and monitor workflows to orchestrate your data pipelines A [DAG (Directed Acyclic Graph)](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dags.html#dags) is the core concept of Airflow, collecting [Tasks](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/tasks.html) together, organized with dependencies and relationships to say how they should run. You declare a DAG in a Python file in the `$AIRFLOW_HOME/dags` folder of your Airflow instance. This page shows you how to use a Python connector in a DAG to integrate Apache Airflow with a Tiger Cloud service. ## Prerequisites for this integration guide To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service). These steps use Tiger Cloud, but the same approach applies to a [self-hosted TimescaleDB](/docs/get-started/choose-your-path/install-timescaledb) instance. - Your [connection details](/docs/integrate/find-connection-details). * [Python3 and pip3](https://docs.python.org/3/using/index.html) installed. * [Apache Airflow](https://airflow.apache.org/docs/apache-airflow/stable/start.html) installed. Ensure that your Airflow instance has network access to Tiger Cloud. This example DAG uses the `company` table you create in [Optimize time-series data in hypertables](/docs/learn/tiger-cloud/tiger-cloud-essentials#optimize-time-series-data-in-hypertables-with-hypercore) ## Install python connectivity libraries [Section titled “Install python connectivity libraries”](#install-python-connectivity-libraries) To install the Python libraries required to connect to Tiger Cloud: 1. **Enable PostgreSQL connections between Airflow and Tiger Cloud** ```bash pip install psycopg2-binary ``` 2. **Enable PostgreSQL connection types in the Airflow UI** ```bash pip install apache-airflow-providers-postgres ``` ## Create a connection between Airflow and your service [Section titled “Create a connection between Airflow and your service”](#create-a-connection-between-airflow-and-your-service) In your Airflow instance, securely connect to your Tiger Cloud service: 1. **Run Airflow** On your development machine, run the following command: ```bash airflow standalone ``` The username and password for Airflow UI are displayed in the `standalone | Login with username` line in the output. 2. **Add a connection from Airflow to your Tiger Cloud service** 1. In your browser, navigate to `localhost:8080`, then select `Admin` > `Connections`. 2. Click `+` (Add a new record), then use your [connection info](/docs/integrate/find-connection-details) to fill in the form. The `Connection Type` is `Postgres`. ## Exchange data between Airflow and your service [Section titled “Exchange data between Airflow and your service”](#exchange-data-between-airflow-and-your-service) To exchange data between Airflow and your Tiger Cloud service: 1. **Create and execute a DAG** To insert data in your Tiger Cloud service from Airflow: 1. In `$AIRFLOW_HOME/dags/timescale_dag.py`, add the following code: ```python from airflow import DAG from airflow.operators.python_operator import PythonOperator from airflow.hooks.postgres_hook import PostgresHook from datetime import datetime def insert_data_to_timescale(): hook = PostgresHook(postgres_conn_id='the ID of the connenction you created') conn = hook.get_conn() cursor = conn.cursor() """ This could be any query. This example inserts data into the table you create in: https://www.tigerdata.com/docs/getting-started/latest/try-key-features-timescale-products/#optimize-time-series-data-in-hypertables-with-hypercore """ cursor.execute("INSERT INTO crypto_assets (symbol, name) VALUES (%s, %s)", ('NEW/Asset','New Asset Name')) conn.commit() cursor.close() conn.close() default_args = { 'owner': 'airflow', 'start_date': datetime(2023, 1, 1), 'retries': 1, } dag = DAG('timescale_dag', default_args=default_args, schedule_interval='@daily') insert_task = PythonOperator( task_id='insert_data', python_callable=insert_data_to_timescale, dag=dag, ) ``` This DAG uses the `company` table created in [Create regular PostgreSQL tables for relational data](/docs/learn/tiger-cloud/tiger-cloud-essentials#optimize-time-series-data-in-hypertables-with-hypercore). 2. In your browser, refresh the Airflow UI. 3. In `Search DAGS`, type `timescale_dag` and press ENTER. 4. Press the play icon and trigger the DAG: ![Apache Airflow DAG results showing daily ETH transaction volume](/docs/_astro/integrations-apache-airflow.CuAODU3S_Ro4x1.webp) 2. **Verify that the data appears in Tiger Cloud** 1. In [Tiger Console](https://console.cloud.tigerdata.com/), navigate to your service and click `SQL editor`. 2. Run a query to view your data. For example: `SELECT symbol, name FROM company;`. You see the new rows inserted in the table. You have successfully integrated Apache Airflow with Tiger Cloud and created a data pipeline. # Integrate Apache Kafka with Tiger Cloud > Stream, process, and analyze real-time event data from Apache Kafka topics [Apache Kafka](https://kafka.apache.org/documentation/) is a distributed event streaming platform used for high-performance data pipelines, streaming analytics, and data integration. [Apache Kafka Connect](https://docs.confluent.io/platform/current/connect/index.html) is a tool to scalably and reliably stream data between Apache Kafka® and other data systems. Kafka Connect is an ecosystem of pre-written and maintained Kafka Producers (source connectors) and Kafka Consumers (sink connectors) for data products and platforms like databases and message brokers. This guide explains how to set up Kafka and Kafka Connect to stream data from a Kafka topic into your Tiger Cloud service. Note This page covers streaming from a self-managed Apache Kafka cluster using Kafka Connect and the Camel PostgreSQL sink. If your Kafka cluster runs on Confluent Cloud, use the fully managed [Apache Kafka source connector](/docs/integrate/connectors/source/sync-from-kafka) in Tiger Console instead. ## Prerequisites for this integration guide To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service). These steps use Tiger Cloud, but the same approach applies to a [self-hosted TimescaleDB](/docs/get-started/choose-your-path/install-timescaledb) instance. - Your [connection details](/docs/integrate/find-connection-details). * [Java8 or higher](https://www.oracle.com/java/technologies/downloads/) installed to run Apache Kafka. ## Install and configure Apache Kafka [Section titled “Install and configure Apache Kafka”](#install-and-configure-apache-kafka) To install and configure Apache Kafka: 1. **Extract the Kafka binaries to a local folder** ```bash curl https://dlcdn.apache.org/kafka/3.9.0/kafka_2.13-3.9.0.tgz | tar -xzf - cd kafka_2.13-3.9.0 ``` From now on, the folder where you extracted the Kafka binaries is called ``. 2. **Configure and run Apache Kafka** ```bash KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)" ./bin/kafka-storage.sh format --standalone -t $KAFKA_CLUSTER_ID -c config/kraft/reconfig-server.properties ./bin/kafka-server-start.sh config/kraft/reconfig-server.properties ``` Use the `-daemon` flag to run this process in the background. 3. **Create Kafka topics** In another terminal window, navigate to ``, then call `kafka-topics.sh` and create the following topics: * `accounts`: publishes JSON messages that are consumed by the timescale-sink connector and inserted into your Tiger Cloud service. * `deadletter`: stores messages that cause errors and that Kafka Connect workers cannot process. ```bash ./bin/kafka-topics.sh \ --create \ --topic accounts \ --bootstrap-server localhost:9092 \ --partitions 10 ./bin/kafka-topics.sh \ --create \ --topic deadletter \ --bootstrap-server localhost:9092 \ --partitions 10 ``` 4. **Test that your topics are working correctly** 1. Run `kafka-console-producer` to send messages to the `accounts` topic: ```bash bin/kafka-console-producer.sh --topic accounts --bootstrap-server localhost:9092 ``` 2. Send some events. For example, type the following: ```bash >Tiger Cloud >How Cool ``` 3. In another terminal window, navigate to ``, then run `kafka-console-consumer` to consume the events you just sent: ```bash bin/kafka-console-consumer.sh --topic accounts --from-beginning --bootstrap-server localhost:9092 ``` You see: ```bash Tiger Cloud How Cool ``` Keep these terminals open, you use them to test the integration later. ## Install the sink connector to communicate with Tiger Cloud [Section titled “Install the sink connector to communicate with Tiger Cloud”](#install-the-sink-connector-to-communicate-with-tiger-cloud) To set up Kafka Connect server, plugins, drivers, and connectors: 1. **Install the PostgreSQL connector** In another terminal window, navigate to ``, then download and configure the PostgreSQL sink and driver. ```bash mkdir -p "plugins/camel-postgresql-sink-kafka-connector" curl https://repo.maven.apache.org/maven2/org/apache/camel/kafkaconnector/camel-postgresql-sink-kafka-connector/3.21.0/camel-postgresql-sink-kafka-connector-3.21.0-package.tar.gz \ | tar -xzf - -C "plugins/camel-postgresql-sink-kafka-connector" --strip-components=1 curl -H "Accept: application/zip" https://jdbc.postgresql.org/download/postgresql-42.7.5.jar -o "plugins/camel-postgresql-sink-kafka-connector/postgresql-42.7.5.jar" echo "plugin.path=`pwd`/plugins/camel-postgresql-sink-kafka-connector" >> "config/connect-distributed.properties" echo "plugin.path=`pwd`/plugins/camel-postgresql-sink-kafka-connector" >> "config/connect-standalone.properties" ``` 2. **Start Kafka Connect** ```bash export CLASSPATH=`pwd`/plugins/camel-postgresql-sink-kafka-connector/* ./bin/connect-standalone.sh config/connect-standalone.properties ``` Use the `-daemon` flag to run this process in the background. 3. **Verify Kafka Connect is running** In yet another another terminal window, run the following command: ```bash curl http://localhost:8083 ``` You see something like: ```bash {"version":"3.9.0","commit":"a60e31147e6b01ee","kafka_cluster_id":"J-iy4IGXTbmiALHwPZEZ-A"} ``` ## Create a table in your service to ingest Kafka events [Section titled “Create a table in your service to ingest Kafka events”](#create-a-table-in-your-service-to-ingest-kafka-events) To prepare your Tiger Cloud service for Kafka integration: 1. **Connect to your Tiger Cloud service** [Connect](/docs/build/data-management/run-queries-from-tiger-console) to your Tiger Cloud service. 2. **Create a hypertable to ingest Kafka events** ```sql CREATE TABLE accounts ( created_at TIMESTAMPTZ DEFAULT NOW(), name TEXT, city TEXT ) WITH ( tsdb.hypertable ); ``` When you create a hypertable using [CREATE TABLE ... WITH ...](/docs/reference/timescaledb/hypertables/create_table), the default partitioning column is automatically the first column with a timestamp data type. Also, TimescaleDB creates a [columnstore policy](/docs/reference/timescaledb/hypercore/add_columnstore_policy) that automatically converts your data to the columnstore, after an interval equal to the value of the [chunk\_interval](/docs/reference/timescaledb/hypertables/set_chunk_time_interval), defined through `after` in the policy. This columnar format enables fast scanning and aggregation, optimizing performance for analytical workloads while also saving significant storage space. In the columnstore conversion, hypertable chunks are compressed by up to 98%, and organized for efficient, large-scale queries. You can customize this policy later using [alter\_job](/docs/reference/timescaledb/jobs-automation/alter_job). However, to change `after` or `created_before`, the compression settings, or the hypertable the policy is acting on, you must [remove the columnstore policy](/docs/reference/timescaledb/hypercore/remove_columnstore_policy) and [add a new one](/docs/reference/timescaledb/hypercore/add_columnstore_policy). You can also manually [convert chunks](/docs/reference/timescaledb/hypercore/convert_to_columnstore) in a hypertable to the columnstore. ## Create the Tiger Cloud sink [Section titled “Create the Tiger Cloud sink”](#create-the-tiger-cloud-sink) To create a Tiger Cloud sink in Apache Kafka: 1. **Create the connection configuration** 1. In the terminal running Kafka Connect, stop the process by pressing `Ctrl+C`. 2. Write the following configuration to `/config/timescale-standalone-sink.properties`, then update the `` with your [connection details](/docs/integrate/find-connection-details). ```properties name=timescale-standalone-sink connector.class=org.apache.camel.kafkaconnector.postgresqlsink.CamelPostgresqlsinkSinkConnector errors.tolerance=all errors.deadletterqueue.topic.name=deadletter tasks.max=10 value.converter=org.apache.kafka.connect.storage.StringConverter key.converter=org.apache.kafka.connect.storage.StringConverter topics=accounts camel.kamelet.postgresql-sink.databaseName= camel.kamelet.postgresql-sink.username= camel.kamelet.postgresql-sink.password= camel.kamelet.postgresql-sink.serverName= camel.kamelet.postgresql-sink.serverPort= camel.kamelet.postgresql-sink.query=INSERT INTO accounts (name,city) VALUES (:#name,:#city) ``` 3. Restart Kafka Connect with the new configuration: ```bash export CLASSPATH=`pwd`/plugins/camel-postgresql-sink-kafka-connector/* ./bin/connect-standalone.sh config/connect-standalone.properties config/timescale-standalone-sink.properties ``` 2. **Test the connection** To see your sink, query the `/connectors` route in a GET request: ```bash curl -X GET http://localhost:8083/connectors ``` You see: ```bash #["timescale-standalone-sink"] ``` ## Test the integration with Tiger Cloud [Section titled “Test the integration with Tiger Cloud”](#test-the-integration-with-tiger-cloud) To test this integration, send some messages onto the `accounts` topic. You can do this using the kafkacat or kcat utility. 1. **In the terminal running `kafka-console-producer.sh` enter the following json strings** ```bash {"name":"Lola","city":"Copacabana"} {"name":"Holly","city":"Miami"} {"name":"Jolene","city":"Tennessee"} {"name":"Barbara Ann ","city":"California"} ``` Look in your terminal running `kafka-console-consumer` to see the messages being processed. 2. **Query your Tiger Cloud service for all rows in the `accounts` table** ```sql SELECT * FROM accounts; ``` You see something like: | created\_at | name | city | | ----------------------------- | ----------- | ---------- | | 2025-02-18 13:55:05.147261+00 | Lola | Copacabana | | 2025-02-18 13:55:05.216673+00 | Holly | Miami | | 2025-02-18 13:55:05.283549+00 | Jolene | Tennessee | | 2025-02-18 13:55:05.35226+00 | Barbara Ann | California | You have successfully integrated Apache Kafka with Tiger Cloud. # Integrate AWS Lambda with Tiger Cloud > Run serverless code to process and store your data without managing infrastructure [AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html) is a serverless computing service provided by Amazon Web Services (AWS) that allows you to run code without provisioning or managing servers, scaling automatically as needed. This page shows you how to integrate AWS Lambda with Tiger Cloud service to process and store time-series data efficiently. ## Prerequisites for this integration guide To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service). These steps use Tiger Cloud, but the same approach applies to a [self-hosted TimescaleDB](/docs/get-started/choose-your-path/install-timescaledb) instance. - Your [connection details](/docs/integrate/find-connection-details). * An [AWS Account](https://signin.aws.amazon.com/signup?request_type=register). * [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) installed and configured. * [NodeJS v18.x or later](https://nodejs.org/en/download) installed. ## Prepare your service to ingest data from AWS Lambda [Section titled “Prepare your service to ingest data from AWS Lambda”](#prepare-your-service-to-ingest-data-from-aws-lambda) Create a table in Tiger Cloud service to store time-series data. 1. **Connect to your Tiger Cloud service** For Tiger Cloud, open an [SQL editor](/docs/build/data-management/run-queries-from-tiger-console) in [Tiger Console](https://console.cloud.tigerdata.com/dashboard/services). For self-hosted TimescaleDB, use [`psql`](/docs/integrate/query-administration/psql). 2. **Create a hypertable to store sensor data** [Hypertables](/docs/learn/hypertables/understand-hypertables) are PostgreSQL tables that automatically partition your data by time. You interact with hypertables in the same way as regular PostgreSQL tables, but with extra features that make managing your time-series data much easier. ```sql CREATE TABLE sensor_data ( time TIMESTAMPTZ NOT NULL, sensor_id TEXT NOT NULL, value DOUBLE PRECISION NOT NULL ) WITH ( tsdb.hypertable ); ``` When you create a hypertable using [CREATE TABLE ... WITH ...](/docs/reference/timescaledb/hypertables/create_table), the default partitioning column is automatically the first column with a timestamp data type. Also, TimescaleDB creates a [columnstore policy](/docs/reference/timescaledb/hypercore/add_columnstore_policy) that automatically converts your data to the columnstore, after an interval equal to the value of the [chunk\_interval](/docs/reference/timescaledb/hypertables/set_chunk_time_interval), defined through `after` in the policy. This columnar format enables fast scanning and aggregation, optimizing performance for analytical workloads while also saving significant storage space. In the columnstore conversion, hypertable chunks are compressed by up to 98%, and organized for efficient, large-scale queries. You can customize this policy later using [alter\_job](/docs/reference/timescaledb/jobs-automation/alter_job). However, to change `after` or `created_before`, the compression settings, or the hypertable the policy is acting on, you must [remove the columnstore policy](/docs/reference/timescaledb/hypercore/remove_columnstore_policy) and [add a new one](/docs/reference/timescaledb/hypercore/add_columnstore_policy). You can also manually [convert chunks](/docs/reference/timescaledb/hypercore/convert_to_columnstore) in a hypertable to the columnstore. ## Create the code to inject data into a service [Section titled “Create the code to inject data into a service”](#create-the-code-to-inject-data-into-a-service) Write an AWS Lambda function in a Node.js project that processes and inserts time-series data into a Tiger Cloud service. 1. **Initialize a new Node.js project to hold your Lambda function** ```shell mkdir lambda-timescale && cd lambda-timescale npm init -y ``` 2. **Install the PostgreSQL client library in your project** ```shell npm install pg ``` 3. **Write a Lambda Function that inserts data into your Tiger Cloud service** Create a file named `index.js`, then add the following code: ```javascript const { Client } = require('pg'); exports.handler = async (event) => { const client = new Client({ host: process.env.TIMESCALE_HOST, port: process.env.TIMESCALE_PORT, user: process.env.TIMESCALE_USER, password: process.env.TIMESCALE_PASSWORD, database: process.env.TIMESCALE_DB, }); try { await client.connect(); // const query = ` INSERT INTO sensor_data (time, sensor_id, value) VALUES ($1, $2, $3); `; const data = JSON.parse(event.body); const values = [new Date(), data.sensor_id, data.value]; await client.query(query, values); return { statusCode: 200, body: JSON.stringify({ message: 'Data inserted successfully!' }), }; } catch (error) { console.error('Error inserting data:', error); return { statusCode: 500, body: JSON.stringify({ error: 'Failed to insert data.' }), }; } finally { await client.end(); } }; ``` ## Deploy your Node project to AWS Lambda [Section titled “Deploy your Node project to AWS Lambda”](#deploy-your-node-project-to-aws-lambda) To create an AWS Lambda function that injects data into your Tiger Cloud service: 1. **Compress your code into a `.zip`** ```shell zip -r lambda-timescale.zip . ``` 2. **Deploy to AWS Lambda** In the following example, replace `` with your [AWS IAM credentials](https://docs.aws.amazon.com/IAM/latest/UserGuide/access-keys-admin-managed.html#admin-list-access-key), then use AWS CLI to create a Lambda function for your project: ```shell aws lambda create-function \ --function-name TimescaleIntegration \ --runtime nodejs14.x \ --role \ --handler index.handler \ --zip-file fileb://lambda-timescale.zip ``` 3. **Set up environment variables** In the following example, use your [connection details](/docs/integrate/find-connection-details) to add your Tiger Cloud service connection settings to your Lambda function: ```shell aws lambda update-function-configuration \ --function-name TimescaleIntegration \ --environment "Variables={TIMESCALE_HOST=,TIMESCALE_PORT=, \ TIMESCALE_USER=,TIMESCALE_PASSWORD=, \ TIMESCALE_DB=}" ``` 4. **Test your AWS Lambda function** 1. Invoke the Lambda function and send some data to your Tiger Cloud service: ```shell aws lambda invoke \ --function-name TimescaleIntegration \ --payload '{"body": "{\"sensor_id\": \"sensor-123\", \"value\": 42.5}"}' \ --cli-binary-format raw-in-base64-out \ response.json ``` 2. Verify that the data is in your service. Open an [SQL editor](/docs/build/data-management/run-queries-from-tiger-console) and check the `sensor_data` table: ```sql SELECT * FROM sensor_data; ``` You see something like: | time | sensor\_id | value | | ----------------------------- | ---------- | ----- | | 2025-02-10 10:58:45.134912+00 | sensor-123 | 42.5 | You can now seamlessly ingest time-series data from AWS Lambda into Tiger Cloud. # Integrate Debezium with TimescaleDB > Capture and stream database changes in real time for event-driven architecture [Debezium](https://debezium.io/) is an open-source distributed platform for change data capture (CDC). It enables you to capture changes in a self-hosted TimescaleDB instance and stream them to other systems in real time. Debezium can capture events about: * [Hypertables](/docs/learn/hypertables/understand-hypertables): captured events are rerouted from their chunk-specific topics to a single logical topic named according to the following pattern: `..` * [Continuous aggregates](/docs/learn/continuous-aggregates): captured events are rerouted from their chunk-specific topics to a single logical topic named according to the following pattern: `..` * [Hypercore](/docs/learn/columnar-storage/understand-hypercore): If you enable hypercore, the Debezium TimescaleDB connector does not apply any special processing to data in the columnstore. Compressed chunks are forwarded unchanged to the next downstream job in the pipeline for further processing as needed. Typically, messages with compressed chunks are dropped, and are not processed by subsequent jobs in the pipeline. This limitation only affects changes to chunks in the columnstore. Changes to data in the rowstore work correctly. This page explains how to capture changes in your database and stream them using Debezium on Apache Kafka. ## Prerequisites for this integration guide To follow these steps, you'll need: * A [self-hosted TimescaleDB](/docs/get-started/choose-your-path/install-timescaledb) instance. - Your [connection details](/docs/integrate/find-connection-details). * [Docker](https://docs.docker.com/engine/install/) installed on your development machine. ## Configure your database to work with Debezium [Section titled “Configure your database to work with Debezium”](#configure-your-database-to-work-with-debezium) To set up self-hosted TimescaleDB to communicate with Debezium: 1. **Configure your self-hosted TimescaleDB deployment** 1. Open `postgresql.conf`. The PostgreSQL configuration files are usually located in: * Docker: `/home/postgres/pgdata/data/` * Linux: `/etc/postgresql//main/` or `/var/lib/pgsql//data/` * MacOS: `/opt/homebrew/var/postgresql@/` * Windows: `C:\Program Files\PostgreSQL\\data\` 2. Enable logical replication. Modify the following settings in `postgresql.conf`: ```ini wal_level = logical max_replication_slots = 10 max_wal_senders = 10 ``` 3. Open `pg_hba.conf` and enable host replication. To allow replication connections, add the following: ```plaintext local replication debezium trust ``` This permission is for the `debezium` PostgreSQL user running on a local or Docker deployment. For more about replication permissions, see [Configuring PostgreSQL to allow replication with the Debezium connector host](https://debezium.io/documentation/reference/3.2/connectors/postgresql.html#postgresql-host-replication-permissions). 4. Restart PostgreSQL. 2. **Connect to your self-hosted TimescaleDB instance** Use [`psql`](/docs/integrate/query-administration/psql#connect-to-your-service). 3. **Create a Debezium user in PostgreSQL** Create a user with the `LOGIN` and `REPLICATION` permissions: ```sql CREATE ROLE debezium WITH LOGIN REPLICATION PASSWORD ''; ``` 4. **Enable a replication spot for Debezium** 1. Create a hypertable for Debezium to listen to: ```sql CREATE TABLE accounts ( created_at TIMESTAMPTZ DEFAULT NOW(), name TEXT, city TEXT ) WITH (tsdb.hypertable); ``` Debezium also works with [continuous aggregates](/docs/build/continuous-aggregates/create-a-continuous-aggregate). 2. Create a publication and enable a replication slot: ```sql CREATE PUBLICATION dbz_publication FOR ALL TABLES WITH (publish = 'insert, update'); ``` ## Configure Debezium to work with your database [Section titled “Configure Debezium to work with your database”](#configure-debezium-to-work-with-your-database) Set up Kafka Connect server, plugins, drivers, and connectors: 1. **Run Zookeeper in Docker** In another Terminal window, run the following command: ```bash docker run -it --rm --name zookeeper -p 2181:2181 -p 2888:2888 -p 3888:3888 quay.io/debezium/zookeeper:3.0 ``` Check the output log to see that zookeeper is running. 2. **Run Kafka in Docker** In another Terminal window, run the following command: ```bash docker run -it --rm --name kafka -p 9092:9092 --link zookeeper:zookeeper quay.io/debezium/kafka:3.0 ``` Check the output log to see that Kafka is running. 3. **Run Kafka Connect in Docker** In another Terminal window, run the following command: ```bash docker run -it --rm --name connect \ -p 8083:8083 \ -e GROUP_ID=1 \ -e CONFIG_STORAGE_TOPIC=accounts \ -e OFFSET_STORAGE_TOPIC=offsets \ -e STATUS_STORAGE_TOPIC=storage \ --link kafka:kafka \ --link timescaledb:timescaledb \ quay.io/debezium/connect:3.0 ``` Check the output log to see that Kafka Connect is running. 4. **Register the Debezium PostgreSQL source connector** Update the `` for the `` you created in your self-hosted TimescaleDB instance in the following command. Then run the command in another Terminal window: ```bash curl -X POST http://localhost:8083/connectors \ -H "Content-Type: application/json" \ -d '{ "name": "timescaledb-connector", "config": { "connector.class": "io.debezium.connector.postgresql.PostgresConnector", "database.hostname": "timescaledb", "database.port": "5432", "database.user": "", "database.password": "", "database.dbname" : "postgres", "topic.prefix": "accounts", "plugin.name": "pgoutput", "schema.include.list": "public,_timescaledb_internal", "transforms": "timescaledb", "transforms.timescaledb.type": "io.debezium.connector.postgresql.transforms.timescaledb.TimescaleDb", "transforms.timescaledb.database.hostname": "timescaledb", "transforms.timescaledb.database.port": "5432", "transforms.timescaledb.database.user": "", "transforms.timescaledb.database.password": "", "transforms.timescaledb.database.dbname": "postgres" } }' ``` 5. **Verify `timescaledb-source-connector` is included in the connector list** Check the tasks associated with `timescaledb-connector`: ```bash curl -i -X GET -H "Accept:application/json" localhost:8083/connectors/timescaledb-connector ``` You see something like: ```bash {"name":"timescaledb-connector","config": { "connector.class":"io.debezium.connector.postgresql.PostgresConnector", "transforms.timescaledb.database.hostname":"timescaledb", "transforms.timescaledb.database.password":"debeziumpassword","database.user":"debezium", "database.dbname":"postgres","transforms.timescaledb.database.dbname":"postgres", "transforms.timescaledb.database.user":"debezium", "transforms.timescaledb.type":"io.debezium.connector.postgresql.transforms.timescaledb.TimescaleDb", "transforms.timescaledb.database.port":"5432","transforms":"timescaledb", "schema.include.list":"public,_timescaledb_internal","database.port":"5432","plugin.name":"pgoutput", "topic.prefix":"accounts","database.hostname":"timescaledb","database.password":"debeziumpassword", "name":"timescaledb-connector"},"tasks":[{"connector":"timescaledb-connector","task":0}],"type":"source"} ``` 6. **Verify `timescaledb-connector` is running** 1. Open the Terminal window running Kafka Connect. When the connector is active, you see something like the following: ```bash 2025-04-30 10:40:15,168 INFO Postgres|accounts|streaming REPLICA IDENTITY for '_timescaledb_internal._hyper_1_1_chunk' is 'DEFAULT'; UPDATE and DELETE events will contain previous values only for PK columns [io.debezium.connector.postgresql.PostgresSchema] 2025-04-30 10:40:15,168 INFO Postgres|accounts|streaming REPLICA IDENTITY for '_timescaledb_internal.bgw_job_stat' is 'DEFAULT'; UPDATE and DELETE events will contain previous values only for PK columns [io.debezium.connector.postgresql.PostgresSchema] 2025-04-30 10:40:15,175 INFO Postgres|accounts|streaming SignalProcessor started. Scheduling it every 5000ms [io.debezium.pipeline.signal.SignalProcessor] 2025-04-30 10:40:15,175 INFO Postgres|accounts|streaming Creating thread debezium-postgresconnector-accounts-SignalProcessor [io.debezium.util.Threads] 2025-04-30 10:40:15,175 INFO Postgres|accounts|streaming Starting streaming [io.debezium.pipeline.ChangeEventSourceCoordinator] 2025-04-30 10:40:15,176 INFO Postgres|accounts|streaming Retrieved latest position from stored offset 'LSN{0/1FCE570}' [io.debezium.connector.postgresql.PostgresStreamingChangeEventSource] 2025-04-30 10:40:15,176 INFO Postgres|accounts|streaming Looking for WAL restart position for last commit LSN 'null' and last change LSN 'LSN{0/1FCE570}' [io.debezium.connector.postgresql.connection.WalPositionLocator] 2025-04-30 10:40:15,176 INFO Postgres|accounts|streaming Initializing PgOutput logical decoder publication [io.debezium.connector.postgresql.connection.PostgresReplicationConnection] 2025-04-30 10:40:15,189 INFO Postgres|accounts|streaming Obtained valid replication slot ReplicationSlot [active=false, latestFlushedLsn=LSN{0/1FCCFF0}, catalogXmin=884] [io.debezium.connector.postgresql.connection.PostgresConnection] 2025-04-30 10:40:15,189 INFO Postgres|accounts|streaming Connection gracefully closed [io.debezium.jdbc.JdbcConnection] 2025-04-30 10:40:15,204 INFO Postgres|accounts|streaming Requested thread factory for component PostgresConnector, id = accounts named = keep-alive [io.debezium.util.Threads] 2025-04-30 10:40:15,204 INFO Postgres|accounts|streaming Creating thread debezium-postgresconnector-accounts-keep-alive [io.debezium.util.Threads] 2025-04-30 10:40:15,216 INFO Postgres|accounts|streaming REPLICA IDENTITY for '_timescaledb_internal.bgw_policy_chunk_stats' is 'DEFAULT'; UPDATE and DELETE events will contain previous values only for PK columns [io.debezium.connector.postgresql.PostgresSchema] 2025-04-30 10:40:15,216 INFO Postgres|accounts|streaming REPLICA IDENTITY for 'public.accounts' is 'DEFAULT'; UPDATE and DELETE events will contain previous values only for PK columns [io.debezium.connector.postgresql.PostgresSchema] 2025-04-30 10:40:15,217 INFO Postgres|accounts|streaming REPLICA IDENTITY for '_timescaledb_internal.bgw_job_stat_history' is 'DEFAULT'; UPDATE and DELETE events will contain previous values only for PK columns [io.debezium.connector.postgresql.PostgresSchema] 2025-04-30 10:40:15,217 INFO Postgres|accounts|streaming REPLICA IDENTITY for '_timescaledb_internal._hyper_1_1_chunk' is 'DEFAULT'; UPDATE and DELETE events will contain previous values only for PK columns [io.debezium.connector.postgresql.PostgresSchema] 2025-04-30 10:40:15,217 INFO Postgres|accounts|streaming REPLICA IDENTITY for '_timescaledb_internal.bgw_job_stat' is 'DEFAULT'; UPDATE and DELETE events will contain previous values only for PK columns [io.debezium.connector.postgresql.PostgresSchema] 2025-04-30 10:40:15,219 INFO Postgres|accounts|streaming Processing messages [io.debezium.connector.postgresql.PostgresStreamingChangeEventSource] ``` 2. Watch the events in the accounts topic on your self-hosted TimescaleDB instance. In another Terminal instance, run the following command: ```bash docker run -it --rm --name watcher --link zookeeper:zookeeper --link kafka:kafka quay.io/debezium/kafka:3.0 watch-topic -a -k accounts ``` You see the topics being streamed. For example: ```bash status-task-timescaledb-connector-0 {"state":"RUNNING","trace":null,"worker_id":"172.17.0.5:8083","generation":31} status-topic-timescaledb.public.accounts:connector-timescaledb-connector {"topic":{"name":"timescaledb.public.accounts","connector":"timescaledb-connector","task":0,"discoverTimestamp":1746009337985}} status-topic-accounts._timescaledb_internal.bgw_job_stat:connector-timescaledb-connector {"topic":{"name":"accounts._timescaledb_internal.bgw_job_stat","connector":"timescaledb-connector","task":0,"discoverTimestamp":1746009338118}} status-topic-accounts._timescaledb_internal.bgw_job_stat:connector-timescaledb-connector {"topic":{"name":"accounts._timescaledb_internal.bgw_job_stat","connector":"timescaledb-connector","task":0,"discoverTimestamp":1746009338120}} status-topic-accounts._timescaledb_internal.bgw_job_stat_history:connector-timescaledb-connector {"topic":{"name":"accounts._timescaledb_internal.bgw_job_stat_history","connector":"timescaledb-connector","task":0,"discoverTimestamp":1746009338243}} status-topic-accounts._timescaledb_internal.bgw_job_stat_history:connector-timescaledb-connector {"topic":{"name":"accounts._timescaledb_internal.bgw_job_stat_history","connector":"timescaledb-connector","task":0,"discoverTimestamp":1746009338245}} status-topic-accounts.public.accounts:connector-timescaledb-connector {"topic":{"name":"accounts.public.accounts","connector":"timescaledb-connector","task":0,"discoverTimestamp":1746009338250}} status-topic-accounts.public.accounts:connector-timescaledb-connector {"topic":{"name":"accounts.public.accounts","connector":"timescaledb-connector","task":0,"discoverTimestamp":1746009338251}} status-topic-accounts.public.accounts:connector-timescaledb-connector {"topic":{"name":"accounts.public.accounts","connector":"timescaledb-connector","task":0,"discoverTimestamp":1746009338251}} status-topic-accounts.public.accounts:connector-timescaledb-connector {"topic":{"name":"accounts.public.accounts","connector":"timescaledb-connector","task":0,"discoverTimestamp":1746009338251}} status-topic-accounts.public.accounts:connector-timescaledb-connector {"topic":{"name":"accounts.public.accounts","connector":"timescaledb-connector","task":0,"discoverTimestamp":1746009338251}} ["timescaledb-connector",{"server":"accounts"}] {"last_snapshot_record":true,"lsn":33351024,"txId":893,"ts_usec":1746009337290783,"snapshot":"INITIAL","snapshot_completed":true} status-connector-timescaledb-connector {"state":"UNASSIGNED","trace":null,"worker_id":"172.17.0.5:8083","generation":31} status-task-timescaledb-connector-0 {"state":"UNASSIGNED","trace":null,"worker_id":"172.17.0.5:8083","generation":31} status-connector-timescaledb-connector {"state":"RUNNING","trace":null,"worker_id":"172.17.0.5:8083","generation":33} status-task-timescaledb-connector-0 {"state":"RUNNING","trace":null,"worker_id":"172.17.0.5:8083","generation":33} ``` And that is it, you have configured Debezium to interact with TimescaleDB. # Integrate Decodable with Tiger Cloud > Build and manage real-time data pipelines to stream and process your data effortlessly [Decodable](https://www.decodable.co/) is a real-time data platform that allows you to build, run, and manage data pipelines effortlessly. ![Decodable pipeline configuration for Tiger Cloud](/docs/_astro/integrations-decodable-configuration.Ckuc85vw_Z22skHG.webp) This page explains how to integrate Decodable with your Tiger Cloud service to enable efficient real-time streaming and analytics. ## Prerequisites for this integration guide To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service). These steps use Tiger Cloud, but the same approach applies to a [self-hosted TimescaleDB](/docs/get-started/choose-your-path/install-timescaledb) instance. - Your [connection details](/docs/integrate/find-connection-details). * A [Decodable](https://auth.decodable.co/u/signup/) account. This page uses the pipeline you create using the [Decodable Quickstart Guide](https://docs.decodable.co/get-started/quickstart.html). ## Connect Decodable to your service [Section titled “Connect Decodable to your service”](#connect-decodable-to-your-service) To stream data gathered in Decodable to a Tiger Cloud service: 1. **Create the sync to pipe a Decodable data stream into your Tiger Cloud service** 1. Log in to your [Decodable account](https://app.decodable.co/-/accounts). 2. Click `Connections`, then click `New Connection`. 3. Select a `PostgreSQL sink` connection type, then click `Connect`. 4. Using your [connection details](/docs/integrate/find-connection-details), fill in the connection information. Leave `schema` and `JDBC options` empty. 5. Select the `http_events` source stream, then click `Next`. Decodable creates the table in your Tiger Cloud service and starts streaming data. 2. **Test the connection** 1. Connect to your Tiger Cloud service. For Tiger Cloud, open an [SQL editor](/docs/build/data-management/run-queries-from-tiger-console) in [Tiger Console](https://console.cloud.tigerdata.com/dashboard/services). For self-hosted TimescaleDB, use [`psql`](/docs/integrate/query-administration/psql). 2. Check the data from Decodable is streaming into your Tiger Cloud service. ```sql SELECT * FROM http_events; ``` You see something like: ![Decodable data visible in a Tiger Cloud service](/docs/_astro/integrations-decodable-data-in-service.De8QoA8y_Z1By9xe.webp) You have successfully integrated Decodable with Tiger Cloud. # Integrate Supabase with Tiger Cloud > Run real-time analytical queries across databases using foreign data wrappers [Supabase](https://supabase.com/) is an open source Firebase alternative. This page shows how to run real-time analytical queries against a Tiger Cloud service through Supabase using a foreign data wrapper (fdw) to bring aggregated data from your Tiger Cloud service. ## Prerequisites for this integration guide To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service). These steps use Tiger Cloud, but the same approach applies to a [self-hosted TimescaleDB](/docs/get-started/choose-your-path/install-timescaledb) instance. - Your [connection details](/docs/integrate/find-connection-details). * A [Supabase project](https://supabase.com/dashboard/new) ## Set up your service [Section titled “Set up your service”](#set-up-your-service) To set up a Tiger Cloud service optimized for analytics to receive data from Supabase: 1. **Optimize time-series data in hypertables** Time-series data represents how a system, process, or behavior changes over time. [Hypertables](/docs/learn/hypertables/understand-hypertables) are PostgreSQL tables that help you improve insert and query performance by automatically partitioning your data by time. [Connect to your Tiger Cloud service](/docs/build/data-management/run-queries-from-tiger-console) and create a table that will point to a Supabase database: ```sql CREATE TABLE signs ( time timestamptz NOT NULL DEFAULT now(), origin_time timestamptz NOT NULL, name TEXT ) WITH ( tsdb.hypertable ); ``` When you create a hypertable using [CREATE TABLE ... WITH ...](/docs/reference/timescaledb/hypertables/create_table), the default partitioning column is automatically the first column with a timestamp data type. Also, TimescaleDB creates a [columnstore policy](/docs/reference/timescaledb/hypercore/add_columnstore_policy) that automatically converts your data to the columnstore, after an interval equal to the value of the [chunk\_interval](/docs/reference/timescaledb/hypertables/set_chunk_time_interval), defined through `after` in the policy. This columnar format enables fast scanning and aggregation, optimizing performance for analytical workloads while also saving significant storage space. In the columnstore conversion, hypertable chunks are compressed by up to 98%, and organized for efficient, large-scale queries. You can customize this policy later using [alter\_job](/docs/reference/timescaledb/jobs-automation/alter_job). However, to change `after` or `created_before`, the compression settings, or the hypertable the policy is acting on, you must [remove the columnstore policy](/docs/reference/timescaledb/hypercore/remove_columnstore_policy) and [add a new one](/docs/reference/timescaledb/hypercore/add_columnstore_policy). You can also manually [convert chunks](/docs/reference/timescaledb/hypercore/convert_to_columnstore) in a hypertable to the columnstore. 2. **Optimize cooling data for analytics** Hypercore is the hybrid row-columnar storage engine in TimescaleDB, designed specifically for real-time analytics and powered by time-series data. The advantage of hypercore is its ability to seamlessly switch between row-oriented and column-oriented storage. This flexibility enables TimescaleDB to deliver the best of both worlds, solving the key challenges in real-time analytics. ```sql ALTER TABLE signs SET ( timescaledb.enable_columnstore = true, timescaledb.segmentby = 'name'); ``` 3. **Create optimized analytical queries** Continuous aggregates are designed to make queries on very large datasets run faster. Continuous aggregates in Tiger Cloud use PostgreSQL [materialized views](https://www.postgresql.org/docs/current/rules-materializedviews.html) to continuously, and incrementally refresh a query in the background, so that when you run the query, only the data that has changed needs to be computed, not the entire dataset. 1. Create a continuous aggregate pointing to the Supabase database. ```sql CREATE MATERIALIZED VIEW IF NOT EXISTS signs_per_minute WITH (timescaledb.continuous) AS SELECT time_bucket('1 minute', time) as ts, name, count(*) as total FROM signs GROUP BY 1, 2 WITH NO DATA; ``` 2. Setup a delay stats comparing `origin_time` to `time`. ```sql CREATE MATERIALIZED VIEW IF NOT EXISTS _signs_per_minute_delay WITH (timescaledb.continuous) AS SELECT time_bucket('1 minute', time) as ts, stats_agg(extract(epoch from origin_time - time)::float8) as delay_agg, candlestick_agg(time, extract(epoch from origin_time - time)::float8, 1) as delay_candlestick FROM signs GROUP BY 1 WITH NO DATA; ``` 3. Setup a view to recieve the data from Supabase. ```sql CREATE VIEW signs_per_minute_delay AS SELECT ts, average(delay_agg) as avg_delay, stddev(delay_agg) as stddev_delay, open(delay_candlestick) as open, high(delay_candlestick) as high, low(delay_candlestick) as low, close(delay_candlestick) as close FROM _signs_per_minute_delay ``` 4. **Add refresh policies for your analytical queries** You use `start_offset` and `end_offset` to define the time range that the continuous aggregate will cover. Assuming that the data is being inserted without any delay, set the `start_offset` to `5 minutes` and the `end_offset` to `1 minute`. This means that the continuous aggregate is refreshed every minute, and the refresh covers the last 5 minutes. You set `schedule_interval` to `INTERVAL '1 minute'` so the continuous aggregate refreshes on your Tiger Cloud service every minute. The data is accessed from Supabase, and the continuous aggregate is refreshed every minute in the other side. ```sql SELECT add_continuous_aggregate_policy('signs_per_minute', start_offset => INTERVAL '5 minutes', end_offset => INTERVAL '1 minute', schedule_interval => INTERVAL '1 minute'); ``` Do the same thing for data inserted with a delay: ```sql SELECT add_continuous_aggregate_policy('_signs_per_minute_delay', start_offset => INTERVAL '5 minutes', end_offset => INTERVAL '1 minute', schedule_interval => INTERVAL '1 minute'); ``` ## Set up a Supabase database [Section titled “Set up a Supabase database”](#set-up-a-supabase-database) To set up a Supabase database that injects data into your Tiger Cloud service: 1. **Connect a foreign server in Supabase to your Tiger Cloud service** 1. Connect to your Supabase project using Supabase dashboard or psql. 2. Enable the `postgres_fdw` extension. ```sql CREATE EXTENSION postgres_fdw; ``` 3. Create a foreign server that points to your Tiger Cloud service. Update the following command with your [connection details](/docs/integrate/find-connection-details), then run it in the Supabase database: ```sql CREATE SERVER timescale FOREIGN DATA WRAPPER postgres_fdw OPTIONS ( host '', port '', dbname '', sslmode 'require', extensions 'timescaledb' ); ``` 2. **Create the user mapping for the foreign server** Update the following command with your [connection details](/docs/integrate/find-connection-details), the run it in the Supabase database: ```sql CREATE USER MAPPING FOR CURRENT_USER SERVER timescale OPTIONS ( user '', password '' ); ``` 3. **Create a foreign table that points to a table in your Tiger Cloud service** This query introduced the following columns: * `time`: with a default value of `now()`. This is because the `time` column is used by Tiger Cloud to optimize data in the columnstore. * `origin_time`: store the original timestamp of the data. Using both columns, you understand the delay between Supabase (`origin_time`) and the time the data is inserted into your Tiger Cloud service (`time`). ```sql CREATE FOREIGN TABLE signs ( TIME timestamptz NOT NULL DEFAULT now(), origin_time timestamptz NOT NULL, NAME TEXT) SERVER timescale OPTIONS ( schema_name 'public', table_name 'signs' ); ``` 4. **Create a foreign table in Supabase** 1. Create a foreign table that matches the `signs_per_minute` view in your Tiger Cloud service. It represents a top level view of the data. ```sql CREATE FOREIGN TABLE signs_per_minute ( ts timestamptz, name text, total int ) SERVER timescale OPTIONS (schema_name 'public', table_name 'signs_per_minute'); ``` 2. Create a foreign table that matches the `signs_per_minute_delay` view in your Tiger Cloud service. ```sql CREATE FOREIGN TABLE signs_per_minute_delay ( ts timestamptz, avg_delay float8, stddev_delay float8, open float8, high float8, low float8, close float8 ) SERVER timescale OPTIONS (schema_name 'public', table_name 'signs_per_minute_delay'); ``` ## Test the integration [Section titled “Test the integration”](#test-the-integration) To inject data into your Tiger Cloud service from a Supabase database using a foreign table: 1. **Insert data into your Supabase database** Connect to Supabase and run the following query: ```sql INSERT INTO signs (origin_time, name) VALUES (now(), 'test') ``` 2. **Check the data in your Tiger Cloud service** [Connect to your Tiger Cloud service](/docs/build/data-management/run-queries-from-tiger-console) and run the following query: ```sql SELECT * from signs; ``` You see something like: | origin\_time | time | name | | ----------------------------- | ----------------------------- | ---- | | 2025-02-27 16:30:04.682391+00 | 2025-02-27 16:30:04.682391+00 | test | You have successfully integrated Supabase with your Tiger Cloud service. # Data ingestion & streaming > Use your data ingestion and streaming tools with Tiger Data The guides below show how to connect data ingestion and streaming tools to TimescaleDB and Tiger Cloud to move data from sources into your database in real time or on a schedule and keep your analytics up to date. ## Integrations [Section titled “Integrations”](#integrations) [HighByte](/docs/integrate/data-ingestion-streaming/highbyte) [Connect operational technology sources, model the data, and stream it to your service](/docs/integrate/data-ingestion-streaming/highbyte) [Fivetran](/docs/integrate/data-ingestion-streaming/fivetran) [Automatically sync data from multiple sources to simplify ETL processes](/docs/integrate/data-ingestion-streaming/fivetran) [HiveMQ](/docs/integrate/data-ingestion-streaming/hivemq) [Stream MQTT messages from a HiveMQ broker into Tiger Cloud](/docs/integrate/data-ingestion-streaming/hivemq) [EMQX](/docs/integrate/data-ingestion-streaming/emqx) [Stream MQTT messages from an EMQX broker into Tiger Cloud](/docs/integrate/data-ingestion-streaming/emqx) [Ignition](/docs/integrate/data-ingestion-streaming/ignition) [Connection Ignition SCADA systems to Tiger Cloud](/docs/integrate/data-ingestion-streaming/ignition) # Integrate EMQX with Tiger Cloud > Connect EMQX to Tiger Cloud and stream MQTT messages into a hypertable [EMQX](https://www.emqx.com/) is a distributed MQTT broker used to move data between IoT devices, applications, and backend systems in real time. This page shows you how to connect EMQX to your Tiger Cloud service using the EMQX [data integration for PostgreSQL](https://docs.emqx.com/en/emqx/latest/data-integration/data-bridge-timescale.html) to stream MQTT messages into TimescaleDB. In this integration guide, you: * prepare a target hypertable for MQTT messages in your Tiger Cloud service service * configure EMQX to send messages to your PostgreSQL database * verify that MQTT tag values arrive in your TimescaleDB hypertable ## Prerequisites for this integration guide To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service). These steps use Tiger Cloud, but the same approach applies to a [self-hosted TimescaleDB](/docs/get-started/choose-your-path/install-timescaledb) instance. - Your [connection details](/docs/integrate/find-connection-details). * An [EMQX](https://www.emqx.com/en/downloads) broker (Open Source or Enterprise) with access to the EMQX Dashboard ## Create a target hypertable in Tiger Cloud 1. **Connect to your service** Use the [Tiger Cloud Console](/docs/build/data-management/run-queries-from-tiger-console), `psql`, or any other SQL editor to connect to your Tiger Cloud service. 2. **Create a target hypertable for MQTT messages** Create a hypertable with the columns you want EMQX to populate from each MQTT message. For example: ```sql CREATE TABLE mqtt_messages ( ts TIMESTAMPTZ NOT NULL, tag_id TEXT NOT NULL, client_id TEXT, qos SMALLINT, value DOUBLE PRECISION ) WITH ( tsdb.hypertable = true, tsdb.partition_column = 'ts', tsdb.chunk_interval = '7 days' ); ``` Choose a chunk interval that matches your message volume and query patterns. Aim for chunk sizes that fit the most recent working set in memory — a good starting point is an interval that produces chunks roughly 25% of your service RAM. For lower-volume streams, that may mean hours instead of days; for very busy MQTT workloads, a longer interval such as 2–7 days may be more appropriate. You can change the chunk interval later with `set_chunk_time_interval()`; it only affects new chunks, so the current data remains unchanged. Note You may need to change the table columns based on the format of the MQTT payload and what you want to store. The instructions here assume a plaintext numeric payload. If you need JSON or Sparkplug B mappings, see the additional examples at the end of this guide. ## Configure an EMQX data integration for PostgreSQL 1. **Sign in to the EMQX Dashboard** Navigate your browser to the EMQX Dashboard (default `http://YOUR_EMQX_HOST:18083`) and sign in with an admin account. 2. **Create a PostgreSQL connector** Open **Integration** / **Connector** and select **Create**. Choose **PostgreSQL** as the connector type. Give the connector a name and add the **host**, **port**, **database**, **username**, and **password** from your Tiger Cloud service connection details. Enable TLS, then select **Test Connectivity** to verify the connection. Warning Tiger Cloud requires TLS. Make sure **Enable TLS** is on, otherwise the connector will not be able to reach your service. 3. **Create a PostgreSQL sink** Open **Integration** / **Rules** and select **Create** to start a new rule. In the **SQL Editor**, write a rule that selects the messages you want to persist. Use the payload mapping example above that matches your MQTT message format. For a simple numeric payload, the rule looks like: ```sql SELECT timestamp as ts, topic as tag_id, clientid as client_id, qos as qos, CAST(payload AS DOUBLE PRECISION) as value FROM "#" ``` Under **Action Outputs**, select **Add Action**, choose **PostgreSQL**, and select the connector you created in the previous step. Set the **SQL template** to insert into your table: ```sql INSERT INTO mqtt_messages (ts, tag_id, client_id, qos, value) VALUES ( to_timestamp(${ts}::double precision / 1000), ${topic}, ${client_id}, ${qos}, ${value} ) ``` Save the action, then save the rule. 4. **Verify messages are flowing** Publish a test message to a topic that matches your rule, then query the table to confirm rows are arriving: ```sql SELECT * FROM mqtt_messages ORDER BY ts DESC LIMIT 10; ``` If no rows appear, check the rule's **Statistics** tab in the EMQX Dashboard for matched and failed counts, and review the EMQX log for connector errors. ## Verify the integration [Section titled “Verify the integration”](#verify-the-integration) To confirm EMQX is sending MQTT data into your Tiger Cloud service: 1. **Publish a test MQTT message** Send a test message to the topic you configured in your EMQX rule. Then check the rule's statistics in the EMQX Dashboard to verify the message was matched and the PostgreSQL action succeeded. 2. **Query the target table in Tiger Cloud service** Use the [Tiger Cloud Console](/docs/build/data-management/run-queries-from-tiger-console), `psql`, or another SQL client to run: ```sql SELECT * FROM mqtt_messages ORDER BY ts DESC LIMIT 10; ``` You should see the tag value and metadata appear in the table rows. You have successfully integrated EMQX with Tiger Cloud. ## Additional payload examples [Section titled “Additional payload examples”](#additional-payload-examples) The following table schemas, EMQX Rules, and Action Outputs can be adapted for different MQTT Payloads depending on the structure of the payload and what values are to be recorded. ### JSON payload with named fields [Section titled “JSON payload with named fields”](#json-payload-with-named-fields) Use this when the MQTT message body is JSON and you want to extract individual fields. Sample payload: ```json { "tag_value": 112.34, "status": "ok", "device": "sensor-1" } ``` Table Schema: ```sql CREATE TABLE mqtt_messages_json ( ts TIMESTAMPTZ NOT NULL, tag_id TEXT NOT NULL, client_id TEXT, qos SMALLINT, value DOUBLE PRECISION, status TEXT ) WITH ( tsdb.hypertable = true, tsdb.partition_column = 'ts', tsdb.chunk_interval = '7 days' ); ``` Rule: ```sql SELECT timestamp as ts, topic as tag_id, clientid as client_id, qos as qos, CAST(payload->>'tag_value' AS DOUBLE PRECISION) as value, payload->>'status' as status FROM "#" ``` Action output: ```sql INSERT INTO mqtt_messages_json (ts, tag_id, client_id, qos, value, status) VALUES ( to_timestamp(${ts}::double precision / 1000), ${topic}, ${client_id}, ${qos}, ${value}, ${status} ) ``` ### Sparkplug B payload [Section titled “Sparkplug B payload”](#sparkplug-b-payload) Use this when EMQX receives Sparkplug B messages with nested metric data. Sample payload: ```json { "namespace": "spBv1.0", "group_id": "factory-1", "edge_node_id": "edge-01", "device_id": "device-01", "metrics": [ { "name": "temperature", "value": 72.5 } ], "timestamp": 1710000000000 } ``` Table Schema: ```sql CREATE TABLE mqtt_messages_sparkplug ( ts TIMESTAMPTZ NOT NULL, namespace TEXT NOT NULL, group_id TEXT NOT NULL, edge_node_id TEXT, device_id TEXT, metric_name TEXT, metric_value DOUBLE PRECISION, metric_ts TIMESTAMPTZ ) WITH ( tsdb.hypertable = true, tsdb.partition_column = 'ts', tsdb.chunk_interval = '7 days' ); ``` Rule: ```sql SELECT timestamp as ts, topic as tag_id, clientid as client_id, qos as qos, payload->>'namespace' as namespace, payload->>'group_id' as group_id, payload->>'edge_node_id' as edge_node_id, payload->>'device_id' as device_id, (payload->'metrics'->0)->>'name' as metric_name, CAST((payload->'metrics'->0)->>'value' AS DOUBLE PRECISION) as metric_value, to_timestamp(CAST(payload->>'timestamp' AS DOUBLE PRECISION) / 1000) as metric_ts FROM "#" ``` Action output: ```sql INSERT INTO mqtt_messages_sparkplug (ts, namespace, group_id, edge_node_id, device_id, metric_name, metric_value, metric_ts) VALUES ( to_timestamp(${ts}::double precision / 1000), ${namespace}, ${group_id}, ${edge_node_id}, ${device_id}, ${metric_name}, ${metric_value}, ${metric_ts} ) ``` ## Troubleshooting [Section titled “Troubleshooting”](#troubleshooting) * If no rows appear in the target table, verify EMQX rule statistics for matched and failed counts, and confirm the connector is using the correct PostgreSQL credentials and database host. * If the rule matches messages but the action fails, check that the payload fields are being extracted with the right JSON path and that the action SQL template uses the same column names as your target table. # Integrate Fivetran with Tiger Cloud > Automatically sync data from multiple sources to simplify ETL processes [Fivetran](https://fivetran.com/docs/getting-started) is a fully managed data pipeline platform that simplifies ETL (Extract, Transform, Load) processes by automatically syncing data from multiple sources to your data warehouse. ![Fivetran synced data in a Tiger Cloud service](/docs/_astro/integrations-fivetran-sync-data.BX_6KKe2_ZveRJy.webp) This page shows you how to inject data from data sources managed by Fivetran into a Tiger Cloud service. ## Prerequisites for this integration guide To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service). These steps use Tiger Cloud, but the same approach applies to a [self-hosted TimescaleDB](/docs/get-started/choose-your-path/install-timescaledb) instance. - Your [connection details](/docs/integrate/find-connection-details). * A [Fivetran](https://www.fivetran.com/) account. ## Set your service as a destination in Fivetran [Section titled “Set your service as a destination in Fivetran”](#set-your-service-as-a-destination-in-fivetran) To be able to inject data into your Tiger Cloud service, set it as a destination in Fivetran: ![Configuring Tiger Cloud as a Fivetran destination](/docs/_astro/integrations-fivetran-destination-timescal-cloud.DAvzkFW3_GHKIp.webp) 1. **Add a destination** In [Fivetran Dashboard > Destinations](https://fivetran.com/dashboard/destinations), click `Add destination`. 2. **Select the PostgreSQL connector** Search for the PostgreSQL connector and click `Select`. Add the destination name and click `Add`. 3. **Connect Fivetran to your service** In the PostgreSQL setup, add your [Tiger Cloud service connection details](/docs/integrate/find-connection-details), then click `Save & Test`. Fivetran validates the connection settings and sets up any security configurations. 4. **Click `View Destination`** The `Destination Connection Details` page opens. ## Set up a Fivetran connection as your data source [Section titled “Set up a Fivetran connection as your data source”](#set-up-a-fivetran-connection-as-your-data-source) In a real world scenario, you can select any of the over 600 connectors available in Fivetran to sync data with your Tiger Cloud service. This section shows you how to inject the logs for your Fivetran connections into your Tiger Cloud service. ![Selecting a data source in Fivetran](/docs/_astro/integrations-fivetran-data-source.Bv9mxu1m_Z1lr3W8.webp) 1. **Add a connection** In [Fivetran Dashboard > Connections](https://fivetran.com/dashboard/connections), click `Add connection`. 2. **Select the `Fivetran Platform` connector** Search for the `Fivetran Platform` connector, then click `Set up`. 3. **Save and test the connection** Leave the default schema name, then click `Save & Test`. You see `All connection tests passed!` 4. **Build data models** Click `Continue`, select `Yes, build data models for me`, and click `Continue`. Your Fivetran connection is connected to your Tiger Cloud service destination. 5. **Start the initial sync** Select `Start syncing all my data now` and click `Start initial sync`. Fivetran creates the log schema in your service and syncs the data to your service. ## View Fivetran data in your service [Section titled “View Fivetran data in your service”](#view-fivetran-data-in-your-service) To see data injected by Fivetran into your Tiger Cloud service: 1. **Query the synced data** In Tiger Console, click `Data view`, select your service, then run the following query: ```sql SELECT * FROM fivetran_log.account LIMIT 10; ``` You see something like the following: ![Viewing Fivetran-synced data in a Tiger Cloud service](/docs/_astro/integrations-fivetran-view-data-in-service.bS7ToIX-_1nLdtx.webp) You have successfully integrated Fivetran with Tiger Cloud. # Integrate HighByte with Tiger Cloud > Connect operational technology sources, model the data, and stream it to your service [HighByte Intelligence Hub](https://www.highbyte.com/) is a DataOps platform for industrial and IoT data integration. It connects to operational technology (OT) sources such as OPC UA, MQTT, and Modbus, models the data, and streams it to destinations like Tiger Cloud. This page shows you how to connect HighByte Intelligence Hub to your Tiger Cloud service to stream industrial time-series data into Tiger Cloud, with support for automatic hypertable creation. ## Prerequisites for this integration guide To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service) with TimescaleDB 2.13+ for automatic hypertable creation. These steps use Tiger Cloud, but the same approach applies to a [self-hosted TimescaleDB](/docs/get-started/choose-your-path/install-timescaledb) instance. - Your [connection details](/docs/integrate/find-connection-details). * A [HighByte Intelligence Hub](https://www.highbyte.com/download-software) instance. * At least one data source configured in HighByte (for example, an OPC UA server or MQTT broker). See the [HighByte documentation](https://guide.highbyte.com/) for details on configuring data sources. ## Create a TimescaleDB connection in HighByte [Section titled “Create a TimescaleDB connection in HighByte”](#create-a-timescaledb-connection-in-highbyte) To stream data from HighByte into your Tiger Cloud service, create a TimescaleDB connection: 1. **Navigate to `Connections`** In HighByte Intelligence Hub, navigate to `Connections`. 2. **Add a new connection** Click `Add Connection` and select `TimescaleDB` as the connection type. 3. **Configure connection settings** Using your [Tiger Cloud service connection details](/docs/integrate/find-connection-details), fill in the connection settings: * `Host`: your service hostname * `Port`: your service port (default `5432`) * `Database`: your database name * `Username`: your database username * `Password`: your database password For Tiger Cloud, add `sslmode=require` in the JDBC connection parameters to enable SSL. 4. **Test the connection** Click `Test Connection` to validate the connection. 5. **Save the connection** Click `Save`. ## Write data to your Tiger Cloud service [Section titled “Write data to your Tiger Cloud service”](#write-data-to-your-tiger-cloud-service) HighByte can write data to your Tiger Cloud service using three modes: * `Insert`: append-only writes, best for time-series ingestion * `Update`: modify existing rows, useful for dimension tables * `Upsert`: insert or update based on conflict, good for slowly changing data When HighByte writes to a table that does not yet exist, it automatically creates the table and converts it to a hypertable. This requires TimescaleDB 2.13 or later. For earlier versions, create the table and hypertable manually before writing data. 1. **Create or select a data model** In HighByte, create or select a `data model` that represents your source data. For example, a temperature sensor model with fields for `time`, `device_id`, and `value`. 2. **Add a TimescaleDB output** Add a `TimescaleDB output` to the data flow and select your connection. 3. **Configure output settings** Configure the output settings: * `Table name`: the target table in your Tiger Cloud service (for example, `sensor_readings`) * `Write mode`: select `Insert` for time-series data 4. **Map data model fields** Map the data model fields to the table columns. 5. **Start the data flow** Start the data flow. HighByte begins streaming data to your Tiger Cloud service. If the target table does not exist, HighByte creates it and converts it to a hypertable automatically. ## Read data from your Tiger Cloud service [Section titled “Read data from your Tiger Cloud service”](#read-data-from-your-tiger-cloud-service) HighByte can also read data from your Tiger Cloud service using SQL queries. This is useful for feeding aggregated or historical data back into other systems. 1. **Add a TimescaleDB input** In HighByte, add a `TimescaleDB input` to a data flow and select your connection. 2. **Write a SQL query** Write a SQL query to retrieve data. For example: ```sql SELECT time, device_id, value FROM sensor_readings WHERE time > :last_read_time ORDER BY time ASC; ``` 3. **Enable caching** Enable `caching` to track the query position. With caching enabled, each successive read retrieves only new data since the last read, avoiding duplicate processing. ## Verify the integration [Section titled “Verify the integration”](#verify-the-integration) To confirm that data is flowing from HighByte into your Tiger Cloud service: 1. **Connect to your service** Use the [Tiger Console](/docs/build/data-management/run-queries-from-tiger-console), `psql`, or any other SQL editor to connect to your Tiger Cloud service. 2. **Check data presence** Check that data from HighByte is present in your Tiger Cloud service: ```sql SELECT * FROM sensor_readings ORDER BY time DESC LIMIT 10; ``` 3. **Confirm hypertable creation** Confirm that the table was created as a hypertable: ```sql SELECT hypertable_name, num_chunks FROM timescaledb_information.hypertables WHERE hypertable_name = 'sensor_readings'; ``` If the query returns a row, HighByte has successfully created the table as a hypertable. You have successfully integrated HighByte with Tiger Cloud. # Integrate HiveMQ with Tiger Cloud > Connect HiveMQ to Tiger Cloud and stream MQTT messages into a hypertable [HiveMQ](https://www.hivemq.com/) is an enterprise MQTT broker used to move data between IoT devices, applications, and backend systems in real time. This page shows you how to connect HiveMQ to your Tiger Cloud service using the [HiveMQ Enterprise Extension for PostgreSQL](https://docs.hivemq.com/hivemq-postgresql-extension/latest/) to stream MQTT messages into Tiger Cloud. ## Prerequisites for this integration guide To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service). These steps use Tiger Cloud, but the same approach applies to a [self-hosted TimescaleDB](/docs/get-started/choose-your-path/install-timescaledb) instance. - Your [connection details](/docs/integrate/find-connection-details). * A [HiveMQ Enterprise](https://www.hivemq.com/downloads/) broker with file system access on the host * The [HiveMQ Enterprise Extension for PostgreSQL](https://docs.hivemq.com/hivemq-postgresql-extension/latest/) downloaded ## Create a target table in Tiger Cloud [Section titled “Create a target table in Tiger Cloud”](#create-a-target-table-in-tiger-cloud) 1. **Connect to your service** Use the [Tiger Console](/docs/build/data-management/run-queries-from-tiger-console), `psql`, or any other SQL editor to connect to your Tiger Cloud service. 2. **Create a hypertable for MQTT messages** Create a `hypertable` for time-series readings. The table stores the event time, a `tag_id`, the numeric `value`, and payload metadata fields: ```sql CREATE TABLE mqtt_messages ( ts TIMESTAMPTZ NOT NULL, tag_id TEXT NOT NULL, value DOUBLE PRECISION, quality TEXT, units TEXT ) WITH ( tsdb.hypertable = true, tsdb.chunk_interval = '1 day' ); ``` This layout (`ts`, `tag_id`, `value`, `quality`, `units`) is recommended for time-series sensor readings when the payload includes metadata. If you want auditability or debugging, add a `payload JSONB` column so you preserve the raw message alongside parsed fields, then adjust the template below accordingly. Aim for 1–20 million rows per chunk. That's roughly 1 day for 10,000 messages/min, or 1 hour for 300,000 messages/min. Adjust `tsdb.chunk_interval` to match your broker's throughput. ## Install and configure the HiveMQ Enterprise Extension for PostgreSQL [Section titled “Install and configure the HiveMQ Enterprise Extension for PostgreSQL”](#install-and-configure-the-hivemq-enterprise-extension-for-postgresql) 1. **Install the extension** Unpack the extension into the `extensions` directory of your HiveMQ installation: ```bash unzip hivemq-postgresql-extension-.zip -d /extensions/ ``` The extension folder is disabled by default. Remove the `DISABLED` marker file once you have finished configuring it in the next steps: ```bash rm /extensions/hivemq-postgresql-extension/DISABLED ``` 2. **Configure the database connection** Edit `/extensions/hivemq-postgresql-extension/conf/config.xml` and add a PostgreSQL connection that points to your Tiger Cloud service. Use the **username**, **password**, **host**, and **port** from your service connection details: ```xml tiger-cloud YOUR_SERVICE_HOST.tsdb.cloud.timescale.com 5432 tsdb tsdbadmin YOUR_PASSWORD true ``` Tiger Cloud requires TLS. Make sure the `true` block is present, otherwise the extension will not be able to connect. 3. **Define a route from MQTT topics to your table** Still in `config.xml`, add an MQTT-to-PostgreSQL route that uses a statement template to parse JSON payload fields and insert them into separate columns on `mqtt_messages`: ```xml mqtt-to-tiger-cloud tiger-cloud uns/tiger-data/# conf/examples/hivemq-mqtt-message-template.sql ``` Narrow the `` to the topics you want to persist. Use `#` only if you want to capture every message on the broker. 4. **Create the statement template file** Create the statement template file at `HIVEMQ_HOME/extensions/hivemq-postgresql-extension/conf/examples/hivemq-mqtt-message-template.sql`. This will need to be customized based on the message format. #### Sample MQTT topic and JSON [Section titled “Sample MQTT topic and JSON”](#sample-mqtt-topic-and-json) Given an mqtt topic and JSON as follows: ```text uns/tiger-data/plant-7/sensor-123/reading ``` ```json { "timestamp": "2026-06-12T14:26:00Z", "value": 74.5, "quality": "GOOD", "units": "°F" } ``` Then the template file might look like this: ```sql INSERT INTO mqtt_messages (ts, tag_id, value, quality, units) VALUES ( ((${mqtt-payload-utf8})::jsonb ->> 'timestamp')::timestamptz, ${mqtt-topic}, ((${mqtt-payload-utf8})::jsonb ->> 'value')::double precision, ((${mqtt-payload-utf8})::jsonb ->> 'quality'), ((${mqtt-payload-utf8})::jsonb ->> 'units') ); ``` The `timestamp` field must be valid ISO 8601 so the cast to `timestamptz` succeeds. `${mqtt-topic}` inserts the full MQTT topic string as `tag_id`. If you only want a sensor identifier, extract the relevant segment from the topic path instead of storing the full topic. See the [HiveMQ extension reference](https://docs.hivemq.com/hivemq-postgresql-extension/latest/) for the full list of placeholders and supported statement templates. In this example, the topic was used as the tag\_id. 5. **Start HiveMQ and verify the extension is loaded** Start (or restart) HiveMQ and check the broker log for a line confirming the extension is started and the route is active: ```bash tail -f /log/hivemq.log ``` Publish a test message to a topic that matches your filter, then query the table to confirm rows are arriving: ```sql SELECT * FROM mqtt_messages ORDER BY ts DESC LIMIT 10; ``` With messages flowing into `mqtt_messages`, add a retention policy to cap how long historical data is kept. The step below can be performed in the [Tiger Console](/docs/build/data-management/run-queries-from-tiger-console) or any other SQL editor. ## Optimize for HiveMQ workloads [Section titled “Optimize for HiveMQ workloads”](#optimize-for-hivemq-workloads) 1. **Optional: add a retention policy** ```sql SELECT add_retention_policy('mqtt_messages', INTERVAL '1 year'); ``` You have successfully integrated HiveMQ with Tiger Cloud. # Integrate Ignition with Tiger Cloud > Connect Ignition SCADA to Tiger Cloud and store tag history [Ignition](https://inductiveautomation.com/) is a SCADA system used to connect data, design industrial systems, and monitor automated processes in real time. This page shows you how to connect Ignition to your Tiger Cloud service to stream industrial data into Tiger Cloud. ## Prerequisites for this integration guide To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service). These steps use Tiger Cloud, but the same approach applies to a [self-hosted TimescaleDB](/docs/get-started/choose-your-path/install-timescaledb) instance. - Your [connection details](/docs/integrate/find-connection-details). * An [Ignition](https://inductiveautomation.com/downloads/) instance with admin access ## Create a Tiger Cloud service connection in Ignition [Section titled “Create a Tiger Cloud service connection in Ignition”](#create-a-tiger-cloud-service-connection-in-ignition) 1. **Sign in to the Ignition Gateway** Navigate your browser to the Ignition Gateway screen and sign in with an admin account. Open `Connections` > `Database Connections` and select `Create Database Connection +`. 2. **Select the PostgreSQL driver** Select `PostgreSQL` as the driver (included in every standard Ignition install). 3. **Configure the database connection** 1. Give the database connection a name. 2. Add the `username`, `password`, and `connection url` using your [Tiger Cloud service connection details](/docs/integrate/find-connection-details). 3. Select `Create Database Connection`. 4. **Verify the connection** Verify that the database connection was created and status is valid. If any errors appear, consult the [Ignition manual](https://www.docs.inductiveautomation.com/). ## Set up your Tiger Cloud service as a historian [Section titled “Set up your Tiger Cloud service as a historian”](#set-up-your-tiger-cloud-service-as-a-historian) 1. **Create a historian** On the Ignition Gateway, select `Services` > `Historians` and `Create Historian +`. 2. **Select SQL Historian** Select `SQL Historian`. 3. **Name the historian and disable partitioning** Add a name for the historian, and select the Tiger Cloud service database connected in section 1 as the Data Source. Disable Partitioning. If you don’t disable partitioning during the initial setup, and actively use the historian beyond the partition length, Ignition will create multiple history tables. This will make setting up Tiger Cloud optimizations more difficult and reduce the convenience of having all historical data in one table. 4. **Verify the historian is running** Verify the new historian is enabled and running. 5. **Start data flowing into the historian** Before proceeding to the next section, make sure there is data flowing into the historian. This can be done by creating any tag and setting `History Enabled` to `True`. That will guarantee that the historian table is created. Ignition will automatically create the table in Tiger Cloud service, but will not set it up as a hypertable. The following instructions can be performed within Ignition's [SQL Editor](https://www.docs.inductiveautomation.com/docs/7.9/sql-in-ignition/common-sql-tasks/simple-database-editor), the [Tiger Console](/docs/build/data-management/run-queries-from-tiger-console), or any other SQL editor. ## Optimize for Ignition workloads [Section titled “Optimize for Ignition workloads”](#optimize-for-ignition-workloads) 1. **Convert the auto-created historian table to a hypertable** ```sql SELECT create_hypertable( 'sqlth_1_data','t_stamp', migrate_data => 'true', chunk_time_interval => 86400000 ); ``` Chunk size must be defined in milliseconds. Recommended chunk size should aim for about 1-20 million records per chunk. That's about 1 day (86,400,000ms) for 10,000 tags/min or 1 hour (3,600,000ms) for 300,000 tags/min. 2. **Tell TimescaleDB what the current time is** Ignition stores `t_stamp` as a Unix epoch timestamp in milliseconds, not as a PostgreSQL `timestamp`. Automated jobs such as columnstore and retention policies work out the age of each chunk relative to the current time, so you must tell TimescaleDB how to read "now" for this integer column with [`set_integer_now_func`](/docs/reference/timescaledb/hypertables/set_integer_now_func). The function must return milliseconds to match `t_stamp`: ```sql CREATE OR REPLACE FUNCTION unix_now_ms() RETURNS BIGINT LANGUAGE SQL STABLE AS $$ SELECT (extract(epoch from now()) * 1000)::BIGINT $$; SELECT set_integer_now_func('sqlth_1_data', 'unix_now_ms'); ``` 3. **Optional: enable the columnstore and add a policy** Because `t_stamp` is an integer column, the policy `after` argument is a number of milliseconds rather than an `INTERVAL`. For example, `604800000` is 7 days: ```sql ALTER TABLE sqlth_1_data SET ( timescaledb.enable_columnstore = true, timescaledb.segmentby = 'tagid', timescaledb.orderby = 't_stamp DESC' ); SELECT add_columnstore_policy('sqlth_1_data', after => 604800000); ``` 4. **Optional: add retention policy** As with the columnstore policy, `drop_after` is a number of milliseconds. For example, `31536000000` is 365 days: ```sql SELECT add_retention_policy('sqlth_1_data', drop_after => 31536000000); ``` You have successfully integrated Ignition with Tiger Cloud. # Integrate Kepware KEPServerEX with Tiger Cloud > Stream OPC tag data from Kepware DataLogger into Tiger Cloud and store it in a hypertable [Kepware KEPServerEX](https://www.kepware.com/en-us/products/kepserverex/) is an industrial connectivity platform that logs OPC tag data from controllers and sensors to ODBC-compliant databases using Kepware DataLogger. This page shows you how to configure Kepware DataLogger to stream OPC data into Tiger Cloud and store it in a hypertable. In this integration guide, you: * Create a target hypertable in your Tiger Cloud service. * Configure a Kepware DataLogger log group and ODBC DSN. * Verify data flows into the target hypertable. ## Prerequisites for this integration guide To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service). These steps use Tiger Cloud, but the same approach applies to a [self-hosted TimescaleDB](/docs/get-started/choose-your-path/install-timescaledb) instance. - Your [connection details](/docs/integrate/find-connection-details). * A running [Kepware Server EX](https://www.ptc.com/en/products/kepware/kepserverex) installation with the DataLogger add-on. * The PostgreSQL ODBC driver installed on the Kepware host. ## Prepare your service [Section titled “Prepare your service”](#prepare-your-service) Create the target table in Tiger Cloud before DataLogger begins writing. Convert that table to a hypertable so your OPC time-series data is partitioned and chunked correctly. Note If Kepware DataLogger creates the table for you, convert it to a hypertable before you enable the log group. 1. **Connect to your Tiger Cloud service** Use the [Tiger Console](/docs/build/data-management/run-queries-from-tiger-console) or another SQL editor to connect to your Tiger Cloud service. 2. **Create the target hypertable** Create the table directly as a hypertable partitioned on the `time` column: ```sql CREATE TABLE kepware_telemetry ( time timestamptz NOT NULL, tag_name varchar(128), value varchar, quality varchar(32) ) WITH ( timescaledb.hypertable, timescaledb.chunk_interval = '7 days' ); ``` The `chunk_interval` setting controls how much data each hypertable chunk holds. A 7-day interval works well for most OPC ingestion workloads. If you have high-throughput tag updates, consider a shorter interval like 1 day. You can adjust this later with `set_chunk_time_interval()`. ## Configure Kepware DataLogger [Section titled “Configure Kepware DataLogger”](#configure-kepware-datalogger) Point DataLogger at your Tiger Cloud hypertable and choose a table write mode that preserves data. 1. **Open the log group configuration** In DataLogger, open your configuration, then add a new log group or edit an existing one. 2. **Configure the ODBC DSN** In the log group settings, click the browse button for `DSN`, then select `Configure DSN…` to open the Microsoft ODBC Data Source Administrator. On the System DSN tab, click `Add`, choose the PostgreSQL ODBC driver for Tiger Data, and complete the driver wizard. 3. **Select the Tiger Cloud DSN** In the log group settings, choose the DSN you created for Tiger Cloud. If required, enter the DSN `Username` and `Password` for the Tiger Cloud service. 4. **Point the log group at the hypertable** Set `TableSelection` to `Create a new table once and always append to this table` and enter `kepware_telemetry` as the table name. Leave `Create Table If Missing` unchecked so DataLogger writes to the existing hypertable instead of creating a plain table. 5. **Add OPC tags to the log group** Add the server items or OPC tags you want to log. Use `Narrow` table format for the schema above, because the target hypertable stores one row per tag update with generic columns such as `tag_name`, `value`, and `quality`. In narrow mode, DataLogger uses a single column-mapping object for all log items. Map `Timestamp` to `time`, `Name` or `NumericID` to `tag_name`, `Value` to `value`, and `Quality` to `quality`. 6. **Enable the log group** Save the log group and set `Enabled` to `Yes`. DataLogger begins sending OPC updates to your Tiger Cloud hypertable when the log group is enabled. ## Stream OPC data to your service [Section titled “Stream OPC data to your service”](#stream-opc-data-to-your-service) A configured log group will only move data after an OPC tag update occurs. 1. **Generate a tag update** Trigger a value change on the configured OPC tag or wait for the polling interval so DataLogger logs a row. 2. **Confirm the data landed in Tiger Cloud** Run the following query in Tiger Console or another SQL editor: ```sql SELECT time, tag_name, value, quality FROM kepware_telemetry ORDER BY time DESC LIMIT 10; ``` You should see recent rows from Kepware, with `time` reflecting when the tag value was logged. You have successfully integrated Kepware KEPServerEX with Tiger Cloud. ## Troubleshooting [Section titled “Troubleshooting”](#troubleshooting) * **The DSN driver is not listed:** install the PostgreSQL ODBC driver and open the 32-bit ODBC Data Source Administrator on Windows. * **`relation "kepware_telemetry" does not exist`:** verify the log group table name is `kepware_telemetry` and that the hypertable exists before enabling the group. * **DataLogger created a plain table:** convert the table with `SELECT create_hypertable('kepware_telemetry', 'time');` and keep `Create Table If Missing` unchecked. * **No rows appear after enabling the group:** confirm the log group is `Enabled` and that the configured OPC tags are updating. # Integrate Litmus Edge with Tiger Cloud > Stream industrial device data from Litmus Edge into your service using the PostgreSQL integration connector [Litmus Edge](https://litmus.io/litmus-edge/) is an industrial edge platform that collects data from PLCs, sensors, and other operational technology assets over protocols such as OPC UA, MQTT, and Modbus, then forwards that data to enterprise systems and databases. This page shows you how to configure the PostgreSQL integration connector in Litmus Edge to stream device data into your Tiger Cloud service, where it is stored as a hypertable for time-series analytics. In this integration guide, you: * Create a target hypertable in your Tiger Cloud service. * Configure the PostgreSQL connector in Litmus Edge to point at your service. * Subscribe a DeviceHub topic to the connector so device data flows into the hypertable. ## Prerequisites for this integration guide To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service). These steps use Tiger Cloud, but the same approach applies to a [self-hosted TimescaleDB](/docs/get-started/choose-your-path/install-timescaledb) instance. - Your [connection details](/docs/integrate/find-connection-details). * A running [Litmus Edge](https://docs.litmus.io/litmusedge) instance that can reach your Tiger Cloud service on port `5432`. * At least one device configured in Litmus Edge [DeviceHub](https://docs.litmus.io/litmusedge/product-features/devicehub) that is publishing tags you want to stream. ## Prepare your service [Section titled “Prepare your service”](#prepare-your-service) Litmus Edge writes every published tag to a fixed schema. Pre-create the table as a hypertable so time-series queries and columnstore policies work out of the box. 1. **Connect to your Tiger Cloud service** Use the [Tiger Console](/docs/build/data-management/run-queries-from-tiger-console), `psql`, or any other SQL editor to connect to your Tiger Cloud service. 2. **Create the target hypertable** Litmus Edge writes one row per tag update with the following columns. Create the table directly as a hypertable partitioned on `arrived_at`: ```sql CREATE TABLE litmus_data ( id bigserial, record uuid NOT NULL, arrived_at timestamptz NOT NULL, device_id varchar(64), register_id varchar(64), tag_name varchar(64), datatype varchar(32), value varchar, success boolean ) WITH ( timescaledb.hypertable, timescaledb.chunk_interval = '7 days', timescaledb.segmentby = 'tag_name, device_id, register_id' ); ``` Litmus Edge's default schema declares `id` as the primary key, but a hypertable primary key must include the partition column. The statement above omits the primary key so that `arrived_at` alone partitions the data. Leave the `Create Table If Missing` option in Litmus Edge unchecked so it writes into this table instead of trying to create its own. If you need row-level uniqueness — for upserts or logical replication, for example — add `PRIMARY KEY (arrived_at, record)`: `record` is already `NOT NULL` and a uuid, and pairing it with the partition column satisfies the hypertable requirement. The `chunk_interval` setting controls how much data each hypertable chunk holds. Aim for chunks that fit the most recent working set in memory — a good starting point is an interval that produces chunks roughly 25% of your service's RAM. For a busy line publishing thousands of tags per second, that may mean hours rather than days; You can change the interval later with [`set_chunk_time_interval()`](/docs/reference/timescaledb/hypertables/set_chunk_time_interval) — it only affects new chunks. ## Configure the PostgreSQL connector in Litmus Edge [Section titled “Configure the PostgreSQL connector in Litmus Edge”](#configure-the-postgresql-connector-in-litmus-edge) Litmus Edge ships two PostgreSQL integration types: `DB - PostgreSQL` for plaintext connections and `DB - PostgreSQL SSL` for TLS-protected connections. Tiger Cloud requires TLS, so use the SSL variant for Tiger Cloud and either variant for self-hosted TimescaleDB. 1. **Open the Integration pane** Log in to Litmus Edge, then select `Integration` from the navigation panel. 2. **Add a PostgreSQL connector** Click the add icon, then select `DB - PostgreSQL SSL` from the list of integration connectors. 3. **Configure connection settings** Using your [Tiger Cloud service connection details](/docs/integrate/find-connection-details), fill in the connector fields: * `Name`: a label for this connector, for example `tiger-cloud`. * `Hostname`: your service hostname. * `Port`: `5432`. * `SSL Mode`: `require`. Use `verify ca` or `verify full` if you want Litmus Edge to validate the server certificate. * `Username`: your service username, typically `tsdbadmin`. * `Password`: your service password. * `Database`: `tsdb`. * `Table`: `litmus_data`. Leave `CA Certificate`, `Certificate`, and `Private Key` blank when `SSL Mode` is `require`. For `verify ca` or `verify full`, paste the Tiger Cloud CA certificate into `CA Certificate`. 4. **Tune the write behavior** For production workloads that require high throughput, set: * `Bulk Insert Count`: number of messages to batch per `INSERT`. A value between `100` and `1000` reduces commit overhead at high message rates. * `Commit Timeout`: maximum milliseconds to wait before flushing a partial batch. `1000` is a good starting point — drop to `200`–`500` if you need sub-second freshness. * `Persistent Storage`: enable to spool messages to disk if the connection to your service drops, so no data is lost during a brief outage. Leave `Create Table If Missing` unchecked — the hypertable you created earlier is the target. 5. **Save the connector** Click `OK`. The new connector appears as a tile in the Integration pane with its status indicator. ## Stream device data to your service [Section titled “Stream device data to your service”](#stream-device-data-to-your-service) A connector by itself does not move data. Subscribe one or more DeviceHub topics to it so tag updates flow into your hypertable. 1. **Open the connector** In the Integration pane, click the tile for the connector you just created. 2. **Add a topic subscription** Select the `Topics` tab, click the add icon, then choose `add a new subscription`. 3. **Map a DeviceHub tag to the connector** Configure the subscription: * `Data direction`: `Local to Remote`. * `Local data topic`: click the search icon and select the DeviceHub device and tag you want to stream. To stream every tag from a device, use a wildcard suffix. * `Remote data topic`: leave the default or enter a label that identifies this stream. * `QoS`: `1` for at-least-once delivery. * `Enable`: toggle on. Click `OK`, then confirm the prompt to restart the connector. ## Verify the integration [Section titled “Verify the integration”](#verify-the-integration) To confirm Litmus Edge is writing data to your Tiger Cloud service: 1. **Trigger a tag update in Litmus Edge** Generate activity on the subscribed tag — let the device publish, or open the tag in DeviceHub and use `Write Value` to push a test value. 2. **Query the target table** From the [Tiger Console](/docs/build/data-management/run-queries-from-tiger-console) or `psql`, run: ```sql SELECT arrived_at, device_id, tag_name, value FROM litmus_data ORDER BY arrived_at DESC LIMIT 10; ``` You see one row per tag update, with `arrived_at` reflecting when Litmus Edge received the value, `device_id` and `tag_name` identifying the source, and `value` holding the payload as a string. You have successfully integrated Litmus Edge with Tiger Cloud. ## Troubleshooting [Section titled “Troubleshooting”](#troubleshooting) * **Connector status shows `disconnected`:** check that the Litmus Edge host can reach your service on port `5432`, and that the `Username`, `Password`, and `Database` match your [connection details](/docs/integrate/find-connection-details). * **`relation "litmus_data" does not exist`:** Litmus Edge is writing to the wrong table. Confirm the `Table` field is set to `litmus_data` and `Create Table If Missing` is unchecked. * **SSL handshake fails:** Tiger Cloud requires TLS — use the `DB - PostgreSQL SSL` connector, not the plain `DB - PostgreSQL` connector. If you selected `verify ca` or `verify full`, make sure the `CA Certificate` field contains the Tiger Cloud CA certificate. * **No rows appearing after a tag update:** confirm that the topic subscription is `Enabled` and that you restarted the connector after adding it. For other connectivity and authentication issues, see [Troubleshoot Tiger Cloud integrations](/docs/integrate/troubleshooting). ## Limitations [Section titled “Limitations”](#limitations) * The PostgreSQL connector is outbound only. Litmus Edge writes to your Tiger Cloud service but cannot read from it. To query the hypertable from inside Litmus Edge, use the Litmus SQL node in a Flow — it can read from your service and feed the results back into DeviceHub or another connector. * Litmus Edge writes the tag payload to a single `varchar` column. Cast `value` to the appropriate type (`(value)::double precision`, `(value)::int`) at query time, or use a continuous aggregate to materialize typed columns. # Integrate Node-RED with Tiger Cloud > Use Node-RED to stream events into Tiger Data and persist them in a hypertable. [Node-RED](https://nodered.org/) is a flow-based programming tool for wiring together hardware devices, APIs, and services using a browser-based editor. This page shows you how to use Node-RED to write events into your Tiger Data service or database so you can persist, query, and analyze them in a hypertable. In this integration guide, you: * Create a hypertable to receive Node-RED events. * Configure Node-RED with a PostgreSQL / TimescaleDB node and connection details. * Send a test event from Node-RED and verify it arrived in your hypertable. ## Prerequisites for this integration guide To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service). These steps use Tiger Cloud, but the same approach applies to a [self-hosted TimescaleDB](/docs/get-started/choose-your-path/install-timescaledb) instance. - Your [connection details](/docs/integrate/find-connection-details). * A running Node-RED instance that can reach your Tiger Cloud service on port `5432` (install instructions: ). * The `node-red-contrib-postgresql` (or equivalent PostgreSQL) node installed in Node-RED. * A database user with INSERT privileges on the target database in your Tiger Cloud service. ## Prepare your database [Section titled “Prepare your database”](#prepare-your-database) Create a hypertable that matches the event columns you plan to send from Node-RED. This example uses `ts` as the time partition, `tag_id` to identify the source, and `value` for the measurement: 1. **Connect to your Tiger Cloud service** Use the [Tiger Console](/docs/build/data-management/run-queries-from-tiger-console), `psql`, or any SQL editor to connect to your service. 2. **Create a hypertable for Node-RED events** Run the following SQL to create the table as a hypertable with explicit chunk interval: ```sql CREATE TABLE node_red_events ( ts TIMESTAMPTZ NOT NULL, tag_id TEXT NOT NULL, value DOUBLE PRECISION ) WITH ( timescaledb.hypertable, timescaledb.chunk_interval = '7 days' ); ``` Modify this schema to match your data. If you send additional fields from Node-RED (such as device ID, location, or unit), add them as columns — for example, `device_id TEXT`, `unit TEXT`, or `location TEXT`. You can also change `value` to `NUMERIC`, `INT`, or `TEXT` if your measurements are non-numeric. The `chunk_interval` setting controls how much data each chunk holds. Aim for chunks that fit the most recent working set in memory — a good starting point is an interval that produces chunks roughly 25% of your service's RAM. For a light event stream publishing a few events per second, 7 days is a reasonable default; for higher throughput, reduce the interval to hours or even minutes. You can change the interval later with [`set_chunk_time_interval()`](/docs/reference/timescaledb/hypertables/set_chunk_time_interval) — it only affects new chunks. ## Configure Node-RED to send data to your database [Section titled “Configure Node-RED to send data to your database”](#configure-node-red-to-send-data-to-your-database) This section shows a minimal Node-RED flow: an `inject` node produces a sample payload, a `function` node formats the values, and a PostgreSQL node performs an `INSERT` using the service connection details. 1. **Install the PostgreSQL node in Node-RED** Open the Node-RED palette manager and install `node-red-contrib-postgresql` (or your preferred PostgreSQL client node). Restart Node-RED if required. 2. **Configure the PostgreSQL node connection** In the PostgreSQL node configuration, create a new server config and enter the `host`, `port`, `database`, `user`, and `password` from your connection details. Enable SSL/TLS if your service requires it. Example connection settings (fill from your connection details): * Host: YOUR\_SERVICE\_HOST * Port: 5432 * Database: tsdb * User: tsdbadmin * Password: (from your connection details) * SSL/TLS: enabled (for Tiger Cloud) The PostgreSQL node typically accepts an SQL string and an array of parameters (for example, `msg.params`). 3. **Build a flow that inserts events** Create a simple flow: 1. `inject` node — sends a test JSON object. 2. `function` node — maps the incoming JSON to `msg.topic` (SQL) and `msg.params` (parameters array). 3. `postgresql` node — configured to run parameterized SQL using `msg.topic` and `msg.params`. **Single-row INSERT (simplest approach):** ```js // msg.payload is the event object const ev = msg.payload || { tag_id: 'sensor-001', value: 22.5 }; msg.topic = 'INSERT INTO node_red_events (ts, tag_id, value) VALUES (NOW(), $1, $2)'; msg.params = [ev.tag_id, ev.value]; return msg; ``` **Batched INSERT (for higher throughput):** For production workloads, buffer multiple events and send them in a single multi-row INSERT to reduce commit overhead. This example flushes every 2 seconds to balance throughput with latency: ```js // Accumulate events into context.batch with timestamps if (!context.batch) { context.batch = []; context.batchStart = null; } const ev = msg.payload; if (ev) { context.batch.push(ev); // Record start time on first event if (!context.batchStart) { context.batchStart = Date.now(); } } // Flush if 2 seconds have elapsed OR if explicitly triggered const elapsed = Date.now() - context.batchStart; const shouldFlush = context.batch.length > 0 && (elapsed >= 2000 || msg.flush); if (shouldFlush) { const batch = context.batch; context.batch = []; context.batchStart = null; // Build multi-row VALUES clause const values = batch.map((_, i) => { const offset = i * 2; return `(NOW(), $${offset + 1}, $${offset + 2})`; }).join(', '); const params = batch.flatMap(e => [e.tag_id, e.value]); msg.topic = `INSERT INTO node_red_events (ts, tag_id, value) VALUES ${values}`; msg.params = params; return msg; } // Don't send if not flushing yet return null; ``` Set up a `delay` node (2 seconds) that feeds back into the function node and inject a message with `msg.flush = true`. This ensures that even during quiet periods, partial batches are flushed every 2 seconds so data never waits too long. Deploy the flow and trigger the `inject` node to send the test event. ## Send a test event and verify data arrived [Section titled “Send a test event and verify data arrived”](#send-a-test-event-and-verify-data-arrived) 1. **Send a test event from Node-RED** Trigger the `inject` node you created. The PostgreSQL node should return success and the flow sidebar will show the node status. 2. **Query Tiger Cloud service to confirm events arrived** Connect to your service and run: ```sql SELECT ts, tag_id, value FROM node_red_events ORDER BY ts DESC LIMIT 10; ``` You should see one or more rows matching the test event you injected, with `payload` containing the JSON you sent. You have successfully integrated Node-RED with Tiger Data. ## Troubleshooting [Section titled “Troubleshooting”](#troubleshooting) * **No rows appear after inject:** Verify the PostgreSQL node connection details (host/port/user/password) and check the Node-RED logs for errors. Confirm TLS is enabled if using Tiger Cloud. * **Parameter/placeholder errors:** Some PostgreSQL nodes expect `$1,$2` parameter placeholders; others use `?`. Use the syntax required by your Node-RED PostgreSQL node. ## Next steps [Section titled “Next steps”](#next-steps) [Connect to Tiger Data](/docs/integrate/find-connection-details) [Where to find the host, port, user, and password for your Tiger Cloud service.](/docs/integrate/find-connection-details) # Connect to Tiger Data > Connect to Tiger Cloud, self-hosted TimescaleDB, or MST using any PostgreSQL client To connect to your service or self-hosted TimescaleDB instance, you need at least the following: * Hostname * Port * Username * Password * Database name Once you have these, connect using any PostgreSQL client. Find the connection methods and details for your product: * Tiger Cloud Connect to your Tiger Cloud service in any of these ways: * **Data view or SQL editor in Tiger Console**: run queries in your browser, with no local client required. * **`psql` or any other PostgreSQL client**: use the connection details below with `psql`, or any PostgreSQL-compatible client or SQL editor. Retrieve the connection details for your Tiger Cloud service: * **In `-credentials.txt`**: All connection details are supplied in the configuration file you download when you create a new service. * **In Tiger Console**: Open the [`Services`](https://console.cloud.tigerdata.com/dashboard/services) page and select your service. The connection details, except the password, are available in `Service info` > `Connection info` > `More details`. If necessary, click `Forgot your password?` to get a new one. ![Service connection details in Tiger Console](/docs/_astro/tiger-service-connection-details.CJPBXPtc_Z23UGAt.webp) ## Find your project and service ID [Section titled “Find your project and service ID”](#find-your-project-and-service-id) To retrieve the connection details for your Tiger Cloud project and Tiger Cloud service: 1. **Retrieve your project ID** In [Tiger Console](https://console.cloud.tigerdata.com/dashboard/services), click the arrows next to the project name in the top left. The project ID is under the project name. ![Retrieving the project ID in Tiger Console](/docs/_astro/tiger-console-change-project.D-LN4vwi_1pfWTk.webp) 2. **Retrieve your service ID** Click the dots next to the service, then click `Copy` next to the service ID. ![Retrieving the service ID in Tiger Console](/docs/_astro/tiger-console-service-id.KxfYxLWI_ZaH4ze.webp) ## Create client credentials [Section titled “Create client credentials”](#create-client-credentials) You use client credentials to obtain access tokens outside of the user context. To retrieve the connection details for your Tiger Cloud project for programmatic usage such as Terraform or the [Tiger Cloud REST API](/docs/get-started/quickstart/cli-rest-api): 1. **Open the settings for your project** In [Tiger Console](https://console.cloud.tigerdata.com/dashboard/services), click `Settings`. 2. **Create client credentials** 1. Click `Create credentials`, then copy `Public key` and `Secret key` locally. ![Creating client credentials in Tiger Console](/docs/_astro/tiger-cloud-console-client-credentials.DPblOdeh_kpDgz.webp) This is the only time you see the `Secret key`. After this, only the `Public key` is visible in this page. 2. Click `Done`. * Self-hosted TimescaleDB Connect using `psql` or any other PostgreSQL-compatible client or SQL editor, with the connection details below. Find the connection details in the [PostgreSQL configuration file](https://www.postgresql.org/docs/current/runtime-config-file-locations.html) or by asking your database administrator. The `postgres` superuser, created during PostgreSQL installation, has all the permissions required to run procedures in this documentation. However, it is recommended to create other users and assign permissions on the need-only basis. * Managed Service for TimescaleDB Connect using `psql` or any other PostgreSQL-compatible client, with the connection details below. In the `Services` page of the Managed Service for TimescaleDB Console, click the MST service you want to connect to. You see the connection details: ![MST service connection details](/docs/_astro/mst-connection-info.Doo0G8db_Z6MTAp.webp) # Observability & alerting > Use your monitoring and alerting tools with Tiger Data The guides below show how to connect monitoring and alerting tools to your database so you can collect metrics, build operational dashboards, and get alerted on service health and performance. ## Integrations [Section titled “Integrations”](#integrations) [Amazon CloudWatch](/docs/integrate/observability-alerting/cloudwatch) [Export telemetry data to monitor and analyze your service performance and health](/docs/integrate/observability-alerting/cloudwatch) [Azure Monitor](/docs/integrate/observability-alerting/azure-monitor) [Export telemetry data to monitor and analyze your service performance and health](/docs/integrate/observability-alerting/azure-monitor) [Datadog](/docs/integrate/observability-alerting/datadog) [Export telemetry data and collect metrics to monitor your service performance](/docs/integrate/observability-alerting/datadog) [Grafana](/docs/integrate/observability-alerting/grafana) [Query, visualize, and explore your data with interactive dashboards](/docs/integrate/observability-alerting/grafana) [Prometheus](/docs/integrate/observability-alerting/prometheus) [Export telemetry metrics from your service to monitor system performance and health](/docs/integrate/observability-alerting/prometheus) [Telegraf](/docs/integrate/observability-alerting/telegraf) [Ingest metrics and events from databases, systems, and IoT sensors](/docs/integrate/observability-alerting/telegraf) ## Reference [Section titled “Reference”](#reference) [Exported metrics](/docs/integrate/observability-alerting/exported-metrics) [Reference for system and PostgreSQL metrics exported by Tiger Cloud](/docs/integrate/observability-alerting/exported-metrics) # Integrate Azure Monitor with Tiger Cloud > Export telemetry data to monitor and analyze your service performance and health with Azure Monitor [Azure Monitor](https://learn.microsoft.com/en-us/azure/azure-monitor/) is a comprehensive monitoring and observability service designed to help you maximize the availability and performance of your applications and services. It collects, analyzes, and acts on telemetry data from your cloud and on-premises environments. You can export telemetry data from your Tiger Cloud services with the time-series and analytics capability enabled to Azure Monitor. See [Exported metrics](/docs/integrate/observability-alerting/exported-metrics) for the full list of default and additional metrics you can export. This page explains how to export telemetry data from your Tiger Cloud service into Azure Monitor by creating a Tiger Cloud data exporter, then attaching it to the service. This integration is available for [Scale and Enterprise](/docs/deploy/tiger-cloud/tiger-cloud-azure/pricing-and-account-management#features-included-in-each-pricing-plan) pricing tiers. ## Prerequisites for this integration guide To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service). - An Azure subscription with access to [Azure Monitor](https://learn.microsoft.com/en-us/azure/azure-monitor/) and [Application Insights](https://learn.microsoft.com/en-us/azure/azure-monitor/app/app-insights-overview). - Your [Azure Monitor Application Insights connection string](https://learn.microsoft.com/en-us/azure/azure-monitor/app/sdk-connection-string). Note This feature is currently not supported for Tiger Cloud on AWS. ## Create a data exporter [Section titled “Create a data exporter”](#create-a-data-exporter) A Tiger Cloud data exporter sends telemetry data from a Tiger Cloud service to third-party monitoring tools. You create an exporter on the [project level](/docs/deploy/tiger-cloud/tiger-cloud-aws/security/members): 1. **In Tiger Console, open Exporters** 2. **Click `New exporter`** 3. **Select `Metrics` for `Data type` and `Azure Monitor` for provider** ![Adding an Azure Monitor exporter in Tiger Console](/docs/_astro/tiger-console-integrations-azure-monitor.BwfUjQBX_y3mq3.webp) 4. **Choose the region for your exporter** 5. **Provide a name for your exporter** The exporter name appears in Tiger Console, best practice is to make this name easily understandable. 6. **Add your connection string** [Learn where to find](https://learn.microsoft.com/en-us/azure/azure-monitor/app/create-workspace-resource?tabs=portal#get-the-connection-string) your connection string. 7. **Export additional metrics** Optionally tick `PostgreSQL metrics` to export [additional metrics](/docs/integrate/observability-alerting/exported-metrics#postgresql-metrics), then click `Create exporter`. Your exporter is created and added to the list of already existing exporters. ## Manage a data exporter [Section titled “Manage a data exporter”](#manage-a-data-exporter) This section shows you how to attach, edit, and delete a data exporter. ### Attach a data exporter to a Tiger Cloud service To send telemetry data to an external monitoring tool, you attach a data exporter to your Tiger Cloud service. You can attach only one exporter to a service. To attach an exporter: 1. **In Tiger Console, choose the service** 2. **Click `Operations` > `Exporters`** 3. **Select the exporter, then click `Attach exporter`** 4. **If you are attaching a first `Logs` data type exporter, restart the service** You can now monitor your service metrics. ### Edit a data exporter To update a data exporter: 1. **In Tiger Console, open Exporters** 2. **Next to the exporter you want to edit, click the menu > `Edit`** 3. **Edit the exporter fields and save your changes** ### Delete a data exporter To remove a data exporter that you no longer need: 1. **Disconnect the data exporter from your Tiger Cloud service** 1. In [Tiger Console](https://console.cloud.tigerdata.com/dashboard/services), choose the service. 2. Click `Operations` > `Exporters`. 3. Click the trash can icon next to the exporter. 4. Repeat for every service attached to the exporter you want to remove. The data exporter is now unattached from all services. However, it still exists in your project. 2. **Delete the exporter on the project level** 1. In Tiger Console, open [Exporters](https://console.cloud.tigerdata.com/dashboard/integrations) 2. Next to the exporter you want to edit, click menu > `Delete` 3. Confirm that you want to delete the data exporter. # Integrate Amazon CloudWatch with Tiger Cloud > Export telemetry data to monitor and analyze your service performance and health [Amazon CloudWatch](https://aws.amazon.com/cloudwatch/) is a monitoring and observability service designed to help collect, analyze, and act on data from applications, infrastructure, and services running in AWS and on-premises environments. You can export telemetry data from your Tiger Cloud services with the time-series and analytics capability enabled to CloudWatch. See [Exported metrics](/docs/integrate/observability-alerting/exported-metrics) for the full list of default and additional metrics you can export. This pages explains how to export telemetry data from your Tiger Cloud service into CloudWatch by creating a Tiger Cloud data exporter, then attaching it to the service. This integration is available for [Scale and Enterprise](/docs/deploy/tiger-cloud/tiger-cloud-aws/pricing-and-account-management#features-included-in-each-pricing-plan) pricing tiers. ## Prerequisites for this integration guide To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service). - An [Amazon CloudWatch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/GettingSetup.html) account. Note This feature is currently not supported for Tiger Cloud on Microsoft Azure. ## Create a data exporter [Section titled “Create a data exporter”](#create-a-data-exporter) A Tiger Cloud data exporter sends telemetry data from a Tiger Cloud service to a third-party monitoring tool. You create an exporter on the [project level](/docs/deploy/tiger-cloud/tiger-cloud-aws/security/members), in the same AWS region as your service: 1. **In Tiger Console, open Exporters** 2. **Click `New exporter`** 3. **Select the data type and specify `AWS CloudWatch` for provider** ![Adding a CloudWatch data exporter in Tiger Console](/docs/_astro/tiger-cloud-integrations-cloudwatch.19Hs9eva_ezt85.webp) 4. **Provide your AWS CloudWatch configuration** * The AWS region must be the same for your Tiger Cloud exporter and AWS CloudWatch Log group. * The exporter name appears in Tiger Console, best practice is to make this name easily understandable. * For CloudWatch credentials, either use an [existing CloudWatch Log group](https://console.aws.amazon.com/cloudwatch/home#logsV2:log-groups) or [create a new one](https://console.aws.amazon.com/cloudwatch/home#logsV2:log-groups/create-log-group). If you're uncertain, use the default values. For more information, see [Working with log groups and log streams](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/Working-with-log-groups-and-streams.html). 5. **Choose the authentication method to use for the exporter** ![Configuring CloudWatch authentication in Tiger Console](/docs/_astro/tiger-cloud-integrations-cloudwatch-authentication.CBcUOfmq_1jCxT0.webp) * IAM role 1. In AWS, navigate to IAM > Identity providers, then click `Add provider`. 2. Update the new identity provider with your details. Set `Provider URL` to the [region where you are creating your exporter](/docs/integrate/observability-alerting/cloudwatch#reference). ![Creating an IAM OIDC identity provider in AWS](/docs/_astro/aws-create-iam-oicd-provider.BYYttbs9_Z4Cydc.webp) 3. Click `Add provider`. 4. In AWS, navigate to IAM > Roles, then click `Create role`. 5. Add your identity provider as a Web identity role and click `Next`. ![Creating a web identity IAM role in AWS](/docs/_astro/aws-create-role-web-identity.DnPD1ceR_Z1YqV2X.webp) 6. Set the following permission and trust policies: * Permission policy: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "logs:PutLogEvents", "logs:CreateLogGroup", "logs:CreateLogStream", "logs:DescribeLogStreams", "logs:DescribeLogGroups", "logs:PutRetentionPolicy", "xray:PutTraceSegments", "xray:PutTelemetryRecords", "xray:GetSamplingRules", "xray:GetSamplingTargets", "xray:GetSamplingStatisticSummaries", "ssm:GetParameters" ], "Resource": "*" } ] } ``` * Role with a Trust Policy: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam::12345678910:oidc-provider/irsa-oidc-discovery-prod.s3.us-east-1.amazonaws.com" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "irsa-oidc-discovery-prod.s3.us-east-1.amazonaws.com:aud": "sts.amazonaws.com" } } }, { "Sid": "Statement1", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::12345678910:role/my-exporter-role" }, "Action": "sts:AssumeRole" } ] } ``` 7. Click `Add role`. * CloudWatch credentials When you use CloudWatch credentials, you link an Identity and Access Management (IAM) user with access to CloudWatch only with your Tiger Cloud service: 1. Retrieve the user information from IAM > Users in AWS console. If you do not have an AWS user with access restricted to CloudWatch only, [create one](https://console.aws.amazon.com/iam/home#/users/create). For more information, see [Creating IAM users (console)](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html#id_users_create_console). 2. Enter the credentials for the AWS IAM user. AWS keys give access to your AWS services. To keep your AWS account secure, restrict users to the minimum required permissions. Always store your keys in a safe location. To avoid this issue, use the IAM role authentication method. 6. **Select the AWS Region your CloudWatch services run in** 7. **Optionally tick `PostgreSQL metrics` to export additional metrics, then click `Create exporter`** See [Exported metrics](/docs/integrate/observability-alerting/exported-metrics#postgresql-metrics) for the full list. ## Manage a data exporter [Section titled “Manage a data exporter”](#manage-a-data-exporter) This section shows you how to attach, edit, and delete a data exporter. ### Attach a data exporter to a Tiger Cloud service To send telemetry data to an external monitoring tool, you attach a data exporter to your Tiger Cloud service. You can attach only one exporter to a service. To attach an exporter: 1. **In Tiger Console, choose the service** 2. **Click `Operations` > `Exporters`** 3. **Select the exporter, then click `Attach exporter`** 4. **If you are attaching a first `Logs` data type exporter, restart the service** You can now monitor your service metrics. ### Edit a data exporter To update a data exporter: 1. **In Tiger Console, open Exporters** 2. **Next to the exporter you want to edit, click the menu > `Edit`** 3. **Edit the exporter fields and save your changes** ### Delete a data exporter To remove a data exporter that you no longer need: 1. **Disconnect the data exporter from your Tiger Cloud service** 1. In [Tiger Console](https://console.cloud.tigerdata.com/dashboard/services), choose the service. 2. Click `Operations` > `Exporters`. 3. Click the trash can icon next to the exporter. 4. Repeat for every service attached to the exporter you want to remove. The data exporter is now unattached from all services. However, it still exists in your project. 2. **Delete the exporter on the project level** 1. In Tiger Console, open [Exporters](https://console.cloud.tigerdata.com/dashboard/integrations) 2. Next to the exporter you want to edit, click menu > `Delete` 3. Confirm that you want to delete the data exporter. ## Reference [Section titled “Reference”](#reference) When you create the IAM OIDC provider, the URL must match the region you create the exporter in. It must be one of the following: | Region | Zone | Location | URL | | ---------------- | ------------- | -------------- | ------------------------------------------------------------------------- | | `ap-southeast-1` | Asia Pacific | Singapore | `irsa-oidc-discovery-prod-ap-southeast-1.s3.ap-southeast-1.amazonaws.com` | | `ap-southeast-2` | Asia Pacific | Sydney | `irsa-oidc-discovery-prod-ap-southeast-2.s3.ap-southeast-2.amazonaws.com` | | `ap-northeast-1` | Asia Pacific | Tokyo | `irsa-oidc-discovery-prod-ap-northeast-1.s3.ap-northeast-1.amazonaws.com` | | `ca-central-1` | Canada | Central | `irsa-oidc-discovery-prod-ca-central-1.s3.ca-central-1.amazonaws.com` | | `eu-central-1` | Europe | Frankfurt | `irsa-oidc-discovery-prod-eu-central-1.s3.eu-central-1.amazonaws.com` | | `eu-west-1` | Europe | Ireland | `irsa-oidc-discovery-prod-eu-west-1.s3.eu-west-1.amazonaws.com` | | `eu-west-2` | Europe | London | `irsa-oidc-discovery-prod-eu-west-2.s3.eu-west-2.amazonaws.com` | | `sa-east-1` | South America | São Paulo | `irsa-oidc-discovery-prod-sa-east-1.s3.sa-east-1.amazonaws.com` | | `us-east-1` | United States | North Virginia | `irsa-oidc-discovery-prod.s3.us-east-1.amazonaws.com` | | `us-east-2` | United States | Ohio | `irsa-oidc-discovery-prod-us-east-2.s3.us-east-2.amazonaws.com` | | `us-west-2` | United States | Oregon | `irsa-oidc-discovery-prod-us-west-2.s3.us-west-2.amazonaws.com` | # Integrate Datadog with Tiger Cloud > Export telemetry data and collect metrics to monitor your service performance [Datadog](https://www.datadoghq.com/) is a cloud-based monitoring and analytics platform that provides comprehensive visibility into applications, infrastructure, and systems through real-time monitoring, logging, and analytics. This page explains how to: * [Monitor Tiger Cloud service metrics with Datadog](#monitor-tiger-cloud-service-metrics-with-datadog) This integration is available for [Scale and Enterprise](/docs/deploy/tiger-cloud/tiger-cloud-aws/pricing-and-account-management#features-included-in-each-pricing-plan) pricing plans. * Configure Datadog Agent to collect metrics for your Tiger Cloud service This integration is available for all pricing plans. ## Prerequisites for this integration guide To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service). - Your [connection details](/docs/integrate/find-connection-details). * A [Datadog](https://www.datadoghq.com/) account. * Your [Datadog API key](https://docs.datadoghq.com/account_management/api-app-keys/#add-an-api-key-or-client-token). * [Datadog Agent](https://docs.datadoghq.com/getting_started/agent/#installation) installed. ## Monitor Tiger Cloud service metrics with Datadog [Section titled “Monitor Tiger Cloud service metrics with Datadog”](#monitor-tiger-cloud-service-metrics-with-datadog) Export telemetry data from your Tiger Cloud services with the time-series and analytics capability enabled to Datadog using a Tiger Cloud data exporter. See [Exported metrics](/docs/integrate/observability-alerting/exported-metrics) for the full list of default and additional metrics you can export. ### Create a data exporter [Section titled “Create a data exporter”](#create-a-data-exporter) A Tiger Cloud data exporter sends telemetry data from a Tiger Cloud service to a third-party monitoring tool. You create an exporter on the [project level](/docs/deploy/tiger-cloud/tiger-cloud-aws/security/members), in the same AWS region as your service: 1. **In Tiger Console, open Exporters** 2. **Click `New exporter`** 3. **Select `Metrics` for `Data type` and `Datadog` for provider** ![Adding a Datadog exporter in Tiger Console](/docs/_astro/tiger-cloud-integrations-datadog.DhcwoTmg_ZUvac9.webp) 4. **Choose your AWS region and provide the API key** The AWS region must be the same for your Tiger Cloud exporter and the Datadog provider. 5. **Set `Site` to your Datadog region** 6. **Export additional metrics** Optionally tick `PostgreSQL metrics` to export [additional metrics](/docs/integrate/observability-alerting/exported-metrics#postgresql-metrics), then click `Create exporter`. This section shows you how to attach, edit, and delete a data exporter. ### Attach a data exporter to a Tiger Cloud service To send telemetry data to an external monitoring tool, you attach a data exporter to your Tiger Cloud service. You can attach only one exporter to a service. To attach an exporter: 1. **In Tiger Console, choose the service** 2. **Click `Operations` > `Exporters`** 3. **Select the exporter, then click `Attach exporter`** 4. **If you are attaching a first `Logs` data type exporter, restart the service** You can now monitor your service metrics. ### Edit a data exporter To update a data exporter: 1. **In Tiger Console, open Exporters** 2. **Next to the exporter you want to edit, click the menu > `Edit`** 3. **Edit the exporter fields and save your changes** ### Delete a data exporter To remove a data exporter that you no longer need: 1. **Disconnect the data exporter from your Tiger Cloud service** 1. In [Tiger Console](https://console.cloud.tigerdata.com/dashboard/services), choose the service. 2. Click `Operations` > `Exporters`. 3. Click the trash can icon next to the exporter. 4. Repeat for every service attached to the exporter you want to remove. The data exporter is now unattached from all services. However, it still exists in your project. 2. **Delete the exporter on the project level** 1. In Tiger Console, open [Exporters](https://console.cloud.tigerdata.com/dashboard/integrations) 2. Next to the exporter you want to edit, click menu > `Delete` 3. Confirm that you want to delete the data exporter. ## Reference [Section titled “Reference”](#reference) When you create the IAM OIDC provider, the URL must match the region you create the exporter in. It must be one of the following: | Region | Zone | Location | URL | | ---------------- | ------------- | -------------- | ------------------------------------------------------------------------- | | `ap-southeast-1` | Asia Pacific | Singapore | `irsa-oidc-discovery-prod-ap-southeast-1.s3.ap-southeast-1.amazonaws.com` | | `ap-southeast-2` | Asia Pacific | Sydney | `irsa-oidc-discovery-prod-ap-southeast-2.s3.ap-southeast-2.amazonaws.com` | | `ap-northeast-1` | Asia Pacific | Tokyo | `irsa-oidc-discovery-prod-ap-northeast-1.s3.ap-northeast-1.amazonaws.com` | | `ca-central-1` | Canada | Central | `irsa-oidc-discovery-prod-ca-central-1.s3.ca-central-1.amazonaws.com` | | `eu-central-1` | Europe | Frankfurt | `irsa-oidc-discovery-prod-eu-central-1.s3.eu-central-1.amazonaws.com` | | `eu-west-1` | Europe | Ireland | `irsa-oidc-discovery-prod-eu-west-1.s3.eu-west-1.amazonaws.com` | | `eu-west-2` | Europe | London | `irsa-oidc-discovery-prod-eu-west-2.s3.eu-west-2.amazonaws.com` | | `sa-east-1` | South America | São Paulo | `irsa-oidc-discovery-prod-sa-east-1.s3.sa-east-1.amazonaws.com` | | `us-east-1` | United States | North Virginia | `irsa-oidc-discovery-prod.s3.us-east-1.amazonaws.com` | | `us-east-2` | United States | Ohio | `irsa-oidc-discovery-prod-us-east-2.s3.us-east-2.amazonaws.com` | | `us-west-2` | United States | Oregon | `irsa-oidc-discovery-prod-us-west-2.s3.us-west-2.amazonaws.com` | ## Configure Datadog Agent to collect metrics for your services [Section titled “Configure Datadog Agent to collect metrics for your services”](#configure-datadog-agent-to-collect-metrics-for-your-services) Datadog Agent includes a [PostgreSQL integration](https://docs.datadoghq.com/integrate/postgres/) that you use to collect detailed PostgreSQL database metrics about your Tiger Cloud services. 1. **Connect to your Tiger Cloud service** For Tiger Cloud, open an SQL editor in [Tiger Console](https://console.cloud.tigerdata.com/dashboard/services). For self-hosted TimescaleDB, use [`psql`](/docs/integrate/query-administration/psql). 2. **Add the `datadog` user to your Tiger Cloud service** ```sql create user datadog with password ''; ``` ```sql grant pg_monitor to datadog; ``` ```sql grant SELECT ON pg_stat_database to datadog; ``` 3. **Test the connection and rights for the datadog user** Update the following command with your [connection details](/docs/integrate/find-connection-details), then run it from the command line: ```bash psql "postgres://datadog:@:/tsdb?sslmode=require" -c \ "select * from pg_stat_database LIMIT(1);" \ && echo -e "\e[0;32mPostgres connection - OK\e[0m" || echo -e "\e[0;31mCannot connect to Postgres\e[0m" ``` You see the output from the `pg_stat_database` table, which means you have given the correct rights to `datadog`. 4. **Connect Datadog to your Tiger Cloud service** 1. Configure the [Datadog Agent PostgreSQL configuration file](https://docs.datadoghq.com/database_monitoring/setup_postgres/selfhosted?tab=postgres15); it is usually located on the Datadog Agent host at: * **Linux**: `/etc/datadog-agent/conf.d/postgres.d/conf.yaml` * **MacOS**: `/opt/datadog-agent/etc/conf.d/postgres.d/conf.yaml` * **Windows**: `C:\ProgramData\Datadog\conf.d\postgres.d\conf.yaml` 2. Integrate Datadog Agent with your Tiger Cloud service. Use your [connection details](/docs/integrate/find-connection-details) to update the following and add it to the Datadog Agent PostgreSQL configuration file: ```yaml init_config: instances: - host: port: username: datadog password: > dbname: tsdb disable_generic_tags: true ``` 5. **Add Tiger Cloud metrics** Tags make it easier to build Datadog dashboards that combine metrics from the Tiger Cloud data exporter and Datadog Agent. Use your [connection details](/docs/integrate/find-connection-details) to update the following and add it to `/datadog.yaml`: ```yaml tags: - project-id: - service-id: - region: ``` 6. **Restart Datadog Agent** See how to [Start, stop, and restart Datadog Agent](https://docs.datadoghq.com/agent/configuration/agent-commands/#start-stop-and-restart-the-agent). Metrics for your Tiger Cloud service are now visible in Datadog. Check the Datadog PostgreSQL integration documentation for a comprehensive list of [metrics](https://docs.datadoghq.com/integrate/postgres/?tab=host#metrics) collected. # Metrics exported by Tiger Cloud exporters > Reference for system and PostgreSQL metrics exported by Tiger Cloud This page provides a list of metrics you can export with all Tiger Cloud exporters. You can filter them by `project_id`, `service_id`, `role`, and `region`. Note The following limitations apply for replica metrics: * For HA replicas, metrics are only exported for the primary and one HA replica. * For read replica sets, metrics are exported for the primary/`standby_leader` and one read replica. ## Tiger Cloud metrics [Section titled “Tiger Cloud metrics”](#tiger-cloud-metrics) These are the metrics exported by default by every Tiger Cloud exporter. | Metric | Description | | --------------------------------------------- | -------------------------------------------------------------------- | | `timescale_cloud_system_cpu_total_millicores` | Total CPU capacity available to the service, in millicores. | | `timescale_cloud_system_cpu_usage_millicores` | Current CPU usage by the service, in millicores. | | `timescale_cloud_system_disk_io_read_bytes` | Total bytes read from disk. | | `timescale_cloud_system_disk_io_read_ops` | Total number of disk read operations. | | `timescale_cloud_system_disk_io_total_bytes` | Total bytes of all disk I/O (reads and writes combined). | | `timescale_cloud_system_disk_io_total_ops` | Total number of all disk I/O operations (reads and writes combined). | | `timescale_cloud_system_disk_io_write_bytes` | Total bytes written to disk. | | `timescale_cloud_system_disk_io_write_ops` | Total number of disk write operations. | | `timescale_cloud_system_disk_usage_bytes` | Current disk space used by the service, in bytes. | | `timescale_cloud_system_memory_total_bytes` | Total memory available to the service, in bytes. | | `timescale_cloud_system_memory_usage_bytes` | Current memory usage by the service, in bytes. | ## PostgreSQL metrics [Section titled “PostgreSQL metrics”](#postgresql-metrics) These additional metrics are collected if you enable **PostgreSQL metrics** when creating Tiger Cloud exporters. All metrics have a scrape interval of 15 seconds. ### Exporter metrics [Section titled “Exporter metrics”](#exporter-metrics) These metrics show the health and status of the metrics collection process, including scrape duration, errors, and total scrape count. | Metric | Description | | ------------------------------------------ | ----------------------------------------------------------------------------------------------------- | | `pg_exporter_last_scrape_duration_seconds` | Duration of the last scrape of metrics from PostgreSQL. | | `pg_exporter_last_scrape_error` | Whether the last scrape of metrics from PostgreSQL resulted in an error (1 for error, 0 for success). | | `pg_exporter_scrapes_total` | Total number of times PostgreSQL was scraped for metrics. | | `pg_exporter_user_queries_load_error` | Whether the user queries file was loaded and parsed successfully (1 for error, 0 for success). | ### Lock metrics [Section titled “Lock metrics”](#lock-metrics) These metrics track current lock activity in the database, helping you detect lock contention that may be blocking queries. | Metric | Description | | ---------------- | ---------------- | | `pg_locks_count` | Number of locks. | ### Replication metrics [Section titled “Replication metrics”](#replication-metrics) These metrics track the status and lag of replication slots, helping you monitor whether replicas are keeping up with the primary. | Metric | Description | | -------------------------------------- | -------------------------------------- | | `pg_replication_slots_active` | Flag indicating if the slot is active. | | `pg_replication_slots_pg_wal_lsn_diff` | Replication lag in bytes. | ### PG settings metrics [Section titled “PG settings metrics”](#pg-settings-metrics) These metrics expose the current values of PostgreSQL configuration parameters as numeric values, covering memory limits, planner costs, autovacuum tuning, WAL configuration, and more. | Metric | Description | | ------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `pg_settings_allow_alter_system` | Allows running the ALTER SYSTEM command. | | `pg_settings_allow_in_place_tablespaces` | Allows tablespaces directly inside `pg_tblspc`, for testing. | | `pg_settings_allow_system_table_mods` | Allows modifications of the structure of system tables. | | `pg_settings_archive_timeout_seconds` | Sets the amount of time to wait before forcing a switch to the next WAL file. Units converted to seconds. | | `pg_settings_array_nulls` | Enables input of NULL elements in arrays. | | `pg_settings_authentication_timeout_seconds` | Sets the maximum allowed time to complete client authentication. Units converted to seconds. | | `pg_settings_auto_explain_log_analyze` | Use EXPLAIN ANALYZE for plan logging. | | `pg_settings_auto_explain_log_buffers` | Log buffers usage. | | `pg_settings_auto_explain_log_min_duration_seconds` | Sets the minimum execution time above which plans will be logged. Units converted to seconds. | | `pg_settings_auto_explain_log_nested_statements` | Log nested statements. | | `pg_settings_auto_explain_log_parameter_max_length_bytes` | Sets the maximum length of query parameter values to log. Units converted to bytes. | | `pg_settings_auto_explain_log_settings` | Log modified configuration parameters affecting query planning. | | `pg_settings_auto_explain_log_timing` | Collect timing data, not just row counts. | | `pg_settings_auto_explain_log_triggers` | Include trigger statistics in plans. | | `pg_settings_auto_explain_log_verbose` | Use EXPLAIN VERBOSE for plan logging. | | `pg_settings_auto_explain_log_wal` | Log WAL usage. | | `pg_settings_auto_explain_sample_rate` | Fraction of queries to process. | | `pg_settings_autovacuum` | Starts the autovacuum subprocess. | | `pg_settings_autovacuum_analyze_scale_factor` | Number of tuple inserts, updates, or deletes prior to analyze as a fraction of `reltuples`. | | `pg_settings_autovacuum_analyze_threshold` | Minimum number of tuple inserts, updates, or deletes prior to analyze. | | `pg_settings_autovacuum_freeze_max_age` | Age at which to autovacuum a table to prevent transaction ID wraparound. | | `pg_settings_autovacuum_max_workers` | Sets the maximum number of simultaneously running autovacuum worker processes. | | `pg_settings_autovacuum_multixact_freeze_max_age` | Multixact age at which to autovacuum a table to prevent multixact wraparound. | | `pg_settings_autovacuum_naptime_seconds` | Time to sleep between autovacuum runs. Units converted to seconds. | | `pg_settings_autovacuum_vacuum_cost_delay_seconds` | Vacuum cost delay in milliseconds, for autovacuum. Units converted to seconds. | | `pg_settings_autovacuum_vacuum_cost_limit` | Vacuum cost amount available before napping, for autovacuum. | | `pg_settings_autovacuum_vacuum_insert_scale_factor` | Number of tuple inserts prior to vacuum as a fraction of `reltuples`. | | `pg_settings_autovacuum_vacuum_insert_threshold` | Minimum number of tuple inserts prior to vacuum. | | `pg_settings_autovacuum_vacuum_max_threshold` | Maximum number of tuple updates or deletes prior to vacuum. | | `pg_settings_autovacuum_vacuum_scale_factor` | Number of tuple updates or deletes prior to vacuum as a fraction of `reltuples`. | | `pg_settings_autovacuum_vacuum_threshold` | Minimum number of tuple updates or deletes prior to vacuum. | | `pg_settings_autovacuum_work_mem_bytes` | Sets the maximum memory to be used by each autovacuum worker process. Units converted to bytes. | | `pg_settings_autovacuum_worker_slots` | Sets the number of backend slots to allocate for autovacuum workers. | | `pg_settings_backend_flush_after_bytes` | Number of pages after which previously performed writes are flushed to disk. Units converted to bytes. | | `pg_settings_bgwriter_delay_seconds` | Background writer sleep time between rounds. Units converted to seconds. | | `pg_settings_bgwriter_flush_after_bytes` | Number of pages after which previously performed writes are flushed to disk. Units converted to bytes. | | `pg_settings_bgwriter_lru_maxpages` | Background writer maximum number of LRU pages to flush per round. | | `pg_settings_bgwriter_lru_multiplier` | Multiple of the average buffer usage to free per round. | | `pg_settings_block_size` | Shows the size of a disk block. | | `pg_settings_bonjour` | Enables advertising the server via Bonjour. | | `pg_settings_check_function_bodies` | Check routine bodies during CREATE FUNCTION and CREATE PROCEDURE. | | `pg_settings_checkpoint_completion_target` | Time spent flushing dirty buffers during checkpoint, as fraction of checkpoint interval. | | `pg_settings_checkpoint_flush_after_bytes` | Number of pages after which previously performed writes are flushed to disk. Units converted to bytes. | | `pg_settings_checkpoint_timeout_seconds` | Sets the maximum time between automatic WAL checkpoints. Units converted to seconds. | | `pg_settings_checkpoint_warning_seconds` | Sets the maximum time before warning if checkpoints triggered by WAL volume happen too frequently. Units converted to seconds. | | `pg_settings_client_connection_check_interval_seconds` | Sets the time interval between checks for disconnection while running queries. Units converted to seconds. | | `pg_settings_commit_delay` | Sets the delay in microseconds between transaction commit and flushing WAL to disk. | | `pg_settings_commit_siblings` | Sets the minimum number of concurrent open transactions required before performing `commit_delay`. | | `pg_settings_commit_timestamp_buffers_bytes` | Sets the size of the dedicated buffer pool used for the commit timestamp cache. Units converted to bytes. | | `pg_settings_cpu_index_tuple_cost` | Sets the planner's estimate of the cost of processing each index entry during an index scan. | | `pg_settings_cpu_operator_cost` | Sets the planner's estimate of the cost of processing each operator or function call. | | `pg_settings_cpu_tuple_cost` | Sets the planner's estimate of the cost of processing each tuple (row). | | `pg_settings_cursor_tuple_fraction` | Sets the planner's estimate of the fraction of a cursor's rows that will be retrieved. | | `pg_settings_data_checksums` | Shows whether data checksums are turned on for this cluster. | | `pg_settings_data_directory_mode` | Shows the mode of the data directory. | | `pg_settings_data_sync_retry` | Whether to continue running after a failure to sync data files. | | `pg_settings_deadlock_timeout_seconds` | Sets the time to wait on a lock before checking for deadlock. Units converted to seconds. | | `pg_settings_debug_assertions` | Shows whether the running server has assertion checks enabled. | | `pg_settings_debug_discard_caches` | Aggressively flush system caches for debugging purposes. | | `pg_settings_debug_pretty_print` | Indents parse and plan tree displays. | | `pg_settings_debug_print_parse` | Logs each query's parse tree. | | `pg_settings_debug_print_plan` | Logs each query's execution plan. | | `pg_settings_debug_print_rewritten` | Logs each query's rewritten parse tree. | | `pg_settings_default_statistics_target` | Sets the default statistics target. | | `pg_settings_default_transaction_deferrable` | Sets the default deferrable status of new transactions. | | `pg_settings_default_transaction_read_only` | Sets the default read-only status of new transactions. | | `pg_settings_effective_cache_size_bytes` | Sets the planner's assumption about the total size of the data caches. Units converted to bytes. | | `pg_settings_effective_io_concurrency` | Number of simultaneous requests that can be handled efficiently by the disk subsystem. | | `pg_settings_enable_async_append` | Enables the planner's use of async append plans. | | `pg_settings_enable_bitmapscan` | Enables the planner's use of bitmap-scan plans. | | `pg_settings_enable_distinct_reordering` | Enables reordering of DISTINCT keys. | | `pg_settings_enable_gathermerge` | Enables the planner's use of gather merge plans. | | `pg_settings_enable_group_by_reordering` | Enables reordering of GROUP BY keys. | | `pg_settings_enable_hashagg` | Enables the planner's use of hashed aggregation plans. | | `pg_settings_enable_hashjoin` | Enables the planner's use of hash join plans. | | `pg_settings_enable_incremental_sort` | Enables the planner's use of incremental sort steps. | | `pg_settings_enable_indexonlyscan` | Enables the planner's use of index-only-scan plans. | | `pg_settings_enable_indexscan` | Enables the planner's use of index-scan plans. | | `pg_settings_enable_material` | Enables the planner's use of materialization. | | `pg_settings_enable_memoize` | Enables the planner's use of memoization. | | `pg_settings_enable_mergejoin` | Enables the planner's use of merge join plans. | | `pg_settings_enable_nestloop` | Enables the planner's use of nested-loop join plans. | | `pg_settings_enable_parallel_append` | Enables the planner's use of parallel append plans. | | `pg_settings_enable_parallel_hash` | Enables the planner's use of parallel hash plans. | | `pg_settings_enable_partition_pruning` | Enables plan-time and execution-time partition pruning. | | `pg_settings_enable_partitionwise_aggregate` | Enables partitionwise aggregation and grouping. | | `pg_settings_enable_partitionwise_join` | Enables partitionwise join. | | `pg_settings_enable_presorted_aggregate` | Enables the planner's ability to produce plans that provide presorted input for ORDER BY / DISTINCT aggregate functions. | | `pg_settings_enable_self_join_elimination` | Enables removal of unique self-joins. | | `pg_settings_enable_seqscan` | Enables the planner's use of sequential-scan plans. | | `pg_settings_enable_sort` | Enables the planner's use of explicit sort steps. | | `pg_settings_enable_tidscan` | Enables the planner's use of TID scan plans. | | `pg_settings_escape_string_warning` | Warn about backslash escapes in ordinary string literals. | | `pg_settings_event_triggers` | Enables event triggers. | | `pg_settings_exit_on_error` | Terminate session on any error. | | `pg_settings_extra_float_digits` | Sets the number of digits displayed for floating-point values. | | `pg_settings_extwlist_extname_from_filename` | Flag allowing a lookup of extension name in custom script filename. | | `pg_settings_from_collapse_limit` | Sets the FROM-list size beyond which subqueries are not collapsed. | | `pg_settings_fsync` | Forces synchronization of updates to disk. | | `pg_settings_full_page_writes` | Writes full pages to WAL when first modified after a checkpoint. | | `pg_settings_geqo` | Enables genetic query optimization. | | `pg_settings_geqo_effort` | GEQO: effort is used to set the default for other GEQO parameters. | | `pg_settings_geqo_generations` | GEQO: number of iterations of the algorithm. | | `pg_settings_geqo_pool_size` | GEQO: number of individuals in the population. | | `pg_settings_geqo_seed` | GEQO: seed for random path selection. | | `pg_settings_geqo_selection_bias` | GEQO: selective pressure within the population. | | `pg_settings_geqo_threshold` | Sets the threshold of FROM items beyond which GEQO is used. | | `pg_settings_gin_fuzzy_search_limit` | Sets the maximum allowed result for exact search by GIN. | | `pg_settings_gin_pending_list_limit_bytes` | Sets the maximum size of the pending list for GIN index. Units converted to bytes. | | `pg_settings_gss_accept_delegation` | Sets whether GSSAPI delegation should be accepted from the client. | | `pg_settings_hash_mem_multiplier` | Multiple of `work_mem` to use for hash tables. | | `pg_settings_hot_standby` | Allows connections and queries during recovery. | | `pg_settings_hot_standby_feedback` | Allows feedback from a hot standby to the primary that will avoid query conflicts. | | `pg_settings_huge_page_size_bytes` | The size of huge page that should be requested. Units converted to bytes. | | `pg_settings_idle_in_transaction_session_timeout_seconds` | Sets the maximum allowed idle time between queries, when in a transaction. Units converted to seconds. | | `pg_settings_idle_replication_slot_timeout_seconds` | Sets the duration a replication slot can remain idle before it is invalidated. Units converted to seconds. | | `pg_settings_idle_session_timeout_seconds` | Sets the maximum allowed idle time between queries, when not in a transaction. Units converted to seconds. | | `pg_settings_ignore_checksum_failure` | Continues processing after a checksum failure. | | `pg_settings_ignore_invalid_pages` | Continues recovery after an invalid pages failure. | | `pg_settings_ignore_system_indexes` | Disables reading from system indexes. | | `pg_settings_in_hot_standby` | Shows whether hot standby is currently active. | | `pg_settings_integer_datetimes` | Shows whether datetimes are integer based. | | `pg_settings_io_combine_limit_bytes` | Limit on the size of data reads and writes. Units converted to bytes. | | `pg_settings_io_max_combine_limit_bytes` | Server-wide limit that clamps `io_combine_limit`. Units converted to bytes. | | `pg_settings_io_max_concurrency` | Max number of IOs that one process can execute simultaneously. | | `pg_settings_io_workers` | Number of IO worker processes, for `io_method=worker`. | | `pg_settings_jit` | Allow JIT compilation. | | `pg_settings_jit_above_cost` | Perform JIT compilation if query is more expensive. | | `pg_settings_jit_debugging_support` | Register JIT-compiled functions with debugger. | | `pg_settings_jit_dump_bitcode` | Write out LLVM bitcode to facilitate JIT debugging. | | `pg_settings_jit_expressions` | Allow JIT compilation of expressions. | | `pg_settings_jit_inline_above_cost` | Perform JIT inlining if query is more expensive. | | `pg_settings_jit_optimize_above_cost` | Optimize JIT-compiled functions if query is more expensive. | | `pg_settings_jit_profiling_support` | Register JIT-compiled functions with perf profiler. | | `pg_settings_jit_tuple_deforming` | Allow JIT compilation of tuple deforming. | | `pg_settings_join_collapse_limit` | Sets the FROM-list size beyond which JOIN constructs are not flattened. | | `pg_settings_krb_caseins_users` | Sets whether Kerberos and GSSAPI user names should be treated as case-insensitive. | | `pg_settings_lo_compat_privileges` | Enables backward compatibility mode for privilege checks on large objects. | | `pg_settings_lock_timeout_seconds` | Sets the maximum allowed duration of any wait for a lock. Units converted to seconds. | | `pg_settings_log_autovacuum_min_duration_seconds` | Sets the minimum execution time above which autovacuum actions will be logged. Units converted to seconds. | | `pg_settings_log_checkpoints` | Logs each checkpoint. | | `pg_settings_log_disconnections` | Logs end of a session, including duration. | | `pg_settings_log_duration` | Logs the duration of each completed SQL statement. | | `pg_settings_log_executor_stats` | Writes executor performance statistics to the server log. | | `pg_settings_log_file_mode` | Sets the file permissions for log files. | | `pg_settings_log_hostname` | Logs the host name in the connection logs. | | `pg_settings_log_lock_failures` | Logs lock failures. | | `pg_settings_log_lock_waits` | Logs long lock waits. | | `pg_settings_log_min_duration_sample_seconds` | Sets the minimum execution time above which a sample of statements will be logged. Sampling is determined by `log_statement_sample_rate`. Units converted to seconds. | | `pg_settings_log_min_duration_statement_seconds` | Sets the minimum execution time above which all statements will be logged. Units converted to seconds. | | `pg_settings_log_parameter_max_length_bytes` | Sets the maximum length in bytes of data logged for bind parameter values when logging statements. Units converted to bytes. | | `pg_settings_log_parameter_max_length_on_error_bytes` | Sets the maximum length in bytes of data logged for bind parameter values when logging statements, on error. Units converted to bytes. | | `pg_settings_log_parser_stats` | Writes parser performance statistics to the server log. | | `pg_settings_log_planner_stats` | Writes planner performance statistics to the server log. | | `pg_settings_log_recovery_conflict_waits` | Logs standby recovery conflict waits. | | `pg_settings_log_replication_commands` | Logs each replication command. | | `pg_settings_log_rotation_age_seconds` | Sets the amount of time to wait before forcing log file rotation. Units converted to seconds. | | `pg_settings_log_rotation_size_bytes` | Sets the maximum size a log file can reach before being rotated. Units converted to bytes. | | `pg_settings_log_startup_progress_interval_seconds` | Time between progress updates for long-running startup operations. Units converted to seconds. | | `pg_settings_log_statement_sample_rate` | Fraction of statements exceeding `log_min_duration_sample` to be logged. | | `pg_settings_log_statement_stats` | Writes cumulative performance statistics to the server log. | | `pg_settings_log_temp_files_bytes` | Log the use of temporary files larger than this number of kilobytes. Units converted to bytes. | | `pg_settings_log_transaction_sample_rate` | Sets the fraction of transactions from which to log all statements. | | `pg_settings_log_truncate_on_rotation` | Truncate existing log files of same name during log rotation. | | `pg_settings_logging_collector` | Start a subprocess to capture stderr, csvlog and/or jsonlog into log files. | | `pg_settings_logical_decoding_work_mem_bytes` | Sets the maximum memory to be used for logical decoding. Units converted to bytes. | | `pg_settings_maintenance_io_concurrency` | A variant of `effective_io_concurrency` that is used for maintenance work. | | `pg_settings_maintenance_work_mem_bytes` | Sets the maximum memory to be used for maintenance operations. Units converted to bytes. | | `pg_settings_max_active_replication_origins` | Sets the maximum number of active replication origins. | | `pg_settings_max_connections` | Sets the maximum number of concurrent connections. | | `pg_settings_max_files_per_process` | Sets the maximum number of files each server process is allowed to open simultaneously. | | `pg_settings_max_function_args` | Shows the maximum number of function arguments. | | `pg_settings_max_identifier_length` | Shows the maximum identifier length. | | `pg_settings_max_index_keys` | Shows the maximum number of index keys. | | `pg_settings_max_locks_per_transaction` | Sets the maximum number of locks per transaction. | | `pg_settings_max_logical_replication_workers` | Maximum number of logical replication worker processes. | | `pg_settings_max_notify_queue_pages` | Sets the maximum number of allocated pages for NOTIFY / LISTEN queue. | | `pg_settings_max_parallel_apply_workers_per_subscription` | Maximum number of parallel apply workers per subscription. | | `pg_settings_max_parallel_maintenance_workers` | Sets the maximum number of parallel processes per maintenance operation. | | `pg_settings_max_parallel_workers` | Sets the maximum number of parallel workers that can be active at one time. | | `pg_settings_max_parallel_workers_per_gather` | Sets the maximum number of parallel processes per executor node. | | `pg_settings_max_pred_locks_per_page` | Sets the maximum number of predicate-locked tuples per page. | | `pg_settings_max_pred_locks_per_relation` | Sets the maximum number of predicate-locked pages and tuples per relation. | | `pg_settings_max_pred_locks_per_transaction` | Sets the maximum number of predicate locks per transaction. | | `pg_settings_max_prepared_transactions` | Sets the maximum number of simultaneously prepared transactions. | | `pg_settings_max_replication_slots` | Sets the maximum number of simultaneously defined replication slots. | | `pg_settings_max_slot_wal_keep_size_bytes` | Sets the maximum WAL size that can be reserved by replication slots. Units converted to bytes. | | `pg_settings_max_stack_depth_bytes` | Sets the maximum stack depth, in kilobytes. Units converted to bytes. | | `pg_settings_max_standby_archive_delay_seconds` | Sets the maximum delay before canceling queries when a hot standby server is processing archived WAL data. Units converted to seconds. | | `pg_settings_max_standby_streaming_delay_seconds` | Sets the maximum delay before canceling queries when a hot standby server is processing streamed WAL data. Units converted to seconds. | | `pg_settings_max_sync_workers_per_subscription` | Maximum number of table synchronization workers per subscription. | | `pg_settings_max_wal_senders` | Sets the maximum number of simultaneously running WAL sender processes. | | `pg_settings_max_wal_size_bytes` | Sets the WAL size that triggers a checkpoint. Units converted to bytes. | | `pg_settings_max_worker_processes` | Maximum number of concurrent worker processes. | | `pg_settings_md5_password_warnings` | Enables deprecation warnings for MD5 passwords. | | `pg_settings_mem_guard_block` | Block memory allocations in query executing processes if the usage is above limit. | | `pg_settings_mem_guard_enable_auth_hook` | The use the authentication hook to indicate if we should block a pid. | | `pg_settings_mem_guard_enable_explain` | Enable collection of data in explain plans. | | `pg_settings_mem_guard_limit_bytes` | The memory limit for all processes in the container. Units converted to bytes. | | `pg_settings_min_dynamic_shared_memory_bytes` | Amount of dynamic shared memory reserved at startup. Units converted to bytes. | | `pg_settings_min_parallel_index_scan_size_bytes` | Sets the minimum amount of index data for a parallel scan. Units converted to bytes. | | `pg_settings_min_parallel_table_scan_size_bytes` | Sets the minimum amount of table data for a parallel scan. Units converted to bytes. | | `pg_settings_min_wal_size_bytes` | Sets the minimum size to shrink the WAL to. Units converted to bytes. | | `pg_settings_multixact_member_buffers_bytes` | Sets the size of the dedicated buffer pool used for the MultiXact member cache. Units converted to bytes. | | `pg_settings_multixact_offset_buffers_bytes` | Sets the size of the dedicated buffer pool used for the MultiXact offset cache. Units converted to bytes. | | `pg_settings_notify_buffers_bytes` | Sets the size of the dedicated buffer pool used for the LISTEN/NOTIFY message cache. Units converted to bytes. | | `pg_settings_num_os_semaphores` | Shows the number of semaphores required for the server. | | `pg_settings_parallel_leader_participation` | Controls whether Gather and Gather Merge also run subplans. | | `pg_settings_parallel_setup_cost` | Sets the planner's estimate of the cost of starting up worker processes for parallel query. | | `pg_settings_parallel_tuple_cost` | Sets the planner's estimate of the cost of passing each tuple (row) from worker to leader backend. | | `pg_settings_pg_prewarm_autoprewarm` | Starts the autoprewarm worker. | | `pg_settings_pg_prewarm_autoprewarm_interval_seconds` | Sets the interval between dumps of shared buffers. Units converted to seconds. | | `pg_settings_pg_stat_statements_max` | Sets the maximum number of statements tracked by `pg_stat_statements`. | | `pg_settings_pg_stat_statements_save` | Save `pg_stat_statements` statistics across server shutdowns. | | `pg_settings_pg_stat_statements_track_planning` | Selects whether planning duration is tracked by `pg_stat_statements`. | | `pg_settings_pg_stat_statements_track_utility` | Selects whether utility commands are tracked by `pg_stat_statements`. | | `pg_settings_pgaudit_log_catalog` | Specifies that session logging should be enabled in the case where all relations in a statement are in `pg_catalog`. Disabling this setting will reduce noise in the log from tools like psql and PgAdmin that query the catalog heavily. | | `pg_settings_pgaudit_log_client` | Specifies whether audit messages should be visible to the client. This setting should generally be left disabled but may be useful for debugging or other purposes. | | `pg_settings_pgaudit_log_parameter` | Specifies that audit logging should include the parameters that were passed with the statement. When parameters are present they will be included in CSV format after the statement text. | | `pg_settings_pgaudit_log_parameter_max_size` | Specifies, in bytes, the maximum length of variable-length parameters to log. If 0 (the default), parameters are not checked for size. If set, when the size of the parameter is longer than the setting, the value in the audit log is replaced with a placeholder. Note that for character types, the length is in bytes for the parameter's encoding, not characters. | | `pg_settings_pgaudit_log_relation` | Specifies whether session audit logging should create a separate log entry for each relation referenced in a SELECT or DML statement. This is a useful shortcut for exhaustive logging without using object audit logging. | | `pg_settings_pgaudit_log_rows` | Specifies whether logging will include the rows retrieved or affected by a statement. | | `pg_settings_pgaudit_log_statement` | Specifies whether logging will include the statement text and parameters. Depending on requirements, the full statement text might not be required in the audit log. | | `pg_settings_pgaudit_log_statement_once` | Specifies whether logging will include the statement text and parameters with the first log entry for a statement/substatement combination or with every entry. Enabling this setting will result in less verbose logging but may make it more difficult to determine the statement that generated a log entry, though the statement/substatement pair along with the process id should suffice to identify the statement text logged with a previous entry. | | `pg_settings_port` | Sets the TCP port the server listens on. | | `pg_settings_post_auth_delay_seconds` | Sets the amount of time to wait after authentication on connection startup. Units converted to seconds. | | `pg_settings_pre_auth_delay_seconds` | Sets the amount of time to wait before authentication on connection startup. Units converted to seconds. | | `pg_settings_quote_all_identifiers` | When generating SQL fragments, quote all identifiers. | | `pg_settings_random_page_cost` | Sets the planner's estimate of the cost of a nonsequentially fetched disk page. | | `pg_settings_recovery_min_apply_delay_seconds` | Sets the minimum delay for applying changes during recovery. Units converted to seconds. | | `pg_settings_recovery_target_inclusive` | Sets whether to include or exclude transaction with recovery target. | | `pg_settings_recursive_worktable_factor` | Sets the planner's estimate of the average size of a recursive query's working table. | | `pg_settings_remove_temp_files_after_crash` | Remove temporary files after backend crash. | | `pg_settings_reserved_connections` | Sets the number of connection slots reserved for roles with privileges of `pg_use_reserved_connections`. | | `pg_settings_restart_after_crash` | Reinitialize server after backend crash. | | `pg_settings_row_security` | Enables row security. | | `pg_settings_scram_iterations` | Sets the iteration count for SCRAM secret generation. | | `pg_settings_segment_size_bytes` | Shows the number of pages per disk file. Units converted to bytes. | | `pg_settings_send_abort_for_crash` | Send SIGABRT not SIGQUIT to child processes after backend crash. | | `pg_settings_send_abort_for_kill` | Send SIGABRT not SIGKILL to stuck child processes. | | `pg_settings_seq_page_cost` | Sets the planner's estimate of the cost of a sequentially fetched disk page. | | `pg_settings_serializable_buffers_bytes` | Sets the size of the dedicated buffer pool used for the serializable transaction cache. Units converted to bytes. | | `pg_settings_server_version_num` | Shows the server version as an integer. | | `pg_settings_shared_buffers_bytes` | Sets the number of shared memory buffers used by the server. Units converted to bytes. | | `pg_settings_shared_memory_size_bytes` | Shows the size of the server's main shared memory area (rounded up to the nearest MB). Units converted to bytes. | | `pg_settings_shared_memory_size_in_huge_pages` | Shows the number of huge pages needed for the main shared memory area. | | `pg_settings_ssl` | Enables SSL connections. | | `pg_settings_ssl_passphrase_command_supports_reload` | Controls whether `ssl_passphrase_command` is called during server reload. | | `pg_settings_ssl_prefer_server_ciphers` | Give priority to server ciphersuite order. | | `pg_settings_standard_conforming_strings` | Causes '...' strings to treat backslashes literally. | | `pg_settings_statement_timeout_seconds` | Sets the maximum allowed duration of any statement. Units converted to seconds. | | `pg_settings_subtransaction_buffers_bytes` | Sets the size of the dedicated buffer pool used for the subtransaction cache. Units converted to bytes. | | `pg_settings_summarize_wal` | Starts the WAL summarizer process to enable incremental backup. | | `pg_settings_superuser_reserved_connections` | Sets the number of connection slots reserved for superusers. | | `pg_settings_sync_replication_slots` | Enables a physical standby to synchronize logical failover replication slots from the primary server. | | `pg_settings_synchronize_seqscans` | Enables synchronized sequential scans. | | `pg_settings_syslog_sequence_numbers` | Add sequence number to syslog messages to avoid duplicate suppression. | | `pg_settings_syslog_split_messages` | Split messages sent to syslog by lines and to fit into 1024 bytes. | | `pg_settings_tcp_keepalives_count` | Maximum number of TCP keepalive retransmits. | | `pg_settings_tcp_keepalives_idle_seconds` | Time between issuing TCP keepalives. Units converted to seconds. | | `pg_settings_tcp_keepalives_interval_seconds` | Time between TCP keepalive retransmits. Units converted to seconds. | | `pg_settings_tcp_user_timeout_seconds` | TCP user timeout. Units converted to seconds. | | `pg_settings_temp_buffers_bytes` | Sets the maximum number of temporary buffers used by each session. Units converted to bytes. | | `pg_settings_temp_file_limit_bytes` | Limits the total size of all temporary files used by each process. Units converted to bytes. | | `pg_settings_timescaledb_bgw_launcher_poll_time` | Launcher timeout value in milliseconds. | | `pg_settings_timescaledb_bgw_scheduler_restart_time_seconds` | Restart time for scheduler. Units converted to seconds. | | `pg_settings_timescaledb_disable_load` | Disable the loading of the actual extension. | | `pg_settings_timescaledb_lake_disable_load` | Disable the loading of the actual extension. | | `pg_settings_timescaledb_max_background_workers` | Maximum background worker processes allocated to TimescaleDB | | `pg_settings_timescaledb_osm_disable_load` | Disable the loading of the actual extension. | | `pg_settings_trace_connection_negotiation` | Logs details of pre-authentication connection handshake. | | `pg_settings_trace_notify` | Generates debugging output for LISTEN and NOTIFY. | | `pg_settings_trace_sort` | Emit information about resource usage in sorting. | | `pg_settings_track_activities` | Collects information about executing commands. | | `pg_settings_track_activity_query_size_bytes` | Sets the size reserved for `pg_stat_activity`.query, in bytes. Units converted to bytes. | | `pg_settings_track_commit_timestamp` | Collects transaction commit time. | | `pg_settings_track_cost_delay_timing` | Collects timing statistics for cost-based vacuum delay. | | `pg_settings_track_counts` | Collects statistics on database activity. | | `pg_settings_track_io_timing` | Collects timing statistics for database I/O activity. | | `pg_settings_track_wal_io_timing` | Collects timing statistics for WAL I/O activity. | | `pg_settings_transaction_buffers_bytes` | Sets the size of the dedicated buffer pool used for the transaction status cache. Units converted to bytes. | | `pg_settings_transaction_deferrable` | Whether to defer a read-only serializable transaction until it can be executed with no possible serialization failures. | | `pg_settings_transaction_read_only` | Sets the current transaction's read-only status. | | `pg_settings_transaction_timeout_seconds` | Sets the maximum allowed duration of any transaction within a session (not a prepared transaction). Units converted to seconds. | | `pg_settings_transform_null_equals` | Treats "expr=NULL" as "expr IS NULL". | | `pg_settings_ts_stat_statements_enable_auth_hook` | Selects whether the auth hook is intercepted by ts\_stat\_statements. | | `pg_settings_ts_stat_statements_enable_log_hook` | Selects whether the log hook is intercepted by ts\_stat\_statements. | | `pg_settings_ts_stat_statements_enable_mem_guard_stats` | Enable reading memory stats from mem\_guard. | | `pg_settings_ts_stat_statements_enable_regression_debug` | Enable debug messages for regression tests. | | `pg_settings_ts_stat_statements_enable_system_stats` | Enable system stat collections. | | `pg_settings_ts_stat_statements_explain_bound` | When tracking, capture an explain plan for queries running longer than this many milliseconds. | | `pg_settings_ts_stat_statements_track_planning` | Selects whether planning duration is tracked by ts\_stat\_statements. | | `pg_settings_ts_stat_statements_track_utility` | Selects whether utility commands are tracked by ts\_stat\_statements. | | `pg_settings_tsdb_admin_read_only_connection` | Enable immutable read-only mode for this connection. | | `pg_settings_tsdb_admin_read_only_role` | Enable immutable read-only mode for this role. | | `pg_settings_unix_socket_permissions` | Sets the access permissions of the Unix-domain socket. | | `pg_settings_update_process_title` | Updates the process title to show the active SQL command. | | `pg_settings_vacuum_buffer_usage_limit_bytes` | Sets the buffer pool size for VACUUM, ANALYZE, and autovacuum. Units converted to bytes. | | `pg_settings_vacuum_cost_delay_seconds` | Vacuum cost delay in milliseconds. Units converted to seconds. | | `pg_settings_vacuum_cost_limit` | Vacuum cost amount available before napping. | | `pg_settings_vacuum_cost_page_dirty` | Vacuum cost for a page dirtied by vacuum. | | `pg_settings_vacuum_cost_page_hit` | Vacuum cost for a page found in the buffer cache. | | `pg_settings_vacuum_cost_page_miss` | Vacuum cost for a page not found in the buffer cache. | | `pg_settings_vacuum_failsafe_age` | Age at which VACUUM should trigger failsafe to avoid a wraparound outage. | | `pg_settings_vacuum_freeze_min_age` | Minimum age at which VACUUM should freeze a table row. | | `pg_settings_vacuum_freeze_table_age` | Age at which VACUUM should scan whole table to freeze tuples. | | `pg_settings_vacuum_max_eager_freeze_failure_rate` | Fraction of pages in a relation vacuum can scan and fail to freeze before disabling eager scanning. | | `pg_settings_vacuum_multixact_failsafe_age` | Multixact age at which VACUUM should trigger failsafe to avoid a wraparound outage. | | `pg_settings_vacuum_multixact_freeze_min_age` | Minimum age at which VACUUM should freeze a MultiXactId in a table row. | | `pg_settings_vacuum_multixact_freeze_table_age` | Multixact age at which VACUUM should scan whole table to freeze tuples. | | `pg_settings_vacuum_truncate` | Enables vacuum to truncate empty pages at the end of the table. | | `pg_settings_wal_block_size` | Shows the block size in the write ahead log. | | `pg_settings_wal_buffers_bytes` | Sets the number of disk-page buffers in shared memory for WAL. Units converted to bytes. | | `pg_settings_wal_decode_buffer_size_bytes` | Buffer size for reading ahead in the WAL during recovery. Units converted to bytes. | | `pg_settings_wal_init_zero` | Writes zeroes to new WAL files before first use. | | `pg_settings_wal_keep_size_bytes` | Sets the size of WAL files held for standby servers. Units converted to bytes. | | `pg_settings_wal_log_hints` | Writes full pages to WAL when first modified after a checkpoint, even for a non-critical modification. | | `pg_settings_wal_receiver_create_temp_slot` | Sets whether a WAL receiver should create a temporary replication slot if no permanent slot is configured. | | `pg_settings_wal_receiver_status_interval_seconds` | Sets the maximum interval between WAL receiver status reports to the sending server. Units converted to seconds. | | `pg_settings_wal_receiver_timeout_seconds` | Sets the maximum wait time to receive data from the sending server. Units converted to seconds. | | `pg_settings_wal_recycle` | Recycles WAL files by renaming them. | | `pg_settings_wal_retrieve_retry_interval_seconds` | Sets the time to wait before retrying to retrieve WAL after a failed attempt. Units converted to seconds. | | `pg_settings_wal_segment_size_bytes` | Shows the size of write ahead log segments. Units converted to bytes. | | `pg_settings_wal_sender_timeout_seconds` | Sets the maximum time to wait for WAL replication. Units converted to seconds. | | `pg_settings_wal_skip_threshold_bytes` | Minimum size of new file to fsync instead of writing WAL. Units converted to bytes. | | `pg_settings_wal_summary_keep_time_seconds` | Time for which WAL summary files should be kept. Units converted to seconds. | | `pg_settings_wal_writer_delay_seconds` | Time between WAL flushes performed in the WAL writer. Units converted to seconds. | | `pg_settings_wal_writer_flush_after_bytes` | Amount of WAL written out by WAL writer that triggers a flush. Units converted to bytes. | | `pg_settings_work_mem_bytes` | Sets the maximum memory to be used for query workspaces. Units converted to bytes. | | `pg_settings_zero_damaged_pages` | Continues processing past damaged page headers. | ### PG stats metrics [Section titled “PG stats metrics”](#pg-stats-metrics) These metrics come from PostgreSQL's internal statistics collector and include connection counts, transaction throughput, buffer usage, replication state, and database-level I/O. | Metric | Description | | ----------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `pg_stat_activity_count` | Number of connections in this state. | | `pg_stat_activity_max_tx_duration` | Max duration in seconds any active transaction has been running. | | `pg_stat_archiver_archived_count_total` | Number of WAL files that have been successfully archived. | | `pg_stat_archiver_failed_count_total` | Number of failed attempts for archiving WAL files. | | `pg_stat_archiver_last_archive_age` | Time in seconds since last WAL segment was successfully archived. | | `pg_stat_bgwriter_buffers_alloc_total` | Number of buffers allocated. | | `pg_stat_bgwriter_buffers_clean_total` | Number of buffers written by the background writer. | | `pg_stat_bgwriter_maxwritten_clean_total` | Number of times the background writer stopped a cleaning scan because it had written too many buffers. | | `pg_stat_bgwriter_stats_reset_total` | Time at which these statistics were last reset. | | `pg_stat_database_active_time` | Unknown metric from `pg_stat_database`. | | `pg_stat_database_blk_read_time_total` | Time spent reading data file blocks by backends in this database, in milliseconds. | | `pg_stat_database_blk_write_time_total` | Time spent writing data file blocks by backends in this database, in milliseconds. | | `pg_stat_database_blks_hit_total` | Number of times disk blocks were found already in the buffer cache, so that a read was not necessary (this only includes hits in the PostgreSQL buffer cache, not the operating system's file system cache). | | `pg_stat_database_blks_read_total` | Number of disk blocks read in this database. | | `pg_stat_database_checksum_failures` | Unknown metric from `pg_stat_database`. | | `pg_stat_database_checksum_last_failure` | Unknown metric from `pg_stat_database`. | | `pg_stat_database_conflicts_confl_active_logicalslot` | Unknown metric from `pg_stat_database_conflicts`. | | `pg_stat_database_conflicts_confl_bufferpin_total` | Number of queries in this database that have been canceled due to pinned buffers. | | `pg_stat_database_conflicts_confl_deadlock_total` | Number of queries in this database that have been canceled due to deadlocks. | | `pg_stat_database_conflicts_confl_lock_total` | Number of queries in this database that have been canceled due to lock timeouts. | | `pg_stat_database_conflicts_confl_snapshot_total` | Number of queries in this database that have been canceled due to old snapshots. | | `pg_stat_database_conflicts_confl_tablespace_total` | Number of queries in this database that have been canceled due to dropped tablespaces. | | `pg_stat_database_conflicts_total` | Number of queries canceled due to conflicts with recovery in this database. (Conflicts occur only on standby servers; see `pg_stat_database_conflicts` for details.) | | `pg_stat_database_deadlocks_total` | Number of deadlocks detected in this database. | | `pg_stat_database_idle_in_transaction_time` | Unknown metric from `pg_stat_database`. | | `pg_stat_database_numbackends` | Number of backends currently connected to this database. This is the only column in this view that returns a value reflecting current state; all other columns return the accumulated values since the last reset. | | `pg_stat_database_parallel_workers_launched` | Unknown metric from `pg_stat_database`. | | `pg_stat_database_parallel_workers_to_launch` | Unknown metric from `pg_stat_database`. | | `pg_stat_database_session_time` | Unknown metric from `pg_stat_database`. | | `pg_stat_database_sessions` | Unknown metric from `pg_stat_database`. | | `pg_stat_database_sessions_abandoned` | Unknown metric from `pg_stat_database`. | | `pg_stat_database_sessions_fatal` | Unknown metric from `pg_stat_database`. | | `pg_stat_database_sessions_killed` | Unknown metric from `pg_stat_database`. | | `pg_stat_database_stats_reset_total` | Time at which these statistics were last reset. | | `pg_stat_database_temp_bytes_total` | Total amount of data written to temporary files by queries in this database. All temporary files are counted, regardless of why the temporary file was created, and regardless of the `log_temp_files` setting. | | `pg_stat_database_temp_files_total` | Number of temporary files created by queries in this database. All temporary files are counted, regardless of why the temporary file was created (e.g., sorting or hashing), and regardless of the `log_temp_files` setting. | | `pg_stat_database_tup_deleted_total` | Number of rows deleted by queries in this database. | | `pg_stat_database_tup_fetched_total` | Number of rows fetched by queries in this database. | | `pg_stat_database_tup_inserted_total` | Number of rows inserted by queries in this database. | | `pg_stat_database_tup_returned_total` | Number of rows returned by queries in this database. | | `pg_stat_database_tup_updated_total` | Number of rows updated by queries in this database. | | `pg_stat_database_xact_commit_total` | Number of transactions in this database that have been committed. | | `pg_stat_database_xact_rollback_total` | Number of transactions in this database that have been rolled back. | | `pg_stat_replication_pg_current_wal_lsn_bytes` | WAL position in bytes. | | `pg_stat_replication_pg_wal_lsn_diff` | Lag in bytes between primary and replica. | | `pg_stat_replication_reply_time` | Timestamp of the last reply message received from the standby server. | | `pg_static` | Version string as reported by PostgreSQL. | | `pg_up` | Whether the last scrape of metrics from PostgreSQL was able to connect to the server (1 for yes, 0 for no). | # Integrate Grafana with Tiger Cloud > Query, visualize, and explore your data with interactive dashboards [Grafana](https://grafana.com/docs/) is an open-source analytics and visualization platform that enables you to query, visualize, alert on, and explore your metrics, logs, and traces wherever they're stored. This page shows you how to integrate Grafana with a Tiger Cloud service, create a dashboard and panel, then visualize geospatial data. ## Prerequisites for this integration guide To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service). These steps use Tiger Cloud, but the same approach applies to a [self-hosted TimescaleDB](/docs/get-started/choose-your-path/install-timescaledb) instance. - Your [connection details](/docs/integrate/find-connection-details). * A [self-managed Grafana](https://grafana.com/get/?tab=self-managed) instance, or a [Grafana Cloud](https://grafana.com/get/) account. ## Connect Grafana to Tiger Cloud To visualize the results of your queries, enable Grafana to read the data in your service: 1. **Log in to Grafana** In your browser, log in to either: * Self-hosted Grafana: at `http://localhost:3000/`. The default credentials are `admin`, `admin`. * Grafana Cloud: use the URL and credentials you set when you created your account. 2. **Add your service as a data source** 1. Open `Connections` > `Data sources`, then click `Add new data source`. 2. Select `PostgreSQL` from the list. 3. Configure the connection: * `Host URL`, `Database name`, `Username`, and `Password`, configure using your [connection details](/docs/integrate/find-connection-details). `Host URL` is in the format `:`. * `TLS/SSL Mode`: select `require`. * `PostgreSQL options`: enable `TimescaleDB`. * Leave the default setting for all other fields. 4. Click `Save & test`. Grafana checks that your details are set correctly. ## Create a Grafana dashboard and panel [Section titled “Create a Grafana dashboard and panel”](#create-a-grafana-dashboard-and-panel) Grafana is organized into dashboards and panels. A dashboard represents a view into the performance of a system, and each dashboard consists of one or more panels, which represent information about a specific metric related to that system. To create a new dashboard: 1. **On the `Dashboards` page, click `New` and select `New dashboard`** 2. **Click `Add visualization`** 3. **Select the data source** Select your service from the list of pre-configured data sources or configure a new one. 4. **Configure your panel** Select the visualization type. The type defines specific fields to configure in addition to standard ones, such as the panel name. 5. **Run your queries** You can edit the queries directly or use the built-in query editor. If you are visualizing time-series data, select `Time series` in the `Format` drop-down. 6. **Click `Save dashboard`** You now have a dashboard with one panel. Add more panels to a dashboard by clicking `Add` at the top right and selecting `Visualization` from the drop-down. ## Use the time filter function [Section titled “Use the time filter function”](#use-the-time-filter-function) Grafana time-series panels include a time filter: 1. **Call `$__timefilter()` to link the user interface construct in a Grafana panel with the query** For example, to set the `pickup_datetime` column as the filtering range for your visualizations: ```sql SELECT --1-- time_bucket('1 day', pickup_datetime) AS "time", --2-- COUNT(*) FROM rides WHERE $__timeFilter(pickup_datetime) ``` 2. **Group your visualizations and order the results by time buckets** Use [`time_bucket`](/docs/reference/timescaledb/hyperfunctions/time-series-utilities/time_bucket) to group results. In this case, the `GROUP BY` and `ORDER BY` statements reference `time`. For example: ```sql SELECT --1-- time_bucket('1 day', pickup_datetime) AS time, --2-- COUNT(*) FROM rides WHERE $__timeFilter(pickup_datetime) GROUP BY time ORDER BY time ``` When you visualize this query in Grafana, you see this: ![Grafana panel showing query results from Tiger Cloud](/docs/_astro/grafana_query_results.DJAqOxI-_Z1xRKp0.webp) You can adjust the `time_bucket` function and compare the graphs: ```sql SELECT --1-- time_bucket('5m', pickup_datetime) AS time, --2-- COUNT(*) FROM rides WHERE $__timeFilter(pickup_datetime) GROUP BY time ORDER BY time ``` When you visualize this query, it looks like this: ![Grafana panel showing query results aggregated in 5-minute time buckets](/docs/_astro/grafana_query_results_5m.BrSMXMw__ZMMOVQ.webp) ## Visualize geospatial data [Section titled “Visualize geospatial data”](#visualize-geospatial-data) Grafana includes a Geomap panel so you can see geospatial data overlaid on a map. This can be helpful to understand how data changes based on its location. This section visualizes taxi rides in Manhattan, where the distance traveled was greater than 5 miles. It uses the same query as the [NYC Taxi Cab](/docs/build/examples/analyze-transport-data) tutorial as a starting point. 1. **Add a geospatial visualization** 1. In your Grafana dashboard, click `Add` > `Visualization`. 2. Select `Geomap` in the visualization type drop-down at the top right. 2. **Configure the data format** 1. In the `Queries` tab below, select your data source. 2. In the `Format` drop-down, select `Table`. 3. In the mode switcher, toggle `Code` and enter the query, then click `Run`. For example: ```sql SELECT time_bucket('5m', rides.pickup_datetime) AS time, rides.trip_distance AS value, rides.pickup_latitude AS latitude, rides.pickup_longitude AS longitude FROM rides WHERE rides.trip_distance > 5 GROUP BY time, rides.trip_distance, rides.pickup_latitude, rides.pickup_longitude ORDER BY time LIMIT 500; ``` 3. **Customize the Geomap settings** With default settings, the visualization uses green circles of the fixed size. Configure at least the following for a more representative view: * `Map layers` > `Styles` > `Size` > `value`. This changes the size of the circle depending on the value, with bigger circles representing bigger values. * `Map layers` > `Styles` > `Color` > `value`. * `Thresholds` > Add `threshold`. Add thresholds for 7 and 10, to mark rides over 7 and 10 miles in different colors, respectively. You now have a visualization that looks like this: ![Grafana dashboard connected to Tiger Cloud](/docs/_astro/timescale-grafana-integration.BnZybwp-_Z2arOYi.webp) # Integrate Prometheus with Tiger Cloud > Export telemetry metrics from your service to monitor system performance and health [Prometheus](https://prometheus.io/docs/introduction/overview/) is an open-source monitoring system with a dimensional data model, flexible query language, and a modern alerting approach. This page shows you how to export your service telemetry to Prometheus: * For Tiger Cloud, using a dedicated Prometheus exporter in Tiger Console. * For self-hosted TimescaleDB, using [PostgreSQL Exporter](https://grafana.com/oss/prometheus/exporters/postgres-exporter/). ## Prerequisites for this integration guide To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service), or a running instance of [self-hosted TimescaleDB](/docs/get-started/choose-your-path/install-timescaledb). - Your [connection details](/docs/integrate/find-connection-details). * [Prometheus](https://prometheus.io/docs/prometheus/latest/installation/) downloaded and running. To reduce latency and potential data transfer costs, run Prometheus on a machine in the same AWS region as your Tiger Cloud service. ## Export Tiger Cloud service telemetry to Prometheus To export your data, do the following: * Tiger Cloud To export metrics from a Tiger Cloud service, you create a dedicated Prometheus exporter in Tiger Console, attach it to your service, then configure Prometheus to scrape metrics using the exposed URL. See [Exported metrics](/docs/integrate/observability-alerting/exported-metrics) for the full list of default and additional metrics you can export. The Prometheus exporter is available for [Scale and Enterprise](/docs/deploy/tiger-cloud/tiger-cloud-aws/pricing-and-account-management#features-included-in-each-pricing-plan) pricing plans. 1. **Create a Prometheus exporter** 1. In [Tiger Console](https://console.cloud.tigerdata.com/dashboard/services), click `Exporters` > `+ New exporter`. 2. Select `Metrics` for data type and `Prometheus` for provider. ![Creating a Prometheus exporter in Tiger Console](/docs/_astro/tiger-cloud-create-prometheus-exporter.B94eNIjE_1amfO0.webp) 3. Choose the region for the exporter. Only services in the same project and region can be attached to this exporter. 4. Name your exporter. 5. Change the auto-generated Prometheus credentials, if needed. See [official documentation](https://prometheus.io/docs/guides/basic-auth/) on basic authentication in Prometheus. 6. Optionally tick `PostgreSQL metrics` to export [additional metrics](/docs/integrate/observability-alerting/exported-metrics#postgresql-metrics), then click `Create exporter`. 2. **Attach the exporter to a service** 1. Select a service, then click `Operations` > `Exporters`. 2. Select the exporter in the drop-down, then click `Attach exporter`. ![Attaching a Prometheus exporter to a service in Tiger Console](/docs/_astro/attach-prometheus-exporter-tiger-console.N2R83WV__1hdbxB.webp) The exporter is now attached to your service. To unattach it, click the trash icon in the exporter list. 3. **Configure the Prometheus scrape target** 1. Select your service, then click `Operations` > `Exporters` and click the information icon next to the exporter. You see the exporter details. ![Prometheus exporter details showing endpoint and credentials](/docs/_astro/prometheus-exporter-details-tiger-cloud.Bkr_vzgL_Sperb.webp) 2. Copy the exporter URL. 3. In your Prometheus installation, update `prometheus.yml` to point to the exporter URL as a scrape target: ```yml scrape_configs: - job_name: "timescaledb-exporter" scheme: https static_configs: - targets: ["my-exporter-url"] basic_auth: username: "user" password: "pass" ``` See the [Prometheus documentation](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config) for details on configuring scrape targets. You can now monitor your service metrics. * Self-hosted TimescaleDB To export metrics from self-hosted TimescaleDB, you import telemetry data about your database to PostgreSQL Exporter, then configure Prometheus to scrape metrics from it. PostgreSQL Exporter exposes metrics that you define, excluding the system metrics. 1. **Install PostgreSQL Exporter** Install [PostgreSQL Exporter](https://grafana.com/oss/prometheus/exporters/postgres-exporter/?tab=installation) on a machine that can reach your self-hosted TimescaleDB instance. 2. **Create a user to access telemetry data about your database** 1. Connect to your database in [`psql`](/docs/integrate/query-administration/psql) using your [connection details](/docs/integrate/find-connection-details). 2. Create a user named `monitoring` with a secure password: ```sql CREATE USER monitoring WITH PASSWORD ''; ``` 3. Grant the `pg_read_all_stats` permission to the `monitoring` user: ```sql GRANT pg_read_all_stats to monitoring; ``` 3. **Import telemetry data about your database to PostgreSQL Exporter** 1. Connect PostgreSQL Exporter to your database: Use your [connection details](/docs/integrate/find-connection-details) to import telemetry data about your database. You connect as the `monitoring` user: * Local installation: ```shell export DATA_SOURCE_NAME="postgres://:@:/?sslmode=" ./postgres_exporter ``` * Docker: ```shell docker run -d \ -e DATA_SOURCE_NAME="postgres://:@:/?sslmode=" \ -p 9187:9187 \ prometheuscommunity/postgres-exporter ``` 2. Check the metrics for your database in the Prometheus format: * Browser: Navigate to `http://:9187/metrics`. * Command line: ```shell curl http://:9187/metrics ``` 4. **Configure Prometheus to scrape metrics** 1. In your Prometheus installation, update `prometheus.yml` to point to your PostgreSQL Exporter instance as a scrape target. In the following example, you replace `` with the hostname or IP address of the PostgreSQL Exporter. ```yaml global: scrape_interval: 15s scrape_configs: - job_name: 'postgresql' static_configs: - targets: [':9187'] ``` If `prometheus.yml` has not been created during installation, create it manually. If you are using Docker, you can find the IPAddress in `Inspect` > `Networks` for the container running PostgreSQL Exporter. 2. Restart Prometheus. 3. Check the Prometheus UI at `http://:9090/targets` and `http://:9090/tsdb-status`. You see the PostgreSQL Exporter target and the metrics scraped from it. You can further [visualize your data](https://grafana.com/docs/grafana-cloud/send-data/metrics/metrics-prometheus/) with Grafana. Use the [Grafana PostgreSQL dashboard](https://grafana.com/oss/prometheus/exporters/postgres-exporter/?tab=dashboards) or [create a custom dashboard](/docs/integrate/observability-alerting/grafana) that suits your needs. Note This feature is currently not supported for Tiger Cloud on Microsoft Azure. # Integrate Telegraf with Tiger Cloud > Ingest metrics and events from databases, systems, and IoT sensors [Telegraf](https://www.influxdata.com/time-series-platform/telegraf/) is a server-based agent that collects and sends metrics and events from databases, systems, and IoT sensors. Telegraf is an open-source, plugin-driven tool for the collection and output of data. To view metrics gathered by Telegraf and stored in a [hypertable](/docs/learn/hypertables/understand-hypertables) in a Tiger Cloud service: * [Link Telegraf to your service](/docs/integrate/observability-alerting/telegraf#link-telegraf-to-your-service): create a Telegraf configuration * [View the metrics collected by Telegraf](/docs/integrate/observability-alerting/telegraf#view-the-metrics-collected-by-telegraf): connect to your service and query the metrics table ## Prerequisites for this integration guide To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service). These steps use Tiger Cloud, but the same approach applies to a [self-hosted TimescaleDB](/docs/get-started/choose-your-path/install-timescaledb) instance. - Your [connection details](/docs/integrate/find-connection-details). * [Telegraf](https://docs.influxdata.com/telegraf/v1/introduction/installation/) installed. ## Link Telegraf to your service [Section titled “Link Telegraf to your service”](#link-telegraf-to-your-service) To create a Telegraf configuration that exports data to a hypertable in your service: 1. **Set up your service connection string** This variable holds the connection information for the target Tiger Cloud service. In the terminal on the source machine, set the following: ```bash export TARGET=postgres://tsdbadmin:@:/tsdb?sslmode=require ``` See where to [find your connection details](/docs/integrate/find-connection-details). 2. **Generate a Telegraf configuration file** In the terminal, run the following: ```bash telegraf --input-filter=cpu --output-filter=postgresql config > telegraf.conf ``` `telegraf.conf` configures a CPU input plugin that samples various metrics about CPU usage, and the PostgreSQL output plugin. `telegraf.conf` also includes all available input, output, processor, and aggregator plugins. These are commented out by default. 3. **Test the configuration** ```bash telegraf --config telegraf.conf --test ``` You see an output similar to the following: ```bash 2022-11-28T12:53:44Z I! Starting Telegraf 1.24.3 2022-11-28T12:53:44Z I! Available plugins: 208 inputs, 9 aggregators, 26 processors, 20 parsers, 57 outputs 2022-11-28T12:53:44Z I! Loaded inputs: cpu 2022-11-28T12:53:44Z I! Loaded aggregators: 2022-11-28T12:53:44Z I! Loaded processors: 2022-11-28T12:53:44Z W! Outputs are not used in testing mode! 2022-11-28T12:53:44Z I! Tags enabled: host=localhost > cpu,cpu=cpu0,host=localhost usage_guest=0,usage_guest_nice=0,usage_idle=90.00000000087311,usage_iowait=0,usage_irq=0,usage_nice=0,usage_softirq=0,usage_steal=0,usage_system=6.000000000040018,usage_user=3.999999999996362 1669640025000000000 > cpu,cpu=cpu1,host=localhost usage_guest=0,usage_guest_nice=0,usage_idle=92.15686274495818,usage_iowait=0,usage_irq=0,usage_nice=0,usage_softirq=0,usage_steal=0,usage_system=5.882352941192206,usage_user=1.9607843136712912 1669640025000000000 > cpu,cpu=cpu2,host=localhost usage_guest=0,usage_guest_nice=0,usage_idle=91.99999999982538,usage_iowait=0,usage_irq=0,usage_nice=0,usage_softirq=0,usage_steal=0,usage_system=3.999999999996362,usage_user=3.999999999996362 1669640025000000000 ``` 4. **Configure the PostgreSQL output plugin** 1. In `telegraf.conf`, in the `[[outputs.postgresql]]` section, set `connection` to the value of $TARGET. ```bash connection = "" ``` 2. Use hypertables when Telegraf creates a new table. In the section that begins with the comment `## Templated statements to execute when creating a new table`, add the following template: ```bash ## Templated statements to execute when creating a new table. create_templates = [ '''CREATE TABLE IF NOT EXISTS {{ .table }} ({{ .columns }}) WITH (tsdb.hypertable, tsdb.chunk_interval='1 week')''', ] ``` ## View the metrics collected by Telegraf [Section titled “View the metrics collected by Telegraf”](#view-the-metrics-collected-by-telegraf) This section shows you how to generate system metrics using Telegraf, then connect to your service and query the metrics [hypertable](/docs/learn/hypertables/understand-hypertables). 1. **Collect system metrics using Telegraf** Run the following command for 30 seconds: ```bash telegraf --config telegraf.conf ``` Telegraf uses loaded inputs `cpu` and outputs `postgresql` along with `global tags`, the intervals when the agent collects data from the inputs, and flushes to the outputs. 2. **View the metrics** 1. Connect to your Tiger Cloud service: ```bash psql $TARGET ``` 2. View the metrics collected in the `cpu` table in `tsdb`: ```sql SELECT*FROM cpu; ``` You see something like: ```sql time | cpu | host | usage_guest | usage_guest_nice | usage_idle | usage_iowait | usage_irq | usage_nice | usage_softirq | usage_steal | usage_system | usage_user ---------------------+-----------+----------------------------------+-------------+------------------+-------------------+--------------+-----------+------------+---------------+-------------+---------------------+--------------------- 2022-12-05 12:25:20 | cpu0 | hostname | 0 | 0 | 83.08605341237833 | 0 | 0 | 0 | 0 | 0 | 6.824925815961274 | 10.089020771444481 2022-12-05 12:25:20 | cpu1 | hostname | 0 | 0 | 84.27299703278959 | 0 | 0 | 0 | 0 | 0 | 5.934718100814769 | 9.792284866395647 2022-12-05 12:25:20 | cpu2 | hostname | 0 | 0 | 87.53709198848934 | 0 | 0 | 0 | 0 | 0 | 4.747774480755411 | 7.715133531241037 2022-12-05 12:25:20 | cpu3 | hostname| 0 | 0 | 86.68639053296472 | 0 | 0 | 0 | 0 | 0 | 4.43786982253345 | 8.875739645039992 2022-12-05 12:25:20 | cpu4 | hostname | 0 | 0 | 96.15384615371369 | 0 | 0 | 0 | 0 | 0 | 1.1834319526667423 | 2.6627218934917614 ``` To view the average usage per CPU core, use `SELECT cpu, avg(usage_user) FROM cpu GROUP BY cpu;`. For more information about the options that you can configure in Telegraf, see the [PostgreSQL output plugin](https://github.com/influxdata/telegraf/blob/release-1.24/plugins/outputs/postgresql/README.md). # Query & administration > Use your query and administration tools with Tiger Data The guides below show how to connect query and administration tools to your database, enabling you to run SQL, manage schemas, and administer your database from the clients and IDEs you already use, without changing your workflow. ## Integrations [Section titled “Integrations”](#integrations) [Azure Data Studio](/docs/integrate/query-administration/azure-data-studio) [Query and manage your database with a cross-platform data analytics tool](/docs/integrate/query-administration/azure-data-studio) [DBeaver](/docs/integrate/query-administration/dbeaver) [Manage and query your data with a cross-platform SQL editor and administration tool](/docs/integrate/query-administration/dbeaver) [pgAdmin](/docs/integrate/query-administration/pgadmin) [Query, visualize, and manage your database with an open-source administration tool](/docs/integrate/query-administration/pgadmin) [PostgreSQL](/docs/integrate/query-administration/postgresql) [Query other Postgres databases using foreign data wrappers](/docs/integrate/query-administration/postgresql) [psql](/docs/integrate/query-administration/psql) [Connect to your Tiger Cloud service and run interactive queries with psql](/docs/integrate/query-administration/psql) [qStudio](/docs/integrate/query-administration/qstudio) [Connect to your Tiger Cloud service and visualize your data with qStudio](/docs/integrate/query-administration/qstudio) # Integrate Azure Data Studio with Tiger Cloud > Query and manage your database with a cross-platform data analytics tool [Azure Data Studio](https://azure.microsoft.com/en-us/products/data-studio) is an open-source, cross-platform hybrid data analytics tool designed to simplify the data landscape. This page explains how to integrate Azure Data Studio with Tiger Cloud. ## Prerequisites for this integration guide To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service). These steps use Tiger Cloud, but the same approach applies to a [self-hosted TimescaleDB](/docs/get-started/choose-your-path/install-timescaledb) instance. - Your [connection details](/docs/integrate/find-connection-details). * [Azure Data Studio](https://learn.microsoft.com/en-us/azure-data-studio/download-azure-data-studio?view=sql-server-ver16#install-azure-data-studio) downloaded and installed. * The [PostgreSQL extension for Azure Data Studio](https://learn.microsoft.com/en-us/azure-data-studio/extensions/postgres-extension?view=sql-server-ver16) installed. ## Connect to your service with Azure Data Studio [Section titled “Connect to your service with Azure Data Studio”](#connect-to-your-service-with-azure-data-studio) To connect to Tiger Cloud: 1. **Start `Azure Data Studio`** 2. **In the `SERVERS` page, click `New Connection`** 3. **Configure the connection** 1. Select `PostgreSQL` for `Connection type`. 2. Configure the server name, database, username, port, and password using your [connection details](/docs/integrate/find-connection-details). 3. Click `Advanced`. If you configured your Tiger Cloud service to connect using [stricter SSL mode](/docs/deploy/tiger-cloud/tiger-cloud-aws/security/strict-ssl), set `SSL mode` to the configured mode, then type the location of your SSL root CA certificate in `SSL root certificate filename`. 4. In the `Port` field, type the port number and click `OK`. 4. **Click `Connect`** You have successfully integrated Azure Data Studio with Tiger Cloud. # Integrate DBeaver with Tiger Cloud > Manage and query your data with a cross-platform SQL editor and administration tool [DBeaver](https://dbeaver.io/) is a free cross-platform database tool for developers, database administrators, analysts, and everyone working with data. DBeaver provides an SQL editor, administration features, data and schema migration, and the ability to monitor database connection sessions. This page explains how to integrate DBeaver with your Tiger Cloud service. ## Prerequisites for this integration guide To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service). These steps use Tiger Cloud, but the same approach applies to a [self-hosted TimescaleDB](/docs/get-started/choose-your-path/install-timescaledb) instance. - Your [connection details](/docs/integrate/find-connection-details). * [DBeaver](https://dbeaver.io/download/) downloaded and installed. ## Connect DBeaver to your service [Section titled “Connect DBeaver to your service”](#connect-dbeaver-to-your-service) To connect to Tiger Cloud: 1. **Start `DBeaver`** 2. **In the toolbar, click the plug+ icon** 3. **In `Connect to a database` search for `TimescaleDB`** 4. **Select `TimescaleDB`, then click `Next`** 5. **Configure the connection** Use your [connection details](/docs/integrate/find-connection-details) to add your connection settings. ![DBeaver connected to a Tiger Cloud service](/docs/_astro/integrations-dbeaver.jLzfPGT3_282ge0.webp) If you configured your service to connect using a [stricter SSL mode](/docs/deploy/tiger-cloud/tiger-cloud-aws/security/strict-ssl), in the `SSL` tab check `Use SSL` and set `SSL mode` to the configured mode. Then, in the `CA Certificate` field type the location of the SSL root CA certificate. 6. **Click `Test Connection`. When the connection is successful, click `Finish`** Your connection is listed in the `Database Navigator`. You have successfully integrated DBeaver with Tiger Cloud. # Integrate pgAdmin with Tiger Cloud > Query, visualize, and manage your database with an open-source administration tool [pgAdmin](https://www.pgadmin.org/) is a feature-rich open-source administration and development platform for PostgreSQL. It is available for Chrome, Firefox, Edge, and Safari browsers, or can be installed on Microsoft Windows, Apple macOS, or various Linux flavors. ![pgAdmin connected to a Tiger Cloud service](/docs/_astro/timescale-cloud-pgadmin.CyQnfiAA_Z26BIfn.webp) This page explains how to integrate pgAdmin with your Tiger Cloud service. ## Prerequisites for this integration guide To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service). These steps use Tiger Cloud, but the same approach applies to a [self-hosted TimescaleDB](/docs/get-started/choose-your-path/install-timescaledb) instance. - Your [connection details](/docs/integrate/find-connection-details). * [pgAdmin](https://www.pgadmin.org/download/) downloaded and installed. ## Connect pgAdmin to your service [Section titled “Connect pgAdmin to your service”](#connect-pgadmin-to-your-service) To connect to Tiger Cloud: 1. **Start pgAdmin** 2. **In the `Quick Links` section of the `Dashboard` tab, click `Add New Server`** 3. **In `Register - Server` > `General`, fill in the `Name` and `Comments` fields with the server name and description, respectively** 4. **Configure the connection** 1. In the `Connection` tab, configure the connection using your [connection details](/docs/integrate/find-connection-details). 2. If you configured your service to connect using a [stricter SSL mode](/docs/deploy/tiger-cloud/tiger-cloud-aws/security/strict-ssl), then in the `SSL` tab check `Use SSL`, set `SSL mode` to the configured mode, and in the `CA Certificate` field type the location of the SSL root CA certificate to use. 5. **Click `Save`** You have successfully integrated pgAdmin with Tiger Cloud. # Integrate Postgres with Tiger Cloud > Query other Postgres data bases using foreign data wrappers You use PostgreSQL foreign data wrappers (FDWs) to query external data sources from a Tiger Cloud service. These external data sources can be one of the following: * Other Tiger Cloud services * PostgreSQL databases outside of Tiger Cloud If you are using VPC peering, you can create FDWs in your Customer VPC to query a service in your Tiger Cloud project. However, you can't create FDWs in your Tiger Cloud services to query a data source in your Customer VPC. This is because Tiger Cloud VPC peering uses AWS PrivateLink for increased security. See [VPC peering documentation](/docs/deploy/tiger-cloud/tiger-cloud-aws/security/vpc) for additional details. PostgreSQL FDWs are particularly useful if you manage multiple Tiger Cloud services with different capabilities, and need to seamlessly access and merge regular and time-series data. ## Prerequisites for this procedure To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service), or a running instance of [self-hosted TimescaleDB](/docs/get-started/choose-your-path/install-timescaledb). - Your [connection details](/docs/integrate/find-connection-details). ## Query another data source To query another data source: * Tiger Cloud You create PostgreSQL FDWs with the `postgres_fdw` extension, which is enabled by default in Tiger Cloud. 1. **Connect to your service** See [how to connect](/docs/build/data-management/run-queries-from-tiger-console). 2. **Create a server** Run the following command using your [connection details](/docs/integrate/find-connection-details): ```sql CREATE SERVER myserver FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '', dbname 'tsdb', port ''); ``` 3. **Create user mapping** Run the following command using your [connection details](/docs/integrate/find-connection-details): ```sql CREATE USER MAPPING FOR tsdbadmin SERVER myserver OPTIONS (user 'tsdbadmin', password ''); ``` 4. **Import a foreign schema (recommended) or create a foreign table** * Import the whole schema: ```sql CREATE SCHEMA foreign_stuff; IMPORT FOREIGN SCHEMA public FROM SERVER myserver INTO foreign_stuff ; ``` * Alternatively, import a limited number of tables: ```sql CREATE SCHEMA foreign_stuff; IMPORT FOREIGN SCHEMA public LIMIT TO (table1, table2) FROM SERVER myserver INTO foreign_stuff; ``` * Create a foreign table. Skip if you are importing a schema: ```sql CREATE FOREIGN TABLE films ( code char(5) NOT NULL, title varchar(40) NOT NULL, did integer NOT NULL, date_prod date, kind varchar(10), len interval hour to minute ) SERVER film_server; ``` A user with the `tsdbadmin` role assigned already has the required `USAGE` permission to create PostgreSQL FDWs. You can enable another user, without the `tsdbadmin` role assigned, to query foreign data. To do so, explicitly grant the permission. For example, for a new `grafana` user: ```sql CREATE USER grafana; GRANT grafana TO tsdbadmin; CREATE SCHEMA fdw AUTHORIZATION grafana; CREATE SERVER db1 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '', dbname 'tsdb', port ''); CREATE USER MAPPING FOR grafana SERVER db1 OPTIONS (user 'tsdbadmin', password ''); GRANT USAGE ON FOREIGN SERVER db1 TO grafana; SET ROLE grafana; IMPORT FOREIGN SCHEMA public FROM SERVER db1 INTO fdw; ``` * Self-hosted TimescaleDB You create PostgreSQL FDWs with the `postgres_fdw` extension. See [documentation](https://www.postgresql.org/docs/current/postgres-fdw.html) on how to enable it. 1. **Connect to your database** Use [`psql`](/docs/integrate/query-administration/psql) to connect to your database. 2. **Create a server** Run the following command using your [connection details](/docs/integrate/find-connection-details): ```sql CREATE SERVER myserver FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '', dbname '', port ''); ``` 3. **Create user mapping** Run the following command using your [connection details](/docs/integrate/find-connection-details): ```sql CREATE USER MAPPING FOR postgres SERVER myserver OPTIONS (user 'postgres', password ''); ``` 4. **Import a foreign schema (recommended) or create a foreign table** * Import the whole schema: ```sql CREATE SCHEMA foreign_stuff; IMPORT FOREIGN SCHEMA public FROM SERVER myserver INTO foreign_stuff ; ``` * Alternatively, import a limited number of tables: ```sql CREATE SCHEMA foreign_stuff; IMPORT FOREIGN SCHEMA public LIMIT TO (table1, table2) FROM SERVER myserver INTO foreign_stuff; ``` * Create a foreign table. Skip if you are importing a schema: ```sql CREATE FOREIGN TABLE films ( code char(5) NOT NULL, title varchar(40) NOT NULL, did integer NOT NULL, date_prod date, kind varchar(10), len interval hour to minute ) SERVER film_server; ``` # Connect to Tiger Cloud with psql > Connect to your Tiger Cloud service and run interactive queries with psql [`psql`](https://www.postgresql.org/docs/current/app-psql.html) is a terminal-based frontend to PostgreSQL that enables you to type in queries interactively, issue them to Postgres, and see the query results. This page shows you how to use the `psql` command line tool to interact with your Tiger Cloud service. ## Prerequisites for this integration guide To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service). These steps use Tiger Cloud, but the same approach applies to a [self-hosted TimescaleDB](/docs/get-started/choose-your-path/install-timescaledb) instance. - Your [connection details](/docs/integrate/find-connection-details). ## Check for an existing installation [Section titled “Check for an existing installation”](#check-for-an-existing-installation) On many operating systems, `psql` is installed by default. To use the functionality described in this page, best practice is to use the latest version of `psql`. To check the version running on your system: * Linux/MacOS ```bash psql --version ``` * Windows ```powershell wmic /output:C:\list.txt product get name, version ``` If you already have the latest version of `psql` installed, proceed to the [Connect to your service](#connect-to-your-service) section. ## Install psql [Section titled “Install psql”](#install-psql) If there is no existing installation, take the following steps to install `psql`: * MacOS Homebrew Install using Homebrew. `libpqxx` is the official C++ client API for PostgreSQL. 1. **Install Homebrew if you don't already have it** ```bash /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" ``` For more information about Homebrew, including installation instructions, see the [Homebrew documentation](https://docs.brew.sh/Installation). 2. **Update your Homebrew repository** ```bash brew doctor brew update ``` 3. **Install `psql`** ```bash brew install libpq ``` 4. **Update your path to include the `psql` tool** ```bash brew link --force libpq ``` On Intel chips, the symbolic link is added to `/usr/local/bin`. On Apple Silicon, the symbolic link is added to `/opt/homebrew/bin`. * MacOS MacPorts Install using MacPorts. `libpqxx` is the official C++ client API for PostgreSQL. 1. **Install MacPorts** [Install MacPorts](https://guide.macports.org/#installing.macports) by downloading and running the package installer. 2. **Update MacPorts** ```bash sudo port selfupdate ``` 3. **Install the latest version of `libpqxx`** ```bash sudo port install libpqxx ``` 4. **View the files installed by `libpqxx`** ```bash port contents libpqxx ``` * Debian and Ubuntu Install `psql` on Debian and Ubuntu with the `apt` package manager. 1. **Update your `apt` repository** ```bash sudo apt-get update ``` 2. **Install the `postgresql-client` package** ```bash sudo apt-get install postgresql-client ``` * Windows `psql` is installed by default when you install PostgreSQL. This procedure uses the interactive installer provided by PostgreSQL and EnterpriseDB. 1. **Download and run the installer** Download and run the PostgreSQL installer from [www.enterprisedb.com](https://www.postgresql.org/download/windows/). 2. **Select `Command Line Tools`** In the `Select Components` dialog, check `Command Line Tools`, along with any other components you want to install, and click `Next`. 3. **Complete the installation wizard** ## Connect to your service [Section titled “Connect to your service”](#connect-to-your-service) To use `psql` to connect to your service, you need the connection details. See [Connect to Tiger Data](/docs/integrate/find-connection-details). Connect to your service with either: * The parameter flags: ```bash psql -h -p -U -W -d ``` * The service URL: ```bash psql "postgres://@:/?sslmode=require" ``` You are prompted to provide the password. * The service URL with the password already included and [a stricter SSL mode](/docs/deploy/tiger-cloud/tiger-cloud-aws/security/strict-ssl) enabled: ```bash psql "postgres://:@:/?sslmode=verify-full" ``` ## Useful psql commands [Section titled “Useful psql commands”](#useful-psql-commands) When you start using `psql`, these are the commands you are likely to use most frequently: | Command | Description | | ---------------------- | ---------------------------------------- | | `\c ` | Connect to a new database | | `\d ` | Show the details of a table | | `\df` | List functions in the current database | | `\df+` | List all functions with more details | | `\di` | List all indexes from all tables | | `\dn` | List all schemas in the current database | | `\dt` | List available tables | | `\du` | List PostgreSQL database roles | | `\dv` | List views in current schema | | `\dv+` | List all views with more details | | `\dx` | Show all installed extensions | | `ef ` | Edit a function | | `\h` | Show help on syntax of SQL commands | | `\l` | List available databases | | `\password ` | Change the password for the user | | `\q` | Quit `psql` | | `\set` | Show system variables list | | `\timing` | Show how long a query took to execute | | `\x` | Show expanded query results | | `\?` | List all `psql` slash commands | For more on `psql` commands, see the [Tiger Data psql cheat sheet](https://www.tigerdata.com/learn/postgres-cheat-sheet) and [psql documentation](https://www.postgresql.org/docs/current/app-psql.html). ## Save query results to a file [Section titled “Save query results to a file”](#save-query-results-to-a-file) When you run queries in `psql`, the results are shown in the terminal by default. If you are running queries that have a lot of results, you might like to save the results into a comma-separated `.csv` file instead. You can do this using the `COPY` command. For example: ```sql \copy (SELECT * FROM ...) TO '/tmp/output.csv' (format CSV); ``` This command sends the results of the query to a new file called `output.csv` in the `/tmp/` directory. You can open the file using any spreadsheet program. ## Run long queries [Section titled “Run long queries”](#run-long-queries) To run multi-line queries in `psql`, use the `EOF` delimiter. For example: ```sql psql -d $TARGET -f -v hypertable= - <<'EOF' SELECT public.alter_job(j.id, scheduled=>true) FROM _timescaledb_config.bgw_job j JOIN _timescaledb_catalog.hypertable h ON h.id = j.hypertable_id WHERE j.proc_schema IN ('_timescaledb_internal', '_timescaledb_functions') AND j.proc_name = 'policy_columnstore' AND j.id >= 1000 AND format('%I.%I', h.schema_name, h.table_name)::text::regclass = :'hypertable'::text::regclass; EOF ``` ## Edit queries in a text editor [Section titled “Edit queries in a text editor”](#edit-queries-in-a-text-editor) Sometimes, queries can get very long, and you might make a mistake when you try typing it the first time around. If you have made a mistake in a long query, instead of retyping it, you can use a built-in text editor, which is based on `Vim`. Launch the query editor with the `\e` command. Your previous query is loaded into the editor. When you have made your changes, press `Esc`, then type `:`＋`w`＋`q` to save the changes, and return to the command prompt. Access the edited query by pressing `↑`, and press `Enter` to run it. # Integrate qStudio with Tiger Cloud > Connect to your Tiger Cloud service and visualize your data with qStudio [qStudio](https://www.timestored.com/qstudio/) is a modern free SQL editor that provides syntax highlighting, code-completion, excel export, charting, and much more. You can use it to run queries, browse tables, and create charts for your Tiger Cloud service. This page explains how to integrate qStudio with Tiger Cloud. ## Prerequisites for this integration guide To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service). These steps use Tiger Cloud, but the same approach applies to a [self-hosted TimescaleDB](/docs/get-started/choose-your-path/install-timescaledb) instance. - Your [connection details](/docs/integrate/find-connection-details). * [qStudio](https://www.timestored.com/qstudio/download) downloaded and installed. ## Connect qStudio to your service [Section titled “Connect qStudio to your service”](#connect-qstudio-to-your-service) To connect to Tiger Cloud: 1. **Start qStudio** 2. **Click `Server` > `Add Server`** 3. **Configure the connection** * For `Server Type`, select `Postgres`. * For `Connect By`, select `Host`. * For `Host`, `Port`, `Database`, `Username`, and `Password`, use your [connection details](/docs/integrate/find-connection-details). ![qStudio connected to a Tiger Cloud service](/docs/_astro/integrations-qstudio.B96JKZDF_tj6WA.webp) 4. **Click `Test`** qStudio indicates whether the connection works. 5. **Click `Add`** The server is listed in the `Server Tree`. You have successfully integrated qStudio with Tiger Cloud. # Secure connectivity > Connect to Tiger Data with VPNs, tunnels, and secure access The guides below show how to connect your cloud or on-prem infrastructure to Tiger Data over private, secure links, so you can keep database traffic off of the public internet, and meet compliance and security requirements. ## Integrations [Section titled “Integrations”](#integrations) [Amazon Web Services](/docs/integrate/secure-connectivity/aws) [Connect your AWS infrastructure to Tiger Cloud using AWS Transit Gateway](/docs/integrate/secure-connectivity/aws) [Corporate data center](/docs/integrate/secure-connectivity/corporate-data-center) [Connect your on-premise infrastructure to Tiger Cloud using AWS Transit Gateway](/docs/integrate/secure-connectivity/corporate-data-center) [Google Cloud](/docs/integrate/secure-connectivity/google-cloud) [Connect your Google Cloud infrastructure to Tiger Cloud using AWS Transit Gateway](/docs/integrate/secure-connectivity/google-cloud) [Microsoft Azure](/docs/integrate/secure-connectivity/microsoft-azure) [Connect your Azure infrastructure to Tiger Cloud on AWS using AWS Transit Gateway](/docs/integrate/secure-connectivity/microsoft-azure) # Integrate Amazon Web Services with Tiger Cloud > Connect your AWS infrastructure to Tiger Cloud using AWS Transit Gateway [Amazon Web Services (AWS)](https://aws.amazon.com/) is a comprehensive cloud computing platform that provides on-demand infrastructure, storage, databases, AI, analytics, and security services to help businesses build, deploy, and scale applications in the cloud. This page explains how to integrate your AWS infrastructure with Tiger Cloud using [AWS Transit Gateway](https://aws.amazon.com/transit-gateway/). ## Prerequisites for this integration guide To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service). - An [AWS Transit Gateway](https://docs.aws.amazon.com/vpc/latest/tgw/tgw-getting-started.html) set up. Note This feature is currently not supported for Tiger Cloud on Microsoft Azure. ## Connect your AWS infrastructure to your services [Section titled “Connect your AWS infrastructure to your services”](#connect-your-aws-infrastructure-to-your-services) To connect to Tiger Cloud: 1. **Create a Peering VPC in Tiger Cloud Console** Open [Tiger Console](https://console.cloud.tigerdata.com/), then: 1. In `Security` > `VPC`, click `Create a VPC`: ![Creating a new peering VPC in Tiger Console](/docs/_astro/add-peering-vpc-tiger-cloud.BHFW98a8_Hcuw1.webp) 2. Choose your region and IP range, name your VPC, then click `Create VPC`: ![Configuring VPC region, IP range, and name](/docs/_astro/configure-peering-vpc-tiger-cloud.DZom78te_Zspfuq.webp) Your service and Peering VPC must be in the same AWS region. The number of Peering VPCs you can create in your project depends on your [pricing plan](/docs/deploy/tiger-cloud/tiger-cloud-aws/pricing-and-account-management). If you need another Peering VPC, either contact or change your plan in [Tiger Console](https://console.cloud.tigerdata.com/). 3. Add a peering connection: 1. In the `VPC Peering` column, click `Add`. 2. Provide your AWS account ID, Transit Gateway ID, CIDR ranges, and AWS region. Tiger Cloud creates a new isolated connection for every unique Transit Gateway ID. ![Adding a peering connection with AWS account ID and Transit Gateway details](/docs/_astro/add-peering-tiger-cloud.9tTfnXvs_Z1YfI1i.webp) 3. Click `Add connection`. 2. **Accept and configure peering connection in your AWS account** Once your peering connection appears as `Processing`, you can accept and configure it in AWS: 1. Accept the peering request coming from Tiger Cloud. The request can take up to 5 min to arrive. Within 5 more minutes after accepting, the peering should appear as `Connected` in Tiger Console. 2. Configure at least the following in your AWS account networking: * Your subnet route table to route traffic to your Transit Gateway for the Peering VPC CIDRs. * Your Transit Gateway route table to route traffic to the newly created Transit Gateway peering attachment for the Peering VPC CIDRs. * Security groups to allow outbound TCP 5432. 3. **Attach a Tiger Cloud service to the Peering VPC** In [Tiger Console](https://console.cloud.tigerdata.com/dashboard/services): 1. Select the service you want to connect to the Peering VPC. 2. Click `Operations` > `Security` > `VPC`. 3. Select the VPC, then click `Attach VPC`. You cannot attach a Tiger Cloud service to multiple Tiger Cloud VPCs at the same time. You have successfully integrated your AWS infrastructure with Tiger Cloud. # Integrate your data center with Tiger Cloud > Connect your on-premise infrastructure to Tiger Cloud using AWS Transit Gateway This page explains how to integrate your corporate on-premise infrastructure with Tiger Cloud using [AWS Transit Gateway](https://aws.amazon.com/transit-gateway/). ## Prerequisites for this integration guide To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service). - Your [connection details](/docs/integrate/find-connection-details). * An [AWS Transit Gateway](https://docs.aws.amazon.com/vpc/latest/tgw/tgw-getting-started.html) set up. Note This feature is currently not supported for Tiger Cloud on Microsoft Azure. ## Connect your on-premise infrastructure to your services [Section titled “Connect your on-premise infrastructure to your services”](#connect-your-on-premise-infrastructure-to-your-services) To connect to Tiger Cloud, first establish connectivity between your on-premise infrastructure and AWS. See [Centralize network connectivity using AWS Transit Gateway](https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/centralize-network-connectivity-using-aws-transit-gateway.html). Then: 1. **Create a Peering VPC in Tiger Cloud Console** Open [Tiger Console](https://console.cloud.tigerdata.com/), then: 1. In `Security` > `VPC`, click `Create a VPC`: ![Creating a new peering VPC in Tiger Console](/docs/_astro/add-peering-vpc-tiger-cloud.BHFW98a8_Hcuw1.webp) 2. Choose your region and IP range, name your VPC, then click `Create VPC`: ![Configuring VPC region, IP range, and name](/docs/_astro/configure-peering-vpc-tiger-cloud.DZom78te_Zspfuq.webp) Your service and Peering VPC must be in the same AWS region. The number of Peering VPCs you can create in your project depends on your [pricing plan](/docs/deploy/tiger-cloud/tiger-cloud-aws/pricing-and-account-management). If you need another Peering VPC, either contact or change your plan in [Tiger Console](https://console.cloud.tigerdata.com/). 3. Add a peering connection: 1. In the `VPC Peering` column, click `Add`. 2. Provide your AWS account ID, Transit Gateway ID, CIDR ranges, and AWS region. Tiger Cloud creates a new isolated connection for every unique Transit Gateway ID. ![Adding a peering connection with AWS account ID and Transit Gateway details](/docs/_astro/add-peering-tiger-cloud.9tTfnXvs_Z1YfI1i.webp) 3. Click `Add connection`. 2. **Accept and configure peering connection in your AWS account** Once your peering connection appears as `Processing`, you can accept and configure it in AWS: 1. Accept the peering request coming from Tiger Cloud. The request can take up to 5 min to arrive. Within 5 more minutes after accepting, the peering should appear as `Connected` in Tiger Console. 2. Configure at least the following in your AWS account networking: * Your subnet route table to route traffic to your Transit Gateway for the Peering VPC CIDRs. * Your Transit Gateway route table to route traffic to the newly created Transit Gateway peering attachment for the Peering VPC CIDRs. * Security groups to allow outbound TCP 5432. 3. **Attach a Tiger Cloud service to the Peering VPC** In [Tiger Console](https://console.cloud.tigerdata.com/dashboard/services): 1. Select the service you want to connect to the Peering VPC. 2. Click `Operations` > `Security` > `VPC`. 3. Select the VPC, then click `Attach VPC`. You cannot attach a Tiger Cloud service to multiple Tiger Cloud VPCs at the same time. You have successfully integrated your corporate data center with Tiger Cloud. # Integrate Google Cloud with Tiger Cloud > Connect your Google Cloud infrastructure to Tiger Cloud using AWS Transit Gateway [Google Cloud](https://cloud.google.com/?hl=en) is a suite of cloud computing services, offering scalable infrastructure, AI, analytics, databases, security, and developer tools to help businesses build, deploy, and manage applications. This page explains how to integrate your Google Cloud infrastructure with Tiger Cloud using [AWS Transit Gateway](https://aws.amazon.com/transit-gateway/). ## Prerequisites for this integration guide To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service). - An [AWS Transit Gateway](https://docs.aws.amazon.com/vpc/latest/tgw/tgw-getting-started.html) set up. - Connectivity between Google Cloud and AWS established. See [Connect HA VPN to AWS peer gateways](https://cloud.google.com/network-connectivity/docs/vpn/how-to/connect-ha-vpn-aws-peer-gateway). Note This feature is currently not supported for Tiger Cloud on Microsoft Azure. ## Connect your Google Cloud infrastructure to your services [Section titled “Connect your Google Cloud infrastructure to your services”](#connect-your-google-cloud-infrastructure-to-your-services) To connect to Tiger Cloud: 1. **Create a Peering VPC in Tiger Cloud Console** Open [Tiger Console](https://console.cloud.tigerdata.com/), then: 1. In `Security` > `VPC`, click `Create a VPC`: ![Creating a new peering VPC in Tiger Console](/docs/_astro/add-peering-vpc-tiger-cloud.BHFW98a8_Hcuw1.webp) 2. Choose your region and IP range, name your VPC, then click `Create VPC`: ![Configuring VPC region, IP range, and name](/docs/_astro/configure-peering-vpc-tiger-cloud.DZom78te_Zspfuq.webp) Your service and Peering VPC must be in the same AWS region. The number of Peering VPCs you can create in your project depends on your [pricing plan](/docs/deploy/tiger-cloud/tiger-cloud-aws/pricing-and-account-management). If you need another Peering VPC, either contact or change your plan in [Tiger Console](https://console.cloud.tigerdata.com/). 3. Add a peering connection: 1. In the `VPC Peering` column, click `Add`. 2. Provide your AWS account ID, Transit Gateway ID, CIDR ranges, and AWS region. Tiger Cloud creates a new isolated connection for every unique Transit Gateway ID. ![Adding a peering connection with AWS account ID and Transit Gateway details](/docs/_astro/add-peering-tiger-cloud.9tTfnXvs_Z1YfI1i.webp) 3. Click `Add connection`. 2. **Accept and configure peering connection in your AWS account** Once your peering connection appears as `Processing`, you can accept and configure it in AWS: 1. Accept the peering request coming from Tiger Cloud. The request can take up to 5 min to arrive. Within 5 more minutes after accepting, the peering should appear as `Connected` in Tiger Console. 2. Configure at least the following in your AWS account networking: * Your subnet route table to route traffic to your Transit Gateway for the Peering VPC CIDRs. * Your Transit Gateway route table to route traffic to the newly created Transit Gateway peering attachment for the Peering VPC CIDRs. * Security groups to allow outbound TCP 5432. 3. **Attach a Tiger Cloud service to the Peering VPC** In [Tiger Console](https://console.cloud.tigerdata.com/dashboard/services): 1. Select the service you want to connect to the Peering VPC. 2. Click `Operations` > `Security` > `VPC`. 3. Select the VPC, then click `Attach VPC`. You cannot attach a Tiger Cloud service to multiple Tiger Cloud VPCs at the same time. You have successfully integrated your Google Cloud infrastructure with Tiger Cloud. # Integrate Microsoft Azure with Tiger Cloud > Connect your Azure infrastructure to Tiger Cloud on Azure using Azure Private Link [Microsoft Azure](https://azure.microsoft.com/en-gb/) is a cloud computing platform and services suite, offering infrastructure, AI, analytics, security, and developer tools to help businesses build, deploy, and manage applications. This page explains how to integrate your Microsoft Azure infrastructure with Tiger Cloud on Azure using [Azure Private Link](https://learn.microsoft.com/en-us/azure/private-link/private-link-overview). Note This feature is currently not supported for Tiger Cloud on AWS. ## Prerequisites for this procedure To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service). - An [Azure account](https://azure.microsoft.com/en-gb/pricing/purchase-options/azure-account) with an active subscription. - [Permissions](https://learn.microsoft.com/en-us/azure/private-link/rbac-permissions) configured to create private endpoints. - An [Azure Virtual Network](https://learn.microsoft.com/en-us/azure/virtual-network/quickstart-create-virtual-network?tabs=portal) with a subnet for the resources you will connect to Tiger Cloud. ## Set up a private endpoint connection Take the following steps to connect your Tiger Cloud service to a Private Link endpoint. 1. **Create an Azure subscription authorization** 1. In [Tiger Console](https://console.cloud.tigerdata.com/dashboard/azure-privatelink), select `Security` > `Private Endpoints` > `Configure Private Endpoint Connection`. ![Azure Private Link authorization](/docs/_astro/configure-private-link-authorization.BVll5xeW_2pKWBB.webp) 2. In `Cloud provider`, select `Azure`. 3. In `Principal ID`, enter your [Azure subscription ID](https://learn.microsoft.com/en-us/azure/azure-portal/get-subscription-tenant-id#find-your-azure-subscription). Give your authorization a name, for convenience. Warning Click the checkmark next to `Connection name` to save your authorization. Otherwise, your input is discarded. 4. Under `Alias`, copy the alias for the region in which you need to create the connection. Choose the region closest to your Azure resources for optimal performance. 5. Click `Done`. Tiger Cloud confirms your authorization. Once it is confirmed, you can connect multiple endpoints from the same authorized subscription. 2. **Create a private endpoint in Azure** 1. In [Azure Portal](https://azure.microsoft.com/en-us/get-started/azure-portal), go to `Private endpoints` and click `Create`. ![Create Azure Private Endpoint](/docs/_astro/create-private-endpoint-azure.BEjIkaB7_Z13XlPp.webp) 2. In `Subscription`, select the subscription you have previously authorized in Tiger Cloud. 3. In `Resource group`, select an existing resource group or create a new one for your private endpoint. 4. Provide a name for your endpoint. 5. Select the region where your Virtual Network is deployed, then click `Next: Resource`. 6. In `Connection method`, select `Connect to an Azure resource by resource ID or alias`. 7. In `Resource ID or alias`, paste the alias you have copied from Tiger Console. 8. In `Request message`, enter your [Tiger Cloud project ID](/docs/integrate/find-connection-details), then click `Next: Virtual Network`. 9. Select the Virtual Network that contains the resources you want to connect to Tiger Cloud, then choose the subnet where the endpoint's private IP will be created. Optionally select an application security group, then click `Next: DNS`. 10. Optionally configure private DNS integration and tags for your endpoint, then click `Next: Review + create`. 11. Review your config and click `Create`. Azure creates your private endpoint. Wait for the deployment to succeed. 12. Go to `Private endpoints` and copy the private endpoint IP from the `Private IP` column. 3. **Sync the connection** 1. In [Tiger Console](https://console.cloud.tigerdata.com/dashboard/azure-privatelink) > `Security` > `Private Endpoints`, click `Refresh`. Tiger Cloud automatically approves connections from authorized subscriptions. Your connection appears in the list. ![Azure Private Endpoint IP](/docs/_astro/private-link-add-ip.CXR8Gl7x_Z2lS5se.webp) 2. Under `IP Address` click `Add IP` and paste the private endpoint IP address you have copied from Azure Portal. 3. Under `Services` click `Attach service`. Select your service from the dropdown and click `Attach`. You can attach a service to one private endpoint. 4. From a VM inside your Azure VNet, connect to your service using a connection string with your [connection details](/docs/integrate/find-connection-details). You should be able to connect successfully. ## Manage connections * To detach a service from a private endpoint connection, go to `Security` > `Private Endpoints`, expand the arrow in the `Services` column, and click the trash icon next to the service connection string. * To edit or remove an endpoint connection, go to `Security` > `Private Endpoints` and click the three dots next to the connection in the list. Select `Edit` or `Disconnect`, respectively. You need to detach all services from a private endpoint connection before deleting it. * To remove an authorization, click `Manage Authorizations` > trash bin icon. You need to disconnect all relevant endpoint connections before removing an authorization. # Troubleshoot > Troubleshoot common problems that occur when integrating Tiger Cloud services with third-party solutions ### JDBC authentication type is not supported **Message**: * Check that your connection definition references your JDBC database with correct URL syntax, username, and password. The authentication type 10 is not supported. When connecting to Tiger Cloud with a Java Database Connectivity (JDBC) driver, you might get this error message. Your Tiger Cloud authentication type doesn't match your JDBC driver's supported authentication types. The recommended approach is to upgrade your JDBC driver to a version that supports `scram-sha-256` encryption. If that isn't an option, you can change the authentication type for your Tiger Cloud service to `md5`. Note that `md5` is less secure, and is provided solely for compatibility with older clients. For information on changing your authentication type, see the documentation on [resetting your service password](/docs/deploy/tiger-cloud/tiger-cloud-aws/service-management/service-management#reset-your-service-password). # Integrate > Integrate Tiger Cloud with third-party solutions to extend what you can do with your data ![https://assets.timescale.com/docs/icons/postgresql-logo.png](https://assets.timescale.com/docs/icons/postgresql-logo.png) A Tiger Cloud service is a PostgreSQL database instance extended by Tiger Data with custom capabilities. This means that any third-party solution that you can integrate with PostgreSQL, you can also integrate with Tiger Cloud. See [the full list](https://slashdot.org/software/p/postgresql/integrations/) of PostgreSQL integrations. Some of the most in-demand integrations are listed below. Connectors Name Description ![tigerlake](https://assets.timescale.com/docs/icons/apache-iceberg-logo.png)[TigerLake](/docs/integrate/connectors/destination/tigerlake) Synchronize data to external destinations like data lakes ![postgres](https://assets.timescale.com/docs/icons/postgresql-logo.png)[Sync from PostgreSQL](/docs/integrate/connectors/source/sync-from-postgres) Ingest data from external PostgreSQL databases ![s3](https://assets.timescale.com/docs/icons/aws-logo.png)[Sync from S3](/docs/integrate/connectors/source/sync-from-s3) Ingest data from Amazon S3 buckets ![kafka](https://assets.timescale.com/docs/icons/kafka-logo.png)[Sync from Kafka](/docs/integrate/connectors/source/sync-from-kafka) Stream data from Apache Kafka topics Coding Name Description ![code](https://assets.timescale.com/docs/icons/code-icon.svg)[Start coding with Tiger Data](/docs/integrate/code) Explore code examples, SDKs, and client libraries in popular programming languages Authentication and security Name Description ![auth](https://assets.timescale.com/docs/icons/auth-logo.png)[Auth.js ↗](https://authjs.dev/getting-started/adapters/pg?framework=next-js) Implement authentication and authorization for web applications ![auth0](https://assets.timescale.com/docs/icons/auth0-logo.png)[Auth0 ↗](https://auth0.com/blog/configuring-postgresql-as-auth0-custom-database/) Securely manage user authentication and access controls for applications ![okta](https://assets.timescale.com/docs/icons/okta-logo.png)[Okta ↗](https://help.okta.com/oag/en-us/content/topics/access-gateway/integrate-app-datastores.htm) Secure authentication and user identity management for applications BI and visualization Name Description ![cubejs](https://assets.timescale.com/docs/icons/cube-js-logo.png)[Cube.js ↗](https://cube.dev/integrations/Timescale-API) Build and optimize data APIs for analytics applications ![looker](https://assets.timescale.com/docs/icons/looker-logo.png)[Looker ↗](https://cloud.google.com/looker/docs/db-config-postgresql) Explore, analyze, and share business insights with a BI platform ![metabase](https://assets.timescale.com/docs/icons/metabase-logo.png)[Metabase ↗](https://www.metabase.com/data_sources/postgresql) Create dashboards and visualize business data without SQL expertise ![power-bi](https://assets.timescale.com/docs/icons/power-bi-logo.png)[Power BI](/docs/integrate/bi-vizualization/power-bi) Visualize data and build interactive dashboards with Microsoft Power BI ![superset](https://assets.timescale.com/docs/icons/superset-logo.png)[Superset ↗](https://superset.apache.org/docs/configuration/databases#timescaledb) Create and explore data visualizations and dashboards ![tableau](https://assets.timescale.com/docs/icons/tableau-logo.png)[Tableau](/docs/integrate/bi-vizualization/tableau) Create interactive visualizations and dashboards with Tableau Configuration and deployment Name Description ![azure-functions](https://assets.timescale.com/docs/icons/azure-functions-logo.png)[Azure Functions ↗](https://github.com/Azure/azure-functions-postgresql-extension) Run event-driven serverless code without managing infrastructure ![deno-deploy](https://assets.timescale.com/docs/icons/deno-deploy-logo.png)[Deno Deploy ↗](https://docs.deno.com/deploy/manual/postgres/) Deploy and run JavaScript and TypeScript applications at the edge ![flyway](https://assets.timescale.com/docs/icons/flyway-logo.png)[Flyway ↗](https://documentation.red-gate.com/flyway/reference/database-driver-reference/timescaledb) Manage and automate database migrations using version control ![kubernetes](https://assets.timescale.com/docs/icons/kubernets-logo.png)[Kubernetes](/docs/integrate/configuration-deployment/kubernetes) Deploy and manage containerized applications with Kubernetes ![liquibase](https://assets.timescale.com/docs/icons/liquibase-logo.png)[Liquibase ↗](https://docs.liquibase.com/start/tutorials/postgresql/postgresql.html) Track, version, and automate database schema changes ![pulumi](https://assets.timescale.com/docs/icons/pulimi-logo.png)[Pulumi ↗](https://www.pulumi.com/registry/packages/timescale/) Define and manage cloud infrastructure using code ![render](https://assets.timescale.com/docs/icons/render-logo.png)[Render ↗](https://render.com/docs/postgresql) Deploy and scale web applications and services easily ![terraform](https://assets.timescale.com/docs/icons/terraform-logo.png)[Terraform](/docs/integrate/configuration-deployment/terraform) Provision and manage infrastructure as code with Terraform Data engineering and extract, transform, load Name Description ![airbyte](https://assets.timescale.com/docs/icons/airbyte-logo.png)[Airbyte ↗](https://docs.airbyte.com/integrations/sources/postgres) Sync data between various sources and destinations ![sagemaker](https://assets.timescale.com/docs/icons/amazon-sagemaker-logo.png)[Amazon SageMaker](/docs/integrate/data-engineering-etl/amazon-sagemaker) Build, train, and deploy ML models with Amazon SageMaker ![airflow](https://assets.timescale.com/docs/icons/airflow-logo.png)[Apache Airflow](/docs/integrate/data-engineering-etl/apache-airflow) Programmatically author, schedule, and monitor workflows ![beam](https://assets.timescale.com/docs/icons/beam-logo.png)[Apache Beam ↗](https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/jdbc/JdbcIO.html) Build and execute batch and streaming data pipelines ![kafka](https://assets.timescale.com/docs/icons/kafka-logo.png)[Apache Kafka](/docs/integrate/data-engineering-etl/apache-kafka) Stream high-performance data pipelines and analytics ![lambda](https://assets.timescale.com/docs/icons/lambda-logo.png)[AWS Lambda](/docs/integrate/data-engineering-etl/aws-lambda) Run serverless code without managing infrastructure ![dbt](https://assets.timescale.com/docs/icons/dbt-logo.png)[dbt ↗](https://dbt-timescaledb.debruyn.dev/) Transform and model data in your warehouse using SQL-based workflows ![debezium](https://assets.timescale.com/docs/icons/debezium-logo.png)[Debezium](/docs/integrate/data-engineering-etl/debezium) Capture and stream real-time database changes ![decodable](https://assets.timescale.com/docs/icons/decodable-logo.png)[Decodable](/docs/integrate/data-engineering-etl/decodable) Build and manage data pipelines effortlessly ![deltalake](https://assets.timescale.com/docs/icons/delta-lake-logo.png)[DeltaLake ↗](https://github.com/delta-io/delta/blob/master/connectors/README.md) Enhance data lakes with ACID transactions and schema enforcement ![firebase](https://assets.timescale.com/docs/icons/firebase-logo.png)[Firebase Wrapper ↗](https://firebase.google.com/products/data-connect) Simplify interactions with Firebase services ![stitch](https://assets.timescale.com/docs/icons/stitch-logo.png)[Stitch ↗](https://stitch-docs.netlify.app/docs/integrations/databases/postgresql) Extract, load, and transform data from various sources ![supabase](https://assets.timescale.com/docs/icons/supabase-logo.svg)[Supabase](/docs/integrate/data-engineering-etl/supabase) Integrate with Supabase for backend services Data ingestion and streaming Name Description ![spark](https://assets.timescale.com/docs/icons/spark-logo.png)[Apache Spark ↗](https://spark.apache.org/docs/latest/sql-data-sources-jdbc.html) Process large-scale data workloads quickly using distributed computing ![confluent](https://assets.timescale.com/docs/icons/confluent-logo.png)[Confluent ↗](https://docs.confluent.io/cloud/current/connectors/cc-postgresql-sink.html) Manage and scale Apache Kafka-based event streaming applications ![electricsql](https://assets.timescale.com/docs/icons/electric-sql-logo.png)[ElectricSQL ↗](https://electric-sql.com/docs/intro) Enable real-time synchronization between databases and frontend applications ![emqx](https://assets.timescale.com/docs/icons/emqx-logo.png)[EMQX](/docs/integrate/data-ingestion-streaming/emqx) Stream MQTT messages from an EMQX broker into Tiger Cloud ![estuary](https://assets.timescale.com/docs/icons/estuary-logo.png)[Estuary ↗](https://docs.estuary.dev/reference/Connectors/materialization-connectors/timescaledb/) Stream and synchronize data in real time between different systems ![fivetran](https://assets.timescale.com/docs/icons/fivetran-logo.png)[Fivetran](/docs/integrate/data-ingestion-streaming/fivetran) Sync data from multiple sources to your data warehouse ![flink](https://assets.timescale.com/docs/icons/flink-logo.png)[Flink ↗](https://nightlies.apache.org/flink/flink-cdc-docs-release-3.1/docs/connectors/flink-sources/postgres-cdc/) Process real-time data streams with fault-tolerant distributed computing ![highbyte](https://assets.timescale.com/docs/icons/highbyte-logo.svg)[HighByte](/docs/integrate/data-ingestion-streaming/highbyte) Connect operational technology sources, model the data, and stream it ![redpanda](https://assets.timescale.com/docs/icons/red-panda-logo.png)[Redpanda ↗](https://www.redpanda.com/blog/build-data-stream-detect-anomalies-timescale-kafka-connect) Stream and process real-time data as a Kafka-compatible platform ![striim](https://assets.timescale.com/docs/icons/striim-logo.png)[Striim ↗](https://www.striim.com/connectors/postgresql/) Ingest, process, and analyze real-time data streams ![ignition](https://assets.timescale.com/docs/icons/ignition-logo.png)[Ignition ↗](https://www.docs.inductiveautomation.com/docs/7.9/database-connections/connecting-to-databases/connecting-to-postgresql) Observability and alerting Name Description ![cloudwatch](https://assets.timescale.com/docs/icons/cloudwatch-logo.png)[Amazon CloudWatch](/docs/integrate/observability-alerting/cloudwatch) Monitor and analyze data from AWS and on-premises environments ![skywalking](https://assets.timescale.com/docs/icons/skywalking-logo.png)[Apache SkyWalking ↗](https://skywalking.apache.org/docs/main/next/en/setup/backend/backend-postgresql-monitoring/) Monitor, trace, and diagnose distributed applications ![azure-monitor](https://assets.timescale.com/docs/icons/azure-monitor-logo.png)[Azure Monitor ↗](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/concepts-monitoring) Collect and analyze telemetry data from cloud and on-premises environments ![dash0](https://assets.timescale.com/docs/icons/dash0-logo.png)[Dash0 ↗](https://www.dash0.com/hub/integrations/int_tiger_service/overview) OpenTelemetry Native Observability built on CNCF Open Standards ![datadog](https://assets.timescale.com/docs/icons/datadog-logo.png)[Datadog](/docs/integrate/observability-alerting/datadog) Comprehensive monitoring and analytics platform ![grafana](https://assets.timescale.com/docs/icons/grafana-logo.png)[Grafana](/docs/integrate/observability-alerting/grafana) Query, visualize, and alert on metrics and logs ![instana](https://assets.timescale.com/docs/icons/instana-logo.png)[IBM Instana ↗](https://www.ibm.com/docs/en/instana-observability/current?topic=technologies-monitoring-postgresql) Monitor application performance and detect issues in real-time ![jaeger](https://assets.timescale.com/docs/icons/jaeger-logo.png)[Jaeger ↗](https://www.jaegertracing.io/docs/2.0/storage/) Trace and diagnose distributed transactions for observability ![new-relic](https://assets.timescale.com/docs/icons/new-relic-logo.png)[New Relic ↗](https://docs.newrelic.com/docs/infrastructure/host-integrations/host-integrations-list/postgresql/postgresql-integration/) Monitor applications, infrastructure, and logs for performance insights ![opentelemetry](https://assets.timescale.com/docs/icons/open-telemetery-logo.png)[OpenTelemetry ↗](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/postgresqlreceiver) Collect and analyze telemetry data for observability across systems ![prometheus](https://assets.timescale.com/docs/icons/prometheus-logo.png)[Prometheus](/docs/integrate/observability-alerting/prometheus) Track performance and health of systems and applications ![signoz](https://assets.timescale.com/docs/icons/signoz-logo.png)[SigNoz ↗](https://signoz.io/docs/integrations/postgresql/) Monitor application performance with an open-source observability tool ![telegraf](https://assets.timescale.com/docs/icons/Influx-telegraf.svg)[Telegraf](/docs/integrate/observability-alerting/telegraf) Collect and ship metrics and events to monitoring platforms Development tools Name Description ![deepnote](https://assets.timescale.com/docs/icons/deepnote-logo.png)[Deepnote ↗](https://deepnote.com/docs/postgresql) Collaborate on data science projects with a cloud-based notebook platform ![django](https://assets.timescale.com/docs/icons/django-logo.png)[Django ↗](https://docs.djangoproject.com/en/5.1/ref/databases/#postgresql-notes) Develop scalable and secure web applications using a Python framework ![langchain](https://assets.timescale.com/docs/icons/long-chain-logo.png)[LangChain ↗](https://api.python.langchain.com/en/latest/postgres/index.html#) Build applications that integrate with language models ![rust](https://assets.timescale.com/docs/icons/rust-logo.png)[Rust ↗](https://github.com/sfackler/rust-postgres) Build high-performance, memory-safe applications ![streamlit](https://assets.timescale.com/docs/icons/streamlit-logo.png)[Streamlit ↗](https://docs.streamlit.io/develop/tutorials/databases/postgresql) Create interactive data applications and dashboards using Python Logging and system administration Name Description ![rsyslog](https://assets.timescale.com/docs/icons/rsyslog-logo.png)[RSyslog ↗](https://www.rsyslog.com/doc/configuration/modules/ompgsql.html) Collect, filter, and forward system logs for centralized logging ![schemaspy](https://assets.timescale.com/docs/icons/schemaspy-logo.png)[SchemaSpy ↗](https://wiki.postgresql.org/wiki/SchemaSpy) Generate database schema documentation and visualization Query and administration Name Description ![azure-data-studio](https://assets.timescale.com/docs/icons/azure-data-studio-logo.png)[Azure Data Studio](/docs/integrate/query-administration/azure-data-studio) Query, manage, and develop PostgreSQL databases ![dbeaver](https://assets.timescale.com/docs/icons/dbeaver-logo.png)[DBeaver](/docs/integrate/query-administration/dbeaver) Universal database tool with SQL editing and visualization ![forest-admin](https://assets.timescale.com/docs/icons/forest-admin-logo.png)[Forest Admin ↗](https://www.forestadmin.com/integrations/postgresql) Create admin panels and dashboards for business applications ![hasura](https://assets.timescale.com/docs/icons/hasura-logo.png)[Hasura ↗](https://hasura.io/docs/2.0/databases/postgres/timescale-cloud/) Instantly generate GraphQL APIs from databases with access control ![mode](https://assets.timescale.com/docs/icons/mode-logo.png)[Mode Analytics ↗](https://mode.com/integrations/postgresql/) Analyze data, create reports, and share insights with teams ![neon](https://assets.timescale.com/docs/icons/neon-logo.png)[Neon ↗](https://neon.com/docs/extensions/timescaledb) Run a cloud-native, serverless PostgreSQL database with automatic scaling ![pgadmin](https://assets.timescale.com/docs/icons/pgadmin-logo.png)[pgAdmin](/docs/integrate/query-administration/pgadmin) Manage and administer PostgreSQL databases through a graphical interface ![postgresql](https://assets.timescale.com/docs/icons/postgresql-logo.png)[PostgreSQL](/docs/integrate/query-administration/postgresql) Connect using native PostgreSQL tools and protocols ![prisma](https://assets.timescale.com/docs/icons/prisma-logo.png)[Prisma ↗](https://www.prisma.io/docs/orm/overview/databases/postgresql) Simplify database access with an open-source ORM for Node.js ![psql](https://assets.timescale.com/docs/icons/psql-logo.png)[psql](/docs/integrate/query-administration/psql) Command-line interface for PostgreSQL ![qlik](https://assets.timescale.com/docs/icons/qlik-logo.png)[Qlik Replicate ↗](https://help.qlik.com/en-US/replicate/November2024/Content/Replicate/Main/%7BC.PG%7D/postgresql.htm) Move and synchronize data across multiple database platforms ![qstudio](https://assets.timescale.com/docs/icons/qstudio-logo.png)[qStudio](/docs/integrate/query-administration/qstudio) SQL editor and database management tool ![redash](https://assets.timescale.com/docs/icons/redash-logo.png)[Redash ↗](https://redash.io/data-sources/postgresql/) Query, visualize, and share data from multiple sources ![sequelize](https://assets.timescale.com/docs/icons/sequelize-logo.png)[Sequelize ↗](https://sequelize.org/docs/v7/databases/postgres/) Interact with SQL databases in Node.js using an ORM ![sqlalchemy](https://assets.timescale.com/docs/icons/sql-alchemy-logo.png)[SQLalchemy ↗](https://docs.sqlalchemy.org/en/20/dialects/postgresql.html) Manage database operations using a Python SQL toolkit and ORM ![stepzen](https://assets.timescale.com/docs/icons/stepzen-logo.png)[StepZen ↗](https://www.ibm.com/docs/en/api-connect-graphql/saas?topic=apis-tutorial-create-graphql-api-postgresql-database) Build and deploy GraphQL APIs with data from multiple sources ![typeorm](https://assets.timescale.com/docs/icons/typeorm-logo.png)[TypeORM ↗](https://typeorm.biunav.com/en/connection-options.html#postgres-cockroachdb-connection-options) Work with databases in TypeScript and JavaScript using an ORM Secure connectivity to Tiger Cloud Name Description ![aws](https://assets.timescale.com/docs/icons/aws-logo.png)[Amazon Web Services](/docs/integrate/secure-connectivity/aws) Connect your AWS infrastructure to Tiger Cloud ![data-center](https://assets.timescale.com/docs/icons/corporate-data-center-logo.png)[Corporate data center](/docs/integrate/secure-connectivity/corporate-data-center) Connect your on-premise data center to Tiger Cloud ![google-cloud](https://assets.timescale.com/docs/icons/google-cloud-logo.png)[Google Cloud](/docs/integrate/secure-connectivity/google-cloud) Connect your Google Cloud infrastructure to Tiger Cloud ![azure](https://assets.timescale.com/docs/icons/azure-logo.png)[Microsoft Azure](/docs/integrate/secure-connectivity/microsoft-azure) Connect your Azure infrastructure to Tiger Cloud Workflow automation and no-code tools Name Description ![appsmith](https://assets.timescale.com/docs/icons/appsmith-logo.png)[Appsmith ↗](https://docs.appsmith.com/connect-data/reference/querying-postgres) Create internal business applications with a low-code platform ![dbos](https://assets.timescale.com/docs/icons/dbos-logo.png)[DBOS ↗](https://docs.dbos.dev/integrations/tigerdata) Add durable workflows to your code and make apps resilient to any failure ![n8n](https://assets.timescale.com/docs/icons/n8n-logo.png)[n8n ↗](https://n8n.io/integrations/redis/and/timescaledb/) Automate workflows and integrate services with a no-code platform ![retool](https://assets.timescale.com/docs/icons/retool-logo.png)[Retool ↗](https://retool.com/integrations/postgresql) Build custom internal tools quickly using a drag-and-drop interface ![tooljet](https://assets.timescale.com/docs/icons/tooljet-logo.png)[Tooljet ↗](https://docs.tooljet.ai/docs/data-sources/postgresql/) Develop internal tools and business applications with a low-code builder ![zapier](https://assets.timescale.com/docs/icons/zapier-logo.png)[Zapier ↗](https://zapier.com/apps/postgresql/integrations) Automate workflows by connecting different applications and services # Component kitchen sink > Internal test page rendering every component and content element used in the docs, for visual and regression checks. This page renders every component and content element used across the docs so that visual and interactivity regressions show up on one preview URL. It is hidden from search engines and from on-site search, and is not in the sidebar. ## Text and inline formatting [Section titled “Text and inline formatting”](#text-and-inline-formatting) Body text with **bold**, *italic*, and `inline code`. A link to [the connection details page](/docs/integrate/find-connection-details) and an external link to [Tiger Data](https://www.tigerdata.com/). Product names resolve from constants: PostgreSQL, Tiger Cloud, self-hosted TimescaleDB, TimescaleDB. > A blockquote. Use it for quoted text or asides. ### Unordered list [Section titled “Unordered list”](#unordered-list) * First item * Second item with `code` * Third item * Nested item * Another nested item ### Ordered list [Section titled “Ordered list”](#ordered-list) 1. First step 2. Second step 3. Third step ## Table [Section titled “Table”](#table) | Column | Type | Description | | ----------- | ------------------ | ----------------- | | `time` | `timestamptz` | Sample timestamp | | `device_id` | `text` | Device identifier | | `value` | `double precision` | Measured value | ## Code blocks [Section titled “Code blocks”](#code-blocks) ```sql SELECT time_bucket('1 hour', time) AS bucket, avg(value) FROM metrics GROUP BY bucket ORDER BY bucket; ``` ```bash psql "postgres://user:pass@host:5432/dbname" ``` ```javascript import { Pool } from "pg"; const pool = new Pool({ connectionString: process.env.DATABASE_URL }); ``` ```typescript const greet = (name: string): string => `Hello, ${name}`; ``` ```json { "host": "example.tsdb.cloud", "port": 5432 } ``` ```yaml service: host: example.tsdb.cloud port: 5432 ``` ```text Plain text block with no syntax highlighting. ``` ## Callouts [Section titled “Callouts”](#callouts) Tips A tip callout for hints and best practices. Note A note callout for supplementary context. Custom title A note callout with an overridden title. Important An important callout for information that should not be skipped. Warning A warning callout for cautions and limitations. Callout with button A call-to-action callout with a button. [Try for free](/) ## Buttons [Section titled “Buttons”](#buttons) Button enabled[Download](/docs/)[Primary CTA](/docs/)Secondary defaultSecondary subtle ## Copy to clipboard [Section titled “Copy to clipboard”](#copy-to-clipboard) Copy Copy query ## Starlight tabs [Section titled “Starlight tabs”](#starlight-tabs) * Tiger Cloud Content for the Tiger Cloud tab. * Self-hosted TimescaleDB Content for the self-hosted TimescaleDB tab. ## Stainless sync tabs [Section titled “Stainless sync tabs”](#stainless-sync-tabs) * First tab Content for the first tab. * Second tab Content for the second tab. ## Numbered list [Section titled “Numbered list”](#numbered-list) 1. **A step with a sub-sequence** Do the first thing, which has lettered sub-steps: 1. Sub-step one. 2. Sub-step two. 3. Sub-step three. 2. **A step with a code block** Run the following: ```sql SELECT now(); ``` Confirm it returns the current time. ## Prerequisites [Section titled “Prerequisites”](#prerequisites) ## Prerequisites for this integration guide To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service), or a running instance of [self-hosted TimescaleDB](/docs/get-started/choose-your-path/install-timescaledb). - Your [connection details](/docs/integrate/find-connection-details). * Any additional tooling the page requires. ## Image [Section titled “Image”](#image) ![Diagram of a hypertable](/docs/_astro/hypertable-light.f-v6IDWg_Z1YDb49.webp) ## Author byline [Section titled “Author byline”](#author-byline) ![](https://avatars.githubusercontent.com/octocat?s=128) Jane Doe Developer advocate [GitHub](https://github.com/octocat) ## Related content cards [Section titled “Related content cards”](#related-content-cards) [Find your connection details](/docs/integrate/find-connection-details) [Locate the host, port, database, user, and password for your service or database.](/docs/integrate/find-connection-details) [Connect your app](/docs/integrate/code/connect-your-app) [Connect to your database from your preferred programming language.](/docs/integrate/code/connect-your-app) # Get to know Tiger Data > Concepts, comparisons, architecture, how features connect, and the glossary ## Explore by topic [Section titled “Explore by topic”](#explore-by-topic) [Overview & architecture](/docs/learn/deep-dive/whitepaper) [Architectural choices and optimizations that power TimescaleDB and Tiger Cloud for real-time analytics](/docs/learn/deep-dive/whitepaper) [Tiger Cloud](/docs/learn/tiger-cloud) [Managed service basics, supported regions, and features only available in the cloud](/docs/learn/tiger-cloud) [Capabilities & comparison](/docs/learn/capabilities-and-comparison/understand-capabilities) [How TimescaleDB capabilities fit together and how Tiger Cloud and self-hosted options compare](/docs/learn/capabilities-and-comparison/understand-capabilities) [Table layout: wide, narrow, or medium](/docs/learn/data-model/wide-narrow-medium-tables) [How table shape affects metrics, schema changes, and queries before you commit to a hypertable](/docs/learn/data-model/wide-narrow-medium-tables) [Data model: keys and time columns](/docs/learn/data-model/primary-keys-time-and-uniqueness) [Partition columns, primary keys, and unique constraints for hypertables, before you load data](/docs/learn/data-model/primary-keys-time-and-uniqueness) [Hypertables & chunks](/docs/learn/hypertables/understand-hypertables) [Time partitioning, chunk intervals, and how data is organized at scale](/docs/learn/hypertables/understand-hypertables) [Hypercore](/docs/learn/columnar-storage/understand-hypercore) [Columnar storage, compression, and how hypercore relates to hypertables and chunks](/docs/learn/columnar-storage/understand-hypercore) [Continuous aggregates (CAGGs)](/docs/learn/continuous-aggregates) [How continuous aggregates work, real-time aggregates and hierarchical aggregates, materialized hypertables, and timezone handling](/docs/learn/continuous-aggregates) [Data lifecycle](/docs/learn/data-lifecycle/data-retention/about-data-retention) [Retention, tiered storage, and Tiger Cloud policies for dropping old data and moving colder data](/docs/learn/data-lifecycle/data-retention/about-data-retention) [Deep dive](/docs/learn/deep-dive) [Hub for advanced topics and longer-form architecture material beyond the whitepaper](/docs/learn/deep-dive) [Glossary](/docs/learn/glossary) [Definitions for engineering, database, and product terms used throughout the docs](/docs/learn/glossary) For quickstarts, examples, and how-tos, see [Build](/docs/build). # Feature comparison > Compare features across Tiger Cloud on AWS, Tiger Cloud on Azure, and self-hosted TimescaleDB This page compares the features available in **Tiger Cloud on AWS**, **Tiger Cloud on Azure**, and **self-hosted TimescaleDB Community Edition**. **Best-in-class PostgreSQL performance** | Feature | Tiger Cloud on AWS | Tiger Cloud on Azure | Self-hosted TimescaleDB | | --------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :-----------------------------------------------------------------------------------------------------------------------------------: | :---------------------: | | Automatic partitioning via hypertables for efficient indexes and faster ingest | ✅ | ✅ | ✅ | | Continuous aggregates | ✅ | ✅ | ✅ | | Time/partition-oriented constraint exclusion for faster queries | ✅ | ✅ | ✅ | | Skip scans, ordered appends, custom optimizations for faster LIMIT and DISTINCT queries | ✅ | ✅ | ✅ | | Columnar storage for accelerated scans | ✅ | ✅ | ✅ | | Vectorized query execution (SIMD) | ✅ | ✅ | ✅ | | Specialized vector indexes for AI applications | ✅ | ✅ | ✅ | | Source connectors | [PostgreSQL](/docs/integrate/connectors/source/sync-from-postgres), [Amazon S3](/docs/integrate/connectors/source/sync-from-s3), [Apache Kafka](/docs/integrate/connectors/source/sync-from-kafka) | [PostgreSQL](/docs/integrate/connectors/source/sync-from-postgres), [Apache Kafka](/docs/integrate/connectors/source/sync-from-kafka) | Manual | | Destination connectors | [Iceberg on S3 Tables](/docs/integrate/connectors/destination/tigerlake) | ❌ | ❌ | | In-Console CSV, Parquet, and text file imports | ✅ | ❌ | ❌ | *** **Flexible analysis with full SQL** | Feature | Tiger Cloud on AWS | Tiger Cloud on Azure | Self-hosted TimescaleDB | | ---------------------------------------------------------------------------------------- | :----------------: | :------------------: | :---------------------: | | Complete PostgreSQL ecosystem (all PostgreSQL features, connectors, third-party drivers) | ✅ | ✅ | ✅ | | Cross-table JOINs for time-series and events with relational tables | ✅ | ✅ | ✅ | | Rich timestamp and timezone support | ✅ | ✅ | ✅ | | Flexible time-bucketing for time-oriented analysis | ✅ | ✅ | ✅ | | Advanced hyperfunctions (interpolation, approximation, visualization) | ✅ | ✅ | ✅ | | Geospatial and vector data types | ✅ | ✅ | ✅ | *** **Automated data management** | Feature | Tiger Cloud on AWS | Tiger Cloud on Azure | Self-hosted TimescaleDB | | --------------------------------------------------- | :----------------: | :------------------: | :---------------------: | | Columnstore compression (up to 98% storage savings) | ✅ | ✅ | ✅ | | Data retention policies | ✅ | ✅ | ✅ | | Data tiering with automated policies | ✅ | ✅ | ❌ | | Data reordering for efficient disk scans | ✅ | ✅ | ✅ | | Data downsampling for efficient historical analysis | ✅ | ✅ | ✅ | | Background job scheduler and user-defined jobs | ✅ | ✅ | ✅ | | Automated chunk interval tuning | ✅ | ✅ | ❌ | *** **Enterprise scalability** | Feature | Tiger Cloud on AWS | Tiger Cloud on Azure | Self-hosted TimescaleDB | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :----------------: | :------------------: | :---------------------: | | Disaggregated compute and storage | ✅ | ✅ | Manual | | Dynamic compute resizing | ✅ | ✅ | ❌ | | Dynamic disk storage with usage-based pricing | ✅ | ✅ | ❌ | | Dynamic I/O provisioning for high-read/high-write performance | ✅ | ✅ | Manual | | Low-cost storage with infinite capacity | ✅ | ✅ | ❌ | | Transparent queries across high-performance and low-cost tiers | ✅ | ✅ | ❌ | | Read replicas with load balancing for seamless read scaling | ✅ | ✅ | Manual | | Connection pooling for connection scaling | ✅ | ✅ | Manual | | Automated resource-aware parameter tuning | ✅ | ✅ | ❌ | | Terraform for infrastructure-as-code control | ✅ | ✅ | ❌ | | Management interfaces ([Console](/docs/deploy/tiger-cloud/tiger-cloud-aws/service-management/service-overview), [REST API](/docs/reference/tiger-cloud-rest), [CLI](/docs/reference/tiger-cloud/tiger-cli), [Tiger MCP](/docs/get-started/quickstart/mcp-cli)) | ✅ | ✅ | ❌ | *** **High availability and reliability** | Feature | Tiger Cloud on AWS | Tiger Cloud on Azure | Self-hosted TimescaleDB | | ----------------------------------------------------------- | :----------------: | :------------------: | :---------------------: | | Multi-AZ deployments for high availability | ✅ | ✅ | Manual | | Continuous incremental backup and automated restore | ✅ | ✅ | Manual | | Cross-region backup and restore | ✅ | ❌ | ❌ | | Point-in-time recovery and forking | ✅ | ✅ | Manual | | Regular database and disk snapshots for fast restore | ✅ | ✅ | ❌ | | Rapid recovery (fast database restart, remote disk remount) | ✅ | ✅ | ❌ | | Memory guard protections to avoid out-of-memory crashes | ✅ | ✅ | ❌ | | Decoupled control/data planes for greater resilience | ✅ | ✅ | ❌ | | Commercial SLAs | ✅ | ✅ | ❌ | *** **Automated upgrades and software patching** | Feature | Tiger Cloud on AWS | Tiger Cloud on Azure | Self-hosted TimescaleDB | | -------------------------------------------------------------------------- | :----------------: | :------------------: | :---------------------: | | Automated upgrades during maintenance windows | ✅ | ✅ | ❌ | | Phased, zero-downtime TimescaleDB and PostgreSQL minor upgrades | ✅ | ✅ | ❌ | | PostgreSQL major version upgrades with forking workflow and disk snapshots | ✅ | ✅ | ❌ | | HA-replica-aware coordinated upgrades | ✅ | ✅ | ❌ | | Fleet-wide version and stability monitoring with staged roll-out/roll-back | ✅ | ✅ | ❌ | *** **Security and compliance** | Feature | Tiger Cloud on AWS | Tiger Cloud on Azure | Self-hosted TimescaleDB | | ------------------------------------------------------------ | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :-----------------------------------------------------------------------------------------: | :---------------------: | | SOC 2 Type 2, GDPR, HIPAA certified compliance | ✅ | ✅ | ❌ | | Data encryption at rest (disk and backup) | ✅ | ✅ | Manual | | Data encryption in transit | ✅ | ✅ | Manual | | Database SSL with fully verifiable certificate chains | ✅ | ✅ | ❌ | | Control plane role-based access control | ✅ | ✅ | ❌ | | Database role-based access control | ✅ | ✅ | ✅ | | Multi-factor authentication | ✅ | ✅ | ❌ | | Corporate SSO and SAML | ✅ | ✅ | ❌ | | IP allow lists | ✅ | ✅ | Manual | | Private connectivity to your cloud network | [AWS PrivateLink](/docs/deploy/tiger-cloud/tiger-cloud-aws/security/aws-privatelink), [Transit Gateway](/docs/deploy/tiger-cloud/tiger-cloud-aws/security/transit-gateway), [VPC peering](/docs/deploy/tiger-cloud/tiger-cloud-aws/security/vpc) | [Azure Private Link](/docs/deploy/tiger-cloud/tiger-cloud-azure/security/azure-privatelink) | ❌ | | Layered database privilege escalation protections | ✅ | ✅ | ❌ | | Secure SDLC, vulnerability scanning, third-party pen testing | ✅ | ✅ | ❌ | *** **Deep observability** | Feature | Tiger Cloud on AWS | Tiger Cloud on Azure | Self-hosted TimescaleDB | | -------------------------------------------------------------------------- | :-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :---------------------: | | Operational database visibility (performance, regressions, optimization) | ✅ | ✅ | ❌ | | Automated query analysis and statistics | ✅ | ✅ | ❌ | | Per-query drill-downs (execution times, row results, plans, memory, cache) | ✅ | ✅ | ❌ | | In-Console metric visualization and system logs | ✅ | ✅ | ❌ | | Metric exporters | [CloudWatch](/docs/integrate/observability-alerting/cloudwatch), [Prometheus](/docs/integrate/observability-alerting/prometheus), [Datadog](/docs/integrate/observability-alerting/datadog) | [Azure Monitor](/docs/integrate/observability-alerting/azure-monitor), [Prometheus](/docs/integrate/observability-alerting/prometheus), [Datadog](/docs/integrate/observability-alerting/datadog) | ❌ | | Connection monitoring and management | ✅ | ✅ | Manual | *** **Production-grade support and operations** | Feature | Tiger Cloud on AWS | Tiger Cloud on Azure | Self-hosted TimescaleDB | | --------------------------------------------------------------------------- | :----------------: | :------------------: | :-----------------------------------------: | | 24/7 follow-the-sun support (APAC, EMEA, Americas) | ✅ | ✅ | [Contact sales](mailto:sales@tigerdata.com) | | Production support (severity 1) | ✅ | ✅ | [Contact sales](mailto:sales@tigerdata.com) | | Architectural reviews, data modeling, query optimization, migration support | ✅ | ✅ | [Contact sales](mailto:sales@tigerdata.com) | | 24/7 operational monitoring and control | ✅ | ✅ | [Contact sales](mailto:sales@tigerdata.com) | | 98%+ customer satisfaction (CSAT scores) | ✅ | ✅ | [Contact sales](mailto:sales@tigerdata.com) | *** Choosing a deployment * **Tiger Cloud on AWS**: Full feature set, including AWS-specific integrations. Private connectivity through AWS PrivateLink, Transit Gateway, and VPC peering, cross-region backup, an Amazon S3 source connector, an Iceberg destination connector for S3 Tables, and metric export to CloudWatch (as well as Prometheus and Datadog). * **Tiger Cloud on Azure**: The same core database and management platform. Private connectivity through Azure Private Link and metric export to Azure Monitor (as well as Prometheus and Datadog). Some AWS-specific integrations, such as the S3 and Iceberg connectors, VPC peering, cross-region backup, and CloudWatch, are not yet available. * **Self-hosted TimescaleDB**: Full SQL and time-series capabilities. You manage backups, high availability, scaling, and observability yourself. Managed operations and enterprise support are available by [contacting sales](mailto:sales@tigerdata.com). # Understand capabilities > Learn how TimescaleDB and Tiger Cloud capabilities work together to power time-series and analytics workloads TimescaleDB and Tiger Cloud extend PostgreSQL with powerful capabilities designed specifically for time-series data, real-time analytics, and event-driven workloads. These capabilities work together to provide a complete solution for ingesting, storing, querying, and analyzing massive datasets efficiently. ## Capabilities overview [Section titled “Capabilities overview”](#capabilities-overview) TimescaleDB and Tiger Cloud capabilities fall into the following categories: ### Data storage and organization [Section titled “Data storage and organization”](#data-storage-and-organization) * **[Hypertables](/docs/learn/hypertables/understand-hypertables)**: automatically partition time-series data into chunks for efficient data management at scale. * **[Hypercore](/docs/learn/columnar-storage/understand-hypercore)**: compress data into columnar storage delivering 90-95% storage reduction. ### Data processing and aggregation [Section titled “Data processing and aggregation”](#data-processing-and-aggregation) * **[Continuous aggregates](/docs/learn/continuous-aggregates)**: maintain pre-computed aggregations that update incrementally as new data arrives. * **[Hyperfunctions](/docs/build/data-management/hyperfunctions)**: analyze data with specialized SQL functions including statistical aggregation and percentiles. ### Data lifecycle management [Section titled “Data lifecycle management”](#data-lifecycle-management) * **[Data retention](/docs/learn/data-lifecycle/data-retention/about-data-retention)**: automatically drop old data based on time intervals, keeping storage costs under control. * **[Data tiering](/docs/learn/data-lifecycle/storage/about-storage-tiers)** (Tiger Cloud-exclusive capability): move older data to low-cost object storage while keeping it queryable with standard SQL. ### Schema optimization and automation [Section titled “Schema optimization and automation”](#schema-optimization-and-automation) * **[Schema optimization](/docs/learn/data-model/understand-database-schemas)**: use PostgreSQL features like indexes, constraints, triggers, tablespaces, and foreign data wrappers. * **[Jobs](/docs/build/data-management/create-and-manage-jobs)**: automate recurring tasks like continuous aggregate refreshes, data retention, and custom maintenance. ## Typical workflow [Section titled “Typical workflow”](#typical-workflow) Here’s how TimescaleDB capabilities work together in a typical time-series application: 1. **Data ingestion** Start by [creating a hypertable](/docs/learn/hypertables/understand-hypertables) for your time-series data. The hypertable automatically partitions data into time-based chunks, enabling efficient inserts and queries. For high-volume ingestion, [optimize your schema](/docs/learn/data-model/understand-database-schemas) with appropriate indexes and constraints. Use bulk insert methods like `COPY` or multi-row `INSERT` statements for best performance. For migrating existing data or importing from external sources, see [Migrate](/docs/migrate). 2. **Data optimization** [Hypercore](/docs/build/columnar-storage/setup-hypercore) is automatically enabled when you create a hypertable, providing columnar storage with advanced compression. This reduces storage by 90-95% while maintaining full query capabilities and delivering 100x to 1000x performance improvements for analytical queries. 3. **Real-time analytics** Create [continuous aggregates](/docs/build/continuous-aggregates/create-a-continuous-aggregate) to automatically maintain pre-computed summaries. Use [hyperfunctions](/docs/build/data-management/hyperfunctions) in your aggregates to calculate statistics, percentiles, time-weighted averages, and other specialized metrics. Query continuous aggregates instead of raw data for instant results on dashboards and reports. [Real-time aggregates](/docs/learn/continuous-aggregates/real-time-aggregates) ensure you see the latest data without waiting for batch processing. 4. **Data lifecycle** [Configure retention policies](/docs/learn/data-lifecycle/data-retention/about-data-retention) to automatically drop old data when it’s no longer needed. Retention works seamlessly with hypercore, removing entire chunks efficiently without impacting performance. Retention policies can preserve aggregated data in continuous aggregates even after dropping raw data, enabling long-term trend analysis without storing every data point. On Tiger Cloud, use [tiered storage](/docs/learn/data-lifecycle/storage/about-storage-tiers) to move older data to low-cost object storage while keeping it queryable. 5. **Automation** [Schedule jobs](/docs/build/data-management/create-and-manage-jobs) to automate hypercore compression, continuous aggregate refreshes, retention, and custom maintenance tasks. Jobs run reliably in the background and provide execution history for monitoring and troubleshooting. ## Capabilities by use case [Section titled “Capabilities by use case”](#capabilities-by-use-case) ### IoT and sensor data [Section titled “IoT and sensor data”](#iot-and-sensor-data) For IoT workloads with millions of devices generating continuous metrics: * **[Hypertables](/docs/learn/hypertables/understand-hypertables)**: partition data by time and optionally by device ID for optimal performance. * **[Hypercore](/docs/learn/columnar-storage/understand-hypercore)**: compress sensor readings with 95%+ storage reduction. * **[Continuous aggregates](/docs/learn/continuous-aggregates)**: pre-compute device statistics and fleet-wide metrics. * **[Hyperfunctions](/docs/build/data-management/hyperfunctions)**: downsample with LTTB for visualization, use time-weighted averages for irregular samples. * **[Data retention](/docs/learn/data-lifecycle/data-retention/about-data-retention)**: drop raw sensor data automatically after a retention period. ### Financial analytics [Section titled “Financial analytics”](#financial-analytics) For financial data with high-frequency trading, market data, and portfolio analytics: * **[Hypertables](/docs/learn/hypertables/understand-hypertables)**: store tick data, OHLCV bars, and trade executions. * **[Hypercore](/docs/learn/columnar-storage/understand-hypercore)**: compress historical market data for cost-effective long-term backtesting. * **[Continuous aggregates](/docs/learn/continuous-aggregates)**: maintain pre-computed OHLCV bars, technical indicators, and portfolio valuations. * **[Hyperfunctions](/docs/build/data-management/hyperfunctions)**: calculate candlestick aggregates, percentiles, and statistical measures. * **[Schema optimization](/docs/learn/data-model/understand-database-schemas)**: use indexes for symbol lookups, constraints for data integrity. ### Observability and monitoring [Section titled “Observability and monitoring”](#observability-and-monitoring) For system metrics, logs, and distributed tracing: * **[Hypertables](/docs/learn/hypertables/understand-hypertables)**: ingest metrics, logs, and traces with automatic partitioning. * **[Hypercore](/docs/learn/columnar-storage/understand-hypercore)**: compress historical metrics and traces for cost-effective long-term retention. * **[Continuous aggregates](/docs/learn/continuous-aggregates)**: maintain service health metrics, error rates, and latency percentiles. * **[Hyperfunctions](/docs/build/data-management/hyperfunctions)**: calculate uptime/downtime via heartbeat aggregation, detect anomalies, and analyze distributions. * **[Data retention](/docs/learn/data-lifecycle/data-retention/about-data-retention)**: drop raw data after the debugging period while keeping aggregated metrics. # Understand chunks > What chunks are, why they matter for real-time analytics, and how to use them effectively In Tiger Data products, **chunks** are the physical partitions behind a hypertable: each one holds rows for a time range (and optional space keys). You always query the hypertable name; the engine routes work to the right chunks. Chunks are what make [hypertables](/docs/learn/hypertables/understand-hypertables) fast and manageable. You rarely interact with chunks directly, but understanding them helps you design schemas, tune performance, and reason about how Tiger Cloud (and self-hosted TimescaleDB) organize your data. For step-by-step tasks (e.g. choosing chunk intervals or listing chunks), see the links in [How to use chunks effectively](#how-to-use-chunks-effectively) and [Learn more](#learn-more). *** ## What are chunks? [Section titled “What are chunks?”](#what-are-chunks) A chunk is a horizontal partition of a [hypertable](/docs/learn/hypertables/understand-hypertables): it is a table that holds rows for a specific time range (and optionally a specific space partition if you use multiple dimensions). When you insert data into a hypertable, TimescaleDB does not store everything in one giant table. Instead, it splits the data into chunks, smaller tables, each responsible for a slice of time (for example one day or one week). That split happens automatically and transparently: you always insert and query against the hypertable name; the system creates and uses chunks behind the scenes. In practice, a hypertable is the logical table you see; chunks are the physical partitions that store the data. You can think of a hypertable as a single table in SQL, and chunks as the internal "drawers" that the database opens only when they’re needed for a given query or write. Note **Inheritance** is not supported for hypertables (and therefore for chunks). Using table inheritance with hypertables can lead to unexpected behavior. *** ## Why chunks matter [Section titled “Why chunks matter”](#why-chunks-matter) Chunks are not just an implementation detail — they are why hypertables can deliver efficient data management, better query performance, and scalable maintenance without you changing how you write SQL. ### Query performance and chunk skipping [Section titled “Query performance and chunk skipping”](#query-performance-and-chunk-skipping) Time-series queries usually filter by time (for example, "last 24 hours" or "September 2024"). Because each chunk covers a known time range, the query planner can skip chunks that don’t contain any data for that range. So instead of scanning the entire table, TimescaleDB only touches the chunks that might contain matching rows. Chunk skipping dramatically reduces work and speeds up queries. On Tiger Cloud, you can also [enable chunk skipping on non-partitioning columns](/docs/build/performance-optimization/improve-hypertable-performance#enable-chunk-skipping) (for example, a "received at" timestamp when the hypertable is partitioned by "event time"), so even more chunks can be skipped when they’re irrelevant to the query. ### Maintenance and lifecycle [Section titled “Maintenance and lifecycle”](#maintenance-and-lifecycle) Many operations in a time-series database are easier when data is partitioned by time: * **Columnstore**: Policies can convert older chunks to [columnstore](/docs/build/columnar-storage/setup-hypercore) while keeping recent chunks in rowstore for fast inserts and updates. Conversion is applied per chunk. * **Data retention**: You can drop old data by dropping entire chunks (e.g. with [`drop_chunks()`](/docs/reference/timescaledb/hypertables/drop_chunks)) instead of running row-by-row `DELETE`, which avoids bloat and is much faster. * **Tiered storage**: On [Tiger Cloud](/docs/get-started/quickstart/create-service), you can move chunks to different storage tiers based on age or access patterns. So chunks matter because they give the system clear boundaries: "this block of time is hot," "this block is cold," "this block can be converted to columnstore or dropped." That makes automation and tuning predictable. ### Parallelism and indexing [Section titled “Parallelism and indexing”](#parallelism-and-indexing) Chunks are independent storage units. The database can process multiple chunks in parallel when executing a query, and indexes are maintained per chunk. That keeps index sizes manageable and allows the planner to use chunk-level statistics. By default, when you create a hypertable, a time-descending index is created; additional indexes you add are also built at the chunk level, which helps both insert and query performance when you have many rows. *** ## How chunks relate to the chunk interval [Section titled “How chunks relate to the chunk interval”](#how-chunks-relate-to-the-chunk-interval) The **chunk interval** is the time span each chunk is responsible for (e.g. one day or seven days). It is set when you create the hypertable (or later via [`set_chunk_time_interval()`](/docs/reference/timescaledb/hypertables/set_chunk_time_interval)) and determines how many chunks you get for a given time range: smaller intervals mean more, smaller chunks; larger intervals mean fewer, larger chunks. * **Default**: Typically each chunk holds about 7 days of data, but you can change this to match how you ingest and query (e.g. one chunk per day for high-volume, time-bound queries). * **Creation**: When you insert a row whose time falls in a range that doesn’t yet have a chunk, TimescaleDB creates a new chunk for that range automatically. So the number of chunks grows as new time ranges receive data. * **Columnstore**: A [columnstore policy](/docs/build/columnar-storage/setup-hypercore) often converts chunks to columnar format after a certain age (e.g. after one chunk interval). So the chunk interval also influences when data moves into the columnstore. See the [Hypertables and chunks](/docs/reference/timescaledb/hypertables) overview for the exact behavior of your version. Choosing the right chunk interval is a trade-off: you want chunks small enough that active (recent) chunks and their indexes fit in memory for fast access, but not so small that you end up with too many chunks (which can slow query planning and reduce columnstore effectiveness). The [Improve hypertable and query performance](/docs/build/performance-optimization/improve-hypertable-performance) guide explains how to choose and tune the chunk interval in practice. *** ## How to use chunks effectively [Section titled “How to use chunks effectively”](#how-to-use-chunks-effectively) You don’t need to create or manage chunks manually for normal use — you create a [hypertable](/docs/learn/hypertables/understand-hypertables) and insert/query it; chunks are created and used automatically. "Using chunks effectively" means designing and tuning so that chunking works in your favor. 1. **Choose an appropriate chunk interval** Set the chunk interval when you [create the hypertable](/docs/learn/hypertables/creating-and-configuring-hypertables) (or change it for new chunks with `set_chunk_time_interval()`). A common guideline is to size the interval so that the indexes of chunks you’re currently ingesting into fit within a reasonable share of main memory (e.g. 25% of RAM). For step-by-step guidance and examples, see [Optimize hypertable chunk intervals](/docs/build/performance-optimization/improve-hypertable-performance#optimize-hypertable-chunk-intervals). 2. **Use time-based filters in queries** Queries that filter on the partitioning (time) column allow the planner to skip irrelevant chunks. If you often filter by another column (e.g. "ingested at"), consider [enabling chunk skipping](/docs/build/performance-optimization/improve-hypertable-performance#enable-chunk-skipping) on that column (where supported) so more chunks can be skipped. 3. **Inspect chunks when tuning or debugging** You can list chunks for a hypertable with [`show_chunks()`](/docs/reference/timescaledb/hypertables/show_chunks) and inspect size with functions like [`chunks_detailed_size()`](/docs/reference/timescaledb/hypertables/chunks_detailed_size). That’s useful when you’re deciding whether to change the chunk interval or investigating query plans. 4. **Prefer chunk-level operations for bulk lifecycle actions** For dropping old data, use retention policies or [`drop_chunks()`](/docs/reference/timescaledb/hypertables/drop_chunks) instead of large `DELETE`s. For columnstore conversion, policies operate per chunk, so your chunk interval and policy settings work together. If you’re just getting started, [Create and configure a hypertable](/docs/learn/hypertables/creating-and-configuring-hypertables) is the right place to begin; the default chunk interval is a reasonable starting point, and you can refine it later using the guides above. *** ## Learn more [Section titled “Learn more”](#learn-more) * [Understand hypertables](/docs/learn/hypertables/understand-hypertables): How hypertables work and why they use chunks. * [Sizing hypertable chunks](/docs/learn/hypertables/sizing-hypertable-chunks): Choosing the right chunk interval. * [Improve hypertable and query performance](/docs/build/performance-optimization/improve-hypertable-performance): Optimize chunk intervals and enable chunk skipping. * [Hypertables and chunks API](/docs/reference/timescaledb/hypertables): `show_chunks`, `drop_chunks`, `move_chunk`, and more. * [Glossary](/docs/learn/glossary): Short definitions for chunk, chunk interval, and chunk skipping. # Compression methods in hypercore > Understand the compression algorithms used when converting data from the rowstore to the columnstore Depending on the data type that is compressed when your data is converted from the rowstore to the columnstore, TimescaleDB uses the following compression algorithms: * **Integers, timestamps, boolean and other integer-like types**: a combination of the following compression methods is used: [delta encoding](#delta-encoding), [delta-of-delta](#delta-of-delta-encoding), [simple-8b](#simple-8b), and [run-length encoding](#run-length-encoding). * **Columns that do not have a high amount of repeated values**: [XOR-based](#xor-based-compression) compression with some [dictionary compression](#dictionary-compression). * **All other types**: [dictionary compression](#dictionary-compression). This page gives an in-depth explanation of the compression methods used in hypercore. ## Integer compression [Section titled “Integer compression”](#integer-compression) For integers, timestamps, and other integer-like types TimescaleDB uses a combination of delta encoding, delta-of-delta, simple 8-b, and run-length encoding. The simple-8b compression method has been extended so that data can be decompressed in reverse order. Backward scanning queries are common in time-series workloads. This means that these types of queries run much faster. ### Delta encoding [Section titled “Delta encoding”](#delta-encoding) Delta encoding reduces the amount of information required to represent a data object by only storing the difference, sometimes referred to as the delta, between that object and one or more reference objects. These algorithms work best where there is a lot of redundant information, and it is often used in workloads like versioned file systems. For example, this is how Dropbox keeps your files synchronized. Applying delta-encoding to time-series data means that you can use fewer bytes to represent a data point, because you only need to store the delta from the previous data point. For example, imagine you had a dataset that collected CPU, free memory, temperature, and humidity over time. If you time column was stored as an integer value, like seconds since UNIX epoch, your raw data would look a little like this: | time | cpu | mem\_free\_bytes | temperature | humidity | | ------------------- | --- | ---------------- | ----------- | -------- | | 2023-04-01 10:00:00 | 82 | 1,073,741,824 | 80 | 25 | | 2023-04-01 10:00:05 | 98 | 858,993,459 | 81 | 25 | | 2023-04-01 10:00:10 | 98 | 858,904,583 | 81 | 25 | With delta encoding, you only need to store how much each value changed from the previous data point, resulting in smaller values to store. So after the first row, you can represent subsequent rows with less information, like this: | time | cpu | mem\_free\_bytes | temperature | humidity | | ------------------- | --- | ---------------- | ----------- | -------- | | 2023-04-01 10:00:00 | 82 | 1,073,741,824 | 80 | 25 | | 5 seconds | 16 | -214,748,365 | 1 | 0 | | 5 seconds | 0 | -88,876 | 0 | 0 | Applying delta encoding to time-series data takes advantage of the fact that most time-series datasets are not random, but instead represent something that is slowly changing over time. The storage savings over millions of rows can be substantial, especially if the value changes very little, or doesn't change at all. ### Delta-of-delta encoding [Section titled “Delta-of-delta encoding”](#delta-of-delta-encoding) Delta-of-delta encoding takes delta encoding one step further and applies delta-encoding over data that has previously been delta-encoded. With time-series datasets where data collection happens at regular intervals, you can apply delta-of-delta encoding to the time column, which results in only needing to store a series of zeroes. In other words, delta encoding stores the first derivative of the dataset, while delta-of-delta encoding stores the second derivative of the dataset. Applied to the example dataset from earlier, delta-of-delta encoding results in this: | time | cpu | mem\_free\_bytes | temperature | humidity | | ------------------- | --- | ---------------- | ----------- | -------- | | 2020-04-01 10:00:00 | 82 | 1,073,741,824 | 80 | 25 | | 5 seconds | 16 | -214,748,365 | 1 | 0 | | 0 seconds | 0 | -88,876 | 0 | 0 | In this example, delta-of-delta further compresses 5 seconds in the time column down to 0 for every entry in the time column after the second row, because the five second gap remains constant for each entry. Note that you see two entries in the table before the delta-delta 0 values, because you need two deltas to compare. This compresses a full timestamp of 8 bytes, or 64 bits, down to just a single bit, resulting in 64x compression. ### Simple-8b [Section titled “Simple-8b”](#simple-8b) With delta and delta-of-delta encoding, you can significantly reduce the number of digits you need to store. But you still need an efficient way to store the smaller integers. The previous examples used a standard integer datatype for the time column, which needs 64 bits to represent the value of 0 when delta-delta encoded. This means that even though you are only storing the integer 0, you are still consuming 64 bits to store it, so you haven't actually saved anything. Simple-8b is one of the simplest and smallest methods of storing variable-length integers. In this method, integers are stored as a series of fixed-size blocks. For each block, every integer within the block is represented by the minimal bit-length needed to represent the largest integer in that block. The first bits of each block denotes the minimum bit-length for the block. This technique has the advantage of only needing to store the length once for a given block, instead of once for each integer. Because the blocks are of a fixed size, you can infer the number of integers in each block from the size of the integers being stored. For example, if you wanted to store a temperature that changed over time, and you applied delta encoding, you might end up needing to store this set of integers: | temperature (deltas) | | -------------------- | | 1 | | 10 | | 11 | | 13 | | 9 | | 100 | | 22 | | 11 | With a block size of 10 digits, you could store this set of integers as two blocks: one block storing 5 2-digit numbers, and a second block storing 3 3-digit numbers, like this: ```text {2: [01, 10, 11, 13, 09]} {3: [100, 022, 011]} ``` In this example, both blocks store about 10 digits worth of data, even though some of the numbers have to be padded with a leading 0. You might also notice that the second block only stores 9 digits, because 10 is not evenly divisible by 3. Simple-8b works in this way, except it uses binary numbers instead of decimal, and it usually uses 64-bit blocks. In general, the longer the integer, the fewer number of integers that can be stored in each block. ### Run-length encoding [Section titled “Run-length encoding”](#run-length-encoding) Simple-8b compresses integers very well, however, if you have a large number of repeats of the same value, you can get even better compression with run-length encoding. This method works well for values that don't change very often, or if an earlier transformation removes the changes. Run-length encoding is one of the classic compression algorithms. For time-series data with billions of contiguous zeroes, or even a document with a million identically repeated strings, run-length encoding works incredibly well. For example, if you wanted to store a temperature that changed minimally over time, and you applied delta encoding, you might end up needing to store this set of integers: | temperature (deltas) | | -------------------- | | 11 | | 12 | | 12 | | 12 | | 12 | | 12 | | 12 | | 1 | | 12 | | 12 | | 12 | | 12 | For values like these, you do not need to store each instance of the value, but rather how long the run, or number of repeats, is. You can store this set of numbers as `{"{run; value}"}` pairs like this: ```text {1; 11}, {6; 12}, {1; 1}, {4; 12} ``` This technique uses 11 digits of storage (1, 1, 1, 6, 1, 2, 1, 1, 4, 1, 2), rather than 23 digits that an optimal series of variable-length integers requires (11, 12, 12, 12, 12, 12, 12, 1, 12, 12, 12, 12). Run-length encoding is also used as a building block for many more advanced algorithms, such as Simple-8b RLE, which is an algorithm that combines run-length and Simple-8b techniques. TimescaleDB implements a variant of Simple-8b RLE. This variant uses different sizes to standard Simple-8b, in order to handle 64-bit values, and RLE. ## Floating point compression [Section titled “Floating point compression”](#floating-point-compression) For columns that do not have a high amount of repeated values, TimescaleDB uses XOR-based compression. The standard XOR-based compression method has been extended so that data can be decompressed in reverse order. Backward scanning queries are common in time-series workloads. This means that queries that use backwards scans run much faster. ### XOR-based compression [Section titled “XOR-based compression”](#xor-based-compression) Floating point numbers are usually more difficult to compress than integers. Fixed-length integers often have leading zeroes, but floating point numbers usually use all of their available bits, especially if they are converted from decimal numbers, which can't be represented precisely in binary. Techniques like delta-encoding don't work well for floats, because they do not reduce the number of bits sufficiently. This means that most floating-point compression algorithms tend to be either complex and slow, or truncate significant digits. One of the few simple and fast lossless floating-point compression algorithms is XOR-based compression, built on top of Facebook's Gorilla compression. XOR is the binary function `exclusive or`. In this algorithm, successive floating point numbers are compared with XOR, and a difference results in a bit being stored. The first data point is stored without compression, and subsequent data points are represented using their XOR'd values. ## Data-agnostic compression [Section titled “Data-agnostic compression”](#data-agnostic-compression) For values that are not integers or floating point, TimescaleDB uses dictionary compression. ### Dictionary compression [Section titled “Dictionary compression”](#dictionary-compression) One of the earliest lossless compression algorithms, dictionary compression is the basis of many popular compression methods. Dictionary compression can also be found in areas outside of computer science, such as medical coding. Instead of storing values directly, dictionary compression works by making a list of the possible values that can appear, and then storing an index into a dictionary containing the unique values. This technique is quite versatile, can be used regardless of data type, and works especially well when you have a limited set of values that repeat frequently. For example, if you had the list of temperatures shown earlier, but you wanted an additional column storing a city location for each measurement, you might have a set of values like this: | City | | ------------- | | New York | | San Francisco | | San Francisco | | Los Angeles | Instead of storing all the city names directly, you can instead store a dictionary, like this: ```text {0: "New York", 1: "San Francisco", 2: "Los Angeles"} ``` You can then store just the indices in your column, like this: | City | | ---- | | 0 | | 1 | | 1 | | 2 | For a dataset with a lot of repetition, this can offer significant compression. In the example, each city name is on average 11 bytes in length, while the indices are never going to be more than 4 bytes long, reducing space usage nearly 3 times. In TimescaleDB, the list of indices is compressed even further with the Simple-8b+RLE method, making the storage cost even smaller. Dictionary compression doesn't always result in savings. If your dataset doesn't have a lot of repeated values, then the dictionary is the same size as the original data. TimescaleDB automatically detects this case, and falls back to not using a dictionary in that scenario. #### JSONB compression [Section titled “JSONB compression”](#jsonb-compression) For JSONB columns, TimescaleDB uses a two-layer compression approach: 1. **TimescaleDB dictionary compression**: TimescaleDB attempts to apply its own dictionary compression to the JSONB data. This works well when JSONB values have high repetition. It does not perform when each JSONB value is unique or nearly unique. 2. **PostgreSQL TOAST compression**: if dictionary compression does not perform, TimescaleDB skips compression and allows PostgreSQL to handle the data using Oversized-Attribute Storage Technique (TOAST) compression. By default, PostgreSQL uses `pglz` compression. To configure PostgreSQL to use more modern compression algorithms like `lz4`, set the `default_toast_compression` configuration parameter. To check your current TOAST compression setting: ```sql SHOW default_toast_compression; ``` ## Learn more [Section titled “Learn more”](#learn-more) * [Set up hypercore](/docs/build/columnar-storage/setup-hypercore): Enable and configure columnstore on your hypertables. * [Improve query and upsert performance](/docs/build/performance-optimization/secondary-indexes): Tune `segmentby` and `orderby` for best compression and query speed. * [Basic compression with hypercore](/docs/build/how-to/basic-compression): Quick-start guide for enabling compression. * [Troubleshoot hypercore](/docs/build/tips-and-tricks/troubleshoot-hypercore): Common issues and solutions. * [Understand hypercore](/docs/learn/columnar-storage/understand-hypercore): How the hybrid row-columnar engine works. * [Hypercore API overview](/docs/reference/timescaledb/hypercore): All columnstore functions and policies. # Understand hypercore > How Tiger Data's hybrid row-columnar engine accelerates analytics while keeping ingest fast Hypercore is Tiger Data's hybrid row-columnar storage engine. New data lands in the rowstore for fast inserts and updates, then automatically converts to the columnstore where analytical queries read fewer bytes and run faster. This conversion includes data compression of up to 98%, which dramatically cuts storage costs. ## How it works [Section titled “How it works”](#how-it-works) Hypercore is a hybrid row-columnar storage engine in TimescaleDB. It is designed specifically for real-time analytics and powered by time-series data. The advantage of hypercore is its ability to seamlessly switch between row-oriented and column-oriented storage, delivering the best of both worlds: ![Hypertable with hypercore enabled: incoming rows are stored in the rowstore and automatically columnized into the columnstore in batches of up to 1000 values per column](/docs/_astro/hypertable-with-hypercore-enabled-light.C2P2rNDf_7VCLT.webp) ![Hypertable with hypercore enabled: incoming rows are stored in the rowstore and automatically columnized into the columnstore in batches of up to 1000 values per column](/docs/_astro/hypertable-with-hypercore-enabled-dark.CSPYbMj5_hrCeU.webp) Hypercore solves the key challenges in real-time analytics: * High ingest throughput * Low-latency ingestion * Fast query performance * Efficient handling of data updates and late-arriving data * Streamlined data management Hypercore's hybrid approach combines the benefits of row-oriented and column-oriented formats: * **Fast ingest with rowstore**: new data is initially written to the rowstore, which is optimized for high-speed inserts and updates. Real-time applications can handle rapid streams of incoming data, including upserts and late-arriving rows. * **Efficient analytics with columnstore**: as the data cools, TimescaleDB automatically converts it to the columnstore. The columnar format enables fast scanning and aggregation. Multiple mechanisms keep queries fast: * [Chunk skipping](/docs/build/performance-optimization/improve-hypertable-performance) skips entire chunks that cannot match the query. * Vectorized execution evaluates aggregate functions directly on columnstore batches. Since 2.27.0, this path also covers queries whose `WHERE` clause uses non-vectorizable functions like `time_bucket()`, including continuous aggregate refreshes — yielding 30%–2× faster execution in many cases. Since [2.28.0](https://github.com/timescale/timescaledb/releases/tag/2.28.0), the vectorized path also evaluates `CASE` expressions in aggregation and grouping. * [Sparse indexes](/docs/build/performance-optimization/indexing) — `bloom` for equality and `minmax` for ranges — let the engine skip individual batches without decompressing them. Bloom indexes accelerate `SELECT`, `UPDATE`, `DELETE`, and `UPSERT` operations on compressed data. * Summary queries like `COUNT`, `MIN`, `MAX`, `FIRST`, and `LAST` read results straight from batch metadata. * **Lower storage costs**: 90–98% compression in the columnstore reduces storage cost dramatically, without sacrificing query performance. * **Fast modification of compressed data in columnstore**: just use SQL. TimescaleDB supports `INSERT`, `UPDATE`, `DELETE`, and `UPSERT` directly on the columnstore, with high-performance paths for each. * **Full mutability with transactional semantics**: regardless of where data is stored, hypercore provides full ACID support. Like in a vanilla PostgreSQL database, inserts and updates to the rowstore and columnstore are always consistent and visible to queries as soon as they complete. For an in-depth explanation of how hypertables and hypercore work, see the [Data model](/docs/learn/deep-dive/whitepaper#data-model). ## Get hands on [Section titled “Get hands on”](#get-hands-on) [Set up hypercore](/docs/build/columnar-storage/setup-hypercore) [Enable and configure columnstore on your hypertables.](/docs/build/columnar-storage/setup-hypercore) [Basic compression with hypercore](/docs/build/how-to/basic-compression) [Quick-start guide for enabling compression.](/docs/build/how-to/basic-compression) [Compression methods](/docs/learn/columnar-storage/compression-methods) [How hypercore compresses your data under the hood.](/docs/learn/columnar-storage/compression-methods) ## Learn more [Section titled “Learn more”](#learn-more) * [Improve query and upsert performance](/docs/build/performance-optimization/secondary-indexes): Segmenting and ordering columnstore data. * [Convert continuous aggregates to the columnstore](/docs/build/continuous-aggregates/compression-on-continuous-aggregates): Compress aggregated data. * [Troubleshoot hypercore](/docs/build/tips-and-tricks/troubleshoot-hypercore): Common issues and solutions. * [Hypercore API overview](/docs/reference/timescaledb/hypercore): All columnstore functions and policies. # Understand continuous aggregates > Learn how TimescaleDB continuous aggregates combine your data into analytic summaries and are refreshed in the background when new data is added In modern applications, data usually grows very quickly. This means that aggregating it into useful summaries can become very slow. If you are collecting data very frequently, you might want to aggregate your data into minutes or hours instead. For example, if an IoT device takes temperature readings every second, you might want to find the average temperature for each hour. Every time you run this query, the database needs to scan the entire table and recalculate the average. TimescaleDB makes aggregating data lightning fast, accurate, and easy with continuous aggregates. ![How a continuous aggregate incrementally materializes time-bucketed rollups from a raw hypertable, with a real-time aggregate query reading both pre-computed and recent raw data](/docs/_astro/continuous-aggregates-light.BhW3GFu2_pFsfL.webp) ![How a continuous aggregate incrementally materializes time-bucketed rollups from a raw hypertable, with a real-time aggregate query reading both pre-computed and recent raw data](/docs/_astro/continuous-aggregates-dark.CnXJ1DFs_2kSGiL.webp) Continuous aggregates in TimescaleDB are a kind of hypertable that is refreshed automatically in the background as new data is added, or old data is modified. Changes to your dataset are tracked, and the hypertable behind the continuous aggregate is automatically updated in the background. Continuous aggregates have a much lower maintenance burden than regular PostgreSQL materialized views, because the whole view is not created from scratch on each refresh. This means that you can get on with working your data instead of maintaining your database. Because continuous aggregates are based on hypertables, you can query them in exactly the same way as your other tables. This includes continuous aggregates in the rowstore, compressed into the [columnstore](/docs/learn/columnar-storage/understand-hypercore), or [tiered to object storage](/docs/build/data-management/storage/manage-storage). You can even create [continuous aggregates on top of your continuous aggregates](/docs/learn/continuous-aggregates/hierarchical-continuous-aggregates), for an even more fine-tuned aggregation. [Real-time aggregation](/docs/learn/continuous-aggregates/real-time-aggregates) enables you to combine pre-aggregated data from the materialized view with the most recent raw data. This gives you up-to-date results on every query. In TimescaleDB v2.13 and later, real-time aggregates are **DISABLED** by default. In earlier versions, real-time aggregates are **ENABLED** by default; when you create a continuous aggregate, queries to that view include the results from the most recent raw data. ## Types of aggregation [Section titled “Types of aggregation”](#types-of-aggregation) There are three main ways to make aggregation easier: materialized views, continuous aggregates, and real-time aggregates. [Materialized views](https://www.postgresql.org/docs/current/rules-materializedviews.html) are a standard PostgreSQL function. They are used to cache the result of a complex query so that you can reuse it later on. Materialized views do not update regularly, although you can manually refresh them as required. [Continuous aggregates](/docs/learn/continuous-aggregates) are a TimescaleDB-only feature. They work in a similar way to a materialized view, but they are updated automatically in the background, as new data is added to your database. Continuous aggregates are updated continuously and incrementally, which means they are less resource intensive to maintain than materialized views. Continuous aggregates are based on hypertables, and you can query them in the same way as you do your other tables. [Real-time aggregates](/docs/learn/continuous-aggregates/real-time-aggregates) are a TimescaleDB-only feature. They are the same as continuous aggregates, but they add the most recent raw data to the previously aggregated data to provide accurate and up-to-date results, without needing to aggregate data as it is being written. ## Continuous aggregates on continuous aggregates [Section titled “Continuous aggregates on continuous aggregates”](#continuous-aggregates-on-continuous-aggregates) You can create a continuous aggregate on top of another continuous aggregate. This allows you to summarize data at different granularity. For example, you might have a raw hypertable that contains second-by-second data. Create a continuous aggregate on the hypertable to calculate hourly data. To calculate daily data, create a continuous aggregate on top of your hourly continuous aggregate. For more information, see the documentation about [continuous aggregates on continuous aggregates](/docs/learn/continuous-aggregates/hierarchical-continuous-aggregates). ## Continuous aggregates with a `JOIN` clause [Section titled “Continuous aggregates with a JOIN clause”](#continuous-aggregates-with-a-join-clause) Continuous aggregates support the following JOIN features: | Feature | TimescaleDB < 2.10.x | TimescaleDB <= 2.15.x | TimescaleDB >= 2.16.x | | ------------------------------------------------------------------------------------------- | -------------------- | --------------------- | --------------------- | | INNER JOIN | ❌ | ✅ | ✅ | | LEFT JOIN | ❌ | ❌ | ✅ | | LATERAL JOIN | ❌ | ❌ | ✅ | | Joins between **ONE** hypertable and **ONE** standard PostgreSQL table | ❌ | ✅ | ✅ | | Joins between **ONE** hypertable and **MANY** standard PostgreSQL tables | ❌ | ❌ | ✅ | | Join conditions must be equality conditions, and there can only be **ONE** `JOIN` condition | ❌ | ✅ | ✅ | | Any join conditions | ❌ | ❌ | ✅ | JOINS in TimescaleDB must meet the following conditions: * Only the changes to the hypertable are tracked, and they are updated in the continuous aggregate when it is refreshed. Changes to standard PostgreSQL table are not tracked. * You can use an `INNER`, `LEFT`, and `LATERAL` joins; no other join type is supported. * Joins on the materialized hypertable of a continuous aggregate are not supported. * Hierarchical continuous aggregates can be created on top of a continuous aggregate with a `JOIN` clause, but cannot themselves have a `JOIN` clause. ### JOIN examples [Section titled “JOIN examples”](#join-examples) Given the following schema: ```sql CREATE TABLE locations ( id TEXT PRIMARY KEY, name TEXT ); CREATE TABLE devices ( id SERIAL PRIMARY KEY, location_id TEXT, name TEXT ); CREATE TABLE conditions ( "time" TIMESTAMPTZ, device_id INTEGER, temperature FLOAT8 ) WITH ( tsdb.hypertable ); ``` When you create a hypertable using [CREATE TABLE ... WITH ...](/docs/reference/timescaledb/hypertables/create_table), the default partitioning column is automatically the first column with a timestamp data type. Also, TimescaleDB creates a [columnstore policy](/docs/reference/timescaledb/hypercore/add_columnstore_policy) that automatically converts your data to the columnstore, after an interval equal to the value of the [chunk\_interval](/docs/reference/timescaledb/hypertables/set_chunk_time_interval), defined through `after` in the policy. This columnar format enables fast scanning and aggregation, optimizing performance for analytical workloads while also saving significant storage space. In the columnstore conversion, hypertable chunks are compressed by up to 98%, and organized for efficient, large-scale queries. You can customize this policy later using [alter\_job](/docs/reference/timescaledb/jobs-automation/alter_job). However, to change `after` or `created_before`, the compression settings, or the hypertable the policy is acting on, you must [remove the columnstore policy](/docs/reference/timescaledb/hypercore/remove_columnstore_policy) and [add a new one](/docs/reference/timescaledb/hypercore/add_columnstore_policy). You can also manually [convert chunks](/docs/reference/timescaledb/hypercore/convert_to_columnstore) in a hypertable to the columnstore. See the following `JOIN` examples on continuous aggregates: * `INNER JOIN` on a single equality condition, using the `ON` clause: ```sql CREATE MATERIALIZED VIEW conditions_by_day WITH (timescaledb.continuous) AS SELECT time_bucket('1 day', time) AS bucket, devices.name, MIN(temperature), MAX(temperature) FROM conditions JOIN devices ON devices.id = conditions.device_id GROUP BY bucket, devices.name WITH NO DATA; ``` * `INNER JOIN` on a single equality condition, using the `ON` clause, with a further condition added in the `WHERE` clause: ```sql CREATE MATERIALIZED VIEW conditions_by_day WITH (timescaledb.continuous) AS SELECT time_bucket('1 day', time) AS bucket, devices.name, MIN(temperature), MAX(temperature) FROM conditions JOIN devices ON devices.id = conditions.device_id WHERE devices.location_id = 'location123' GROUP BY bucket, devices.name WITH NO DATA; ``` * `INNER JOIN` on a single equality condition specified in `WHERE` clause: ```sql CREATE MATERIALIZED VIEW conditions_by_day WITH (timescaledb.continuous) AS SELECT time_bucket('1 day', time) AS bucket, devices.name, MIN(temperature), MAX(temperature) FROM conditions, devices WHERE devices.id = conditions.device_id GROUP BY bucket, devices.name WITH NO DATA; ``` * `INNER JOIN` on multiple equality conditions: ```sql CREATE MATERIALIZED VIEW conditions_by_day WITH (timescaledb.continuous) AS SELECT time_bucket('1 day', time) AS bucket, devices.name, MIN(temperature), MAX(temperature) FROM conditions JOIN devices ON devices.id = conditions.device_id AND devices.location_id = 'location123' GROUP BY bucket, devices.name WITH NO DATA; ``` TimescaleDB v2.16.x and higher. * `INNER JOIN` with a single equality condition specified in `WHERE` clause can be combined with further conditions in the `WHERE` clause: ```sql CREATE MATERIALIZED VIEW conditions_by_day WITH (timescaledb.continuous) AS SELECT time_bucket('1 day', time) AS bucket, devices.name, MIN(temperature), MAX(temperature) FROM conditions, devices WHERE devices.id = conditions.device_id AND devices.location_id = 'location123' GROUP BY bucket, devices.name WITH NO DATA; ``` TimescaleDB v2.16.x and higher. * `INNER JOIN` between a hypertable and multiple PostgreSQL tables: ```sql CREATE MATERIALIZED VIEW conditions_by_day WITH (timescaledb.continuous) AS SELECT time_bucket('1 day', time) AS bucket, devices.name AS device, locations.name AS location, MIN(temperature), MAX(temperature) FROM conditions JOIN devices ON devices.id = conditions.device_id JOIN locations ON locations.id = devices.location_id GROUP BY bucket, devices.name, locations.name WITH NO DATA; ``` TimescaleDB v2.16.x and higher. * `LEFT JOIN` between a hypertable and a PostgreSQL table: ```sql CREATE MATERIALIZED VIEW conditions_by_day WITH (timescaledb.continuous) AS SELECT time_bucket('1 day', time) AS bucket, devices.name, MIN(temperature), MAX(temperature) FROM conditions LEFT JOIN devices ON devices.id = conditions.device_id GROUP BY bucket, devices.name WITH NO DATA; ``` TimescaleDB v2.16.x and higher. * `LATERAL JOIN` between a hypertable and a subquery: ```sql CREATE MATERIALIZED VIEW conditions_by_day WITH (timescaledb.continuous) AS SELECT time_bucket('1 day', time) AS bucket, devices.name, MIN(temperature), MAX(temperature) FROM conditions, LATERAL (SELECT * FROM devices WHERE devices.id = conditions.device_id) AS devices GROUP BY bucket, devices.name WITH NO DATA; ``` TimescaleDB v2.16.x and higher. ## Function support [Section titled “Function support”](#function-support) In TimescaleDB v2.7 and later, continuous aggregates support all PostgreSQL aggregate functions. This includes both parallelizable aggregates, such as `SUM` and `AVG`, and non-parallelizable aggregates, such as `RANK`. In TimescaleDB v2.10.0 and later, the `FROM` clause supports `JOINS`, with some restrictions. For more information, see the [`JOIN` support section](#continuous-aggregates-with-a-join-clause). In older versions of TimescaleDB, continuous aggregates only support [aggregate functions that can be parallelized by PostgreSQL](https://www.postgresql.org/docs/current/parallel-plans.html#PARALLEL-AGGREGATION). You can work around this by aggregating the other parts of your query in the continuous aggregate, then [using the window function to query the aggregate](/docs/build/continuous-aggregates/create-a-continuous-aggregate#create-a-continuous-aggregate). The following table summarizes the aggregate functions supported in continuous aggregates: | Function, clause, or feature | TimescaleDB 2.6 and earlier | TimescaleDB 2.7, 2.8, and 2.9 | TimescaleDB 2.10 and later | | --------------------------------------------------------------------------------------------------------------------- | --------------------------- | ----------------------------- | -------------------------- | | Parallelizable aggregate functions | ✅ | ✅ | ✅ | | [Non-parallelizable SQL aggregates](https://www.postgresql.org/docs/current/parallel-plans.html#PARALLEL-AGGREGATION) | ❌ | ✅ | ✅ | | `ORDER BY` | ❌ | ✅ | ✅ | | Ordered-set aggregates | ❌ | ✅ | ✅ | | Hypothetical-set aggregates | ❌ | ✅ | ✅ | | `DISTINCT` in aggregate functions | ❌ | ✅ | ✅ | | `FILTER` in aggregate functions | ❌ | ✅ | ✅ | | `FROM` clause supports `JOINS` | ❌ | ❌ | ✅ | DISTINCT works in aggregate functions, not in the query definition. For example, for the table: ```sql CREATE TABLE public.candle( symbol_id uuid NOT NULL, symbol text NOT NULL, "time" timestamp with time zone NOT NULL, open double precision NOT NULL, high double precision NOT NULL, low double precision NOT NULL, close double precision NOT NULL, volume double precision NOT NULL ); ``` * The following works: ```sql CREATE MATERIALIZED VIEW candles_start_end WITH (timescaledb.continuous) AS SELECT time_bucket('1 hour', "time"), COUNT(DISTINCT symbol), first(time, time) as first_candle, last(time, time) as last_candle FROM candle GROUP BY 1; ``` * This does not: ```sql CREATE MATERIALIZED VIEW candles_start_end WITH (timescaledb.continuous) AS SELECT DISTINCT ON (symbol) symbol,symbol_id, first(time, time) as first_candle, last(time, time) as last_candle FROM candle GROUP BY symbol_id; ``` If you want the old behavior in later versions of TimescaleDB, set the `timescaledb.finalized` parameter to `false` when you create your continuous aggregate. ## Components of a continuous aggregate [Section titled “Components of a continuous aggregate”](#components-of-a-continuous-aggregate) Continuous aggregates consist of: * Materialization hypertable to store the aggregated data in * Materialization engine to aggregate data from the raw, underlying, table to the materialization hypertable * Invalidation engine to determine when data needs to be re-materialized, due to changes in the data * Query engine to access the aggregated data ### Materialization hypertable [Section titled “Materialization hypertable”](#materialization-hypertable) Continuous aggregates take raw data from the original hypertable, aggregate it, and store the aggregated data in a materialization hypertable. When you query the continuous aggregate view, the aggregated data is returned to you as needed. Using the same temperature example, the materialization table looks like this: | day | location | chunk | avg temperature | | ---------- | --------- | ----- | --------------- | | 2021/01/01 | New York | 1 | 73 | | 2021/01/01 | Stockholm | 1 | 70 | | 2021/01/02 | New York | 2 | | | 2021/01/02 | Stockholm | 2 | 69 | The materialization table is stored as a TimescaleDB hypertable, to take advantage of the scaling and query optimizations that hypertables offer. Materialization tables contain a column for each group-by clause in the query, and an `aggregate` column for each aggregate in the query. For more information, see [materialized hypertables](/docs/learn/continuous-aggregates/materialized-hypertables). ### Materialization engine [Section titled “Materialization engine”](#materialization-engine) The materialization engine performs two transactions. The first transaction blocks all INSERTs, UPDATEs, and DELETEs, determines the time range to materialize, and updates the invalidation threshold. The second transaction unblocks other transactions, and materializes the aggregates. The first transaction is very quick, and most of the work happens during the second transaction, to ensure that the work does not interfere with other operations. ### Invalidation engine [Section titled “Invalidation engine”](#invalidation-engine) Any change to the data in a hypertable could potentially invalidate some materialized rows. The invalidation engine checks to ensure that the system does not become swamped with invalidations. Fortunately, time-series data means that nearly all INSERTs and UPDATEs have a recent timestamp, so the invalidation engine does not materialize all the data, but to a set point in time called the materialization threshold. This threshold is set so that the vast majority of INSERTs contain more recent timestamps. These data points have never been materialized by the continuous aggregate, so there is no additional work needed to notify the continuous aggregate that they have been added. When the materializer next runs, it is responsible for determining how much new data can be materialized without invalidating the continuous aggregate. It then materializes the more recent data and moves the materialization threshold forward in time. This ensures that the threshold lags behind the point-in-time where data changes are common, and that most INSERTs do not require any extra writes. When data older than the invalidation threshold is changed, each transaction logs the minimum and maximum timestamps of the rows it modified. The continuous aggregate then identifies which complete time buckets are affected based on this per-transaction tracking. The range of buckets that are recalculated depends on transaction boundaries: * If you modify rows in the 10:00 bucket and rows in the 15:00 bucket within a **single transaction**, all buckets from 10:00 to 15:00 (including intermediate buckets 11:00, 12:00, 13:00, and 14:00) are recalculated during refresh. * If you modify rows in the 10:00 bucket in one transaction and rows in the 15:00 bucket in a **separate transaction**, only the 10:00 and 15:00 buckets are recalculated. The intermediate buckets (11:00, 12:00, 13:00, 14:00) are not affected. This logging does cause some write load. However, the threshold lags behind the area of data that is currently changing, so the writes are small and rare. ## Get hands on [Section titled “Get hands on”](#get-hands-on) [Create a continuous aggregate](/docs/build/continuous-aggregates/create-a-continuous-aggregate) [Step-by-step guide to creating and querying a continuous aggregate.](/docs/build/continuous-aggregates/create-a-continuous-aggregate) [Refresh continuous aggregates](/docs/build/continuous-aggregates/refresh-policies) [Set up automatic refresh policies to keep aggregates current.](/docs/build/continuous-aggregates/refresh-policies) [Convert to the columnstore](/docs/build/continuous-aggregates/compression-on-continuous-aggregates) [Compress older aggregated data for storage savings.](/docs/build/continuous-aggregates/compression-on-continuous-aggregates) ## Learn more [Section titled “Learn more”](#learn-more) * [`CREATE MATERIALIZED VIEW` reference](/docs/reference/timescaledb/continuous-aggregates/create_materialized_view): Full API reference. * [Continuous aggregates API overview](/docs/reference/timescaledb/continuous-aggregates): All continuous aggregate functions and policies. # Hierarchical continuous aggregates > Create continuous aggregates on top of continuous aggregates to summarize data at different levels of granularity Since [2.9.0](https://github.com/timescale/timescaledb/releases/tag/2.9.0) The more data you have, the more likely you are to run a more sophisticated analysis on it. When a simple one-level aggregation is not enough, TimescaleDB lets you create continuous aggregates on top of other continuous aggregates. This way, you summarize data at different levels of granularity, while still saving resources with precomputing. For example, you might have an hourly continuous aggregate that summarizes minute-by-minute data. To get a daily summary, you can create a new continuous aggregate on top of your hourly aggregate. This is more efficient than creating the daily aggregate on top of the original hypertable, because you can reuse the calculations from the hourly aggregate. ![Hierarchical continuous aggregates: raw table rows are rolled up into hourly buckets, which are then rolled up into daily summaries](/docs/_astro/hierarchical-continuous-aggregates-light.Dux1JWZs_Z1xC6P5.webp) ![Hierarchical continuous aggregates: raw table rows are rolled up into hourly buckets, which are then rolled up into daily summaries](/docs/_astro/hierarchical-continuous-aggregates-dark.3Opn8wcz_eWY0g.webp) ## Create a continuous aggregate on top of another continuous aggregate [Section titled “Create a continuous aggregate on top of another continuous aggregate”](#create-a-continuous-aggregate-on-top-of-another-continuous-aggregate) Creating a continuous aggregate on top of another continuous aggregate works the same way as creating it on top of a hypertable. In your query, select from a continuous aggregate rather than from the hypertable, and use the time-bucketed column from the existing continuous aggregate as your time column. For more information, see the instructions for [creating a continuous aggregate](/docs/build/continuous-aggregates/create-a-continuous-aggregate). ## Use real-time aggregate with hierarchical continuous aggregates [Section titled “Use real-time aggregate with hierarchical continuous aggregates”](#use-real-time-aggregate-with-hierarchical-continuous-aggregates) In TimescaleDB v2.13 and later, real-time aggregates are **DISABLED** by default. In earlier versions, real-time aggregates are **ENABLED** by default; when you create a continuous aggregate, queries to that view include the results from the most recent raw data. Real-time aggregates always return up-to-date data in response to queries. They accomplish this by joining the materialized data in the continuous aggregate with unmaterialized raw data from the source table or view. When continuous aggregates are stacked, each continuous aggregate is only aware of the layer immediately below. The joining of unmaterialized data happens recursively until it reaches the bottom layer, giving you access to recent data down to that layer. If you keep all continuous aggregates in the stack as real-time aggregates, the bottom layer is the source hypertable. That means every continuous aggregate in the stack has access to all recent data. If there is a non-real-time continuous aggregate somewhere in the stack, the recursive joining stops at that non-real-time continuous aggregate. Higher-level continuous aggregates don't receive any unmaterialized data from lower levels. For example, say you have the following continuous aggregates: * A real-time hourly continuous aggregate on the source hypertable * A real-time daily continuous aggregate on the hourly continuous aggregate * A non-real-time, or materialized-only, monthly continuous aggregate on the daily continuous aggregate * A real-time yearly continuous aggregate on the monthly continuous aggregate Queries on the hourly and daily continuous aggregates include real-time, non-materialized data from the source hypertable. Queries on the monthly continuous aggregate only return already-materialized data. Queries on the yearly continuous aggregate return materialized data from the yearly continuous aggregate itself, plus more recent data from the monthly continuous aggregate. However, the data is limited to what is already materialized in the monthly continuous aggregate, and doesn't get even more recent data from the source hypertable. This happens because the materialized-only continuous aggregate provides a stopping point, and the yearly continuous aggregate is unaware of any layers beyond that stopping point. This is similar to [how stacked views work in PostgreSQL](https://www.postgresql.org/docs/current/rules-views.html). To make queries on the yearly continuous aggregate access all recent data, you can either: * Make the monthly continuous aggregate real-time, or * Redefine the yearly continuous aggregate on top of the daily continuous aggregate. ## Roll up calculations [Section titled “Roll up calculations”](#roll-up-calculations) When summarizing already-summarized data, be aware of how stacked calculations work. Not all calculations return the correct result if you stack them. For example, if you take the maximum of several subsets, then take the maximum of the maximums, you get the maximum of the entire set. But if you take the average of several subsets, then take the average of the averages, that can result in a different figure than the average of all the data. To simplify such calculations when using continuous aggregates on top of continuous aggregates, you can use the [hyperfunctions](/docs/build/data-management/hyperfunctions) from TimescaleDB Toolkit, such as the [statistical aggregates](/docs/reference/toolkit/statistical-and-regression-analysis/stats_agg-one-variable/stats_agg). These hyperfunctions are designed with a two-step aggregation pattern that allows you to roll them up into larger buckets. The first step creates a summary aggregate that can be rolled up, just as a maximum can be rolled up. You can store this aggregate in your continuous aggregate. Then, you can call an accessor function as a second step when you query from your continuous aggregate. This accessor takes the stored data from the summary aggregate and returns the final result. For example, you can create an hourly continuous aggregate using `percentile_agg` over a hypertable, like this: ```sql CREATE MATERIALIZED VIEW response_times_hourly WITH (timescaledb.continuous) AS SELECT time_bucket('1 h'::interval, ts) as bucket, api_id, avg(response_time_ms), percentile_agg(response_time_ms) as percentile_hourly FROM response_times GROUP BY 1, 2; ``` To then stack another daily continuous aggregate over it, you can use a `rollup` function, like this: ```sql CREATE MATERIALIZED VIEW response_times_daily WITH (timescaledb.continuous) AS SELECT time_bucket('1 d'::interval, bucket) as bucket_daily, api_id, mean(rollup(percentile_hourly)) as mean, rollup(percentile_hourly) as percentile_daily FROM response_times_hourly GROUP BY 1, 2; ``` The `mean` function of the TimescaleDB Toolkit is used to calculate the concrete mean value of the rolled up values. The additional `percentile_daily` attribute contains the raw rolled up values, which can be used in an additional continuous aggregate on top of this continuous aggregate (for example a continuous aggregate for the daily values). For more information and examples about using `rollup` functions to stack calculations, see the [`percentile_agg` API documentation](/docs/reference/toolkit/percentile-approximation/uddsketch/percentile_agg). ## Restrictions [Section titled “Restrictions”](#restrictions) There are some restrictions when creating a continuous aggregate on top of another continuous aggregate. In most cases, these restrictions are in place to ensure valid time-bucketing: * You can only create a continuous aggregate on top of a finalized continuous aggregate. This new finalized format is the default for all continuous aggregates created since TimescaleDB 2.7. If you need to create a continuous aggregate on top of a continuous aggregate in the old format, you need to [migrate your continuous aggregate](/docs/build/continuous-aggregates/migrate-to-new-form) to the new format first. * The time bucket of a continuous aggregate should be greater than or equal to the time bucket of the underlying continuous aggregate. It also needs to be a multiple of the underlying time bucket. For example, you can rebucket an hourly continuous aggregate into a new continuous aggregate with time buckets of 6 hours. You can't rebucket the hourly continuous aggregate into a new continuous aggregate with time buckets of 90 minutes, because 90 minutes is not a multiple of 1 hour. * A continuous aggregate with a fixed-width time bucket can't be created on top of a continuous aggregate with a variable-width time bucket. Fixed-width time buckets are time buckets defined in seconds, minutes, hours, and days, because those time intervals are always the same length. Variable-width time buckets are time buckets defined in months or years, because those time intervals vary by the month or on leap years. This limitation prevents a case such as trying to rebucket monthly buckets into `61 day` buckets, where there is no good mapping between time buckets for month combinations such as July/August (62 days). Note that even though weeks are fixed-width intervals, you can't use monthly or yearly time buckets on top of weekly time buckets for the same reason. The number of weeks in a month or year is usually not an integer. However, you can stack a variable-width time bucket on top of a fixed-width time bucket. For example, creating a monthly continuous aggregate on top of a daily continuous aggregate works, and is one of the main use cases for this feature. ## Learn more [Section titled “Learn more”](#learn-more) * [Understand continuous aggregates](/docs/learn/continuous-aggregates): How continuous aggregates work, JOINs, and function support. * [Create a continuous aggregate](/docs/build/continuous-aggregates/create-a-continuous-aggregate): Step-by-step guide including hierarchical examples. * [Refresh continuous aggregates](/docs/build/continuous-aggregates/refresh-policies): Set up automatic refresh policies for each level. * [Materialized hypertables](/docs/learn/continuous-aggregates/materialized-hypertables): How aggregated data is stored. * [`CREATE MATERIALIZED VIEW` reference](/docs/reference/timescaledb/continuous-aggregates/create_materialized_view): Full API reference. # Materialized hypertables > Continuous aggregates store aggregated data in materialized hypertables. Learn how to manage them Continuous aggregates take raw data from the original hypertable, aggregate it, and store the aggregated data in a materialization hypertable. You can modify this materialized hypertable in the same way as any other hypertable. ## Discover the name of a materialized hypertable [Section titled “Discover the name of a materialized hypertable”](#discover-the-name-of-a-materialized-hypertable) To change a materialized hypertable, you need to use its fully qualified name. To find the correct name, use the [`timescaledb_information.continuous_aggregates` view](/docs/reference/timescaledb/informational-views/continuous_aggregates). You can then use the name to modify it in the same way as any other hypertable. 1. **Query the `continuous_aggregates` view** ```sql SELECT view_name, format('%I.%I', materialization_hypertable_schema, materialization_hypertable_name) AS materialization_hypertable FROM timescaledb_information.continuous_aggregates; ``` 2. **Locate the hypertable name** The results look like this: ```sql view_name | materialization_hypertable ---------------------------+--------------------------------------------------- conditions_summary_hourly | _timescaledb_internal._materialized_hypertable_30 conditions_summary_daily | _timescaledb_internal._materialized_hypertable_31 (2 rows) ``` ## Learn more [Section titled “Learn more”](#learn-more) * [Understand continuous aggregates](/docs/learn/continuous-aggregates): How continuous aggregates work, JOINs, and function support. * [Hierarchical continuous aggregates](/docs/learn/continuous-aggregates/hierarchical-continuous-aggregates): Create continuous aggregates on top of other continuous aggregates. * [Convert continuous aggregates to the columnstore](/docs/build/continuous-aggregates/compression-on-continuous-aggregates): Compress older aggregated data. * [Drop data from continuous aggregates](/docs/build/continuous-aggregates/drop-data): Remove materialized data. * [`ALTER MATERIALIZED VIEW` reference](/docs/reference/timescaledb/continuous-aggregates/alter_materialized_view): Modify continuous aggregate settings. # Real-time aggregates > Real-time aggregates combine pre-aggregated data with the most recent raw data for up-to-date results High-ingest workloads need aggregates that blend historical rollups with the live tail without blocking writers. By default, continuous aggregates do not include the most recent data chunk from the underlying hypertable. Real-time aggregates, however, use the aggregated data **and** add the most recent raw data to it. This provides accurate and up-to-date results, without needing to aggregate data as it is being written. In TimescaleDB v2.13 and later, real-time aggregates are **DISABLED** by default. In earlier versions, real-time aggregates are **ENABLED** by default; when you create a continuous aggregate, queries to that view include the results from the most recent raw data. For more detail on the comparison between continuous aggregates and real-time aggregates, see the [real-time aggregate blog post](https://tigerdata.com/blog/achieving-the-best-of-both-worlds-ensuring-up-to-date-results-with-real-time-aggregation/). ## Use real-time aggregates [Section titled “Use real-time aggregates”](#use-real-time-aggregates) You can enable and disable real-time aggregation by setting the `materialized_only` parameter when you create or alter the view. 1. **Enable real-time aggregation** ```sql ALTER MATERIALIZED VIEW table_name set (timescaledb.materialized_only = false); ``` 2. **Disable real-time aggregation** ```sql ALTER MATERIALIZED VIEW table_name set (timescaledb.materialized_only = true); ``` ## Real-time aggregates and refreshing historical data [Section titled “Real-time aggregates and refreshing historical data”](#real-time-aggregates-and-refreshing-historical-data) Real-time aggregates automatically add the most recent data when you query your continuous aggregate. In other words, they include data *more recent than* your last materialized bucket. If you add new *historical* data to an already-materialized bucket, it won't be reflected in a real-time aggregate. You should wait for the next scheduled refresh, or manually refresh by calling `refresh_continuous_aggregate`. You can think of real-time aggregates as being eventually consistent for historical data. ## Learn more [Section titled “Learn more”](#learn-more) * [Understand continuous aggregates](/docs/learn/continuous-aggregates): How continuous aggregates work, JOINs, and function support. * [Create a continuous aggregate](/docs/build/continuous-aggregates/create-a-continuous-aggregate): Step-by-step guide to creating a continuous aggregate with real-time aggregation. * [Refresh continuous aggregates](/docs/build/continuous-aggregates/refresh-policies): Control how often materialized data is updated. * [Troubleshoot continuous aggregates](/docs/build/tips-and-tricks/troubleshoot-continuous-aggregates): Common issues and solutions. * [`ALTER MATERIALIZED VIEW` reference](/docs/reference/timescaledb/continuous-aggregates/alter_materialized_view): Toggle `materialized_only` and other settings. For more information, see the [troubleshooting section](/docs/build/tips-and-tricks/troubleshoot-continuous-aggregates). # Time and continuous aggregates > Learn to work with timezones and continuous aggregates Functions that depend on a local timezone setting inside a continuous aggregate are not supported. You cannot adjust to a local time because the timezone setting changes from user to user. To manage this, you can use explicit timezones in the view definition. Alternatively, you can create your own custom aggregation scheme for tables that use an integer time column. ## Declare an explicit timezone [Section titled “Declare an explicit timezone”](#declare-an-explicit-timezone) The most common method of working with timezones is to declare an explicit timezone in the view query. 1. **Create the view with an explicit timezone** ```sql CREATE MATERIALIZED VIEW device_summary WITH (timescaledb.continuous) AS SELECT time_bucket('1 hour', observation_time) AS bucket, min(observation_time AT TIME ZONE 'EST') AS min_time, device_id, avg(metric) AS metric_avg, max(metric) - min(metric) AS metric_spread FROM device_readings GROUP BY bucket, device_id; ``` 2. **Cast to a timestamp after the view** Alternatively, you can cast to a timestamp using `SELECT`: ```sql SELECT min_time::timestamp FROM device_summary; ``` ## Integer-based time [Section titled “Integer-based time”](#integer-based-time) Date and time is usually expressed as year-month-day and hours:minutes:seconds. Most TimescaleDB databases use a [PostgreSQL date/time-type](https://www.postgresql.org/docs/current/datatype-datetime.html) column to express the date and time. However, in some cases, you might need to convert these common time and date formats to a format that uses an integer. The most common integer time is Unix epoch time, which is the number of seconds since the Unix epoch of 1970-01-01, but other types of integer-based time formats are possible. These examples use a hypertable called `devices` that contains CPU and disk usage information. The devices measure time using the Unix epoch. To create a hypertable that uses an integer-based column as time, you need to provide the chunk time interval. In this case, each chunk is 10 minutes. Define the integer-based time column and chunk time interval: ```sql CREATE TABLE devices( time BIGINT, -- Time in minutes since epoch cpu_usage INTEGER, -- Total CPU usage disk_usage INTEGER, -- Total disk usage PRIMARY KEY (time) ) WITH ( tsdb.hypertable, tsdb.partition_column='time', tsdb.chunk_interval='10' ); ``` For TimescaleDB [v2.23.0](https://github.com/timescale/timescaledb/releases/tag/2.23.0) and higher, the table is automatically partitioned on the first column in the table with a timestamp data type. If multiple columns are suitable candidates as a partitioning column, TimescaleDB throws an error and asks for an explicit definition. For earlier versions, set `partition_column` to a time column. If you are self-hosting TimescaleDB [v2.20.0](https://github.com/timescale/timescaledb/releases/tag/2.23.0) to [v2.22.1](https://github.com/timescale/timescaledb/releases/tag/2.23.0), to convert your data to the columnstore after a specific time interval, you have to call [add\_columnstore\_policy](/docs/reference/timescaledb/hypercore/add_columnstore_policy) after you call [CREATE TABLE](/docs/reference/timescaledb/hypertables/create_table) If you are self-hosting TimescaleDB [v2.19.3](https://github.com/timescale/timescaledb/releases/tag/2.19.3) and below, create a [PostgreSQL relational table](https://www.postgresql.org/docs/current/sql-createtable.html), then convert it using [create\_hypertable](/docs/reference/timescaledb/hypertables/create_hypertable). You then enable hypercore with a call to [ALTER TABLE](/docs/reference/timescaledb/hypercore/alter_table). To define a continuous aggregate on a hypertable that uses integer-based time, you need to have a function to get the current time in the correct format, and set it for the hypertable. You can do this with the [`set_integer_now_func`](/docs/reference/timescaledb/hypertables/set_integer_now_func) function. It can be defined as a regular PostgreSQL function, but needs to be [`STABLE`](https://www.postgresql.org/docs/current/sql-createfunction.html), take no arguments, and return an integer value of the same type as the time column in the table. When you have set up the time-handling, you can create the continuous aggregate. 1. **Set up the time conversion function** ```sql CREATE FUNCTION current_epoch() RETURNS BIGINT LANGUAGE SQL STABLE AS $$ SELECT EXTRACT(EPOCH FROM CURRENT_TIMESTAMP)::bigint;$$; SELECT set_integer_now_func('devices', 'current_epoch'); ``` 2. **Create the continuous aggregate for the devices table** ```sql CREATE MATERIALIZED VIEW devices_summary WITH (timescaledb.continuous) AS SELECT time_bucket('500', time) AS bucket, avg(cpu_usage) AS avg_cpu, avg(disk_usage) AS avg_disk FROM devices GROUP BY bucket; ``` 3. **Insert some rows** ```sql CREATE EXTENSION tablefunc; INSERT INTO devices(time, cpu_usage, disk_usage) SELECT time, normal_rand(1,70,10) AS cpu_usage, normal_rand(1,2,1) * (row_number() over()) AS disk_usage FROM generate_series(1,10000) AS time; ``` This command uses the `tablefunc` extension to generate a normal distribution, and uses the `row_number` function to turn it into a cumulative sequence. 4. **Check the view data** ```sql postgres=# SELECT * FROM devices_summary ORDER BY bucket LIMIT 10; bucket | avg_cpu | avg_disk --------+---------------------+---------------------- 0 | 63.0000000000000000 | 6.0000000000000000 5 | 69.8000000000000000 | 9.6000000000000000 10 | 70.8000000000000000 | 24.0000000000000000 15 | 75.8000000000000000 | 37.6000000000000000 20 | 71.6000000000000000 | 26.8000000000000000 25 | 67.6000000000000000 | 56.0000000000000000 30 | 68.8000000000000000 | 90.2000000000000000 35 | 71.6000000000000000 | 88.8000000000000000 40 | 66.4000000000000000 | 81.2000000000000000 45 | 68.2000000000000000 | 106.0000000000000000 (10 rows) ``` ## Learn more [Section titled “Learn more”](#learn-more) * [Understand continuous aggregates](/docs/learn/continuous-aggregates): How continuous aggregates work, JOINs, and function support. * [Create a continuous aggregate](/docs/build/continuous-aggregates/create-a-continuous-aggregate): Step-by-step guide to creating and querying a continuous aggregate. * [Refresh continuous aggregates](/docs/build/continuous-aggregates/refresh-policies): Set up automatic refresh policies. * [Understand time buckets](/docs/learn/data-lifecycle/time-buckets/about-time-buckets): How time bucket origins and timezones work. * [`CREATE MATERIALIZED VIEW` reference](/docs/reference/timescaledb/continuous-aggregates/create_materialized_view): Full API reference. # Understand the data lifecycle > Learn how to manage time-series data from ingestion to deletion using hypertables, continuous aggregates, tiered storage, and retention policies Time-series data is relentless. It grows at an ever-increasing rate and quickly becomes expensive and unwieldy to store and query. Managing your time-series data across its entire lifecycle is essential to building high-performance, cost-effective applications. TimescaleDB, a time-series database built on PostgreSQL, provides purpose-built features for every stage of the data lifecycle, from the moment data is ingested to when it is archived or dropped. These features work together so you can maintain query performance, control storage costs, and automate routine maintenance — without building custom pipelines or relying on external tools. For definitions of all terms used in this guide, see the [glossary](/docs/learn/glossary). ## The Five phases [Section titled “The Five phases”](#the-five-phases) The data lifecycle describes the journey your data takes as it ages. Each phase maps to a set of TimescaleDB features that help you manage that stage efficiently: | Phase | What happens | TimescaleDB features | | -------------------------------- | ----------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | **1. Ingest and store** | Data arrives and is written to disk | [Hypertables](/docs/learn/hypertables/understand-hypertables), [chunks](/docs/learn/chunks/understanding-chunks) | | **2. Query recent data** | Applications read and analyze fresh data | SQL, [`time_bucket`](/docs/reference/timescaledb/hyperfunctions/time-series-utilities/time_bucket), [indexes](/docs/learn/hypertables/hypertable-indexes) | | **3. Create aggregated rollups** | Summarize data at coarser intervals | [Continuous aggregates](/docs/learn/continuous-aggregates), [downsampling](/docs/build/data-management/example-downsample-and-compress) | | **4. Compress and archive** | Compress and move aging data to cheaper storage | [Columnstore](/docs/learn/columnar-storage/compression-methods), [tiered storage](/docs/learn/data-lifecycle/storage/about-storage-tiers) | | **5. Drop old data** | Remove data that is no longer valuable | [Retention policies](/docs/learn/data-lifecycle/data-retention/about-data-retention), [`drop_chunks`](/docs/reference/timescaledb/hypertables/drop_chunks) | ![The five phases of the Tiger Data lifecycle: ingest and store, query recent data, create aggregated rollups, compress and archive, and drop old data](/docs/_astro/2026-04-Lifecycle-Main-Light.BZAm_KyW_Z1S2ocL.webp) ![The five phases of the Tiger Data lifecycle: ingest and store, query recent data, create aggregated rollups, compress and archive, and drop old data](/docs/_astro/2026-04-Lifecycle-Main-Dark.DVk9Wtn8_ZnqBTS.webp) ## Phase 1: Ingest and store data efficiently [Section titled “Phase 1: Ingest and store data efficiently”](#phase-1-ingest-and-store-data-efficiently) ![Phase 1 of the data lifecycle: incoming time-series data is written to a hypertable, which automatically partitions it into chunks by time range for fast ingest](/docs/_astro/2026-04-Lifecycle-Phase-1-Light.y2k5DFsC_Z1XNBhg.webp) ![Phase 1 of the data lifecycle: incoming time-series data is written to a hypertable, which automatically partitions it into chunks by time range for fast ingest](/docs/_astro/2026-04-Lifecycle-Phase-1-Dark.BKr2gpEI_Z2gmEh9.webp) When it comes to time-series data, storing data as quickly and as efficiently as possible is the foundation of everything else. TimescaleDB achieves this through [hypertables](/docs/learn/hypertables/understand-hypertables), PostgreSQL tables that are automatically partitioned by time into smaller tables called [chunks](/docs/learn/chunks/understanding-chunks). Each chunk holds data for a specific time range, such as one week. Because only recent chunks are in memory, ingest throughput stays high even as overall data volume grows. You interact with hypertables using standard SQL — there is no new query language to learn. Key considerations at this stage: * **Choose the right chunk interval.** The default is 7 days. Aim for each chunk to be about 25% of available memory. See [sizing hypertable chunks](/docs/learn/hypertables/sizing-hypertable-chunks). * **Plan for columnstore conversion.** As chunks age past their "hot" query window, you can convert them to the [columnstore](/docs/learn/columnar-storage/compression-methods) for significant storage savings. See [Phase 4](#phase-4-compress-and-archive-older-data). ## Phase 2: Query recent data [Section titled “Phase 2: Query recent data”](#phase-2-query-recent-data) ![Phase 2 of the data lifecycle: queries against recent data use chunk exclusion and time\_bucket aggregations to read only the relevant hot chunks](/docs/_astro/2026-04-Lifecycle-Phase-2-Light.DFjVZpLt_1wbWTa.webp) ![Phase 2 of the data lifecycle: queries against recent data use chunk exclusion and time\_bucket aggregations to read only the relevant hot chunks](/docs/_astro/2026-04-Lifecycle-Phase-2-Dark.BpoODnkE_Z5T8w3.webp) Once data is stored, you need to query it efficiently. TimescaleDB extends the PostgreSQL query planner with time-series-aware optimizations. Queries on recent data benefit from chunk exclusion, an optimization where the planner automatically skips any chunks that fall outside your query's time range. For time-series analysis, the [`time_bucket`](/docs/reference/timescaledb/hyperfunctions/time-series-utilities/time_bucket) function is a TimescaleDB SQL function that groups rows into fixed time intervals (1 minute, 1 hour, 1 day), making it easy to compute aggregates over time windows: ```sql SELECT time_bucket('1 hour', time) AS bucket, device_id, avg(temperature) AS avg_temp FROM conditions WHERE time > now() - INTERVAL '7 days' GROUP BY bucket, device_id ORDER BY bucket DESC; ``` As your dataset grows and queries over longer time ranges slow down, it is time to move to the next phase. ## Phase 3: Create aggregated rollups [Section titled “Phase 3: Create aggregated rollups”](#phase-3-create-aggregated-rollups) ![Phase 3 of the data lifecycle: a continuous aggregate incrementally summarizes raw hypertable data into pre-computed rollups that refresh automatically on a schedule](/docs/_astro/2026-04-Lifecycle-Phase-3-Light.kFLH1RHn_4OrD6.webp) ![Phase 3 of the data lifecycle: a continuous aggregate incrementally summarizes raw hypertable data into pre-computed rollups that refresh automatically on a schedule](/docs/_astro/2026-04-Lifecycle-Phase-3-Dark.C9WX4wNh_1kf97y.webp) As data ages, individual data points often matter less than statistical summaries. [Continuous aggregates](/docs/learn/continuous-aggregates) solve this. A continuous aggregate is a materialized view that TimescaleDB keeps up to date automatically — it pre-computes and stores the results of an aggregate query (such as averages or counts) and incrementally refreshes only the time buckets where underlying data changed, not the entire view. Unlike standard PostgreSQL materialized views that recompute everything on each refresh, continuous aggregates are efficient even over very large datasets. ```sql CREATE MATERIALIZED VIEW conditions_daily WITH (timescaledb.continuous) AS SELECT time_bucket('1 day', time) AS day, device_id, avg(temperature) AS avg_temp, min(temperature) AS min_temp, max(temperature) AS max_temp FROM conditions GROUP BY day, device_id; SELECT add_continuous_aggregate_policy('conditions_daily', start_offset => INTERVAL '7 days', end_offset => INTERVAL '1 day', schedule_interval => INTERVAL '1 day'); ``` Continuous aggregates are stored as hypertables themselves, so they benefit from all the same features — columnstore conversion, tiered storage, and retention policies. You can also create [hierarchical continuous aggregates](/docs/learn/continuous-aggregates/hierarchical-continuous-aggregates), where one continuous aggregate reads from another instead of from raw data, to build multi-resolution rollup pipelines (for example, hourly to daily to monthly). ## Phase 4: Compress and archive older data [Section titled “Phase 4: Compress and archive older data”](#phase-4-compress-and-archive-older-data) ![Phase 4 of the data lifecycle: aging chunks are converted to the columnstore for compression and then tiered to low-cost object storage while remaining queryable](/docs/_astro/2026-04-Lifecycle-Phase-4-Light.Z8ijYcJE_Z1H5z1G.webp) ![Phase 4 of the data lifecycle: aging chunks are converted to the columnstore for compression and then tiered to low-cost object storage while remaining queryable](/docs/_astro/2026-04-Lifecycle-Phase-4-Dark.dBv1oecD_2lpeTU.webp) As chunks age past their "hot" query window, you can dramatically reduce storage costs in two ways: convert chunks to the columnstore for compression, and move them to cheaper storage tiers. ### Convert to the columnstore [Section titled “Convert to the columnstore”](#convert-to-the-columnstore) The [columnstore](/docs/learn/columnar-storage/compression-methods) is a columnar storage format that compresses your data. Converting older chunks to the columnstore can reduce storage by up to 98% while improving analytical query performance. Use a [columnstore policy](/docs/reference/timescaledb/hypercore/add_columnstore_policy) to automatically convert chunks after a set interval: ```sql SELECT add_columnstore_policy('conditions', after => INTERVAL '7 days'); ``` Once a chunk is in the columnstore, it still supports inserts and updates — [hypercore](/docs/learn/columnar-storage/understand-hypercore), the underlying storage engine, handles this transparently. ### Tier to low-cost storage [Section titled “Tier to low-cost storage”](#tier-to-low-cost-storage) Even with the columnstore and continuous aggregates, there are cases where you need access to raw data for long periods — for compliance, model training, or historical analysis. Rather than paying for high-performance storage you rarely query, you can move older chunks to cheaper storage. #### On Tiger Cloud [Section titled “On Tiger Cloud”](#on-tiger-cloud) [Tiered storage](/docs/learn/data-lifecycle/storage/about-storage-tiers) is a Tiger Cloud feature that automatically moves older chunks from high-performance storage to a low-cost object storage tier backed by S3 and Azure Blob. Data is stored in Apache Parquet format with automatic chunk, row group, and column pruning for efficient queries. You set a policy and Tiger Cloud handles the rest: ```sql SELECT add_tiering_policy('conditions', INTERVAL '6 months'); ``` Tiered data remains queryable via standard SQL — no separate query path or external pipeline required. #### On self-hosted TimescaleDB [Section titled “On self-hosted TimescaleDB”](#on-self-hosted-timescaledb) Use PostgreSQL [tablespaces](https://www.postgresql.org/docs/current/manage-ag-tablespaces.html) with the [`move_chunk`](/docs/reference/timescaledb/hypertables/move_chunk) function to relocate chunks to slower, cheaper disks. You can automate this with a [custom job](/docs/build/data-management/example-tiered-storage). ## Phase 5: Drop old data [Section titled “Phase 5: Drop old data”](#phase-5-drop-old-data) ![Phase 5 of the data lifecycle: a scheduled retention policy automatically drops entire chunks older than a defined threshold to reclaim storage](/docs/_astro/2026-04-Lifecycle-Phase-5-Light.CU0vYzPy_Z1mHjYf.webp) ![Phase 5 of the data lifecycle: a scheduled retention policy automatically drops entire chunks older than a defined threshold to reclaim storage](/docs/_astro/2026-04-Lifecycle-Phase-5-Dark.Bd9icdvW_Z21os5E.webp) At some point, raw data is no longer valuable. Keeping it around slows down queries, inflates storage costs, and adds maintenance overhead. [Retention policies](/docs/learn/data-lifecycle/data-retention/about-data-retention), scheduled rules that automatically drop chunks older than a threshold you define, automate this process. ```sql SELECT add_retention_policy('conditions', INTERVAL '12 months'); ``` Because TimescaleDB drops entire chunks rather than deleting rows one by one, retention is nearly instantaneous and avoids the performance overhead of `DELETE` + `VACUUM`. Warning If you use continuous aggregates, make sure your retention policy interval is longer than the continuous aggregate refresh window. Otherwise, the refresh sees the deleted raw data and removes the aggregate too. For details, see [data retention with continuous aggregates](/docs/learn/data-lifecycle/data-retention/data-retention-with-continuous-aggregates). ## Putting it all together [Section titled “Putting it all together”](#putting-it-all-together) A complete lifecycle strategy combines multiple phases. Here is an example configuration for a `conditions` hypertable that stores device telemetry: ```sql -- Phase 1: Create the hypertable SELECT create_hypertable('conditions', by_range('time')); -- Phase 3: Create a daily rollup CREATE MATERIALIZED VIEW conditions_daily WITH (timescaledb.continuous) AS SELECT time_bucket('1 day', time) AS day, device_id, avg(temperature), min(temperature), max(temperature) FROM conditions GROUP BY day, device_id; SELECT add_continuous_aggregate_policy('conditions_daily', start_offset => INTERVAL '7 days', end_offset => INTERVAL '1 day', schedule_interval => INTERVAL '1 day'); -- Phase 4a: Convert to columnstore after 7 days SELECT add_columnstore_policy('conditions', after => INTERVAL '7 days'); -- Phase 4b: Tier data to low-cost storage after 3 months (Tiger Cloud) SELECT add_tiering_policy('conditions', INTERVAL '3 months'); -- Phase 5: Drop raw data after 12 months SELECT add_retention_policy('conditions', INTERVAL '12 months'); ``` With this configuration: * Raw data is converted to the columnstore after 7 days, reducing storage by up to 98% * Daily summaries are always available via the continuous aggregate * After 3 months, raw chunks move to low-cost object storage * After 12 months, raw chunks are dropped entirely * The daily continuous aggregate retains its data independently of the raw data All of this runs automatically via scheduled jobs — no cron, no external scripts, no manual intervention. ## Automate and customize [Section titled “Automate and customize”](#automate-and-customize) Every lifecycle feature supports scheduled automation through the TimescaleDB [job scheduler](/docs/build/data-management/about-automation). A job is a user-defined or built-in stored procedure that TimescaleDB runs on a schedule you configure — policies like retention, columnstore conversion, and continuous aggregate refresh are all implemented as jobs under the hood. If the built-in policies don't cover your use case, you can create [custom jobs](/docs/build/data-management/create-and-manage-jobs). Common examples include: * [Generic retention across all hypertables](/docs/build/data-management/example-generic-retention) * [Automated tablespace management](/docs/build/data-management/example-tiered-storage) * [Downsampling and converting to columnstore in one step](/docs/build/data-management/example-downsample-and-compress) ## Get hands on [Section titled “Get hands on”](#get-hands-on) [Your first hypertable](/docs/build/how-to/your-first-hypertable) [Create a hypertable, insert data, and run your first queries.](/docs/build/how-to/your-first-hypertable) [Create a continuous aggregate](/docs/build/continuous-aggregates/create-a-continuous-aggregate) [Build pre-computed rollups that refresh automatically.](/docs/build/continuous-aggregates/create-a-continuous-aggregate) [Set up hypercore](/docs/build/columnar-storage/setup-hypercore) [Enable columnstore compression on your hypertables.](/docs/build/columnar-storage/setup-hypercore) [Manage storage and tiering](/docs/build/data-management/storage/manage-storage) [Move older data to low-cost object storage.](/docs/build/data-management/storage/manage-storage) [Create a retention policy](/docs/build/data-management/data-retention/create-a-retention-policy) [Automate dropping chunks past a defined age.](/docs/build/data-management/data-retention/create-a-retention-policy) [Create and manage custom jobs](/docs/build/data-management/create-and-manage-jobs) [Build your own scheduled automation with the job scheduler.](/docs/build/data-management/create-and-manage-jobs) ## Learn more [Section titled “Learn more”](#learn-more) ### By lifecycle phase [Section titled “By lifecycle phase”](#by-lifecycle-phase) * **Ingest:** [Understand hypertables](/docs/learn/hypertables/understand-hypertables) | [Size chunks](/docs/learn/hypertables/sizing-hypertable-chunks) * **Query:** [Understand time buckets](/docs/learn/data-lifecycle/time-buckets/about-time-buckets) | [Hypertable indexes](/docs/learn/hypertables/hypertable-indexes) * **Rollup:** [Understand continuous aggregates](/docs/learn/continuous-aggregates) | [Hierarchical continuous aggregates](/docs/learn/continuous-aggregates/hierarchical-continuous-aggregates) * **Compress and archive:** [Understand hypercore](/docs/learn/columnar-storage/understand-hypercore) | [Columnstore methods](/docs/learn/columnar-storage/compression-methods) | [Understand tiered storage](/docs/learn/data-lifecycle/storage/about-storage-tiers) * **Drop:** [Understand data retention](/docs/learn/data-lifecycle/data-retention/about-data-retention) | [Retention with continuous aggregates](/docs/learn/data-lifecycle/data-retention/data-retention-with-continuous-aggregates) ### Automation and reference [Section titled “Automation and reference”](#automation-and-reference) * [About automation](/docs/build/data-management/about-automation) * [`add_columnstore_policy()`](/docs/reference/timescaledb/hypercore/add_columnstore_policy) * [`add_job()` reference](/docs/reference/timescaledb/jobs-automation/add_job) # Understand data retention > Discard old data as it reaches a certain age, manually or by setting up automated policies In modern applications, data grows exponentially. As data gets older, it often becomes less useful in day-to-day operations. However, you still need it for analysis. TimescaleDB elegantly solves this problem with [automated data retention policies](/docs/build/data-management/data-retention/create-a-retention-policy). Data retention policies delete raw old data for you on a schedule that you define. By [combining retention policies with continuous aggregates](/docs/learn/data-lifecycle/data-retention/data-retention-with-continuous-aggregates), you can downsample your data and keep useful summaries of it instead. This lets you analyze historical data - while also saving on storage. ## Drop data by chunk [Section titled “Drop data by chunk”](#drop-data-by-chunk) TimescaleDB data retention works on chunks, not on rows. Deleting data row-by-row, for example, with the PostgreSQL `DELETE` command, can be slow. But dropping data by the chunk is faster, because it deletes an entire file from disk. It doesn't need garbage collection and defragmentation. Whether you use a policy or manually drop chunks, TimescaleDB drops data by the chunk. It only drops chunks where all the data is within the specified time range. For example, consider the setup where you have 3 chunks containing data: 1. More than 36 hours old 2. Between 12 and 36 hours old 3. From the last 12 hours You manually drop chunks older than 24 hours. Only the oldest chunk is deleted. The middle chunk is retained, because it contains some data newer than 24 hours. ## Get hands on [Section titled “Get hands on”](#get-hands-on) [Create a retention policy](/docs/build/data-management/data-retention/create-a-retention-policy) [Step-by-step guide to automating data deletion.](/docs/build/data-management/data-retention/create-a-retention-policy) [Data retention with continuous aggregates](/docs/learn/data-lifecycle/data-retention/data-retention-with-continuous-aggregates) [Keep aggregated data while dropping raw data.](/docs/learn/data-lifecycle/data-retention/data-retention-with-continuous-aggregates) [Manually drop chunks](/docs/learn/data-lifecycle/data-retention/manually-drop-chunks) [One-time deletion of chunks by time range.](/docs/learn/data-lifecycle/data-retention/manually-drop-chunks) ## Learn more [Section titled “Learn more”](#learn-more) * [Understand tiered storage](/docs/learn/data-lifecycle/storage/about-storage-tiers): Move old data to low-cost storage instead of deleting it. * [`add_retention_policy()` reference](/docs/reference/timescaledb/data-retention/add_retention_policy): Full API reference. * [`drop_chunks()` reference](/docs/reference/timescaledb/hypertables/drop_chunks): Manual chunk deletion API. # About data retention with continuous aggregates > Combine continuous aggregates with data retention to save on raw data storage while keeping summarized data for historical analysis You can downsample your data by combining a data retention policy with [continuous aggregates](/docs/build/continuous-aggregates/create-a-continuous-aggregate). If you set your refresh policies correctly, you can delete old data from a hypertable without deleting it from any continuous aggregates. This lets you save on raw data storage while keeping summarized data for historical analysis. Warning To keep your aggregates while dropping raw data, you must be careful about refreshing your aggregates. You can delete raw data from the underlying table without deleting data from continuous aggregates, so long as you don't refresh the aggregate over the deleted data. When you refresh a continuous aggregate, TimescaleDB updates the aggregate based on changes in the raw data for the refresh window. If it sees that the raw data was deleted, it also deletes the aggregate data. To prevent this, make sure that the aggregate's refresh window doesn't overlap with any deleted data. For more information, see the following example. As an example, say that you add a continuous aggregate to a `conditions` hypertable that stores device temperatures: ```sql CREATE MATERIALIZED VIEW conditions_summary_daily (day, device, temp) WITH (timescaledb.continuous) AS SELECT time_bucket('1 day', time), device, avg(temperature) FROM conditions GROUP BY (1, 2); SELECT add_continuous_aggregate_policy('conditions_summary_daily', '7 days', '1 day', '1 day'); ``` This creates a `conditions_summary_daily` aggregate which stores the daily temperature per device. The aggregate refreshes every day. Every time it refreshes, it updates with any data changes from 7 days ago to 1 day ago. You should **not** set a 24-hour retention policy on the `conditions` hypertable. If you do, chunks older than 1 day are dropped. Then the aggregate refreshes based on data changes. Since the data change was to delete data older than 1 day, the aggregate also deletes the data. You end up with no data in the `conditions_summary_daily` table. To fix this, set a longer retention policy, for example 30 days: ```sql SELECT add_retention_policy('conditions', INTERVAL '30 days'); ``` Now, chunks older than 30 days are dropped. But when the aggregate refreshes, it doesn't look for changes older than 30 days. It only looks for changes between 7 days and 1 day ago. The raw hypertable still contains data for that time period. So your aggregate retains the data. ## Data retention on a continuous aggregate itself [Section titled “Data retention on a continuous aggregate itself”](#data-retention-on-a-continuous-aggregate-itself) You can also apply data retention on a continuous aggregate itself. For example, you can keep raw data for 30 days, as mentioned earlier. Meanwhile, you can keep daily data for 600 days, and no data beyond that. ## Learn more [Section titled “Learn more”](#learn-more) * [Create a retention policy](/docs/build/data-management/data-retention/create-a-retention-policy): Step-by-step guide to automating data deletion. * [Create a continuous aggregate](/docs/build/continuous-aggregates/create-a-continuous-aggregate): Set up the aggregation that preserves your summarized data. * [Refresh continuous aggregates](/docs/build/continuous-aggregates/refresh-policies): Control how often materialized data is updated. * [Understand data retention](/docs/learn/data-lifecycle/data-retention/about-data-retention): How retention policies and chunk deletion work. * [`add_retention_policy()` reference](/docs/reference/timescaledb/data-retention/add_retention_policy): Full API reference. # Manually drop chunks > Manually drop chunks from your hypertables based on time value Drop chunks manually by time value. For example, drop chunks containing data older than 30 days. Note Dropping chunks manually is a one-time operation. To automatically drop chunks as they age, set up a [data retention policy](/docs/build/data-management/data-retention/create-a-retention-policy). ## Drop chunks older than a certain date [Section titled “Drop chunks older than a certain date”](#drop-chunks-older-than-a-certain-date) To drop chunks older than a certain date, use the [`drop_chunks`](/docs/reference/timescaledb/hypertables/drop_chunks) function. Provide the name of the hypertable to drop chunks from, and a time interval beyond which to drop chunks. For example, to drop chunks with data older than 24 hours: ```sql SELECT drop_chunks('conditions', INTERVAL '24 hours'); ``` ## Drop chunks between 2 dates [Section titled “Drop chunks between 2 dates”](#drop-chunks-between-2-dates) You can also drop chunks between 2 dates. For example, drop chunks with data between 3 and 4 months old. Supply a second `INTERVAL` argument for the `newer_than` cutoff: ```sql SELECT drop_chunks( 'conditions', older_than => INTERVAL '3 months', newer_than => INTERVAL '4 months' ) ``` ## Drop chunks in the future [Section titled “Drop chunks in the future”](#drop-chunks-in-the-future) You can also drop chunks in the future, for example, to correct data with the wrong timestamp. To drop all chunks that are more than 3 months in the future, from a hypertable called `conditions`: ```sql SELECT drop_chunks( 'conditions', newer_than => now() + INTERVAL '3 months' ); ``` ## Learn more [Section titled “Learn more”](#learn-more) * [Create a data retention policy](/docs/build/data-management/data-retention/create-a-retention-policy): Automate chunk deletion based on age. * [Understand chunks](/docs/learn/chunks/understanding-chunks): How chunks work and why they matter. * [`drop_chunks()` reference](/docs/reference/timescaledb/hypertables/drop_chunks): Full API reference with all parameters. * [`show_chunks()` reference](/docs/reference/timescaledb/hypertables/show_chunks): List chunks before dropping them. # Understand tiered storage > Learn how Tiger Cloud helps you save on storage costs with a high-performance storage tier and a low-cost object storage tier The tiered storage architecture in Tiger Cloud includes a high-performance storage tier and a low-cost object storage tier. You use the high-performance tier for data that requires quick access, and the object tier for rarely used historical data. Tiering policies move older data asynchronously and periodically from high-performance to low-cost storage, sparing you the need to do it manually. Chunks from a single hypertable, including compressed chunks, can stretch across these two storage tiers. ![Tiger Cloud tiered storage architecture with high-performance and object storage](/docs/_astro/timescale-tiered-storage-architecture.CBsqA3nu_cHDU.webp) ![Tiger Cloud tiered storage architecture with high-performance and object storage](/docs/_astro/timescale-tiered-storage-architecture-dark.D4kUlVNC_1tQRvC.webp) ## High-performance storage [Section titled “High-performance storage”](#high-performance-storage) High-performance storage is where your data is stored by default, until you [enable tiered storage](/docs/build/data-management/storage/manage-storage#low-cost-object-storage-tier) and [move older data to the low-cost tier](/docs/build/data-management/storage/manage-storage#automate-tiering-with-policies). It provides you with up to 64 TB of storage and 80,000 IOPS (depending on your pricing plan and the use of the I/O Boost feature). In the high-performance storage, your data is stored on a block device that is optimized for frequent querying. The [hypercore row-columnar storage engine](/docs/learn/columnar-storage/understand-hypercore) enables you to compress the data in the high-performance storage by up to 98%, while improving performance. Coupled with other optimizations, Tiger Cloud high-performance storage makes sure your data is durable and available, and your queries run as quickly as they can. ## Low-cost storage [Section titled “Low-cost storage”](#low-cost-storage) Note Low-cost tiered storage is available on Scale and Enterprise pricing plans. Once you [enable tiered storage](/docs/build/data-management/storage/manage-storage#low-cost-object-storage-tier), you can start moving rarely used data to the object tier. The object tier is based on AWS S3 and Azure Blob storage and stores your data in the [Apache Parquet](https://parquet.apache.org/) format. Within a Parquet file, a set of rows is grouped together to form a row group. Within a row group, values for a single column across multiple rows are stored together. The original size of the data in your service, compressed or uncompressed, does not correspond directly to its size in the object store. For low-cost storage, Tiger Data charges for data tiered based on its original uncompressed size in the high-performance storage tier. There are no additional expenses, such as data transfer or compute. Apache Parquet allows for more efficient scans across longer time periods, and Tiger Cloud uses other metadata and query optimizations to reduce the amount of data that needs to be fetched to satisfy a query, such as: * **Chunk skipping**: exclude the chunks that fall outside the query time window. * **Row group skipping**: identify the row groups within the Parquet object that satisfy the query. * **Column skipping**: fetch only columns that are requested by the query. The following query is against a tiered dataset and illustrates the optimizations: ```sql EXPLAIN ANALYZE SELECT count(*) FROM ( SELECT device_uuid, sensor_id FROM public.device_readings WHERE observed_at > '2023-08-28 00:00+00' and observed_at < '2023-08-29 00:00+00' GROUP BY device_uuid, sensor_id ) q; QUERY PLAN ------------------------------------------------------------------------------------------------- Aggregate (cost=7277226.78..7277226.79 rows=1 width=8) (actual time=234993.749..234993.750 rows=1 loops=1) -> HashAggregate (cost=4929031.23..7177226.78 rows=8000000 width=68) (actual time=184256.546..234913.067 rows=1651523 loops=1) Group Key: osm_chunk_1.device_uuid, osm_chunk_1.sensor_id Planned Partitions: 128 Batches: 129 Memory Usage: 20497kB Disk Usage: 4429832kB -> Foreign Scan on osm_chunk_1 (cost=0.00..0.00 rows=92509677 width=68) (actual time=345.890..128688.459 rows=92505457 loops=1) Filter: ((observed_at > '2023-08-28 00:00:00+00'::timestamp with time zone) AND (observed_at < '2023-08-29 00:00:00+00'::timestamp with time zone)) Rows Removed by Filter: 4220 Match tiered objects: 3 Row Groups: _timescaledb_internal._hyper_1_42_chunk: 0-74 _timescaledb_internal._hyper_1_43_chunk: 0-29 _timescaledb_internal._hyper_1_44_chunk: 0-71 S3 requests: 177 S3 data: 224423195 bytes Planning Time: 6.216 ms Execution Time: 235372.223 ms (16 rows) ``` `EXPLAIN` illustrates which chunks are being pulled in from the object storage tier: 1. Fetch data from chunks 42, 43, and 44 from the object storage tier. 2. Skip row groups and limit the fetch to a subset of the offsets in the Parquet object that potentially match the query filter. Only fetch the data for `device_uuid`, `sensor_id`, and `observed_at` as the query needs only these 3 columns. The object storage tier is more than an archiving solution. It is also: * **Cost-effective:** store high volumes of data at a lower cost. You pay only for what you store, with no extra cost for queries. * **Scalable:** scale past the restrictions of even the enhanced high-performance storage tier. * **Online:** your data is always there and can be [queried when needed](/docs/build/data-management/storage/query-tiered-data). By default, tiered data is not included when you query from a Tiger Cloud service. To access tiered data, you [enable tiered reads](/docs/build/data-management/storage/query-tiered-data) for a query, a session, or even for all sessions. After you enable tiered reads, when you run regular SQL queries, a behind-the-scenes process transparently pulls data from wherever it's located: the standard high-performance storage tier, the object storage tier, or both. You can `JOIN` against tiered data, build views, and even define continuous aggregates on it. In fact, because the implementation of continuous aggregates also uses hypertables, they can be tiered to low-cost storage as well. The low-cost storage tier comes with the following limitations: * **Limited schema modifications**: some schema modifications are not allowed on hypertables with tiered chunks. *Allowed* modifications include: renaming the hypertable, adding columns without defaults, adding indexes, changing or renaming the hypertable schema, and adding `CHECK` constraints. For `CHECK` constraints, only untiered data is verified. Columns can also be deleted, but you cannot subsequently add a new column to a tiered hypertable with the same name as the now-deleted column. *Disallowed* modifications include: adding a column with any default value (including `NULL`), renaming a column, changing the data type of a column, and adding a `NOT NULL` constraint to the column. * **Limited data changes**: you cannot insert data into, update, or delete a tiered chunk. These limitations take effect as soon as the chunk is scheduled for tiering. * **Inefficient query planner filtering for non-native data types**: the query planner speeds up reads from our object storage tier by using metadata to filter out columns and row groups that don't satisfy the query. This works for all native data types, but not for non-native types, such as `JSON`, `JSONB`, and `GIS`. * **Latency**: The object store has higher access latency than local storage. This can affect the execution time of queries in latency-sensitive environments, especially lighter queries. * **Number of dimensions**: you cannot use tiered storage with hypertables partitioned on more than one dimension. Make sure your hypertables are partitioned on time only, before you enable tiered storage. ## The tiered storage workflow [Section titled “The tiered storage workflow”](#the-tiered-storage-workflow) The typical workflow to use tiered storage in Tiger Cloud is: 1. **Enable tiered storage** You [enable tiered storage](/docs/build/data-management/storage/manage-storage#low-cost-object-storage-tier) for each service individually. 2. **Tier your data** Choose how to [move data to the low-cost tier](/docs/build/data-management/storage/manage-storage#automate-tiering-with-policies): * **Automated tiering**: create an interval-based policy using `add_tiering_policy()` to automatically tier chunks older than a specified age. By default, policies run hourly and continuously manage data placement. * **Manual tiering**: identify specific chunks to tier by querying the `timescaledb_information.chunks` view, then use the `tier_chunk()` function to move individual chunks. The tiering process is asynchronous, chunks are scheduled for migration and handled by background services. 3. **Query your data** 1. To access data stored in the object tier, set `timescaledb.enable_tiered_reads = true` for your session, query, or all future sessions. Without this setting, queries only access data in the high-performance tier. For more information, see [querying tiered data](/docs/build/data-management/storage/query-tiered-data). 2. Run standard SQL queries against your hypertables. The query planner automatically determines which chunks to access across both storage tiers based on your query filters. Chunk pruning, row group pruning, and column pruning optimize query performance. 4. **Monitor and manage** Track tiered chunks using the `timescaledb_osm.chunks_queued_for_tiering` view. Modify or remove tiering policies as needed using [`alter_job`](/docs/reference/timescaledb/jobs-automation/alter_job) and `remove_tiering_policy`. ## Get hands on [Section titled “Get hands on”](#get-hands-on) [Manage storage and tiering](/docs/build/data-management/storage/manage-storage) [Configure high-performance storage, object storage, and tiering policies.](/docs/build/data-management/storage/manage-storage) [Query tiered data](/docs/build/data-management/storage/query-tiered-data) [Access and query data stored in the object storage tier.](/docs/build/data-management/storage/query-tiered-data) [Replicas and forks with tiered data](/docs/build/data-management/storage/tiered-data-replicas-forks) [How tiered storage works with replicas and forks.](/docs/build/data-management/storage/tiered-data-replicas-forks) ## Learn more [Section titled “Learn more”](#learn-more) * [Data tiering API](/docs/reference/tiger-cloud/data-tiering): Tiering policy and chunk management functions. * [Understand data retention](/docs/learn/data-lifecycle/data-retention/about-data-retention): Drop old data instead of tiering it. * [Data retention API](/docs/reference/timescaledb/data-retention): Retention policy functions. # Understand time buckets > Learn how time buckets help you aggregate data by time interval for efficient and simple real-time analytics The [`time_bucket`](/docs/reference/timescaledb/hyperfunctions/time-series-utilities/time_bucket) function enables you to aggregate data in a [hypertable](/docs/learn/hypertables/understand-hypertables) into buckets of time. For example, 5 minutes, 1 hour, or 3 days. It's similar to PostgreSQL's [`date_bin`](https://www.postgresql.org/docs/current/functions-datetime.html#FUNCTIONS-DATETIME-BIN) function, but it gives you more flexibility in the bucket size and start time. You can use it to roll up data for analysis or downsampling. For example, you can calculate 5-minute averages for a sensor reading over the last day. You can perform these rollups as needed, or pre-calculate them in [continuous aggregates](/docs/build/continuous-aggregates/create-a-continuous-aggregate). For examples of the `time_bucket` function, see [Aggregate time-series data with `time_bucket`](/docs/learn/data-lifecycle/time-buckets/use-time-buckets). ## How time bucketing works [Section titled “How time bucketing works”](#how-time-bucketing-works) Time bucketing groups data into time intervals. With `time_bucket`, the interval length can be any number of microseconds, milliseconds, seconds, minutes, hours, days, weeks, months, or years. The `time_bucket` function is usually used in combination with `GROUP BY` to aggregate data. For example, you can calculate the average, maximum, minimum, or sum of values within a bucket. ![Diagram showing time\_bucket aggregating data into daily buckets and calculating the daily sum](/docs/_astro/time-bucket.B55ZDUcx_ZXDMBe.webp) ![Diagram showing time\_bucket aggregating data into daily buckets and calculating the daily sum](/docs/_astro/time-bucket-dark.5YC19Bnb_ZJClhm.webp) ### Origin [Section titled “Origin”](#origin) The origin determines when time buckets start and end. By default, a time bucket doesn't start at the earliest timestamp in your data. There is often a more logical time. For example, you might collect your first data point at `00:37`, but you probably want your daily buckets to start at midnight. Similarly, you might collect your first data point on a Wednesday, but you might want your weekly buckets calculated from Sunday or Monday. Instead, time is divided into buckets based on intervals from the origin. The following diagram shows how, using the example of 2-week buckets. The first possible start date for a bucket is `origin`. The next possible start date for a bucket is `origin + bucket interval`. If your first timestamp does not fall exactly on a possible start date, the immediately preceding start date is used for the beginning of the bucket. ![Diagram showing how time bucket boundaries are calculated from the origin timestamp](/docs/_astro/time-bucket-origin.De4qvp0I_1XQI96.webp) ![Diagram showing how time bucket boundaries are calculated from the origin timestamp](/docs/_astro/time-bucket-origin-dark.Cmbud1B4_YxUIw.webp) For example, say that your data's earliest timestamp is April 24, 2020. If you bucket by an interval of two weeks, the first bucket doesn't start on April 24, which is a Friday. It also doesn't start on April 20, which is the immediately preceding Monday. It starts on April 13, because you can get to April 13, 2020, by counting in two-week increments from January 3, 2000, which is the default origin in this case. #### Default origins [Section titled “Default origins”](#default-origins) For intervals that don't include months or years, the default origin is January 3, 2000. For month or year intervals, the default origin is January 1, 2000. For integer time values, the default origin is 0. These choices make the time ranges of time buckets more intuitive. Because January 3, 2000, is a Monday, weekly time buckets start on Monday. This is compliant with the ISO standard for calculating calendar weeks. Monthly and yearly time buckets use January 1, 2000, as an origin. This allows them to start on the first day of the calendar month or year. If you prefer another origin, you can set it yourself using the [`origin` parameter](/docs/reference/timescaledb/hyperfunctions/time-series-utilities/time_bucket#arguments). For example, to start weeks on Sunday, set the origin to Sunday, January 2, 2000. ### Timezones [Section titled “Timezones”](#timezones) The origin time depends on the data type of your time values. If you use `TIMESTAMP`, by default, bucket start times are aligned with `00:00:00`. Daily and weekly buckets start at `00:00:00`. Shorter buckets start at a time that you can get to by counting in bucket increments from `00:00:00` on the origin date. If you use `TIMESTAMPTZ`, by default, bucket start times are aligned with `00:00:00 UTC`. To align time buckets to another timezone, set the `timezone` parameter. ## Learn more [Section titled “Learn more”](#learn-more) * [Use time buckets](/docs/learn/data-lifecycle/time-buckets/use-time-buckets): SQL examples for grouping, offsetting, and calculating time buckets. * [Create a continuous aggregate](/docs/build/continuous-aggregates/create-a-continuous-aggregate): Pre-compute time bucket aggregations with continuous aggregates. * [`time_bucket()` reference](/docs/reference/timescaledb/hyperfunctions/time-series-utilities/time_bucket): Full API reference with all parameters. * [Gapfilling and interpolation](/docs/build/data-management/hyperfunctions/gapfilling-interpolation): Fill gaps in time bucket results with `time_bucket_gapfill()`. # Use time buckets > Group data by time interval using the time_bucket function to perform aggregate calculations over arbitrary time intervals The `time_bucket` function helps you group data in a [hypertable](/docs/learn/hypertables/understand-hypertables) so you can perform aggregate calculations over arbitrary time intervals. It is usually used in combination with `GROUP BY` for this purpose. This section shows examples of `time_bucket` use. To learn how time buckets work, see the [about time buckets section](/docs/learn/data-lifecycle/time-buckets/about-time-buckets). ## Group data by time buckets and calculate a summary value [Section titled “Group data by time buckets and calculate a summary value”](#group-data-by-time-buckets-and-calculate-a-summary-value) Group data into time buckets and calculate a summary value for a column. For example, calculate the average daily temperature in a table named `weather_conditions`. The table has a time column named `time` and a `temperature` column: ```sql SELECT time_bucket('1 day', time) AS bucket, avg(temperature) AS avg_temp FROM weather_conditions GROUP BY bucket ORDER BY bucket ASC; ``` The `time_bucket` function returns the start time of the bucket. In this example, the first bucket starts at midnight on November 15, 2016, and aggregates all the data from that day: ```sql bucket | avg_temp -----------------------+--------------------- 2016-11-15 00:00:00+00 | 68.3704391666665821 2016-11-16 00:00:00+00 | 67.0816684374999347 ``` ## Group data by time buckets and show the end time of the bucket [Section titled “Group data by time buckets and show the end time of the bucket”](#group-data-by-time-buckets-and-show-the-end-time-of-the-bucket) By default, the `time_bucket` column shows the start time of the bucket. If you prefer to show the end time, you can shift the displayed time using a mathematical operation on `time`. For example, you can calculate the minimum and maximum CPU usage for 5-minute intervals, and show the end of time of the interval. The example table is named `metrics`. It has a time column named `time` and a CPU usage column named `cpu`: ```sql SELECT time_bucket('5 min', time) + '5 min' AS bucket, min(cpu), max(cpu) FROM metrics GROUP BY bucket ORDER BY bucket DESC; ``` The addition of `+ '5 min'` changes the displayed timestamp to the end of the bucket. It doesn't change the range of times spanned by the bucket. ## Group data by time buckets and change the time range of the bucket [Section titled “Group data by time buckets and change the time range of the bucket”](#group-data-by-time-buckets-and-change-the-time-range-of-the-bucket) To change the time range spanned by the buckets, use the `offset` parameter, which takes an `INTERVAL` argument. A positive offset shifts the start and end time of the buckets later. A negative offset shifts the start and end time of the buckets earlier. For example, you can calculate the average CPU usage for 5-hour intervals, and shift the start and end times of all buckets 1 hour later: ```sql SELECT time_bucket('5 hours', time, '1 hour'::INTERVAL) AS bucket, avg(cpu) FROM metrics GROUP BY bucket ORDER BY bucket DESC; ``` ## Calculate the time bucket of a single value [Section titled “Calculate the time bucket of a single value”](#calculate-the-time-bucket-of-a-single-value) Time buckets are usually used together with `GROUP BY` to aggregate data. But you can also run `time_bucket` on a single time value. This is useful for testing and learning, because you can see what bucket a value falls into. For example, to see the 1-week time bucket into which January 5, 2021 would fall, run: ```sql SELECT time_bucket(INTERVAL '1 week', TIMESTAMP '2021-01-05'); ``` The function returns `2021-01-04 00:00:00`. The start time of the time bucket is the Monday of that week, at midnight. ## Learn more [Section titled “Learn more”](#learn-more) * [Understand time buckets](/docs/learn/data-lifecycle/time-buckets/about-time-buckets): How time buckets work, origins, and timezones. * [Create a continuous aggregate](/docs/build/continuous-aggregates/create-a-continuous-aggregate): Pre-compute time bucket aggregations with continuous aggregates. * [`time_bucket()` reference](/docs/reference/timescaledb/hyperfunctions/time-series-utilities/time_bucket): Full API reference with all parameters. * [Troubleshoot time buckets](/docs/build/tips-and-tricks/troubleshoot-time-buckets): Common issues and solutions. # Understand the data lifecycle > Learn how to manage time-series data from ingestion to deletion using hypertables, continuous aggregates, tiered storage, and retention policies Time-series data is relentless. It grows at an ever-increasing rate and quickly becomes expensive and unwieldy to store and query. Managing your time-series data across its entire lifecycle is essential to building high-performance, cost-effective applications. TimescaleDB, a time-series database built on PostgreSQL, provides purpose-built features for every stage of the data lifecycle, from the moment data is ingested to when it is archived or dropped. These features work together so you can maintain query performance, control storage costs, and automate routine maintenance — without building custom pipelines or relying on external tools. For definitions of all terms used in this guide, see the [glossary](/docs/learn/glossary). ## The five phases [Section titled “The five phases”](#the-five-phases) The data lifecycle describes the journey your data takes as it ages. Each phase maps to a set of TimescaleDB features that help you manage that stage efficiently: | Phase | What happens | TimescaleDB features | | -------------------------------- | ----------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | **1. Ingest and store** | Data arrives and is written to disk | [Hypertables](/docs/learn/hypertables/understand-hypertables), [chunks](/docs/learn/chunks/understanding-chunks) | | **2. Query recent data** | Applications read and analyze fresh data | SQL, [`time_bucket`](/docs/reference/timescaledb/hyperfunctions/time-series-utilities/time_bucket), [indexes](/docs/learn/hypertables/hypertable-indexes) | | **3. Create aggregated rollups** | Summarize data at coarser intervals | [Continuous aggregates](/docs/learn/continuous-aggregates), [downsampling](/docs/build/data-management/example-downsample-and-compress) | | **4. Compress and archive** | Compress and move aging data to cheaper storage | [Columnstore](/docs/learn/columnar-storage/compression-methods), [tiered storage](/docs/learn/data-lifecycle/storage/about-storage-tiers) | | **5. Drop old data** | Remove data that is no longer valuable | [Retention policies](/docs/learn/data-lifecycle/data-retention/about-data-retention), [`drop_chunks`](/docs/reference/timescaledb/hypertables/drop_chunks) | ## Phase 1: Ingest and store data efficiently [Section titled “Phase 1: Ingest and store data efficiently”](#phase-1-ingest-and-store-data-efficiently) When it comes to time-series data, storing data as quickly and as efficiently as possible is the foundation of everything else. TimescaleDB achieves this through [hypertables](/docs/learn/hypertables/understand-hypertables), PostgreSQL tables that are automatically partitioned by time into smaller tables called [chunks](/docs/learn/chunks/understanding-chunks). Each chunk holds data for a specific time range, such as one week. Because only recent chunks are in memory, ingest throughput stays high even as overall data volume grows. You interact with hypertables using standard SQL — there is no new query language to learn. Key considerations at this stage: * **Choose the right chunk interval.** The default is 7 days. Aim for each chunk to be about 25% of available memory. See [sizing hypertable chunks](/docs/learn/hypertables/sizing-hypertable-chunks). * **Plan for columnstore conversion.** As chunks age past their "hot" query window, you can convert them to the [columnstore](/docs/learn/columnar-storage/compression-methods) for significant storage savings. See [Phase 4](#phase-4-compress-and-archive-older-data). ## Phase 2: Query recent data [Section titled “Phase 2: Query recent data”](#phase-2-query-recent-data) Once data is stored, you need to query it efficiently. TimescaleDB extends the PostgreSQL query planner with time-series-aware optimizations. Queries on recent data benefit from chunk exclusion, an optimization where the planner automatically skips any chunks that fall outside your query's time range. For time-series analysis, the [`time_bucket`](/docs/reference/timescaledb/hyperfunctions/time-series-utilities/time_bucket) function is a TimescaleDB SQL function that groups rows into fixed time intervals (1 minute, 1 hour, 1 day), making it easy to compute aggregates over time windows: ```sql SELECT time_bucket('1 hour', time) AS bucket, device_id, avg(temperature) AS avg_temp FROM conditions WHERE time > now() - INTERVAL '7 days' GROUP BY bucket, device_id ORDER BY bucket DESC; ``` As your dataset grows and queries over longer time ranges slow down, it is time to move to the next phase. ## Phase 3: Create aggregated rollups [Section titled “Phase 3: Create aggregated rollups”](#phase-3-create-aggregated-rollups) As data ages, individual data points often matter less than statistical summaries. [Continuous aggregates](/docs/learn/continuous-aggregates) solve this. A continuous aggregate is a materialized view that TimescaleDB keeps up to date automatically — it pre-computes and stores the results of an aggregate query (such as averages or counts) and incrementally refreshes only the time buckets where underlying data changed, not the entire view. Unlike standard PostgreSQL materialized views that recompute everything on each refresh, continuous aggregates are efficient even over very large datasets. ```sql CREATE MATERIALIZED VIEW conditions_daily WITH (timescaledb.continuous) AS SELECT time_bucket('1 day', time) AS day, device_id, avg(temperature) AS avg_temp, min(temperature) AS min_temp, max(temperature) AS max_temp FROM conditions GROUP BY day, device_id; SELECT add_continuous_aggregate_policy('conditions_daily', start_offset => INTERVAL '7 days', end_offset => INTERVAL '1 day', schedule_interval => INTERVAL '1 day'); ``` Continuous aggregates are stored as hypertables themselves, so they benefit from all the same features — columnstore conversion, tiered storage, and retention policies. You can also create [hierarchical continuous aggregates](/docs/learn/continuous-aggregates/hierarchical-continuous-aggregates), where one continuous aggregate reads from another instead of from raw data, to build multi-resolution rollup pipelines (for example, hourly to daily to monthly). ## Phase 4: Compress and archive older data [Section titled “Phase 4: Compress and archive older data”](#phase-4-compress-and-archive-older-data) As chunks age past their "hot" query window, you can dramatically reduce storage costs in two ways: convert chunks to the columnstore for compression, and move them to cheaper storage tiers. ### Convert to the columnstore [Section titled “Convert to the columnstore”](#convert-to-the-columnstore) The [columnstore](/docs/learn/columnar-storage/compression-methods) is a columnar storage format that compresses your data. Converting older chunks to the columnstore can reduce storage by up to 98% while improving analytical query performance. Use a [columnstore policy](/docs/reference/timescaledb/hypercore/add_columnstore_policy) to automatically convert chunks after a set interval: ```sql SELECT add_columnstore_policy('conditions', after => INTERVAL '7 days'); ``` Once a chunk is in the columnstore, it still supports inserts and updates — [hypercore](/docs/learn/columnar-storage/understand-hypercore), the underlying storage engine, handles this transparently. ### Tier to low-cost storage [Section titled “Tier to low-cost storage”](#tier-to-low-cost-storage) Even with the columnstore and continuous aggregates, there are cases where you need access to raw data for long periods — for compliance, model training, or historical analysis. Rather than paying for high-performance storage you rarely query, you can move older chunks to cheaper storage. #### On Tiger Cloud [Section titled “On Tiger Cloud”](#on-tiger-cloud) [Tiered storage](/docs/learn/data-lifecycle/storage/about-storage-tiers) is a Tiger Cloud feature that automatically moves older chunks from high-performance storage to a low-cost object storage tier backed by S3 and Azure Blob. Data is stored in Apache Parquet format with automatic chunk, row group, and column pruning for efficient queries. You set a policy and Tiger Cloud handles the rest: ```sql SELECT add_tiering_policy('conditions', INTERVAL '6 months'); ``` Tiered data remains queryable via standard SQL — no separate query path or external pipeline required. #### On self-hosted TimescaleDB [Section titled “On self-hosted TimescaleDB”](#on-self-hosted-timescaledb) Use PostgreSQL [tablespaces](https://www.postgresql.org/docs/current/manage-ag-tablespaces.html) with the [`move_chunk`](/docs/reference/timescaledb/hypertables/move_chunk) function to relocate chunks to slower, cheaper disks. You can automate this with a [custom job](/docs/build/data-management/example-tiered-storage). ## Phase 5: Drop old data [Section titled “Phase 5: Drop old data”](#phase-5-drop-old-data) At some point, raw data is no longer valuable. Keeping it around slows down queries, inflates storage costs, and adds maintenance overhead. [Retention policies](/docs/learn/data-lifecycle/data-retention/about-data-retention), scheduled rules that automatically drop chunks older than a threshold you define, automate this process. ```sql SELECT add_retention_policy('conditions', INTERVAL '12 months'); ``` Because TimescaleDB drops entire chunks rather than deleting rows one by one, retention is nearly instantaneous and avoids the performance overhead of `DELETE` + `VACUUM`. Warning If you use continuous aggregates, make sure your retention policy interval is longer than the continuous aggregate refresh window. Otherwise, the refresh sees the deleted raw data and removes the aggregate too. For details, see [data retention with continuous aggregate s](/docs/learn/data-lifecycle/data-retention/data-retention-with-continuous-aggregates). ## Putting it all together [Section titled “Putting it all together”](#putting-it-all-together) A complete lifecycle strategy combines multiple phases. Here is an example configuration for a `conditions` hypertable that stores device telemetry: ```sql -- Phase 1: Create the hypertable SELECT create_hypertable('conditions', by_range('time')); -- Phase 3: Create a daily rollup CREATE MATERIALIZED VIEW conditions_daily WITH (timescaledb.continuous) AS SELECT time_bucket('1 day', time) AS day, device_id, avg(temperature), min(temperature), max(temperature) FROM conditions GROUP BY day, device_id; SELECT add_continuous_aggregate_policy('conditions_daily', start_offset => INTERVAL '7 days', end_offset => INTERVAL '1 day', schedule_interval => INTERVAL '1 day'); -- Phase 4a: Convert to columnstore after 7 days SELECT add_columnstore_policy('conditions', after => INTERVAL '7 days'); -- Phase 4b: Tier data to low-cost storage after 3 months (Tiger Cloud) SELECT add_tiering_policy('conditions', INTERVAL '3 months'); -- Phase 5: Drop raw data after 12 months SELECT add_retention_policy('conditions', INTERVAL '12 months'); ``` With this configuration: * Raw data is converted to the columnstore after 7 days, reducing storage by up to 98% * Daily summaries are always available via the continuous aggregate * After 3 months, raw chunks move to low-cost object storage * After 12 months, raw chunks are dropped entirely * The daily continuous aggregate retains its data independently of the raw data All of this runs automatically via scheduled jobs — no cron, no external scripts, no manual intervention. ## Automate and customize [Section titled “Automate and customize”](#automate-and-customize) Every lifecycle feature supports scheduled automation through the TimescaleDB [job scheduler](/docs/build/data-management/about-automation). A job is a user-defined or built-in stored procedure that TimescaleDB runs on a schedule you configure — policies like retention, columnstore conversion, and continuous aggregate refresh are all implemented as jobs under the hood. If the built-in policies don't cover your use case, you can create [custom jobs](/docs/build/data-management/create-and-manage-jobs). Common examples include: * [Generic retention across all hypertables](/docs/build/data-management/example-generic-retention) * [Automated tablespace management](/docs/build/data-management/example-tiered-storage) * [Downsampling and converting to columnstore in one step](/docs/build/data-management/example-downsample-and-compress) ## Get hands on [Section titled “Get hands on”](#get-hands-on) [Your first hypertable](/docs/build/how-to/your-first-hypertable) [Create a hypertable, insert data, and run your first queries.](/docs/build/how-to/your-first-hypertable) [Create a continuous aggregate](/docs/build/continuous-aggregates/create-a-continuous-aggregate) [Build pre-computed rollups that refresh automatically.](/docs/build/continuous-aggregates/create-a-continuous-aggregate) [Set up hypercore](/docs/build/columnar-storage/setup-hypercore) [Enable columnstore compression on your hypertables.](/docs/build/columnar-storage/setup-hypercore) [Manage storage and tiering](/docs/build/data-management/storage/manage-storage) [Move older data to low-cost object storage.](/docs/build/data-management/storage/manage-storage) [Create a retention policy](/docs/build/data-management/data-retention/create-a-retention-policy) [Automate dropping chunks past a defined age.](/docs/build/data-management/data-retention/create-a-retention-policy) [Create and manage custom jobs](/docs/build/data-management/create-and-manage-jobs) [Build your own scheduled automation with the job scheduler.](/docs/build/data-management/create-and-manage-jobs) ## Learn more [Section titled “Learn more”](#learn-more) ### By lifecycle phase [Section titled “By lifecycle phase”](#by-lifecycle-phase) * **Ingest:** [Understand hypertables](/docs/learn/hypertables/understand-hypertables) | [Size chunks](/docs/learn/hypertables/sizing-hypertable-chunks) * **Query:** [Understand time buckets](/docs/learn/data-lifecycle/time-buckets/about-time-buckets) | [Hypertable indexes](/docs/learn/hypertables/hypertable-indexes) * **Rollup:** [Understand continuous aggregates](/docs/learn/continuous-aggregates) | [Hierarchical continuous aggregates](/docs/learn/continuous-aggregates/hierarchical-continuous-aggregates) * **Compress and archive:** [Understand hypercore](/docs/learn/columnar-storage/understand-hypercore) | [Columnstore methods](/docs/learn/columnar-storage/compression-methods) | [Understand tiered storage](/docs/learn/data-lifecycle/storage/about-storage-tiers) * **Drop:** [Understand data retention](/docs/learn/data-lifecycle/data-retention/about-data-retention) | [Retention with continuous aggregates](/docs/learn/data-lifecycle/data-retention/data-retention-with-continuous-aggregates) ### Automation and reference [Section titled “Automation and reference”](#automation-and-reference) * [About automation](/docs/build/data-management/about-automation) * [`add_columnstore_policy()`](/docs/reference/timescaledb/hypercore/add_columnstore_policy) * [`add_job()` reference](/docs/reference/timescaledb/jobs-automation/add_job) # Design your data model > Overview of the data model section covering table layouts, partition columns, keys, and schema optimization This section covers the decisions you make when designing schemas for time-series data on PostgreSQL with TimescaleDB. * [Wide, narrow, and medium tables](/docs/learn/data-model/wide-narrow-medium-tables): Choose a table layout based on how many data points you track and how often the schema changes. * [Primary keys, time columns, and uniqueness](/docs/learn/data-model/primary-keys-time-and-uniqueness): Pick a partition column, define keys that include it, and understand how constraints work with hypertables. * [Schema optimization](/docs/learn/data-model/understand-database-schemas): Indexes, constraints, triggers, tablespaces, JSON, and foreign data wrappers. # Primary keys, time columns, and uniqueness for hypertables > Choose a time partition column, primary keys, and unique constraints so your hypertable matches TimescaleDB rules and your query patterns Every hypertable has a **partitioning column**, usually **time**. That choice drives chunk boundaries, retention, columnstore, and which **unique** and **primary key** constraints are valid. Decide these when you model the table, not after data is loaded. ## Time column [Section titled “Time column”](#time-column) The partition column defines how rows are grouped into **chunks**. It is almost always: * A **`timestamptz`** (or `timestamp` / `date`) column for event time * An **integer** (`smallint`, `int`, or `bigint`) column for Unix epoch seconds, milliseconds, or other monotonic keys ### `timestamptz` vs integer time [Section titled “timestamptz vs integer time”](#timestamptz-vs-integer-time) * **`timestamptz`**: Natural for real-world clocks, time zones, and human-readable queries. Prefer this for most applications. * **Integer epoch**: Useful when upstream systems already emit epoch integers. If you use background jobs like retention or columnstore policies, set [`integer_now_func`](/docs/reference/timescaledb/hypertables/set_integer_now_func) so those jobs know what "current" means. Pick a type you can keep **stable** — changing the partition column later requires a migration. ## Primary keys and unique constraints [Section titled “Primary keys and unique constraints”](#primary-keys-and-unique-constraints) TimescaleDB requires that **every unique constraint or primary key includes the partitioning column**. Additionally, if your hypertable uses multiple partitioning dimensions (for example, space partitioning on `device_id`), all partitioning columns must appear in the constraint. This rule exists because uniqueness is enforced **per chunk**. Including all partitioning columns ensures that PostgreSQL can check the constraint within a single chunk without scanning the entire hypertable. ### Valid examples [Section titled “Valid examples”](#valid-examples) A sensor table where each device writes at most one row per timestamp: ```sql CREATE TABLE sensor_data ( time TIMESTAMPTZ NOT NULL, device_id INT NOT NULL, temperature DOUBLE PRECISION, humidity DOUBLE PRECISION, PRIMARY KEY (device_id, time) ) WITH (timescaledb.hypertable); ``` An order events table with a surrogate ID that still includes the partition column: ```sql CREATE TABLE order_events ( time TIMESTAMPTZ NOT NULL, order_id BIGINT NOT NULL, status TEXT, UNIQUE (order_id, time) ) WITH (timescaledb.hypertable); ``` ### Invalid example [Section titled “Invalid example”](#invalid-example) A surrogate key that does not include the partition column fails: ```sql -- This fails because 'id' alone cannot guarantee uniqueness per chunk CREATE TABLE bad_example ( id SERIAL PRIMARY KEY, time TIMESTAMPTZ NOT NULL, value DOUBLE PRECISION ) WITH (timescaledb.hypertable); ``` ### Unique constraints and columnstore [Section titled “Unique constraints and columnstore”](#unique-constraints-and-columnstore) When you insert into a columnstore chunk that has a unique constraint or primary key, TimescaleDB decompresses data in memory to check for constraint violations. Hypertables **without** unique constraints skip this check, so inserts into columnstore chunks are faster. If your workload is append-only with no duplicates, consider whether you truly need a unique constraint. ### Indexes [Section titled “Indexes”](#indexes) Unique constraints create indexes. Those indexes affect **ingest cost** and **storage**. See [Hypertable indexes](/docs/learn/hypertables/hypertable-indexes) and [Hypertables and unique indexes](/docs/build/performance-optimization/hypertables-and-unique-indexes) for tuning and edge cases. ## Modeling checklist [Section titled “Modeling checklist”](#modeling-checklist) * **Partition column**: One clear time or integer column used consistently in queries and policies. * **Uniqueness**: Express real-world identity (`sensor + time`, `order_id + time`) in keys that **include** all partitioning columns. * **Nulls**: TimescaleDB enforces `NOT NULL` on the partition column automatically. Make sure your ingest pipeline never sends null values for this column. * **Future timestamps**: Far-future `time` values create unexpected chunks and affect retention behavior. Validate timestamps at ingest. ## Learn more [Section titled “Learn more”](#learn-more) * [Wide, narrow, and medium tables](/docs/learn/data-model/wide-narrow-medium-tables): Choose a table layout for your time-series data. * [Create and configure a hypertable](/docs/learn/hypertables/creating-and-configuring-hypertables): `CREATE TABLE` options and configuration. * [Partition a hypertable](/docs/learn/hypertables/partitioning-hypertables): Time, integer, and space partitioning. * [Enforce constraints with unique indexes](/docs/build/performance-optimization/hypertables-and-unique-indexes): Unique index rules and limitations. * [CREATE TABLE reference](/docs/reference/timescaledb/hypertables/create_table): Full API reference. # Understand schema optimization > Schema optimization options available in TimescaleDB and Tiger Cloud, including indexes, constraints, triggers, tablespaces, JSON, and FDW TimescaleDB inherits the full schema toolbox from PostgreSQL — indexes, constraints, triggers, tablespaces, foreign data wrappers, and JSON/JSONB types — and applies them to hypertables and chunks. A well-designed schema keeps hot paths fast; a poor fit shows up as wasted I/O, bloated indexes, or painful migrations later. ## Indexes [Section titled “Indexes”](#indexes) Indexes are the primary tool for [accelerating queries](/docs/build/performance-optimization/indexing). TimescaleDB automatically creates a time-based index on every hypertable. You can add [secondary indexes](/docs/build/performance-optimization/secondary-indexes) (B-tree, hash, GiST, GIN) to speed up filters on non-time columns such as device IDs, tags, or status codes. For hypertables with hypercore enabled, secondary indexes also support chunk skipping, which lets the query planner skip entire chunks that don't match a filter. ## Constraints [Section titled “Constraints”](#constraints) Use PostgreSQL [constraints](/docs/build/performance-optimization/ensure-data-integrity-with-constraints) — `NOT NULL`, `CHECK`, `UNIQUE`, and foreign keys — to enforce data integrity at the database level. Hypertables support all standard constraint types, with the requirement that `UNIQUE` and `PRIMARY KEY` constraints must include the partitioning column. ## Triggers [Section titled “Triggers”](#triggers) [Triggers](/docs/build/performance-optimization/automate-tasks-with-triggers) let you run custom logic automatically when rows are inserted, updated, or deleted. Common use cases include auditing changes, enforcing business rules, and transforming data on write. ## Tablespaces [Section titled “Tablespaces”](#tablespaces) [Tablespaces](/docs/build/performance-optimization/manage-tablespaces) let you control where PostgreSQL stores data on disk. You can place hot chunks on fast SSDs and move older chunks to slower, cheaper storage. This is primarily useful for self-hosted deployments — Tiger Cloud manages storage tiers automatically. ## JSON and JSONB [Section titled “JSON and JSONB”](#json-and-jsonb) When your data has varying or unpredictable fields — for example, IoT sensors that report different measurements — use PostgreSQL [`JSONB` columns](/docs/build/performance-optimization/handle-semi-structured-data-with-json) for the flexible parts and regular columns for the fields you always filter or aggregate on. This gives you schema flexibility without sacrificing query performance on known fields. ## Foreign data wrappers (FDW) [Section titled “Foreign data wrappers (FDW)”](#foreign-data-wrappers-fdw) [Foreign data wrappers](/docs/build/performance-optimization/query-external-data-sources-with-fdw) let you query external PostgreSQL data sources as if they were local tables. This is useful for joining time-series data with reference data stored elsewhere. ## Schema changes [Section titled “Schema changes”](#schema-changes) You can [alter hypertable schemas](/docs/build/performance-optimization/alter-update-table-schema) after creation: add or drop columns, change data types, rename columns, and modify constraints. Schema changes propagate automatically across all chunks. ## Best practices [Section titled “Best practices”](#best-practices) * Set the time column datatype to `timestamptz`, not `timestamp`. See [PostgreSQL timestamp best practices](https://wiki.postgresql.org/wiki/Don't_Do_This#Don.27t_use_timestamp_.28without_time_zone.29). * Add secondary indexes only on columns you actively filter or join on — unused indexes slow down writes. * Use `JSONB` over `JSON` for semi-structured data — `JSONB` supports indexing and is more efficient to query. * Include the partitioning column in any `UNIQUE` or `PRIMARY KEY` constraint. ## Learn more [Section titled “Learn more”](#learn-more) * [Primary keys, time columns, and uniqueness](/docs/learn/data-model/primary-keys-time-and-uniqueness): Partition column and constraint rules for hypertables. * [Wide, narrow, and medium tables](/docs/learn/data-model/wide-narrow-medium-tables): Choose a table layout for your time-series data. * [Accelerate queries using indexes](/docs/build/performance-optimization/indexing): How-to guide for creating and tuning indexes. * [Alter and update table schemas](/docs/build/performance-optimization/alter-update-table-schema): Modify schemas on existing hypertables. * [Handle semi-structured data with JSON](/docs/build/performance-optimization/handle-semi-structured-data-with-json): Using JSONB columns effectively. # Wide, narrow, and medium tables > Choose a narrow, wide, or medium table layout for your time-series data in PostgreSQL When you design a PostgreSQL schema for time-series data, choosing between **narrow**, **wide**, and **medium** table layouts affects how you query, how easily you add new metrics, and how well your data compresses into columnstore. This page explains each layout with examples and helps you pick the right one. ## Narrow table layout [Section titled “Narrow table layout”](#narrow-table-layout) A narrow table has few value columns but may include multiple metadata columns. Each metric gets its own table and data type. ```sql -- One table per metric CREATE TABLE cpu_usage ( time TIMESTAMPTZ NOT NULL, host_id INT, usage DOUBLE PRECISION ); CREATE TABLE memory_usage ( time TIMESTAMPTZ NOT NULL, host_id INT, bytes_used BIGINT ); ``` **Advantages:** * Adding a new metric means creating a new table — no schema changes to existing tables. * Each metric has its own precise data type. * Columns are uniform, so compression ratios are typically high. **Disadvantages:** * Many tables become hard to manage operationally. * Querying several metrics at once requires joins or unions and more I/O. ## Wide table layout [Section titled “Wide table layout”](#wide-table-layout) A wide table has many columns, often one per metric. Related values live in a single row, so common queries avoid joins. ```sql CREATE TABLE host_metrics ( time TIMESTAMPTZ NOT NULL, host_id INT, cpu_usage DOUBLE PRECISION, mem_bytes BIGINT, disk_io_read BIGINT, disk_io_write BIGINT, net_rx_bytes BIGINT, net_tx_bytes BIGINT ); ``` **Advantages:** * Simpler queries — all metrics for a host are in one row. * Efficient column access, especially with columnar compression. **Disadvantages:** * New metrics require `ALTER TABLE ADD COLUMN`, which can be costly on large tables. * Dropped columns do not immediately reclaim disk space. ## Medium table layout [Section titled “Medium table layout”](#medium-table-layout) A medium layout uses one column per **data type** rather than one column per metric. A separate metadata column identifies which metric each row represents. ```sql CREATE TABLE metrics ( time TIMESTAMPTZ NOT NULL, host_id INT, metric_name TEXT, value_double DOUBLE PRECISION, value_bigint BIGINT ); ``` **Advantages:** * New metrics don't require schema changes — just insert rows with a new `metric_name`. * Works well for multi-tenancy where each tenant may have different metrics. **Disadvantages:** * You need a mapping from logical metric names to typed columns. * Querying metrics of different types requires filtering and casting. ## Comparison [Section titled “Comparison”](#comparison) | | Narrow | Wide | Medium | | -------------------- | --------------------------------------------- | --------------------------------------- | ------------------------------------------- | | **Best for** | Few, well-defined metrics with distinct types | Many related metrics queried together | Dynamic or unknown metrics with mixed types | | **Adding metrics** | Create a new table | `ALTER TABLE` | Insert rows with a new name | | **Query complexity** | Joins/unions across tables | Simple single-table queries | Filtering on `metric_name` | | **Schema changes** | Rare per table | Frequent as metrics grow | Rare | | **Compression** | Excellent (uniform columns) | Good (NULL-heavy columns compress well) | Good (uniform typed columns) | ## Which layout should you choose? [Section titled “Which layout should you choose?”](#which-layout-should-you-choose) * **You know all metrics up front and they rarely change** → wide. You get simple queries and good compression. * **You have a small, fixed set of distinct metric types** → narrow. Each table stays focused and compresses well. * **Metrics are dynamic, user-defined, or vary by tenant** → medium. You avoid constant schema changes. In practice, many systems combine layouts. For example, use a wide table for core host metrics and a medium table for custom application-level metrics. ## Learn more [Section titled “Learn more”](#learn-more) * [Primary keys, time columns, and uniqueness](/docs/learn/data-model/primary-keys-time-and-uniqueness): Partition column and constraint rules for hypertables. * [Schema optimization](/docs/learn/data-model/understand-database-schemas): Indexes, constraints, and other schema tools. * [Create and configure a hypertable](/docs/learn/hypertables/creating-and-configuring-hypertables): `CREATE TABLE` options including `segmentby` and `orderby`. * [Accelerate queries using indexes](/docs/build/performance-optimization/indexing): How-to guide for creating and tuning indexes. * [CREATE TABLE reference](/docs/reference/timescaledb/hypertables/create_table): Full API reference. # Deep dive overview > Advanced topics and architecture deep dives Explore advanced concepts and architectural details of Tiger Data. ## Topics [Section titled “Topics”](#topics) * Architecture whitepapers * Query planning and execution * Storage internals * Performance optimization deep dives ## Coming soon [Section titled “Coming soon”](#coming-soon) This section will include detailed technical content for users who want to understand Tiger Data at a deeper level. # Tiger Data architecture for real-time analytics > A whitepaper detailing the architectural choices and optimizations for real-time analytics that power TimescaleDB and Tiger Cloud Tiger Data has created a powerful application database for real-time analytics on time-series data. It integrates seamlessly with the PostgreSQL ecosystem and enhances it with automatic time-based partitioning, hybrid row-columnar storage, and vectorized execution, enabling high-ingest performance, sub-second queries, and full SQL support at scale. Tiger Cloud offers managed database services that provide a stable and reliable environment for your applications. Each service is based on a PostgreSQL database instance and the TimescaleDB extension. By making use of incrementally updated materialized views and advanced analytical functions, TimescaleDB reduces compute overhead and improves query efficiency. Developers can continue using familiar SQL workflows and tools, while benefiting from a database purpose-built for fast, scalable analytics. This document outlines the architectural choices and optimizations that power TimescaleDB and Tiger Cloud's performance and scalability while preserving PostgreSQL's reliability and transactional guarantees. Want to read this whitepaper from the comfort of your own computer? Download the PDF: [Tiger Data architecture for real-time analytics](https://assets.timescale.com/docs/downloads/tigerdata-whitepaper.pdf) ## Introduction [Section titled “Introduction”](#introduction) ### What is real-time analytics? [Section titled “What is real-time analytics?”](#what-is-real-time-analytics) Real-time analytics enables applications to process and query data as it is generated and as it accumulates, delivering immediate and ongoing insights for decision-making. Unlike traditional analytics, which relies on batch processing and delayed reporting, real-time analytics supports *both* instant queries on fresh data and fast exploration of historical trends, powering applications with sub-second query performance across vast, continuously growing datasets. Many modern applications depend on real-time analytics to drive critical functionality: * **IoT monitoring systems** track sensor data over time, identifying long-term performance patterns while still surfacing anomalies as they arise. This allows businesses to optimize maintenance schedules, reduce costs, and improve reliability. * **Financial and business intelligence platforms** analyze both current and historical data to detect trends, assess risk, and uncover opportunities, from tracking stock performance over a day, week, or year to identifying spending patterns across millions of transactions. * **Interactive customer dashboards** empower users to explore live and historical data in a seamless experience, whether it's a SaaS product providing real-time analytics on business operations, a media platform analyzing content engagement, or an e-commerce site surfacing personalized recommendations based on recent and past behavior. Real-time analytics isn't just about reacting to the latest data, although that is critically important. It's also about delivering fast, interactive, and scalable insights across all your data, enabling better decision-making and richer user experiences. Unlike traditional ad-hoc analytics used by analysts, real-time analytics powers applications, driving dynamic dashboards, automated decisions, and user-facing insights at scale. To achieve this, real-time analytics systems must meet several key requirements: * **Low-latency queries** ensure sub-second response times even under high load, enabling fast insights for dashboards, monitoring, and alerting. * **Low-latency ingest** minimizes the lag between when data is created and when it becomes available for analysis, ensuring fresh and accurate insights. * **Data mutability** allows for efficient updates, corrections, and backfills, ensuring analytics reflect the most accurate state of the data. * **Concurrency and scalability** enable systems to handle high query volumes and growing workloads without degradation in performance. * **Seamless access to both recent and historical data** ensures fast queries across time, whether analyzing live, streaming data, or running deep historical queries on days or months of information. * **Query flexibility** provides full SQL support, allowing for complex queries with joins, filters, aggregations, and analytical functions. ### Tiger Cloud: real-time analytics from PostgreSQL [Section titled “Tiger Cloud: real-time analytics from PostgreSQL”](#tiger-cloud-real-time-analytics-from-postgresql) Tiger Cloud is a high-performance database that brings real-time analytics to applications. It combines fast queries, high ingest performance, and full SQL support, all while ensuring scalability and reliability. Tiger Cloud extends PostgreSQL with the TimescaleDB extension. It enables sub-second queries on vast amounts of incoming data while providing optimizations designed for continuously updating datasets. Tiger Cloud achieves this through the following optimizations: * **Efficient data partitioning:** automatically and transparently partitioning data into chunks, ensuring fast queries, minimal indexing overhead, and seamless scalability * **Row-columnar storage:** providing the flexibility of a row store for transactions and the performance of a column store for analytics * **Optimized query execution:** using techniques like chunk and batch exclusion, columnar storage, and vectorized execution to minimize latency * **Continuous aggregates:** precomputing analytical results for fast insights without expensive reprocessing * **Cloud-native operation:** compute/compute separation, elastic usage-based storage, horizontal scale out, data tiering to object storage * **Operational simplicity:** offering high availability, connection pooling, and automated backups for reliable and scalable real-time applications With Tiger Cloud, developers can build low-latency, high-concurrency applications that seamlessly handle streaming data, historical queries, and real-time analytics while leveraging the familiarity and power of PostgreSQL. ## Data model [Section titled “Data model”](#data-model) Today's applications demand a database that can handle real-time analytics and transactional queries without sacrificing speed, flexibility, or SQL compatibility (including joins between tables). TimescaleDB achieves this with **hypertables**, which provide an automatic partitioning engine, and **hypercore**, a hybrid row-columnar storage engine designed to deliver high-performance queries and efficient compression (up to 95%) within PostgreSQL. ### Efficient data partitioning [Section titled “Efficient data partitioning”](#efficient-data-partitioning) TimescaleDB provides hypertables, a table abstraction that automatically partitions data into chunks in real time (using time stamps or incrementing IDs) to ensure fast queries and predictable performance as datasets grow. Unlike traditional relational databases that require manual partitioning, hypertables automate all aspects of partition management, keeping locking minimal even under high ingest load. At ingest time, hypertables ensure that PostgreSQL can deal with a constant stream of data without suffering from table bloat and index degradation by automatically partitioning data across time. Because each chunk is ordered by time and has its own indexes and storage, writes are usually isolated to small, recent chunks, keeping index sizes small, improving cache locality, and reducing the overhead of vacuum and background maintenance operations. This localized write pattern minimizes write amplification and ensures consistently high ingest performance, even as total data volume grows. At query time, hypertables efficiently exclude irrelevant chunks from the execution plan when the partitioning column is used in a `WHERE` clause. This architecture ensures fast query execution, avoiding the gradual slowdowns that affect non-partitioned tables as they accumulate millions of rows. Chunk-local indexes keep indexing overhead minimal, ensuring index operations scans remain efficient regardless of dataset size. ![Tiger Data hypertable architecture: a normal Postgres table is automatically partitioned by time into chunks based on the chunk\_time\_interval setting](/docs/_astro/hypertable-light.f-v6IDWg_Z1YDb49.webp) ![Tiger Data hypertable architecture: a normal Postgres table is automatically partitioned by time into chunks based on the chunk\_time\_interval setting](/docs/_astro/hypertable-dark.CNX8sC6Y_Z23WxHL.webp) Hypertables are the foundation for all of TimescaleDB's real-time analytics capabilities. They enable seamless data ingestion, high-throughput writes, optimized query execution, and chunk-based lifecycle management, including automated data retention (drop a chunk) and data tiering (move a chunk to object storage). ### Row-columnar storage [Section titled “Row-columnar storage”](#row-columnar-storage) Traditional databases force a trade-off between fast inserts (row-based storage) and efficient analytics (columnar storage). Hypercore eliminates this trade-off, allowing real-time analytics without sacrificing transactional capabilities. Hypercore dynamically stores data in the most efficient format for its lifecycle: * **Row-based storage for recent data**: the most recent chunk (and possibly more) is always stored in the rowstore, ensuring fast inserts, updates, and low-latency single record queries. Additionally, row-based storage is used as a writethrough for inserts and updates to columnar storage. * **Columnar storage for analytical performance**: chunks are automatically compressed into the columnstore, optimizing storage efficiency and accelerating analytical queries. Unlike traditional columnar databases, hypercore allows data to be inserted or modified at any stage, making it a flexible solution for both high-ingest transactional workloads and real-time analytics, within a single database. ### Columnar storage layout [Section titled “Columnar storage layout”](#columnar-storage-layout) TimescaleDB's columnar storage layout optimizes analytical query performance by structuring data efficiently on disk, reducing scan times, and maximizing compression rates. Unlike traditional row-based storage, where data is stored sequentially by row, columnar storage organizes and compresses data by column, allowing queries to retrieve only the necessary fields in batches rather than scanning entire rows. But unlike many column store implementations, TimescaleDB's columnstore supports full mutability, inserts, upserts, updates, and deletes, even at the individual record level, with transactional guarantees. Data is also immediately visible to queries as soon as it is written. ![Hypertable with hypercore enabled: incoming rows are stored in the rowstore and automatically columnized into the columnstore in batches of up to 1000 values per column](/docs/_astro/hypertable-with-hypercore-enabled-light.C2P2rNDf_7VCLT.webp) ![Hypertable with hypercore enabled: incoming rows are stored in the rowstore and automatically columnized into the columnstore in batches of up to 1000 values per column](/docs/_astro/hypertable-with-hypercore-enabled-dark.CSPYbMj5_hrCeU.webp) #### Columnar batches [Section titled “Columnar batches”](#columnar-batches) TimescaleDB uses columnar collocation and columnar compression within row-based storage to optimize analytical query performance while maintaining full PostgreSQL compatibility. This approach ensures efficient storage, high compression ratios, and rapid query execution. ![Tiger Data automatic conversion of a rowstore chunk into a columnstore chunk, where rows are grouped into batches and stored as compressed column arrays ordered by time](/docs/_astro/columnstore-light.CNoPisWd_Z2tlPDO.webp) ![Tiger Data automatic conversion of a rowstore chunk into a columnstore chunk, where rows are grouped into batches and stored as compressed column arrays ordered by time](/docs/_astro/columnstore-dark.DzSeelcs_ZjRRLM.webp) A rowstore chunk is converted to a columnstore chunk by successfully grouping together sets of rows (typically up to 1000) into a single batch, then converting the batch into columnar form. Each compressed batch does the following: * Encapsulates columnar data in compressed arrays of up to 1,000 values per column, stored as a single entry in the underlying compressed table * Uses a column-major format within the batch, enabling efficient scans by co-locating values of the same column and allowing the selection of individual columns without reading the entire batch * Applies advanced compression techniques at the column level, including run-length encoding, delta encoding, and Gorilla compression, to significantly reduce storage footprint (by up to 95%) and improve I/O performance. While the chunk interval of rowstore and columnstore batches usually remains the same, TimescaleDB can also combine columnstore batches so they use a different chunk interval. This architecture provides the benefits of columnar storage, optimized scans, reduced disk I/O, and improved analytical performance, while seamlessly integrating with PostgreSQL's row-based execution model. #### Segmenting and ordering data [Section titled “Segmenting and ordering data”](#segmenting-and-ordering-data) To optimize query performance, TimescaleDB allows explicit control over how data is physically organized within columnar storage. By structuring data effectively, queries can minimize disk reads and execute more efficiently, using vectorized execution for parallel batch processing where possible. ![Columnstore chunk organized with SEGMENTBY and ORDERBY: rows grouped into batches per segment key and ordered by time within each batch](/docs/_astro/columnstore-segmentby-light.Csu9SmJQ_2lxGlQ.webp) ![Columnstore chunk organized with SEGMENTBY and ORDERBY: rows grouped into batches per segment key and ordered by time within each batch](/docs/_astro/columnstore-segmentby-dark.Bxa3vqFc_ZldADl.webp) * **Group related data together to improve scan efficiency**: organizing rows into logical segments ensures that queries filtering by a specific value only scan relevant data sections. For example, in the above, querying for a specific ID is particularly fast. *(Implemented with `SEGMENTBY`.)* * **Sort data within segments to accelerate range queries**: defining a consistent order reduces the need for post-query sorting, making time-based queries and range scans more efficient. *(Implemented with `ORDERBY`.)* * **Reduce disk reads and maximize vectorized execution**: a well-structured storage layout enables efficient batch processing (Single Instruction, Multiple Data, or SIMD vectorization) and parallel execution, optimizing query performance. By combining segmentation and ordering, TimescaleDB ensures that columnar queries are not only fast but also resource-efficient, enabling high-performance real-time analytics. ### Data mutability [Section titled “Data mutability”](#data-mutability) Traditional databases force a trade-off between fast updates and efficient analytics. Fully immutable storage is impractical in real-world applications, where data needs to change. Asynchronous mutability, where updates only become visible after batch processing, introduces delays that break real-time workflows. In-place mutability, while theoretically ideal, is prohibitively slow in columnar storage, requiring costly decompression, segmentation, ordering, and recompression cycles. Hypercore navigates these trade-offs with a hybrid approach that enables immediate updates without modifying compressed columnstore data in place. By staging changes in an interim rowstore chunk, hypercore allows updates and deletes to happen efficiently while preserving the analytical performance of columnar storage. ![Data mutability in hypercore: inserts, updates, and deletes are routed to an interim rowstore chunk that periodically recompresses into the columnstore, supporting full mutability without rewriting compressed data](/docs/_astro/hypercore-data-mutability-light.CNUSPO4Z_Z1AvahI.webp) ![Data mutability in hypercore: inserts, updates, and deletes are routed to an interim rowstore chunk that periodically recompresses into the columnstore, supporting full mutability without rewriting compressed data](/docs/_astro/hypercore-data-mutability-dark.0BwC6tck_Z1u7eJM.webp) #### Real-time writes without delays [Section titled “Real-time writes without delays”](#real-time-writes-without-delays) All new data which is destined for a columnstore chunk is first written to an interim rowstore chunk, ensuring high-speed ingestion and immediate queryability. Unlike fully columnar systems that require ingestion to go through compression pipelines, hypercore allows fresh data to remain in a fast row-based structure before being later compressed into columnar format in ordered batches as normal. Queries transparently access both the rowstore and columnstore chunks, meaning applications always see the latest data instantly, regardless of its storage format. #### Efficient updates and deletes without performance penalties [Section titled “Efficient updates and deletes without performance penalties”](#efficient-updates-and-deletes-without-performance-penalties) When modifying or deleting existing data, hypercore avoids the inefficiencies of both asynchronous updates and in-place modifications. Instead of modifying compressed storage directly, affected batches are decompressed and staged in the interim rowstore chunk, where changes are applied immediately. These modified batches remain in row storage until they are recompressed and reintegrated into the columnstore (which happens automatically via a background process). This approach ensures updates are immediately visible, but without the expensive overhead of decompressing and rewriting entire chunks. This approach avoids: * The rigidity of immutable storage, which requires workarounds like versioning or copy-on-write strategies * The delays of asynchronous updates, where modified data is only visible after batch processing * The performance hit of in-place mutability, which makes compressed storage prohibitively slow for frequent updates * The restrictions some databases have on not altering the segmentation or ordering keys ## Query optimizations [Section titled “Query optimizations”](#query-optimizations) Real-time analytics isn't just about raw speed, it's about executing queries efficiently, reducing unnecessary work, and maximizing performance. TimescaleDB optimizes every step of the query lifecycle to ensure that queries scan only what's necessary, make use of data locality, and execute in parallel for sub-second response times over large datasets. ### Skip unnecessary data [Section titled “Skip unnecessary data”](#skip-unnecessary-data) TimescaleDB minimizes the amount of data a query touches, reducing I/O and improving execution speed: #### Primary partition exclusion (row and columnar) [Section titled “Primary partition exclusion (row and columnar)”](#primary-partition-exclusion-row-and-columnar) Queries automatically skip irrelevant partitions (chunks) based on the primary partitioning key (usually a timestamp), ensuring they only scan relevant data. ![Primary partition exclusion: skipping irrelevant chunks based on time range](/docs/_astro/constraint-exclusion-partitioning-column.BGshtrGU_r9H1p.webp) ![Primary partition exclusion: skipping irrelevant chunks based on time range](/docs/_astro/constraint-exclusion-partitioning-column-dark.B1uqGNmk_1t7K65.webp) #### Secondary partition exclusion (columnar) [Section titled “Secondary partition exclusion (columnar)”](#secondary-partition-exclusion-columnar) Min/max metadata allows queries filtering on correlated dimensions (e.g., `order_id` or secondary timestamps) to exclude chunks that don't contain relevant data. ![Secondary partition exclusion: using min/max metadata to skip chunks on non-partitioning columns](/docs/_astro/chunk-skipping-on-secondary-columns.D4xM0nlH_Z9fgfL.webp) ![Secondary partition exclusion: using min/max metadata to skip chunks on non-partitioning columns](/docs/_astro/chunk-skipping-on-secondary-columns-dark.wCjAq7k__2i0VfL.webp) #### PostgreSQL indexes (row and columnar) [Section titled “PostgreSQL indexes (row and columnar)”](#postgresql-indexes-row-and-columnar) Unlike many databases, TimescaleDB supports sparse indexes on columnstore data, allowing queries to efficiently locate specific values within both row-based and compressed columnar storage. These indexes enable fast lookups, range queries, and filtering operations that further reduce unnecessary data scans. ![PostgreSQL indexes on both rowstore and columnstore data](/docs/_astro/rowstore-indexes.CrJ-w7B2_1q1gzw.webp) ![PostgreSQL indexes on both rowstore and columnstore data](/docs/_astro/rowstore-indexes-dark.CgemkZSr_Z137WVo.webp) #### Batch-level filtering (columnar) [Section titled “Batch-level filtering (columnar)”](#batch-level-filtering-columnar) Within each chunk, compressed columnar batches are organized using `SEGMENTBY` keys and ordered by `ORDERBY` columns. Indexes and min/max metadata can be used to quickly exclude batches that don't match the query criteria. ![Batch-level filtering: excluding compressed batches using SEGMENTBY keys and min/max metadata](/docs/_astro/batch-skipping-indexes.DsnaICTH_ozWqv.webp) ![Batch-level filtering: excluding compressed batches using SEGMENTBY keys and min/max metadata](/docs/_astro/batch-skipping-indexes-dark.BO7PlFHC_Z7rFoN.webp) ### Maximize locality [Section titled “Maximize locality”](#maximize-locality) Organizing data for efficient access ensures queries are read in the most optimal order, reducing unnecessary random reads and reducing scans of unneeded data. ![Data locality: segmentation groups related data, ordering sorts within segments](/docs/_astro/segment-and-order.Br-W1hE-_1mjfaN.webp) ![Data locality: segmentation groups related data, ordering sorts within segments](/docs/_astro/segment-and-order-dark.KUX_Z6HW_Z2wcTzw.webp) * **Segmentation**: Columnar batches are grouped using `SEGMENTBY` to keep related data together, improving scan efficiency. * **Ordering**: Data within each batch is physically sorted using `ORDERBY`, increasing scan efficiency (and reducing I/O operations), enabling efficient range queries, and minimizing post-query sorting. * **Column selection**: Queries read only the necessary columns, reducing disk I/O, decompression overhead, and memory usage. ### Parallelize execution [Section titled “Parallelize execution”](#parallelize-execution) Once a query is scanning only the required columnar data in the optimal order, TimescaleDB is able to maximize performance through parallel execution. As well as using multiple workers, TimescaleDB accelerates columnstore query execution by using Single Instruction, Multiple Data (SIMD) vectorization, allowing modern CPUs to process multiple data points in parallel. ![SIMD vectorized execution: decompression, filtering, and aggregation in parallel](/docs/_astro/simd.mFJT4jP3_2q6wMG.webp) ![SIMD vectorized execution: decompression, filtering, and aggregation in parallel](/docs/_astro/simd-dark.BYdJhvdH_5Et8k.webp) The TimescaleDB implementation of SIMD vectorization currently allows: * **Vectorized decompression**, which efficiently restores compressed data into a usable form for analysis. * **Vectorized filtering**, which rapidly applies filter conditions across data sets. * **Vectorized aggregation**, which performs aggregate calculations, such as sum or average, across multiple data points concurrently. ## Accelerating queries with continuous aggregates [Section titled “Accelerating queries with continuous aggregates”](#accelerating-queries-with-continuous-aggregates) Aggregating large datasets in real time can be expensive, requiring repeated scans and calculations that strain CPU and I/O. While some databases attempt to brute-force these queries at runtime, compute and I/O are always finite resources, leading to high latency, unpredictable performance, and growing infrastructure costs as data volume increases. **Continuous aggregates**, the TimescaleDB implementation of incrementally updated materialized views, solve this by shifting computation from every query run to a single, asynchronous step after data is ingested. Only the time buckets that receive new or modified data are updated, and queries read precomputed results instead of scanning raw data, dramatically improving performance and efficiency. ![How a continuous aggregate incrementally materializes time-bucketed rollups from a raw hypertable, with a real-time aggregate query reading both pre-computed and recent raw data](/docs/_astro/continuous-aggregates-light.BhW3GFu2_pFsfL.webp) ![How a continuous aggregate incrementally materializes time-bucketed rollups from a raw hypertable, with a real-time aggregate query reading both pre-computed and recent raw data](/docs/_astro/continuous-aggregates-dark.CnXJ1DFs_2kSGiL.webp) When you know the types of queries you'll need ahead of time, continuous aggregates allow you to pre-aggregate data along meaningful time intervals, such as per-minute, hourly, or daily summaries, delivering instant results without on-the-fly computation. Continuous aggregates also avoid the time-consuming and error-prone process of maintaining manual rollups, while continuing to offer data mutability to support efficient updates, corrections, and backfills. Whenever new data is inserted or modified in chunks which have been materialized, TimescaleDB stores invalidation records reflecting that these results are stale and need to be recomputed. Then, an asynchronous process re-computes regions that include invalidated data, and updates the materialized results. TimescaleDB tracks the lineage and dependencies between continuous aggregates and their underlying data, to ensure the continuous aggregates are regularly kept up-to-date. This happens in a resource-efficient manner, and where multiple invalidations can be coalesced into a single refresh (as opposed to refreshing any dependencies at write time, such as via a trigger-based approach). Continuous aggregates themselves are stored in hypertables, and they can be converted to columnar storage for compression, and raw data can be dropped, reducing storage footprint and processing cost. Continuous aggregates also support hierarchical rollups (e.g., hourly to daily to monthly) and real-time mode, which merges precomputed results with the latest ingested data to ensure accurate, up-to-date analytics. This architecture enables scalable, low-latency analytics while keeping resource usage predictable, ideal for dashboards, monitoring systems, and any workload with known query patterns. ### Hyperfunctions for real-time analytics [Section titled “Hyperfunctions for real-time analytics”](#hyperfunctions-for-real-time-analytics) Real-time analytics requires more than basic SQL functions, efficient computation is essential as datasets grow in size and complexity. Hyperfunctions, available through the `timescaledb_toolkit` extension, provide high-performance, SQL-native functions tailored for time-series analysis. These include advanced tools for gap-filling, percentile estimation, time-weighted averages, counter correction, and state tracking, among others. A key innovation of hyperfunctions is their support for partial aggregation, which allows TimescaleDB to store intermediate computational states rather than just final results. These partials can later be merged to compute rollups efficiently, avoiding expensive reprocessing of raw data and reducing compute overhead. This is especially effective when combined with continuous aggregates. Consider a real-world example: monitoring request latencies across thousands of application instances. You might want to compute p95 latency per minute, then roll that up into hourly and daily percentiles for dashboards or alerts. With traditional SQL, calculating percentiles requires a full scan and sort of all underlying data, making multi-level rollups computationally expensive. With TimescaleDB, you can use the `percentile_agg` hyperfunction in a continuous aggregate to compute and store a partial aggregation state for each minute. This state efficiently summarizes the distribution of latencies for that time bucket, without storing or sorting all individual values. Later, to produce an hourly or daily percentile, you simply combine the stored partials, no need to reprocess the raw latency values. This approach provides a scalable, efficient solution for percentile-based analytics. By combining hyperfunctions with continuous aggregates, TimescaleDB enables real-time systems to deliver fast, resource-efficient insights across high-ingest, high-resolution datasets, without sacrificing accuracy or flexibility. ## Cloud-native architecture [Section titled “Cloud-native architecture”](#cloud-native-architecture) Real-time analytics requires a scalable, high-performance, and cost-efficient database that can handle high-ingest rates and low-latency queries without overprovisioning. Tiger Cloud is designed for elasticity, enabling independent scaling of storage and compute, workload isolation, and intelligent data tiering. ### Independent storage and compute scaling [Section titled “Independent storage and compute scaling”](#independent-storage-and-compute-scaling) Real-time applications generate continuous data streams while requiring instant querying of both fresh and historical data. Traditional databases force users to pre-provision fixed storage, leading to unnecessary costs or unexpected limits. Tiger Cloud eliminates this constraint by dynamically scaling storage based on actual usage: * Storage expands and contracts automatically as data is added or deleted, avoiding manual intervention. * Usage-based billing ensures costs align with actual storage consumption, eliminating large upfront allocations. * Compute can be scaled independently to optimize query execution, ensuring fast analytics across both recent and historical data. With this architecture, databases grow alongside data streams, enabling seamless access to real-time and historical insights while efficiently managing storage costs. ### Workload isolation for real-time performance [Section titled “Workload isolation for real-time performance”](#workload-isolation-for-real-time-performance) Balancing high-ingest rates and low-latency analytical queries on the same system can create contention, slowing down performance. Tiger Cloud mitigates this by allowing read and write workloads to scale independently: * The primary database efficiently handles both ingestion and real-time rollups without disruption. * Read replicas scale query performance separately, ensuring fast analytics even under heavy workloads. ![Horizontal read scaling in Tiger Cloud: writes go to a primary, while reads are distributed across read replicas through a single load-balanced endpoint](/docs/_astro/horizontal-read-scaling-in-tiger-cloud-light.FKvfNjnh_1Ew1eU.webp) ![Horizontal read scaling in Tiger Cloud: writes go to a primary, while reads are distributed across read replicas through a single load-balanced endpoint](/docs/_astro/horizontal-read-scaling-in-tiger-cloud-dark.CujqDL0m_ZAil5b.webp) This separation ensures that frequent queries on fresh data don't interfere with ingestion, making it easier to support live monitoring, anomaly detection, interactive dashboards, and alerting systems. ### Intelligent data tiering for cost-efficient real-time analytics [Section titled “Intelligent data tiering for cost-efficient real-time analytics”](#intelligent-data-tiering-for-cost-efficient-real-time-analytics) Not all real-time data is equally valuable, recent data is queried constantly, while older data is accessed less frequently. Tiger Cloud can be configured to automatically tier data to cheaper bottomless object storage, ensuring that hot data remains instantly accessible, while historical data is still available. ![Tiered storage in Tiger Cloud: a hypertable's rowstore and columnstore chunks live on high-performance block storage, while older chunks tier to low-cost bottomless object storage and remain transparently queryable](/docs/_astro/tiered-storage-light.DWY9mRTx_Z2qmhC0.webp) ![Tiered storage in Tiger Cloud: a hypertable's rowstore and columnstore chunks live on high-performance block storage, while older chunks tier to low-cost bottomless object storage and remain transparently queryable](/docs/_astro/tiered-storage-dark.D_8Dc-sh_1E6TXQ.webp) * **Recent, high-velocity data** stays in high-performance storage for ultra-fast queries. * **Older, less frequently accessed data** is automatically moved to cost-efficient object storage but remains queryable and available for building continuous aggregates. While many systems support this concept of data cooling, TimescaleDB ensures that the data can still be queried from the same hypertable regardless of its current location. For real-time analytics, this means applications can analyze live data streams without worrying about storage constraints, while still maintaining access to long-term trends when needed. ### Cloud-native database observability [Section titled “Cloud-native database observability”](#cloud-native-database-observability) Real-time analytics doesn't just require fast queries, it requires the ability to understand why queries are fast or slow, where resources are being used, and how performance changes over time. That's why Tiger Cloud is built with deep observability features, giving developers and operators full visibility into their database workloads. At the core of this observability is Insights, Tiger Cloud's built-in query monitoring tool. Insights captures per-query statistics from our whole fleet in real time, showing you exactly how your database is behaving under load. It tracks key metrics like execution time, planning time, number of rows read and returned, I/O usage, and buffer cache hit rates, not just for the database as a whole, but for each individual query. Insights lets you do the following: * Identify slow or resource-intensive queries instantly * Spot long-term performance regressions or trends * Understand query patterns and how they evolve over time * See the impact of schema changes, indexes, or continuous aggregates on workload performance * Monitor and compare different versions of the same query to optimize execution All this is surfaced through an intuitive interface, available directly in Tiger Cloud, with no instrumentation or external monitoring infrastructure required. Beyond query-level visibility, Tiger Cloud also exposes metrics around service resource consumption, compression, continuous aggregates, and data tiering, allowing you to track how data moves through the system, and how those background processes impact storage and query performance. Together, these observability features give you the insight and control needed to operate a real-time analytics database at scale, with confidence, clarity, and performance you can trust. ## Ensuring reliability and scalability [Section titled “Ensuring reliability and scalability”](#ensuring-reliability-and-scalability) Maintaining high availability, efficient resource utilization, and data durability is essential for real-time applications. Tiger Cloud provides robust operational features to ensure seamless performance under varying workloads. * **High-availability replicas**: deploy multi-AZ HA replicas to provide fault tolerance and ensure minimal downtime. In the event of a primary node failure, replicas are automatically promoted to maintain service continuity. * **Connection pooling**: optimize database connections by efficiently managing and reusing them, reducing overhead and improving performance for high-concurrency applications. * **Backup and recovery**: leverage continuous backups, Point-in-Time Recovery (PITR), and automated snapshotting to protect against data loss. Restore data efficiently to minimize downtime in case of failures or accidental deletions. These operational capabilities ensure Tiger Cloud remains reliable, scalable, and resilient, even under demanding real-time workloads. ## Conclusion [Section titled “Conclusion”](#conclusion) Real-time analytics is critical for modern applications, but traditional databases struggle to balance high-ingest performance, low-latency queries, and flexible data mutability. Tiger Cloud extends PostgreSQL to solve this challenge, combining automatic partitioning, hybrid row-columnar storage, and intelligent compression to optimize both transactional and analytical workloads. With continuous aggregates, hyperfunctions, and advanced query optimizations, Tiger Cloud ensures sub-second queries even on massive datasets that combine current and historic data. Its cloud-native architecture further enhances scalability with independent compute and storage scaling, workload isolation, and cost-efficient data tiering, allowing applications to handle real-time and historical queries seamlessly. For developers, this means building high-performance, real-time analytics applications without sacrificing SQL compatibility, transactional guarantees, or operational simplicity. Tiger Cloud delivers the best of PostgreSQL, optimized for real-time analytics. # Glossary > Technical terms and concepts for Tiger Data products, TimescaleDB, and real-time analytics This glossary defines technical terms, concepts, and terminology used in Tiger Data documentation, the database industry, and real-time analytics. Use the search and filters below to find terms. Search terms... ABCDEFGHIJKLMNOPQRSTUVWXYZ FilterAllTimescaleDBStorageTime-seriesCloudSecurityOperationsObservabilityAI & vectorsPostgreSQLData & migration A12 terms ### ACL (Access Control List) Security a table that tells a computer operating system which access rights each user has to a particular system object, such as a file directory or individual file. ### ACID PostgreSQL a set of properties (atomicity, consistency, isolation, durability) that guarantee database transactions are processed reliably. ### ACID compliance PostgreSQL a set of database properties, Atomicity, Consistency, Isolation, Durability, ensuring reliable and consistent transactions. Inherited from PostgreSQL. ### Adaptive query optimization PostgreSQL dynamic query plan adjustment based on actual execution statistics and data distribution patterns, improving performance over time. ### Aggregate (Continuous Aggregate) TimescaleDB a materialized, precomputed summary of query results over time-series data, providing faster access to analytics. ### Alerting Observability the process of automatically notifying administrators when predefined conditions or thresholds are met in system monitoring. ### Analytics database Data & migration a system optimized for large-scale analytical queries, supporting complex aggregations, time-based queries, and data exploration. ### Anomaly detection Time-series the identification of abnormal patterns or outliers within time-series datasets, common in observability, IoT, and finance. ### Append-only storage Storage a storage pattern where data is only added, never modified in place. Ideal for time-series workloads and audit trails. ### Archival Storage the process of moving old or infrequently accessed data to long-term, cost-effective storage solutions. ### Auto-partitioning TimescaleDB automatic division of a hypertable into chunks based on partitioning dimensions to optimize scalability and performance. ### Availability zone Cloud an isolated location within a cloud region that provides redundant power, networking, and connectivity. B10 terms ### B-tree PostgreSQL a self-balancing tree data structure that maintains sorted data and allows searches, sequential access, insertions, and deletions in logarithmic time. ### Background job Operations an automated task that runs in the background without user intervention, typically for maintenance operations like compression or data retention. ### Background worker PostgreSQL a PostgreSQL process that runs background tasks independently of client sessions. ### Batch processing Data & migration handling data in grouped batches rather than as individual real-time events, often used for historical data processing. ### Backfill Data & migration the process of filling in historical data that was missing or needs to be recalculated, often used during migrations or after schema changes. ### Backup Operations a copy of data stored separately from the original data to protect against data loss, corruption, or system failure. ### Bloom filter TimescaleDB a probabilistic data structure that tests set membership with possible false positives but no false negatives. TimescaleDB uses blocked bloom filters to speed up point lookups by eliminating chunks that don’t contain queried values. ### Buffer pool PostgreSQL memory area where frequently accessed data pages are cached to reduce disk I/O operations. ### BRIN (Block Range Index) PostgreSQL a PostgreSQL index type that stores summaries about ranges of table blocks, useful for large tables with naturally ordered data. ### Bytea PostgreSQL a PostgreSQL data type for storing binary data as a sequence of bytes. C25 terms ### Cache hit ratio Operations the percentage of data requests served from memory cache rather than disk, indicating query performance efficiency. ### Cardinality PostgreSQL the number of unique values in a dataset or database column. ### Check constraint PostgreSQL a database constraint that limits the values that can be stored in a column by checking them against a specified condition. ### Chunk TimescaleDB a horizontal partition of a hypertable that contains data for a specific time interval and space partition. ### Chunk interval TimescaleDB the time period covered by each chunk in a hypertable, which affects query performance and storage efficiency. ### Chunk skipping TimescaleDB a query optimization technique that skips chunks not relevant to the query’s time range, dramatically improving performance. ### CIDR (Classless Inter-Domain Routing) Security a method for allocating IP addresses and routing IP packets. ### Client credentials Security authentication tokens used by applications to access services programmatically without user interaction. ### Close Time-series in financial data, the closing price of a security at the end of a trading period. ### Cloud Cloud computing services delivered over the internet, including servers, storage, databases, networking, software, analytics, and intelligence. ### Cloud deployment Cloud the use of public, private, or hybrid cloud infrastructure to host TimescaleDB, enabling elastic scalability and managed services. ### Cloud-native Cloud an approach to building applications that leverage cloud infrastructure, scalability, and services like Kubernetes. ### Cold storage Storage a tier of data storage for infrequently accessed data that offers lower costs but higher access times. ### Columnar Storage a data storage format that stores data column by column rather than row by row, optimizing for analytical queries. ### Columnstore TimescaleDB TimescaleDB‘s columnar storage engine optimized for analytical workloads and compression. ### Compression Storage the process of reducing data size by encoding information using fewer bits, improving storage efficiency and query performance. ### Connection pooling Operations a technique for managing multiple database connections efficiently, reducing overhead for high-concurrency environments. ### Consensus algorithm Operations protocols ensuring distributed systems agree on data state, critical for multi-node database deployments. ### Compression policy TimescaleDB an automated rule that compresses hypertable chunks after they reach a specified age or size threshold. ### Compression ratio Storage the ratio between the original data size and the compressed data size, indicating compression effectiveness. ### Constraint PostgreSQL a rule enforced by the database to maintain data integrity and consistency. ### Continuous aggregate TimescaleDB a materialized view that incrementally updates with new data, providing fast access to pre-computed aggregations. ### Counter aggregation Time-series aggregating monotonic counter data, handling counter resets and extrapolation. ### Cron Operations a time-based job scheduler in Unix-like computer operating systems. ### Cross-region backup Operations a backup stored in a different geographical region from the primary data for disaster recovery. D19 terms ### Data lake Data & migration a centralized repository storing structured and unstructured data at scale, often integrated with time-series databases for analytics. ### Data lineage Data & migration the tracking of data flow from source to destination, including transformations, essential for compliance and debugging. ### Data pipeline Data & migration automated workflows for moving, transforming, and loading data between systems, often using tools like Apache Kafka or Apache Airflow. ### Data migration Data & migration the process of moving data from one system, storage type, or format to another. ### Data retention Operations the practice of storing data for a specified period before deletion, often governed by compliance requirements or storage optimization. ### Data rollup Time-series the process of summarizing detailed historical data into higher-level aggregates, balancing storage needs with query efficiency. ### Data skew Data & migration uneven distribution of data across partitions or nodes, potentially causing performance bottlenecks. ### Data tiering Storage a storage management strategy that places data on different storage tiers based on access patterns and performance requirements. ### Data type PostgreSQL a classification that specifies which type of value a variable can hold, such as integer, string, or boolean. ### Decompress Storage the process of restoring compressed data to its original, uncompressed state. ### Delta Time-series the difference between two values, commonly used in counter aggregations to calculate the change over time. ### DHCP (Dynamic Host Configuration Protocol) Security a network management protocol used to automatically assign IP addresses and other network configuration parameters. ### Dimension TimescaleDB a partitioning key in a hypertable that determines how data is distributed across chunks. ### Disaster recovery Operations the process and procedures for recovering and protecting a business’s IT infrastructure in the event of a disaster. ### Double precision PostgreSQL a floating-point data type that provides more precision than the standard float type. ### Downsample Time-series the process of reducing the temporal resolution of time-series data by aggregating data points over longer time intervals. ### Downtime Operations the period during which a system, service, or application is unavailable or not operational. ### Dual-write and backfill Data & migration a migration approach where new data is written to both the source and target databases simultaneously, followed by backfilling historical data to ensure completeness. ### Dual-write Data & migration a migration pattern where applications write data to both the source and target systems simultaneously. E11 terms ### Edge computing Cloud processing data at or near the data source such as IoT devices, rather than solely in centralized servers, reducing latency. ### Edge gateway Cloud a device that aggregates data from sensors and performs preprocessing before sending data to cloud or centralized databases. ### ELT (Extract, Load, Transform) Data & migration a data pipeline pattern where raw data is loaded first, then transformed within the target system, leveraging database processing power. ### Embedding AI & vectors a vector representation of data such as text or images, that captures semantic meaning in a high-dimensional space. ### Error rate Observability the percentage of requests or operations that result in errors over a given time period. ### Euclidean distance AI & vectors a measure of the straight-line distance between two points in multidimensional space. ### Exactly-once Data & migration a message is delivered and processed precisely once. There is no loss and no duplicates. ### Explain PostgreSQL a PostgreSQL command that shows the execution plan for a query, useful for performance analysis. ### Event sourcing Time-series an architectural pattern storing all changes as a sequence of events, naturally fitting time-series database capabilities. ### Event-driven architecture Data & migration a design pattern where components react to events such as sensor readings, requiring real-time data pipelines and storage. ### Extension PostgreSQL a PostgreSQL add-on that extends the database’s functionality beyond the core features. F7 terms ### Fact table PostgreSQL the central table in a star schema containing quantitative measures, often time-series data with foreign keys to dimension tables. ### Failover Operations the automatic switching to a backup system, server, or network upon the failure or abnormal termination of the primary system. ### Financial time-series Time-series high-volume, timestamped datasets like stock market feeds or trade logs, requiring low-latency, scalable databases like TimescaleDB. ### Foreign key PostgreSQL a database constraint that establishes a link between data in two tables by referencing the primary key of another table. ### Fork Cloud a copy of a database service that shares the same data but can diverge independently through separate writes. ### Free service Cloud a free instance of Tiger Cloud with limited resources. You can create up to two free services under any pricing plan. When a free service reaches the resource limit, it converts to the read-only state. You can convert a free service to a standard one under paid pricing plans. ### FTP (File Transfer Protocol) Security a standard network protocol used for transferring files between a client and server on a computer network. G6 terms ### Gap filling Time-series a technique for handling missing data points in time-series by interpolation or other methods, often implemented with hyperfunctions. ### GIN (Generalized Inverted Index) PostgreSQL a PostgreSQL index type designed for indexing composite values and supporting fast searches. ### GiST (Generalized Search Tree) PostgreSQL a PostgreSQL index type that provides a framework for implementing custom index types. ### GP-LTTB Time-series an advanced downsampling algorithm that extends Largest-Triangle-Three-Buckets with Gaussian Process modeling. ### GUC (Grand Unified Configuration) PostgreSQL PostgreSQL‘s configuration parameter system that controls various aspects of database behavior. ### GUID (Globally Unique Identifier) PostgreSQL a unique identifier used in software applications, typically represented as a 128-bit value. H15 terms ### Hash PostgreSQL an index type that provides constant-time lookups for equality comparisons but doesn’t support range queries. ### High-cardinality Time-series refers to datasets with a large number of unique values, which can strain storage and indexing in time-series applications. ### Histogram bucket Observability a predefined range of metrics organized for statistical analysis, commonly visualized in monitoring tools. ### Hot standby Operations a replication configuration where the standby server can serve read-only queries while staying synchronized with the primary. ### High availability Operations a system design that ensures an agreed level of operational performance, usually uptime, for a higher than normal period. ### High Time-series in financial data, the highest price of a security during a specific time period. ### Histogram Observability a graphical representation of the distribution of numerical data, showing the frequency of data points in different ranges. ### Historical data Time-series previously recorded data that provides context and trends for analysis and decision-making. ### HNSW (Hierarchical Navigable Small World) AI & vectors a graph-based algorithm for approximate nearest neighbor search in high-dimensional spaces. ### Hot storage Storage a tier of data storage for frequently accessed data that provides the fastest access times but at higher cost. ### Hypercore TimescaleDB TimescaleDB‘s hybrid storage engine that seamlessly combines row and column storage for optimal performance. ### Hyperfunction TimescaleDB an SQL function in TimescaleDB designed for time-series analysis, statistics, and specialized computations. ### HyperLogLog PostgreSQL a probabilistic data structure used for estimating the cardinality of large datasets with minimal memory usage. ### Hypershift Data & migration a migration tool and strategy for moving data to TimescaleDB with minimal downtime. ### Hypertable TimescaleDB TimescaleDB‘s core abstraction that automatically partitions time-series data for scalability. I10 terms ### Idempotency Data & migration the property where repeated operations produce the same result, crucial for reliable data ingestion and processing. ### Ingest rate Observability the speed at which new data is written to the system, measured in rows per second. Critical for IoT and observability. ### Inner product AI & vectors a mathematical operation that combines two vectors to produce a scalar, used in similarity calculations. ### Insert PostgreSQL an SQL operation that adds new rows of data to a database table. ### Integer PostgreSQL a data type that represents whole numbers without decimal points. ### Intercept Time-series a statistical measure representing the y-intercept in linear regression analysis. ### Internet gateway Cloud an AWS VPC component that enables communication between instances in a VPC and the internet. ### Interpolation Time-series a method of estimating unknown values that fall between known data points. ### IP allow list Security a security feature that restricts access to specified IP addresses or ranges. ### Isolation level PostgreSQL a database transaction property that defines the degree to which operations in one transaction are isolated from those in other concurrent transactions. J6 terms ### Job Operations an automated task scheduled to run at specific intervals or triggered by certain conditions. ### Job execution Operations the process of running scheduled background tasks or automated procedures. ### JIT (Just-In-Time) compilation PostgreSQL PostgreSQL feature that compiles frequently executed query parts for improved performance, available in TimescaleDB. ### Job history Operations a record of past job executions, including their status, duration, and any errors encountered. ### JSON (JavaScript Object Notation) PostgreSQL a lightweight data interchange format that is easy for humans to read and write. ### JWT (JSON Web Token) Security a compact, URL-safe means of representing claims to be transferred between two parties. L12 terms ### Latency Observability the time delay between a request being made and the response being received. ### Lifecycle policy Operations a set of rules that automatically manage data throughout its lifecycle, including retention and deletion. ### Live migration Data & migration a data migration technique that moves data with minimal or zero downtime. ### Load balancer Operations a service distributing traffic across servers or database nodes to optimize resource use and avoid single points of failure. ### Log-Structured Merge (LSM) Tree PostgreSQL a data structure optimized for write-heavy workloads, though TimescaleDB primarily uses B-tree indexes for balanced read/write performance. ### LlamaIndex AI & vectors a framework for building applications with large language models, providing tools for data ingestion and querying. ### LOCF (Last Observation Carried Forward) Time-series a method for handling missing data by using the most recent known value. ### Logical backup Operations a backup method that exports data in a human-readable format, allowing for selective restoration. ### Logical replication Operations a PostgreSQL feature that replicates data changes at the logical level rather than the physical level. ### Logging Observability the process of recording events, errors, and system activities for monitoring and troubleshooting purposes. ### Low Time-series in financial data, the lowest price of a security during a specific time period. ### LTTB (Largest-Triangle-Three-Buckets) Time-series a downsampling algorithm that preserves the visual characteristics of time-series data. M12 terms ### Manhattan distance AI & vectors a distance metric calculated as the sum of the absolute differences of their coordinates. ### Manual compression Storage the process of compressing chunks manually rather than through automated policies. ### Materialization PostgreSQL the process of computing and storing the results of a query or view for faster access. ### Materialized view PostgreSQL a database object that stores the result of a query and can be refreshed periodically. ### Memory-optimized query PostgreSQL a query pattern designed to minimize disk I/O by leveraging available RAM and efficient data structures. ### Metric Observability a quantitative measurement used to assess system performance, business outcomes, or operational efficiency. ### MFA (Multi-Factor Authentication) Security a security method that requires two or more verification factors to grant access. ### Migration Data & migration the process of moving data, applications, or systems from one environment to another. ### Monitoring Observability the continuous observation and measurement of system performance and health. ### Multi-tenancy Cloud an architecture pattern supporting multiple customers or applications within a single database instance, with proper isolation. ### MQTT (Message Queuing Telemetry Transport) Data & migration a lightweight messaging protocol designed for small sensors and mobile devices. ### MST (Managed Service for TimescaleDB) Cloud a fully managed TimescaleDB service that handles infrastructure and maintenance tasks. N5 terms ### NAT Gateway Cloud a network address translation service that enables instances in a private subnet to connect to the internet. ### Node (database node) Operations an individual server within a distributed system, contributing to storage, compute, or replication tasks. ### Normalization PostgreSQL database design technique organizing data to reduce redundancy, though time-series data often benefits from denormalized structures. ### Not null PostgreSQL a database constraint that ensures a column cannot contain empty values. ### Numeric PostgreSQL a PostgreSQL data type for storing exact numeric values with user-defined precision. O9 terms ### OAuth Security an open standard for access delegation commonly used for token-based authentication and authorization. ### Observability Observability the ability to measure the internal states of a system by examining its outputs. ### OLAP (Online Analytical Processing) Data & migration systems or workloads focused on large-scale, multidimensional, and complex analytical queries. ### OLTP (Online Transaction Processing) Data & migration high-speed transactional systems optimized for data inserts, updates, and short queries. ### OHLC Time-series an acronym for Open, High, Low, Close prices, commonly used in financial data analysis. ### OHLCV Time-series an extension of OHLC that includes Volume data for complete candlestick analysis. ### Open Time-series in financial data, the opening price of a security at the beginning of a trading period. ### OpenTelemetry Observability open standard for collecting, processing, and exporting telemetry data, often stored in time-series databases. ### Optimization PostgreSQL the process of making systems, queries, or operations more efficient and performant. P20 terms ### Parallel copy Data & migration a technique for copying large amounts of data using multiple concurrent processes to improve performance. ### Parallel Query Execution PostgreSQL a PostgreSQL feature that uses multiple CPU cores to execute single queries faster, inherited by TimescaleDB. ### Partitioning PostgreSQL the practice of dividing large tables into smaller, more manageable pieces based on certain criteria. ### Percentile Time-series a statistical measure that indicates the value below which a certain percentage of observations fall. ### Performance Observability a measure of how efficiently a system operates, often quantified by metrics like throughput, latency, and resource utilization. ### pg\\\_basebackup Operations a PostgreSQL utility for taking base backups of a running PostgreSQL cluster. ### pg\\\_dump Operations a PostgreSQL utility for backing up database objects and data in various formats. ### pg\\\_restore Operations a PostgreSQL utility for restoring databases from backup files created by \`pg\_dump\`. ### pgVector AI & vectors a PostgreSQL extension that adds vector similarity search capabilities for AI and machine learning applications. ### pgai on Tiger Cloud AI & vectors a cloud solution for building search, RAG, and AI agents with PostgreSQL. Enables calling AI embedding and generation models directly from the database using SQL. ### pgvectorscale AI & vectors a performance enhancement for pgvector featuring StreamingDiskANN indexing, binary quantization compression, and label-based filtering. ### pgvectorizer AI & vectors a TimescaleDB tool for automatically vectorizing and indexing data for similarity search. ### Physical backup Operations a backup method that copies the actual database files at the storage level. ### PITR (Point-in-Time Recovery) Operations the ability to restore a database to a specific moment in time. ### Policy TimescaleDB an automated rule or procedure that performs maintenance tasks like compression, retention, or refresh operations. ### Predictive maintenance Time-series the use of time-series data to forecast equipment failure, common in IoT and industrial applications. ### PostgreSQL PostgreSQL an open-source object-relational database system known for its reliability, robustness, and performance. ### PostGIS PostgreSQL a PostgreSQL extension that adds support for geographic objects and spatial queries. ### Primary key PostgreSQL a database constraint that uniquely identifies each row in a table. ### psql PostgreSQL an interactive terminal-based front-end to PostgreSQL that allows users to type queries interactively. Q5 terms ### QPS (Queries Per Second) Observability a measure of database performance indicating how many queries a database can process per second. ### Query PostgreSQL a request for data or information from a database, typically written in SQL. ### Query performance PostgreSQL a measure of how efficiently database queries execute, including factors like execution time and resource usage. ### Query planner/optimizer PostgreSQL a component determining the most efficient strategy for executing SQL queries based on database structure and indexes. ### Query planning PostgreSQL the database process of determining the most efficient way to execute a query. R24 terms ### RBAC (Role-Based Access Control) Security a security model that assigns permissions to users based on their roles within an organization. ### Read committed PostgreSQL an isolation level where transactions can read committed changes made by other transactions. ### Read scaling Operations a technique for improving database performance by distributing read queries across multiple database replicas. ### Read uncommitted PostgreSQL the lowest isolation level where transactions can read uncommitted changes from other transactions. ### Read-only role Security a database role with permissions limited to reading data without modification capabilities. ### Read replica Operations a copy of the primary database that serves read-only queries, improving read scalability and geographic distribution. ### Real-time analytics Time-series the immediate analysis of incoming data streams, crucial for observability, trading platforms, and IoT monitoring. ### Real PostgreSQL a PostgreSQL data type for storing single-precision floating-point numbers. ### Real-time aggregate TimescaleDB a continuous aggregate that includes both materialized historical data and real-time calculations on recent data. ### Reciprocal Rank Fusion (RRF) AI & vectors a technique for combining ranked results from multiple search methods into a single list. Instead of comparing raw scores across different systems, RRF uses rank position: each result receives a score of 1 / (k + rank), where k is a smoothing constant (typically 60). Scores are summed across all search methods and sorted by total. Commonly used in hybrid search to fuse keyword (BM25) and vector similarity results. ### Refresh policy TimescaleDB an automated rule that determines when and how continuous aggregates are updated with new data. ### Region Cloud a geographical area containing multiple data centers, used in cloud computing for data locality and compliance. ### Repeatable read PostgreSQL an isolation level that ensures a transaction sees a consistent snapshot of data throughout its execution. ### Replica Operations a copy of a database that can be used for read scaling, backup, or disaster recovery purposes. ### Replication Operations the process of copying and maintaining data across multiple database instances to ensure availability and durability. ### Response time Observability the time it takes for a system to respond to a request, measured from request initiation to response completion. ### REST API Data & migration a web service architecture that uses HTTP methods to enable communication between applications. ### Restore Operations the process of recovering data from backups to restore a database to a previous state. ### Restore point Operations a snapshot of database state that can be used as a reference point for recovery operations. ### Retention policy TimescaleDB an automated rule that determines how long data is kept before being deleted from the system. ### Route table Cloud a set of rules that determine where network traffic is directed within a cloud network. ### RTO (Recovery Time Objective) Operations the maximum acceptable time that systems can be down after a failure or disaster. ### RPO (Recovery Point Objective) Operations the maximum acceptable amount of data loss measured in time after a failure or disaster. ### Rowstore Storage traditional row-oriented data storage where data is stored row by row, optimized for transactional workloads. S27 terms ### SAML (Security Assertion Markup Language) Security an XML-based standard for exchanging authentication and authorization data between security domains. ### Scheduled job Operations an automated task that runs at predetermined times or intervals. ### Schema evolution PostgreSQL the process of modifying database structure over time while maintaining compatibility with existing applications. ### Schema PostgreSQL the structure of a database, including tables, columns, relationships, and constraints. ### Security group Cloud a virtual firewall that controls inbound and outbound traffic for cloud resources. ### Service discovery Cloud mechanisms allowing applications to dynamically locate services like database endpoints, often used in distributed environments. ### Segmentwise recompression TimescaleDB a TimescaleDB compression technique that recompresses data segments to improve compression ratios. ### Serializable PostgreSQL the highest isolation level that ensures transactions appear to run serially even when executed concurrently. ### Sharding Data & migration horizontal partitioning of data across multiple database instances, distributing load and enabling linear scalability. ### SFTP (SSH File Transfer Protocol) Security a secure version of FTP that encrypts both commands and data during transmission. ### SkipScan TimescaleDB query optimization for DISTINCT operations that incrementally jumps between ordered values without reading intermediate rows. Uses a Custom Scan node to efficiently traverse ordered indexes, dramatically improving performance over traditional DISTINCT queries. ### Similarity search AI & vectors a technique for finding items that are similar to a given query item, often used with vector embeddings. ### SLA (Service Level Agreement) Operations a contract that defines the expected level of service between a provider and customer. ### SLI (Service Level Indicator) Observability a quantitative measure of some aspect of service quality. ### SLO (Service Level Objective) Observability a target value or range for service quality measured by an SLI. ### Slope Time-series a statistical measure representing the rate of change in linear regression analysis. ### SMTP (Simple Mail Transfer Protocol) Security an internet standard for email transmission across networks. ### Snapshot Operations a point-in-time copy of data that can be used for backup and recovery purposes. ### SP-GiST (Space-Partitioned Generalized Search Tree) PostgreSQL a PostgreSQL index type for data structures that naturally partition search spaces. ### Storage optimization Storage techniques for reducing storage costs and improving performance through compression, tiering, and efficient data organization. ### Streaming data Time-series continuous flows of data generated by devices, logs, or sensors, requiring high-ingest, real-time storage solutions. ### SQL (Structured Query Language) PostgreSQL a programming language designed for managing and querying relational databases. ### SSH (Secure Shell) Security a cryptographic network protocol for secure communication over an unsecured network. ### SSL (Secure Sockets Layer) Security a security protocol that establishes encrypted links between networked computers. ### Standard service Cloud a regular Tiger Cloud service that includes the resources and features according to the pricing plan. You can create standard services under any of the paid plans. ### Streaming replication Operations a PostgreSQL replication method that continuously sends write-ahead log records to standby servers. ### Synthetic monitoring Observability simulated transactions or probes used to test system health, generating time-series metrics for performance analysis. T25 terms ### Tiger Cloud service Cloud an optimized PostgreSQL instance extended with database engine innovations such as TimescaleDB, in a cloud infrastructure that delivers speed without sacrifice. ### Table PostgreSQL a database object that stores data in rows and columns, similar to a spreadsheet. ### Tablespace PostgreSQL a PostgreSQL storage structure that defines where database objects are physically stored on disk. ### TCP (Transmission Control Protocol) Security a connection-oriented protocol that ensures reliable data transmission between applications. ### TDigest Time-series a probabilistic data structure for accurate estimation of percentiles in distributed systems. ### Telemetry Observability the collection of real-time data from systems or devices for monitoring and analysis. ### Text PostgreSQL a PostgreSQL data type for storing variable-length character strings. ### Throughput Observability a measure of system performance indicating the amount of work performed or data processed per unit of time. ### Tiered storage Storage a storage strategy that automatically moves data between different storage classes based on access patterns and age. ### Tiger Cloud Cloud Tiger Data ‘s managed cloud platform that provides TimescaleDB as a fully managed solution with additional features. ### TigerLake Cloud Tiger Data ‘s service for integrating operational databases with data lake architectures. ### Tiger Cloud service Cloud an optimized PostgreSQL instance extended with database engine innovations such as TimescaleDB, in a cloud infrastructure that delivers speed without sacrifice. ### Time series Time-series data points indexed and ordered by time, typically representing how values change over time. ### Time-weighted average Time-series a statistical calculation that gives more weight to values based on the duration they were held. ### Time bucketing Time-series grouping timestamps into uniform intervals for analysis, commonly used with hyperfunctions. ### Time-series forecasting Time-series the application of statistical models to time-series data to predict future trends or events. ### TimescaleDB TimescaleDB an open-source PostgreSQL extension for real-time analytics that provides scalability and performance optimizations. ### Timestamp PostgreSQL a data type that stores date and time information without timezone data. ### Timestamptz PostgreSQL a PostgreSQL data type that stores timestamp with timezone information. ### TLS (Transport Layer Security) Security a cryptographic protocol that provides security for communication over networks. ### Tombstone Storage marker indicating deleted data in append-only systems, requiring periodic cleanup processes. ### Transaction isolation PostgreSQL the database property controlling the visibility of uncommitted changes between concurrent transactions. ### TPS (Transactions Per Second) Observability a measure of database performance indicating transaction processing capacity. ### Transaction PostgreSQL a unit of work performed against a database that must be completed entirely or not at all. ### Trigger PostgreSQL a database procedure that automatically executes in response to certain events on a table or view. U5 terms ### UDP (User Datagram Protocol) Security a connectionless communication protocol that provides fast but unreliable data transmission. ### Unique PostgreSQL a database constraint that ensures all values in a column or combination of columns are distinct. ### Uptime Observability the amount of time that a system has been operational and available for use. ### Usage-based storage Cloud a billing model where storage costs are based on actual data stored rather than provisioned capacity. ### UUID (Universally Unique Identifier) PostgreSQL a 128-bit identifier used to uniquely identify information without central coordination. V8 terms ### Vacuum PostgreSQL a PostgreSQL maintenance operation that reclaims storage and updates database statistics. ### Varchar PostgreSQL a variable-length character data type that can store strings up to a specified maximum length. ### Vector operations AI & vectors SIMD (Single Instruction, Multiple Data) optimizations for processing arrays of data, improving analytical query performance. ### Vertical scaling (scale up) Operations increasing system capacity by adding more power (CPU, RAM) to existing machines, as opposed to horizontal scaling. ### Visualization tool Observability a platform or dashboard used to display time-series data in charts, graphs, and alerts for easier monitoring and analysis. ### Vector AI & vectors a mathematical object with magnitude and direction, used in machine learning for representing data as numerical arrays. ### VPC (Virtual Private Cloud) Cloud a virtual network dedicated to your cloud account that provides network isolation. ### VWAP (Volume Weighted Average Price) Time-series a financial indicator that shows the average price weighted by volume over a specific time period. W6 terms ### WAL (Write-Ahead Log) PostgreSQL PostgreSQL‘s method for ensuring data integrity by writing changes to a log before applying them to data files. ### Warm storage Storage a storage tier that balances access speed and cost, suitable for data accessed occasionally. ### Watermark TimescaleDB a timestamp that tracks the progress of continuous aggregate materialization. ### WebSocket Data & migration a communication protocol that provides full-duplex communication channels over a single TCP connection. ### Window function PostgreSQL an SQL function that performs calculations across related rows, particularly useful for time-series analytics and trend analysis. ### Workload management Operations techniques for prioritizing and scheduling different types of database operations to optimize overall system performance. X1 term ### XML (eXtensible Markup Language) PostgreSQL a markup language that defines rules for encoding documents in a format that is both human-readable and machine-readable. Y1 term ### YAML (YAML Ain’t Markup Language) PostgreSQL a human-readable data serialization standard commonly used for configuration files. Z2 terms ### Zero downtime Operations a system design goal where services remain available during maintenance, upgrades, or migrations without interruption. ### Zero-downtime migration Data & migration migration strategies that maintain service availability throughout the transition process, often using techniques like dual-write and gradual cutover. 283 terms10 categories # About TimescaleDB hyperfunctions > Use hyperfunctions for enhanced real-time analytics TimescaleDB hyperfunctions package common time-series analytics patterns (percentiles, counters, sessionization, approximate distincts) as composable SQL functions instead of hand-rolled window logic. They help across IoT, IT telemetry, product analytics, finance, and many other event-heavy domains. Hyperfunctions aim to keep those queries fast enough to feel real time. Tiger Cloud includes all hyperfunctions by default, while self-hosted TimescaleDB includes a subset of them. For additional hyperfunctions, install the [TimescaleDB Toolkit](/docs/deploy/self-hosted/tooling/install-toolkit) PostgreSQL extension. ## Available hyperfunctions [Section titled “Available hyperfunctions”](#available-hyperfunctions) Here is a list of all the hyperfunctions provided by TimescaleDB. Hyperfunctions with a tick in the `Toolkit` column require an installation of TimescaleDB Toolkit for self-hosted deployments. Hyperfunctions with a tick in the `Experimental` column are still under development. Warning Experimental features could have bugs. They might not be backwards compatible, and could be removed in future releases. Use these features at your own risk, and do not use any experimental features in production. Tips When you upgrade the `timescaledb` extension, the experimental schema is removed by default. To use experimental features after an upgrade, you need to add the experimental schema again. ### Approximate count distinct functions [Section titled “Approximate count distinct functions”](#approximate-count-distinct-functions) | Type | Name | Toolkit | Experimental | | ------------------- | ----------------------------------------------------------------------------------------------------- | ------- | ------------ | | Aggregate | [`hyperloglog()`](/docs/reference/toolkit/approximate-count-distinct/hyperloglog) | ✅ | | | Alternate aggregate | [`approx_count_distinct()`](/docs/reference/toolkit/approximate-count-distinct/approx_count_distinct) | ✅ | | | Accessor | [`distinct_count()`](/docs/reference/toolkit/approximate-count-distinct/distinct_count) | ✅ | | | | [`stderror()`](/docs/reference/toolkit/approximate-count-distinct/stderror) | ✅ | | | Rollup | [`rollup()`](/docs/reference/toolkit/approximate-count-distinct/rollup) | ✅ | | ### Counters and gauges functions [Section titled “Counters and gauges functions”](#counters-and-gauges-functions) **`counter_agg()` functions** | Type | Name | Toolkit | Experimental | | --------- | ---------------------------------------------------------------------------------------------------- | ------- | ------------ | | Aggregate | [`counter_agg()`](/docs/reference/toolkit/counters-and-gauges/counter_agg/counter_agg) | ✅ | | | Accessor | [`corr()`](/docs/reference/toolkit/counters-and-gauges/counter_agg/corr) | ✅ | | | | [`counter_zero_time()`](/docs/reference/toolkit/counters-and-gauges/counter_agg/counter_zero_time) | ✅ | | | | [`delta()`](/docs/reference/toolkit/counters-and-gauges/counter_agg/delta) | ✅ | | | | [`extrapolated_delta()`](/docs/reference/toolkit/counters-and-gauges/counter_agg/extrapolated_delta) | ✅ | | | | [`extrapolated_rate()`](/docs/reference/toolkit/counters-and-gauges/counter_agg/extrapolated_rate) | ✅ | | | | [`first_time()`](/docs/reference/toolkit/counters-and-gauges/counter_agg/first_time) | ✅ | | | | [`first_val()`](/docs/reference/toolkit/counters-and-gauges/counter_agg/first_val) | ✅ | | | | [`idelta_left()`](/docs/reference/toolkit/counters-and-gauges/counter_agg/idelta_left) | ✅ | | | | [`idelta_right()`](/docs/reference/toolkit/counters-and-gauges/counter_agg/idelta_right) | ✅ | | | | [`intercept()`](/docs/reference/toolkit/counters-and-gauges/counter_agg/intercept) | ✅ | | | | [`interpolated_delta()`](/docs/reference/toolkit/counters-and-gauges/counter_agg/interpolated_delta) | ✅ | | | | [`interpolated_rate()`](/docs/reference/toolkit/counters-and-gauges/counter_agg/interpolated_rate) | ✅ | | | | [`irate_left()`](/docs/reference/toolkit/counters-and-gauges/counter_agg/irate_left) | ✅ | | | | [`irate_right()`](/docs/reference/toolkit/counters-and-gauges/counter_agg/irate_right) | ✅ | | | | [`last_time()`](/docs/reference/toolkit/counters-and-gauges/counter_agg/last_time) | ✅ | | | | [`last_val()`](/docs/reference/toolkit/counters-and-gauges/counter_agg/last_val) | ✅ | | | | [`num_changes()`](/docs/reference/toolkit/counters-and-gauges/counter_agg/num_changes) | ✅ | | | | [`num_elements()`](/docs/reference/toolkit/counters-and-gauges/counter_agg/num_elements) | ✅ | | | | [`num_resets()`](/docs/reference/toolkit/counters-and-gauges/counter_agg/num_resets) | ✅ | | | | [`rate()`](/docs/reference/toolkit/counters-and-gauges/counter_agg/rate) | ✅ | | | | [`slope()`](/docs/reference/toolkit/counters-and-gauges/counter_agg/slope) | ✅ | | | | [`time_delta()`](/docs/reference/toolkit/counters-and-gauges/counter_agg/time_delta) | ✅ | | | Rollup | [`rollup()`](/docs/reference/toolkit/counters-and-gauges/counter_agg/rollup) | ✅ | | | Mutator | [`with_bounds()`](/docs/reference/toolkit/counters-and-gauges/counter_agg/with_bounds) | ✅ | | **`gauge_agg()` functions** | Type | Name | Toolkit | Experimental | | --------- | -------------------------------------------------------------------------------------------------- | ------- | ------------ | | Aggregate | [`gauge_agg()`](/docs/reference/toolkit/counters-and-gauges/gauge_agg/gauge_agg) | ✅ | ✅ | | Accessor | [`corr()`](/docs/reference/toolkit/counters-and-gauges/gauge_agg/corr) | ✅ | ✅ | | | [`delta()`](/docs/reference/toolkit/counters-and-gauges/gauge_agg/delta) | ✅ | ✅ | | | [`extrapolated_delta()`](/docs/reference/toolkit/counters-and-gauges/gauge_agg/extrapolated_delta) | ✅ | ✅ | | | [`extrapolated_rate()`](/docs/reference/toolkit/counters-and-gauges/gauge_agg/extrapolated_rate) | ✅ | ✅ | | | [`gauge_zero_time()`](/docs/reference/toolkit/counters-and-gauges/gauge_agg/gauge_zero_time) | ✅ | ✅ | | | [`idelta_left()`](/docs/reference/toolkit/counters-and-gauges/gauge_agg/idelta_left) | ✅ | ✅ | | | [`idelta_right()`](/docs/reference/toolkit/counters-and-gauges/gauge_agg/idelta_right) | ✅ | ✅ | | | [`intercept()`](/docs/reference/toolkit/counters-and-gauges/gauge_agg/intercept) | ✅ | ✅ | | | [`interpolated_delta()`](/docs/reference/toolkit/counters-and-gauges/gauge_agg/interpolated_delta) | ✅ | ✅ | | | [`interpolated_rate()`](/docs/reference/toolkit/counters-and-gauges/gauge_agg/interpolated_rate) | ✅ | ✅ | | | [`irate_left()`](/docs/reference/toolkit/counters-and-gauges/gauge_agg/irate_left) | ✅ | ✅ | | | [`irate_right()`](/docs/reference/toolkit/counters-and-gauges/gauge_agg/irate_right) | ✅ | ✅ | | | [`num_changes()`](/docs/reference/toolkit/counters-and-gauges/gauge_agg/num_changes) | ✅ | ✅ | | | [`num_elements()`](/docs/reference/toolkit/counters-and-gauges/gauge_agg/num_elements) | ✅ | ✅ | | | [`slope()`](/docs/reference/toolkit/counters-and-gauges/gauge_agg/slope) | ✅ | ✅ | | | [`rate()`](/docs/reference/toolkit/counters-and-gauges/gauge_agg/rate) | ✅ | ✅ | | | [`time_delta()`](/docs/reference/toolkit/counters-and-gauges/gauge_agg/time_delta) | ✅ | ✅ | | Rollup | [`rollup()`](/docs/reference/toolkit/counters-and-gauges/gauge_agg/rollup) | ✅ | ✅ | | Mutator | [`with_bounds()`](/docs/reference/toolkit/counters-and-gauges/gauge_agg/with_bounds) | ✅ | ✅ | ### Downsampling functions [Section titled “Downsampling functions”](#downsampling-functions) | Type | Name | Toolkit | Experimental | | -------- | ------------------------------------------------------------------- | ------- | ------------ | | Function | [`asap_smooth()`](/docs/reference/toolkit/downsampling/asap_smooth) | ✅ | | | | [`lttb()`](/docs/reference/toolkit/downsampling/lttb) | ✅ | | | | [`gp_lttb()`](/docs/reference/toolkit/downsampling/gp_lttb) | ✅ | ✅ | ### Financial analysis functions [Section titled “Financial analysis functions”](#financial-analysis-functions) **`candlestick_agg()` functions** | Type | Name | Toolkit | Experimental | | ---------------- | ------------------------------------------------------------------------------ | ------- | ------------ | | Aggregate | [`candlestick_agg()`](/docs/reference/toolkit/candlestick_agg/candlestick_agg) | ✅ | | | Pseudo aggregate | [`candlestick()`](/docs/reference/toolkit/candlestick_agg/candlestick) | ✅ | | | Accessor | [`close()`](/docs/reference/toolkit/candlestick_agg/close) | ✅ | | | | [`close_time()`](/docs/reference/toolkit/candlestick_agg/close_time) | ✅ | | | | [`high()`](/docs/reference/toolkit/candlestick_agg/high) | ✅ | | | | [`high_time()`](/docs/reference/toolkit/candlestick_agg/high_time) | ✅ | | | | [`low()`](/docs/reference/toolkit/candlestick_agg/low) | ✅ | | | | [`low_time()`](/docs/reference/toolkit/candlestick_agg/low_time) | ✅ | | | | [`open()`](/docs/reference/toolkit/candlestick_agg/open) | ✅ | | | | [`open_time()`](/docs/reference/toolkit/candlestick_agg/open_time) | ✅ | | | | [`volume()`](/docs/reference/toolkit/candlestick_agg/volume) | ✅ | | | | [`vwap()`](/docs/reference/toolkit/candlestick_agg/vwap) | ✅ | | | Rollup | [`rollup()`](/docs/reference/toolkit/candlestick_agg/rollup) | ✅ | | ### Frequency analysis functions [Section titled “Frequency analysis functions”](#frequency-analysis-functions) **`count_min_sketch()` functions** | Type | Name | Toolkit | Experimental | | --------- | ---------------------------------------------------------------------------------------------------- | ------- | ------------ | | Aggregate | [`count_min_sketch()`](/docs/reference/toolkit/frequency-analysis/count_min_sketch/count_min_sketch) | ✅ | ✅ | | Accessor | [`approx_count()`](/docs/reference/toolkit/frequency-analysis/count_min_sketch/approx_count) | ✅ | ✅ | **`freq_agg()` functions** | Type | Name | Toolkit | Experimental | | ------------------- | -------------------------------------------------------------------------------------- | ------- | ------------ | | Aggregate | [`freq_agg()`](/docs/reference/toolkit/frequency-analysis/freq_agg/freq_agg) | ✅ | ✅ | | Accessor | [`into_values()`](/docs/reference/toolkit/frequency-analysis/freq_agg/into_values) | ✅ | | | | [`max_frequency()`](/docs/reference/toolkit/frequency-analysis/freq_agg/max_frequency) | ✅ | | | | [`min_frequency()`](/docs/reference/toolkit/frequency-analysis/freq_agg/min_frequency) | ✅ | | | | [`topn()`](/docs/reference/toolkit/frequency-analysis/freq_agg/topn) | ✅ | | | Alternate aggregate | [`mcv_agg()`](/docs/reference/toolkit/frequency-analysis/freq_agg/mcv_agg) | ✅ | | | Rollup | [`rollup()`](/docs/reference/toolkit/frequency-analysis/freq_agg/rollup) | ✅ | | ### Gapfilling functions [Section titled “Gapfilling functions”](#gapfilling-functions) | Type | Name | Toolkit | Experimental | | ------------ | ------------------------------------------------------------------------------------------------------------- | ------- | ------------ | | Bucket | [`time_bucket_gapfill()`](/docs/reference/timescaledb/hyperfunctions/time_bucket_gapfill/time_bucket_gapfill) | | | | Interpolator | [`interpolate()`](/docs/reference/timescaledb/hyperfunctions/time_bucket_gapfill/interpolate) | | | | | [`locf()`](/docs/reference/timescaledb/hyperfunctions/time_bucket_gapfill/locf) | | | ### Minimum and maximum functions [Section titled “Minimum and maximum functions”](#minimum-and-maximum-functions) **`max_n()` functions** | Type | Name | Toolkit | Experimental | | --------- | -------------------------------------------------------------------------------- | ------- | ------------ | | Aggregate | [`max_n()`](/docs/reference/toolkit/minimum-and-maximum/max_n/max_n) | ✅ | | | Accessor | [`into_values()`](/docs/reference/toolkit/minimum-and-maximum/max_n/into_values) | ✅ | | | | [`into_array()`](/docs/reference/toolkit/minimum-and-maximum/max_n/into_array) | ✅ | | | Rollup | [`rollup()`](/docs/reference/toolkit/minimum-and-maximum/max_n/rollup) | ✅ | | **`max_n_by()` functions** | Type | Name | Toolkit | Experimental | | --------- | ----------------------------------------------------------------------------------- | ------- | ------------ | | Aggregate | [`max_n_by()`](/docs/reference/toolkit/minimum-and-maximum/max_n_by/max_n_by) | ✅ | | | Accessor | [`into_values()`](/docs/reference/toolkit/minimum-and-maximum/max_n_by/into_values) | ✅ | | | Rollup | [`rollup()`](/docs/reference/toolkit/minimum-and-maximum/max_n_by/rollup) | ✅ | | **`min_n()` functions** | Type | Name | Toolkit | Experimental | | --------- | -------------------------------------------------------------------------------- | ------- | ------------ | | Aggregate | [`min_n()`](/docs/reference/toolkit/minimum-and-maximum/min_n/min_n) | ✅ | | | Accessor | [`into_array()`](/docs/reference/toolkit/minimum-and-maximum/min_n/into_array) | ✅ | | | | [`into_values()`](/docs/reference/toolkit/minimum-and-maximum/min_n/into_values) | ✅ | | | Rollup | [`rollup()`](/docs/reference/toolkit/minimum-and-maximum/min_n/rollup) | ✅ | | **`min_n_by()` functions** | Type | Name | Toolkit | Experimental | | --------- | ----------------------------------------------------------------------------------- | ------- | ------------ | | Aggregate | [`min_n_by()`](/docs/reference/toolkit/minimum-and-maximum/min_n_by/min_n_by) | ✅ | | | Accessor | [`into_values()`](/docs/reference/toolkit/minimum-and-maximum/min_n_by/into_values) | ✅ | | | Rollup | [`rollup()`](/docs/reference/toolkit/minimum-and-maximum/min_n_by/rollup) | ✅ | | ### Percentile approximation functions [Section titled “Percentile approximation functions”](#percentile-approximation-functions) **`tdigest()` functions** | Type | Name | Toolkit | Experimental | | --------- | ------------------------------------------------------------------------------------------------------------- | ------- | ------------ | | Aggregate | [`tdigest()`](/docs/reference/toolkit/percentile-approximation/tdigest/tdigest) | ✅ | | | Accessor | [`approx_percentile()`](/docs/reference/toolkit/percentile-approximation/tdigest/approx_percentile) | ✅ | | | | [`approx_percentile_rank()`](/docs/reference/toolkit/percentile-approximation/tdigest/approx_percentile_rank) | ✅ | | | | [`max_val()`](/docs/reference/toolkit/percentile-approximation/tdigest/max_val) | ✅ | | | | [`mean()`](/docs/reference/toolkit/percentile-approximation/tdigest/mean) | ✅ | | | | [`min_val()`](/docs/reference/toolkit/percentile-approximation/tdigest/min_val) | ✅ | | | | [`num_vals()`](/docs/reference/toolkit/percentile-approximation/tdigest/num_vals) | ✅ | | | | [`total()`](/docs/reference/toolkit/percentile-approximation/tdigest/total) | ✅ | | | Rollup | [`rollup()`](/docs/reference/toolkit/percentile-approximation/tdigest/rollup) | ✅ | | **`percentile_agg()` / `uddsketch()` functions** | Type | Name | Toolkit | Experimental | | ------------------- | ----------------------------------------------------------------------------------------------------------------- | ------- | ------------ | | Aggregate | [`percentile_agg()`](/docs/reference/toolkit/percentile-approximation/uddsketch/percentile_agg) | ✅ | | | Alternate aggregate | [`uddsketch()`](/docs/reference/toolkit/percentile-approximation/uddsketch/uddsketch) | ✅ | | | Accessor | [`approx_percentile()`](/docs/reference/toolkit/percentile-approximation/uddsketch/approx_percentile) | ✅ | | | | [`approx_percentile_array()`](/docs/reference/toolkit/percentile-approximation/uddsketch/approx_percentile_array) | ✅ | | | | [`approx_percentile_rank()`](/docs/reference/toolkit/percentile-approximation/uddsketch/approx_percentile_rank) | ✅ | | | | [`error()`](/docs/reference/toolkit/percentile-approximation/uddsketch/error) | ✅ | | | | [`mean()`](/docs/reference/toolkit/percentile-approximation/uddsketch/mean) | ✅ | | | | [`num_vals()`](/docs/reference/toolkit/percentile-approximation/uddsketch/num_vals) | ✅ | | | | [`total()`](/docs/reference/toolkit/percentile-approximation/uddsketch/total) | ✅ | | | Rollup | [`rollup()`](/docs/reference/toolkit/percentile-approximation/uddsketch/rollup) | ✅ | | ### Saturating math functions [Section titled “Saturating math functions”](#saturating-math-functions) | Type | Name | Toolkit | Experimental | | -------- | ------------------------------------------------------------------------------------ | ------- | ------------ | | Function | [`saturating_add()`](/docs/reference/toolkit/saturating-math/saturating_add) | ✅ | | | | [`saturating_add_pos()`](/docs/reference/toolkit/saturating-math/saturating_add_pos) | ✅ | | | | [`saturating_mul()`](/docs/reference/toolkit/saturating-math/saturating_mul) | ✅ | | | | [`saturating_sub()`](/docs/reference/toolkit/saturating-math/saturating_sub) | ✅ | | | | [`saturating_sub_pos()`](/docs/reference/toolkit/saturating-math/saturating_sub_pos) | ✅ | | ### State tracking functions [Section titled “State tracking functions”](#state-tracking-functions) **`compact_state_agg()` functions** | Type | Name | Toolkit | Experimental | | --------- | ----------------------------------------------------------------------------------------------------------------- | ------- | ------------ | | Aggregate | [`compact_state_agg()`](/docs/reference/toolkit/state-tracking/compact_state_agg/compact_state_agg) | ✅ | ✅ | | Accessor | [`duration_in()`](/docs/reference/toolkit/state-tracking/compact_state_agg/duration_in) | ✅ | ✅ | | | [`interpolated_duration_in()`](/docs/reference/toolkit/state-tracking/compact_state_agg/interpolated_duration_in) | ✅ | ✅ | | | [`into_values()`](/docs/reference/toolkit/state-tracking/compact_state_agg/into_values) | ✅ | ✅ | | Rollup | [`rollup()`](/docs/reference/toolkit/state-tracking/compact_state_agg/rollup) | ✅ | ✅ | **`heartbeat_agg()` functions** | Type | Name | Toolkit | Experimental | | --------- | ------------------------------------------------------------------------------------------------------- | ------- | ------------ | | Aggregate | [`heartbeat_agg()`](/docs/reference/toolkit/state-tracking/heartbeat_agg/heartbeat_agg) | ✅ | | | Accessor | [`dead_ranges()`](/docs/reference/toolkit/state-tracking/heartbeat_agg/dead_ranges) | ✅ | | | | [`downtime()`](/docs/reference/toolkit/state-tracking/heartbeat_agg/downtime) | ✅ | | | | [`interpolate()`](/docs/reference/toolkit/state-tracking/heartbeat_agg/interpolate) | ✅ | | | | [`interpolated_downtime()`](/docs/reference/toolkit/state-tracking/heartbeat_agg/interpolated_downtime) | ✅ | | | | [`interpolated_uptime()`](/docs/reference/toolkit/state-tracking/heartbeat_agg/interpolated_uptime) | ✅ | | | | [`live_ranges()`](/docs/reference/toolkit/state-tracking/heartbeat_agg/live_ranges) | ✅ | | | | [`num_gaps()`](/docs/reference/toolkit/state-tracking/heartbeat_agg/num_gaps) | ✅ | | | | [`live_at()`](/docs/reference/toolkit/state-tracking/heartbeat_agg/live_at) | ✅ | | | | [`num_live_ranges()`](/docs/reference/toolkit/state-tracking/heartbeat_agg/num_live_ranges) | ✅ | | | | [`trim_to()`](/docs/reference/toolkit/state-tracking/heartbeat_agg/trim_to) | ✅ | | | | [`uptime()`](/docs/reference/toolkit/state-tracking/heartbeat_agg/uptime) | ✅ | | | Rollup | [`rollup()`](/docs/reference/toolkit/state-tracking/heartbeat_agg/rollup) | ✅ | | **`state_agg()` functions** | Type | Name | Toolkit | Experimental | | --------- | --------------------------------------------------------------------------------------------------------------- | ------- | ------------ | | Aggregate | [`state_agg()`](/docs/reference/toolkit/state-tracking/state_agg/state_agg) | ✅ | | | Accessor | [`duration_in()`](/docs/reference/toolkit/state-tracking/state_agg/duration_in) | ✅ | | | | [`interpolated_duration_in()`](/docs/reference/toolkit/state-tracking/state_agg/interpolated_duration_in) | ✅ | | | | [`interpolated_state_periods()`](/docs/reference/toolkit/state-tracking/state_agg/interpolated_state_periods) | ✅ | | | | [`interpolated_state_timeline()`](/docs/reference/toolkit/state-tracking/state_agg/interpolated_state_timeline) | ✅ | | | | [`into_values()`](/docs/reference/toolkit/state-tracking/state_agg/into_values) | ✅ | | | | [`state_at()`](/docs/reference/toolkit/state-tracking/state_agg/state_at) | ✅ | | | | [`state_periods()`](/docs/reference/toolkit/state-tracking/state_agg/state_periods) | ✅ | | | | [`state_timeline()`](/docs/reference/toolkit/state-tracking/state_agg/state_timeline) | ✅ | | | Rollup | [`rollup()`](/docs/reference/toolkit/state-tracking/state_agg/rollup) | ✅ | | ### Statistical and regression analysis functions [Section titled “Statistical and regression analysis functions”](#statistical-and-regression-analysis-functions) **`stats_agg()` (one variable) functions** | Type | Name | Toolkit | Experimental | | --------- | ---------------------------------------------------------------------------------------------------------------------------- | ------- | ------------ | | Aggregate | [`stats_agg()`](/docs/reference/toolkit/statistical-and-regression-analysis/stats_agg-one-variable/stats_agg) (one variable) | ✅ | | | Accessor | [`average()`](/docs/reference/toolkit/statistical-and-regression-analysis/stats_agg-one-variable/average) | ✅ | | | | [`num_vals()`](/docs/reference/toolkit/statistical-and-regression-analysis/stats_agg-one-variable/num_vals) | ✅ | | | | [`kurtosis()`](/docs/reference/toolkit/statistical-and-regression-analysis/stats_agg-one-variable/kurtosis) | ✅ | | | | [`skewness()`](/docs/reference/toolkit/statistical-and-regression-analysis/stats_agg-one-variable/skewness) | ✅ | | | | [`stddev()`](/docs/reference/toolkit/statistical-and-regression-analysis/stats_agg-one-variable/stddev) | ✅ | | | | [`sum()`](/docs/reference/toolkit/statistical-and-regression-analysis/stats_agg-one-variable/sum) | ✅ | | | | [`variance()`](/docs/reference/toolkit/statistical-and-regression-analysis/stats_agg-one-variable/variance) | ✅ | | | Rollup | [`rolling()`](/docs/reference/toolkit/statistical-and-regression-analysis/stats_agg-one-variable/rolling) | ✅ | | | | [`rollup()`](/docs/reference/toolkit/statistical-and-regression-analysis/stats_agg-one-variable/rollup) | ✅ | | **`stats_agg()` (two variables) functions** | Type | Name | Toolkit | Experimental | | --------- | ---------------------------------------------------------------------------------------------------------------------------------- | ------- | ------------ | | Aggregate | [`stats_agg()`](/docs/reference/toolkit/statistical-and-regression-analysis/stats_agg-two-variables/stats_agg) (two variables) | ✅ | | | Accessor | [`average_y()`, `average_x()`](/docs/reference/toolkit/statistical-and-regression-analysis/stats_agg-two-variables/average_y_x) | ✅ | | | | [`corr()`](/docs/reference/toolkit/statistical-and-regression-analysis/stats_agg-two-variables/corr) | ✅ | | | | [`covariance()`](/docs/reference/toolkit/statistical-and-regression-analysis/stats_agg-two-variables/covariance) | ✅ | | | | [`determination_coeff()`](/docs/reference/toolkit/statistical-and-regression-analysis/stats_agg-two-variables/determination_coeff) | ✅ | | | | [`intercept()`](/docs/reference/toolkit/statistical-and-regression-analysis/stats_agg-two-variables/intercept) | ✅ | | | | [`kurtosis_y()`, `kurtosis_x()`](/docs/reference/toolkit/statistical-and-regression-analysis/stats_agg-two-variables/kurtosis_y_x) | ✅ | | | | [`num_vals()`](/docs/reference/toolkit/statistical-and-regression-analysis/stats_agg-two-variables/num_vals) | ✅ | | | | [`skewness_y()`, `skewness_x()`](/docs/reference/toolkit/statistical-and-regression-analysis/stats_agg-two-variables/skewness_y_x) | ✅ | | | | [`slope()`](/docs/reference/toolkit/statistical-and-regression-analysis/stats_agg-two-variables/slope) | ✅ | | | | [`stddev_y()`, `stddev_x()`](/docs/reference/toolkit/statistical-and-regression-analysis/stats_agg-two-variables/stddev_y_x) | ✅ | | | | [`sum_y()`, `sum_x()`](/docs/reference/toolkit/statistical-and-regression-analysis/stats_agg-two-variables/sum_y_x) | ✅ | | | | [`variance_y()`, `variance_x()`](/docs/reference/toolkit/statistical-and-regression-analysis/stats_agg-two-variables/variance_y_x) | ✅ | | | | [`x_intercept()`](/docs/reference/toolkit/statistical-and-regression-analysis/stats_agg-two-variables/x_intercept) | ✅ | | | Rollup | [`rolling()`](/docs/reference/toolkit/statistical-and-regression-analysis/stats_agg-two-variables/rolling) | ✅ | | | | [`rollup()`](/docs/reference/toolkit/statistical-and-regression-analysis/stats_agg-two-variables/rollup) | ✅ | | ### Time weighted calculations functions [Section titled “Time weighted calculations functions”](#time-weighted-calculations-functions) **`time_weight()` functions** | Type | Name | Toolkit | Experimental | | --------- | -------------------------------------------------------------------------------------- | ------- | ------------ | | Aggregate | [`time_weight()`](/docs/reference/toolkit/time_weight/time_weight) | ✅ | | | Accessor | [`average()`](/docs/reference/toolkit/time_weight/average) | ✅ | | | | [`first_time()`](/docs/reference/toolkit/time_weight/first_time) | ✅ | | | | [`first_val()`](/docs/reference/toolkit/time_weight/first_val) | ✅ | | | | [`integral()`](/docs/reference/toolkit/time_weight/integral) | ✅ | | | | [`interpolated_average()`](/docs/reference/toolkit/time_weight/interpolated_average) | ✅ | | | | [`interpolated_integral()`](/docs/reference/toolkit/time_weight/interpolated_integral) | ✅ | | | | [`last_time()`](/docs/reference/toolkit/time_weight/last_time) | ✅ | | | | [`last_val()`](/docs/reference/toolkit/time_weight/last_val) | ✅ | | | Rollup | [`rollup()`](/docs/reference/toolkit/time_weight/rollup) | ✅ | | ### General functions [Section titled “General functions”](#general-functions) | Type | Name | Toolkit | Experimental | | ------------------ | ------------------------------------------------------------------------------------------------------------------- | ------- | ------------ | | Bucket | [`time_bucket()`](/docs/reference/timescaledb/hyperfunctions/time-series-utilities/time_bucket) | | | | One step aggregate | [`approximate_row_count()`](/docs/reference/timescaledb/hyperfunctions/distribution-analysis/approximate_row_count) | | | | | [`first()`](/docs/reference/timescaledb/hyperfunctions/time-series-utilities/first) | | | | | [`histogram()`](/docs/reference/timescaledb/hyperfunctions/distribution-analysis/histogram) | | | | | [`last()`](/docs/reference/timescaledb/hyperfunctions/time-series-utilities/last) | | | | One step operation | [`days_in_month()`](/docs/reference/timescaledb/hyperfunctions/time-series-utilities/days_in_month) | ✅ | | | | [`month_normalize()`](/docs/reference/timescaledb/hyperfunctions/time-series-utilities/month_normalize) | ✅ | | For more information about each of the API calls listed in this table, see the [hyperfunction API documentation](/docs/reference/timescaledb/hyperfunctions). ## Function pipelines [Section titled “Function pipelines”](#function-pipelines) Function pipelines are an experimental feature, designed to radically improve the developer ergonomics of analyzing data in PostgreSQL and SQL, by applying principles from functional programming and popular tools like Python's Pandas, and PromQL. SQL is the best language for data analysis, but it is not perfect, and at times can get quite unwieldy. For example, this query gets data from the last day from the measurements table, sorts the data by the time column, calculates the delta between the values, takes the absolute value of the delta, and then takes the sum of the result of the previous steps: ```sql SELECT device_id, sum(abs_delta) as volatility FROM ( SELECT device_id, abs(val - lag(val) OVER last_day) as abs_delta FROM measurements WHERE ts >= now()-'1 day'::interval) calc_delta GROUP BY device_id; ``` You can express the same query with a function pipeline like this: ```sql SELECT device_id, timevector(ts, val) -> sort() -> delta() -> abs() -> sum() as volatility FROM measurements WHERE ts >= now()-'1 day'::interval GROUP BY device_id; ``` Function pipelines are completely SQL compliant, meaning that any tool that speaks SQL is able to support data analysis using function pipelines. For more information about how function pipelines work, read the [blog post](https://www.tigerdata.com/blog/function-pipelines-building-functional-programming-into-postgresql-using-custom-operators). ## TimescaleDB Toolkit feature development [Section titled “TimescaleDB Toolkit feature development”](#timescaledb-toolkit-feature-development) TimescaleDB Toolkit features are developed in the open. As features are developed they are categorized as experimental, beta, stable, or deprecated. This documentation covers the stable features, but more information on experimental features in development can be found in the [Toolkit repository](https://github.com/timescale/timescale-analytics/tree/main/docs). ## Contribute to TimescaleDB Toolkit [Section titled “Contribute to TimescaleDB Toolkit”](#contribute-to-timescaledb-toolkit) We want and need your feedback! What are the frustrating parts of analyzing time-series data? What takes far more code than you feel it should? What runs slowly, or only runs quickly after many rewrites? We want to solve community-wide problems and incorporate as much feedback as possible. * Join the [discussion](https://github.com/timescale/timescale-analytics/discussions). * Check out the [proposed features](https://github.com/timescale/timescale-analytics/labels/proposed-feature). * Explore the current [feature requests](https://github.com/timescale/timescale-analytics/labels/feature-request). * Add your own [feature request](https://github.com/timescale/timescale-analytics/issues/new?assignees=\&labels=feature-request\&template=feature-request.md\&title=). ## Learn more [Section titled “Learn more”](#learn-more) * [Gapfilling and interpolation](/docs/build/data-management/hyperfunctions/gapfilling-interpolation): Fill gaps in time-series query results. * [Function pipelines](/docs/build/data-management/hyperfunctions/function-pipelines): Chain hyperfunctions together for cleaner analytics queries. * [Use time buckets](/docs/learn/data-lifecycle/time-buckets/use-time-buckets): SQL examples for grouping data by time intervals. * [Install TimescaleDB Toolkit](/docs/deploy/self-hosted/tooling/install-toolkit): Set up the Toolkit extension for self-hosted deployments. * [Hyperfunctions API reference](/docs/reference/timescaledb/hyperfunctions): Full API reference for all built-in hyperfunctions. * [TimescaleDB Toolkit API reference](/docs/reference/toolkit): Full API reference for Toolkit hyperfunctions. # Create and configure a hypertable > Create a hypertable, set chunk intervals, control default indexes, and migrate existing data Hypertables turn ordinary PostgreSQL tables into time-partitioned tables made of **chunks**, so you can scale to very large datasets. Add `WITH (timescaledb.hypertable)` to a standard `CREATE TABLE` statement to create one. ## Create a hypertable [Section titled “Create a hypertable”](#create-a-hypertable) Create a hypertable from scratch in Tiger Console or with SQL. * Tiger Console To create a hypertable in Tiger Console: 1. **Select your service, then click `Actions`** 2. **Under `Goals`, find `Create a hypertable` and click `Get started`** In the wizard, select the `Create new table` tab. ![Create a hypertable wizard in Tiger Console](/docs/_astro/create-hypertable-name.CiPftzfu_18Qx6v.webp) 3. **Name the table** Under `Name table`, enter a name, then click `Continue`. The name must start with a letter or an underscore, and can contain only letters, numbers, and underscores. 4. **Configure the partition column** Under `Configure`, set the `Name` and `Type` of the column used to partition data into chunks, for example `created_at` with type `TIMESTAMP WITH TIME ZONE`. Optionally, select an `Additional dimension` to partition on a [secondary column](/docs/learn/hypertables/partitioning-hypertables#space-partitioning-hash-dimension). Click `Continue`. 5. **Create the hypertable** Under `Create hypertable`, review the generated SQL, then click `Run and proceed`. Your new table is created as a hypertable. ![Generated SQL in the Create a hypertable wizard](/docs/_astro/create-hypertable-create.V9zfc9Sm_1T8upP.webp) * SQL The simplest form auto-detects the partition column from the first `timestamptz` column and adds `NOT NULL` automatically if needed: ```sql CREATE TABLE sensor_data ( time TIMESTAMPTZ NOT NULL, device_id INT, temperature DOUBLE PRECISION ) WITH (timescaledb.hypertable); ``` If your table has multiple timestamp columns or uses a non-standard name, specify the partition column explicitly: ```sql CREATE TABLE order_events ( created_at TIMESTAMPTZ NOT NULL, order_id BIGINT, status TEXT ) WITH ( timescaledb.hypertable, timescaledb.partition_column = 'created_at' ); ``` ## Configuration options [Section titled “Configuration options”](#configuration-options) Set these options in the `WITH` clause to control how the hypertable is created: | Option | Default | Description | | ------------------------------------ | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `timescaledb.hypertable` | — | Required. Marks the table as a hypertable. | | `timescaledb.partition_column` | Auto-detected | The column used to partition data into chunks. Auto-detected from the first `timestamptz` column. Required if the table has no `timestamptz` column or has more than one. | | `timescaledb.chunk_interval` | `7 days` | The time range each chunk covers. Tune this for your ingest rate and memory. See [Sizing hypertable chunks](/docs/learn/hypertables/sizing-hypertable-chunks). | | `timescaledb.create_default_indexes` | `true` | Creates an index on the partition column automatically. Set to `false` for large bulk loads where you want to add indexes after loading. | ### Set chunk interval [Section titled “Set chunk interval”](#set-chunk-interval) Choose a chunk interval so that all active chunks fit in memory. The default is 7 days, which works well for many workloads. For high-throughput ingestion, a shorter interval like 1 day may be better: ```sql CREATE TABLE high_throughput_metrics ( time TIMESTAMPTZ NOT NULL, device_id INT, value DOUBLE PRECISION ) WITH ( timescaledb.hypertable, timescaledb.chunk_interval = '1 day' ); ``` ### Disable default indexes [Section titled “Disable default indexes”](#disable-default-indexes) For large bulk loads into a new hypertable, skip automatic index creation and add indexes after the data is loaded: ```sql CREATE TABLE bulk_load_target ( time TIMESTAMPTZ NOT NULL, device_id INT, value DOUBLE PRECISION ) WITH ( timescaledb.hypertable, timescaledb.create_default_indexes = false ); -- Load data first, then create indexes CREATE INDEX ON bulk_load_target (time DESC); CREATE INDEX ON bulk_load_target (device_id, time DESC); ``` ## Convert an existing table [Section titled “Convert an existing table”](#convert-an-existing-table) Convert an existing PostgreSQL table that already has data to a hypertable in Tiger Console or with SQL. * Tiger Console To convert an existing table in Tiger Console: 1. **Select your service, then click `Actions`** 2. **Under `Goals`, find `Create a hypertable` and click `Get started`** In the wizard, select the `Convert existing table` tab. ![Create a hypertable wizard in Tiger Console](/docs/_astro/convert-hypertable-start.H1y-GrgH_Z1kIM9b.webp) 3. **Choose the table to convert** Under `Choose a table`, select the PostgreSQL table you want to convert. 4. **Configure the partition column** Under `Configure`, select the column used to partition data into chunks, for example `time`. If the wizard reports `Primary key change required`, click `Run` to update the primary key, then continue. 5. **Transform the table into a hypertable** Your table is converted to a hypertable. ![A table converted to a hypertable in Tiger Console](/docs/_astro/convert-hypertable-complete.DQUxB5jw_Z1lvOAc.webp) * SQL To convert a PostgreSQL table that already has data, use the `create_hypertable()` function with `migrate_data`: ```sql SELECT create_hypertable('existing_table', by_range('time'), migrate_data => true); ``` For very large tables, this can be slow and lock-heavy. A safer approach is to create an empty hypertable and load data in batches: ```sql CREATE TABLE new_table (LIKE existing_table INCLUDING ALL) WITH (timescaledb.hypertable); INSERT INTO new_table SELECT * FROM existing_table; ``` ## Learn more [Section titled “Learn more”](#learn-more) * [Partition a hypertable](/docs/learn/hypertables/partitioning-hypertables): Time, integer, and space partitioning. * [Sizing hypertable chunks](/docs/learn/hypertables/sizing-hypertable-chunks): Choosing the right chunk interval. * [Hypertable indexes](/docs/learn/hypertables/hypertable-indexes): Default and custom indexes. * [Primary keys, time columns, and uniqueness](/docs/learn/data-model/primary-keys-time-and-uniqueness): Partition column and constraint rules. * [CREATE TABLE reference](/docs/reference/timescaledb/hypertables/create_table): Full API reference. * [`create_hypertable()` reference](/docs/reference/timescaledb/hypertables/create_hypertable): For converting existing tables. # Hypertable indexes > Default indexes, supported index types, and links to indexing guides for hypertables Indexes on hypertables work the same as on regular PostgreSQL tables, with a few TimescaleDB-specific behaviors. ## Default indexes [Section titled “Default indexes”](#default-indexes) When you create a hypertable, TimescaleDB automatically creates a B-tree index on the partition column (time descending). If the hypertable has a space dimension, a second composite index on (space column, time descending) is also created. You can disable this with `timescaledb.create_default_indexes = false` if you plan to add custom indexes after loading data. Indexes speed up `SELECT` queries but slow down `INSERT` operations and increase storage. Drop any indexes you don't actively use. ## Supported index types [Section titled “Supported index types”](#supported-index-types) hypertables support all PostgreSQL index types: * **B-tree** — the default. Best for equality and range filters on time, device IDs, and other scalar columns. * **Hash** — useful for equality-only lookups. * **GIN** — for JSONB, arrays, and full-text search columns. * **GiST** — for geometric, range, and full-text search data. * **BRIN** — block range indexes, useful for very large append-only tables. ## Unique indexes and primary keys [Section titled “Unique indexes and primary keys”](#unique-indexes-and-primary-keys) Any `UNIQUE` or `PRIMARY KEY` index must include the partition column. This is a TimescaleDB requirement because uniqueness is enforced per chunk. See [Primary keys, time columns, and uniqueness](/docs/learn/data-model/primary-keys-time-and-uniqueness) for details. ## Per-chunk index builds [Section titled “Per-chunk index builds”](#per-chunk-index-builds) Use the `timescaledb.transaction_per_chunk` option when creating indexes to build per chunk in separate transactions, reducing lock contention: ```sql CREATE INDEX idx_device_time ON sensor_data (device_id, time DESC) WITH (timescaledb.transaction_per_chunk = true); ``` ## Columnstore and indexes [Section titled “Columnstore and indexes”](#columnstore-and-indexes) PostgreSQL indexes (including B-tree) are kept on chunks converted to columnstore and continue to work for queries. TimescaleDB also uses batch-level metadata (min/max values) to skip irrelevant batches during scans. For best columnstore query performance, tune segmenting and ordering in addition to indexes. See [Improve query and upsert performance](/docs/build/performance-optimization/secondary-indexes) for details. ## Learn more [Section titled “Learn more”](#learn-more) * [Accelerate queries using indexes](/docs/build/performance-optimization/indexing): How-to guide for creating and tuning indexes. * [Improve query and upsert performance](/docs/build/performance-optimization/secondary-indexes): Segmenting and ordering columnstore data. * [Enforce constraints with unique indexes](/docs/build/performance-optimization/hypertables-and-unique-indexes): Unique index rules and limitations. * [Primary keys, time columns, and uniqueness](/docs/learn/data-model/primary-keys-time-and-uniqueness): Partition column requirements for unique constraints. * [`CREATE INDEX` reference](/docs/reference/timescaledb/hypertables/create_index): Full API reference including `transaction_per_chunk`. # Hypertable operations > Create, alter, and drop a hypertable, and speed up data ingest with direct compress Hypertables are designed for real-time analytics, they are PostgreSQL tables that automatically partition your data by time. Typically, you partition hypertables on columns that hold time values. [Best practice is to use `timestamptz`](https://wiki.postgresql.org/wiki/Don't_Do_This#Don.27t_use_timestamp_.28without_time_zone.29) column type. However, you can also partition on `date`, `integer`, `timestamp` and [UUIDv7](/docs/reference/timescaledb/uuid-functions) types. ## Prerequisites for this page To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service), or a running instance of [self-hosted TimescaleDB](/docs/get-started/choose-your-path/install-timescaledb). ## Create a hypertable [Section titled “Create a hypertable”](#create-a-hypertable) Create a [hypertable](/docs/learn/hypertables/understand-hypertables) for your time-series data using [CREATE TABLE](/docs/reference/timescaledb/hypertables/create_table). For [efficient queries](/docs/build/performance-optimization/secondary-indexes), remember to `segmentby` the column you will use most often to filter your data: ```sql CREATE TABLE conditions ( time TIMESTAMPTZ NOT NULL, location TEXT NOT NULL, device TEXT NOT NULL, temperature DOUBLE PRECISION NULL, humidity DOUBLE PRECISION NULL ) WITH ( tsdb.hypertable, tsdb.segmentby = 'device', tsdb.orderby = 'time DESC' ); ``` When you create a hypertable using [CREATE TABLE ... WITH ...](/docs/reference/timescaledb/hypertables/create_table), the default partitioning column is automatically the first column with a timestamp data type. Also, TimescaleDB creates a [columnstore policy](/docs/reference/timescaledb/hypercore/add_columnstore_policy) that automatically converts your data to the columnstore, after an interval equal to the value of the [chunk\_interval](/docs/reference/timescaledb/hypertables/set_chunk_time_interval), defined through `after` in the policy. This columnar format enables fast scanning and aggregation, optimizing performance for analytical workloads while also saving significant storage space. In the columnstore conversion, hypertable chunks are compressed by up to 98%, and organized for efficient, large-scale queries. You can customize this policy later using [alter\_job](/docs/reference/timescaledb/jobs-automation/alter_job). However, to change `after` or `created_before`, the compression settings, or the hypertable the policy is acting on, you must [remove the columnstore policy](/docs/reference/timescaledb/hypercore/remove_columnstore_policy) and [add a new one](/docs/reference/timescaledb/hypercore/add_columnstore_policy). You can also manually [convert chunks](/docs/reference/timescaledb/hypercore/convert_to_columnstore) in a hypertable to the columnstore. To convert an existing table with data in it, call `create_hypertable` on that table with [`migrate_data` to `true`](/docs/reference/timescaledb/hypertables/create_hypertable#arguments). However, if you have a lot of data, this may take a long time. ## Speed up data ingestion [Section titled “Speed up data ingestion”](#speed-up-data-ingestion) When you set `timescaledb.enable_direct_compress_insert` or `timescaledb.enable_direct_compress_copy`, TimescaleDB compresses data in memory during ingestion and writes the compressed batches directly to the columnstore. This significantly reduces the I/O footprint, and `INSERT` and `COPY` produce columnstore chunks on the spot — the [columnstore policy](/docs/reference/timescaledb/hypercore/add_columnstore_policy) is less important. Warning This feature is a **tech preview** and not production-ready. Using direct compress can regress query performance or storage ratio if the ingested rows are not sorted by the table's `orderby` columns or if the data has very high cardinality. To enable in-memory compression during ingestion, set the appropriate GUC: ```sql SET timescaledb.enable_direct_compress_insert = on; -- or SET timescaledb.enable_direct_compress_copy = on; ``` **Important facts** * High cardinality data does not produce good batches and degrades query performance. * The columnstore is optimized for 1000 rows per batch per `segmentby` value. * WAL records are written for the compressed batches rather than the individual tuples. * Best results are achieved for batch ingestion with 1000 rows or more. * Tables with unique constraints cannot use direct compress. * If your hypertable has no `segmentby` column configured, TimescaleDB picks one for you based on the first batch of rows you ingest. Turn this off with `SET timescaledb.enable_direct_compress_auto_segmentby = off`. ## Alter a hypertable [Section titled “Alter a hypertable”](#alter-a-hypertable) You can alter a hypertable, for example to add a column, by using the PostgreSQL [`ALTER TABLE`](https://www.postgresql.org/docs/current/sql-altertable.html) command. Some operations are not supported for hypertables with columnstore enabled. See [Altering hypertables with columnstore enabled](/docs/build/performance-optimization/alter-update-table-schema). ### Add a column to a hypertable [Section titled “Add a column to a hypertable”](#add-a-column-to-a-hypertable) You add a column to a hypertable using the `ALTER TABLE` command. In this example, the hypertable is named `conditions` and the new column is named `humidity`: ```sql ALTER TABLE conditions ADD COLUMN humidity DOUBLE PRECISION NULL; ``` Adding a column is fast regardless of the default value. PostgreSQL 11+ stores the default in the catalog without rewriting existing rows. ### Rename a hypertable [Section titled “Rename a hypertable”](#rename-a-hypertable) You can change the name of a hypertable using the `ALTER TABLE` command. In this example, the hypertable is called `conditions`, and is being changed to the new name, `weather`: ```sql ALTER TABLE conditions RENAME TO weather; ``` ### Change a column data type [Section titled “Change a column data type”](#change-a-column-data-type) You can change the data type of a column in a hypertable using the `ALTER TABLE` command. In this example, the `temperature` column data type is changed from `DOUBLE PRECISION` to `NUMERIC`: ```sql ALTER TABLE conditions ALTER COLUMN temperature TYPE NUMERIC; ``` The following restrictions apply: * For time dimension columns, you can only change to `TIMESTAMPTZ`, `TIMESTAMP`, `DATE`, `INTEGER` (smallint, integer, or bigint), or `UUID`. * You cannot change the type of columns with custom partitioning functions. * You cannot change column types when the hypertable has columnstore chunks. Convert those chunks back to rowstore first. See [Altering hypertables with columnstore enabled](/docs/build/performance-optimization/alter-update-table-schema). * For columns with statistics enabled, you can only change to integer or timestamp types. To change to other types, first disable statistics using `disable_column_stats`. ### Drop a column [Section titled “Drop a column”](#drop-a-column) You can drop a column from a hypertable using the `ALTER TABLE` command. In this example, the `humidity` column is dropped from the `conditions` hypertable: ```sql ALTER TABLE conditions DROP COLUMN humidity; ``` You cannot drop partitioning columns. ## Delete data [Section titled “Delete data”](#delete-data) Prefer [`drop_chunks()`](/docs/reference/timescaledb/hypertables/drop_chunks) to remove whole time ranges instead of running large `DELETE` statements. Dropping chunks is faster and avoids table bloat: ```sql SELECT drop_chunks('conditions', older_than => INTERVAL '30 days'); ``` For automated deletion, use a [data retention policy](/docs/reference/timescaledb/data-retention). ## Vacuum and bloat [Section titled “Vacuum and bloat”](#vacuum-and-bloat) Heavy `INSERT`, `UPDATE`, and `DELETE` operations cause table bloat over time. PostgreSQL autovacuum runs by default, but you may need to tune its settings for high write-rate hypertables. Run `VACUUM ANALYZE` manually when you need to reclaim space or refresh query planner statistics: ```sql VACUUM ANALYZE conditions; ``` ## Drop a hypertable [Section titled “Drop a hypertable”](#drop-a-hypertable) Drop a hypertable using a standard PostgreSQL [`DROP TABLE`](https://www.postgresql.org/docs/current/sql-droptable.html) command: ```sql DROP TABLE conditions; ``` All chunks belonging to the hypertable are deleted. For very large hypertables that may time out, drop chunks first with `drop_chunks()`, then drop the table. ## Learn more [Section titled “Learn more”](#learn-more) * [Create and configure a hypertable](/docs/learn/hypertables/creating-and-configuring-hypertables): `CREATE TABLE` options and configuration. * [Alter and update table schemas](/docs/build/performance-optimization/alter-update-table-schema): Detailed guide for ALTER operations with columnstore. * [Improve hypertable performance](/docs/build/performance-optimization/improve-hypertable-performance): Optimize chunk intervals and enable chunk skipping. * [`CREATE TABLE` reference](/docs/reference/timescaledb/hypertables/create_table): Full API reference. * [`drop_chunks()` reference](/docs/reference/timescaledb/hypertables/drop_chunks): Remove chunks by age or time range. * [Data retention policies](/docs/reference/timescaledb/data-retention): Automate data deletion. * [`show_chunks()` reference](/docs/reference/timescaledb/hypertables/show_chunks): List chunks for a hypertable. # Partition a hypertable > Partition a hypertable by time or integer columns, and understand when to use space partitioning Every hypertable is partitioned on at least one column, called a **dimension**. The primary dimension is almost always **time**. Each chunk is assigned a range of time and only contains data from that range. ## Time partitioning [Section titled “Time partitioning”](#time-partitioning) Partition by a `timestamptz`, `timestamp`, `date`, or [UUIDv7](/docs/reference/timescaledb/uuid-functions) column. [Best practice is to use `timestamptz`](https://wiki.postgresql.org/wiki/Don't_Do_This#Don.27t_use_timestamp_.28without_time_zone.29). ```sql CREATE TABLE sensor_data ( time TIMESTAMPTZ NOT NULL, device_id INT, temperature DOUBLE PRECISION ) WITH ( timescaledb.hypertable, timescaledb.chunk_interval = '1 day' ); ``` By default, each chunk holds data for 7 days. You can change this with `timescaledb.chunk_interval` to better suit your workload. TimescaleDB divides time into potential chunk ranges based on the interval. When you insert data from a time range that doesn't yet have a chunk, TimescaleDB automatically creates one. This means the start time of your earliest chunk does not necessarily equal the earliest timestamp in your data — there might be a gap. This doesn't affect queries, but you may notice it when inspecting chunks. ## Integer partitioning [Section titled “Integer partitioning”](#integer-partitioning) For tables where the partition column is an integer (Unix epoch, sequence, or other monotonic key), specify the chunk interval as a number: ```sql CREATE TABLE events ( epoch_ms BIGINT NOT NULL, device_id INT, payload JSONB ) WITH ( timescaledb.hypertable, timescaledb.partition_column = 'epoch_ms', timescaledb.chunk_interval = 86400000 ); ``` If you use background policies like retention or columnstore, set [`integer_now_func`](/docs/reference/timescaledb/hypertables/set_integer_now_func) so those policies know what "current" means: ```sql CREATE OR REPLACE FUNCTION epoch_ms_now() RETURNS BIGINT LANGUAGE SQL STABLE AS $$ SELECT (extract(epoch FROM now()) * 1000)::BIGINT $$; SELECT set_integer_now_func('events', 'epoch_ms_now'); ``` For **Unix time**, you can reuse the same function across hypertables. For **sequence-style** keys, you need a dedicated function per hypertable. The default chunk interval of 7 days works for many workloads, but tuning it can significantly improve performance. See [Sizing hypertable chunks](/docs/learn/hypertables/sizing-hypertable-chunks) for guidance. ## Space partitioning (hash dimension) [Section titled “Space partitioning (hash dimension)”](#space-partitioning-hash-dimension) TimescaleDB supports adding a hash dimension to a hypertable with `add_dimension()`, which distributes rows across multiple partitions within each chunk. For example, you could partition by `tenant_id` in a multi-tenant setup to isolate tenants into separate partitions. Warning Space partitioning multiplies the number of chunks, increases query planning overhead, and requires all unique constraints to include the hash column. Do not use it without extensive testing and a deep understanding of how it affects your specific workload. For most use cases, time-based chunk sizing and indexing are a better fit. If you have a specific use case that requires hash partitioning, add a hash dimension after creating the hypertable: ```sql SELECT add_dimension('my_table', by_hash('device_id', 4)); ``` The second argument to `by_hash()` is the number of partitions. You must specify it explicitly — there is no default. ## Learn more [Section titled “Learn more”](#learn-more) * [Sizing hypertable chunks](/docs/learn/hypertables/sizing-hypertable-chunks): Choosing the right chunk interval for your workload. * [Create and configure a hypertable](/docs/learn/hypertables/creating-and-configuring-hypertables): `CREATE TABLE` options including `chunk_interval`. * [Primary keys, time columns, and uniqueness](/docs/learn/data-model/primary-keys-time-and-uniqueness): Constraint rules for partitioned tables. * [Improve hypertable performance](/docs/build/performance-optimization/improve-hypertable-performance): Optimize chunk intervals and enable chunk skipping. * [`set_integer_now_func()` reference](/docs/reference/timescaledb/hypertables/set_integer_now_func): Required for integer-partitioned policies. * [`add_dimension()` reference](/docs/reference/timescaledb/hypertables/add_dimension): Add hash or range dimensions. * [`set_chunk_time_interval()` reference](/docs/reference/timescaledb/hypertables/set_chunk_time_interval): Change the chunk interval on an existing hypertable. # Size hypertable chunks > Choose a chunk interval that balances memory usage, query planning, and columnstore effectiveness Chunk size affects insert and query performance. You want chunks small enough to fit into memory so you can insert and query recent data without reading from disk. However, too many small chunks can slow query planning and reduce columnstore effectiveness. ## How the chunk interval works [Section titled “How the chunk interval works”](#how-the-chunk-interval-works) The chunk interval is the time range each chunk covers. It determines how many chunks you get for a given time range: * **Default**: 7 days for timestamp columns. For integer columns, defaults vary by type (`smallint`: 10,000, `integer`: 100,000, `bigint`: 1,000,000). * **New chunks only**: Changing the interval with [`set_chunk_time_interval()`](/docs/reference/timescaledb/hypertables/set_chunk_time_interval) affects only chunks created after the change. Existing chunks keep their original boundaries. * **Minimum**: For timestamp types, intervals shorter than 1 second trigger a warning. ## Sizing guideline [Section titled “Sizing guideline”](#sizing-guideline) Set `chunk_interval` so that the indexes of chunks currently being ingested into fit within 25% of main memory (`shared_buffers`). PostgreSQL builds indexes on the fly during ingestion. When an index doesn't fit in memory, it is constantly flushed to disk and read back, wasting I/O that would otherwise be used for writes. **Example**: On a system with 64 GB of memory, `shared_buffers` is typically \~16 GB. * If index growth is approximately 2 GB per day, a 7-day interval is appropriate (2 GB x 7 = 14 GB, fits in 16 GB). * If index growth is around 10 GB per day, use a 1-day interval (10 GB, fits in 16 GB). ## Active vs inactive chunks [Section titled “Active vs inactive chunks”](#active-vs-inactive-chunks) * **Active** chunks receive new writes and their indexes should fit comfortably in cache. * **Inactive** chunks are older and mostly read-only — they can rely more on disk, especially after conversion to columnstore. ## Too many chunks [Section titled “Too many chunks”](#too-many-chunks) Having too many chunks in the system has costs: * **Query planning**: The planner must evaluate each chunk to decide whether to include or skip it. More chunks means more planning time. * **Columnstore**: Very small chunks compress less effectively because there is less data to find patterns in. * **System limits**: Limit the number of hypertables in your service — tens of thousands of hypertables is not recommended. ## Setting the chunk interval [Section titled “Setting the chunk interval”](#setting-the-chunk-interval) Set the interval at creation time: ```sql CREATE TABLE sensor_data ( time TIMESTAMPTZ NOT NULL, device_id INT, value DOUBLE PRECISION ) WITH ( timescaledb.hypertable, timescaledb.chunk_interval = '1 day' ); ``` Or change it for an existing hypertable: ```sql SELECT set_chunk_time_interval('sensor_data', INTERVAL '1 day'); ``` You can also set a global default for all new hypertables: ```sql SET timescaledb.default_chunk_time_interval = '1 day'; ``` ## When you need to retrofit the chunk interval [Section titled “When you need to retrofit the chunk interval”](#when-you-need-to-retrofit-the-chunk-interval) [`set_chunk_time_interval()`](/docs/reference/timescaledb/hypertables/set_chunk_time_interval) only affects **new** chunks. If an undersized `chunk_interval` has already fragmented your hypertable into thousands of small chunks, calling `set_chunk_time_interval()` alone does not heal historical data — the old fragments remain and keep dragging on planner time and memory. Watch for these symptoms, which indicate historical chunks need to be rebuilt: * **Over 1,000 chunks in a single hypertable**. Scanning chunk catalog metadata starts to dominate query planning. * **Intermittent out-of-memory errors** on wide time-range scans, caused by the volume of chunk metadata held in memory. * **Low columnstore compression ratios** on historical data — very small chunks compress poorly because there is less data to find patterns in. When this happens, the fix is a table-swap migration: create a new hypertable with the correct `chunk_interval`, atomically swap names so ingestion is never blocked, then backfill historical data in batches. For a step-by-step walkthrough, see [Retrofit chunk intervals on a production hypertable](/docs/build/performance-optimization/retrofit-chunk-intervals). ## Learn more [Section titled “Learn more”](#learn-more) * [Partition a hypertable](/docs/learn/hypertables/partitioning-hypertables): Time and integer partitioning options. * [Create and configure a hypertable](/docs/learn/hypertables/creating-and-configuring-hypertables): `CREATE TABLE` options including `chunk_interval`. * [Improve hypertable and query performance](/docs/build/performance-optimization/improve-hypertable-performance): Optimize chunk intervals and enable chunk skipping. * [Retrofit chunk intervals on a production hypertable](/docs/build/performance-optimization/retrofit-chunk-intervals): Rebuild historical chunks when `set_chunk_time_interval()` alone is not enough. * [Chunk time intervals blog post](https://www.tigerdata.com/blog/timescale-cloud-tips-testing-your-chunk-size): Detailed analysis of choosing intervals. * [`set_chunk_time_interval()` reference](/docs/reference/timescaledb/hypertables/set_chunk_time_interval): Change the chunk interval on an existing hypertable. * [`show_chunks()` reference](/docs/reference/timescaledb/hypertables/show_chunks): Inspect existing chunks and their ranges. # Understand hypertables > Hypertables are PostgreSQL tables with special features that power real-time analytics on time-series and event data Tiger Cloud supercharges your real-time analytics by letting you run complex queries continuously, with near-zero latency. Under the hood, this is achieved by using hypertables, PostgreSQL tables that automatically partition your time-series data by time and optionally by other dimensions. When you run a query, Tiger Cloud identifies the correct partition, called chunk, and runs the query on it, instead of going through the entire table. ![Tiger Data hypertable architecture: a normal Postgres table is automatically partitioned by time into chunks based on the chunk\_time\_interval setting](/docs/_astro/hypertable-light.f-v6IDWg_Z1YDb49.webp) ![Tiger Data hypertable architecture: a normal Postgres table is automatically partitioned by time into chunks based on the chunk\_time\_interval setting](/docs/_astro/hypertable-dark.CNX8sC6Y_Z23WxHL.webp) Hypertables offer the following benefits: * **Efficient data management with [automated partitioning by time](/docs/build/performance-optimization/improve-hypertable-performance#optimize-hypertable-chunk-intervals)**: Tiger Cloud splits your data into chunks that hold data from a specific time range. For example, one day or one week. You can configure this range to better suit your needs. * **Better performance with [strategic indexing](/docs/build/performance-optimization/hypertables-and-unique-indexes)**: an index on time in the descending order is automatically created when you create a hypertable. More indexes are created on the chunk level, to optimize performance. You can create additional indexes, including unique indexes, on the columns you need. * **Faster queries with [chunk skipping](/docs/build/performance-optimization/improve-hypertable-performance)**: Tiger Cloud skips the chunks that are irrelevant in the context of your query, dramatically reducing the time and resources needed to fetch results. Even more, you can enable chunk skipping on non-partitioning columns. * **Advanced data analysis with [hyperfunctions](/docs/learn/hyperfunctions/about-hyperfunctions)**: Tiger Cloud enables you to efficiently process, aggregate, and analyze significant volumes of data while maintaining high performance. To top it all, there is no added complexity, you interact with hypertables in the same way as you would with regular PostgreSQL tables. All the optimization magic happens behind the scenes. ## Get hands on [Section titled “Get hands on”](#get-hands-on) [Your first hypertable](/docs/build/how-to/your-first-hypertable) [Create your first hypertable step by step.](/docs/build/how-to/your-first-hypertable) [Create and configure a hypertable](/docs/learn/hypertables/creating-and-configuring-hypertables) [Set up a hypertable with CREATE TABLE ... WITH (timescaledb.hypertable).](/docs/learn/hypertables/creating-and-configuring-hypertables) [Size hypertable chunks](/docs/learn/hypertables/sizing-hypertable-chunks) [Choose the right chunk interval for your workload.](/docs/learn/hypertables/sizing-hypertable-chunks) ## Learn more [Section titled “Learn more”](#learn-more) * [Partition a hypertable](/docs/learn/hypertables/partitioning-hypertables): Time, integer, and space partitioning. * [Hypertable indexes](/docs/learn/hypertables/hypertable-indexes): Default indexes, supported types, and unique constraints. * [Hypertable operations](/docs/learn/hypertables/optimize-data-in-hypertables): Alter, drop, delete data, and vacuum. * [CREATE TABLE reference](/docs/reference/timescaledb/hypertables/create_table): Full API reference. * [Informational views](/docs/reference/timescaledb/informational-views): Query hypertable metadata from SQL. # Key vector concepts for pgvector > Learn how pgvector stores embeddings in ordinary tables, how nearest-neighbor queries and distance operators work, and why approximate indexes matter on Tiger Cloud. This page explains the ideas behind **pgvector** in PostgreSQL: the **`vector`** column type, typical table shapes, **k**-nearest-neighbor queries, distance metrics, and **approximate nearest neighbor (ANN)** indexes. For how these pieces fit with **pgvectorscale** and Tiger Cloud, see **[Understand pgvector and pgvectorscale](/docs/learn/search/pgvector-pgvectorsearch)**. ## The `vector` type and table design [Section titled “The vector type and table design”](#the-vector-type-and-table-design) Embeddings live in normal tables: you add a column whose type is provided by the **pgvector** extension (for example `vector(1536)` when your model outputs 1,536 dimensions). A common pattern is to store each embedding next to the text it represents and optional **JSONB** metadata: ```sql CREATE TABLE IF NOT EXISTS document_embedding ( id BIGINT PRIMARY KEY GENERATED BY DEFAULT AS IDENTITY, document_id BIGINT REFERENCES document (id), metadata JSONB, contents TEXT, embedding vector(1536) ); ``` The row has a primary key, an optional foreign key to a parent document, metadata, the chunk text (if you keep it in the database), and the embedding vector. ### Why a separate table instead of a column on `document` [Section titled “Why a separate table instead of a column on document”](#why-a-separate-table-instead-of-a-column-on-document) Embedding models and LLMs enforce **context limits**. Long documents are usually **chunked**; each chunk gets its own embedding. That is naturally a **one-to-many** relationship (one document, many embedding rows), modeled with `document_id` (or similar) on the embedding table. If you only store vectors and not source text, you can drop the foreign key or keep a loose link inside **JSONB**. Both patterns are common. ## Nearest-neighbor queries [Section titled “Nearest-neighbor queries”](#nearest-neighbor-queries) The usual question is: **which stored vectors are closest to this query vector?** That is **k**-nearest neighbors (**k-NN**). In SQL you order by a distance expression and cap rows with **`LIMIT`**. Here `$1` is a parameter holding the query embedding. The **`<=>`** operator is the **cosine distance** operator when you use the cosine operator class on the column (see [Distance metrics](#distance-metrics)): ```sql SELECT * FROM document_embedding ORDER BY embedding <=> $1 LIMIT 10; ``` This returns the ten rows whose embeddings are most similar to the query under that metric. Because this is PostgreSQL, you can add **`WHERE`** filters (including on **JSONB**), **JOIN**s, and everything else you already use. ## Distance metrics [Section titled “Distance metrics”](#distance-metrics) The example above uses **cosine distance** via **`<=>`**. pgvector also supports **L2** (**`<->`**) and **inner product** (**`<#>`**) for the matching operator classes. Your **index** must be built for the same distance you use in **`ORDER BY`**, and you should follow your **embedding provider’s** guidance (many text models are tuned for cosine or inner product on **unit-normalized** vectors). In practice, **cosine distance on normalized vectors** is a solid default for many text-embedding pipelines. ### Cosine distance [Section titled “Cosine distance”](#cosine-distance) Cosine-related scores measure how aligned two directions are in high-dimensional space. For **unit vectors** (length 1), cosine distance and inner-product-based rankings line up in useful ways; pgvector’s operator-class docs spell out the exact definitions for **`<=>`** and friends. For background on cosine similarity, see a standard reference such as [Cosine similarity](https://en.wikipedia.org/wiki/Cosine_similarity) (Wikipedia). ### Inner product [Section titled “Inner product”](#inner-product) The **inner product** (dot product) scales with vector **length** as well as angle. With **unit** vectors, ordering by inner product often matches ordering by cosine similarity. ### Euclidean (L2) distance [Section titled “Euclidean (L2) distance”](#euclidean-l2-distance) **L2** is straight-line distance in the embedding space. It is sensitive to magnitude and is widely used in clustering and geometry-heavy workflows. ### When rankings match [Section titled “When rankings match”](#when-rankings-match) For **unit-length** embeddings, cosine distance, negative inner product, and Euclidean distance can yield the **same ranking** of neighbors (up to ties). That is why many teams standardize on **normalization + cosine** unless a model vendor documents a different contract. ## Why approximate indexes exist [Section titled “Why approximate indexes exist”](#why-approximate-indexes-exist) Classic multidimensional indexes (think **R-tree**-style ideas) work well at **low** dimensionality. Embeddings are often **hundreds or thousands** of dimensions, where exact structures stop scaling (**curse of dimensionality**). Exact **k-NN** over huge tables without an index is too expensive for interactive search. **ANN** indexes trade a small amount of recall for much faster queries. pgvector ships **HNSW** and **IVFFlat**; **pgvectorscale** adds **StreamingDiskANN** for large or disk-heavy workloads. Deep dive: **[pgvectorscale on GitHub](https://github.com/timescale/pgvectorscale)**. ## Index options on Tiger Cloud [Section titled “Index options on Tiger Cloud”](#index-options-on-tiger-cloud) On Tiger Cloud you typically use **pgvector** for the **`vector`** type and for **HNSW** / **IVFFlat**. **StreamingDiskANN** comes from **pgvectorscale** (it builds on pgvector). The table is a coarse comparison (your data size, recall, and update pattern matter more than any one row): | Index | Build time | Typical query speed | Rebuild after many updates? | | ------------------------------------ | ---------- | -------------------------- | ----------------------------------------------------- | | **StreamingDiskANN** (pgvectorscale) | Fast | Often strongest at scale | Usually **no** (check release notes for your version) | | **HNSW** (pgvector) | Fast | Fast | Usually **no** | | **IVFFlat** (pgvector) | Fastest | Often slowest of the three | Often **yes** (lists can go stale) | For methodology and benchmark context, see the discussion in **[pgvectorscale on GitHub](https://github.com/timescale/pgvectorscale)** and **[Understand pgvector and pgvectorscale](/docs/learn/search/pgvector-pgvectorsearch)**. ### Practical default [Section titled “Practical default”](#practical-default) If you are on Tiger Cloud and expect **large** embedding tables or strict latency goals, evaluate **StreamingDiskANN** from pgvectorscale alongside **HNSW**. For smaller datasets or simpler ops, **HNSW** alone is often enough. ## Next steps [Section titled “Next steps”](#next-steps) | Goal | Where to read | | --------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Tiger Cloud stack (pgvector + pgvectorscale + pgai) | **[Understand pgvector and pgvectorscale](/docs/learn/search/pgvector-pgvectorsearch)** | | StreamingDiskANN parameters, filtered search | **[pgvectorscale on GitHub](https://github.com/timescale/pgvectorscale)** | | Enable extensions and walk through a flow | **[Create a chatbot using pgvector](/docs/deploy/tiger-cloud/tiger-cloud-aws/tiger-cloud-extensions/pgvector)** ([Azure](/docs/deploy/tiger-cloud/tiger-cloud-azure/tiger-cloud-extensions/pgvector)) | | Lexical (BM25) search and hybrid ideas | **[Understand pg\_textsearch and BM25 search](/docs/learn/search/using-pg-textsearch)** | # Understand pgvector and pgvectorscale > Learn how vector embeddings power semantic search, how pgvector and pgvectorscale work together on Tiger Cloud, and where to read about indexes, pgai, and BM25 hybrid search. **pgvector** is a widely used open source extension for storing vectors and running similarity search in PostgreSQL. **pgvectorscale** builds on pgvector with indexing aimed at large-scale, high-performance workloads (including **StreamingDiskANN**, inspired by Microsoft’s DiskANN research). On Tiger Cloud, **pgai** is available alongside these extensions. You can rely on familiar pgvector capabilities (for example **HNSW** and **IVFFlat** indexes) and add **StreamingDiskANN** from pgvectorscale when you want to optimize vector search further. That combination makes it easier to migrate an existing pgvector deployment and still pick the index types that match your latency, recall, and cost goals. For index types, tuning, and tradeoffs, see **[pgvectorscale on GitHub](https://github.com/timescale/pgvectorscale)** (StreamingDiskANN, filtered search, and build or query parameters). ## Why embeddings matter [Section titled “Why embeddings matter”](#why-embeddings-matter) Embeddings are numeric representations of data (usually high-dimensional **vectors**). Similar meanings tend to land closer together in that space, so the database can retrieve “nearby” rows instead of relying only on exact tokens. In practice, many teams describe this as **semantic search**: matching intent and context, not just shared keywords. Pairing lexical search (keywords, BM25) with vectors is often called **hybrid search**. If you are focused on keyword ranking first, start with **[Understand pg\_textsearch and BM25 search](/docs/learn/search/using-pg-textsearch)**; this page centers on the vector side. ## Common use cases [Section titled “Common use cases”](#common-use-cases) * **Semantic search**: Return results that align with the meaning of a query even when phrasing differs from your stored text. * **Recommendations**: Suggest items whose embeddings sit close to embeddings of content the user already engaged with, beyond simple tags or categories. * **Retrieval-augmented generation (RAG)**: Fetch relevant rows from the database and pass them to an LLM as context so answers stay grounded in your data. * **Clustering and exploration**: Group entities in embedding space (for example with **k-means** or hierarchical methods) to surface structure that raw attributes might not show. ## How vector similarity search works [Section titled “How vector similarity search works”](#how-vector-similarity-search-works) At a high level: 1. **Ingest**: Generate embeddings for your content (in your application or inside the database, depending on your pipeline) and store them (typically in a `vector` column). 2. **Query**: Turn the user’s question or search text into an embedding with the **same model** you used for storage. 3. **Retrieve**: Ask the database for the stored rows whose embeddings are **closest** to the query embedding under a chosen distance metric. Internally, each embedding is a vector (a fixed-length list of floats). The database compares vectors with a **distance** or **similarity** function (common choices include **cosine** distance, **L2** (Euclidean), or **inner product**, depending on your model and index). A similarity search returns rows with the smallest distance (or highest similarity) to the query vector. ## Embedding models and limits [Section titled “Embedding models and limits”](#embedding-models-and-limits) **pgai** on Tiger Cloud integrates with popular hosted and open embedding models. Always match your table’s declared **dimensions** and your index **distance operator** to the model you use; supported providers, APIs, and limits are documented in the **[pgai repository](https://github.com/timescale/pgai)**. **Text embeddings** (examples): * **OpenAI**: Models such as **text-embedding-3-small** or **text-embedding-3-large** (and older generations like **text-embedding-ada-002**) are common choices for English and multilingual text. * **Cohere**: Several representation models support English and multilingual text. **Image and multimodal embeddings** (examples): * **OpenAI CLIP**: Useful when you need shared space for **text and images**. * **VGG** and **Vision Transformer (ViT)**: Classic and transformer-based image backbones often used when your pipeline produces image vectors. Always keep **model version**, **dimensionality**, and **distance metric** consistent between indexing and query time. ## Next steps [Section titled “Next steps”](#next-steps) | Goal | Where to read | | -------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Enable pgvector, walk through a chatbot-style flow, platform-specific notes | **[Create a chatbot using pgvector](/docs/deploy/tiger-cloud/tiger-cloud-aws/tiger-cloud-extensions/pgvector)** (same pattern on [Azure](/docs/deploy/tiger-cloud/tiger-cloud-azure/tiger-cloud-extensions/pgvector)) | | StreamingDiskANN, filtered vector search, index parameters | **[pgvectorscale on GitHub](https://github.com/timescale/pgvectorscale)** | | pgai install, vectorizer, and SQL interfaces | **[pgai on GitHub](https://github.com/timescale/pgai)** | | Tiger Cloud vectorizer worker or in-database LLM SQL deprecation (June 30, 2026) | **[Vectorizer and LLM calls migration guide](/docs/deploy/tiger-cloud/vectorizer-deprecation)** | | Lexical search with BM25 | **[Understand pg\_textsearch and BM25 search](/docs/learn/search/using-pg-textsearch)** | # Understand pg_textsearch and BM25 search > Learn why BM25 full-text search matters, how pg_textsearch ranks results, and when to open the Tiger Cloud extension guide for setup and tuning. **`pg_textsearch`** brings **BM25** full-text search into PostgreSQL so you can rank documents by relevance with SQL, close to how many web search engines score results. **v1.1.0** is **production-ready** on Tiger Cloud; the extension targets **PostgreSQL 17 and 18** (see [compatibility](https://github.com/timescale/pg_textsearch#postgresql-version-compatibility) in the open source repo). Start here if you want the **ideas** behind the extension. When you need **steps** (install the extension, create indexes, run queries, tune settings), open the Deploy guide linked at the end of this page. ## When built-in search stops being enough [Section titled “When built-in search stops being enough”](#when-built-in-search-stops-being-enough) PostgreSQL ships a capable full-text stack (`tsvector`, `tsquery`, `ts_rank`, and related types). For many apps, that stack works well. As your corpus grows, you may hit limits: ranking can diverge from what users expect, and **top-k** queries (return the *k* best matches) can get costly to compute fairly. **`pg_textsearch`** targets that gap with a dedicated BM25 index and query operators designed for **ranked retrieval at scale**. ## What BM25 does for you [Section titled “What BM25 does for you”](#what-bm25-does-for-you) **BM25** is a standard relevance function from information retrieval. **`pg_textsearch`** exposes BM25 scores as **negative** floats: **a lower (more negative) score means a better match**, which fits natural `ORDER BY ... ASC` ranking. BM25 encodes a few behaviors people usually want from keyword search: * **Rare terms count more**: the less often a term appears across your corpus, the more it boosts a match (inverse document frequency). * **Repetition levels off**: a document cannot dominate results only by repeating the same word. * **Length stays fair**: long documents do not automatically win only because they contain more words. You still write ordinary SQL; you trade some of the manual tuning of `ts_rank` for a ranking model that behaves more like familiar search products. ## What you get from the extension [Section titled “What you get from the extension”](#what-you-get-from-the-extension) In practice, **`pg_textsearch`** lets you: * **Index a text column or expression per BM25 index** with a **`text_config`** (for example `english`) so stemming and tokenization match your language. The indexed expression can concatenate columns, extract a JSONB field, or normalize text, and you can scope an index to a subset of rows with a partial `WHERE`. * **Score queries with `<@>`**, typically with `ORDER BY` and `LIMIT` for top-k results, and with explicit index naming when you filter in `WHERE` or need planner-friendly forms (see the Deploy guide). * **Operate the index** alongside the rest of your stack: parallel index builds on large tables, optional **hybrid** workflows with vector extensions (for example pgvector or pgvectorscale), and settings that control memory, segments, and top-k performance. Constraints matter too: phrase-only queries, certain compressed data paths, PL/pgSQL caveats, and other limits are **documented as product behavior**, not as tutorials on this page. Read the extension guide or the [upstream README](https://github.com/timescale/pg_textsearch/blob/main/README.md) for the full list. ## Next steps [Section titled “Next steps”](#next-steps) | Goal | Where to read | | -------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Install `pg_textsearch`, create BM25 indexes, query examples, configuration, limitations, self-hosted `shared_preload_libraries` | **[Optimize full text search with BM25](/docs/deploy/tiger-cloud/tiger-cloud-aws/tiger-cloud-extensions/pg-textsearch)** ([AWS](/docs/deploy/tiger-cloud/tiger-cloud-aws/tiger-cloud-extensions/pg-textsearch) and [Azure](/docs/deploy/tiger-cloud/tiger-cloud-azure/tiger-cloud-extensions/pg-textsearch) use the same article) | | Releases, source, and deep reference | **[pg\_textsearch on GitHub](https://github.com/timescale/pg_textsearch)** | | Same story on the public site | **[Tiger Data: Optimize full text search with BM25](https://www.tigerdata.com/docs/use-timescale/latest/extensions/pg-textsearch)** | # Tiger Cloud > Learn about Tiger Cloud, a fully managed cloud database service built on TimescaleDB and PostgreSQL for time-series and analytics workloads Tiger Cloud runs TimescaleDB on PostgreSQL as a managed service. You use hypertables, hypercore, continuous aggregates, and hyperfunctions as usual, while backups, upgrades, scaling, and infrastructure are handled for you, along with tiered storage, high-availability replicas and read replicas, and observability in Tiger Console. ## What Tiger Cloud adds [Section titled “What Tiger Cloud adds”](#what-tiger-cloud-adds) On top of the core TimescaleDB capabilities, Tiger Cloud includes: * **Tiered storage**: automatically move older data to low-cost object storage while keeping it queryable. This enables virtually unlimited data retention at a fraction of the cost of high-performance storage. * **High-availability replicas**: automatic failover replicas that keep your service online if the primary instance becomes unavailable. * **Read replicas**: offload read-heavy workloads like dashboards, reporting, and analytics queries to dedicated read replicas. * **Built-in observability**: Tiger Console provides query-level insights, resource monitoring, and alerting without additional tooling. * **Automated operations**: backups, PostgreSQL upgrades, TimescaleDB updates, and security patches are handled automatically. * **Data connectors**: ingest data from sources like S3, Kafka, and PostgreSQL using built-in connectors, and export to Apache Iceberg for lakehouse integration. ## Available on AWS and Azure [Section titled “Available on AWS and Azure”](#available-on-aws-and-azure) Tiger Cloud runs on Amazon Web Services (AWS) and Microsoft Azure, with services available in regions across North America, Europe, Asia Pacific, and South America. See [supported regions](/docs/learn/tiger-cloud/regions) for the full list of available regions by cloud provider. ## Learn more [Section titled “Learn more”](#learn-more) [Supported regions](/docs/learn/tiger-cloud/regions) [Where Tiger Cloud services are available](/docs/learn/tiger-cloud/regions) [Cloud-exclusive features](/docs/learn/tiger-cloud/cloud-exclusive-features) [Capabilities only available in Tiger Cloud](/docs/learn/tiger-cloud/cloud-exclusive-features) [Tiger Data architecture](/docs/learn/deep-dive/whitepaper) [Deep dive into the platform architecture](/docs/learn/deep-dive/whitepaper) [Feature comparison](/docs/get-started/feature-comparison) [Compare features across Tiger Data products](/docs/get-started/feature-comparison) # Cloud-exclusive features > Learn about the features and capabilities available exclusively in Tiger Cloud ## Compare TimescaleDB Community Edition and Tiger Cloud TimescaleDB Community Edition is the open-source TimescaleDB engine you install and run yourself. Tiger Cloud hosts that engine as a service and adds the [managed platform capabilities](#managed-platform-and-operations). For a full matrix including AWS, Azure, and self-hosted deployment details, see the [feature comparison](/docs/get-started/feature-comparison). ### Core engine capabilities Both offerings include the same TimescaleDB database features. | Capability | TimescaleDB Community Edition | Tiger Cloud | | -------------------------------- | :---------------------------: | :---------: | | Hypertables | ✓ | ✓ | | Hypercore (rowstore/columnstore) | ✓ | ✓ | | Continuous aggregates | ✓ | ✓ | | Full PostgreSQL compatibility | ✓ | ✓ | | Data retention policies | ✓ | ✓ | | Hyperfunctions | ✓ | ✓ | ### Managed platform and operations | Capability | TimescaleDB Community Edition | Tiger Cloud | | -------------------------------------------------------------- | :---------------------------: | :---------: | | Tiered storage (high-performance and object tiers) | ✗ | ✓ | | Iceberg connector (sync to Iceberg tables in S3) | ✗ | ✓ | | Managed source connectors (S3, Kafka, PostgreSQL) | Manual | ✓ | | Performance insights | ✗ | ✓ | | Automatic high-availability replicas and managed read replicas | Manual | ✓ | | Metrics, dashboards, and alerting | ✗ | ✓ | | Platform-managed backups, upgrades, and patching | Manual | ✓ | | SOC 2 Type II (service organization controls) | ✗ | ✓ | **Legend:** * `✓` included, * `✗` not available, * `Manual` you can approximate with self-hosted tooling and your own operations. ## Tiered storage Tiger Cloud provides a [hierarchical storage architecture](/docs/learn/data-lifecycle/storage/about-storage-tiers) with two tiers: * **High-performance tier**: block storage optimized for fast reads and writes, providing up to 64 TB and 80,000 IOPS. * **Object storage tier**: low-cost, virtually unlimited storage backed by AWS S3 or Azure Blob, using Apache Parquet format for efficient analytical queries. Tiering policies automate data movement between tiers based on age, so recent data stays fast and older data stays affordable. Tiered data remains fully queryable with standard SQL. Learn how to [manage storage](/docs/build/data-management/storage/manage-storage) and [query tiered data](/docs/build/data-management/storage/query-tiered-data). ## High-availability replicas Tiger Cloud supports automatic high-availability replicas that maintain a synchronized copy of your service. If the primary instance becomes unavailable, the replica is promoted automatically, minimizing downtime. HA replicas are available on Scale and Enterprise pricing plans. Learn more about high availability on [AWS](/docs/deploy/tiger-cloud/tiger-cloud-aws/high-availability/high-availability) or [Azure](/docs/deploy/tiger-cloud/tiger-cloud-azure/high-availability/high-availability). ## Read replicas Offload read-heavy workloads — dashboards, reporting queries, and analytics — to dedicated read replicas. Read replicas receive streaming replication from the primary and serve read-only queries independently, which helps avoid resource contention on the primary. Set up read replicas on [AWS](/docs/deploy/tiger-cloud/tiger-cloud-aws/high-availability/read-scaling) or [Azure](/docs/deploy/tiger-cloud/tiger-cloud-azure/high-availability/read-scaling). ## Built-in observability Tiger Console provides observability tools without requiring additional monitoring infrastructure: * **Insights**: analyze query performance, identify slow queries, and understand resource utilization patterns directly from Tiger Console. * **Service metrics**: monitor CPU, memory, storage, and connection usage with built-in dashboards. * **External integrations**: export metrics to CloudWatch, Datadog, Prometheus, or Azure Monitor to integrate with your existing monitoring stack. Learn more about monitoring on [AWS](/docs/deploy/tiger-cloud/tiger-cloud-aws/monitoring) or [Azure](/docs/deploy/tiger-cloud/tiger-cloud-azure/monitoring). ## Data connectors Tiger Cloud provides built-in connectors for ingesting and exporting data: * **[Source PostgreSQL connector](/docs/integrate/connectors/source/sync-from-postgres)**: replicate data from an external PostgreSQL database into your service. * **[Source S3 connector](/docs/integrate/connectors/source/sync-from-s3)**: import data from CSV and Parquet files stored in Amazon S3. * **[Kafka connector](/docs/integrate/connectors/source/sync-from-kafka)**: stream data from Apache Kafka topics directly into hypertables. * **[Iceberg connector](/docs/integrate/connectors/destination/tigerlake)**: export data from Tiger Cloud to Apache Iceberg tables for lakehouse analytics workflows. Note Connector availability varies by cloud provider and pricing plan. See the [feature comparison](/docs/get-started/feature-comparison) for details. ## Managed security and compliance Tiger Cloud includes security features managed at the platform level: * **Encryption**: data is encrypted at rest and in transit by default. * **Network isolation**: [VPC peering](/docs/deploy/tiger-cloud/tiger-cloud-aws/security/vpc) (AWS), [AWS Transit Gateway](/docs/deploy/tiger-cloud/tiger-cloud-aws/security/transit-gateway), and [Azure Private Link](/docs/deploy/tiger-cloud/tiger-cloud-azure/security/azure-privatelink) provide private connectivity between your application and service. * **SOC 2 compliance**: Tiger Cloud is SOC 2 Type II certified. Learn more about security on [AWS](/docs/deploy/tiger-cloud/tiger-cloud-aws/security/overview) or [Azure](/docs/deploy/tiger-cloud/tiger-cloud-azure/security/overview). ## Automated operations Tiger Cloud manages the operational overhead of running a production database: * **Automated backups**: continuous backups with point-in-time recovery (PITR). See backup and restore on [AWS](/docs/deploy/tiger-cloud/tiger-cloud-aws/high-availability/backup-restore) or [Azure](/docs/deploy/tiger-cloud/tiger-cloud-azure/high-availability/backup-restore). * **Upgrades**: PostgreSQL major version upgrades, TimescaleDB updates, and security patches are applied automatically with minimal disruption. See upgrades on [AWS](/docs/deploy/tiger-cloud/tiger-cloud-aws/upgrades) or [Azure](/docs/deploy/tiger-cloud/tiger-cloud-azure/upgrades). * **Scaling**: resize compute and storage independently through Tiger Console or the Tiger REST API. See scaling on [AWS](/docs/deploy/tiger-cloud/tiger-cloud-aws/service-management/change-resources) or [Azure](/docs/deploy/tiger-cloud/tiger-cloud-azure/service-management/change-resources). ## Compare features across products For a detailed side-by-side comparison of Tiger Cloud on AWS, Tiger Cloud on Azure, and self-hosted TimescaleDB, see the [feature comparison](/docs/get-started/feature-comparison). # Supported regions > Learn which regions are available for Tiger Cloud services Tiger Cloud runs on AWS and Azure. You choose the provider and region when you create a service. ## Available regions [Section titled “Available regions”](#available-regions) * AWS Tiger Cloud services run in the following Amazon Web Services (AWS) regions: | Region | Zone | Location | | ---------------- | ------------- | -------------- | | `ap-south-1` | Asia Pacific | Mumbai | | `ap-southeast-1` | Asia Pacific | Singapore | | `ap-southeast-2` | Asia Pacific | Sydney | | `ap-northeast-1` | Asia Pacific | Tokyo | | `ca-central-1` | Canada | Central | | `eu-central-1` | Europe | Frankfurt | | `eu-central-2` | Europe | Zurich | | `eu-west-1` | Europe | Ireland | | `eu-west-2` | Europe | London | | `sa-east-1` | South America | São Paulo | | `us-east-1` | United States | North Virginia | | `us-east-2` | United States | Ohio | | `us-west-2` | United States | Oregon | * Azure Tiger Cloud services run in the following Microsoft Azure regions: | Region | Location | | -------------------- | --------- | | `eastus2` | Virginia | | `westeurope` | Amsterdam | | `germanywestcentral` | Frankfurt | | `southeastasia` | Singapore | ## Choosing a region [Section titled “Choosing a region”](#choosing-a-region) * **Proximity**: pick a region near your app or users to keep network latency low. * **Data residency**: meet regulatory or compliance needs (for example EU storage for GDPR). * **Existing infrastructure**: matching your current AWS or Azure footprint cuts transfer cost and latency. * **Features**: some options (for example VPC peering, AWS Transit Gateway) depend on the provider. See the [feature comparison](/docs/get-started/feature-comparison). ## Cross-region backups [Section titled “Cross-region backups”](#cross-region-backups) On AWS, you can store backup copies in a different region than your primary service for extra resilience to regional outages. # Tiger Cloud essentials > Technical walkthrough covering hypertables, hypercore, continuous aggregates, and tiered storage in a single workflow Tiger Cloud offers managed database services that provide a stable and reliable environment for your applications. Each Tiger Cloud service is a single optimised PostgreSQL instance extended with innovations such as TimescaleDB in the database engine, in a cloud infrastructure that delivers speed without sacrifice. A radically faster PostgreSQL for transactional, analytical, and agentic workloads at scale. Tiger Cloud scales PostgreSQL to ingest and query vast amounts of live data. Tiger Cloud provides a range of features and optimizations that supercharge your queries while keeping the costs down. For example: * The hypercore row-columnar engine in TimescaleDB makes queries up to 350x faster, ingests 44% faster, and reduces storage by 90%. * Tiered storage in Tiger Cloud seamlessly moves your data from high performance storage for frequently accessed data to low cost bottomless storage for rarely accessed data. The following figure shows how TimescaleDB optimizes your data for superfast real-time analytics: ![Data mutability in hypercore: inserts, updates, and deletes are routed to an interim rowstore chunk that periodically recompresses into the columnstore, supporting full mutability without rewriting compressed data](/docs/_astro/hypercore-data-mutability-light.CNUSPO4Z_Z1AvahI.webp) ![Data mutability in hypercore: inserts, updates, and deletes are routed to an interim rowstore chunk that periodically recompresses into the columnstore, supporting full mutability without rewriting compressed data](/docs/_astro/hypercore-data-mutability-dark.0BwC6tck_Z1u7eJM.webp) This page shows you how to rapidly implement the features in Tiger Cloud that enable you to ingest and query data faster while keeping the costs low. ## Prerequisites for this page To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service). ## Optimize time-series data in hypertables with hypercore [Section titled “Optimize time-series data in hypertables with hypercore”](#optimize-time-series-data-in-hypertables-with-hypercore) Time-series data represents the way a system, process, or behavior changes over time. Hypertables are PostgreSQL tables that help you improve insert and query performance by automatically partitioning your data by time. Each hypertable is made up of child tables called chunks. Each chunk is assigned a range of time, and only contains data from that range. When you run a query, TimescaleDB identifies the correct chunk and runs the query on it, instead of going through the entire table. You can also tune hypertables to increase performance even more. ![Hypertable structure](/docs/_astro/hypertable-structure.Ze5G6DtA_IDjHi.webp) ![Hypertable structure](/docs/_astro/hypertable-structure-dark.QO9KP5rD_1SGIbH.webp) [Hypercore](/docs/learn/columnar-storage/understand-hypercore) is the hybrid row-columnar storage engine in TimescaleDB used by hypertables. Traditional databases force a trade-off between fast inserts (row-based storage) and efficient analytics (columnar storage). Hypercore eliminates this trade-off, allowing real-time analytics without sacrificing transactional capabilities. Hypercore dynamically stores data in the most efficient format for its lifecycle: ![Move from rowstore to columstore in hypercore](/docs/_astro/hypercore_intro.DmLiqZGV_1VM5z7.webp) ![Move from rowstore to columstore in hypercore](/docs/_astro/hypercore_intro-dark.qPmjkBqW_19V2rF.webp) * **Row-based storage for recent data**: the most recent chunk (and possibly more) is always stored in the rowstore, ensuring fast inserts, updates, and low-latency single record queries. Additionally, row-based storage is used as a writethrough for inserts and updates to columnar storage. * **Columnar storage for analytical performance**: chunks are automatically compressed into the columnstore, optimizing storage efficiency and accelerating analytical queries. Unlike traditional columnar databases, hypercore allows data to be inserted or modified at any stage, making it a flexible solution for both high-ingest transactional workloads and real-time analytics, within a single database. Hypertables exist alongside regular PostgreSQL tables. You use regular PostgreSQL tables for relational data, and interact with hypertables and regular PostgreSQL tables in the same way. This section shows you how to create regular tables and hypertables, and import relational and time-series data from external files. 1. **Import some time-series data into hypertables** 1. Unzip [crypto\_sample.zip](https://assets.timescale.com/docs/downloads/candlestick/crypto_sample.zip) to a ``. This test dataset contains: * Second-by-second data for the most-traded crypto-assets. This time-series data is best suited for optimization in a [hypertable](/docs/learn/hypertables/understand-hypertables). * A list of asset symbols and company names. This is best suited for a regular relational table. To import up to 100 GB of data directly from your current PostgreSQL-based database, [migrate with downtime](/docs/migrate/migrate-with-downtime) using native PostgreSQL tooling. To seamlessly import 100GB-10TB+ of data, use the [live migration](/docs/migrate/live-migration) tooling supplied by Tiger Data. To add data from non-PostgreSQL data sources, see [Import and ingest data](/docs/migrate). 2. Upload data into a hypertable: To more fully understand how to create a hypertable, how hypertables work, and how to optimize them for performance by tuning chunk intervals and enabling chunk skipping, see [the hypertables documentation](/docs/learn/hypertables/understand-hypertables). * Tiger Console Tiger Console data upload creates hypertables and relational tables from the data you are uploading: 1. In [Tiger Console](https://console.cloud.tigerdata.com/dashboard/services), select the service to add data to, then click `Actions` > `Import data` > `Upload .CSV`. 2. Click to browse, or drag and drop `/tutorial_sample_tick.csv` to upload. 3. Leave the default settings for the delimiter, skipping the header, and creating a new table. 4. In `Table`, provide `crypto_ticks` as the new table name. 5. Enable `hypertable partition` for the `time` column and click `Process CSV file`. The upload wizard creates a hypertable containing the data from the CSV file. 6. When the data is uploaded, close `Upload .CSV`. If you want to have a quick look at your data, press `Run`. 7. Repeat the process with `/tutorial_sample_assets.csv` and rename to `crypto_assets`. There is no time-series data in this table, so you don't see the `hypertable partition` option. * psql 1. In the terminal, navigate to `` and connect to your service. ```bash psql -d "postgres://:@:/" ``` You use your [connection details](/docs/integrate/find-connection-details) to fill in this PostgreSQL connection string. 2. Create tables for the data to import: * For the time-series data: In your sql client, create a hypertable: Create a [hypertable](/docs/learn/hypertables/understand-hypertables) for your time-series data using CREATE TABLE. For [efficient queries](/docs/build/performance-optimization/secondary-indexes), remember to `segmentby` the column you will use most often to filter your data. For example: ```sql CREATE TABLE crypto_ticks ( "time" TIMESTAMPTZ, symbol TEXT, price DOUBLE PRECISION, day_volume NUMERIC ) WITH ( tsdb.hypertable, tsdb.segmentby = 'symbol' ); ``` When you create a hypertable using [CREATE TABLE ... WITH ...](/docs/reference/timescaledb/hypertables/create_table), the default partitioning column is automatically the first column with a timestamp data type. Also, TimescaleDB creates a [columnstore policy](/docs/reference/timescaledb/hypercore/add_columnstore_policy) that automatically converts your data to the columnstore, after an interval equal to the value of the [chunk\_interval](/docs/reference/timescaledb/hypertables/set_chunk_time_interval), defined through `after` in the policy. This columnar format enables fast scanning and aggregation, optimizing performance for analytical workloads while also saving significant storage space. In the columnstore conversion, hypertable chunks are compressed by up to 98%, and organized for efficient, large-scale queries. You can customize this policy later using [alter\_job](/docs/reference/timescaledb/jobs-automation/alter_job). However, to change `after` or `created_before`, the compression settings, or the hypertable the policy is acting on, you must [remove the columnstore policy](/docs/reference/timescaledb/hypercore/remove_columnstore_policy) and [add a new one](/docs/reference/timescaledb/hypercore/add_columnstore_policy). You can also manually [convert chunks](/docs/reference/timescaledb/hypercore/convert_to_columnstore) in a hypertable to the columnstore. * For the relational data: In your sql client, create a normal PostgreSQL table: ```sql CREATE TABLE crypto_assets ( symbol TEXT NOT NULL, name TEXT NOT NULL ); ``` 3. Speed up data ingestion: When you set `timescaledb.enable_direct_compress_insert` or `timescaledb.enable_direct_compress_copy`, TimescaleDB compresses data in memory during ingestion and writes the compressed batches directly to the columnstore. This significantly reduces the I/O footprint, and `INSERT` and `COPY` produce columnstore chunks on the spot — the [columnstore policy](/docs/reference/timescaledb/hypercore/add_columnstore_policy) is less important. Warning This feature is a **tech preview** and not production-ready. Using direct compress can regress query performance or storage ratio if the ingested rows are not sorted by the table's `orderby` columns or if the data has very high cardinality. To enable in-memory compression during ingestion, set the appropriate GUC: ```sql SET timescaledb.enable_direct_compress_insert = on; -- or SET timescaledb.enable_direct_compress_copy = on; ``` **Important facts** * High cardinality data does not produce good batches and degrades query performance. * The columnstore is optimized for 1000 rows per batch per `segmentby` value. * WAL records are written for the compressed batches rather than the individual tuples. * Best results are achieved for batch ingestion with 1000 rows or more. * Tables with unique constraints cannot use direct compress. * If your hypertable has no `segmentby` column configured, TimescaleDB picks one for you based on the first batch of rows you ingest. Turn this off with `SET timescaledb.enable_direct_compress_auto_segmentby = off`. 4. Upload the dataset to your service: ```sql \COPY crypto_ticks from './tutorial_sample_tick.csv' DELIMITER ',' CSV HEADER; ``` ```sql \COPY crypto_assets from './tutorial_sample_assets.csv' DELIMITER ',' CSV HEADER; ``` 2. **Have a quick look at your data** You query hypertables in exactly the same way as you would a relational PostgreSQL table. Use one of the following SQL editors to run a query and see the data you uploaded: * **Data view**: write queries, visualize data, and share your results in [Tiger Console](https://console.cloud.tigerdata.com/dashboard/services?popsql) for all your Tiger Cloud services. * **SQL editor**: write, fix, and organize SQL faster and more accurately in [Tiger Console](https://console.cloud.tigerdata.com/dashboard/services) for a Tiger Cloud service. * **psql**: easily run queries on your Tiger Cloud services or self-hosted TimescaleDB deployment from the terminal. ```sql SELECT * FROM crypto_ticks ORDER BY time DESC LIMIT 10; ``` ## Write fast and efficient analytical queries [Section titled “Write fast and efficient analytical queries”](#write-fast-and-efficient-analytical-queries) Aggregation is a way of combing data to get insights from it. Average, sum, and count are all examples of simple aggregates. However, with large amounts of data, aggregation slows things down, quickly. Continuous aggregates are a kind of hypertable that is refreshed automatically in the background as new data is added, or old data is modified. Changes to your dataset are tracked, and the hypertable behind the continuous aggregate is automatically updated in the background. ![How a continuous aggregate incrementally materializes time-bucketed rollups from a raw hypertable, with a real-time aggregate query reading both pre-computed and recent raw data](/docs/_astro/continuous-aggregates-light.BhW3GFu2_pFsfL.webp) ![How a continuous aggregate incrementally materializes time-bucketed rollups from a raw hypertable, with a real-time aggregate query reading both pre-computed and recent raw data](/docs/_astro/continuous-aggregates-dark.CnXJ1DFs_2kSGiL.webp) You create continuous aggregates on uncompressed data in high-performance storage. They continue to work on [data in the columnstore](#write-fast-and-efficient-analytical-queries) and [rarely accessed data in tiered storage](#slash-storage-charges). You can even create [continuous aggregates on top of your continuous aggregates](/docs/learn/continuous-aggregates/hierarchical-continuous-aggregates). You use time buckets to create a continuous aggregate. Time buckets aggregate data in hypertables by time interval. For example, a 5-minute, 1-hour, or 3-day bucket. The data grouped in a time bucket uses a single timestamp. Continuous aggregates minimize the number of records that you need to look up to perform your query. This section shows you how to run fast analytical queries using time buckets and continuous aggregates in Tiger Console. You can also do this using psql. * Data view 1. **In Tiger Console, toggle Data view at the top right** 2. **Connect to your service** In the connection drop-down in the top right, select your service. 3. **Create a continuous aggregate** For a continuous aggregate, data grouped using a time bucket is stored in a PostgreSQL `MATERIALIZED VIEW` in a hypertable. `timescaledb.continuous` ensures that this data is always up to date. In Data view, use the following code to create a continuous aggregate on the real-time data in the `crypto_ticks` table: ```sql CREATE MATERIALIZED VIEW assets_candlestick_daily WITH (timescaledb.continuous) AS SELECT time_bucket('1 day', "time") AS day, symbol, max(price) AS high, first(price, time) AS open, last(price, time) AS close, min(price) AS low FROM crypto_ticks srt GROUP BY day, symbol; ``` This continuous aggregate creates the [candlestick chart](https://www.investopedia.com/terms/c/candlestick.asp) data you use to visualize the price change of an asset. 4. **Create a policy to refresh the view every hour** ```sql SELECT add_continuous_aggregate_policy('assets_candlestick_daily', start_offset => INTERVAL '3 weeks', end_offset => INTERVAL '24 hours', schedule_interval => INTERVAL '3 hours'); ``` 5. **Have a quick look at your data** You query continuous aggregates exactly the same way as your other tables. To query the `assets_candlestick_daily` continuous aggregate for all assets: ```sql SELECT * FROM assets_candlestick_daily ORDER BY day DESC, symbol LIMIT 10; ``` * Continuous aggregate wizard 1. **In Tiger Console, select the service you uploaded data to** 2. **Click `Explorer` > `Continuous Aggregates` > `Create a Continuous Aggregate` next to the `crypto_ticks` hypertable** 3. **Create a view called `assets_candlestick_daily` on the `time` column with an interval of `1 day`, then click `Next step`** ![Continuous aggregate wizard](/docs/_astro/continuous-aggregate-wizard-tiger-console.uDYTFl9A_2cOj4P.webp) 4. **Update the view SQL with the following functions, then click `Run`** ```sql CREATE MATERIALIZED VIEW assets_candlestick_daily WITH (timescaledb.continuous) AS SELECT time_bucket('1 day', "time") AS bucket, symbol, max(price) AS high, first(price, time) AS open, last(price, time) AS close, min(price) AS low FROM "public"."crypto_ticks" srt GROUP BY bucket, symbol; ``` 5. **When the view is created, click `Next step`** 6. **Define a refresh policy with the following values:** * `How far back do you want to materialize?`: `3 weeks` * `What recent data to exclude?`: `24 hours` * `How often do you want the job to run?`: `3 hours` 7. **Click `Next step`, then click `Run`** Tiger Cloud creates the continuous aggregate and displays the aggregate ID in Tiger Console. Click `DONE` to close the wizard. To see the change in terms of query time and data returned between a regular query and a continuous aggregate, run the query part of the continuous aggregate ( `SELECT ...GROUP BY day, symbol;` ) and compare the results. ## Slash storage charges [Section titled “Slash storage charges”](#slash-storage-charges) In the previous sections, you used continuous aggregates to make fast analytical queries, and hypercore to reduce storage costs on frequently accessed data. To reduce storage costs even more, you create tiering policies to move rarely accessed data to the object store. The object store is low-cost bottomless data storage built on Amazon S3 or Azure Blob storage. However, no matter the tier, you can [query your data when you need](/docs/build/data-management/storage/query-tiered-data). Tiger Cloud seamlessly accesses the correct storage tier and generates the response. ![Tiered storage in Tiger Cloud: a hypertable's rowstore and columnstore chunks live on high-performance block storage, while older chunks tier to low-cost bottomless object storage and remain transparently queryable](/docs/_astro/tiered-storage-light.DWY9mRTx_Z2qmhC0.webp) ![Tiered storage in Tiger Cloud: a hypertable's rowstore and columnstore chunks live on high-performance block storage, while older chunks tier to low-cost bottomless object storage and remain transparently queryable](/docs/_astro/tiered-storage-dark.D_8Dc-sh_1E6TXQ.webp) To set up data tiering: 1. **Enable data tiering** 1. In [Tiger Console](https://console.cloud.tigerdata.com/dashboard/services), select the service to modify. 2. In `Explorer`, click `Data tiering` > `Enable tiered storage`. ![Enable tiered storage](/docs/_astro/enable-tiered-storage-tiger-console.DOgO4kJn_Z14tATB.webp) When tiered storage is enabled, you see the amount of data in the tiered object storage. 2. **Set the time interval when data is tiered** In Tiger Console, toggle Data view, then set up a tiering policy on a hypertable with the following query: ```sql SELECT add_tiering_policy('assets_candlestick_daily', INTERVAL '3 weeks'); ``` 3. **Query tiered data** You enable reads from tiered data for each query, for a session or for all future sessions. To run a single query on tiered data: 1. Enable reads on tiered data: ```sql set timescaledb.enable_tiered_reads = true ``` 2. Query the data: ```sql SELECT * FROM crypto_ticks srt LIMIT 10 ``` 3. Disable reads on tiered data: ```sql set timescaledb.enable_tiered_reads = false; ``` For more information, see [Querying tiered data](/docs/build/data-management/storage/query-tiered-data). ## Reduce the risk of downtime and data loss [Section titled “Reduce the risk of downtime and data loss”](#reduce-the-risk-of-downtime-and-data-loss) By default, all Tiger Cloud services have rapid recovery enabled. However, if your app has very low tolerance for downtime, Tiger Cloud offers high-availability replicas. HA replicas are exact, up-to-date copies of your database hosted in multiple AWS availability zones (AZ) within the same region as your primary node. HA replicas automatically take over operations if the original primary data node becomes unavailable. The primary node streams its write-ahead log (WAL) to the replicas to minimize the chances of data loss during failover. 1. **In Tiger Console, select the service to enable replication for.** 2. **Click `Operations`, then select `High availability`.** 3. **Choose your replication strategy, then click `Change configuration`.** ![High availability replica configuration in Tiger Console](/docs/_astro/tiger-console-ha-replicas.BsuEJHbW_Znnyey.webp) 4. **In `Change high availability configuration`, click `Change config`.** For more information, see [High-availability replica](/docs/deploy/tiger-cloud/tiger-cloud-aws/high-availability/high-availability). ## Tiger Cloud support [Section titled “Tiger Cloud support”](#tiger-cloud-support) Tiger Data runs a global support organization with Customer Satisfaction (CSAT) scores above 99%. Support covers all timezones and is fully staffed at weekend hours. All paid pricing plans have free Developer Support through email with a target response time of 1 business day; we are often faster. If you need 24x7 responsiveness, talk to us about [Production Support](https://www.tigerdata.com/support). You can open, view, reply to, and close support tickets from the `Support` tab in Tiger Console: ![Manage support in Tiger Cloud](/docs/_astro/tiger-cloud-manage-support.oAio5bba_Z1Ot3bO.webp) What next? See [Build](/docs/build) for more advanced tutorials, interact with the data in your Tiger Cloud service using [your favorite programming language](/docs/integrate/code/connect-your-app), integrate your Tiger Cloud service with a range of [third-party tools](/docs/integrate), [build with Tiger Data products](/docs/build), or dive into [the API reference](/docs/reference). # Data > Postgres for time-series, real-time analytics and events Quickstart [Begin your journey with TimescaleDB. Learn the basics of time-series data management, create your first hypertable, and understand how to read and write time-series data efficiently.](/docs/get-started/quickstart/quickstart-5-minutes) By use case: [(todo)](/docs/build/examples/simulate-iot-sensor-data) [(todo)](/docs/build/examples/analyze-financial-tick-data) By feature: [Explore TimescaleDB's architecture and design principles. Learn how hypertables work, understand chunk-based partitioning, and discover best practices for designing your data model.](/docs/learn/hypertables/understand-hypertables) [Master TimescaleDB's powerful data management features. Work with hypertables, leverage Hypercore's columnar storage, create continuous aggregates, optimize schemas, and automate tasks with background jobs.](/docs/learn/columnar-storage/understand-hypercore) [Get your data into TimescaleDB efficiently. Migrate from existing databases, import data from the console or terminal, and set up continuous data ingestion pipelines.](/docs/learn/continuous-aggregates) [Resolve common issues and optimize performance. Find solutions for query problems, data ingestion challenges, and configuration issues in your TimescaleDB deployment.](/docs/reference/timescaledb/hyperfunctions) [(todo)](/docs/reference/timescaledb/data-retention) # Migrate to Tiger Cloud > Move your data to Tiger Cloud from other databases Whether you're moving an existing database to Tiger Cloud or loading data from files and streaming sources, this section has you covered. There are three ways to get your data in: a one-time [migration](/docs/migrate/livesync-replication) that ends with Tiger Cloud as your primary database, ongoing [sync and stream](/docs/migrate/livesync-for-postgresql) that keeps your source primary and Tiger Cloud as a live replica, or a one-off [file upload](/docs/migrate/import-console). Not sure which fits your source database, size, and downtime budget? See [Choose a migration approach](/docs/migrate/choose-your-approach) to compare the tools, or jump straight to a method below. ## I want to... [Section titled “I want to...”](#i-want-to) [Migrate to Tiger Cloud](/docs/migrate/livesync-replication) [Make a one-time move from PostgreSQL or TimescaleDB to Tiger Cloud and cut over.](/docs/migrate/livesync-replication) [Sync and stream](/docs/migrate/livesync-for-postgresql) [Continuously replicate data from PostgreSQL, S3, or Kafka into Tiger Cloud while the source stays primary.](/docs/migrate/livesync-for-postgresql) [Upload files](/docs/migrate/import-console) [Load CSV, Parquet, text, and MySQL files into a service from your machine or S3.](/docs/migrate/import-console) ## Troubleshooting [Section titled “Troubleshooting”](#troubleshooting) Running into issues? See [FAQ and troubleshooting](/docs/migrate/troubleshooting) for common migration problems and solutions. # Choose a migration approach > Compare the tools for moving data to Tiger Cloud and pick the right approach for your source database, size, and downtime budget This guide helps you pick a migration approach based on your source database, its size, and how much downtime you can tolerate. Once you have chosen an approach, follow the linked guide for step-by-step instructions. For PostgreSQL and TimescaleDB sources, [Livesync replication](/docs/migrate/livesync-replication) is the default low-downtime path. For non-PostgreSQL sources, use dual writes and backfill for low downtime, or a CSV export and backfill when downtime is acceptable. For small databases where downtime is acceptable, [migrating with downtime](/docs/migrate/migrate-with-downtime) using `pg_dump` is the simplest option. ## Migrate from PostgreSQL or TimescaleDB [Section titled “Migrate from PostgreSQL or TimescaleDB”](#migrate-from-postgresql-or-timescaledb) [Livesync replication](/docs/migrate/livesync-replication) is the recommended default for most migrations. The table below shows when another approach fits better. | Scenario | Approach | Tool | Estimated effort | | ----------------------------------------------- | ----------------------------------------------- | ------------------------------------------------------------------------------------------------------------ | --------------------- | | Low or zero downtime, any size | Continuous sync | [Livesync replication](/docs/migrate/livesync-replication) | Days (setup and sync) | | Low downtime with very large tables (over 1 TB) | Hybrid: sync new data, backfill historical data | [Livesync replication](/docs/migrate/livesync-replication) plus a [backfill tool](#backfill-tool-comparison) | Days to weeks | | Under 100 GB, downtime acceptable | Full dump and restore | [Migrate with downtime](/docs/migrate/migrate-with-downtime) (`pg_dump`) | Hours | ## Migrate from a non-PostgreSQL database [Section titled “Migrate from a non-PostgreSQL database”](#migrate-from-a-non-postgresql-database) For non-PostgreSQL sources, the application layer handles the transition. New data is written to both the old and new systems while historical data is exported and backfilled. | Scenario | Approach | New data | Historical data | | ----------------------------- | ----------------------------------------------------------------- | ---------------------------------------------- | ------------------------------------------------------------- | | Low or zero downtime required | [Dual writes and backfill](/docs/migrate/dual-write-and-backfill) | Application writes to both old and new systems | CSV export, then a [backfill tool](#backfill-tool-comparison) | | Downtime acceptable | Offline CSV export and import | Stop writes, then cut over | CSV export, then a [backfill tool](#backfill-tool-comparison) | ## Backfill tool comparison [Section titled “Backfill tool comparison”](#backfill-tool-comparison) When you backfill historical data, choose the tool based on your source data type and infrastructure. | Tool | Source type | Requires S3 | Columnstore chunks | Managed UI | Notes | | ------------------------------------------------------------------------------------- | -------------- | ----------- | ------------------ | ---------- | ------------------------------------------------- | | [`timescaledb-backfill`](/docs/migrate/dual-write-and-backfill/timescaledb-backfill) | Hypertable | No | Yes | No | Requires matching TimescaleDB versions | | [S3 connector](/docs/integrate/connectors/source/sync-from-s3) | CSV or Parquet | Yes | N/A | Yes | Preferred for CSV or Parquet when S3 is available | | [`timescaledb-parallel-copy`](https://github.com/timescale/timescaledb-parallel-copy) | CSV | No | N/A | No | Self-hosted; more manual control over parallelism | ## Tool reference [Section titled “Tool reference”](#tool-reference) ### Livesync replication [Section titled “Livesync replication”](#livesync-replication) Recommended for PostgreSQL or TimescaleDB sources with a low or near-zero downtime requirement, at any database size. **Capabilities:** * Continuous replication with low downtime. * PostgreSQL and TimescaleDB versions do not need to match between source and target. * Handles multi-tenant consolidation and splitting (many databases to one, or multi-tenant logical databases to a schema per tenant). * Can convert native partitions (`pg_partman` or similar) to hypertables on the fly. * Supports phased migration: selectively add or remove tables, and stream only new data while you backfill historical data separately. **Limitations:** * Schema migration is handled separately. Use your existing schema management solution, or fall back to `pg_dump --schema-only`. * Sequences must be updated manually during cutover. * Large tables require a backfill plan for historical data. * More operational setup and monitoring than a simple downtime migration. **Prerequisites:** * Network connectivity from the source to Tiger Cloud (firewall rules, VPN, or peering). * A PostgreSQL user with replication privileges on the source. * The schema pre-created on the target (using a schema management tool or `pg_dump --schema-only`). * Sufficient disk on the target to accommodate incoming data during sync. **Cutover sequence:** once Livesync replication has caught up and lag is minimal: 1. Verify that sync lag is near zero. 2. Stop writes to the source database by putting the application in read-only or maintenance mode. 3. Wait for the final sync to complete. This typically takes seconds. 4. Update sequences on the target to match or exceed the source values. 5. Validate row counts and spot-check data integrity. 6. Switch application connection strings to Tiger Cloud. 7. Verify application health, then tear down the sync. For setup instructions, see [Livesync replication](/docs/migrate/livesync-replication) and [Advanced topics](/docs/migrate/livesync-replication-advanced). If sync falls behind, see [Troubleshooting](/docs/migrate/livesync-replication-troubleshooting). ### Migrate with downtime using pg\_dump [Section titled “Migrate with downtime using pg\_dump”](#migrate-with-downtime-using-pg_dump) Best for databases under 100 GB where downtime is acceptable. This is the simplest possible migration path. **Capabilities:** * Well-understood, standard PostgreSQL tooling with broad community support, using [`pg_dump`](https://www.postgresql.org/docs/current/app-pgdump.html) and [`pg_restore`](https://www.postgresql.org/docs/current/app-pgrestore.html). * Single-command dump and restore, with minimal operational overhead. **Limitations:** * Requires full downtime during dump and restore. * Fragile with large datasets: network interruptions or transient errors require restarting from scratch. * Not suitable for databases approaching or exceeding 100 GB, because of the extended downtime window. **Rough time estimates:** | Database size | Approximate downtime | | ------------- | -------------------- | | Under 10 GB | 30 minutes to 1 hour | | 10 to 50 GB | 1 to 4 hours | | 50 to 100 GB | 4 to 12 hours | Times vary significantly based on network bandwidth, table complexity, and index count. For step-by-step instructions, see [Migrate with downtime](/docs/migrate/migrate-with-downtime). ### Live migration [Section titled “Live migration”](#live-migration) Warning Live migration is deprecated in favor of [Livesync replication](/docs/migrate/livesync-replication). It is not recommended for new migrations. **Capabilities:** * Migrates both schema and data in a single operation. * Low downtime with continuous sync. * Can copy chunks in the columnstore directly, without converting them back to the rowstore. **Limitations:** * Strict version matching is required: the TimescaleDB version must match between source and target. * Only supports 1:1 migrations, with no multi-tenant consolidation or splitting. If you are continuing an existing migration with this tool, see [Live migration](/docs/migrate/live-migration) and its [FAQ and troubleshooting](/docs/migrate/troubleshooting). ### Dual writes and backfill [Section titled “Dual writes and backfill”](#dual-writes-and-backfill) Use when the source is a non-PostgreSQL database (for example, MySQL, DynamoDB, or InfluxDB) and low or near-zero downtime is required. The application is modified to write to both the old database and Tiger Cloud simultaneously. While dual writes handle new data, historical data is exported to CSV and loaded with a backfill tool. Once historical data is loaded and validated, the application cuts over to Tiger Cloud exclusively. **Capabilities:** * Enables near-zero downtime for non-PostgreSQL sources. * Allows a phased cutover, with time for data validation and reconciliation. **Limitations:** * Requires application-level code changes to implement dual writes. * Careful validation and reconciliation are needed to ensure data consistency between the old and new systems. * More operationally complex than offline approaches. For step-by-step instructions, see [Dual-write and backfill](/docs/migrate/dual-write-and-backfill). ### CSV export and backfill [Section titled “CSV export and backfill”](#csv-export-and-backfill) Use when downtime is acceptable and the source can export to CSV. This is the simplest path for non-PostgreSQL migrations. **Capabilities:** * Operationally simpler than dual-write approaches. * Works for any source database that can export CSV. **Limitations:** * Full downtime during the export and import cycle. * Not suitable when near-zero downtime is a requirement. Export your data to CSV, then load it with the [S3 connector](/docs/integrate/connectors/source/sync-from-s3) or [`timescaledb-parallel-copy`](https://github.com/timescale/timescaledb-parallel-copy). See the [Backfill tool comparison](#backfill-tool-comparison) to choose between them. For a worked `timescaledb-parallel-copy` example in a backfill, see [Dual-write from other databases](/docs/migrate/dual-write-and-backfill/dual-write-from-other). ## Backfill best practices [Section titled “Backfill best practices”](#backfill-best-practices) Whichever backfill tool you use, follow these practices for a smooth and performant data load. * **Insert in chronological order.** Always load data from oldest to newest. This aligns with the natural order of chunk creation in TimescaleDB and significantly improves write performance. * **Tune batch size and parallelization.** Insert data in manageable batches to avoid overwhelming memory or exceeding transaction limits. A batch size of 10,000 to 100,000 rows per insert is a good starting point. Use parallel workers to improve throughput, but ensure time ranges across workers do not overlap, to prevent contention. * **Pause retention policies and automated jobs.** Pause retention policies and any automated background jobs, such as continuous aggregate refresh, during the backfill. This prevents data from being deleted or transformed mid-load. * **Disable the columnstore during backfill.** Writing into the columnstore adds processing overhead and can significantly slow ingestion. Re-enable columnstore policies after the backfill is complete and validated. * **Start small, then scale up.** Begin with a small batch, such as one day of data, to observe how the target handles the load. If performance looks good and resources are not strained, increase the batch size to cover larger time ranges, reducing the total number of import cycles. Important If the target service is already handling production traffic, [open a support ticket](https://console.cloud.tigerdata.com/dashboard/support) before starting a large backfill to ensure sufficient resource provisioning for the additional ingestion load. ## Frequently asked questions [Section titled “Frequently asked questions”](#frequently-asked-questions) ### What if the source is Amazon RDS or Aurora? [Section titled “What if the source is Amazon RDS or Aurora?”](#what-if-the-source-is-amazon-rds-or-aurora) RDS and Aurora are PostgreSQL-compatible, so the PostgreSQL decision flow applies. [Livesync replication](/docs/migrate/livesync-replication) works with RDS and Aurora as long as logical replication is enabled. Set `rds.logical_replication = 1` in the parameter group. Aurora may require a reboot after you enable this setting. ### What about foreign keys and constraints during backfill? [Section titled “What about foreign keys and constraints during backfill?”](#what-about-foreign-keys-and-constraints-during-backfill) For large backfills, consider temporarily disabling foreign key checks and non-essential indexes on the target to improve throughput. Re-enable and validate them after the backfill completes. Document which constraints you disable so nothing is missed. ### What if Livesync replication falls behind? [Section titled “What if Livesync replication falls behind?”](#what-if-livesync-replication-falls-behind) Monitor replication lag. If lag grows persistently, check for long-running transactions on the source, network bandwidth bottlenecks, or target write throughput limits. In extreme cases, you may need to increase target service resources or pause non-critical workloads during catch-up. See [Livesync replication troubleshooting](/docs/migrate/livesync-replication-troubleshooting). ### What is the rollback strategy? [Section titled “What is the rollback strategy?”](#what-is-the-rollback-strategy) * **Livesync replication and live migration:** the source database remains untouched throughout. If the migration fails, the application continues using the source and no data is lost. * **`pg_dump`:** the source is read-only during the dump. If the restore fails on the target, the source is still intact. Retry the dump and restore cycle. * **Dual writes:** disable the dual-write code path and revert to writing to the old database only. Because the application is already writing to both, rolling back is a configuration change. ### Can I migrate only specific tables or schemas? [Section titled “Can I migrate only specific tables or schemas?”](#can-i-migrate-only-specific-tables-or-schemas) Yes. [Livesync replication](/docs/migrate/livesync-replication) supports selective table migration. You can add or remove tables from the sync configuration at any time. `pg_dump` also supports per-table and per-schema exports with the `-t` and `-n` flags. ### What about extensions? [Section titled “What about extensions?”](#what-about-extensions) Check that all required PostgreSQL extensions are available on Tiger Cloud before you start. Most common extensions, such as PostGIS and `pg_stat_statements`, are supported. See [PostgreSQL extensions](/docs/deploy/tiger-cloud/tiger-cloud-aws/tiger-cloud-extensions) for the full list, and [open a support ticket](https://console.cloud.tigerdata.com/dashboard/support) for uncommon extensions. # Dual-write and backfill > Migrate a hypertable or your entire database to Tiger Cloud with low downtime using dual-write and backfill Dual-write and backfill is a migration strategy to move a large amount of time-series data (100 GB-10 TB+) with low downtime (on the order of minutes of downtime). It is significantly more complicated to execute than a migration with downtime using [pg\_dump/restore](/docs/migrate/migrate-with-downtime), and has some prerequisites on the data ingest patterns of your application, so it may not be universally applicable. Note In the context of migrations, your existing production database is referred to as the SOURCE database, the Tiger Cloud service that you are migrating your data to is the TARGET. Roughly, it consists of three steps: 1. Clone schema and relational data from source to target 2. Dual-write to source and target 3. Backfill time-series data Dual-write and backfill can be used for any source database type, as long as it can provide data in csv format. It can be used to move data from a PostgreSQL source, and from TimescaleDB to TimescaleDB. Dual-write and backfill works well when: * The bulk of the (on-disk) data is in time-series tables. * Writes by the application do not reference historical time-series data. * Writes to time-series data are append-only. * No `UPDATE` or `DELETE` queries will be run on time-series data in the source database during the migration process (or if they are, it happens in a controlled manner, such that it's possible to either ignore, or re-backfill). * Either the relational (non-time-series) data is small enough to be copied from source to target in an acceptable amount of time for this to be done with downtime, or the relational data can be copied asynchronously while the application continues to run (that is, changes relatively infrequently). For more information, consult the step-by-step guide for your source database: * [Dual-write and backfill from TimescaleDB](/docs/migrate/dual-write-and-backfill/dual-write-from-timescaledb) * [Dual-write and backfill from PostgreSQL](/docs/migrate/dual-write-and-backfill/dual-write-from-postgres) * [Dual-write and backfill from other](/docs/migrate/dual-write-and-backfill/dual-write-from-other) Note If you get stuck, you can get help by either opening a support request, or take your issue to the `#migration` channel in the [community slack](https://slack.timescale.com/), where the developers of this migration method are there to help. You can open a support request directly from [Tiger Console](https://console.cloud.tigerdata.com/dashboard/support/cases), or by email to . # Migrate from non-PostgreSQL using dual-write and backfill > Migrate from a non-PostgreSQL database to Tiger Cloud using the low-downtime dual-write and backfill method This document provides detailed step-by-step instructions to migrate data using the [dual-write and backfill](/docs/migrate/dual-write-and-backfill) migration method from a source database which is not using PostgreSQL to Tiger Cloud. Note In the context of migrations, your existing production database is referred to as the SOURCE database, the Tiger Cloud service that you are migrating your data to is the TARGET. In detail, the migration process consists of the following steps: 1. Set up a target Tiger Cloud service. 2. Modify the application to write to a secondary database. 3. Set up schema and migrate relational data to target database. 4. Start the application in dual-write mode. 5. Determine the completion point `T`. 6. Backfill time-series data from source to target. 7. Enable background jobs (policies) in the target database. 8. Validate that all data is present in target database. 9. Validate that target database can handle production load. 10. Switch application to treat target database as primary (potentially continuing to write into source database, as a backup). Note If you get stuck, you can get help by either opening a support request, or take your issue to the `#migration` channel in the [community slack](https://slack.timescale.com/), where the developers of this migration method are there to help. You can open a support request directly from [Tiger Console](https://console.cloud.tigerdata.com/dashboard/support/cases), or by email to . ## 1. Set up a target database instance in Tiger Cloud [Create a Tiger Cloud service](/docs/get-started/quickstart/create-service). If you intend on migrating more than 400 GB, open a support request to ensure that enough disk is pre-provisioned on your Tiger Cloud service. You can open a support request directly from [Tiger Console](https://console.cloud.tigerdata.com/dashboard/support/cases), or by email to . ## 2. Modify the application to write to the target database How exactly to do this is dependent on the language that your application is written in, and on how exactly your ingestion and application function. In the simplest case, you simply execute two inserts in parallel. In the general case, you must think about how to handle the failure to write to either the source or target database, and what mechanism you want to or can build to recover from such a failure. Should your time-series data have foreign-key references into a plain table, you must ensure that your application correctly maintains the foreign key relations. If the referenced column is a `*SERIAL` type, the same row inserted into the source and target *may not* obtain the same autogenerated id. If this happens, the data backfilled from the source to the target is internally inconsistent. In the best case it causes a foreign key violation, in the worst case, the foreign key constraint is maintained, but the data references the wrong foreign key. To avoid these issues, best practice is to follow [live migration](/docs/migrate/live-migration). You may also want to execute the same read queries on the source and target database to evaluate the correctness and performance of the results which the queries deliver. Bear in mind that the target database spends a certain amount of time without all data being present, so you should expect that the results are not the same for some period (potentially a number of days). ## 3. set up schema and migrate relational data to target database [Section titled “3. set up schema and migrate relational data to target database”](#3-set-up-schema-and-migrate-relational-data-to-target-database) Describing exactly how to migrate your data from every possible source is not feasible, instead we tell you what needs to be done, and hope that you find resources to support you. In this step, you need to prepare the database to receive time-series data which is dual-written from your application. If you're migrating from another time-series database then you only need to worry about setting up the schema for the hypertables which will contain time-series data. For some background on what hypertables are, consult the [hypertable documentation](/docs/learn/hypertables/understand-hypertables). If you're migrating from a relational database containing both relational and time-series data, you also need to set up the schema for the relational data, and copy it over in this step, excluding any of the time-series data. The time-series data is backfilled in a subsequent step. Our assumption in the dual-write and backfill scenario is that the volume of relational data is either very small in relation to the time-series data, so that it is not problematic to briefly stop your production application while you copy the relational data, or that it changes infrequently, so you can get a snapshot of the relational metadata without stopping your application. If this is not the case for your application, you should reconsider using the dual-write and backfill method. Tips If you're planning on experimenting with continuous aggregates, we recommend that you first complete the dual-write and backfill migration, and only then create continuous aggregates on the data. If you create continuous aggregates on a hypertable before backfilling data into it, you must refresh the continuous aggregate over the whole time range to ensure that there are no holes in the aggregated data. ## 4. Start application in dual-write mode With the target database set up, your application can now be started in dual-write mode. ## 5. Determine the completion point `T` After dual-writes have been executing for a while, the target hypertable contains data in three time ranges: missing writes, late-arriving data, and the "consistency" range ![Hypertable dual-write time ranges: missing writes, late-arriving data, and consistency range](/docs/_astro/hypertable_backfill_consistency.Cr3t3p1E_3HSNV.webp) ![Hypertable dual-write time ranges: missing writes, late-arriving data, and consistency range](/docs/_astro/hypertable_backfill_consistency-dark.CXQ2ihep_PQ7f5.webp) ### Missing writes If the application is made up of multiple writers, and these writers did not all simultaneously start writing into the target hypertable, there is a period of time in which not all writes have made it into the target hypertable. This period starts when the first writer begins dual-writing, and ends when the last writer begins dual-writing. ### Late-arriving data Some applications have late-arriving data: measurements which have a timestamp in the past, but which weren't written yet (for example from devices which had intermittent connectivity issues). The window of late-arriving data is between the present moment, and the maximum lateness. ### Consistency range The consistency range is the range in which there are no missing writes, and in which all data has arrived, that is between the end of the missing writes range and the beginning of the late-arriving data range. The length of these ranges is defined by the properties of the application, there is no one-size-fits-all way to determine what they are. ### Completion point The completion point `T` is an arbitrarily chosen time in the consistency range. It is the point in time to which data can safely be backfilled, ensuring that there is no data loss. The completion point should be expressed as the type of the `time` column of the hypertables to be backfilled. For instance, if you're using a `TIMESTAMPTZ` `time` column, then the completion point may be `2023-08-10T12:00:00.00Z`. If you're using a `BIGINT` column it may be `1695036737000`. If you are using a mix of types for the `time` columns of your hypertables, you must determine the completion point for each type individually, and backfill each set of hypertables with the same type independently from those of other types. ## 6. backfill data from source to target [Section titled “6. backfill data from source to target”](#6-backfill-data-from-source-to-target) Dump the data from your source database on a per-table basis into CSV format, and restore those CSVs into the target database using the `timescaledb-parallel-copy` tool. ### 6a. Determine the time range of data to be copied Determine the window of data that to be copied from the source database to the target. Depending on the volume of data in the source table, it may be sensible to split the source table into multiple chunks of data to move independently. In the following steps, this time range is called `` and ``. Usually the `time` column is of type `timestamp with time zone`, so the values of `` and `` must be something like `2023-08-01T00:00:00Z`. If the `time` column is not a `timestamp with time zone` then the values of `` and `` must be the correct type for the column. If you intend to copy all historic data from the source table, then the value of `` can be `'-infinity'`, and the `` value is the value of the completion point `T` that you determined. ### 6b. Remove overlapping data in the target The dual-write process may have already written data into the target database in the time range that you want to move. In this case, the dual-written data must be removed. This can be achieved with a `DELETE` statement, as follows: ```bash psql $TARGET -c "DELETE FROM WHERE time >= AND time < );" ``` Tips The BETWEEN operator is inclusive of both the start and end ranges, so it is not recommended to use it. ### 6c. copy the data [Section titled “6c. copy the data”](#6c-copy-the-data) Refer to the documentation for your source database in order to determine how to dump a table into a CSV. You must ensure the CSV contains only data before the completion point. You should apply this filter when dumping the data from the source database. You can load a CSV file into a hypertable using `timescaledb-parallel-copy` as follows. Set the number of workers equal to the number of CPU cores in your target database: ```plaintext timescaledb-parallel-copy \ --connection $TARGET \ --table \ --workers 8 \ --file
``` The above command is not transactional. If there is a connection issue, or some other issue which causes it to stop copying, the partially copied rows must be removed from the target (using the instructions in step 6b above), and then the copy can be restarted. ### 6e. Enable policies that compress data in the target hypertable In the following command, replace `` with the fully qualified table name of the target hypertable, for example `public.metrics`: ```bash psql -d $TARGET -f -v hypertable= - <<'EOF' SELECT public.alter_job(j.id, scheduled=>true) FROM _timescaledb_config.bgw_job j JOIN _timescaledb_catalog.hypertable h ON h.id = j.hypertable_id WHERE j.proc_schema IN ('_timescaledb_internal', '_timescaledb_functions') AND j.proc_name = 'policy_compression' AND j.id >= 1000 AND format('%I.%I', h.schema_name, h.table_name)::text::regclass = :'hypertable'::text::regclass; EOF ``` ## 7. validate that all data is present in target database [Section titled “7. validate that all data is present in target database”](#7-validate-that-all-data-is-present-in-target-database) Now that all data has been backfilled, and the application is writing data to both databases, the contents of both databases should be the same. How exactly this should best be validated is dependent on your application. If you are reading from both databases in parallel for every production query, you could consider adding an application-level validation that both databases are returning the same data. Another option is to compare the number of rows in the source and target tables, although this reads all data in the table which may have an impact on your production workload. ## 8. validate that target database can handle production load [Section titled “8. validate that target database can handle production load”](#8-validate-that-target-database-can-handle-production-load) Now that dual-writes have been in place for a while, the target database should be holding up to production write traffic. Now would be the right time to determine if the target database can serve all production traffic (both reads *and* writes). How exactly this is done is application-specific and up to you to determine. ## 9. switch production workload to target database [Section titled “9. switch production workload to target database”](#9-switch-production-workload-to-target-database) Once you've validated that all the data is present, and that the target database can handle the production workload, the final step is to switch to the target database as your primary. You may want to continue writing to the source database for a period, until you are certain that the target database is holding up to all production traffic. # Migrate from PostgreSQL using dual-write and backfill > Migrate from a PostgreSQL database to Tiger Cloud using the low-downtime dual-write and backfill method This document provides detailed step-by-step instructions to migrate data using the [dual-write and backfill](/docs/migrate/dual-write-and-backfill) migration method from a source database which is using PostgreSQL to Tiger Cloud. Note In the context of migrations, your existing production database is referred to as the SOURCE database, the Tiger Cloud service that you are migrating your data to is the TARGET. In detail, the migration process consists of the following steps: 1. Set up a target Tiger Cloud service. 2. Modify the application to write to the target database. 3. Migrate schema and relational data from source to target. 4. Start the application in dual-write mode. 5. Determine the completion point `T`. 6. Backfill time-series data from source to target. 7. Validate that all data is present in target database. 8. Validate that target database can handle production load. 9. Switch application to treat target database as primary (potentially continuing to write into source database, as a backup). Note If you get stuck, you can get help by either opening a support request, or take your issue to the `#migration` channel in the [community slack](https://slack.timescale.com/), where the developers of this migration method are there to help. You can open a support request directly from [Tiger Console](https://console.cloud.tigerdata.com/dashboard/support/cases), or by email to . ## 1. Set up a target database instance in Tiger Cloud [Create a Tiger Cloud service](/docs/get-started/quickstart/create-service). If you intend on migrating more than 400 GB, open a support request to ensure that enough disk is pre-provisioned on your Tiger Cloud service. You can open a support request directly from [Tiger Console](https://console.cloud.tigerdata.com/dashboard/support/cases), or by email to . ## 2. Modify the application to write to the target database How exactly to do this is dependent on the language that your application is written in, and on how exactly your ingestion and application function. In the simplest case, you simply execute two inserts in parallel. In the general case, you must think about how to handle the failure to write to either the source or target database, and what mechanism you want to or can build to recover from such a failure. Should your time-series data have foreign-key references into a plain table, you must ensure that your application correctly maintains the foreign key relations. If the referenced column is a `*SERIAL` type, the same row inserted into the source and target *may not* obtain the same autogenerated id. If this happens, the data backfilled from the source to the target is internally inconsistent. In the best case it causes a foreign key violation, in the worst case, the foreign key constraint is maintained, but the data references the wrong foreign key. To avoid these issues, best practice is to follow [live migration](/docs/migrate/live-migration). You may also want to execute the same read queries on the source and target database to evaluate the correctness and performance of the results which the queries deliver. Bear in mind that the target database spends a certain amount of time without all data being present, so you should expect that the results are not the same for some period (potentially a number of days). ## 3. set up schema and migrate relational data to target database [Section titled “3. set up schema and migrate relational data to target database”](#3-set-up-schema-and-migrate-relational-data-to-target-database) You would probably like to convert some of your large tables which contain time-series data into hypertables. This step consists of identifying those tables, excluding their data from the database dump, copying the database schema and tables, and setting up the time-series tables as hypertables. The data is backfilled into these hypertables in a subsequent step. Note For the sake of convenience, connection strings to the source and target databases are referred to as `$SOURCE` and `$TARGET` throughout this guide. This can be set in your shell, for example: ```bash export SOURCE="postgres://:@:/" export TARGET="postgres://:@:/" ``` ### 3a. dump the database roles from the source database [Section titled “3a. dump the database roles from the source database”](#3a-dump-the-database-roles-from-the-source-database) ```bash pg_dumpall -d "$SOURCE" \ -l $DB_NAME \ --quote-all-identifiers \ --roles-only \ --file=roles.sql ``` Tiger Cloud services do not support roles with superuser access. If your SQL dump includes roles that have such permissions, you'll need to modify the file to be compliant with the security model. You can use the following `sed` command to remove unsupported statements and permissions from your roles.sql file: ```bash sed -i -E \ -e '/CREATE ROLE "postgres";/d' \ -e '/ALTER ROLE "postgres"/d' \ -e '/CREATE ROLE "tsdbadmin";/d' \ -e '/ALTER ROLE "tsdbadmin"/d' \ -e 's/(NO)*SUPERUSER//g' \ -e 's/(NO)*REPLICATION//g' \ -e 's/(NO)*BYPASSRLS//g' \ -e 's/GRANTED BY "[^"]*"//g' \ roles.sql ``` Note This command works only with the GNU implementation of sed (sometimes referred to as gsed). For the BSD implementation (the default on macOS), you need to add an extra argument to change the `-i` flag to `-i ''`. To check the sed version, you can use the command `sed --version`. While the GNU version explicitly identifies itself as GNU, the BSD version of sed generally doesn't provide a straightforward --version flag and simply outputs an "illegal option" error. A brief explanation of this script is: * `CREATE ROLE "postgres"`; and `ALTER ROLE "postgres"`: These statements are removed because they require superuser access, which is not supported by Timescale. * `(NO)SUPERUSER` | `(NO)REPLICATION` | `(NO)BYPASSRLS`: These are permissions that require superuser access. * `GRANTED BY role_specification`: The GRANTED BY clause can also have permissions that require superuser access and should therefore be removed. Note: according to the TimescaleDB documentation, the GRANTOR in the GRANTED BY clause must be the current user, and this clause mainly serves the purpose of SQL compatibility. Therefore, it's safe to remove it. ### 3b. determine which tables to convert to hypertables [Section titled “3b. determine which tables to convert to hypertables”](#3b-determine-which-tables-to-convert-to-hypertables) Ideal candidates for hypertables are large tables containing time-series data. This is usually data with some form of timestamp value (`TIMESTAMPTZ`, `TIMESTAMP`, `BIGINT`, `INT` etc.) as the primary dimension, and some other measurement values. ### 3c. dump all tables from the source database, excluding data from hypertable candidates [Section titled “3c. dump all tables from the source database, excluding data from hypertable candidates”](#3c-dump-all-tables-from-the-source-database-excluding-data-from-hypertable-candidates) ```plaintext pg_dump -d "$SOURCE" \ --format=plain \ --quote-all-identifiers \ --no-tablespaces \ --no-owner \ --no-privileges \ --exclude-table-data=
\ --file=dump.sql ``` * `--exclude-table-data` is used to exclude all data from hypertable candidates. You can either specify a table pattern, or specify `--exclude-table-data` multiple times, once for each table to be converted. - `--no-tablespaces` is required because Tiger Cloud does not support tablespaces other than the default. This is a known limitation. - `--no-owner` is required because Tiger Cloud's `tsdbadmin` user is not a superuser and cannot assign ownership in all cases. This flag means that everything is owned by the user used to connect to the target, regardless of ownership in the source. This is a known limitation. - `--no-privileges` is required because the `tsdbadmin` user for your Tiger Cloud service is not a superuser and cannot assign privileges in all cases. This flag means that privileges assigned to other users must be reassigned in the target database as a manual clean-up task. This is a known limitation. ### 3d. load the roles and schema into the target database [Section titled “3d. load the roles and schema into the target database”](#3d-load-the-roles-and-schema-into-the-target-database) ```plaintext psql -X -d "$TARGET" \ -v ON_ERROR_STOP=1 \ --echo-errors \ -f roles.sql \ -f dump.sql ``` ### 3e. convert the plain tables to hypertables, optionally compress data in the columnstore [Section titled “3e. convert the plain tables to hypertables, optionally compress data in the columnstore”](#3e-convert-the-plain-tables-to-hypertables-optionally-compress-data-in-the-columnstore) For each table which should be converted to a hypertable in the target database, execute: ```sql SELECT create_hypertable('
', by_range('
', '
REPLICA IDENTITY USING INDEX <_index_name>' ``` * **No primary key or viable unique index**: use brute force. For each table, set `REPLICA IDENTITY` to `FULL`: ``` psql -X -d $SOURCE -c 'ALTER TABLE REPLICA IDENTITY FULL' ``` For each `UPDATE` or `DELETE` statement, PostgreSQL reads the whole table to find all matching rows. This results in significantly slower replication. If you are expecting a large number of `UPDATE` or `DELETE` operations on the table, best practice is to not use `FULL`. ## Migrate your data, then start downtime 1. **Pull the live-migration Docker image to your migration machine** ```shell sudo docker pull timescale/live-migration:latest ``` To list the available commands, run: ```shell sudo docker run --rm -it -e PGCOPYDB_SOURCE_PGURI=$SOURCE timescale/live-migration:latest --help ``` To see the available flags for each command, run `--help` for that command. For example: ```shell sudo docker run --rm -it -e PGCOPYDB_SOURCE_PGURI=$SOURCE timescale/live-migration:latest migrate --help ``` 2. **Create a snapshot image of your source database in your Tiger Cloud service** This process checks that you have tuned your source database and target service correctly for replication, then creates a snapshot of your data on the migration machine: ```shell docker run --rm -it --name live-migration-snapshot \ -e PGCOPYDB_SOURCE_PGURI=$SOURCE \ -e PGCOPYDB_TARGET_PGURI=$TARGET \ --pid=host \ -v ~/live-migration:/opt/timescale/ts_cdc \ timescale/live-migration:latest snapshot ``` Live-migration supplies information about updates you need to make to the source database and target service. For example: ```shell 2024-03-25T12:40:40.884 WARNING: The following tables in the Source DB have neither a primary key nor a REPLICA IDENTITY (FULL/INDEX) 2024-03-25T12:40:40.884 WARNING: UPDATE and DELETE statements on these tables will not be replicated to the Target DB 2024-03-25T12:40:40.884 WARNING: - public.metrics ``` If you have warnings, stop live-migration, make the suggested changes and start again. 3. **Synchronize data between your source database and your Tiger Cloud service** This command migrates data from the snapshot to your Tiger Cloud service, then streams transactions from the source to the target. ```shell docker run --rm -it --name live-migration-migrate \ -e PGCOPYDB_SOURCE_PGURI=$SOURCE \ -e PGCOPYDB_TARGET_PGURI=$TARGET \ --pid=host \ -v ~/live-migration:/opt/timescale/ts_cdc \ timescale/live-migration:latest migrate ``` Tips If the source PostgreSQL version is 17 or later, you need to pass additional flag `-e PGVERSION=17` to the `migrate` command. During this process, you see the migration process: ```shell Live-replay will complete in 1 minute 38.631 seconds (source_wal_rate: 106.0B/s, target_replay_rate: 589.0KiB/s, replay_lag: 56MiB) ``` If `migrate` stops add `--resume` to start from where it left off. Once the data in your target Tiger Cloud service has almost caught up with the source database, you see the following message: ```shell Target has caught up with source (source_wal_rate: 751.0B/s, target_replay_rate: 0B/s, replay_lag: 7KiB) To stop replication, hit 'c' and then ENTER ``` Wait until `replay_lag` is down to a few kilobytes before you move to the next step. Otherwise, data replication may not have finished. 4. **Start app downtime** 1. Stop your app writing to the source database, then let the remaining transactions finish to fully sync with the target. You can use tools like the `pg_top` CLI or `pg_stat_activity` to view the current transaction on the source database. 2. Stop Live-migration. ```shell hit 'c' and then ENTER ``` Live-migration continues the remaining work. This includes copying TimescaleDB metadata, sequences, and run policies. When the migration completes, you see the following message: ```sh Migration successfully completed ``` ## Validate your data, then restart your app 1. **Validate the migrated data** The contents of both databases should be the same. To check this you could compare the number of rows, or an aggregate of columns. However, the best validation method depends on your app. 2. **Stop app downtime** Once you are confident that your data is successfully replicated, configure your apps to use your Tiger Cloud service. 3. **Cleanup resources associated with live-migration from your migration machine** This command removes all resources and temporary files used in the migration process. When you run this command, you can no longer resume live-migration. ```shell docker run --rm -it --name live-migration-clean \ -e PGCOPYDB_SOURCE_PGURI=$SOURCE \ -e PGCOPYDB_TARGET_PGURI=$TARGET \ --pid=host \ -v ~/live-migration:/opt/timescale/ts_cdc \ timescale/live-migration:latest clean --prune ``` * From PostgreSQL This section shows you how to move your data from self-hosted PostgreSQL to a Tiger Cloud service using live migration from Terminal. ## Set your connection strings These variables hold the connection information for the source database and target Tiger Cloud service. In Terminal on your migration machine, set the following: ```bash export SOURCE="postgres://:@:/" export TARGET="postgres://tsdbadmin:@:/tsdb?sslmode=require" ``` You find the connection information for your Tiger Cloud service in the configuration file you downloaded when you created the service. Important Avoid using connection strings that route through connection poolers like PgBouncer or similar tools. This tool requires a direct connection to the database to function properly. ## Align the extensions on the source and target 1. **Ensure that the Tiger Cloud service is running the PostgreSQL extensions used in your source database** 1. Check the extensions on the source database: ```bash psql $SOURCE -c "SELECT * FROM pg_extension;" ``` 2. For each extension, enable it on your target Tiger Cloud service: ```bash psql $TARGET -c "CREATE EXTENSION IF NOT EXISTS CASCADE;" ``` ## Tune your source database You need admin rights to to update the configuration on your source database. If you are using a managed service, follow the instructions in the `From AWS RDS/Aurora` tab on this page. 1. **Install the `wal2json` extension on your source database** [Install wal2json](https://github.com/eulerto/wal2json) on your source database. 2. **Prevent PostgreSQL from treating the data in a snapshot as outdated** ```shell psql -X -d $SOURCE -c 'alter system set old_snapshot_threshold=-1' ``` This is not applicable if the source database is PostgreSQL 17 or later. 3. **Set the Write-Ahead Log (WAL) to record the information needed for logical decoding** ```shell psql -X -d $SOURCE -c 'alter system set wal_level=logical' ``` 4. **Restart the source database** Your configuration changes are now active. However, verify that the settings are live in your database. 5. **Enable live-migration to replicate `DELETE` and `UPDATE` operations** Replica identity assists data replication by identifying the rows being modified. Your options are that each table and hypertable in the source database should either have: * **A primary key**: data replication defaults to the primary key of the table being replicated. Nothing to do. * **A viable unique index**: each table has a unique, non-partial, non-deferrable index that includes only columns marked as `NOT NULL`. If a UNIQUE index does not exist, create one to assist the migration. You can delete it after migration. For each table, set `REPLICA IDENTITY` to the viable unique index: ``` psql -X -d $SOURCE -c 'ALTER TABLE
REPLICA IDENTITY USING INDEX <_index_name>' ``` * **No primary key or viable unique index**: use brute force. For each table, set `REPLICA IDENTITY` to `FULL`: ``` psql -X -d $SOURCE -c 'ALTER TABLE REPLICA IDENTITY FULL' ``` For each `UPDATE` or `DELETE` statement, PostgreSQL reads the whole table to find all matching rows. This results in significantly slower replication. If you are expecting a large number of `UPDATE` or `DELETE` operations on the table, best practice is to not use `FULL`. ## Migrate your data, then start downtime 1. **Pull the live-migration Docker image to your migration machine** ```shell sudo docker pull timescale/live-migration:latest ``` To list the available commands, run: ```shell sudo docker run --rm -it -e PGCOPYDB_SOURCE_PGURI=$SOURCE timescale/live-migration:latest --help ``` To see the available flags for each command, run `--help` for that command. For example: ```shell sudo docker run --rm -it -e PGCOPYDB_SOURCE_PGURI=$SOURCE timescale/live-migration:latest migrate --help ``` 2. **Create a snapshot image of your source database in your Tiger Cloud service** This process checks that you have tuned your source database and target service correctly for replication, then creates a snapshot of your data on the migration machine: ```shell docker run --rm -it --name live-migration-snapshot \ -e PGCOPYDB_SOURCE_PGURI=$SOURCE \ -e PGCOPYDB_TARGET_PGURI=$TARGET \ --pid=host \ -v ~/live-migration:/opt/timescale/ts_cdc \ timescale/live-migration:latest snapshot ``` Live-migration supplies information about updates you need to make to the source database and target service. For example: ```shell 2024-03-25T12:40:40.884 WARNING: The following tables in the Source DB have neither a primary key nor a REPLICA IDENTITY (FULL/INDEX) 2024-03-25T12:40:40.884 WARNING: UPDATE and DELETE statements on these tables will not be replicated to the Target DB 2024-03-25T12:40:40.884 WARNING: - public.metrics ``` If you have warnings, stop live-migration, make the suggested changes and start again. 3. **Synchronize data between your source database and your Tiger Cloud service** This command migrates data from the snapshot to your Tiger Cloud service, then streams transactions from the source to the target. ```shell docker run --rm -it --name live-migration-migrate \ -e PGCOPYDB_SOURCE_PGURI=$SOURCE \ -e PGCOPYDB_TARGET_PGURI=$TARGET \ --pid=host \ -v ~/live-migration:/opt/timescale/ts_cdc \ timescale/live-migration:latest migrate ``` Tips If the source PostgreSQL version is 17 or later, you need to pass additional flag `-e PGVERSION=17` to the `migrate` command. After migrating the schema, live-migration prompts you to create hypertables for tables that contain time-series data in your Tiger Cloud service. Run `create_hypertable()` to convert these tables. For more information, see the [hypertable docs](/docs/learn/hypertables/understand-hypertables). During this process, you see the migration process: ```shell Live-replay will complete in 1 minute 38.631 seconds (source_wal_rate: 106.0B/s, target_replay_rate: 589.0KiB/s, replay_lag: 56MiB) ``` If `migrate` stops add `--resume` to start from where it left off. Once the data in your target Tiger Cloud service has almost caught up with the source database, you see the following message: ```shell Target has caught up with source (source_wal_rate: 751.0B/s, target_replay_rate: 0B/s, replay_lag: 7KiB) To stop replication, hit 'c' and then ENTER ``` Wait until `replay_lag` is down to a few kilobytes before you move to the next step. Otherwise, data replication may not have finished. 4. **Start app downtime** 1. Stop your app writing to the source database, then let the remaining transactions finish to fully sync with the target. You can use tools like the `pg_top` CLI or `pg_stat_activity` to view the current transaction on the source database. 2. Stop Live-migration. ```shell hit 'c' and then ENTER ``` Live-migration continues the remaining work. This includes copying TimescaleDB metadata, sequences, and run policies. When the migration completes, you see the following message: ```sh Migration successfully completed ``` ## Validate your data, then restart your app 1. **Validate the migrated data** The contents of both databases should be the same. To check this you could compare the number of rows, or an aggregate of columns. However, the best validation method depends on your app. 2. **Stop app downtime** Once you are confident that your data is successfully replicated, configure your apps to use your Tiger Cloud service. 3. **Cleanup resources associated with live-migration from your migration machine** This command removes all resources and temporary files used in the migration process. When you run this command, you can no longer resume live-migration. ```shell docker run --rm -it --name live-migration-clean \ -e PGCOPYDB_SOURCE_PGURI=$SOURCE \ -e PGCOPYDB_TARGET_PGURI=$TARGET \ --pid=host \ -v ~/live-migration:/opt/timescale/ts_cdc \ timescale/live-migration:latest clean --prune ``` * From AWS RDS/Aurora To migrate your data from an Amazon RDS/Aurora PostgreSQL instance to a Tiger Cloud service, you extract the data to an intermediary EC2 Ubuntu instance in the same AWS region as your RDS/Aurora instance. You then upload your data to a Tiger Cloud service. To make this process as painless as possible, ensure that the intermediary machine has enough CPU and disk space to rapidly extract and store your data before uploading to Tiger Cloud. Migration from RDS/Aurora gives you the opportunity to create hypertables before copying the data. Once the migration is complete, you can manually enable Tiger Cloud features like [data compression](/docs/learn/columnar-storage/understand-hypercore) or [data retention](/docs/learn/data-lifecycle/data-retention/about-data-retention). This section shows you how to move your data from an Amazon RDS/Aurora instance to a Tiger Cloud service using live migration. ## Create an intermediary EC2 Ubuntu instance 1. **Select the RDS/Aurora instance to migrate** In , select the RDS/Aurora PostgreSQL instance to migrate. 2. **Click `Actions` > `Set up EC2 connection`** Press `Create EC2 instance` and use the following settings: * **AMI**: Ubuntu Server. * **Key pair**: use an existing pair or create a new one that you will use to access the intermediary machine. * **VPC**: by default, this is the same as the database instance. * **Configure Storage**: adjust the volume to at least the size of RDS/Aurora PostgreSQL instance you are migrating from. You can reduce the space used by your data on Tiger Cloud using [Hypercore](/docs/learn/columnar-storage/understand-hypercore). 3. **Click `Launch instance`, then connect via SSH** AWS creates your EC2 instance. Click `Connect to instance` > `SSH client` and follow the instructions to create the connection to your intermediary EC2 instance. ## Install the psql client tools on the intermediary instance 1. **Connect to your intermediary EC2 instance. For example:** ```sh ssh -i ".pem" ubuntu@ ``` 2. **On your intermediary EC2 instance, install the PostgreSQL client.** ```sh sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list' wget -qO- https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo tee /etc/apt/trusted.gpg.d/pgdg.asc &>/dev/null sudo apt update sudo apt install postgresql-client-16 -y # "postgresql-client-16" if your source DB is using PG 16. psql --version && pg_dump --version ``` Keep this terminal open, you need it to connect to the RDS/Aurora PostgreSQL instance for migration. ## Set up secure connectivity between your RDS/Aurora PostgreSQL and EC2 instances 1. **Select the RDS/Aurora instance to migrate** In , select the RDS/Aurora PostgreSQL instance to migrate. 2. **Open the inbound rules for the security group** Scroll down to `Security group rules (1)` and select the `EC2 Security Group - Inbound` group. The `Security Groups (1)` window opens. Click the `Security group ID`, then click `Edit inbound rules`. ![Creating a security group rule to enable RDS/Aurora to EC2 connection](/docs/_astro/rds-add-security-rule-to-ec2-instance.BXp5di3z.svg) 3. **On your intermediary EC2 instance, get your local IP address:** ```sh ec2metadata --local-ipv4 ``` Bear with me on this one, you need this IP address to enable access to your RDS/Aurora PostgreSQL instance. 4. **Add inbound rule for your EC2 instance** In `Edit inbound rules`, click `Add rule`, then create a PostgreSQL `TCP` rule granting access to the local IP address for your EC2 instance. Then click `Save rules`. ![Adding an inbound security rule for the EC2 instance](/docs/_astro/rds-add-inbound-rule-for-ec2-instance.BElzIqvO_Z2Ld7z.webp) ## Test the connection between your RDS/Aurora PostgreSQL and EC2 instances 1. **Select the RDS/Aurora instance to migrate** In , select the RDS/Aurora PostgreSQL instance to migrate. 2. **Create the source connection string** On your intermediary EC2 instance, use the values of `Endpoint`, `Port`, `Master username`, and `DB name` to create the PostgreSQL connectivity string for the `SOURCE` variable. ![Recording the RDS endpoint, port, and VPC details](/docs/_astro/migrate-source-rds-instance.BHStcVYP.svg) ```sh export SOURCE="postgres://:@:/" ``` The value of `Master password` was supplied when this RDS/Aurora PostgreSQL instance was created. 3. **Test your connection:** ```sh psql -d $SOURCE ``` You are connected to your RDS/Aurora PostgreSQL instance from your intermediary EC2 instance. ## Set your connection strings These variables hold the connection information for the source database and target Tiger Cloud service. In Terminal on your migration machine, set the following: ```bash export SOURCE="postgres://:@:/" export TARGET="postgres://tsdbadmin:@:/tsdb?sslmode=require" ``` You find the connection information for your Tiger Cloud service in the configuration file you downloaded when you created the service. Important Avoid using connection strings that route through connection poolers like PgBouncer or similar tools. This tool requires a direct connection to the database to function properly. ## Align the extensions on the source and target 1. **Ensure that the Tiger Cloud service is running the PostgreSQL extensions used in your source database** 1. Check the extensions on the source database: ```bash psql $SOURCE -c "SELECT * FROM pg_extension;" ``` 2. For each extension, enable it on your target Tiger Cloud service: ```bash psql $TARGET -c "CREATE EXTENSION IF NOT EXISTS CASCADE;" ``` ## Tune your source database Updating parameters on a PostgreSQL instance will cause an outage. Choose a time that will cause the least issues to tune this database. 1. **Update the DB instance parameter group for your source database** 1. In , select the RDS instance to migrate. 2. Click `Configuration`, scroll down and note the `DB instance parameter group`, then click `Parameter groups` ![RDS parameter groups in the AWS console](/docs/_astro/awsrds-parameter-groups.CiXrFBVV_1Diea.webp) 3. Click `Create parameter group`, fill in the form with the following values, then click `Create`. * **Parameter group name** - whatever suits your fancy. * **Description** - knock yourself out with this one. * **Engine type** - `PostgreSQL` * **Parameter group family** - the same as `DB instance parameter group` in your `Configuration`. 4. In `Parameter groups`, select the parameter group you created, then click `Edit`. 5. Update the following parameters, then click `Save changes`. * `rds.logical_replication` set to `1`: record the information needed for logical decoding. * `wal_sender_timeout` set to `0`: disable the timeout for the sender process. 6. In RDS, navigate back to your [databases](https://console.aws.amazon.com/rds/home#databases:), select the RDS instance to migrate, and click `Modify`. 7. Scroll down to `Database options`, select your new parameter group, and click `Continue`. 8. Click `Apply immediately` or choose a maintenance window, then click `Modify DB instance`. Changing parameters will cause an outage. Wait for the database instance to reboot before continuing. 9. Verify that the settings are live in your database. 2. **Enable replication `DELETE` and `UPDATE` operations** Replica identity assists data replication by identifying the rows being modified. Your options are that each table and hypertable in the source database should either have: * **A primary key**: data replication defaults to the primary key of the table being replicated. Nothing to do. * **A viable unique index**: each table has a unique, non-partial, non-deferrable index that includes only columns marked as `NOT NULL`. If a UNIQUE index does not exist, create one to assist the migration. You can delete it after migration. For each table, set `REPLICA IDENTITY` to the viable unique index: ``` psql -X -d $SOURCE -c 'ALTER TABLE
REPLICA IDENTITY USING INDEX <_index_name>' ``` * **No primary key or viable unique index**: use brute force. For each table, set `REPLICA IDENTITY` to `FULL`: ``` psql -X -d $SOURCE -c 'ALTER TABLE REPLICA IDENTITY FULL' ``` For each `UPDATE` or `DELETE` statement, PostgreSQL reads the whole table to find all matching rows. This results in significantly slower replication. If you are expecting a large number of `UPDATE` or `DELETE` operations on the table, best practice is to not use `FULL`. ## Migrate your data, then start downtime 1. **Pull the live-migration Docker image to your migration machine** ```shell sudo docker pull timescale/live-migration:latest ``` To list the available commands, run: ```shell sudo docker run --rm -it -e PGCOPYDB_SOURCE_PGURI=$SOURCE timescale/live-migration:latest --help ``` To see the available flags for each command, run `--help` for that command. For example: ```shell sudo docker run --rm -it -e PGCOPYDB_SOURCE_PGURI=$SOURCE timescale/live-migration:latest migrate --help ``` 2. **Create a snapshot image of your source database in your Tiger Cloud service** This process checks that you have tuned your source database and target service correctly for replication, then creates a snapshot of your data on the migration machine: ```shell docker run --rm -it --name live-migration-snapshot \ -e PGCOPYDB_SOURCE_PGURI=$SOURCE \ -e PGCOPYDB_TARGET_PGURI=$TARGET \ --pid=host \ -v ~/live-migration:/opt/timescale/ts_cdc \ timescale/live-migration:latest snapshot ``` Live-migration supplies information about updates you need to make to the source database and target service. For example: ```shell 2024-03-25T12:40:40.884 WARNING: The following tables in the Source DB have neither a primary key nor a REPLICA IDENTITY (FULL/INDEX) 2024-03-25T12:40:40.884 WARNING: UPDATE and DELETE statements on these tables will not be replicated to the Target DB 2024-03-25T12:40:40.884 WARNING: - public.metrics ``` If you have warnings, stop live-migration, make the suggested changes and start again. 3. **Synchronize data between your source database and your Tiger Cloud service** This command migrates data from the snapshot to your Tiger Cloud service, then streams transactions from the source to the target. ```shell docker run --rm -it --name live-migration-migrate \ -e PGCOPYDB_SOURCE_PGURI=$SOURCE \ -e PGCOPYDB_TARGET_PGURI=$TARGET \ --pid=host \ -v ~/live-migration:/opt/timescale/ts_cdc \ timescale/live-migration:latest migrate ``` Tips If the source PostgreSQL version is 17 or later, you need to pass additional flag `-e PGVERSION=17` to the `migrate` command. After migrating the schema, live-migration prompts you to create hypertables for tables that contain time-series data in your Tiger Cloud service. Run `create_hypertable()` to convert these tables. For more information, see the [hypertable docs](/docs/learn/hypertables/understand-hypertables). During this process, you see the migration process: ```shell Live-replay will complete in 1 minute 38.631 seconds (source_wal_rate: 106.0B/s, target_replay_rate: 589.0KiB/s, replay_lag: 56MiB) ``` If `migrate` stops add `--resume` to start from where it left off. Once the data in your target Tiger Cloud service has almost caught up with the source database, you see the following message: ```shell Target has caught up with source (source_wal_rate: 751.0B/s, target_replay_rate: 0B/s, replay_lag: 7KiB) To stop replication, hit 'c' and then ENTER ``` Wait until `replay_lag` is down to a few kilobytes before you move to the next step. Otherwise, data replication may not have finished. 4. **Start app downtime** 1. Stop your app writing to the source database, then let the remaining transactions finish to fully sync with the target. You can use tools like the `pg_top` CLI or `pg_stat_activity` to view the current transaction on the source database. 2. Stop Live-migration. ```shell hit 'c' and then ENTER ``` Live-migration continues the remaining work. This includes copying TimescaleDB metadata, sequences, and run policies. When the migration completes, you see the following message: ```sh Migration successfully completed ``` ## Validate your data, then restart your app 1. **Validate the migrated data** The contents of both databases should be the same. To check this you could compare the number of rows, or an aggregate of columns. However, the best validation method depends on your app. 2. **Stop app downtime** Once you are confident that your data is successfully replicated, configure your apps to use your Tiger Cloud service. 3. **Cleanup resources associated with live-migration from your migration machine** This command removes all resources and temporary files used in the migration process. When you run this command, you can no longer resume live-migration. ```shell docker run --rm -it --name live-migration-clean \ -e PGCOPYDB_SOURCE_PGURI=$SOURCE \ -e PGCOPYDB_TARGET_PGURI=$TARGET \ --pid=host \ -v ~/live-migration:/opt/timescale/ts_cdc \ timescale/live-migration:latest clean --prune ``` * From MST This section shows you how to move your data from a MST instance to a Tiger Cloud service using live migration from Terminal. ## Set your connection strings These variables hold the connection information for the source database and target Tiger Cloud service. In Terminal on your migration machine, set the following: ```bash export SOURCE="postgres://:@:/" export TARGET="postgres://tsdbadmin:@:/tsdb?sslmode=require" ``` You find the connection information for your Tiger Cloud service in the configuration file you downloaded when you created the service. Important Avoid using connection strings that route through connection poolers like PgBouncer or similar tools. This tool requires a direct connection to the database to function properly. ## Align the version of TimescaleDB on the source and target 1. **Ensure that the source and target databases are running the same version of TimescaleDB** 1. Check the version of TimescaleDB running on your Tiger Cloud service: ```bash psql $TARGET -c "SELECT extversion FROM pg_extension WHERE extname = 'timescaledb';" ``` 2. Update the TimescaleDB extension in your source database to match the target service: If the TimescaleDB extension is the same version on the source database and target service, you do not need to do this. ```bash psql $SOURCE -c "ALTER EXTENSION timescaledb UPDATE TO '';" ``` For more information and guidance, see [Upgrade TimescaleDB](/docs/deploy/self-hosted/upgrades). 2. **Ensure that the Tiger Cloud service is running the PostgreSQL extensions used in your source database** 1. Check the extensions on the source database: ```bash psql $SOURCE -c "SELECT * FROM pg_extension;" ``` 2. For each extension, enable it on your target Tiger Cloud service: ```bash psql $TARGET -c "CREATE EXTENSION IF NOT EXISTS CASCADE;" ``` ## Tune your source database 1. **Enable live-migration to replicate `DELETE` and `UPDATE` operations** Replica identity assists data replication by identifying the rows being modified. Your options are that each table and hypertable in the source database should either have: * **A primary key**: data replication defaults to the primary key of the table being replicated. Nothing to do. * **A viable unique index**: each table has a unique, non-partial, non-deferrable index that includes only columns marked as `NOT NULL`. If a UNIQUE index does not exist, create one to assist the migration. You can delete it after migration. For each table, set `REPLICA IDENTITY` to the viable unique index: ``` psql -X -d $SOURCE -c 'ALTER TABLE
REPLICA IDENTITY USING INDEX <_index_name>' ``` * **No primary key or viable unique index**: use brute force. For each table, set `REPLICA IDENTITY` to `FULL`: ``` psql -X -d $SOURCE -c 'ALTER TABLE REPLICA IDENTITY FULL' ``` For each `UPDATE` or `DELETE` statement, PostgreSQL reads the whole table to find all matching rows. This results in significantly slower replication. If you are expecting a large number of `UPDATE` or `DELETE` operations on the table, best practice is to not use `FULL`. ## Migrate your data, then start downtime 1. **Pull the live-migration Docker image to your migration machine** ```shell sudo docker pull timescale/live-migration:latest ``` To list the available commands, run: ```shell sudo docker run --rm -it -e PGCOPYDB_SOURCE_PGURI=$SOURCE timescale/live-migration:latest --help ``` To see the available flags for each command, run `--help` for that command. For example: ```shell sudo docker run --rm -it -e PGCOPYDB_SOURCE_PGURI=$SOURCE timescale/live-migration:latest migrate --help ``` 2. **Create a snapshot image of your source database in your Tiger Cloud service** This process checks that you have tuned your source database and target service correctly for replication, then creates a snapshot of your data on the migration machine: ```shell docker run --rm -it --name live-migration-snapshot \ -e PGCOPYDB_SOURCE_PGURI=$SOURCE \ -e PGCOPYDB_TARGET_PGURI=$TARGET \ --pid=host \ -v ~/live-migration:/opt/timescale/ts_cdc \ timescale/live-migration:latest snapshot ``` Live-migration supplies information about updates you need to make to the source database and target service. For example: ```shell 2024-03-25T12:40:40.884 WARNING: The following tables in the Source DB have neither a primary key nor a REPLICA IDENTITY (FULL/INDEX) 2024-03-25T12:40:40.884 WARNING: UPDATE and DELETE statements on these tables will not be replicated to the Target DB 2024-03-25T12:40:40.884 WARNING: - public.metrics ``` If you have warnings, stop live-migration, make the suggested changes and start again. 3. **Synchronize data between your source database and your Tiger Cloud service** This command migrates data from the snapshot to your Tiger Cloud service, then streams transactions from the source to the target. ```shell docker run --rm -it --name live-migration-migrate \ -e PGCOPYDB_SOURCE_PGURI=$SOURCE \ -e PGCOPYDB_TARGET_PGURI=$TARGET \ --pid=host \ -v ~/live-migration:/opt/timescale/ts_cdc \ timescale/live-migration:latest migrate ``` Tips If the source PostgreSQL version is 17 or later, you need to pass additional flag `-e PGVERSION=17` to the `migrate` command. During this process, you see the migration process: ```shell Live-replay will complete in 1 minute 38.631 seconds (source_wal_rate: 106.0B/s, target_replay_rate: 589.0KiB/s, replay_lag: 56MiB) ``` If `migrate` stops add `--resume` to start from where it left off. Once the data in your target Tiger Cloud service has almost caught up with the source database, you see the following message: ```shell Target has caught up with source (source_wal_rate: 751.0B/s, target_replay_rate: 0B/s, replay_lag: 7KiB) To stop replication, hit 'c' and then ENTER ``` Wait until `replay_lag` is down to a few kilobytes before you move to the next step. Otherwise, data replication may not have finished. 4. **Start app downtime** 1. Stop your app writing to the source database, then let the remaining transactions finish to fully sync with the target. You can use tools like the `pg_top` CLI or `pg_stat_activity` to view the current transaction on the source database. 2. Stop Live-migration. ```shell hit 'c' and then ENTER ``` Live-migration continues the remaining work. This includes copying TimescaleDB metadata, sequences, and run policies. When the migration completes, you see the following message: ```sh Migration successfully completed ``` ## Validate your data, then restart your app 1. **Validate the migrated data** The contents of both databases should be the same. To check this you could compare the number of rows, or an aggregate of columns. However, the best validation method depends on your app. 2. **Stop app downtime** Once you are confident that your data is successfully replicated, configure your apps to use your Tiger Cloud service. 3. **Cleanup resources associated with live-migration from your migration machine** This command removes all resources and temporary files used in the migration process. When you run this command, you can no longer resume live-migration. ```shell docker run --rm -it --name live-migration-clean \ -e PGCOPYDB_SOURCE_PGURI=$SOURCE \ -e PGCOPYDB_TARGET_PGURI=$TARGET \ --pid=host \ -v ~/live-migration:/opt/timescale/ts_cdc \ timescale/live-migration:latest clean --prune ``` And you are done, your data is now in your Tiger Cloud service. ## Troubleshooting [Section titled “Troubleshooting”](#troubleshooting) This section shows you how to work around frequently seen issues when using live migration. ### ERROR: relation "xxx.yy" does not exist This may happen when a relation is removed after executing the `snapshot` command. A relation can be a table, index, view, or materialized view. When you see you this error: * Do not perform any explicit DDL operation on the source database during the course of migration. * If you are migrating from self-hosted TimescaleDB or MST, disable the chunk retention policy on your source database until you have finished migration. ### FATAL: remaining connection slots are reserved for non-replication superuser connections This may happen when the number of connections exhaust `max_connections` defined in your target Tiger Cloud service. By default, live-migration needs around \~6 connections on the source and \~12 connections on the target. ### Migration seems to be stuck with "x GB copied to Target DB (Source DB is y GB)" When you are migrating a lot of data involved in aggregation, or there are many materialized views taking time to complete the materialization, this may be due to `REFRESH MATERIALIZED VIEWS` happening at the end of initial data migration. To resolve this issue: 1. **See what is happening on the target Tiger Cloud service** ```shell psql $TARGET -c "select * from pg_stat_activity where application_name ilike '%pgcopydb%';" ``` 2. **When you run migrate, add flags to exclude specific materialized views** ```shell --skip-table-data " ``` 3. **When `migrate` has finished, manually refresh the materialized views you excluded** ### Restart migration from scratch after a non-resumable failure If the migration halts due to a failure, such as a misconfiguration of the source or target database, you may need to restart the migration from scratch. In such cases, you can reuse the original target Tiger Cloud service created for the migration by utilizing the `--drop-if-exists` flag with the migrate command. This flag ensures that the existing target objects created by the previous migration are dropped, allowing the migration to proceed without trouble. Note: This flag also requires you to manually recreate the TimescaleDB extension on the target. Here's an example command sequence to restart the migration: ```shell psql $TARGET -c "DROP EXTENSION timescaledb CASCADE" psql $TARGET -c 'CREATE EXTENSION timescaledb VERSION ""' docker run --rm -it --name live-migration-migrate \ -e PGCOPYDB_SOURCE_PGURI=$SOURCE \ -e PGCOPYDB_TARGET_PGURI=$TARGET \ --pid=host \ -v ~/live-migration:/opt/timescale/ts_cdc \ timescale/live-migration:latest migrate --drop-if-exists ``` This approach provides a clean slate for the migration process while reusing the existing target instance. ### Inactive or lagging replication slots If you encounter an "Inactive or lagging replication slots" warning on your cloud provider console after using live-migration, it might be due to lingering replication slots created by the live-migration tool on your source database. To clean up resources associated with live migration, use the following command: ```sh docker run --rm -it --name live-migration-clean \ -e PGCOPYDB_SOURCE_PGURI=$SOURCE \ -e PGCOPYDB_TARGET_PGURI=$TARGET \ --pid=host \ -v ~/live-migration:/opt/timescale/ts_cdc \ timescale/live-migration:latest clean --prune ``` The `--prune` flag is used to delete temporary files in the `~/live-migration` directory that were needed for the migration process. It's important to note that executing the `clean` command means you cannot resume the interrupted live migration. ### Role passwords Because of issues dumping passwords from various managed service providers, Live-migration migrates roles without passwords. You have to migrate passwords manually. ### Table privileges Live-migration does not migrate table privileges. After completing Live-migration: 1. **Grant all roles to `tsdbadmin`** ```shell psql -d $SOURCE -t -A -c "SELECT FORMAT('GRANT %I TO tsdbadmin;', rolname) FROM pg_catalog.pg_roles WHERE rolname not like 'pg_%' AND rolname != 'tsdbadmin' AND NOT rolsuper" | psql -d $TARGET -f - ``` 2. **On your migration machine, edit `/tmp/grants.psql` to match table privileges on your source database** ```shell pg_dump --schema-only --quote-all-identifiers --exclude-schema=_timescaledb_catalog --format=plain --dbname "$SOURCE" | grep "(ALTER.*OWNER.*|GRANT|REVOKE)" > /tmp/grants.psql ``` 3. **Run `grants.psql` on your target Tiger Cloud service** ```shell psql -d $TARGET -f /tmp/grants.psql ``` ### PostgreSQL to Tiger Cloud: "live-replay not keeping up with source load" 1. **Find the slow query** Go to Tiger Console -> `Monitoring` -> `Insights` tab and find the query which takes significant time. 2. **Check indexes for UPDATE/DELETE queries** If the query is either UPDATE/DELETE, make sure the columns used on the WHERE clause have necessary indexes. 3. **Check REPLICA IDENTITY for hypertables** If the query is either UPDATE/DELETE on the tables which are converted as hypertables, make sure the REPLICA IDENTITY (defaults to primary key) on the source is compatible with the target primary key. If not, create a UNIQUE index on the source database by including the hypertable partition column and make it a REPLICA IDENTITY. Also, create the same UNIQUE index on the target. ### ERROR: out of memory (or) Failed on request of size xxx in memory context "yyy" on a Tiger Cloud service This error occurs when the Out of Memory (OOM) guard is triggered due to memory allocations exceeding safe limits. It typically happens when multiple concurrent connections to the TimescaleDB instance are performing memory-intensive operations. For example, during live migrations, this error can occur when large indexes are being created simultaneously. The live-migration tool includes a retry mechanism to handle such errors. However, frequent OOM crashes may significantly delay the migration process. One of the following can be used to avoid the OOM errors: 1. **Upgrade to higher memory spec instances** To mitigate memory constraints, consider using a TimescaleDB instance with higher specifications, such as an instance with 8 CPUs and 32 GB RAM (or more). Higher memory capacity can handle larger workloads and reduce the likelihood of OOM errors. 2. **Reduce concurrency** If upgrading your instance is not feasible, you can reduce the concurrency of the index migration process using the `--index-jobs=` flag in the migration command. By default, the value of `--index-jobs` matches the GUC max\_parallel\_workers. Lowering this value reduces the memory usage during migration but may increase the total migration time. # Stream data from Kafka > Stream data from Kafka into a Tiger Cloud service to store, query, and analyze your Kafka events efficiently Early access Note The Kafka source connector is not yet supported for production use. You use the Kafka source connector in Tiger Cloud to stream events from Kafka into your service. Tiger Cloud connects to your Confluent Cloud Kafka cluster and Schema Registry using SASL/SCRAM authentication and service account–based API keys. Only the Avro format is currently supported [with some limitations](#known-limitations-and-unsupported-types). This page explains how to connect Tiger Cloud to your Confluent Cloud Kafka cluster. Note This page covers the fully managed Kafka source connector, which streams from Confluent Cloud clusters into your service. To stream from a self-managed Apache Kafka cluster using your own Kafka Connect deployment, see [Integrate Apache Kafka with Tiger Cloud](/docs/integrate/data-engineering-etl/apache-kafka). ## Prerequisites for this procedure To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service). - A [Confluent Cloud](https://www.confluent.io/get-started/) account. - A [Kafka cluster](https://docs.confluent.io/cloud/current/clusters/create-cluster.html) created in Confluent Cloud. ## Access your Kafka cluster in Confluent Cloud Take the following steps to prepare your Kafka cluster for connection to Tiger Cloud: 1. **Create a service account** If you already have a service account for Tiger Cloud, you can reuse it. To create a new service account: 1. Log in to [Confluent Cloud](https://confluent.cloud/). 2. Click the burger menu at the top-right of the pane, then press `Access control` > `Service accounts` > `Add service account`. 3. Enter the following details: * Name: `tigerdata-access` * Description: `Service account for the Tiger Cloud source connector` 4. Add the service account owner role, then click `Next`. 5. Select a role assignment, then click `Add`. 6. Click `Next`, then click `Create service account`. 2. **Create API keys** 1. In Confluent Cloud, click `Home` > `Environments` > Select your environment > Select your cluster. 2. Under `Cluster overview` in the left sidebar, select `API Keys`. 3. Click `Add key`, choose `Service Account` and click `Next`. 4. Select `tigerdata-access`, then click `Next`. 5. For your cluster, choose the `Operation` and select the following `Permission`s, then click `Next`: * `Resource type`: `Cluster` * `Operation`: `DESCRIBE` * `Permission`: `ALLOW` 6. Click `Download and continue`, then securely store the ACL. 7. Use the same procedure to add the following keys: * ACL 2: Topic access * `Resource type`: `Topic` * `Topic name`: Select the topics that Tiger Cloud should read * `Pattern type`: `LITERAL` * `Operation`: `READ` * `Permission`: `ALLOW` * ACL 3: Consumer group access * `Resource type`: `Consumer group` * `Consumer group ID`: `tigerdata-kafka/`. See [Connect to Tiger Data](/docs/integrate/find-connection-details) for where to find your project ID * `Pattern type`: `PREFIXED` * `Operation`: `READ` * `Permission`: `ALLOW` You need these to configure your Kafka source connector in Tiger Cloud. ## Configure Confluent Cloud Schema Registry Tiger Cloud requires access to the Schema Registry to fetch schemas for Kafka topics. To configure the Schema Registry: 1. **Navigate to `Schema Registry`** In Confluent Cloud, click `Environments` and select your environment, then click `Stream Governance`. 2. **Create a Schema Registry API key** 1. Click `API Keys`, then click `Add API Key`. 2. Choose `Service Account`, select `tigerdata-access`, then click `Next`. 3. Under `Resource scope`, choose `Schema Registry`, select the `default` environment, then click `Next`. 4. In `Create API Key`, add the following, then click `Create API Key`: * `Name`: `tigerdata-schema-registry-access` * `Description`: `API key for Tiger Cloud schema registry access` 5. Click `Download API Key` and securely store the API key and secret, then click `Complete`. 3. **Assign roles for Schema Registry** 1. Click the burger menu at the top-right of the pane, then press `Access control` > `Accounts & access` > `Service accounts`. 2. Select the `tigerdata-access` service account. 3. In the `Access` tab, add the following role assignments for `All schema subjects`: * `ResourceOwner` on the service account. * `DeveloperRead` on schema subjects. Choose `All schema subjects` or restrict to specific subjects as required. 4. Save the role assignments. Your Confluent Cloud Schema Registry is now accessible to Tiger Cloud using the API key and secret. ## Add Kafka source connector in Tiger Cloud Take the following steps to create a Kafka source connector in Tiger Console. 1. **In Console, select your service** Open [Console](https://console.cloud.tigerdata.com/) and select your service. 2. **Go to Connectors > Source connectors** Click `New Connector`, then select `Kafka`. 3. **Set the connector name** Click the pencil icon, then set the connector name. 4. **Set up Kafka authentication** Enter the name of your cluster in Confluent Cloud and the information from the first `api-key-*.txt` that you downloaded, then click `Authenticate`. 5. **Set up the Schema Registry** Enter the service account ID and the information from the second `api-key-*.txt` that you downloaded, then click `Authenticate`. 6. **Select topics to sync** Add the schema and table, map the columns in the table, and click `Create connector`. Your Kafka connector is configured and ready to stream events. ## Known limitations and unsupported types The following Avro schema types are not supported: ### Union types Multi-type non-nullable unions are blocked. Examples: * Multiple type union: ```json { "type": "record", "name": "Message", "fields": [ {"name": "content", "type": ["string", "bytes", "null"]} ] } ``` * Union as root schema: ```json ["null", "string"] ``` ### Reference types (named type references) Referencing a previously defined named type by name, instead of inline, is not supported. Examples: * Named type definition: ```json { "type": "record", "name": "Address", "fields": [ {"name": "street", "type": "string"}, {"name": "city", "type": "string"} ] } ``` * Failing reference: ```json { "type": "record", "name": "Person", "fields": [ {"name": "name", "type": "string"}, {"name": "address", "type": "Address"} ] } ``` ### Unsupported logical types Only the logical types in the hardcoded supported list are supported. This includes: * decimal, date, time-millis, time-micros * timestamp-millis, timestamp-micros, timestamp-nanos * local-timestamp-millis, local-timestamp-micros, local-timestamp-nanos * uuid, duration Unsupported examples: ```json { "type": "int", "logicalType": "date-time" } { "type": "string", "logicalType": "json" } { "type": "bytes", "logicalType": "custom-type" } ``` # Sync data from PostgreSQL > Sync PostgreSQL tables to Tiger Cloud in real time using the source PostgreSQL connector Tips **Source PostgreSQL connector vs. Livesync replication:** The source PostgreSQL connector provides **continuous ongoing replication** where PostgreSQL stays the primary and Tiger Cloud acts as a logical replica. For a **one-time migration** to Tiger Cloud with a cutover at the end, see [Livesync replication](/docs/migrate/livesync-replication) instead. Both features share the same underlying technology, but the workflows and end states differ. You use the source PostgreSQL connector in Tiger Cloud to synchronize all data or specific tables from a PostgreSQL database instance to your service, in real time. You run the connector continuously, turning PostgreSQL into a primary database with your service as a logical replica. This enables you to leverage Tiger Cloud's real-time analytics capabilities on your replica data. ![Connectors overview in Tiger Console](/docs/_astro/tiger-console-connector-overview.C3brL-kO_Z2bCaMM.webp) The source PostgreSQL connector in Tiger Cloud leverages the well-established PostgreSQL logical replication protocol. By relying on this protocol, Tiger Cloud ensures compatibility, familiarity, and a broader knowledge base, making it easier for you to adopt the connector and integrate your data. You use the source PostgreSQL connector for data synchronization, rather than migration. This includes: * Copy existing data from a PostgreSQL instance to a Tiger Cloud service: * Copy data at up to 150 GB/hr. You need at least a 4 CPU/16 GB source database, and a 4 CPU/16 GB target service. * Copy the publication tables in parallel. Large individual tables still use a single connection, except PostgreSQL declarative partitioned tables published with `publish_via_partition_root = false` (the PostgreSQL default), which are copied leaf partition by leaf partition in parallel since v0.23.0. * Forget foreign key relationships. The connector disables foreign key validation during the sync. For example, if a `metrics` table refers to the `id` column on the `tags` table, you can still sync only the `metrics` table without worrying about their foreign key relationships. * Track progress. PostgreSQL exposes `COPY` progress under `pg_stat_progress_copy`. * Synchronize real-time changes from a PostgreSQL instance to a Tiger Cloud service. * Add and remove tables on demand using the [PostgreSQL PUBLICATION interface](https://www.postgresql.org/docs/current/sql-createpublication.html). * Enable features such as [hypertables](/docs/learn/hypertables/understand-hypertables), [columnstore](/docs/learn/columnar-storage/understand-hypercore), and [continuous aggregates](/docs/learn/continuous-aggregates) on your logical replica. * Indexes, primary key, unique constraints, and sequences are **not** migrated. Create needed indexes on the target for your queries. * TimescaleDB as source has limited support (for example, no continuous aggregates). * Schema changes must be coordinated: apply compatible changes on the target first, then on the source. * WAL volume on the source increases during large table copy. * **Continuous aggregates:** The connector uses `session_replication_role=replica` during copy, so triggers (including continuous aggregate invalidation) do not run. Data synced during initial load below a continuous aggregate's materialization watermark may not appear in the aggregate until you manually refresh. If the aggregate exists on the source, include it in the connector's publication; if only on the target, use the `force` option of [refresh\_continuous\_aggregate](/docs/reference/timescaledb/continuous-aggregates/refresh_continuous_aggregate) to refresh affected ranges. - Tiger Console ## Prerequisites for this integration guide To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service). - Your [connection details](/docs/integrate/find-connection-details). * The [PostgreSQL client tools](/docs/integrate/query-administration/psql) installed on your sync machine. * The source PostgreSQL instance and the target Tiger Cloud service must have the same extensions installed. The source PostgreSQL connector does not create extensions on the target. If the table uses column types from an extension, first create the extension on the target Tiger Cloud service before syncing the table. ## Limitations * Indexes, including the primary key, unique constraints, and sequences are not migrated to the target Tiger Cloud service. We recommend that, depending on your query patterns, you create only the necessary indexes on the target Tiger Cloud service. - Using TimescaleDB as the source has limited support (no CAGGs). - The source must be running PostgreSQL 13 or later. - Schema changes must be co-ordinated. Make compatible changes to the schema in your Tiger Cloud service first, then make the same changes to the source PostgreSQL instance. - Ensure that the source PostgreSQL instance and the target Tiger Cloud service have the same extensions installed. The source PostgreSQL connector does not create extensions on the target. If the table uses column types from an extension, first create the extension on the target Tiger Cloud service before syncing the table. - There is WAL volume growth on the source PostgreSQL instance during large table copy. - Continuous aggregate invalidation The connector uses `session_replication_role=replica` during data replication, which prevents table triggers from firing. This includes the internal triggers that mark continuous aggregates as invalid when underlying data changes. If you have continuous aggregates on your target database, they do not automatically refresh for data inserted during the migration. This limitation only applies to data below the continuous aggregate's materialization watermark. For example, backfilled data. New rows synced above the continuous aggregate watermark are used correctly when refreshing. This can lead to: * Missing data in continuous aggregates for the migration period. * Stale aggregate data. * Queries returning incomplete results. If the continuous aggregate exists in the source database, best practice is to add it to the PostgreSQL connector publication. If it only exists on the target database, manually refresh the continuous aggregate using the `force` option of [refresh\_continuous\_aggregate](/docs/reference/timescaledb/continuous-aggregates/refresh_continuous_aggregate#samples). ## Set your connection string This variable holds the connection information for the source database. In the terminal on your migration machine, set the following: ```bash export SOURCE="postgres://:@:/" ``` Important Avoid using connection strings that route through connection poolers like PgBouncer or similar tools. This tool requires a direct connection to the database to function properly. ## Tune your source database * From AWS RDS/Aurora Updating parameters on a PostgreSQL instance will cause an outage. Choose a time that will cause the least issues to tune this database. 1. **Tune the Write Ahead Log (WAL) on the RDS/Aurora PostgreSQL source database** 1. In [RDS console](https://console.aws.amazon.com/rds/home#databases:), select the RDS instance to migrate. 2. Click `Configuration`, scroll down and note the `DB instance parameter group`, then click `Parameter Groups`. ![RDS parameter groups in the AWS console](/docs/_astro/awsrds-parameter-groups.CiXrFBVV_1Diea.webp) 3. Click `Create parameter group`, fill in the form with the following values, then click `Create`: * `Parameter group name`, whatever suits your fancy. * `Description`, knock yourself out with this one. * `Engine type`, `PostgreSQL` * `Parameter group family`, the same as `DB instance parameter group` in your `Configuration`. 4. In `Parameter groups`, select the parameter group you created, then click `Edit`. 5. Update the following parameters, then click `Save changes`: * `rds.logical_replication` set to `1`: record the information needed for logical decoding. * `wal_sender_timeout` set to `0`: disable the timeout for the sender process. 6. In RDS, navigate back to your [databases](https://console.aws.amazon.com/rds/home#databases:), select the RDS instance to migrate, and click `Modify`. 7. Scroll down to `Database options`, select your new parameter group, and click `Continue`. 8. Click `Apply immediately` or choose a maintenance window, then click `Modify DB instance`. Changing parameters will cause an outage. Wait for the database instance to reboot before continuing. After it comes back up, verify that the new settings are in effect on your database. 2. **Create a user for the source PostgreSQL connector and assign permissions** 1. Create ``: ```bash psql $SOURCE -c "CREATE USER PASSWORD ''" ``` You can use an existing user. However, you must ensure that the user has the following permissions. 2. Grant permissions to create a replication slot: ```bash psql $SOURCE -c "GRANT rds_replication TO " ``` 3. Grant permissions to create a publication: ```bash psql $SOURCE -c "GRANT CREATE ON DATABASE TO " ``` 4. Assign the user permissions on the source database: ```bash psql $SOURCE <; GRANT SELECT ON ALL TABLES IN SCHEMA "public" TO ; ALTER DEFAULT PRIVILEGES IN SCHEMA "public" GRANT SELECT ON TABLES TO ; EOF ``` If the tables you are syncing are not in the `public` schema, grant the user permissions for each schema you are syncing: ```bash psql $SOURCE < TO ; GRANT SELECT ON ALL TABLES IN SCHEMA TO ; ALTER DEFAULT PRIVILEGES IN SCHEMA GRANT SELECT ON TABLES TO ; EOF ``` 5. On each table you want to sync, make `` the owner: ```bash psql $SOURCE -c 'ALTER TABLE
OWNER TO ;' ``` You can skip this step if the replicating user is already the owner of the tables. 3. **Enable replication DELETE and UPDATE operations** Replica identity assists data replication by identifying the rows being modified. Your options are that each table and hypertable in the source database should either have: * **A primary key**: data replication defaults to the primary key of the table being replicated. Nothing to do. * **A viable unique index**: each table has a unique, non-partial, non-deferrable index that includes only columns marked as `NOT NULL`. If a UNIQUE index does not exist, create one to assist the migration. You can delete it after migration. For each table, set `REPLICA IDENTITY` to the viable unique index: ``` psql -X -d $SOURCE -c 'ALTER TABLE
REPLICA IDENTITY USING INDEX <_index_name>' ``` * **No primary key or viable unique index**: use brute force. For each table, set `REPLICA IDENTITY` to `FULL`: ``` psql -X -d $SOURCE -c 'ALTER TABLE REPLICA IDENTITY FULL' ``` For each `UPDATE` or `DELETE` statement, PostgreSQL reads the whole table to find all matching rows. This results in significantly slower replication. If you are expecting a large number of `UPDATE` or `DELETE` operations on the table, best practice is to not use `FULL`. * From PostgreSQL 1. **Tune the Write Ahead Log (WAL) on the PostgreSQL source database** ```sql psql $SOURCE <`: ```sql psql $SOURCE -c "CREATE USER PASSWORD ''" ``` You can use an existing user. However, you must ensure that the user has the following permissions. 2. Grant permissions to create a replication slot: ```sql psql $SOURCE -c "ALTER ROLE REPLICATION" ``` 3. Grant permissions to create a publication: ```sql psql $SOURCE -c "GRANT CREATE ON DATABASE TO " ``` 4. Assign the user permissions on the source database: ```sql psql $SOURCE <; GRANT SELECT ON ALL TABLES IN SCHEMA "public" TO ; ALTER DEFAULT PRIVILEGES IN SCHEMA "public" GRANT SELECT ON TABLES TO ; EOF ``` If the tables you are syncing are not in the `public` schema, grant the user permissions for each schema you are syncing: ```sql psql $SOURCE < TO ; GRANT SELECT ON ALL TABLES IN SCHEMA TO ; ALTER DEFAULT PRIVILEGES IN SCHEMA GRANT SELECT ON TABLES TO ; EOF ``` 5. On each table you want to sync, make `` the owner: ```sql psql $SOURCE -c 'ALTER TABLE
OWNER TO ;' ``` You can skip this step if the replicating user is already the owner of the tables. 3. **Enable replication DELETE and UPDATE operations** Replica identity assists data replication by identifying the rows being modified. Your options are that each table and hypertable in the source database should either have: * **A primary key**: data replication defaults to the primary key of the table being replicated. Nothing to do. * **A viable unique index**: each table has a unique, non-partial, non-deferrable index that includes only columns marked as `NOT NULL`. If a UNIQUE index does not exist, create one to assist the migration. You can delete it after migration. For each table, set `REPLICA IDENTITY` to the viable unique index: ``` psql -X -d $SOURCE -c 'ALTER TABLE
REPLICA IDENTITY USING INDEX <_index_name>' ``` * **No primary key or viable unique index**: use brute force. For each table, set `REPLICA IDENTITY` to `FULL`: ``` psql -X -d $SOURCE -c 'ALTER TABLE REPLICA IDENTITY FULL' ``` For each `UPDATE` or `DELETE` statement, PostgreSQL reads the whole table to find all matching rows. This results in significantly slower replication. If you are expecting a large number of `UPDATE` or `DELETE` operations on the table, best practice is to not use `FULL`. ## Synchronize data to your Tiger Cloud service To sync data from your PostgreSQL database to your Tiger Cloud service using Tiger Console: 1. **Connect to your Tiger Cloud service** In [Tiger Console](https://console.cloud.tigerdata.com/dashboard/services), select the service to sync live data to. 2. **Prepare the source database** ![PostgreSQL connector wizard in Tiger Console](/docs/_astro/pg-connector-wizard-tiger-console.BPVoP0DK_Z1kktnR.webp) 1. Click `Connectors` > `Postgres`. 2. Set the name for the new connector by clicking the pencil icon. 3. Check the boxes for `Set wal_level to logical` and `Update your credentials`, then click `Continue`. 3. **Connect the source database and the target service** ![PostgreSQL connector connection string in Tiger Console](/docs/_astro/pg-connector-connection-string.DUeVjYuI_1Kk98E.webp) 1. Enter your database credentials or a PostgreSQL connection string. This is the connection string for [``](#tune-your-source-database). 2. **(Recommended for private databases)** If the source database is not reachable from the public Internet, toggle `Enable SSH tunneling` and route the connector through a bastion host: 1. Copy the public key shown in the wizard and append it to the `authorized_keys` file of the user the connector will log in as on the bastion host: ```bash echo "" >> ~/.ssh/authorized_keys # Verify cat ~/.ssh/authorized_keys ``` 2. Enter the bastion `Hostname`, `User`, and `Port` in the wizard. 3. Click `Connect to database`. Tiger Console connects to the source database and retrieves the schema information. 4. **Select the data to synchronize** ![Starting the PostgreSQL connector in Tiger Console](/docs/_astro/pg-connector-start-tiger-console.DPSXCOf0_AQ7bm.webp) Choose where the connector picks tables from: * **From an existing publication**: select a PostgreSQL `PUBLICATION` name, then pick schemas and tables from those it includes. Use this when you cannot grant the connector user `CREATE` privilege on the source — your DBA can pre-create the publication — or when you want to apply [row or column filters](https://www.postgresql.org/docs/current/logical-replication-row-filter.html) on published tables. * **Directly from the source**: select schemas, then pick the tables to sync from them. Tiger Console creates the publication for you. Click `Select tables +`. For each selected table, Tiger Console checks the schema and, if possible, suggests the column to use as the time dimension in a hypertable. 5. **Configure Initial Data Copy workers** ![Configure Initial Data Copy worker count in Tiger Console](/docs/_astro/pg-connector-configure-idc-worker-count.D6aj0UEK_Z12BfdB.webp) Specify the number of parallel connections — Initial Data Copy (IDC) workers — used to process the initial data copy. Higher values can speed up the initial copy but may increase load on the source database. Defaults to `4`. Tune this based on the resources available on the source database and the number of tables being synced. Large individual tables still copy through a single connection, so higher worker counts help most when you have many small to medium tables. Click `Create Connector`. Tiger Console starts the source PostgreSQL connector between the source database and the target service and displays the progress. 6. **Monitor and manage the connector** ![Editing a PostgreSQL connector in Tiger Console](/docs/_astro/edit-pg-connector-tiger-console.CAiEG08p_ZedUDY.webp) 1. To review the syncing progress for each table, click `Connectors` > `Source connectors`, then select the name of your connector in the table. 2. To edit the connector, click `Connectors` > `Source connectors`, then select the name of your connector in the table. You can rename the connector, delete or add new tables for syncing. 3. To pause a connector, click `Connectors` > `Source connectors`, then open the three-dot menu on the right and select `Pause`. 4. To delete a connector, click `Connectors` > `Source connectors`, then open the three-dot menu on the right and select `Delete`. You must pause the connector before deleting it. And that is it, you are using the source PostgreSQL connector to synchronize all the data, or specific tables, from a PostgreSQL database instance to your Tiger Cloud service, in real time. - Self-hosted PostgreSQL connector Updating parameters on a PostgreSQL instance will cause an outage. Choose a time that will cause the least issues to tune this database. 1. **Tune the Write Ahead Log (WAL) on the RDS/Aurora PostgreSQL source database** 1. In [RDS console](https://console.aws.amazon.com/rds/home#databases:), select the RDS instance to migrate. 2. Click `Configuration`, scroll down and note the `DB instance parameter group`, then click `Parameter Groups`. ![RDS parameter groups in the AWS console](/docs/_astro/awsrds-parameter-groups.CiXrFBVV_1Diea.webp) 3. Click `Create parameter group`, fill in the form with the following values, then click `Create`: * `Parameter group name`, whatever suits your fancy. * `Description`, knock yourself out with this one. * `Engine type`, `PostgreSQL` * `Parameter group family`, the same as `DB instance parameter group` in your `Configuration`. 4. In `Parameter groups`, select the parameter group you created, then click `Edit`. 5. Update the following parameters, then click `Save changes`: * `rds.logical_replication` set to `1`: record the information needed for logical decoding. * `wal_sender_timeout` set to `0`: disable the timeout for the sender process. 6. In RDS, navigate back to your [databases](https://console.aws.amazon.com/rds/home#databases:), select the RDS instance to migrate, and click `Modify`. 7. Scroll down to `Database options`, select your new parameter group, and click `Continue`. 8. Click `Apply immediately` or choose a maintenance window, then click `Modify DB instance`. Changing parameters will cause an outage. Wait for the database instance to reboot before continuing. After it comes back up, verify that the new settings are in effect on your database. 2. **Create a user for the source PostgreSQL connector and assign permissions** 1. Create ``: ```bash psql $SOURCE -c "CREATE USER PASSWORD ''" ``` You can use an existing user. However, you must ensure that the user has the following permissions. 2. Grant permissions to create a replication slot: ```bash psql $SOURCE -c "GRANT rds_replication TO " ``` 3. Grant permissions to create a publication: ```bash psql $SOURCE -c "GRANT CREATE ON DATABASE TO " ``` 4. Assign the user permissions on the source database: ```bash psql $SOURCE <; GRANT SELECT ON ALL TABLES IN SCHEMA "public" TO ; ALTER DEFAULT PRIVILEGES IN SCHEMA "public" GRANT SELECT ON TABLES TO ; EOF ``` If the tables you are syncing are not in the `public` schema, grant the user permissions for each schema you are syncing: ```bash psql $SOURCE < TO ; GRANT SELECT ON ALL TABLES IN SCHEMA TO ; ALTER DEFAULT PRIVILEGES IN SCHEMA GRANT SELECT ON TABLES TO ; EOF ``` 5. On each table you want to sync, make `` the owner: ```bash psql $SOURCE -c 'ALTER TABLE
OWNER TO ;' ``` You can skip this step if the replicating user is already the owner of the tables. 3. **Enable replication DELETE and UPDATE operations** Replica identity assists data replication by identifying the rows being modified. Your options are that each table and hypertable in the source database should either have: * **A primary key**: data replication defaults to the primary key of the table being replicated. Nothing to do. * **A viable unique index**: each table has a unique, non-partial, non-deferrable index that includes only columns marked as `NOT NULL`. If a UNIQUE index does not exist, create one to assist the migration. You can delete it after migration. For each table, set `REPLICA IDENTITY` to the viable unique index: ``` psql -X -d $SOURCE -c 'ALTER TABLE
REPLICA IDENTITY USING INDEX <_index_name>' ``` * **No primary key or viable unique index**: use brute force. For each table, set `REPLICA IDENTITY` to `FULL`: ``` psql -X -d $SOURCE -c 'ALTER TABLE REPLICA IDENTITY FULL' ``` For each `UPDATE` or `DELETE` statement, PostgreSQL reads the whole table to find all matching rows. This results in significantly slower replication. If you are expecting a large number of `UPDATE` or `DELETE` operations on the table, best practice is to not use `FULL`. - From AWS RDS/Aurora 1. **Tune the Write Ahead Log (WAL) on the PostgreSQL source database** ```sql psql $SOURCE <`: ```sql psql $SOURCE -c "CREATE USER PASSWORD ''" ``` You can use an existing user. However, you must ensure that the user has the following permissions. 2. Grant permissions to create a replication slot: ```sql psql $SOURCE -c "ALTER ROLE REPLICATION" ``` 3. Grant permissions to create a publication: ```sql psql $SOURCE -c "GRANT CREATE ON DATABASE TO " ``` 4. Assign the user permissions on the source database: ```sql psql $SOURCE <; GRANT SELECT ON ALL TABLES IN SCHEMA "public" TO ; ALTER DEFAULT PRIVILEGES IN SCHEMA "public" GRANT SELECT ON TABLES TO ; EOF ``` If the tables you are syncing are not in the `public` schema, grant the user permissions for each schema you are syncing: ```sql psql $SOURCE < TO ; GRANT SELECT ON ALL TABLES IN SCHEMA TO ; ALTER DEFAULT PRIVILEGES IN SCHEMA GRANT SELECT ON TABLES TO ; EOF ``` 5. On each table you want to sync, make `` the owner: ```sql psql $SOURCE -c 'ALTER TABLE
OWNER TO ;' ``` You can skip this step if the replicating user is already the owner of the tables. 3. **Enable replication DELETE and UPDATE operations** Replica identity assists data replication by identifying the rows being modified. Your options are that each table and hypertable in the source database should either have: * **A primary key**: data replication defaults to the primary key of the table being replicated. Nothing to do. * **A viable unique index**: each table has a unique, non-partial, non-deferrable index that includes only columns marked as `NOT NULL`. If a UNIQUE index does not exist, create one to assist the migration. You can delete it after migration. For each table, set `REPLICA IDENTITY` to the viable unique index: ``` psql -X -d $SOURCE -c 'ALTER TABLE
REPLICA IDENTITY USING INDEX <_index_name>' ``` * **No primary key or viable unique index**: use brute force. For each table, set `REPLICA IDENTITY` to `FULL`: ``` psql -X -d $SOURCE -c 'ALTER TABLE REPLICA IDENTITY FULL' ``` For each `UPDATE` or `DELETE` statement, PostgreSQL reads the whole table to find all matching rows. This results in significantly slower replication. If you are expecting a large number of `UPDATE` or `DELETE` operations on the table, best practice is to not use `FULL`. - From PostgreSQL ## Prerequisites for this integration guide To follow these steps, you'll need: * A target [Tiger Cloud service](/docs/get-started/quickstart/create-service). Best practice is to create a Tiger Cloud service with at least 8 CPUs for a smoother experience. A higher-spec instance can significantly reduce the overall migration window. * A migration machine to run the commands that move data from your source database to your target Tiger Cloud service. Best practice: use an [Ubuntu EC2 instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EC2_GetStarted.html#ec2-launch-instance) hosted in the same region as your Tiger Cloud service. * An [adjusted maintenance window](/docs/deploy/tiger-cloud/tiger-cloud-aws/upgrades#define-your-maintenance-window) to prevent maintenance from running during migration. - The source PostgreSQL instance and the target Tiger Cloud service must have the same extensions installed. The source PostgreSQL connector does not create extensions on the target. If the table uses column types from an extension, first create the extension on the target Tiger Cloud service before syncing the table. - [Docker](https://docs.docker.com/engine/install/) installed on your sync machine. For a better experience, use a 4 CPU/16GB EC2 instance or greater to run the source PostgreSQL connector. - The [PostgreSQL client tools](/docs/integrate/query-administration/psql) installed on your sync machine. This includes `psql`, `pg_dump`, `pg_dumpall`, and `vacuumdb` commands. ## Limitations * The schema is not migrated by the source PostgreSQL connector, you use `pg_dump`/`pg_restore` to migrate it. - Using TimescaleDB as the source has limited support (no CAGGs). - The source must be running PostgreSQL 13 or later. - Schema changes must be co-ordinated. Make compatible changes to the schema in your Tiger Cloud service first, then make the same changes to the source PostgreSQL instance. - Ensure that the source PostgreSQL instance and the target Tiger Cloud service have the same extensions installed. The source PostgreSQL connector does not create extensions on the target. If the table uses column types from an extension, first create the extension on the target Tiger Cloud service before syncing the table. - There is WAL volume growth on the source PostgreSQL instance during large table copy. - Continuous aggregate invalidation The connector uses `session_replication_role=replica` during data replication, which prevents table triggers from firing. This includes the internal triggers that mark continuous aggregates as invalid when underlying data changes. If you have continuous aggregates on your target database, they do not automatically refresh for data inserted during the migration. This limitation only applies to data below the continuous aggregate's materialization watermark. For example, backfilled data. New rows synced above the continuous aggregate watermark are used correctly when refreshing. This can lead to: * Missing data in continuous aggregates for the migration period. * Stale aggregate data. * Queries returning incomplete results. If the continuous aggregate exists in the source database, best practice is to add it to the PostgreSQL connector publication. If it only exists on the target database, manually refresh the continuous aggregate using the `force` option of [refresh\_continuous\_aggregate](/docs/reference/timescaledb/continuous-aggregates/refresh_continuous_aggregate#samples). ## Set your connection strings The `` in the `SOURCE` connection must have the replication role granted in order to create a replication slot. These variables hold the connection information for the source database and target Tiger Cloud service. In Terminal on your migration machine, set the following: ```bash export SOURCE="postgres://:@:/" export TARGET="postgres://tsdbadmin:@:/tsdb?sslmode=require" ``` You find the connection information for your Tiger Cloud service in the configuration file you downloaded when you created the service. Important Avoid using connection strings that route through connection poolers like PgBouncer or similar tools. This tool requires a direct connection to the database to function properly. ## Tune your source database * From AWS RDS/Aurora Updating parameters on a PostgreSQL instance will cause an outage. Choose a time that will cause the least issues to tune this database. 1. **Update the DB instance parameter group for your source database** 1. In , select the RDS instance to migrate. 2. Click `Configuration`, scroll down and note the `DB instance parameter group`, then click `Parameter groups` ![RDS parameter groups in the AWS console](/docs/_astro/awsrds-parameter-groups.CiXrFBVV_1Diea.webp) 3. Click `Create parameter group`, fill in the form with the following values, then click `Create`. * **Parameter group name** - whatever suits your fancy. * **Description** - knock yourself out with this one. * **Engine type** - `PostgreSQL` * **Parameter group family** - the same as `DB instance parameter group` in your `Configuration`. 4. In `Parameter groups`, select the parameter group you created, then click `Edit`. 5. Update the following parameters, then click `Save changes`. * `rds.logical_replication` set to `1`: record the information needed for logical decoding. * `wal_sender_timeout` set to `0`: disable the timeout for the sender process. 6. In RDS, navigate back to your [databases](https://console.aws.amazon.com/rds/home#databases:), select the RDS instance to migrate, and click `Modify`. 7. Scroll down to `Database options`, select your new parameter group, and click `Continue`. 8. Click `Apply immediately` or choose a maintenance window, then click `Modify DB instance`. Changing parameters will cause an outage. Wait for the database instance to reboot before continuing. 9. Verify that the settings are live in your database. 2. **Enable replication `DELETE` and `UPDATE` operations** Replica identity assists data replication by identifying the rows being modified. Your options are that each table and hypertable in the source database should either have: * **A primary key**: data replication defaults to the primary key of the table being replicated. Nothing to do. * **A viable unique index**: each table has a unique, non-partial, non-deferrable index that includes only columns marked as `NOT NULL`. If a UNIQUE index does not exist, create one to assist the migration. You can delete it after migration. For each table, set `REPLICA IDENTITY` to the viable unique index: ``` psql -X -d $SOURCE -c 'ALTER TABLE
REPLICA IDENTITY USING INDEX <_index_name>' ``` * **No primary key or viable unique index**: use brute force. For each table, set `REPLICA IDENTITY` to `FULL`: ``` psql -X -d $SOURCE -c 'ALTER TABLE REPLICA IDENTITY FULL' ``` For each `UPDATE` or `DELETE` statement, PostgreSQL reads the whole table to find all matching rows. This results in significantly slower replication. If you are expecting a large number of `UPDATE` or `DELETE` operations on the table, best practice is to not use `FULL`. * From PostgreSQL 1. **Tune the Write Ahead Log (WAL) on the PostgreSQL source database** ```sql psql $SOURCE <`: ```sql psql $SOURCE -c "CREATE USER PASSWORD ''" ``` You can use an existing user. However, you must ensure that the user has the following permissions. 2. Grant permissions to create a replication slot: ```sql psql $SOURCE -c "ALTER ROLE REPLICATION" ``` 3. Grant permissions to create a publication: ```sql psql $SOURCE -c "GRANT CREATE ON DATABASE TO " ``` 4. Assign the user permissions on the source database: ```sql psql $SOURCE <; GRANT SELECT ON ALL TABLES IN SCHEMA "public" TO ; ALTER DEFAULT PRIVILEGES IN SCHEMA "public" GRANT SELECT ON TABLES TO ; EOF ``` If the tables you are syncing are not in the `public` schema, grant the user permissions for each schema you are syncing: ```sql psql $SOURCE < TO ; GRANT SELECT ON ALL TABLES IN SCHEMA TO ; ALTER DEFAULT PRIVILEGES IN SCHEMA GRANT SELECT ON TABLES TO ; EOF ``` 5. On each table you want to sync, make `` the owner: ```sql psql $SOURCE -c 'ALTER TABLE
OWNER TO ;' ``` You can skip this step if the replicating user is already the owner of the tables. 3. **Enable replication DELETE and UPDATE operations** Replica identity assists data replication by identifying the rows being modified. Your options are that each table and hypertable in the source database should either have: * **A primary key**: data replication defaults to the primary key of the table being replicated. Nothing to do. * **A viable unique index**: each table has a unique, non-partial, non-deferrable index that includes only columns marked as `NOT NULL`. If a UNIQUE index does not exist, create one to assist the migration. You can delete it after migration. For each table, set `REPLICA IDENTITY` to the viable unique index: ``` psql -X -d $SOURCE -c 'ALTER TABLE
REPLICA IDENTITY USING INDEX <_index_name>' ``` * **No primary key or viable unique index**: use brute force. For each table, set `REPLICA IDENTITY` to `FULL`: ``` psql -X -d $SOURCE -c 'ALTER TABLE REPLICA IDENTITY FULL' ``` For each `UPDATE` or `DELETE` statement, PostgreSQL reads the whole table to find all matching rows. This results in significantly slower replication. If you are expecting a large number of `UPDATE` or `DELETE` operations on the table, best practice is to not use `FULL`. ## Migrate the table schema to the Tiger Cloud service Use `pg_dump` to: 1. **Download the schema from the source database** ```shell pg_dump $SOURCE \ --no-privileges \ --no-owner \ --no-publications \ --no-subscriptions \ --no-table-access-method \ --no-tablespaces \ --schema-only \ --file=schema.sql ``` 2. **Apply the schema on the target service** ```shell psql $TARGET -f schema.sql ``` ## Convert partitions and tables with time-series data into hypertables For efficient querying and analysis, you can convert tables which contain time-series or events data, and tables that are already partitioned using PostgreSQL declarative partition into [hypertables](/docs/learn/hypertables/understand-hypertables). 1. **Convert tables to hypertables** Run the following on each table in the target Tiger Cloud service to convert it to a hypertable: ```shell psql -X -d $TARGET -c "SELECT public.create_hypertable('
', by_range('', ''::interval));" ``` For example, to convert the *metrics* table into a hypertable with *time* as a partition column and *1 day* as a partition interval: ```shell psql -X -d $TARGET -c "SELECT public.create_hypertable('public.metrics', by_range('time', '1 day'::interval));" ``` 2. **Convert PostgreSQL partitions to hypertables** Rename the partition and create a new regular table with the same name as the partitioned table, then convert to a hypertable: ```shell psql $TARGET -f - <<'EOF' BEGIN; ALTER TABLE public.events RENAME TO events_part; CREATE TABLE public.events(LIKE public.events_part INCLUDING ALL); SELECT create_hypertable('public.events', by_range('time', '1 day'::interval)); COMMIT; EOF ``` ## Specify the tables to synchronize After the schema is migrated, you [`CREATE PUBLICATION`](https://www.postgresql.org/docs/current/sql-createpublication.html) on the source database that specifies the tables to synchronize. 1. **Create a publication that specifies the table to synchronize** A `PUBLICATION` enables you to synchronize some or all the tables in the schema or database. ```sql CREATE PUBLICATION FOR TABLE , ; ``` To add tables after to an existing publication, use [ALTER PUBLICATION](https://www.postgresql.org/docs/current/sql-alterpublication.html)\*\* ```sql ALTER PUBLICATION ADD TABLE ; ``` 2. **Publish the PostgreSQL declarative partitioned table** Leave `publish_via_partition_root` at its default (`false`). Since live-sync v0.23.0, this copies the partitioned table's leaf partitions in parallel and makes the initial copy resumable, instead of streaming the whole table through a single connection: ```sql ALTER PUBLICATION SET(publish_via_partition_root=false); ``` Only set it to `true` if you need the older serial, whole-table copy: ```sql ALTER PUBLICATION SET(publish_via_partition_root=true); ``` To convert a partitioned table to a hypertable, follow [Convert partitions and tables with time-series data into hypertables](#convert-partitions-and-tables-with-time-series-data-into-hypertables). 3. **Stop syncing a table in the `PUBLICATION`, use `DROP TABLE`** ```sql ALTER PUBLICATION DROP TABLE ; ``` ## Synchronize data to your Tiger Cloud service You use the source PostgreSQL connector docker image to synchronize changes in real time from a PostgreSQL database instance to a Tiger Cloud service: 1. **Start the source PostgreSQL connector** As you run the source PostgreSQL connector continuously, best practice is to run it as a Docker daemon. ```shell docker run -d --rm --name livesync timescale/live-sync:latest run \ --publication --subscription \ --source $SOURCE --target $TARGET --table-map ``` * `version-tag`: The latest available version tag of the live-sync image. See [Docker Hub](https://hub.docker.com/r/timescale/live-sync). * `--publication`: The name of the publication as you created in the previous step. To use multiple publications, repeat the `--publication` flag. * `--subscription`: The name that identifies the subscription on the target Tiger Cloud service. * `--source`: The connection string to the source PostgreSQL database. * `--target`: The connection string to the target Tiger Cloud service. * `--table-map`: (Optional) A JSON string that maps source tables to target tables. If not provided, the source and target table names are assumed to be the same. For example, to map the source table `metrics` to the target table `metrics_data`: `--table-map '{"source": {"schema": "public", "table": "metrics"}, "target": {"schema": "public", "table": "metrics_data"}}'` To map only the schema, use: `--table-map '{"source": {"schema": "public"}, "target": {"schema": "analytics"}}'` This flag can be repeated for multiple table mappings. * `--table-sync-workers`: (Optional) The number of parallel workers to use for initial table sync. Default is 4. * `--copy-data`: (Optional) By default, the initial table data is copied from source to target before starting logical replication. Set to `false` so only changes made after replication slot creation are replicated. Best practice is to set to `false` during dry-run livesync so you do not copy table data. 2. **Capture logs** Once the source PostgreSQL connector is running as a docker daemon, you can also capture the logs: ```shell docker logs -f livesync ``` 3. **View the progress of tables being synchronized** List the tables being synchronized by the source PostgreSQL connector using the `_ts_live_sync.subscription_rel` table in the target Tiger Cloud service: ```bash psql $TARGET -c "SELECT * FROM _ts_live_sync.subscription_rel" ``` You see something like the following: ```txt subname | pubname | schemaname | tablename | rrelid | state | lsn | updated_at | last_error | created_at | rows_copied | approximate_rows | bytes_copied | approximate_size | target_schema | target_table ----------+-----------+------------+-----------+--------+-------+------------+-------------------------------+------------+-------------------------------+-------------+------------------+--------------+------------------+---------------+------------- livesync | analytics | public | metrics | 20856 | r | 6/1A8CBA48 | 2025-06-24 06:16:21.434898+00 | | 2025-06-24 06:03:58.172946+00 | 18225440 | 18225440 | 1387359359 | 1387359359 | public | metrics ``` The `state` column indicates the current state of the table synchronization. Possible values for `state` are: | state | description | | ----- | ----------------------------------------------------------------------- | | i | initial state, table data sync not started | | d | initial table data sync is in progress | | f | initial table data sync completed, catching up with incremental changes | | s | synchronized, waiting for the main apply worker to take over | | r | table is ready, applying changes in real-time | To see the replication lag, run the following against the SOURCE database: ```bash psql $SOURCE -f - <<'EOF' SELECT slot_name, pg_size_pretty(pg_current_wal_flush_lsn() - confirmed_flush_lsn) AS lag FROM pg_replication_slots WHERE slot_name LIKE 'live_sync_%' AND slot_type = 'logical' EOF ``` 4. **Add or remove tables from the publication** To add tables, use [ALTER PUBLICATION .. ADD TABLE](https://www.postgresql.org/docs/current/sql-alterpublication.html)\*\* ```sql ALTER PUBLICATION ADD TABLE ; ``` To remove tables, use [ALTER PUBLICATION .. DROP TABLE](https://www.postgresql.org/docs/current/sql-alterpublication.html)\*\* ```sql ALTER PUBLICATION DROP TABLE ; ``` 5. **Update table statistics** If you have a large table, you can run `ANALYZE` on the target Tiger Cloud service to update the table statistics after the initial sync is complete. This helps the query planner make better decisions for query execution plans. ```bash vacuumdb --analyze --verbose --dbname=$TARGET ``` 6. **Stop the source PostgreSQL connector** ```shell docker stop live-sync ``` 7. **(Optional) Reset sequence nextval on the target Tiger Cloud service** The source PostgreSQL connector does not automatically reset the sequence nextval on the target Tiger Cloud service. Run the following script to reset the sequence for all tables that have a serial or identity column in the target Tiger Cloud service: ```bash psql $TARGET -f - <<'EOF' DO $$ DECLARE rec RECORD; BEGIN FOR rec IN ( SELECT sr.target_schema AS table_schema, sr.target_table AS table_name, col.column_name, pg_get_serial_sequence( sr.target_schema || '.' || sr.target_table, col.column_name ) AS seqname FROM _ts_live_sync.subscription_rel AS sr JOIN information_schema.columns AS col ON col.table_schema = sr.target_schema AND col.table_name = sr.target_table WHERE col.column_default LIKE 'nextval(%' -- only serial/identity columns ) LOOP EXECUTE format( 'SELECT setval(%L, COALESCE((SELECT MAX(%I) FROM %I.%I), 0) + 1, false );', rec.seqname, -- the sequence identifier rec.column_name, -- the column to MAX() rec.table_schema, -- schema for MAX() rec.table_name -- table for MAX() ); END LOOP; END; $$ LANGUAGE plpgsql; EOF ``` 8. **Clean up** When you are permanently done syncing (for example, you have fully cut over to Tiger Cloud, or you want to restart the sync from scratch), use the `drop` sub-command to remove the replication slot from the source and all subscription state from the target: ```shell docker run -it --rm --name livesync timescale/live-sync:latest drop \ --subscription --source $SOURCE --target $TARGET ``` `drop` removes: * The replication slot (`live_sync_`) on the **source** database. * The subscription and internal tracking tables (`_ts_live_sync.*`) on the **target** Tiger Cloud service. Warning Run `drop` only when you intend to permanently stop or fully restart the sync. Dropping the replication slot while the source is still active causes WAL to be retained until the slot is recreated, which can grow disk usage on the source. - From AWS RDS/Aurora Updating parameters on a PostgreSQL instance will cause an outage. Choose a time that will cause the least issues to tune this database. 1. **Update the DB instance parameter group for your source database** 1. In , select the RDS instance to migrate. 2. Click `Configuration`, scroll down and note the `DB instance parameter group`, then click `Parameter groups` ![RDS parameter groups in the AWS console](/docs/_astro/awsrds-parameter-groups.CiXrFBVV_1Diea.webp) 3. Click `Create parameter group`, fill in the form with the following values, then click `Create`. * **Parameter group name** - whatever suits your fancy. * **Description** - knock yourself out with this one. * **Engine type** - `PostgreSQL` * **Parameter group family** - the same as `DB instance parameter group` in your `Configuration`. 4. In `Parameter groups`, select the parameter group you created, then click `Edit`. 5. Update the following parameters, then click `Save changes`. * `rds.logical_replication` set to `1`: record the information needed for logical decoding. * `wal_sender_timeout` set to `0`: disable the timeout for the sender process. 6. In RDS, navigate back to your [databases](https://console.aws.amazon.com/rds/home#databases:), select the RDS instance to migrate, and click `Modify`. 7. Scroll down to `Database options`, select your new parameter group, and click `Continue`. 8. Click `Apply immediately` or choose a maintenance window, then click `Modify DB instance`. Changing parameters will cause an outage. Wait for the database instance to reboot before continuing. 9. Verify that the settings are live in your database. 2. **Enable replication `DELETE` and `UPDATE` operations** Replica identity assists data replication by identifying the rows being modified. Your options are that each table and hypertable in the source database should either have: * **A primary key**: data replication defaults to the primary key of the table being replicated. Nothing to do. * **A viable unique index**: each table has a unique, non-partial, non-deferrable index that includes only columns marked as `NOT NULL`. If a UNIQUE index does not exist, create one to assist the migration. You can delete it after migration. For each table, set `REPLICA IDENTITY` to the viable unique index: ``` psql -X -d $SOURCE -c 'ALTER TABLE
REPLICA IDENTITY USING INDEX <_index_name>' ``` * **No primary key or viable unique index**: use brute force. For each table, set `REPLICA IDENTITY` to `FULL`: ``` psql -X -d $SOURCE -c 'ALTER TABLE REPLICA IDENTITY FULL' ``` For each `UPDATE` or `DELETE` statement, PostgreSQL reads the whole table to find all matching rows. This results in significantly slower replication. If you are expecting a large number of `UPDATE` or `DELETE` operations on the table, best practice is to not use `FULL`. - From PostgreSQL 1. **Tune the Write Ahead Log (WAL) on the PostgreSQL source database** ```sql psql $SOURCE <`: ```sql psql $SOURCE -c "CREATE USER PASSWORD ''" ``` You can use an existing user. However, you must ensure that the user has the following permissions. 2. Grant permissions to create a replication slot: ```sql psql $SOURCE -c "ALTER ROLE REPLICATION" ``` 3. Grant permissions to create a publication: ```sql psql $SOURCE -c "GRANT CREATE ON DATABASE TO " ``` 4. Assign the user permissions on the source database: ```sql psql $SOURCE <; GRANT SELECT ON ALL TABLES IN SCHEMA "public" TO ; ALTER DEFAULT PRIVILEGES IN SCHEMA "public" GRANT SELECT ON TABLES TO ; EOF ``` If the tables you are syncing are not in the `public` schema, grant the user permissions for each schema you are syncing: ```sql psql $SOURCE < TO ; GRANT SELECT ON ALL TABLES IN SCHEMA TO ; ALTER DEFAULT PRIVILEGES IN SCHEMA GRANT SELECT ON TABLES TO ; EOF ``` 5. On each table you want to sync, make `` the owner: ```sql psql $SOURCE -c 'ALTER TABLE
OWNER TO ;' ``` You can skip this step if the replicating user is already the owner of the tables. 3. **Enable replication DELETE and UPDATE operations** Replica identity assists data replication by identifying the rows being modified. Your options are that each table and hypertable in the source database should either have: * **A primary key**: data replication defaults to the primary key of the table being replicated. Nothing to do. * **A viable unique index**: each table has a unique, non-partial, non-deferrable index that includes only columns marked as `NOT NULL`. If a UNIQUE index does not exist, create one to assist the migration. You can delete it after migration. For each table, set `REPLICA IDENTITY` to the viable unique index: ``` psql -X -d $SOURCE -c 'ALTER TABLE
REPLICA IDENTITY USING INDEX <_index_name>' ``` * **No primary key or viable unique index**: use brute force. For each table, set `REPLICA IDENTITY` to `FULL`: ``` psql -X -d $SOURCE -c 'ALTER TABLE REPLICA IDENTITY FULL' ``` For each `UPDATE` or `DELETE` statement, PostgreSQL reads the whole table to find all matching rows. This results in significantly slower replication. If you are expecting a large number of `UPDATE` or `DELETE` operations on the table, best practice is to not use `FULL`. # Sync data from S3 > Sync CSV and Parquet files from an S3 bucket to Tiger Cloud in real time using the source S3 connector This guide walks you through syncing CSV and Parquet files from an Amazon S3 bucket to your Tiger Cloud service in real time using the source S3 connector in Tiger Cloud. The connector runs continuously so you can use Tiger Cloud as your analytics database with data constantly synced from S3, without building or operating custom ETL between S3 and Tiger Cloud. **What you get:** The connector syncs from an S3 bucket to a Tiger Cloud service using glob patterns, watches for new files and imports them on a configurable schedule, supports CSV (including GZ and ZIP compression) and Parquet (converted to CSV then processed with timescaledb-parallel-copy), lets you enable a hypertable during file-to-table mapping and add columnstore and continuous aggregates later in the SQL editor, and polls on a default 1-minute interval (configurable with a cron expression). **File ordering:** The connector processes files in **lexicographical order**. It uses the name of the last file processed as a marker and only fetches files that sort later in the alphabet. Files whose names sort earlier than the marker are skipped and never synced (for example, if the marker is at `Elephant`, a file named `Bob` added later is never processed). For large backlogs, use a 1-minute check until caught up. Note The connector only syncs existing and new files. It does not update or delete records in your Tiger Cloud tables when files are updated or deleted in S3. ![Connectors overview in Tiger Console](/docs/_astro/tiger-console-connector-overview.C3brL-kO_Z2bCaMM.webp) ## Before you start [Section titled “Before you start”](#before-you-start) ## Prerequisites for this procedure To follow these steps, you'll need: * A [Tiger Cloud service](/docs/get-started/quickstart/create-service). - Access to a standard Amazon S3 bucket containing your data files. **Directory buckets are not supported.** - Access credentials for the S3 bucket configured: * **IAM Role:** Set the trust policy `Principal` to `arn:aws:iam::142548018081:role/timescale-s3-connections` and `ExternalID` to your Tiger Cloud project and service ID in the form `/` (to avoid the confused deputy problem). Grant `s3:GetObject` and `s3:ListBucket`. * **Public anonymous user:** Supported if the bucket is public. Note This feature is currently not supported for Tiger Cloud on Microsoft Azure. ### Limitations [Section titled “Limitations”](#limitations) * **File naming:** Files must follow lexicographical ordering; names that sort earlier than already-processed files are permanently skipped (for example, after `file_2024_01_15.csv` is processed, `file_2024_01_10.csv` added later is never synced). Use timestamps (for example, `YYYY-MM-DD-HHMMSS`) or sequential names with fixed padding (for example, `file_00001`, `file_00002`). * **CSV:** Max file size 1 GB (contact sales\@tigerdata.com to increase), max row size 2 MB. Compressed formats: GZ, ZIP. Delimiter and “skip header” are configurable in advanced settings. * **Parquet:** Max file size 1 GB, max row size 2 MB. * **Sync iteration:** Up to 100 files per sync iteration; additional checks only fill empty queue slots. ## How to set up S3 sync [Section titled “How to set up S3 sync”](#how-to-set-up-s3-sync) Follow these steps in [Tiger Console](https://console.cloud.tigerdata.com/dashboard/services) to connect your S3 bucket and start syncing. 1. **Connect to your Tiger Cloud service** In [Tiger Console](https://console.cloud.tigerdata.com/dashboard/services), select the service you want to sync live data into. 2. **Connect the source S3 bucket to the target service** 1. Click `Connectors` → `Amazon S3`. 2. Click the pencil icon and set the name for the new connector. 3. Set `Bucket name` and `Authentication method`, then click `Continue`. For IAM role setup, click `Learn how` in the UI. Tiger Console connects to the source bucket. 4. In `Define files to sync`, choose `File type` and set the `Glob pattern`. Examples: * `/*`, all files in a folder (patterns ending with `/` are treated as `/*`). * `/**`: all files recursively. * `/**/*.csv`, a specific file type. Keep filters toward the end of the pattern; complex filtering can cause S3 list operations to time out. 5. Click the search icon to see the files that match, then click `Continue`. ![Connecting Tiger Cloud to an S3 bucket](/docs/_astro/s3-connector-tiger-console.DDuCo9oV_ZBF0v1.webp) 3. **Map schema and optimize for hypertables** Tiger Console infers the file schema and, when possible, suggests the time column for a hypertable. ![S3 connector table selection in Tiger Console](/docs/_astro/tiger-console-s3-connector-create-tables.CWZ2J2w5_Z1XzU94.webp) * Choose `Create a new table for your data` or `Ingest data to an existing table`. * Set the `Data type` for each column, then click `Continue`. * Configure insert behavior on conflict, then click `Continue`. * Choose the polling interval (for example, minute, hour, or a cron expression). * Click `Start Connector`. Tiger Console starts the connector and shows progress. You now have continuous sync from your S3 bucket to your Tiger Cloud service. ## How to monitor synchronization [Section titled “How to monitor synchronization”](#how-to-monitor-synchronization) * **Connector data flow:** Click `Connectors` to see the diagram with connector status and amount of data replicated. * **Per-connector stats:** Click `Connectors` → `Source connectors`, then select your connector. ![S3 connector import details and statistics](/docs/_astro/tiger-console-s3-connector-import-details.DbkXHMp-_1BMtgk.webp) From the connector dashboard you can: * **Search by file name** to find specific imports. * **Filter by status:** All statuses, Cancelled, Failure, In Queue, Paused, Pending Retry, Running, Success. * **Bulk retry** for files in Error status. * View **Lifecycle history** for time spent in each status. * **Refresh every minute** (auto-refresh). ## How to manage the connector [Section titled “How to manage the connector”](#how-to-manage-the-connector) * **Pause:** `Connectors` → `Source connectors` → three-dot menu next to your connector → `Pause`. ![Pausing an S3 connector in Tiger Console](/docs/_astro/tiger-console-s3-connector-pause.C5OYyjFX_1JBHaD.webp) * **Edit:** Same table → three-dot menu → `Edit` → `Connector settings`. You must pause the connector before editing. ![Editing S3 connector settings in Tiger Console](/docs/_astro/tiger-console-s3-connector-edit.Dwm-LblS_XvOVP.webp) * **Delete:** Three-dot menu → `Delete` (pause the connector first). ## Summary [Section titled “Summary”](#summary) You now have continuous sync from your S3 bucket to your Tiger Cloud service: new files that match your glob pattern are picked up on the schedule you chose, and you can monitor imports and manage the connector (pause, edit, or delete) from Tiger Console anytime. To extend the pipeline, enable [hypertables](/docs/learn/hypertables/understand-hypertables), [columnstore](/docs/learn/columnar-storage/understand-hypercore), or [continuous aggregates](/docs/learn/continuous-aggregates) on the synced table via the SQL editor. ## Related [Section titled “Related”](#related) * [Sync data from PostgreSQL](/docs/migrate/livesync-for-postgresql): Continuously replicate from a PostgreSQL database instead of S3. * [Upload a file using Tiger Console](/docs/migrate/import-console): Upload CSV/TSV directly in the web console without a connector. * [Upload a file using the terminal](/docs/migrate/import-terminal): Load data using the command line (for example, psql, `COPY`). * [Live import from a database](/docs/migrate/live-migration): Migrate with minimal or no downtime from databases. * [Source S3 connector reference](/docs/integrate/connectors/source/sync-from-s3): Technical details and capabilities of the S3 connector. # Livesync replication > Migrate your entire PostgreSQL or TimescaleDB database to Tiger Cloud with near-zero downtime using Livesync Tips **Livesync replication vs. source PostgreSQL connector:** Livesync replication is a **one-time migration** to Tiger Cloud with a cutover at the end. For **continuous ongoing replication** where PostgreSQL stays the primary and Tiger Cloud acts as a logical replica, see [Sync from PostgreSQL](/docs/migrate/livesync-for-postgresql) instead. Both features share the same underlying technology, but the workflows and end states differ. Important The Livesync replication workflow is in **early access** and is not yet supported for production use. For questions or feedback, visit [#livesync in the Tiger Cloud Community](https://app.slack.com/client/T4GT3N2JK/C086NU9EZ88). Livesync replication uses the PostgreSQL logical replication protocol to synchronize data from a source database to a Tiger Cloud service, then cut over once the target is caught up. Because it relies on the standard PostgreSQL protocol, you get compatibility, familiarity, and a broad knowledge base. The same technology powers two distribution channels: the **self-hosted Docker image** (`timescale/live-sync`) is commonly referred to as **Livesync**, and the **managed Tiger Console integration** is referred to as the **Source PostgreSQL connector**. Both produce the same migration result; this page covers both. This page replaces the older [Live migration](/docs/migrate/live-migration) workflow. The migration runs in four stages: 1. **Schema migration:** export structure from the source, apply it to the target. 2. **Initial data copy:** bulk `COPY` at 300 GB/hr to 2.2 TB/hr depending on `--table-sync-workers` tuning, target bandwidth, and IOPS. Publication tables are copied in parallel; large individual tables still use a single connection, except PostgreSQL declarative partitioned tables published with `publish_via_partition_root = false`, which Livesync (v0.23.0 and later) copies leaf partition by leaf partition in parallel, the same way hypertable chunks are copied. Foreign key validation is disabled during sync. 3. **CDC:** real-time replication of ongoing changes up to 50K operations per second. 4. **Cutover:** validate, stop sync, switch the application connection strings. You can drive Livesync replication from the Tiger Console UI or from a self-hosted Docker container. Tiger Console supports multiple Livesync instances per service, dropdown-based schema and table selection, and live progress metadata (rows copied during initial data copy, replication lag during CDC). ## Prerequisites for this procedure To follow these steps, you'll need: * **Source:** PostgreSQL 13 or later with `wal_level = logical`. The source must be a primary, not a read replica or standby. * **Target:** A Tiger Cloud service with real-time analytics enabled. Target spec should be close to the source spec for a full migration. Minimum 4 CPU / 16 GB RAM on both source and target for roughly 300 GB/hr to 2.2 TB/hr transfer. `tsdbadmin` credentials. * **Migration host:** A separate machine to run the migration commands, with Docker, `psql`, `pg_dump`, and `vacuumdb` installed and network access to both source and target databases. * An [adjusted maintenance window](/docs/deploy/tiger-cloud/tiger-cloud-aws/upgrades#define-your-maintenance-window) to prevent maintenance from running during migration. * Environment variables for the source and target connections: ```bash export SOURCE="postgres://:@:/" export TARGET="postgres://tsdbadmin:@:/tsdb?sslmode=require" ``` ## Limitations [Section titled “Limitations”](#limitations) * **TimescaleDB source must be on 2.16.0 or newer** if any chunk is ever compressed or decompressed during the migration. Earlier versions don't emit the decompression-boundary markers that Livesync needs to deduplicate the WAL stream, so the target ends up with duplicate records. This affects all of: * `INSERT`s into a chunk that the compression policy later compresses. * Explicit `convert_to_columnstore()` or `convert_to_rowstore()` calls. * `UPDATE` or `DELETE` on a compressed chunk, which decompresses, mutates, then recompresses. The pre-2.16.0 GUC `timescaledb.enable_decompression_logrep_markers` is incomplete (it doesn't cover compression policies or explicit compress/decompress) and requires PostgreSQL 14 or later to do anything. If you can't upgrade, the practical mitigations are: keep chunks that may receive writes uncompressed (for example, the last three months), pause compression policies during the migration, or plan to re-sync affected hypertables. ## Source setup [Section titled “Source setup”](#source-setup) 1. **Database parameters** Livesync requires logical replication on the source. How you enable it depends on where your source database runs. * Self-hosted Apply the parameters with `ALTER SYSTEM`. A restart is required afterwards. ```bash psql "$SOURCE" <<'EOF' ALTER SYSTEM SET wal_level = 'logical'; ALTER SYSTEM SET max_wal_senders = 30; ALTER SYSTEM SET max_replication_slots = 30; ALTER SYSTEM SET wal_sender_timeout = 0; ALTER SYSTEM SET max_locks_per_transaction = 1200; EOF ``` After restart, validate: ```bash psql "$SOURCE" <<'EOF' SHOW wal_level; SHOW max_wal_senders; SHOW max_replication_slots; SHOW wal_sender_timeout; SHOW max_locks_per_transaction; EOF ``` * AWS RDS / Aurora RDS and Aurora do not permit `ALTER SYSTEM`. Set the parameters in a custom parameter group: 1. In the RDS console, create a parameter group for your PostgreSQL major version, or edit the cluster parameter group for Aurora. 2. Set: * `rds.logical_replication = 1` (enables `wal_level=logical`) * `max_wal_senders = 30` * `max_replication_slots = 30` * `wal_sender_timeout = 0` * `max_locks_per_transaction = 1200` 3. Attach the parameter group to your instance or cluster. 4. Reboot the instance or cluster (for Aurora, reboot the writer) to apply. Validate: ```bash psql "$SOURCE" -c "SHOW rds.logical_replication;" -- expect: on psql "$SOURCE" -c "SHOW wal_level;" -- expect: logical ``` * Neon Neon manages logical replication settings through its console. 1. Open the Neon Console, go to your project, then click `Settings` > `Logical Replication`. 2. Click `Enable`. No restart or `ALTER SYSTEM` is required. Validate: ```bash psql "$SOURCE" <<'EOF' SHOW wal_level; -- expect: logical SHOW max_wal_senders; -- expect: 10 SHOW max_replication_slots;-- expect: 10 EOF ``` Neon computes may auto-suspend on idle. For long initial copies, raise the auto-suspend timeout in `Settings` > `Compute`, or pick a plan that disables it. `wal_sender_timeout` can only be changed via a Neon support case. Ask support to disable it (set to `0`) or set it to a high value such as 12 hours to avoid timeouts during the initial copy of large tables. * Supabase Supabase PostgreSQL has logical replication enabled by default; no `ALTER SYSTEM` or reboot is required. Validate: ```bash psql "$SOURCE" -c "SHOW wal_level;" -- expect: logical ``` Recommended changes (apply via the [Supabase custom Postgres config](https://supabase.com/docs/guides/database/custom-postgres-config)): * `max_slot_wal_keep_size = 102400` (100 GB) prevents the replication slot from being dropped during long-running transactions. Use the **direct connection string**, not the connection pooler: pgBouncer doesn't support logical replication. In `Project Settings` > `Database` > `Connection parameters`, disable `Display connection pooler`. * Azure Flexible Server Azure does not permit `ALTER SYSTEM`. Set the parameters via the `Server parameters` blade in the Azure portal, or with `az postgres flexible-server parameter set`: 1. Set `wal_level = logical`. 2. Set `max_wal_senders = 30`, `max_replication_slots = 30`, `wal_sender_timeout = 0`, `max_locks_per_transaction = 1200`. 3. Save. Azure prompts to restart the server. Restart now. Validate: ```bash psql "$SOURCE" -c "SHOW wal_level;" -- expect: logical ``` 2. **Migration user** Create a dedicated migration user on the source: ```sql psql "$SOURCE" <<'EOF' CREATE USER livesync_migrate PASSWORD ''; ALTER ROLE livesync_migrate WITH SUPERUSER REPLICATION; GRANT CREATE ON DATABASE TO livesync_migrate; EOF ``` `SUPERUSER` is only needed on self-hosted TimescaleDB. New hypertable chunks are created on the fly by the extension, and PostgreSQL requires the role calling `ALTER PUBLICATION ... ADD TABLE` to either own the table or be a superuser. Without `SUPERUSER`, chunks created during the migration won't get added to the publication and their data is skipped. If you can't grant `SUPERUSER`, make `livesync_migrate` the owner of every hypertable in the publication instead. Note **Azure Database for PostgreSQL:** managed Azure instances don't allow creating a new `SUPERUSER`. Skip the `CREATE USER` step and use the existing server administrator account instead, after granting it replication: ```sql ALTER ROLE WITH REPLICATION; ``` Then set: ```bash export SOURCE="postgres://:@:/" ``` If the admin lacks ownership of the application's hypertables, transfer ownership (or use `GRANT ... TO WITH ADMIN OPTION`) before creating the publication. Grant the migration user access to all user schemas: ```bash psql "$SOURCE" -t -A <<'EOF' | psql "$SOURCE" SELECT 'GRANT USAGE ON SCHEMA ' || quote_ident(nspname) || ' TO livesync_migrate;' || chr(10) || 'GRANT SELECT ON ALL TABLES IN SCHEMA ' || quote_ident(nspname) || ' TO livesync_migrate;' || chr(10) || 'ALTER DEFAULT PRIVILEGES IN SCHEMA ' || quote_ident(nspname) || ' GRANT SELECT ON TABLES TO livesync_migrate;' FROM pg_namespace WHERE nspname NOT IN ('information_schema') AND nspname NOT LIKE '_timescale%' AND nspname NOT LIKE 'pg\_%' ESCAPE '\' AND nspname NOT LIKE 'pg_toast%' AND nspname NOT LIKE 'pg_temp_%' AND nspname NOT LIKE 'timescaledb%' AND nspname NOT LIKE 'toolkit%' ORDER BY nspname; EOF ``` Switch to this user for all subsequent steps: ```bash export SOURCE="postgres://livesync_migrate:@:/" ``` ## Roles [Section titled “Roles”](#roles) Dump roles from the source and apply to the target: ```bash pg_dumpall --roles-only -d "$SOURCE" > roles.sql ``` Review and apply: ```bash cat roles.sql psql "$TARGET" -f roles.sql ``` `pg_dumpall --roles-only` includes passwords (from `pg_authid`), role attributes, and memberships. You may need to edit out `postgres`, `tsdbadmin`, and other system roles before applying to the target. ## Schema migration [Section titled “Schema migration”](#schema-migration) Choose the section matching your source database type. * From TimescaleDB 1. **Dump and restore the schema** ```bash pg_dump "$SOURCE" \ --schema-only \ --no-privileges \ --no-owner \ --no-publications \ --no-subscriptions \ --no-table-access-method \ --no-tablespaces \ --exclude-schema='_timescaledb_cache' \ --exclude-schema='_timescaledb_catalog' \ --exclude-schema='_timescaledb_config' \ --exclude-schema='_timescaledb_debug' \ --exclude-schema='_timescaledb_functions' \ --exclude-schema='_timescaledb_internal' \ --exclude-schema='timescaledb_experimental' \ --exclude-schema='timescaledb_information' \ --file=schema_dump.sql ``` This dumps hypertables as regular tables. continuous aggregates are included as regular views; they are dropped and recreated properly in the continuous aggregates step. ```bash psql "$TARGET" -f schema_dump.sql ``` 2. **Create hypertables** **Primary dimension** ```bash psql "$SOURCE" -t -A <<'EOF' > create_hypertables.sql SELECT format( 'SELECT create_hypertable(%L, by_range(%L%s), create_default_indexes => false, if_not_exists => true);', format('%I.%I', d.hypertable_schema, d.hypertable_name), d.column_name, CASE WHEN d.time_interval IS NOT NULL THEN format(', %L::interval', d.time_interval::text) ELSE '' END ) FROM timescaledb_information.dimensions d JOIN timescaledb_information.hypertables h ON d.hypertable_schema = h.hypertable_schema AND d.hypertable_name = h.hypertable_name WHERE d.dimension_number = 1 AND h.hypertable_name NOT LIKE '_materialized_hypertable_%' ORDER BY d.hypertable_schema, d.hypertable_name; EOF cat create_hypertables.sql psql "$TARGET" -f create_hypertables.sql ``` **Additional dimensions (optional)** TimescaleDB generally discourages using more than one dimension. Additional dimensions carry a risk of [partition explosion](/docs/learn/hypertables/partitioning-hypertables#space-partitioning-hash-dimension) and prevent the use of [data tiering](/docs/learn/data-lifecycle/storage/about-storage-tiers#low-cost-storage) (tiered storage requires single-dimension hypertables). Only add extra dimensions if the source explicitly uses them and you have validated the need. ```bash psql "$SOURCE" -t -A <<'EOF' > add_dimensions.sql SELECT format( 'SELECT add_dimension(%L, %s);', format('%I.%I', d.hypertable_schema, d.hypertable_name), CASE WHEN d.num_partitions IS NOT NULL THEN format('by_hash(%L, %s)', d.column_name, d.num_partitions) ELSE format('by_range(%L%s)', d.column_name, CASE WHEN d.time_interval IS NOT NULL THEN format(', %L::interval', d.time_interval::text) ELSE '' END) END ) FROM timescaledb_information.dimensions d JOIN timescaledb_information.hypertables h ON d.hypertable_schema = h.hypertable_schema AND d.hypertable_name = h.hypertable_name WHERE d.dimension_number > 1 AND h.hypertable_name NOT LIKE '_materialized_hypertable_%' ORDER BY d.hypertable_schema, d.hypertable_name, d.dimension_number; EOF cat add_dimensions.sql psql "$TARGET" -f add_dimensions.sql ``` 3. **Continuous aggregates** Drop the views that `pg_dump` created (they are not proper continuous aggregates), then recreate them correctly: ```bash psql "$SOURCE" -t -A <<'EOF' > continuous_aggregates.sql SELECT format( E'DROP VIEW IF EXISTS %I.%I CASCADE;\nCREATE MATERIALIZED VIEW %I.%I\nWITH (timescaledb.continuous) AS\n%s\nWITH NO DATA;\n', view_schema, view_name, view_schema, view_name, rtrim(view_definition, ';') ) FROM timescaledb_information.continuous_aggregates ORDER BY view_schema, view_name; EOF cat continuous_aggregates.sql psql "$TARGET" -f continuous_aggregates.sql ``` 4. **Columnstore** ```bash psql "$SOURCE" -t -A <<'EOF' > enable_columnstore.sql SELECT format( E'ALTER TABLE %s SET (\n timescaledb.enable_columnstore = true%s%s\n);\n', hypertable::text, CASE WHEN segmentby IS NOT NULL AND segmentby <> '' THEN E',\n timescaledb.segmentby = ''' || segmentby || '''' ELSE '' END, CASE WHEN orderby IS NOT NULL AND orderby <> '' THEN E',\n timescaledb.orderby = ''' || orderby || '''' ELSE '' END ) FROM timescaledb_information.hypertable_columnstore_settings WHERE hypertable::text NOT LIKE '\_timescaledb%' ESCAPE '\' ORDER BY hypertable::text; EOF cat enable_columnstore.sql psql "$TARGET" -f enable_columnstore.sql ``` `timescaledb_information.hypertable_columnstore_settings` requires TimescaleDB 2.18.0 or later on the source. 5. **Save policies for later** Generate the columnstore, refresh, and retention policy scripts now, but apply them only after data sync completes (see [Post-sync](#post-sync)). ```bash # Columnstore policies psql "$SOURCE" -t -A <<'EOF' > columnstore_policies.sql SELECT format( E'CALL add_columnstore_policy(%L, after => INTERVAL %L);\n', format('%I.%I', j.hypertable_schema, j.hypertable_name), j.config->>'compress_after' ) FROM timescaledb_information.jobs j WHERE j.proc_name = 'policy_compression' ORDER BY j.hypertable_schema, j.hypertable_name; EOF # Continuous aggregate refresh policies psql "$SOURCE" -t -A <<'EOF' > refresh_policies.sql SELECT format( E'SELECT add_continuous_aggregate_policy(%L,\n start_offset => INTERVAL %L,\n end_offset => INTERVAL %L,\n schedule_interval => INTERVAL %L\n);\n', format('%I.%I', ca.view_schema, ca.view_name), coalesce(j.config->>'start_offset', '30 days'), coalesce(j.config->>'end_offset', '1 hour'), j.schedule_interval::text ) FROM timescaledb_information.jobs j JOIN timescaledb_information.continuous_aggregates ca ON j.hypertable_schema = ca.view_schema AND j.hypertable_name = ca.view_name WHERE j.proc_name = 'policy_refresh_continuous_aggregate' ORDER BY ca.view_schema, ca.view_name; EOF # Retention policies psql "$SOURCE" -t -A <<'EOF' > retention_policies.sql SELECT format( E'SELECT add_retention_policy(%L, drop_after => INTERVAL %L);\n', format('%I.%I', j.hypertable_schema, j.hypertable_name), j.config->>'drop_after' ) FROM timescaledb_information.jobs j WHERE j.proc_name = 'policy_retention' ORDER BY j.hypertable_schema, j.hypertable_name; EOF ``` * From PostgreSQL 1. **Dump and restore the schema** ```bash pg_dump "$SOURCE" \ --schema-only \ --no-privileges \ --no-owner \ --no-publications \ --no-subscriptions \ --no-table-access-method \ --no-tablespaces \ --file=schema_dump.sql psql "$TARGET" -f schema_dump.sql ``` 2. **Convert tables to hypertables** Good candidates for [hypertables](/docs/learn/hypertables/understand-hypertables): * **Tables with a time-based column:** any column of type `TIMESTAMPTZ`, `TIMESTAMP`, `DATE`, or `INTEGER`/`BIGINT` (epoch). See [supported partition column types](/docs/learn/hypertables/creating-and-configuring-hypertables). * **Append-mostly tables:** sensor readings, metrics, events, logs, or any table where rows are predominantly inserted in time order and rarely updated. * **PostgreSQL declarative partitioned tables:** these can be converted directly to hypertables. Rename the partitioned table, create a regular table with the same name, then convert (see example below). * **Large tables with time-range queries:** tables frequently queried with `WHERE ts > now() - interval '...'` benefit most from chunk exclusion. Convert on the target: ```sql -- Regular table SELECT create_hypertable( '.
', by_range('', INTERVAL '7 days'), create_default_indexes => false, if_not_exists => true ); -- PostgreSQL declarative partitioned table BEGIN; ALTER TABLE .
RENAME TO
_part; CREATE TABLE .
(LIKE .
_part INCLUDING ALL); SELECT create_hypertable('.
', by_range('', INTERVAL '7 days')); COMMIT; ``` Primary key and unique constraints must include the partition column. 3. **Enable columnstore** ```sql ALTER TABLE .
SET ( timescaledb.enable_columnstore = true, timescaledb.segmentby = '', timescaledb.orderby = ' DESC' ); ``` Columnstore policies are added after data sync. See [Hypercore (columnstore)](/docs/learn/columnar-storage/understand-hypercore) for details on compression methods and configuration. 4. **Create continuous aggregates (optional)** ```sql CREATE MATERIALIZED VIEW . WITH (timescaledb.continuous) AS SELECT time_bucket('1 hour', ) AS bucket, , avg() AS avg_value, count(*) AS cnt FROM .
GROUP BY 1, 2 WITH NO DATA; ``` 5. **Save policies for later** Define columnstore, refresh, and retention policies now, but apply them only after data sync (see [Post-sync](#post-sync)): ```sql -- Columnstore policy CALL add_columnstore_policy('.
', after => INTERVAL '7 days'); -- Refresh policy SELECT add_continuous_aggregate_policy('.', start_offset => INTERVAL '3 days', end_offset => INTERVAL '1 hour', schedule_interval => INTERVAL '1 hour' ); -- Retention (optional) SELECT add_retention_policy('.
', drop_after => INTERVAL '90 days'); ``` Whichever source type you migrated from, finish schema migration by applying privileges. ### Apply privileges [Section titled “Apply privileges”](#apply-privileges) After restoring the schema on the target, export privileges from the source and apply them to the migrated objects: ```bash psql "$SOURCE" -t -A <<'EOF' > privileges.sql -- Schema privileges SELECT 'GRANT ' || acl.privilege_type || ' ON SCHEMA ' || quote_ident(nspname) || ' TO ' || CASE WHEN acl.grantee = 0 THEN 'PUBLIC' ELSE acl.grantee::regrole::text END || CASE WHEN acl.is_grantable THEN ' WITH GRANT OPTION' ELSE '' END || ';' FROM pg_namespace, LATERAL aclexplode(nspacl) AS acl WHERE nspacl IS NOT NULL AND nspname NOT IN ('pg_catalog', 'information_schema') AND nspname NOT LIKE '_timescale%' AND nspname NOT LIKE 'timescaledb%' AND nspname NOT LIKE 'toolkit%' AND acl.grantee::regrole::text NOT LIKE 'pg_%' AND acl.grantee::regrole::text NOT LIKE 'timescaledb%' AND acl.grantee::regrole::text NOT IN ('postgres') ORDER BY nspname; -- Table privileges SELECT 'GRANT ' || acl.privilege_type || ' ON TABLE ' || format('%I.%I', n.nspname, c.relname) || ' TO ' || CASE WHEN acl.grantee = 0 THEN 'PUBLIC' ELSE acl.grantee::regrole::text END || CASE WHEN acl.is_grantable THEN ' WITH GRANT OPTION' ELSE '' END || ';' FROM pg_class c JOIN pg_namespace n ON c.relnamespace = n.oid, LATERAL aclexplode(relacl) AS acl WHERE relacl IS NOT NULL AND relkind IN ('r', 'v', 'm', 'f', 'p') AND n.nspname NOT IN ('pg_catalog', 'information_schema') AND n.nspname NOT LIKE '_timescale%' AND n.nspname NOT LIKE 'timescaledb%' AND n.nspname NOT LIKE 'toolkit%' AND acl.grantee::regrole::text NOT LIKE 'pg_%' AND acl.grantee::regrole::text NOT LIKE 'timescaledb%' AND acl.grantee::regrole::text NOT IN ('postgres') ORDER BY n.nspname, c.relname; EOF cat privileges.sql psql "$TARGET" -f privileges.sql ``` ## Publication [Section titled “Publication”](#publication) Create a [PostgreSQL `PUBLICATION`](https://www.postgresql.org/docs/current/sql-createpublication.html) on the source. This defines what Livesync syncs. You can create more than one publication (for example, to group tables by domain), but avoid creating one per table. Important **Do not use `FOR ALL TABLES`.** It includes the `_timescaledb_catalog.*` tables, and replicating those onto the target corrupts the TimescaleDB catalog there. Always list tables explicitly (`FOR TABLE ...`) or scope to user schemas (`FOR TABLES IN SCHEMA , ...`). ```sql psql "$SOURCE" <<'EOF' CREATE PUBLICATION FOR TABLE , ; EOF ``` To sync every table across one or more user schemas (PostgreSQL 15 or later only), list the schemas explicitly and exclude the TimescaleDB ones: ```sql psql "$SOURCE" <<'EOF' CREATE PUBLICATION FOR TABLES IN SCHEMA , ; EOF ``` To add or remove tables after creating the publication: ```sql ALTER PUBLICATION ADD TABLE ; ALTER PUBLICATION DROP TABLE ; ``` Publication changes are picked up only after a Livesync stop/start, or by running the `refresh-publication` command: ```bash docker run -it --rm --name livesync-refresh \ timescale/live-sync:latest refresh-publication \ --subscription \ --source "$SOURCE" \ --target "$TARGET" ``` If you have PostgreSQL declarative partitioned tables in the publication, leave `publish_via_partition_root` at its PostgreSQL default of `false`. Since Livesync v0.23.0, this makes the initial copy of the partitioned table both parallel (each leaf partition is copied by its own worker, like hypertable chunks) and resumable (already-copied leaves aren't recopied if the sync is interrupted): ```sql ALTER PUBLICATION SET (publish_via_partition_root = false); ``` Set `publish_via_partition_root = true` only if you need the older serial, whole-table copy, for example for compatibility with a Livesync version older than v0.23.0: ```sql ALTER PUBLICATION SET (publish_via_partition_root = true); ``` To selectively replicate only specific DML events (by default all `INSERT`, `UPDATE`, `DELETE`, and `TRUNCATE` are replicated): ```sql CREATE PUBLICATION FOR TABLE WITH (publish = 'insert, update'); ``` Validate: ```bash psql "$SOURCE" -c "SELECT * FROM pg_publication_tables WHERE pubname = '';" ``` ## Replication identity [Section titled “Replication identity”](#replication-identity) Skip this section for `INSERT`-only workloads. Tables without a primary key need an explicit replica identity for `UPDATE` and `DELETE` replication. The query below finds such tables and generates the appropriate `ALTER TABLE` statement, using a unique index if one exists, otherwise falling back to `REPLICA IDENTITY FULL`. ```bash psql "$SOURCE" -t -A <<'EOF' SELECT CASE WHEN i.indexrelid IS NOT NULL THEN format('ALTER TABLE %I.%I REPLICA IDENTITY USING INDEX %I;', n.nspname, c.relname, ic.relname) ELSE format('ALTER TABLE %I.%I REPLICA IDENTITY FULL;', n.nspname, c.relname) END FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace LEFT JOIN pg_constraint pk ON pk.conrelid = c.oid AND pk.contype = 'p' LEFT JOIN LATERAL ( SELECT i.indrelid, i.indexrelid FROM pg_index i WHERE i.indrelid = c.oid AND i.indisunique AND i.indisvalid AND i.indpred IS NULL AND NOT EXISTS ( SELECT 1 FROM unnest(i.indkey) k JOIN pg_attribute a ON a.attrelid = i.indrelid AND a.attnum = k WHERE NOT a.attnotnull ) LIMIT 1 ) i ON true LEFT JOIN pg_class ic ON ic.oid = i.indexrelid WHERE c.relkind = 'r' AND pk.oid IS NULL AND n.nspname NOT IN ('information_schema') AND n.nspname NOT LIKE '_timescale%' AND n.nspname NOT LIKE 'pg\_%' ESCAPE '\' ORDER BY n.nspname, c.relname; EOF ``` Review the output, then pipe it to `psql "$SOURCE"` to apply. ## Data migration [Section titled “Data migration”](#data-migration) Choose one of the two approaches below. * Self-hosted Docker 1. **Start Livesync** Best practice is to run as a Docker daemon: ```bash docker run -d --rm --name livesync timescale/live-sync:latest run \ --publication \ --subscription \ --source "$SOURCE" \ --target "$TARGET" ``` `latest` is the latest available version. See [Docker Hub](https://hub.docker.com/r/timescale/live-sync). | Flag | Description | | ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--publication` | Publication name as created in the publication step. Repeat the flag for multiple publications. | | `--subscription` | Name that identifies the subscription on the target. | | `--source` | Connection string to the source PostgreSQL database. | | `--target` | Connection string to the target Tiger Cloud service. | | `--table-map` | (Optional) JSON mapping source tables to target tables. Repeat for multiple mappings. Example: `--table-map '{"source": {"schema": "public", "table": "metrics"}, "target": {"schema": "public", "table": "metrics_data"}}'` | | `--table-sync-workers` | (Optional) Number of parallel workers for initial table sync. Default: 4. | | `--copy-data` | (Optional) Set to `false` to skip initial data copy and only replicate changes made after replication slot creation. Useful for dry-run testing. Default: `true`. | 2. **Monitor** ```bash docker logs -f livesync ``` **Table sync status** ```bash psql "$TARGET" -c " SELECT state, last_error, count(*) FROM _ts_live_sync.subscription_rel GROUP BY 1, 2 ORDER BY 1, 2; " ``` Tables with errors appear as separate rows grouped by `last_error`. | State | Meaning | | ----- | ------------------------------------------------------------------------ | | `i` | Initial state; table data sync not started. | | `d` | Initial table data sync is in progress. | | `f` | Initial table data sync completed; catching up with incremental changes. | | `s` | Synchronized; waiting for the main apply worker to take over. | | `r` | Table is ready; applying changes in real time. | **COPY progress during initial sync** Monitor that the initial data copy is progressing as expected. The number of rows returned should usually match the `--table-sync-workers` value. ```bash psql "$SOURCE" -c "SELECT * FROM pg_stat_progress_copy;" ``` **Replication lag** On the source using `pg_replication_slots`: ```bash psql "$SOURCE" -c " SELECT slot_name, pg_size_pretty(pg_current_wal_flush_lsn() - confirmed_flush_lsn) AS lag FROM pg_replication_slots WHERE slot_name LIKE 'live_sync_%' AND slot_type = 'logical'; " ``` On the target using `_ts_live_sync.subscription` (also shows `last_error` if any): ```bash psql "$TARGET" -c " SELECT pg_size_pretty(source_flush_lsn - last_replicated_lsn) AS lag_bytes, (metrics_updated_at - last_replicated_txn_time) AS lag_duration, * FROM _ts_live_sync.subscription; " ``` * Tiger Console 1. **Create the connector** 1. In Tiger Console, select the target service. 2. Click `Connectors` > `Source Postgres Connector`. 3. Set a name for the connector (click the pencil icon). 4. Check `Set wal_level to logical` and `Update your credentials`, then click `Continue`. 5. Enter the source database credentials or a PostgreSQL connection string, then click `Connect to database`. Tiger Console connects and retrieves the schema. 2. **Select tables** 1. In the `Select table` dropdown, choose the tables to sync. 2. Click `Select tables +`. 3. Tiger Console checks each table schema and suggests a time-dimension column for hypertable conversion where possible. 4. Click `Create Connector`. Tiger Console starts the connector and displays progress. 3. **Monitor** * **Data flow:** click `Connectors` to view the overall diagram showing connector status and data replicated. * **Per-table progress:** click `Connectors` > `Source connectors`, then select your connector name. 4. **Manage** * **Edit** (rename, add or remove tables): click `Connectors` > `Source connectors`, then select the connector name. * **Pause:** open the three-dot menu on the right > `Pause`. * **Delete:** pause first, then three-dot menu > `Delete`. ## Post-sync [Section titled “Post-sync”](#post-sync) After all tables reach state `r` or `s`: 1. **Apply policies** Apply the columnstore, refresh, and retention policies saved earlier: ```bash # From TimescaleDB source psql "$TARGET" -f columnstore_policies.sql psql "$TARGET" -f refresh_policies.sql psql "$TARGET" -f retention_policies.sql ``` For continuous aggregates, do a full initial refresh first: ```sql SELECT add_continuous_aggregate_policy('.', start_offset => NULL, end_offset => INTERVAL '30 seconds', schedule_interval => INTERVAL '1 hour' ); ``` Wait for completion, then switch to production intervals: ```sql SELECT remove_continuous_aggregate_policy('.'); -- Apply the saved refresh policy ``` 2. **Update statistics** ```bash vacuumdb --analyze --verbose --jobs=8 --dbname="$TARGET" ``` 3. **Apply database and role settings** Settings applied on the source via `ALTER DATABASE ... SET` or `ALTER ROLE ... IN DATABASE ... SET` (for example, `search_path`, `timezone`, `statement_timeout`, `default_transaction_isolation`) are **not** included in `pg_dump --schema-only` or `pg_dumpall --roles-only`. If your application relies on them, generate equivalent statements from the source catalogs and apply them on the target. Both `ALTER DATABASE ... SET` and `ALTER ROLE ... IN DATABASE ... SET` settings live in `pg_db_role_setting`, distinguished by `setrole = 0` (database-level) versus `setrole != 0` (role-specific). The query below reads settings from the source and generates statements that reference the target database name. Source and target database names typically differ (for example, source `migrate`, target `tsdb`): ```bash # Resolve the target database name TGT_DB=$(psql "$TARGET" -t -A -c "SELECT current_database();") psql "$SOURCE" -t -A -v tgt_db="$TGT_DB" <<'EOF' > db_settings.sql WITH pairs AS ( SELECT drs.setrole, r.rolname, split_part(s, '=', 1) AS guc, CASE WHEN substring(s FROM position('=' IN s) + 1) LIKE '"%"' THEN replace( substring(s FROM position('=' IN s) + 2 FOR length(s) - position('=' IN s) - 2), '""', '"') ELSE substring(s FROM position('=' IN s) + 1) END AS val FROM pg_db_role_setting drs JOIN pg_database d ON d.oid = drs.setdatabase LEFT JOIN pg_roles r ON r.oid = drs.setrole, LATERAL unnest(drs.setconfig) s WHERE d.datname = current_database() AND (drs.setrole = 0 OR (r.rolname NOT LIKE 'pg_%' AND r.rolname NOT IN ('postgres', 'tsdbadmin'))) ) SELECT CASE WHEN setrole = 0 THEN format('ALTER DATABASE %I SET %I = %L;', :'tgt_db', guc, val) ELSE format('ALTER ROLE %I IN DATABASE %I SET %I = %L;', rolname, :'tgt_db', guc, val) END FROM pairs ORDER BY setrole, guc; EOF cat db_settings.sql psql "$TARGET" -f db_settings.sql ``` Review the generated statements before applying. Some settings (for example, shared library paths, filesystem paths, replication-specific parameters) may not apply to a managed Tiger Cloud service and should be removed. ## Cutover [Section titled “Cutover”](#cutover) 1. **Validate** Confirm all tables are in state `r` or `s`: ```bash psql "$TARGET" -c " SELECT state, last_error, count(*) FROM _ts_live_sync.subscription_rel GROUP BY 1, 2 ORDER BY 1, 2; " ``` Confirm replication lag is near zero: ```bash psql "$SOURCE" -c " SELECT slot_name, pg_size_pretty(pg_current_wal_flush_lsn() - confirmed_flush_lsn) AS lag FROM pg_replication_slots WHERE slot_name LIKE 'live_sync_%' AND slot_type = 'logical'; " ``` 2. **Stop write traffic to the source** Stop all application writes to the source database. Then wait for replication lag to reach near zero: ```bash psql "$SOURCE" -c " SELECT slot_name, pg_size_pretty(pg_current_wal_flush_lsn() - confirmed_flush_lsn) AS lag FROM pg_replication_slots WHERE slot_name LIKE 'live_sync_%' AND slot_type = 'logical'; " ``` Re-run until lag is near 0 bytes. 3. **Reset sequences** Important Livesync does not replicate sequence values. Skipping this step causes serial or identity columns to reuse already-existing values after cutover, resulting in primary key or unique constraint violations. * copy-sequences command ```bash docker run -it --rm --name livesync-copy-sequences \ timescale/live-sync:latest copy-sequences \ --subscription \ --source "$SOURCE" \ --target "$TARGET" ``` * Manual SQL ```sql psql "$TARGET" <<'EOF' DO $$ DECLARE rec RECORD; BEGIN FOR rec IN ( SELECT sr.target_schema AS table_schema, sr.target_table AS table_name, col.column_name, pg_get_serial_sequence( format('%I.%I', sr.target_schema, sr.target_table), col.column_name ) AS seqname FROM _ts_live_sync.subscription_rel AS sr JOIN information_schema.columns AS col ON col.table_schema = sr.target_schema AND col.table_name = sr.target_table WHERE col.column_default LIKE 'nextval(%' ) LOOP EXECUTE format( 'SELECT setval(%L, COALESCE((SELECT MAX(%I) FROM %I.%I), 0) + 1, false);', rec.seqname, rec.column_name, rec.table_schema, rec.table_name ); END LOOP; END; $$ LANGUAGE plpgsql; EOF ``` 4. **Stop Livesync** ```bash docker stop livesync ``` 5. **Connect application** Update connection strings to point to Tiger Cloud. ## Cleanup [Section titled “Cleanup”](#cleanup) After the application is fully running on Tiger Cloud: ```bash docker run -it --rm --name livesync-cleanup \ timescale/live-sync:latest drop \ --subscription \ --source "$SOURCE" \ --target "$TARGET" ``` Then drop the migration user on the source. ## Next steps [Section titled “Next steps”](#next-steps) * [Troubleshooting](/docs/migrate/livesync-replication-troubleshooting): common issues you may hit during migration and how to resolve them. * [Advanced topics](/docs/migrate/livesync-replication-advanced): continuous aggregates, compressed hypertables on older TimescaleDB versions, and `GENERATED ALWAYS AS IDENTITY` columns. # Livesync replication advanced topics > Advanced Livesync replication scenarios for migrating to Tiger Cloud Advanced scenarios for [Livesync replication](/docs/migrate/livesync-replication) migrations: continuous aggregates, compressed hypertables on older TimescaleDB versions, and identity columns. ## Migrating continuous aggregates [Section titled “Migrating continuous aggregates”](#migrating-continuous-aggregates) Livesync replicates the continuous aggregate's underlying materialization hypertable, not the view. The continuous aggregate must already exist with data on the source and be defined (empty) on the target. 1. **Resolve materialization hypertable names on both source and target** The `_materialized_hypertable_` suffix differs per environment: ```sql SELECT materialization_hypertable_schema AS mat_schema, materialization_hypertable_name AS mat_name FROM timescaledb_information.continuous_aggregates WHERE view_schema = '' AND view_name = ''; ``` 2. **Set replica identity on the source materialization hypertable** Required when the publication replicates `UPDATE`/`DELETE` (the default). Continuous aggregate refreshes issue `DELETE`s; without a replica identity the next refresh fails with `cannot delete from table ... because it does not have a replica identity and publishes deletes`. ```sql ALTER TABLE _timescaledb_internal._materialized_hypertable_ REPLICA IDENTITY FULL; ``` 3. **Add the materialization hypertable to the publication** ```sql ALTER PUBLICATION ADD TABLE _timescaledb_internal._materialized_hypertable_; ``` 4. **Map source matview hypertable to target matview hypertable** * **Tiger Console:** edit the connector, select the source `_materialized_hypertable_`, and set its target mapping to the target's `_materialized_hypertable_`. * **Docker:** add a `--table-map` entry and refresh: ```text --table-map '{ "source": {"schema": "_timescaledb_internal", "table": "_materialized_hypertable_"}, "target": {"schema": "_timescaledb_internal", "table": "_materialized_hypertable_"} }' ``` 5. **Do not attach a refresh policy on the target** While Livesync is running, the target continuous aggregate is kept current via replication. Attach the refresh policy on the target only after cutover. ## Replica identity on compressed hypertables (older TimescaleDB) [Section titled “Replica identity on compressed hypertables (older TimescaleDB)”](#replica-identity-on-compressed-hypertables-older-timescaledb) On older TimescaleDB versions, `ALTER TABLE ... REPLICA IDENTITY ...` fails on compressed hypertables: ```text ERROR: operation not supported on hypertables that have compression enabled ``` Newer versions accept the command directly on the hypertable and propagate it to chunks. If you can upgrade TimescaleDB on the source first, do that and skip this section. If upgrading isn't an option, wrap the work in `timescaledb_pre_restore()` / `timescaledb_post_restore()` (these disable the catalog hooks that block the operation). Iterate over all chunks of the hypertable, set `REPLICA IDENTITY FULL` on each, and set it on the hypertable itself so new chunks inherit: ```bash psql "$SOURCE" -v schema='' -v hypertable='' <<'EOF' BEGIN; SELECT timescaledb_pre_restore(); SELECT format('ALTER TABLE %I.%I REPLICA IDENTITY FULL;', chunk_schema, chunk_name) FROM timescaledb_information.chunks WHERE hypertable_schema = :'schema' AND hypertable_name = :'hypertable' \gexec ALTER TABLE :"schema".:"hypertable" REPLICA IDENTITY FULL; SELECT timescaledb_post_restore(); COMMIT; EOF ``` Run as a single transaction. Dry-run on a non-production instance first. ## Tables with `GENERATED ALWAYS AS IDENTITY` columns [Section titled “Tables with GENERATED ALWAYS AS IDENTITY columns”](#tables-with-generated-always-as-identity-columns) CDC `UPDATE` replication currently fails on target tables whose columns are defined as `GENERATED ALWAYS AS IDENTITY`, because the apply worker tries to write the source value into a column that PostgreSQL forbids assigning to. Until Livesync handles this automatically, work around it by relaxing the column to `GENERATED BY DEFAULT` for the duration of the migration. This does not rewrite the column or change runtime behavior for existing sequences. 1. **Find affected columns on the source** ```sql psql "$SOURCE" <<'EOF' SELECT n.nspname AS schema, c.relname AS table, a.attname AS column FROM pg_attribute a JOIN pg_class c ON c.oid = a.attrelid JOIN pg_namespace n ON n.oid = c.relnamespace WHERE a.attidentity = 'a' -- 'a' = ALWAYS, 'd' = BY DEFAULT AND c.relkind IN ('r', 'p') AND n.nspname NOT IN ('pg_catalog', 'information_schema') ORDER BY n.nspname, c.relname, a.attnum; EOF ``` 2. **Relax to `BY DEFAULT` before migration** For each row returned above, run on **both** source and target: ```sql ALTER TABLE .
ALTER COLUMN SET GENERATED BY DEFAULT; ``` 3. **Restore to `ALWAYS` after cutover** After the application is running on Tiger Cloud, restore the original constraint on the target: ```sql ALTER TABLE .
ALTER COLUMN SET GENERATED ALWAYS; ``` # Livesync replication troubleshooting > Troubleshoot Livesync replication when migrating to Tiger Cloud Common issues you may hit while running a [Livesync replication](/docs/migrate/livesync-replication) migration to Tiger Cloud, and how to resolve them. ## `_ts_live_sync.subscription_rel` empty [Section titled “\_ts\_live\_sync.subscription\_rel empty”](#_ts_live_syncsubscription_rel-empty) It may take up to 30 minutes for large publications before `_ts_live_sync.subscription_rel` is populated. Check Docker logs: ```bash docker logs -f livesync ``` ## Replay not keeping up [Section titled “Replay not keeping up”](#replay-not-keeping-up) Add indexes on `UPDATE`/`DELETE` `WHERE`-clause columns on the target. Ensure `REPLICA IDENTITY` includes the partition column on hypertables. ## Lag growing [Section titled “Lag growing”](#lag-growing) Scale up the target. Check for long-running transactions on the source that hold `xmin` and prevent WAL cleanup: ```sql SELECT pid, age(backend_xmin) AS xmin_age, query FROM pg_stat_activity WHERE backend_xmin IS NOT NULL ORDER BY xmin_age DESC; ``` ## Initial data copy is slow or target IOPS saturated [Section titled “Initial data copy is slow or target IOPS saturated”](#initial-data-copy-is-slow-or-target-iops-saturated) If the target's IOPS or disk throughput is pinned during the initial data copy, the most common cause is index maintenance on the destination tables. Each non-PK index has to be updated for every `COPY`'d row, which multiplies the I/O cost. Drop the non-PK indexes on the target before the copy and recreate them after all tables reach state `r` or `s`. Primary-key and unique-constraint indexes must stay; Livesync uses them for `INSERT ... ON CONFLICT` deduplication. ```sql -- On the TARGET, before initial copy: DROP INDEX IF EXISTS .; DROP INDEX IF EXISTS .; -- ... repeat for each non-PK index ``` After the initial copy completes, recreate the indexes (preferably in parallel sessions for large tables): ```sql SET maintenance_work_mem = '2GB'; SET max_parallel_maintenance_workers = 4; CREATE INDEX ON .
(...); -- ... etc. ``` Other knobs to consider: lower `--table-sync-workers`, scale up target IOPS or CPU, or stagger heavy tables. ## Restart a single table [Section titled “Restart a single table”](#restart-a-single-table) 1. **Remove the table from the publication on the source** ```sql ALTER PUBLICATION DROP TABLE .
; ``` 2. **Stop and restart Livesync to remove catalog objects for the dropped table** 3. **Confirm removal (this should return empty)** ```sql psql "$TARGET" -c " SELECT * FROM _ts_live_sync.subscription_rel WHERE tablename = '
'; " ``` 4. **Truncate the table on the target** ```sql TRUNCATE .
; ``` 5. **Add the table back to the publication** ```sql ALTER PUBLICATION ADD TABLE .
; ``` 6. **Stop and restart Livesync to begin syncing the re-added table** ## Restart everything [Section titled “Restart everything”](#restart-everything) Stop Livesync, drop the subscription, truncate all target tables, then restart the container. Important Failing to truncate target tables leads to duplicate-key violation errors or duplicate records on the target. # Migrate with downtime > Migrate a hypertable or an entire database to Tiger Cloud with native PostgreSQL commands using pg_dump and pg_restore You use downtime migration to move less than 100GB of data from a self-hosted database to a Tiger Cloud service. Downtime migration uses the native PostgreSQL [`pg_dump`](https://www.postgresql.org/docs/current/app-pgdump.html) and [`pg_restore`](https://www.postgresql.org/docs/current/app-pgrestore.html) commands. If you are migrating from self-hosted TimescaleDB, this method works for hypertables compressed into the columnstore without having to convert the data back to the rowstore before you begin. Tips If you want to migrate more than 400GB of data, create a [Tiger Console support request](https://console.cloud.tigerdata.com/dashboard/support), or send us an email at saying how much data you want to migrate. We pre-provision your Tiger Cloud service for you. However, downtime migration for large amounts of data takes a large amount of time. For more than 100GB of data, best practice is to follow [live migration](/docs/migrate/live-migration). This page shows you how to move your data from a self-hosted database to a Tiger Cloud service using shell commands. ## Prerequisites for this procedure To follow these steps, you'll need: * A target [Tiger Cloud service](/docs/get-started/quickstart/create-service). Best practice is to create a Tiger Cloud service with at least 8 CPUs for a smoother experience. A higher-spec instance can significantly reduce the overall migration window. * A migration machine to run the commands that move data from your source database to your target Tiger Cloud service. Best practice: use an [Ubuntu EC2 instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EC2_GetStarted.html#ec2-launch-instance) hosted in the same region as your Tiger Cloud service. * An [adjusted maintenance window](/docs/deploy/tiger-cloud/tiger-cloud-aws/upgrades#define-your-maintenance-window) to prevent maintenance from running during migration. - The PostgreSQL client tools installed on your migration machine. This includes `psql`, `pg_dump`, and `pg_dumpall`. - The GNU implementation of `sed` installed. Run `sed --version` on your migration machine. GNU sed identifies itself as GNU software, BSD sed returns `sed: illegal option -- -`. ## Migrate to Tiger Cloud [Section titled “Migrate to Tiger Cloud”](#migrate-to-tiger-cloud) To move your data from a self-hosted database to a Tiger Cloud service: * From TimescaleDB This section shows you how to move your data from self-hosted TimescaleDB to a Tiger Cloud service using `pg_dump` and `psql` from Terminal. ## Prepare to migrate 1. **Take the applications that connect to the source database offline** The duration of the migration is proportional to the amount of data stored in your database. By disconnecting your app from your database you avoid any possible data loss. 2. **Set your connection strings** These variables hold the connection information for the source database and target Tiger Cloud service: ```bash export SOURCE="postgres://:@:/" export TARGET="postgres://tsdbadmin:@:/tsdb?sslmode=require" ``` You find the connection information for your Tiger Cloud service in the configuration file you downloaded when you created the service. ## Align the version of TimescaleDB on the source and target 1. **Ensure that the source and target databases are running the same version of TimescaleDB** 1. Check the version of TimescaleDB running on your Tiger Cloud service: ```bash psql $TARGET -c "SELECT extversion FROM pg_extension WHERE extname = 'timescaledb';" ``` 2. Update the TimescaleDB extension in your source database to match the target service: If the TimescaleDB extension is the same version on the source database and target service, you do not need to do this. ```bash psql $SOURCE -c "ALTER EXTENSION timescaledb UPDATE TO '';" ``` For more information and guidance, see [Upgrade TimescaleDB](/docs/deploy/self-hosted/upgrades). 2. **Ensure that the Tiger Cloud service is running the PostgreSQL extensions used in your source database** 1. Check the extensions on the source database: ```bash psql $SOURCE -c "SELECT * FROM pg_extension;" ``` 2. For each extension, enable it on your target Tiger Cloud service: ```bash psql $TARGET -c "CREATE EXTENSION IF NOT EXISTS CASCADE;" ``` ## Migrate the roles from TimescaleDB to your Tiger Cloud service Roles manage database access permissions. To migrate your role-based security hierarchy to your Tiger Cloud service: 1. **Dump the roles from your source database** Export your role-based security hierarchy. `` has the same value as `` in `$SOURCE`. I know, it confuses me as well. ```bash pg_dumpall -d "$SOURCE" \ -l --quote-all-identifiers \ --roles-only \ --file=roles.sql ``` If you only use the default `postgres` role, this step is not necessary. 2. **Remove roles with superuser access** Tiger Cloud service do not support roles with superuser access. Run the following script to remove statements, permissions and clauses that require superuser permissions from `roles.sql`: ```bash sed -i -E \ -e '/CREATE ROLE "postgres";/d' \ -e '/ALTER ROLE "postgres"/d' \ -e '/CREATE ROLE "tsdbadmin";/d' \ -e '/ALTER ROLE "tsdbadmin"/d' \ -e 's/(NO)*SUPERUSER//g' \ -e 's/(NO)*REPLICATION//g' \ -e 's/(NO)*BYPASSRLS//g' \ -e 's/GRANTED BY "[^"]*"//g' \ roles.sql ``` 3. **Dump the source database schema and data** The `pg_dump` flags remove superuser access and tablespaces from your data. When you run `pg_dump`, check the run time, [a long-running `pg_dump` can cause issues](/docs/build/tips-and-tricks/troubleshoot-import-ingest). ```bash pg_dump -d "$SOURCE" \ --format=plain \ --quote-all-identifiers \ --no-tablespaces \ --no-owner \ --no-privileges \ --file=dump.sql ``` To dramatically reduce the time taken to dump the source database, use multiple connections. For more information, see [dumping with concurrency](/docs/build/tips-and-tricks/troubleshoot-import-ingest) and [restoring with concurrency](/docs/build/tips-and-tricks/troubleshoot-import-ingest). ## Upload your data to the target Tiger Cloud service This command uses the \[timescaledb\_pre\_restore] and \[timescaledb\_post\_restore] functions to put your database in the correct state. ```bash psql $TARGET -v ON_ERROR_STOP=1 --echo-errors \ -f roles.sql \ -c "SELECT timescaledb_pre_restore();" \ -f dump.sql \ -c "SELECT timescaledb_post_restore();" ``` ## Validate your Tiger Cloud service and restart your app 1. **Update the table statistics** ```bash psql $TARGET -c "ANALYZE;" ``` 2. **Verify the data in the target Tiger Cloud service** Check that your data is correct, and returns the results that you expect. 3. **Enable any Tiger Cloud features you want to use** Migration from PostgreSQL moves the data only. Now manually enable Tiger Cloud features like [hypertables](/docs/learn/hypertables/understand-hypertables), [hypercore](/docs/learn/columnar-storage/understand-hypercore) or [data retention](/docs/learn/data-lifecycle/data-retention/about-data-retention) while your database is offline. 4. **Reconfigure your app to use the target database, then restart it** And that is it, you have migrated your data from a self-hosted instance running TimescaleDB to a Tiger Cloud service. * From PostgreSQL This section shows you how to move your data from self-hosted PostgreSQL to a Tiger Cloud service using `pg_dump` and `psql` from Terminal. Migration from PostgreSQL moves the data only. You must manually enable Tiger Cloud features like [hypertables](/docs/learn/hypertables/understand-hypertables), [hypercore](/docs/learn/columnar-storage/understand-hypercore) or [data retention](/docs/learn/data-lifecycle/data-retention/about-data-retention) after the migration is complete. You enable Tiger Cloud features while your database is offline. ## Prepare to migrate 1. **Take the applications that connect to the source database offline** The duration of the migration is proportional to the amount of data stored in your database. By disconnecting your app from your database you avoid any possible data loss. 2. **Set your connection strings** These variables hold the connection information for the source database and target Tiger Cloud service: ```bash export SOURCE="postgres://:@:/" export TARGET="postgres://tsdbadmin:@:/tsdb?sslmode=require" ``` You find the connection information for your Tiger Cloud service in the configuration file you downloaded when you created the service. ## Align the extensions on the source and target 1. **Ensure that the Tiger Cloud service is running the PostgreSQL extensions used in your source database** 1. Check the extensions on the source database: ```bash psql $SOURCE -c "SELECT * FROM pg_extension;" ``` 2. For each extension, enable it on your target Tiger Cloud service: ```bash psql $TARGET -c "CREATE EXTENSION IF NOT EXISTS CASCADE;" ``` ## Migrate the roles from TimescaleDB to your Tiger Cloud service Roles manage database access permissions. To migrate your role-based security hierarchy to your Tiger Cloud service: 1. **Dump the roles from your source database** Export your role-based security hierarchy. `` has the same value as `` in `$SOURCE`. I know, it confuses me as well. ```bash pg_dumpall -d "$SOURCE" \ -l --quote-all-identifiers \ --roles-only \ --file=roles.sql ``` If you only use the default `postgres` role, this step is not necessary. 2. **Remove roles with superuser access** Tiger Cloud service do not support roles with superuser access. Run the following script to remove statements, permissions and clauses that require superuser permissions from `roles.sql`: ```bash sed -i -E \ -e '/CREATE ROLE "postgres";/d' \ -e '/ALTER ROLE "postgres"/d' \ -e '/CREATE ROLE "tsdbadmin";/d' \ -e '/ALTER ROLE "tsdbadmin"/d' \ -e 's/(NO)*SUPERUSER//g' \ -e 's/(NO)*REPLICATION//g' \ -e 's/(NO)*BYPASSRLS//g' \ -e 's/GRANTED BY "[^"]*"//g' \ roles.sql ``` 3. **Dump the source database schema and data** The `pg_dump` flags remove superuser access and tablespaces from your data. When you run `pg_dump`, check the run time, [a long-running `pg_dump` can cause issues](/docs/build/tips-and-tricks/troubleshoot-import-ingest). ```bash pg_dump -d "$SOURCE" \ --format=plain \ --quote-all-identifiers \ --no-tablespaces \ --no-owner \ --no-privileges \ --file=dump.sql ``` To dramatically reduce the time taken to dump the source database, use multiple connections. For more information, see [dumping with concurrency](/docs/build/tips-and-tricks/troubleshoot-import-ingest) and [restoring with concurrency](/docs/build/tips-and-tricks/troubleshoot-import-ingest). ## Upload your data to the target Tiger Cloud service ```bash psql $TARGET -v ON_ERROR_STOP=1 --echo-errors \ -f roles.sql \ -f dump.sql ``` ## Validate your Tiger Cloud service and restart your app 1. **Update the table statistics** ```bash psql $TARGET -c "ANALYZE;" ``` 2. **Verify the data in the target Tiger Cloud service** Check that your data is correct, and returns the results that you expect. 3. **Enable any Tiger Cloud features you want to use** Migration from PostgreSQL moves the data only. Now manually enable Tiger Cloud features like [hypertables](/docs/learn/hypertables/understand-hypertables), [hypercore](/docs/learn/columnar-storage/understand-hypercore) or [data retention](/docs/learn/data-lifecycle/data-retention/about-data-retention) while your database is offline. 4. **Reconfigure your app to use the target database, then restart it** And that is it, you have migrated your data from a self-hosted instance running PostgreSQL to a Tiger Cloud service. * From AWS RDS/Aurora To migrate your data from an Amazon RDS/Aurora PostgreSQL instance to a Tiger Cloud service, you extract the data to an intermediary EC2 Ubuntu instance in the same AWS region as your RDS/Aurora PostgreSQL instance. You then upload your data to a Tiger Cloud service. To make this process as painless as possible, ensure that the intermediary machine has enough CPU and disk space to rapidly extract and store your data before uploading to Tiger Cloud. Migration from RDS/Aurora PostgreSQL moves the data only. You must manually enable Tiger Cloud features like [hypertables](/docs/learn/hypertables/understand-hypertables), [data compression](/docs/learn/columnar-storage/understand-hypercore) or [data retention](/docs/learn/data-lifecycle/data-retention/about-data-retention) after the migration is complete. You enable Tiger Cloud features while your database is offline. This section shows you how to move your data from a PostgreSQL database running in an Amazon RDS/Aurora PostgreSQL instance to a Tiger Cloud service using `pg_dump` and `psql` from Terminal. ## Create an intermediary EC2 Ubuntu instance 1. **Select the RDS/Aurora instance to migrate** In , select the RDS/Aurora PostgreSQL instance to migrate. 2. **Click `Actions` > `Set up EC2 connection`** Press `Create EC2 instance` and use the following settings: * **AMI**: Ubuntu Server. * **Key pair**: use an existing pair or create a new one that you will use to access the intermediary machine. * **VPC**: by default, this is the same as the database instance. * **Configure Storage**: adjust the volume to at least the size of RDS/Aurora PostgreSQL instance you are migrating from. You can reduce the space used by your data on Tiger Cloud using [Hypercore](/docs/learn/columnar-storage/understand-hypercore). 3. **Click `Launch instance`, then connect via SSH** AWS creates your EC2 instance. Click `Connect to instance` > `SSH client` and follow the instructions to create the connection to your intermediary EC2 instance. ## Install the psql client tools on the intermediary instance 1. **Connect to your intermediary EC2 instance. For example:** ```sh ssh -i ".pem" ubuntu@ ``` 2. **On your intermediary EC2 instance, install the PostgreSQL client.** ```sh sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list' wget -qO- https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo tee /etc/apt/trusted.gpg.d/pgdg.asc &>/dev/null sudo apt update sudo apt install postgresql-client-16 -y # "postgresql-client-16" if your source DB is using PG 16. psql --version && pg_dump --version ``` Keep this terminal open, you need it to connect to the RDS/Aurora PostgreSQL instance for migration. ## Set up secure connectivity between your RDS/Aurora PostgreSQL and EC2 instances 1. **Select the RDS/Aurora instance to migrate** In , select the RDS/Aurora PostgreSQL instance to migrate. 2. **Open the inbound rules for the security group** Scroll down to `Security group rules (1)` and select the `EC2 Security Group - Inbound` group. The `Security Groups (1)` window opens. Click the `Security group ID`, then click `Edit inbound rules`. ![Creating a security group rule to enable RDS/Aurora to EC2 connection](/docs/_astro/rds-add-security-rule-to-ec2-instance.BXp5di3z.svg) 3. **On your intermediary EC2 instance, get your local IP address:** ```sh ec2metadata --local-ipv4 ``` Bear with me on this one, you need this IP address to enable access to your RDS/Aurora PostgreSQL instance. 4. **Add inbound rule for your EC2 instance** In `Edit inbound rules`, click `Add rule`, then create a PostgreSQL `TCP` rule granting access to the local IP address for your EC2 instance. Then click `Save rules`. ![Adding an inbound security rule for the EC2 instance](/docs/_astro/rds-add-inbound-rule-for-ec2-instance.BElzIqvO_Z2Ld7z.webp) ## Test the connection between your RDS/Aurora PostgreSQL and EC2 instances 1. **Select the RDS/Aurora instance to migrate** In , select the RDS/Aurora PostgreSQL instance to migrate. 2. **Create the source connection string** On your intermediary EC2 instance, use the values of `Endpoint`, `Port`, `Master username`, and `DB name` to create the PostgreSQL connectivity string for the `SOURCE` variable. ![Recording the RDS endpoint, port, and VPC details](/docs/_astro/migrate-source-rds-instance.BHStcVYP.svg) ```sh export SOURCE="postgres://:@:/" ``` The value of `Master password` was supplied when this RDS/Aurora PostgreSQL instance was created. 3. **Test your connection:** ```sh psql -d $SOURCE ``` You are connected to your RDS/Aurora PostgreSQL instance from your intermediary EC2 instance. ## Migrate your data to your Tiger Cloud service To securely migrate data from your RDS instance: ## Prepare to migrate 1. **Take the applications that connect to the RDS instance offline** The duration of the migration is proportional to the amount of data stored in your database.\ By disconnection your app from your database you avoid and possible data loss. You should also ensure that your source RDS instance is not receiving any DML queries. 2. **Connect to your intermediary EC2 instance** For example: ```sh ssh -i ".pem" ubuntu@ ``` 3. **Set your connection strings** These variables hold the connection information for the RDS instance and target Tiger Cloud service: ```bash export SOURCE="postgres://:@:/" export TARGET=postgres://tsdbadmin:@:/tsdb?sslmode=require ``` You find the connection information for `SOURCE` in your RDS configuration. For `TARGET` in the configuration file you downloaded when you created the Tiger Cloud service. ## Align the extensions on the source and target 1. **Ensure that the Tiger Cloud service is running the PostgreSQL extensions used in your source database** 1. Check the extensions on the source database: ```bash psql $SOURCE -c "SELECT * FROM pg_extension;" ``` 2. For each extension, enable it on your target Tiger Cloud service: ```bash psql $TARGET -c "CREATE EXTENSION IF NOT EXISTS CASCADE;" ``` ## Migrate roles from RDS to your Tiger Cloud service Roles manage database access permissions. To migrate your role-based security hierarchy to your Tiger Cloud service: 1. **Dump the roles from your RDS instance** Export your role-based security hierarchy. If you only use the default `postgres` role, this step is not necessary. ```bash pg_dumpall -d "$SOURCE" \ --quote-all-identifiers \ --roles-only \ --no-role-passwords \ --file=roles.sql ``` AWS RDS does not allow you to export passwords with roles. You assign passwords to these roles when you have uploaded them to your Tiger Cloud service. 2. **Remove roles with superuser access** Tiger Cloud services do not support roles with superuser access. Run the following script to remove statements, permissions and clauses that require superuser permissions from `roles.sql`: ```bash sed -i -E \ -e '/CREATE ROLE "postgres";/d' \ -e '/ALTER ROLE "postgres"/d' \ -e '/CREATE ROLE "rds/d' \ -e '/ALTER ROLE "rds/d' \ -e '/TO "rds/d' \ -e '/GRANT "rds/d' \ -e 's/(NO)*SUPERUSER//g' \ -e 's/(NO)*REPLICATION//g' \ -e 's/(NO)*BYPASSRLS//g' \ -e 's/GRANTED BY "[^"]*"//g' \ roles.sql ``` 3. **Upload the roles to your Tiger Cloud service** ```bash psql -X -d "$TARGET" \ -v ON_ERROR_STOP=1 \ --echo-errors \ -f roles.sql ``` 4. **Manually assign passwords to the roles** AWS RDS did not allow you to export passwords with roles. For each role, use the following command to manually assign a password to a role: ```bash psql $TARGET -c "ALTER ROLE WITH PASSWORD '';" ``` ## Migrate data from your RDS instance to your Tiger Cloud service 1. **Dump the data from your RDS instance to your intermediary EC2 instance** The `pg_dump` flags remove superuser access and tablespaces from your data. When you run `pgdump`, check the run time, [a long-running `pg_dump` can cause issues](/docs/migrate/troubleshooting#dumping-and-locks). ```bash pg_dump -d "$SOURCE" \ --format=plain \ --quote-all-identifiers \ --no-tablespaces \ --no-owner \ --no-privileges \ --file=dump.sql ``` To dramatically reduce the time taken to dump the RDS instance, using multiple connections. For more information, see [dumping with concurrency](/docs/migrate/troubleshooting#dumping-with-concurrency) and [restoring with concurrency](/docs/migrate/troubleshooting#restoring-with-concurrency). 2. **Upload your data to your Tiger Cloud service** ```bash psql -d $TARGET -v ON_ERROR_STOP=1 --echo-errors \ -f dump.sql ``` ## Validate your Tiger Cloud service and restart your app 1. **Update the table statistics** ```bash psql $TARGET -c "ANALYZE;" ``` 2. **Verify the data in the target Tiger Cloud service** Check that your data is correct, and returns the results that you expect. 3. **Enable any Tiger Cloud features you want to use** Migration from PostgreSQL moves the data only. Now manually enable Tiger Cloud features like [hypertables](/docs/learn/hypertables/understand-hypertables), [hypercore](/docs/learn/columnar-storage/understand-hypercore) or [data retention](/docs/learn/data-lifecycle/data-retention/about-data-retention) while your database is offline. 4. **Reconfigure your app to use the target database, then restart it** And that is it, you have migrated your data from an RDS/Aurora PostgreSQL instance to a Tiger Cloud service. * From MST This section shows you how to move your data from a Managed Service for TimescaleDB instance to a Tiger Cloud service using `pg_dump` and `psql` from Terminal. ## Prepare to migrate 1. **Take the applications that connect to the source database offline** The duration of the migration is proportional to the amount of data stored in your database. By disconnecting your app from your database you avoid any possible data loss. 2. **Set your connection strings** These variables hold the connection information for the source database and target Tiger Cloud service: ```bash export SOURCE="postgres://:@:/" export TARGET="postgres://tsdbadmin:@:/tsdb?sslmode=require" ``` You find the connection information for your Tiger Cloud service in the configuration file you downloaded when you created the service. ## Align the version of TimescaleDB on the source and target 1. **Ensure that the source and target databases are running the same version of TimescaleDB** 1. Check the version of TimescaleDB running on your Tiger Cloud service: ```bash psql $TARGET -c "SELECT extversion FROM pg_extension WHERE extname = 'timescaledb';" ``` 2. Update the TimescaleDB extension in your source database to match the target service: If the TimescaleDB extension is the same version on the source database and target service, you do not need to do this. ```bash psql $SOURCE -c "ALTER EXTENSION timescaledb UPDATE TO '';" ``` For more information and guidance, see [Upgrade TimescaleDB](/docs/deploy/self-hosted/upgrades). 2. **Ensure that the Tiger Cloud service is running the PostgreSQL extensions used in your source database** 1. Check the extensions on the source database: ```bash psql $SOURCE -c "SELECT * FROM pg_extension;" ``` 2. For each extension, enable it on your target Tiger Cloud service: ```bash psql $TARGET -c "CREATE EXTENSION IF NOT EXISTS CASCADE;" ``` ## Migrate the roles from TimescaleDB to your Tiger Cloud service Roles manage database access permissions. To migrate your role-based security hierarchy to your Tiger Cloud service: 1. **Dump the roles from your source database** Export your role-based security hierarchy. `` has the same value as `` in `$SOURCE`. I know, it confuses me as well. ```bash pg_dumpall -d "$SOURCE" \ -l \ --quote-all-identifiers \ --roles-only \ --no-role-passwords \ --file=roles.sql ``` MST does not allow you to export passwords with roles. You assign passwords to these roles when you have uploaded them to your Tiger Cloud service. 2. **Remove roles with superuser access** Tiger Cloud services do not support roles with superuser access. Run the following script to remove statements, permissions and clauses that require superuser permissions from `roles.sql`: ```bash sed -i -E \ -e '/DROP ROLE IF EXISTS "postgres";/d' \ -e '/DROP ROLE IF EXISTS "tsdbadmin";/d' \ -e '/CREATE ROLE "postgres";/d' \ -e '/ALTER ROLE "postgres"/d' \ -e '/CREATE ROLE "rds/d' \ -e '/ALTER ROLE "rds/d' \ -e '/TO "rds/d' \ -e '/GRANT "rds/d' \ -e '/GRANT "pg_read_all_stats" TO "tsdbadmin"/d' \ -e 's/(NO)*SUPERUSER//g' \ -e 's/(NO)*REPLICATION//g' \ -e 's/(NO)*BYPASSRLS//g' \ -e 's/GRANTED BY "[^"]*"//g' \ -e '/CREATE ROLE "tsdbadmin";/d' \ -e '/ALTER ROLE "tsdbadmin"/d' \ -e 's/WITH ADMIN OPTION,/WITH /g' \ -e 's/WITH ADMIN OPTION//g' \ -e 's/GRANTED BY ".*"//g' \ -e '/GRANT "pg_.*" TO/d' \ -e '/CREATE ROLE "_aiven";/d' \ -e '/ALTER ROLE "_aiven"/d' \ -e '/GRANT SET ON PARAMETER "pgaudit\.[^"]+" TO "_tsdbadmin_auditing"/d' \ -e '/GRANT SET ON PARAMETER "anon\.[^"]+" TO "tsdbadmin_group"/d' \ -e '/ALTER ROLE .* SET "pg_stat_statements\./d' \ -e '/ALTER ROLE .* SET "pg_stat_monitor\./d' \ -e '/ALTER ROLE "_tsdbadmin_monitor" SET/d' \ roles.sql ``` 3. **Dump the source database schema and data** The `pg_dump` flags remove superuser access and tablespaces from your data. When you run `pg_dump`, check the run time, [a long-running `pg_dump` can cause issues](/docs/build/tips-and-tricks/troubleshoot-import-ingest). ```bash pg_dump -d "$SOURCE" \ --format=plain \ --quote-all-identifiers \ --no-tablespaces \ --no-owner \ --no-privileges \ --file=dump.sql ``` To dramatically reduce the time taken to dump the source database, use multiple connections. For more information, see [dumping with concurrency](/docs/build/tips-and-tricks/troubleshoot-import-ingest) and [restoring with concurrency](/docs/build/tips-and-tricks/troubleshoot-import-ingest). ## Upload your data to the target Tiger Cloud service This command uses the \[timescaledb\_pre\_restore] and \[timescaledb\_post\_restore] functions to put your database in the correct state. 1. **Upload your data** ```bash psql $TARGET -v ON_ERROR_STOP=1 --echo-errors \ -f roles.sql \ -c "SELECT timescaledb_pre_restore();" \ -f dump.sql \ -c "SELECT timescaledb_post_restore();" ``` 2. **Manually assign passwords to the roles** MST did not allow you to export passwords with roles. For each role, use the following command to manually assign a password to a role: ```bash psql $TARGET -c "ALTER ROLE WITH PASSWORD '';" ``` ## Validate your Tiger Cloud service and restart your app 1. **Update the table statistics** ```bash psql $TARGET -c "ANALYZE;" ``` 2. **Verify the data in the target Tiger Cloud service** Check that your data is correct, and returns the results that you expect. 3. **Enable any Tiger Cloud features you want to use** Migration from PostgreSQL moves the data only. Now manually enable Tiger Cloud features like [hypertables](/docs/learn/hypertables/understand-hypertables), [hypercore](/docs/learn/columnar-storage/understand-hypercore) or [data retention](/docs/learn/data-lifecycle/data-retention/about-data-retention) while your database is offline. 4. **Reconfigure your app to use the target database, then restart it** And that is it, you have migrated your data from a Managed Service for TimescaleDB instance to a Tiger Cloud service. # FAQ and troubleshooting > Troubleshooting known issues in database migrations to Tiger Cloud ## Unsupported in live migration [Section titled “Unsupported in live migration”](#unsupported-in-live-migration) Live migration tooling is currently experimental. You may run into the following shortcomings: * Live migration does not yet support mutable columnstore compression (`INSERT`, `UPDATE`, `DELETE` on data in the columnstore). * By default, numeric fields containing `NaN`/`+Inf`/`-Inf` values are not correctly replicated, and will be converted to `NULL`. A workaround is available, but is not enabled by default. Should you run into any problems, please open a support request before losing any time debugging issues. You can open a support request directly from [Tiger Console](https://console.cloud.tigerdata.com/dashboard/support/cases), or by email to . ## Where can I find logs for processes running during live migration? [Section titled “Where can I find logs for processes running during live migration?”](#where-can-i-find-logs-for-processes-running-during-live-migration) Live migration involves several background processes to manage different stages of the migration. The logs of these processes can be helpful for troubleshooting unexpected behavior. You can find these logs in the `/logs` directory. ## Source and target databases have different TimescaleDB versions [Section titled “Source and target databases have different TimescaleDB versions”](#source-and-target-databases-have-different-timescaledb-versions) When you migrate a self-hosted or Managed Service for TimescaleDB (MST) database to Tiger Cloud, the source database and the destination Tiger Cloud service must run the same version of TimescaleDB. Before you start [live migration](/docs/migrate/live-migration): 1. **Check the version of TimescaleDB running on both databases** ```sql select extversion from pg_extension where extname = 'timescaledb'; ``` 2. **If the version on the source is lower than your Tiger Cloud service, either downgrade or upgrade** * **Downgrade**: reinstall an older version of TimescaleDB on your Tiger Cloud service that matches the source database: 1. Connect to your Tiger Cloud service and check the versions of TimescaleDB available: ```sql SELECT version FROM pg_available_extension_versions WHERE name = 'timescaledb' ORDER BY 1 DESC; ``` 2. If an available TimescaleDB release matches your source database, uninstall and reinstall: ```sql DROP EXTENSION timescaledb; CREATE EXTENSION timescaledb VERSION ''; ``` Tips You may need to reconnect to your Tiger Cloud service using `psql -X` when you're creating the TimescaleDB extension. * **Upgrade**: for self-hosted databases, [upgrade TimescaleDB](/docs/deploy/self-hosted/upgrades) to match your Tiger Cloud service. ## Why does live migration log "no tuple identifier" warning? [Section titled “Why does live migration log "no tuple identifier" warning?”](#why-does-live-migration-log-no-tuple-identifier-warning) Live migration logs a warning `WARNING: no tuple identifier for UPDATE in table` when it cannot determine which specific rows should be updated after receiving an `UPDATE` statement from the source database during replication. This occurs when tables in the source database that receive `UPDATE` statements lack either a `PRIMARY KEY` or a `REPLICA IDENTITY` setting. For live migration to successfully replicate `UPDATE` and `DELETE` statements, tables must have either a `PRIMARY KEY` or `REPLICA IDENTITY` set as a prerequisite. ## Set REPLICA IDENTITY on PostgreSQL partitioned tables [Section titled “Set REPLICA IDENTITY on PostgreSQL partitioned tables”](#set-replica-identity-on-postgresql-partitioned-tables) If your PostgreSQL tables use native partitioning, setting `REPLICA IDENTITY` on the root (parent) table will not automatically apply it to the partitioned child tables. You must manually set `REPLICA IDENTITY` on each partitioned child table. ## Can I use read/failover replicas as source database for live migration? [Section titled “Can I use read/failover replicas as source database for live migration?”](#can-i-use-readfailover-replicas-as-source-database-for-live-migration) Live migration does not support replication from read or failover replicas. You must provide a connection string that points directly to your source database for live migration. ## Can I use live migration with a PostgreSQL connection pooler like PgBouncer? [Section titled “Can I use live migration with a PostgreSQL connection pooler like PgBouncer?”](#can-i-use-live-migration-with-a-postgresql-connection-pooler-like-pgbouncer) Live migration does not support connection poolers. You must provide a connection string that points directly to your source and target databases for live migration to work smoothly. ## Can I use Tiger Cloud instance as source for live migration? [Section titled “Can I use Tiger Cloud instance as source for live migration?”](#can-i-use-tiger-cloud-instance-as-source-for-live-migration) Yes, but logical replication must be enabled first. [Contact us](mailto:support@tigerdata.com) to enable. ## How can I exclude a schema/table from being replicated in live migration? [Section titled “How can I exclude a schema/table from being replicated in live migration?”](#how-can-i-exclude-a-schematable-from-being-replicated-in-live-migration) At present, live migration does not allow for excluding schemas or tables from replication, but this feature is expected to be added in future releases. However, a workaround is available for skipping table data using the `--skip-table-data` flag. For more information, please refer to the help text under the `migrate` subcommand. ## Large migrations blocked [Section titled “Large migrations blocked”](#large-migrations-blocked) Tiger Cloud automatically manages the underlying disk volume. Due to platform limitations, it is only possible to resize the disk once every six hours. Depending on the rate at which you're able to copy data, you may be affected by this restriction. Affected instances are unable to accept new data and error with: `FATAL: terminating connection due to administrator command`. If you intend on migrating more than 400 GB of data to Tiger Cloud, open a support request requesting the required storage to be pre-allocated in your Tiger Cloud service. You can open a support request directly from [Tiger Console](https://console.cloud.tigerdata.com/dashboard/support/cases), or by email to . ## Dumping and locks [Section titled “Dumping and locks”](#dumping-and-locks) When `pg_dump` starts, it takes an `ACCESS SHARE` lock on all tables which it dumps. This ensures that tables aren't dropped before `pg_dump` is able to drop them. A side effect of this is that any query which tries to take an `ACCESS EXCLUSIVE` lock on a table is blocked by the `ACCESS SHARE` lock. A number of Tiger Cloud-internal processes require taking `ACCESS EXCLUSIVE` locks to ensure consistency of the data. The following is a non-exhaustive list of potentially affected operations: * converting a chunk into the columnstore/rowstore and back * continuous aggregate refresh (before 2.12) * create hypertable with foreign keys, truncate hypertable * enable hypercore on a hypertable * drop chunks The most likely impact of the above is that background jobs for retention policies, columnstore compression policies, and continuous aggregate refresh policies are blocked for the duration of the `pg_dump` command. This may have unintended consequences for your database performance. ## Dumping with concurrency [Section titled “Dumping with concurrency”](#dumping-with-concurrency) When using the `pg_dump` directory format, it is possible to use concurrency to use multiple connections to the source database to dump data. This speeds up the dump process. Due to the fact that there are multiple connections, it is possible for `pg_dump` to end up in a deadlock situation. When it detects a deadlock it aborts the dump. In principle, any query which takes an `ACCESS EXCLUSIVE` lock on a table causes such a deadlock. As mentioned above, some common operations which take an `ACCESS EXCLUSIVE` lock are: * retention policies * columnstore compression policies * continuous aggregate refresh policies If you would like to use concurrency nonetheless, turn off all background jobs in the source database before running `pg_dump`, and turn them on once the dump is complete. If the dump procedure takes longer than the continuous aggregate refresh policy's window, you must manually refresh the continuous aggregate in the correct time range. For more information, consult the [refresh policies documentation](/docs/build/continuous-aggregates/refresh-policies). To turn off the jobs: ```sql SELECT public.alter_job(id::integer, scheduled=>false) FROM _timescaledb_config.bgw_job WHERE id >= 1000; ``` To turn on the jobs: ```sql SELECT public.alter_job(id::integer, scheduled=>true) FROM _timescaledb_config.bgw_job WHERE id >= 1000; ``` ## Restoring with concurrency [Section titled “Restoring with concurrency”](#restoring-with-concurrency) If the directory format is used for `pg_dump` and `pg_restore`, concurrency can be employed to speed up the process. Unfortunately, loading the tables in the `timescaledb_catalog` schema concurrently causes errors. Furthermore, the `tsdbadmin` user does not have sufficient privileges to turn off triggers in this schema. To get around this limitation, load this schema serially, and then load the rest of the database concurrently. ```bash # first, serially load JUST the _timescaledb_catalog pg_restore -d "$TARGET" \ --format=directory \ --schema='_timescaledb_catalog' \ --exit-on-error \ --no-tablespaces \ --no-owner \ --no-privileges \ dump # next, concurrently load everything EXCEPT the _timescaledb_catalog pg_restore -d "$TARGET" \ --format=directory \ --jobs=8 \ --exclude-schema='_timescaledb_catalog' \ --exit-on-error \ --disable-triggers \ --no-tablespaces \ --no-owner \ --no-privileges \ dump ``` ## Ownership of background jobs [Section titled “Ownership of background jobs”](#ownership-of-background-jobs) The `_timescaledb_config.bgw_jobs` table is used to manage background jobs. This includes custom jobs, columnstore compression policies, retention policies, and continuous aggregate refresh policies. On Tiger Cloud, this table has a trigger which ensures that no database user can create or modify jobs owned by another database user. This trigger can provide an obstacle for migrations. If the `--no-owner` flag is used with `pg_dump` and `pg_restore`, all objects in the target database are owned by the user that ran `pg_restore`, likely `tsdbadmin`. If all the background jobs in the source database were owned by a user of the same name as the user running the restore (again likely `tsdbadmin`), then loading the `_timescaledb_config.bgw_jobs` table should work. If the background jobs in the source were owned by the `postgres` user, they are automatically changed to be owned by the `tsdbadmin` user. In this case, one just needs to verify that the jobs do not make use of privileges that the `tsdbadmin` user does not possess. If background jobs are owned by one or more users other than the user employed in restoring, then there could be issues. To work around this issue, do not dump this table with `pg_dump`. Provide either `--exclude-table-data='_timescaledb_config.bgw_job'` or `--exclude-table='_timescaledb_config.bgw_job'` to `pg_dump` to skip this table. Then, use `psql` and the `COPY` command to dump and restore this table with modified values for the `owner` column. ```bash # dump the _timescaledb_config.bgw_job table to a csv file replacing the owner # values with tsdbadmin psql -d "$SOURCE" -X -v ON_ERROR_STOP=1 --echo-errors -f - <<'EOF' begin; select string_agg ( case attname when 'owner' then $$'tsdbadmin' as "owner"$$ else format('%I', attname) end , ', ' ) as cols from pg_namespace n inner join pg_class c on (n.nspname = '_timescaledb_config' and n.oid = c.relnamespace and c.relname = 'bgw_job') inner join pg_attribute a on (c.oid = a.attrelid and a.attnum > 0) \gset copy ( select :cols from _timescaledb_config.bgw_job where id >= 1000 ) to stdout with (format csv, header true) \g bgw_job.csv rollback; \q EOF # copy the csv file into the target's _timescaledb_config.bgw_job psql -X -d "$TARGET" -v ON_ERROR_STOP=1 --echo-errors \ -c "\copy _timescaledb_config.bgw_job from 'bgw_job.csv' with (format csv, header match)" ``` Once the table has been loaded and the restore completed, you may then use SQL to adjust the ownership of the jobs and/or the associated stored procedures and functions as you wish. ## Extension availability [Section titled “Extension availability”](#extension-availability) There are a vast number of PostgreSQL extensions available in the wild. Tiger Cloud supports many of the most popular extensions, but not all extensions. Before migrating, check that the extensions you are using are supported on Tiger Cloud. ## TimescaleDB extension in the public schema [Section titled “TimescaleDB extension in the public schema”](#timescaledb-extension-in-the-public-schema) When self-hosting, the TimescaleDB extension may be installed in an arbitrary schema. Tiger Cloud only supports installing the TimescaleDB extension in the `public` schema. How to go about resolving this depends heavily on the particular details of the source schema and the migration approach chosen. ## Tablespaces [Section titled “Tablespaces”](#tablespaces) Tiger Cloud does not support using custom tablespaces. Providing the `--no-tablespaces` flag to `pg_dump` and `pg_restore` when dumping/restoring the schema results in all objects being in the default tablespace as desired. ## Only one database per instance [Section titled “Only one database per instance”](#only-one-database-per-instance) While PostgreSQL clusters can contain many databases, Tiger Cloud services are limited to a single database. When migrating a cluster with multiple databases to Tiger Cloud, one can either migrate each source database to a separate Tiger Cloud service or "merge" source databases to target schemas. ## Superuser privileges [Section titled “Superuser privileges”](#superuser-privileges) The `tsdbadmin` database user is the most powerful available on Tiger Cloud, but it is not a true superuser. Review your application for use of superuser privileged operations and mitigate before migrating. ## Migrate partial continuous aggregates [Section titled “Migrate partial continuous aggregates”](#migrate-partial-continuous-aggregates) In order to improve the performance and compatibility of continuous aggregates, TimescaleDB v2.7 replaces *partial* continuous aggregates with *finalized* continuous aggregates. To test your database for partial continuous aggregates, run the following query: ```sql SELECT exists (SELECT 1 FROM timescaledb_information.continuous_aggregates WHERE NOT finalized); ``` If you have partial continuous aggregates in your database, [migrate them](/docs/migrate/live-migration) from partial to finalized before you migrate your database. If you accidentally migrate partial continuous aggregates across PostgreSQL versions, you see the following error when you query any continuous aggregates: ```plaintext ERROR: insufficient data left in message. ``` # Reference > SQL function reference, analytics functions, configuration, and the tools and APIs for managing Tiger Cloud Complete reference documentation for TimescaleDB, TimescaleDB Toolkit, and Tiger Cloud. Use it to look up exact SQL function signatures, configuration parameters, command syntax, and API endpoints. For step-by-step guidance, see the task-based sections such as Get started, Build, and Deploy. ## TimescaleDB [Section titled “TimescaleDB”](#timescaledb) SQL functions, views, and settings for TimescaleDB, on both Tiger Cloud and self-hosted TimescaleDB. See the [TimescaleDB reference](/docs/reference/timescaledb) for: * [Hypertables and chunks](/docs/reference/timescaledb/hypertables): create and manage hypertables and chunks, including size and statistics, tablespaces, reordering, and query optimization. * [Hypercore](/docs/reference/timescaledb/hypercore): the hypercore columnstore, including columnstore policies, manual conversion, and per-chunk settings. * [Continuous aggregates](/docs/reference/timescaledb/continuous-aggregates): create, modify, and schedule refresh policies for continuous aggregates. * [Hyperfunctions](/docs/reference/timescaledb/hyperfunctions): time-series utilities, distribution analysis, and gapfilling. * [Data retention](/docs/reference/timescaledb/data-retention): drop old data on a schedule. * [Jobs and automation](/docs/reference/timescaledb/jobs-automation): schedule and manage user-defined actions. * [UUIDv7 functions](/docs/reference/timescaledb/uuid-functions): generate and inspect time-ordered UUIDs. * [Informational views](/docs/reference/timescaledb/informational-views): inspect hypertables, chunks, columnstore settings, jobs, and policies. * [Configuration](/docs/reference/timescaledb/configuration): GUCs and PostgreSQL configuration parameters. * [Administration](/docs/reference/timescaledb/administration): administrative functions and commands. ## TimescaleDB Toolkit [Section titled “TimescaleDB Toolkit”](#timescaledb-toolkit) Advanced analytics hyperfunctions for statistical analysis and time-series workflows. See the [TimescaleDB Toolkit reference](/docs/reference/toolkit) for approximate count distinct, statistical and regression analysis, minimum and maximum, financial analysis, percentile approximation, counters and gauges, state tracking, and saturating math. ## Tiger Cloud [Section titled “Tiger Cloud”](#tiger-cloud) Everything you need to manage Tiger Cloud services programmatically. See the [Tiger Cloud reference](/docs/reference/tiger-cloud) for: * [Data tiering](/docs/reference/tiger-cloud/data-tiering): SQL functions that move older chunks to low-cost object storage while keeping them queryable. * [Tiger CLI](/docs/reference/tiger-cloud/tiger-cli): every Tiger CLI command, configuration parameter, and global flag. * [Tiger MCP](/docs/reference/tiger-cloud/tiger-mcp): the Tiger MCP tools and `tiger mcp` commands that let your AI agent manage Tiger Cloud. * [Tiger REST API](/docs/reference/tiger-cloud-rest): create, resize, and configure services over HTTP. # Configuration reference > PostgreSQL and TimescaleDB configuration settings # Configuration reference [Section titled “Configuration reference”](#configuration-reference) Complete reference for configuring Tiger Data. ## PostgreSQL settings [Section titled “PostgreSQL settings”](#postgresql-settings) Standard PostgreSQL configuration parameters. ## TimescaleDB settings [Section titled “TimescaleDB settings”](#timescaledb-settings) TimescaleDB-specific configuration: * Chunk management * Compression settings * Background workers * Memory tuning ## Tiger Cloud settings [Section titled “Tiger Cloud settings”](#tiger-cloud-settings) Tiger Cloud-specific configuration options. ## Query planning & execution [Section titled “Query planning & execution”](#query-planning--execution) TimescaleDB query planning and execution settings. # Tiger Cloud API reference > API reference for Tiger Cloud, including data tiering SQL functions and the Tiger Cloud REST API for managing services programmatically ## [Data tiering](/docs/reference/tiger-cloud/data-tiering) [Move older chunks to low-cost object storage on a schedule you define, while keeping them queryable](/docs/reference/tiger-cloud/data-tiering) ## [Tiger CLI](/docs/reference/tiger-cloud/tiger-cli) [Every Tiger CLI command, configuration parameter, and global flag for managing Tiger Cloud](/docs/reference/tiger-cloud/tiger-cli) ## [Tiger MCP](/docs/reference/tiger-cloud/tiger-mcp) [The Tiger MCP tools and `tiger mcp` commands that let your AI agent manage Tiger Cloud](/docs/reference/tiger-cloud/tiger-mcp) ## [Tiger Cloud REST API](/docs/reference/tiger-cloud-rest) [Create, resize, and configure Tiger Cloud services programmatically](/docs/reference/tiger-cloud-rest) # Tiger Cloud REST API (local preview) > Generated REST reference is disabled in this build ## Generated API reference is off [Section titled “Generated API reference is off”](#generated-api-reference-is-off) This page appears when you run the docs with **`DOCS_LOCAL_WITHOUT_STAINLESS=1`**. The Tiger Cloud REST API reference is generated by the Stainless Docs plugin, which needs a [Stainless API key](https://app.stainless.com/org/default/settings) in `.env` as `STAINLESS_API_KEY`, or a logged-in [Stainless CLI](https://www.stainless.com/docs/getting-started/quickstart-cli) (`stl auth login`). Tips To browse the full REST reference locally, unset `DOCS_LOCAL_WITHOUT_STAINLESS`, add credentials, then run `pnpm dev` again. All other reference sections (TimescaleDB, Toolkit, pgai, and so on) work without Stainless. Only the generated Tiger Cloud REST pages require access to Stainless. # Data tiering overview > Tiger Cloud API reference for data tiering. Includes SQL functions for moving older chunks to object storage on a schedule that you define Time-series data accumulates rapidly and the value of older data typically decreases over time. To control storage costs without losing access to historical data, you can move older chunks from local storage to lower-cost object storage while keeping them fully queryable through the same hypertable. With Tiger Cloud, you can manually tier individual chunks or implement automated policies using these APIs. For more information about working with tiered data, see [Manage storage and tiering](/docs/build/data-management/storage/manage-storage). ## Samples [Section titled “Samples”](#samples) ### Create a tiering policy to move chunks older than 3 weeks to object storage [Section titled “Create a tiering policy to move chunks older than 3 weeks to object storage”](#create-a-tiering-policy-to-move-chunks-older-than-3-weeks-to-object-storage) ```sql SELECT add_tiering_policy('assets_candlestick_daily', INTERVAL '3 weeks'); ``` When you call `add_tiering_policy`, chunks fully older than the supplied interval are frozen and queued for upload to object storage in the background. ### Create a tiering policy with an integer-based time column [Section titled “Create a tiering policy with an integer-based time column”](#create-a-tiering-policy-with-an-integer-based-time-column) ```sql SELECT add_tiering_policy('events', BIGINT '1000000'); ``` ### Manually tier an individual chunk [Section titled “Manually tier an individual chunk”](#manually-tier-an-individual-chunk) ```sql SELECT tier_chunk('_timescaledb_internal._hyper_1_1_chunk'); ``` ### Bring a tiered chunk back into local storage [Section titled “Bring a tiered chunk back into local storage”](#bring-a-tiered-chunk-back-into-local-storage) ```sql CALL untier_chunk('_hyper_1_1_chunk'); ``` ### Remove a tiering policy [Section titled “Remove a tiering policy”](#remove-a-tiering-policy) ```sql SELECT remove_tiering_policy('assets_candlestick_daily'); ``` Removes the existing tiering policy from the `assets_candlestick_daily` hypertable. Chunks already uploaded to object storage remain tiered. ## Available functions [Section titled “Available functions”](#available-functions) * [`add_tiering_policy()`](/docs/reference/tiger-cloud/data-tiering/add_tiering_policy): create a policy to tier chunks older than a given interval * [`remove_tiering_policy()`](/docs/reference/tiger-cloud/data-tiering/remove_tiering_policy): remove the tiering policy from a hypertable or continuous aggregate * [`tier_chunk()`](/docs/reference/tiger-cloud/data-tiering/tier_chunk): manually tier an individual chunk to object storage * [`untier_chunk()`](/docs/reference/tiger-cloud/data-tiering/untier_chunk): move a tiered chunk back to local storage * [`disable_tiering()`](/docs/reference/tiger-cloud/data-tiering/disable_tiering): disable tiering on a hypertable and remove its object storage catalog entries # add_tiering_policy() > Add a policy to automatically tier older chunks to object storage Create a policy that periodically moves chunks older than a given age from local storage to tiered object storage. The policy runs as a Tiger Cloud background job and applies to all chunks of the target hypertable, including the materialized hypertables that back continuous aggregates. Chunks selected for tiering are first frozen and then queued. A separate asynchronous worker uploads them to the configured object store and deletes the local copy when the upload completes. Frozen chunks remain queryable until the upload finishes. By default the policy runs once per hour. To change the interval, call [`alter_job`](/docs/reference/timescaledb/jobs-automation/alter_job) against the returned `job_id`. Note For hypertables and continuous aggregates partitioned on an integer column, an `integer_now_func` must be configured on the underlying hypertable before adding a tiering policy. If `integer_now_func` is not set, `add_tiering_policy` raises an exception. ## Samples [Section titled “Samples”](#samples) * **Tier chunks older than three weeks on the `assets_candlestick_daily` hypertable**: ```sql SELECT add_tiering_policy('assets_candlestick_daily', INTERVAL '3 weeks'); ``` * **Tier chunks older than 1,000,000 ticks on an integer-partitioned hypertable**: ```sql SELECT add_tiering_policy('events', BIGINT '1000000'); ``` * **Add a tiering policy to a continuous aggregate, ignoring the call when one already exists**: ```sql SELECT add_tiering_policy('daily_summary', INTERVAL '90 days', if_not_exists => true); ``` ## Arguments [Section titled “Arguments”](#arguments) The syntax is: ```sql SELECT add_tiering_policy( hypertable = '', move_after = , if_not_exists = true | false ); ``` | Name | Type | Default | Required | Description | | --------------- | ------------ | ------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `hypertable` | `REGCLASS` | - | ✔ | Name of the hypertable or continuous aggregate to create the tiering policy for. | | `move_after` | `ANYELEMENT` | - | ✔ | Chunks fully older than this value are tiered. The type must match the type of the primary partitioning column: `INTERVAL` for `TIMESTAMP`, `TIMESTAMPTZ`, or `DATE`; `BIGINT`, `INTEGER`, or `SMALLINT` for integer-based dimensions. | | `if_not_exists` | `BOOLEAN` | `false` | - | When `true`, emit a notice rather than an error if a tiering policy already exists on the relation, and return the existing `job_id`. | ## Returns [Section titled “Returns”](#returns) | Column | Type | Description | | -------- | --------- | --------------------------------------------------------------- | | `job_id` | `INTEGER` | Tiger Cloud background job ID created to implement this policy. | ## Errors [Section titled “Errors”](#errors) `add_tiering_policy` raises an exception when: * The supplied relation is neither a hypertable nor a continuous aggregate. * A tiering policy already exists on the relation and `if_not_exists` is `false`. * The type of `move_after` does not match the primary dimension type. * The hypertable or its underlying hypertable is integer-partitioned and no `integer_now_func` is set. # disable_tiering() > Disable tiering on a hypertable and remove its object storage catalog entries Disable tiering on a hypertable. The foreign table that represents tiered chunks is dropped and all object storage catalog entries for the hypertable are removed. After the call returns, the hypertable behaves as a regular Tiger Cloud hypertable with no tiered storage attached. `disable_tiering` is only allowed when no tiered data exists for the hypertable. Before calling it you must: 1. Remove the tiering policy with [`remove_tiering_policy`](/docs/reference/tiger-cloud/data-tiering/remove_tiering_policy). 2. Bring every tiered chunk back with [`untier_chunk`](/docs/reference/tiger-cloud/data-tiering/untier_chunk). 3. Clear any chunks still waiting in the tiering queue with [`untier_chunk`](/docs/reference/tiger-cloud/data-tiering/untier_chunk). ## Samples [Section titled “Samples”](#samples) * **Disable tiering on the `my_hypertable_name` hypertable**: ```sql SELECT disable_tiering('my_hypertable_name'); ``` ## Arguments [Section titled “Arguments”](#arguments) The syntax is: ```sql SELECT disable_tiering( hypertable = '' ); ``` | Name | Type | Default | Required | Description | | ------------ | ---------- | ------- | -------- | --------------------------------------------- | | `hypertable` | `REGCLASS` | - | ✔ | Name of the hypertable to disable tiering on. | ## Returns [Section titled “Returns”](#returns) This function returns void. ## Errors [Section titled “Errors”](#errors) `disable_tiering` raises an exception when: * The supplied relation is not a hypertable. * A tiering policy is still attached to the hypertable. Remove it with [`remove_tiering_policy`](/docs/reference/tiger-cloud/data-tiering/remove_tiering_policy) first. * The hypertable has tiered chunks. Untier them with [`untier_chunk`](/docs/reference/tiger-cloud/data-tiering/untier_chunk) first. * The hypertable has chunks queued for tiering. Clear them with [`untier_chunk`](/docs/reference/tiger-cloud/data-tiering/untier_chunk) first. # remove_tiering_policy() > Remove the tiering policy from a hypertable or continuous aggregate Remove the tiering policy from a hypertable or continuous aggregate. The underlying Tiger Cloud background job is deleted, so no further chunks are tiered automatically. Chunks that the policy had already frozen and queued for upload but had not yet shipped to object storage are unfrozen and removed from the tiering queue. Chunks that have already been uploaded to object storage remain tiered; use [`untier_chunk`](/docs/reference/tiger-cloud/data-tiering/untier_chunk) to bring them back. ## Samples [Section titled “Samples”](#samples) * **Remove the tiering policy from the `assets_candlestick_daily` hypertable**: ```sql SELECT remove_tiering_policy('assets_candlestick_daily'); ``` * **Remove the tiering policy from a continuous aggregate, ignoring the call when no policy exists**: ```sql SELECT remove_tiering_policy('daily_summary', if_exists => true); ``` ## Arguments [Section titled “Arguments”](#arguments) The syntax is: ```sql SELECT remove_tiering_policy( hypertable = '', if_exists = true | false ); ``` | Name | Type | Default | Required | Description | | ------------ | ---------- | ------- | -------- | -------------------------------------------------------------------------------------------- | | `hypertable` | `REGCLASS` | - | ✔ | Name of the hypertable or continuous aggregate the tiering policy is attached to. | | `if_exists` | `BOOLEAN` | `false` | - | When `true`, emit a notice rather than an error if no tiering policy exists on the relation. | ## Returns [Section titled “Returns”](#returns) This function returns void. ## Errors [Section titled “Errors”](#errors) `remove_tiering_policy` raises an exception when: * The supplied relation is neither a hypertable nor a continuous aggregate. * No tiering policy exists on the relation and `if_exists` is `false`. # tier_chunk() > Manually tier an individual chunk to object storage Manually tier a single chunk to object storage. Use this to ship a specific chunk to the object store without waiting for the tiering policy to pick it up, or in environments where no tiering policy is configured. The chunk is frozen in place and added to the tiering queue. An asynchronous worker uploads the chunk to the configured object store and removes the local copy when the upload completes. If tiering has not yet been set up for the hypertable, the foreign table backing the tiered chunks is created on first call. ## Samples [Section titled “Samples”](#samples) * **Tier a specific chunk**: ```sql SELECT tier_chunk('_timescaledb_internal._hyper_1_1_chunk'); ``` * **Tier a chunk, ignoring the call when the chunk is already in the tiering queue**: ```sql SELECT tier_chunk('_timescaledb_internal._hyper_1_1_chunk', if_not_tiered => true); ``` ## Arguments [Section titled “Arguments”](#arguments) The syntax is: ```sql SELECT tier_chunk( chunk = '', if_not_tiered = true | false ); ``` | Name | Type | Default | Required | Description | | --------------- | ---------- | ------- | -------- | ------------------------------------------------------------------------------------------------------------------------ | | `chunk` | `REGCLASS` | - | ✔ | Name of the chunk to tier. Must be a real chunk of a hypertable, not the foreign table representing already-tiered data. | | `if_not_tiered` | `BOOLEAN` | `false` | - | When `true`, emit a notice rather than an error if the chunk is already queued for tiering. | ## Returns [Section titled “Returns”](#returns) This function returns void. ## Errors [Section titled “Errors”](#errors) `tier_chunk` raises an exception when: * The supplied relation cannot be found as a chunk of a hypertable. * The chunk is already queued for tiering and `if_not_tiered` is `false`. * The chunk could not be frozen. # untier_chunk() > Move a tiered chunk from object storage back to local storage Move a previously tiered chunk from object storage back to local Tiger Cloud storage. After the call returns, the chunk behaves like any other local chunk and is no longer billed against object storage. If the chunk is still in the tiering queue and has not yet been uploaded to object storage, `untier_chunk` simply unfreezes it and removes it from the queue without any data copy. Otherwise, the chunk's contents are copied from object storage into a temporary table named `osm_tmp` in the chunk's original schema. The copied row count is compared against the count recorded in the catalog; if they differ the procedure aborts and the temporary table is left in place for inspection. Once the copy is verified, the chunk is re-attached to its hypertable and the tiered copy is removed from the object store catalog. `untier_chunk` is a `PROCEDURE` and must be invoked with `CALL`. Note If a check constraint defined on the parent hypertable is violated by rows in the tiered chunk, `untier_chunk` raises an exception. The temporary `osm_tmp` table is preserved so you can fix the offending rows and re-run `untier_chunk` to resume. ## Samples [Section titled “Samples”](#samples) * **Move a tiered chunk back into local storage**: ```sql CALL untier_chunk('_hyper_1_1_chunk'); ``` ## Arguments [Section titled “Arguments”](#arguments) The syntax is: ```sql CALL untier_chunk( osm_chunk_name = '' ); ``` | Name | Type | Default | Required | Description | | ---------------- | ------ | ------- | -------- | ------------------------------------------------------------------------------------------------------------------------------ | | `osm_chunk_name` | `TEXT` | - | ✔ | Name of the tiered chunk, as recorded in the object storage catalog. This is the chunk's table name without the schema prefix. | ## Returns [Section titled “Returns”](#returns) This procedure returns no value. ## Errors [Section titled “Errors”](#errors) `untier_chunk` raises an exception when: * The supplied chunk name does not exist in the object storage catalog. * The number of rows copied from object storage does not match the row count recorded in the catalog. * A check constraint on the parent hypertable is violated by rows in the chunk. * The chunk could not be unfrozen. # Tiger CLI reference > Command reference for Tiger CLI, including every command, configuration parameter, and global flag for managing Tiger Cloud Tiger CLI is a command-line interface that you use to manage Tiger Cloud resources including VPCs, services, read replicas, and related infrastructure. Tiger CLI calls Tiger REST API to communicate with Tiger Cloud. To install CLI, configure authentication, and create your first service, see [Get started with Tiger CLI](/docs/get-started/quickstart/tiger-cli). This page is the reference for every CLI command and configuration parameter. Tiger CLI (which bundles Tiger MCP) is open source. [File an issue in the repo](https://github.com/timescale/tiger-cli/issues/new) to report bugs or request features and help shape the product. Use the following commands to manage Tiger Cloud from the terminal. Every command supports `-h` for inline help, for example `tiger service create -h`. ## Authentication ### `tiger auth login` Create an authenticated connection to your Tiger Cloud account. This opens a browser to authorize. For non-interactive login, pass credentials as flags. ```bash tiger auth login ``` * `--public-key`: public key for non-interactive login. * `--secret-key`: secret key for non-interactive login. You can also set the `TIGER_PUBLIC_KEY` and `TIGER_SECRET_KEY` environment variables; the project is auto-detected from your credentials. See [Authentication parameters](#authentication-parameters). ### `tiger auth logout` Remove the credentials used to connect to Tiger Cloud. ```bash tiger auth logout ``` ### `tiger auth status` Show your current authentication status and project ID. ```bash tiger auth status ``` * `--output, -o`: output format: `json`, `yaml`, or `table`. ## Version ### `tiger version` Show the installed Tiger CLI version. ```bash tiger version ``` * `--check`: force a check for updates, regardless of the last check time. * `--output, -o`: output format: `table`, `json`, `yaml`, or `bare`. ## Configuration ### `tiger config show` Show the current configuration. ```bash tiger config show ``` * `--output, -o`: output format: `json`, `yaml`, or `table`. * `--no-defaults`: do not show default values for unset fields. * `--with-env`: apply environment variable overrides. ### `tiger config set` Set a configuration value. See [Configuration parameters](#configuration-parameters) for the available keys. ```bash tiger config set # for example tiger config set read_only true ``` ### `tiger config unset` Clear a configuration value. ```bash tiger config unset ``` ### `tiger config reset` Reset the configuration to defaults. This also logs you out of the current Tiger Cloud project. ```bash tiger config reset ``` ## Services ### `tiger service create` Create a new service in the current project. ```bash tiger service create --name analytics --region us-east-1 ``` * `--name`: service name (auto-generated if omitted). * `--addons`: addons to enable: `time-series`, `ai`. Set to `none` for vanilla PostgreSQL. * `--region`: region code. [Free services](/docs/get-started/quickstart/create-service#what-is-a-tiger-cloud-service) (shared CPU/memory) must use `us-east-1`. * `--cpu`: CPU allocation in millicores. Set to `shared` for a free service. * `--memory`: memory allocation in gigabytes. Set to `shared` for a free service. * `--replicas`: number of high-availability replicas. * `--environment`: environment tag: `DEV` or `PROD` (default: `DEV`). * `--no-wait`: return without waiting for the operation to complete. * `--wait-timeout`: wait timeout (for example, `30m`, `1h30m`, `90s`). * `--no-set-default`: do not set this service as the default. * `--with-password`: include the password in the output. * `--output, -o`: output format: `json`, `yaml`, `env`, or `table`. Allowed compute configurations: `shared`/`shared` (only in `us-east-1`), 0.5 CPU/2 GB, 1/4, 2/8, 4/16, 8/32, 16/64, 32/128. Specify `--cpu` and `--memory` together, or set one and the other is configured automatically. ### `tiger service list` List the services in the current project. ```bash tiger service list ``` * `--output, -o`: output format: `json`, `yaml`, or `table`. ### `tiger service get` Show detailed information about a service. Aliases: `describe`, `show`. ```bash tiger service get ``` * `--with-password`: include the password in the output. * `--output, -o`: output format: `json`, `yaml`, `env`, or `table`. ### `tiger service fork` Fork an existing service into a new, independent copy. Choose exactly one timing option. ```bash tiger service fork --now ``` * `--now`: fork at the current database state. * `--last-snapshot`: fork at the last snapshot (faster). * `--to-timestamp`: fork at a point in time (RFC3339). * `--cpu`, `--memory`: compute for the fork (inherits from the source if omitted). See [`service create`](#tiger-service-create) for allowed configurations. * `--name`: fork name (default: `{source-service-name}-fork`). * `--environment`: environment tag: `DEV` or `PROD` (default: `DEV`). * `--no-wait`, `--wait-timeout`: wait behavior (default timeout: `30m`). * `--no-set-default`, `--with-password`, `--output, -o`: default-service and output options. ### `tiger service resize` Change a service's CPU and memory. The service may be briefly unavailable during the resize. ```bash tiger service resize --cpu 4 --memory 16 ``` * `--cpu`, `--memory`: new allocation. See [`service create`](#tiger-service-create) for allowed configurations. * `--no-wait`, `--wait-timeout`: wait behavior (default timeout: `10m`). ### `tiger service start` Start an inactive service. ```bash tiger service start ``` * `--no-wait`, `--wait-timeout`: wait behavior (default timeout: `10m`). ### `tiger service stop` Stop an active service. After stopping, the service no longer accepts connections. ```bash tiger service stop ``` * `--no-wait`, `--wait-timeout`: wait behavior (default timeout: `10m`). ### `tiger service update-password` Update the master password for a service. ```bash tiger service update-password ``` * `--new-password`: new password for the `tsdbadmin` user. * `--auto-generate`: auto-generate a secure password (mutually exclusive with `--new-password`). ### `tiger service delete` Delete a service. This is irreversible and prompts for confirmation before proceeding. ```bash tiger service delete ``` * `--confirm`: skip the confirmation prompt. AI agents must confirm with the user first. * `--no-wait`, `--wait-timeout`: wait behavior (default timeout: `30m`). ### `tiger service logs` View the logs for a service. Alias: `log`. ```bash tiger service logs ``` * `--tail`: number of log lines to show (default: `100`). * `--since`, `--until`: fetch logs within a time range (RFC3339, for example `2024-01-15T09:00:00Z`). * `--node`: specific node to fetch logs from (for services with HA replicas; `0` is valid). * `--output, -o`: output format: `text`, `json`, or `yaml`. ## Database ### `tiger db connect` Connect to a service with `psql`. Pass extra `psql` flags after `--`, for example `tiger db connect -- --single-transaction`. Alias: `psql`. ```bash tiger db connect ``` * `--pooled`: use connection pooling (default: `false`). * `--role`: database role (default: `tsdbadmin`). * `--read-only`: open the session in Tiger Cloud's immutable read-only mode, so writes and DDL are rejected by the server. The [`read_only` config option](#configuration-parameters) or `TIGER_READ_ONLY=true` forces the same behavior. * `--no-replica-prompt`: do not prompt to connect to a read replica. ### `tiger db connection-string` Print the connection string for a service. ```bash tiger db connection-string ``` * `--pooled`: use connection pooling (default: `false`). * `--role`: database role (default: `tsdbadmin`). * `--with-password`: include the password (default: `false`, less secure). * `--read-only`: emit a read-only connection string. ### `tiger db create role` Create a database role. ```bash tiger db create role --name app_role ``` * `--name` (required): the role to create. * `--read-only`: enforce permanent read-only for the role using `tsdb_admin.read_only_role`. * `--from`: inherit grants from one or more roles, for example `--from app_role,readonly_role`. * `--statement-timeout`: statement timeout for the role, for example `30s`, `5m`. * `--password`: role password (falls back to `TIGER_NEW_PASSWORD`, otherwise auto-generated). * `--output, -o`: output format: `json`, `yaml`, or `table`. ### `tiger db schema` Display the schema of a service database as readable text. ```bash tiger db schema ``` * `--schema`: restrict output to a single schema. * `--definitions`: include full object definitions (view `SELECT`s, function and procedure bodies). * `--comments`: include object comments (`COMMENT ON` text). * `--internal`: include system schemas (`pg_*`, `information_schema`, TimescaleDB internals) and extension-owned objects. * `--pooled`: use connection pooling (default: `false`). * `--role`: database role (default: `tsdbadmin`). ### `tiger db save-password` Save the password for a service to the keychain. ```bash tiger db save-password ``` * `--role`: database role (default: `tsdbadmin`). * `--password`: password value (or use `TIGER_NEW_PASSWORD`). ### `tiger db test-connection` Test connectivity to a service. ```bash tiger db test-connection ``` * `--timeout, -t`: connection timeout (default: `3s`, `0` for none). * `--pooled`: use connection pooling (default: `false`). * `--role`: database role (default: `tsdbadmin`). ## MCP ### `tiger mcp install` Install and configure Tiger MCP for an AI agent. Supported clients: `claude-code`, `codex`, `cursor`, `gemini`, `vscode`, `windsurf`, `antigravity`, `kiro-cli`. If no client is given, you are prompted to choose. ```bash tiger mcp install claude-code ``` * `--no-backup`: do not back up the client's existing config before writing. * `--config-path`: path to the client config file to update. ### `tiger mcp list` List the available Tiger MCP tools, prompts, and resources. ```bash tiger mcp list ``` * `--output, -o`: output format: `json`, `yaml`, or `table`. ### `tiger mcp get` Show detailed information about a Tiger MCP tool, prompt, or resource, including skills. Aliases: `describe`, `show`. ```bash tiger mcp get service_create tiger mcp get setup-timescaledb-hypertables ``` * `--output, -o`: output format: `json`, `yaml`, or `table`. ### `tiger mcp start` Start Tiger MCP. `tiger mcp start` is the same as `tiger mcp start stdio`. ```bash tiger mcp start ``` * `stdio`: stdio transport (default). * `http`: HTTP transport, with `--port` (default: `8080`) and `--host` (default: `localhost`). ## Configuration parameters By default, Tiger CLI stores your configuration in `~/.config/tiger/config.yaml`. Change the config directory with the `--config-dir` flag or the `TIGER_CONFIG_DIR` environment variable. ### Configuration options Set these with `tiger config set `: | Flag | Default | Description | | ------------------ | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `analytics` | `true` | Enable or disable usage analytics | | `color` | `true` | Enable or disable colored output | | `debug` | `false` | Enable or disable debug logging | | `docs_mcp` | `true` | Enable or disable the Tiger Data documentation MCP proxy | | `mcp_max_rows` | `100` | Set the maximum number of rows returned by the `db_execute_query` Tiger MCP tool | | `output` | `table` | Set the output format to `json`, `yaml`, or `table` | | `password_storage` | `keyring` | Set the password storage method. Options are `keyring`, `pgpass`, or `none` | | `read_only` | `false` | Enable or disable read-only mode. When `true`, mutating CLI commands and Tiger MCP tools (`service create`, `fork`, `start`, `stop`, `resize`, `update-password`, and `delete`) are refused, and the database sessions opened by `db connect`, `db connection-string`, and `db_execute_query` run in Tiger Cloud's immutable read-only mode, so writes and DDL are rejected by the server. Read commands and tools are unaffected | | `service_id` | - | Set the default service to manage | | `version_check` | `true` | Enable or disable checking for a new version of Tiger CLI on startup | You can also set these configuration options as environment variables. Environment variables: * Take precedence over configuration parameters values. * Are in upper case and use the `TIGER_` prefix. For example, `TIGER_ANALYTICS` ### Global flags These flags are available on all commands and take precedence over both environment variables and configuration file values: | Flag | Default | Description | | --------------------- | ----------------- | --------------------------------------------------------------------------- | | `--analytics` | `true` | Enable or disable usage analytics | | `--color` | `true` | Enable or disable colored output | | `--config-dir` | `~/.config/tiger` | Set the directory that holds `config.yaml` | | `--debug` | `false` | Enable or disable debug logging | | `--help`, `-h` | - | Print help about the current command. For example, `tiger service --help` | | `--password-storage` | `keyring` | Set the password storage method. Options are `keyring`, `pgpass`, or `none` | | `--service-id` | - | Set the default service to manage | | `--skip-update-check` | `false` | Skip checking if a new version of Tiger CLI is available | ### Authentication parameters You can authenticate using client credentials instead of interactive browser login. Client credentials are a public key and secret key pair that you create in the Tiger Cloud Console. [Learn how to create client credentials](/docs/integrate/find-connection-details#create-client-credentials). Once you have your credentials, either set them as environment variables, then log in: ```shell TIGER_PUBLIC_KEY= TIGER_SECRET_KEY= \ tiger auth login ``` Or pass them to the `login` command: ```shell tiger auth login --public-key= --secret-key= ``` The project is auto-detected from your credentials. # Tiger MCP reference > Tool reference for Tiger MCP, the Model Context Protocol server bundled with Tiger CLI Tiger MCP gives your AI agent access to Tiger Cloud so you can manage services and query your data in natural language. It is built into the Tiger CLI binary. To install and configure Tiger MCP for your AI agent, see [Integrate Tiger Cloud with your AI agent](/docs/get-started/quickstart/mcp-cli). This page is the reference for the Tiger MCP tools your agent can call. ## Tiger MCP tools [Section titled “Tiger MCP tools”](#tiger-mcp-tools) Tiger MCP exposes the following tools to your AI agent. You do not call these directly; you describe what you want and the agent selects the tool. Definitions can change Parameter names and required or optional fields can change with new Tiger MCP versions. Run `tiger mcp get ` for the current definition of any tool. ## Service tools Note The following tools are disabled when using read-only mode: `service_create`, `service_fork`, `service_start`, `service_stop`, `service_resize`, `service_update_password`. ### `service_list` List the services in the current project. No parameters. ### `service_get` Show detailed information about one service. * `service_id` (required): the target service. * `with_password`: include the password. Only set this if you explicitly ask for the password. ### `service_create` Create a new service. Addons: `time-series` (TimescaleDB) and `ai` (AI/vector). * `name`, `addons`, `region`, `cpu_memory`, `replicas`, `wait`, `set_default`, `with_password`. Creates billable resources `service_create` provisions billable infrastructure. ### `service_fork` Fork a service into an independent copy. * `service_id` (required): the source service. * `fork_strategy` (required): `NOW`, `LAST_SNAPSHOT`, or `PITR`. * `target_time`: point in time, for `PITR`. * `name`, `cpu_memory`, `wait`, `set_default`, `with_password`. Creates billable resources `service_fork` provisions billable infrastructure. ### `service_resize` Change a service's CPU and memory. * `service_id` (required), `cpu_memory` (required), `wait`. Affects billing `service_resize` affects billing, and the service may be briefly unavailable. ### `service_start` Start a stopped service. * `service_id` (required), `wait`. ### `service_stop` Stop a running service. * `service_id` (required), `wait`. ### `service_update_password` Update the `tsdbadmin` password. May disconnect existing sessions. * `service_id` (required), `password` (required). ### `service_logs` Fetch service logs. * `service_id` (required), `tail`, `since`, `until`, `node`. ## Database tools ### `db_execute_query` Run a single SQL statement against a service. * `service_id` (required), `query` (required), `parameters`, `timeout_seconds`, `role`, `pooled`. Can run destructive SQL `db_execute_query` can run `INSERT`, `UPDATE`, `DELETE`, and DDL. Multi-statement queries (semicolon-separated) are supported when no `parameters` are provided. In read-only mode, writes and DDL are rejected by the server. ### `db_schema` Return the schema of a service database as readable text (tables, views, materialized views, and more). * `service_id` (required), `schema`, `definitions`, `comments`, `internal`, `role`, `pooled`. ## Documentation and skills tools ### `search_docs` Search Tiger Data documentation with hybrid semantic (vector) and keyword search. * `source` (required), `query` (required), `limit` (required), `semanticWeight` (required): `0` for keyword-only, `1` for semantic-only, or a value in between to blend the two. ### `view_skill` Retrieve a built-in skill for TimescaleDB operations and best practices (for example, schema design, hypertable setup, and migration planning). * `skill_name` (required), `path` (required). ## Manage the Tiger MCP server [Section titled “Manage the Tiger MCP server”](#manage-the-tiger-mcp-server) To install, list, inspect, and start Tiger MCP, use the `tiger mcp` commands, documented in the [Tiger CLI reference](/docs/reference/tiger-cloud/tiger-cli#mcp). Global flags that apply when running Tiger MCP are in the [configuration parameters](/docs/reference/tiger-cloud/tiger-cli#configuration-parameters). Tiger MCP ships inside the open-source Tiger CLI. [File an issue in the repo](https://github.com/timescale/tiger-cli/issues/new) to report bugs or request features and help shape the product. # TimescaleDB API reference > Complete API reference for TimescaleDB functions, SQL commands, and time-series data management ## [Hypertables and chunks](/docs/reference/timescaledb/hypertables) [Create and manage hypertables, chunks, dimensions, and indexes for time-series data](/docs/reference/timescaledb/hypertables) ## [Hypercore](/docs/reference/timescaledb/hypercore) [Manage columnstore tables for analytical workloads with hybrid row and column storage](/docs/reference/timescaledb/hypercore) ## [Continuous aggregates](/docs/reference/timescaledb/continuous-aggregates) [Create and manage automatically refreshed materialized views for time-series data](/docs/reference/timescaledb/continuous-aggregates) ## [Hyperfunctions](/docs/reference/timescaledb/hyperfunctions) [Analyze time-series data with time bucketing, gapfilling, and distribution analysis](/docs/reference/timescaledb/hyperfunctions) ## [Data retention](/docs/reference/timescaledb/data-retention) [Automatically drop old data with retention policies](/docs/reference/timescaledb/data-retention) ## [Jobs and automation](/docs/reference/timescaledb/jobs-automation) [Schedule and manage automated background jobs](/docs/reference/timescaledb/jobs-automation) ## [UUIDv7 functions](/docs/reference/timescaledb/uuid-functions) [Generate and manipulate UUIDv7 identifiers with built-in timestamp ordering](/docs/reference/timescaledb/uuid-functions) ## [Informational views](/docs/reference/timescaledb/informational-views) [Query metadata about hypertables, chunks, jobs, and system configuration](/docs/reference/timescaledb/informational-views) ## [Configuration](/docs/reference/timescaledb/configuration) [Configure TimescaleDB settings and parameters](/docs/reference/timescaledb/configuration) ## [Administration](/docs/reference/timescaledb/administration) [Administrative functions for backup, restore, and system management](/docs/reference/timescaledb/administration) TimescaleDB provides SQL commands and functions for managing time-series data efficiently. For additional hyperfunctionss for advanced time-series analysis, see the [TimescaleDB Toolkit API reference](/docs/reference/toolkit). # Administrative functions > Administration functions help you manage your service before and after recovery, as well as keeping track of your data Administrative APIs help you prepare a database before and after a restore event. They also help you keep track of your TimescaleDB setup data. ## Samples [Section titled “Samples”](#samples) ### Prepare database for restore [Section titled “Prepare database for restore”](#prepare-database-for-restore) Before restoring a database backup, prepare the database: ```sql SELECT timescaledb_pre_restore(); ``` Then perform your restore operation: ```bash psql -d your_database < backup.sql ``` ### Complete restore operation [Section titled “Complete restore operation”](#complete-restore-operation) After restoring a database backup, complete the restore: ```sql SELECT timescaledb_post_restore(); ``` ### View telemetry report [Section titled “View telemetry report”](#view-telemetry-report) Check what telemetry data is being collected and sent: ```sql SELECT get_telemetry_report(); ``` ### Full backup and restore workflow [Section titled “Full backup and restore workflow”](#full-backup-and-restore-workflow) Complete workflow for backing up and restoring a TimescaleDB database: ```bash # On source database psql -d source_db -c "SELECT timescaledb_pre_restore();" pg_dump -Fc -f backup.dump source_db # On target database createdb target_db psql -d target_db -c "CREATE EXTENSION IF NOT EXISTS timescaledb;" psql -d target_db -c "SELECT timescaledb_pre_restore();" pg_restore -d target_db backup.dump psql -d target_db -c "SELECT timescaledb_post_restore();" ``` ### Check TimescaleDB version and installation [Section titled “Check TimescaleDB version and installation”](#check-timescaledb-version-and-installation) Verify the TimescaleDB extension is installed and check version: ```sql SELECT default_version, installed_version FROM pg_available_extensions WHERE name = 'timescaledb'; SELECT extversion FROM pg_extension WHERE extname = 'timescaledb'; ``` ## Dump TimescaleDB meta data [Section titled “Dump TimescaleDB meta data”](#dump-timescaledb-meta-data) To help when asking for support and reporting bugs, TimescaleDB includes an SQL dump script. It outputs metadata from the internal TimescaleDB tables, along with version information. This script is available in the source distribution in `scripts/`. To use it, run: ```bash psql [your connect flags] -d your_timescale_db < dump_meta_data.sql > dumpfile.txt ``` Inspect `dumpfile.txt` before sending it together with a bug report or support question. ## Available functions [Section titled “Available functions”](#available-functions) * [`get_telemetry_report()`](/docs/reference/timescaledb/administration/get_telemetry_report): view the background telemetry string sent to Timescale * [`timescaledb_post_restore()`](/docs/reference/timescaledb/administration/timescaledb_post_restore): perform required operations after finishing a database restore * [`timescaledb_pre_restore()`](/docs/reference/timescaledb/administration/timescaledb_pre_restore): prepare the database for a restore operation # get_telemetry_report() > View the background telemetry string sent to Timescale Since [0.6.0](https://github.com/timescale/timescaledb/releases/tag/0.6.0) Returns the background [telemetry](/docs/deploy/self-hosted/configuration/telemetry) string sent to Timescale. If telemetry is turned off, it sends the string that would be sent if telemetry were enabled. ## Samples [Section titled “Samples”](#samples) View the telemetry report: ```sql SELECT get_telemetry_report(); ``` ## Returns [Section titled “Returns”](#returns) | Column | Type | Description | | ----------------- | ----- | ---------------------------------------------------------- | | telemetry\_report | JSONB | The telemetry report that is or would be sent to Timescale | # timescaledb_post_restore() > Perform required operations after finishing a database restore Since [0.9.0](https://github.com/timescale/timescaledb/releases/tag/0.9.0) Perform the required operations after you have finished restoring the database using `pg_restore`. Specifically, this resets the `timescaledb.restoring` GUC and restarts any background workers. For more information, see [Migrate using pg\_dump and pg\_restore](/docs/deploy/self-hosted/migration/entire-database). ## Samples [Section titled “Samples”](#samples) Prepare the database for normal use after a restore: ```sql SELECT timescaledb_post_restore(); ``` ## Returns [Section titled “Returns”](#returns) | Column | Type | Description | | ------- | ------- | -------------------------------------------- | | success | BOOLEAN | TRUE if the operation completed successfully | # timescaledb_pre_restore() > Prepare the database for a restore operation Since [0.9.0](https://github.com/timescale/timescaledb/releases/tag/0.9.0) Perform the required operations so that you can restore the database using `pg_restore`. Specifically, this sets the `timescaledb.restoring` GUC to `on` and stops any background workers which could have been performing tasks. The background workers are stopped until the [`timescaledb_post_restore()`](/docs/reference/timescaledb/administration/timescaledb_post_restore) function is run, after the restore operation is complete. For more information, see [Migrate using pg\_dump and pg\_restore](/docs/deploy/self-hosted/migration/entire-database). Important After using `timescaledb_pre_restore()`, you need to run [`timescaledb_post_restore()`](/docs/reference/timescaledb/administration/timescaledb_post_restore) before you can use the database normally. ## Samples [Section titled “Samples”](#samples) Prepare to restore the database: ```sql SELECT timescaledb_pre_restore(); ``` ## Returns [Section titled “Returns”](#returns) | Column | Type | Description | | ------- | ------- | -------------------------------------------- | | success | BOOLEAN | TRUE if the operation completed successfully | # Service configuration > Use the default PostgreSQL server configuration settings for your Tiger Cloud service, or customize them as needed TimescaleDB uses the default PostgreSQL server configuration settings. You can optimize your service configuration using the following TimescaleDB and Grand Unified Configuration (GUC) parameters. ## Samples [Section titled “Samples”](#samples) ### View current TimescaleDB settings [Section titled “View current TimescaleDB settings”](#view-current-timescaledb-settings) Check all TimescaleDB-specific configuration settings: ```sql SELECT name, setting, unit, short_desc FROM pg_settings WHERE name LIKE 'timescaledb%' ORDER BY name; ``` ### Enable chunkwise aggregation [Section titled “Enable chunkwise aggregation”](#enable-chunkwise-aggregation) Enable query optimization for aggregations: ```sql ALTER DATABASE your_database SET timescaledb.enable_chunkwise_aggregation = 'on'; ``` Or set it for your current session: ```sql SET timescaledb.enable_chunkwise_aggregation = on; ``` ### Enable vectorized aggregation [Section titled “Enable vectorized aggregation”](#enable-vectorized-aggregation) Enable vectorized optimizations for compressed chunks: ```sql ALTER DATABASE your_database SET timescaledb.vectorized_aggregation = 'on'; ``` ### Configure continuous aggregate refresh optimization [Section titled “Configure continuous aggregate refresh optimization”](#configure-continuous-aggregate-refresh-optimization) Enable merge optimization for continuous aggregate refreshes: ```sql SET timescaledb.enable_merge_on_cagg_refresh = on; ``` ### Disable telemetry [Section titled “Disable telemetry”](#disable-telemetry) Turn off telemetry reporting: ```sql ALTER SYSTEM SET timescaledb.telemetry_level = 'off'; SELECT pg_reload_conf(); ``` ### Check TimescaleDB version and license [Section titled “Check TimescaleDB version and license”](#check-timescaledb-version-and-license) View the current TimescaleDB version and license: ```sql SELECT extname, extversion FROM pg_extension WHERE extname = 'timescaledb'; SHOW timescaledb.license; ``` To interpret the license value and see which features each edition includes, see [Compare TimescaleDB editions](/docs/get-started/choose-your-path/timescaledb-editions#check-your-installed-edition). ## Available configuration options [Section titled “Available configuration options”](#available-configuration-options) * [TimescaleDB configuration and tuning](/docs/reference/timescaledb/configuration/tiger-postgres): configure the TimescaleDB settings related to policies, query planning and execution, and administration * [Grand Unified Configuration (GUC) parameters](/docs/reference/timescaledb/configuration/gucs): optimize the behavior of TimescaleDB using Grand Unified Configuration (GUC) parameters # Grand Unified Configuration (GUC) parameters > Optimize the behavior of TimescaleDB using Grand Unified Configuration (GUC) parameters You use the following Grand Unified Configuration (GUC) parameters to optimize the behavior of your Tiger Cloud service. The namespace of each GUC is `timescaledb`. To set a GUC you specify `.`. For example: ```sql SET timescaledb.enable_tiered_reads = true; ``` | Name | Type | Default | Description | | --------------------------------------------- | --------- | ---------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `auto_sparse_indexes` | `BOOLEAN` | `true` | The hypertable columns that are used as index keys will have suitable sparse indexes when compressed. Must be set at the moment of chunk compression, e.g. when the `compress_chunk()` is called. | | `bgw_log_level` | `ENUM` | `WARNING` | Log level for the scheduler and workers of the background worker subsystem. Requires configuration reload to change. | | `cagg_rewrites_debug_info` | `BOOLEAN` | `false` | Print debug info about whether queries can be rewritten with Caggs | | `compress_truncate_behaviour` | `ENUM` | `COMPRESS_TRUNCATE_ONLY` | Defines how truncate behaves at the end of compression. 'truncate\_only' forces truncation. 'truncate\_disabled' deletes rows instead of truncate. 'truncate\_or\_delete' allows falling back to deletion. | | `compression_batch_size_limit` | `INTEGER` | `TARGET_COMPRESSED_BATCH_SIZE` | Setting this option to a number between 1 and 32767 will force compression to limit the size of compressed batches to that amount of uncompressed tuples. The setting influences only the compression process itself. The value of the setting is taken from the context of the session where the compression is performed. It is not persisted in any way. min: `1`, max: `GLOBAL_MAX_ROWS_PER_COMPRESSION` | | `compression_orderby_default_function` | `STRING` | `"_timescaledb_functions.get_orderby_defaults"` | Function to use for calculating default order\_by setting for compression | | `compression_segmentby_default_function` | `STRING` | `"_timescaledb_functions.get_segmentby_defaults"` | Function to use for calculating default segment\_by setting for compression | | `current_timestamp_mock` | `STRING` | `NULL` | this is for debugging purposes | | `debug_bgw_scheduler_exit_status` | `INTEGER` | `0` | this is for debugging purposes min: `0`, max: `255` | | `debug_compression_path_info` | `BOOLEAN` | `false` | this is for debugging/information purposes | | `debug_have_int128` | `BOOLEAN` | `#ifdef HAVE_INT128 true` | this is for debugging purposes | | `debug_require_batch_sorted_merge` | `ENUM` | `DRO_Allow` | this is for debugging purposes | | `debug_require_vector_agg` | `ENUM` | `DRO_Allow` | this is for debugging purposes | | `debug_require_vector_qual` | `ENUM` | `DRO_Allow` | this is for debugging purposes, to let us check if the vectorized quals are used or not. EXPLAIN differs after PG15 for custom nodes, and using the test templates is a pain | | `debug_skip_scan_info` | `BOOLEAN` | `false` | Print debug info about SkipScan distinct columns | | `debug_toast_tuple_target` | `INTEGER` | `/* bootValue = */ 128` | this is for debugging purposes min: `/* minValue = */ 1`, max: `/* maxValue = */ 65535` | | `default_chunk_time_interval` | `STRING` | `NULL` | Chunk time interval to use for a new hypertable, unless a specific chunk time interval is set on the hypertable. The default chunk interval is only used for hypertables with a compatible time type, e.g., timestamp, date, and UUID (v7). Hypertables using an integer partitioning column have hard-coded defaults.Expert-level setting. These parameters are optimized for internal workflows; incorrect configurations can negatively impact query performance and system efficiency. | | `direct_compress_copy_tuple_sort_limit` | `INTEGER` | `100000` | This is mainly used to keep the memory footprint down for operations like importing large amounts of data in single transaction. Setting this to 0 would make it unlimited. min: `0`, max: `2147483647` | | `direct_compress_insert_tuple_sort_limit` | `INTEGER` | `30000` | This is mainly used to keep the memory footprint down for operations like importing large amounts of data in single transaction. Setting this to 0 would make it unlimited. min: `0`, max: `2147483647` | | `direct_compress_segmentby_batch_size_limit` | `INTEGER` | `500` | Each distinct value of a candidate segmentby column must have at least this many rows for the column to be selected. min: `1`, max: `TARGET_COMPRESSED_BATCH_SIZE` | | `direct_compress_segmentby_min_rows` | `INTEGER` | `5000` | During direct compress, automatic segmentby selection only runs when the number of buffered tuples meets this threshold. min: `0`, max: `2147483647` | | `enable_bool_compression` | `BOOLEAN` | `true` | Enable bool compression | | `enable_bulk_decompression` | `BOOLEAN` | `true` | Increases throughput of decompression, but might increase query memory usage | | `enable_cagg_reorder_groupby` | `BOOLEAN` | `true` | Enable group by clause reordering for continuous aggregates | | `enable_cagg_rewrites` | `BOOLEAN` | `false` | Enable rewriting queries with eligible continuous aggregates | | `enable_cagg_sort_pushdown` | `BOOLEAN` | `true` | Enable pushdown of ORDER BY clause for continuous aggregates | | `enable_cagg_watermark_constify` | `BOOLEAN` | `true` | Enable constifying cagg watermark for real-time caggs | | `enable_cagg_window_functions` | `BOOLEAN` | `false` | Allow window functions in continuous aggregate views | | `enable_chunk_append` | `BOOLEAN` | `true` | Enable using chunk append node | | `enable_chunk_auto_publication` | `BOOLEAN` | `false` | Enable automatically adding newly created chunks to the publication of their hypertable | | `enable_chunk_skipping` | `BOOLEAN` | `false` | Enable using chunk column stats to filter chunks based on column filters | | `enable_chunkwise_aggregation` | `BOOLEAN` | `true` | Enable the pushdown of aggregations to the chunk level | | `enable_columnar_scan_filter_pushdown` | `BOOLEAN` | `true` | Enable pushing down the filters into the compressed scan part of the columnar scan | | `enable_columnarindexscan` | `BOOLEAN` | `true` | Enable experimental support for returning results directly from compression metadata without decompression | | `enable_columnarscan` | `BOOLEAN` | `true` | Transparently decompress columnar data using ColumnarScan custom node. Disabling columnar scan will ignore data stored in columnar format in queries. | | `enable_composite_bloom_indexes` | `BOOLEAN` | `true` | This composite index speeds up the equality queries on compressed columns, and can be disabled when not desired. | | `enable_compressed_direct_batch_delete` | `BOOLEAN` | `true` | Enable direct batch deletion in compressed chunks | | `enable_compressed_merge` | `BOOLEAN` | `false` | Enable MERGE support for compressed hypertables. This is only available in debug builds and will currently do full decompression | | `enable_compressed_skipscan` | `BOOLEAN` | `true` | Enable SkipScan for distinct inputs over compressed chunks | | `enable_compression_indexscan` | `BOOLEAN` | `false` | Enable indexscan during compression, if matching index is found | | `enable_compression_ratio_warnings` | `BOOLEAN` | `true` | Enable warnings for poor compression ratio | | `enable_compression_wal_markers` | `BOOLEAN` | `true` | Enable the generation of markers in the WAL stream which mark the start and end of compression operations | | `enable_compressor_batch_limit` | `BOOLEAN` | `false` | Enable compressor batch limit for compressors which can go over the allocation limit (1 GB). This feature will limit those compressors by reducing the size of the batch and thus avoid hitting the limit. | | `enable_constraint_aware_append` | `BOOLEAN` | `true` | Enable constraint exclusion at execution time | | `enable_constraint_exclusion` | `BOOLEAN` | `true` | Enable planner constraint exclusion | | `enable_decompression_sorted_merge` | `BOOLEAN` | `true` | Enable the merge of compressed batches to preserve the compression order by | | `enable_delete_after_compression` | `BOOLEAN` | `false` | Delete all rows after compression instead of truncate | | `enable_deprecation_warnings` | `BOOLEAN` | `true` | Enable warnings when using deprecated functionality | | `enable_direct_compress_auto_segmentby` | `BOOLEAN` | `true` | When enabled, automatically analyzes buffered tuples to pick an optimal segmentby column if none is configured. | | `enable_direct_compress_copy` | `BOOLEAN` | `false` | Enable experimental support for direct compression during COPY | | `enable_direct_compress_copy_client_sorted` | `BOOLEAN` | `false` | Correct handling of data sorting by the user is required for this option. | | `enable_direct_compress_copy_sort_batches` | `BOOLEAN` | `true` | Enable batch sorting during direct compress COPY | | `enable_direct_compress_insert` | `BOOLEAN` | `false` | Enable experimental support for direct compression during INSERT | | `enable_direct_compress_insert_client_sorted` | `BOOLEAN` | `false` | Correct handling of data sorting by the user is required for this option. | | `enable_direct_compress_insert_sort_batches` | `BOOLEAN` | `true` | Enable batch sorting during direct compress INSERT | | `enable_direct_compress_on_cagg_refresh` | `BOOLEAN` | `false` | Enable experimental support for direct compression during Continuous Aggregate refresh | | `enable_dml_bloom_filter` | `BOOLEAN` | `true` | When enabled, bloom filters are used to skip compressed batches that definitely do not contain matching rows during DELETE and UPDATE operations, reducing decompression overhead. | | `enable_dml_decompression` | `BOOLEAN` | `true` | Enable DML decompression when modifying compressed hypertable | | `enable_dml_decompression_tuple_filtering` | `BOOLEAN` | `true` | Recheck tuples during DML decompression to only decompress batches with matching tuples | | `enable_event_triggers` | `BOOLEAN` | `false` | Enable event triggers for chunks creation | | `enable_exclusive_locking_recompression` | `BOOLEAN` | `false` | Enable getting exclusive lock on chunk during segmentwise recompression | | `enable_foreign_key_propagation` | `BOOLEAN` | `true` | Adjust foreign key lookup queries to target whole hypertable | | `enable_in_memory_recompression` | `BOOLEAN` | `true` | Enable in-memory recompression | | `enable_job_execution_logging` | `BOOLEAN` | `false` | Retain job run status in logging table | | `enable_merge_on_cagg_refresh` | `BOOLEAN` | `false` | Enable MERGE statement on cagg refresh | | `enable_multikey_skipscan` | `BOOLEAN` | `true` | Enable SkipScan for multiple distinct inputs | | `enable_now_constify` | `BOOLEAN` | `true` | Enable constifying now() in query constraints | | `enable_null_compression` | `BOOLEAN` | `true` | Enable null compression | | `enable_optimizations` | `BOOLEAN` | `true` | Enable TimescaleDB query optimizations | | `enable_ordered_append` | `BOOLEAN` | `true` | Enable ordered append optimization for queries that are ordered by the time dimension | | `enable_parallel_chunk_append` | `BOOLEAN` | `true` | Enable using parallel aware chunk append node | | `enable_partitioned_hypertables` | `BOOLEAN` | `false` | Enable experimental support for creating hypertables using PostgreSQL's native declarative partitioning | | `enable_qual_filtering` | `BOOLEAN` | `true` | Filter qualifiers on chunks when complete chunk would be included by filter | | `enable_qual_propagation` | `BOOLEAN` | `true` | Enable propagation of qualifiers in JOINs | | `enable_rowlevel_compression_locking` | `BOOLEAN` | `false` | Use only if you know what you are doing | | `enable_runtime_exclusion` | `BOOLEAN` | `true` | Enable runtime chunk exclusion in ChunkAppend node | | `enable_segmentwise_recompression` | `BOOLEAN` | `true` | Enable segmentwise recompression | | `enable_skipscan` | `BOOLEAN` | `true` | Enable SkipScan for DISTINCT queries | | `enable_skipscan_for_distinct_aggregates` | `BOOLEAN` | `true` | Enable SkipScan for DISTINCT aggregates | | `enable_sparse_index_bloom` | `BOOLEAN` | `true` | This sparse index speeds up the equality queries on compressed columns, and can be disabled when not desired. | | `enable_tiered_reads` | `BOOLEAN` | `true` | Enable reading of tiered data by including a foreign table representing the data in the object storage into the query plan | | `enable_tss_callbacks` | `BOOLEAN` | `true` | Enable ts\_stat\_statements callbacks | | `enable_uuid_compression` | `BOOLEAN` | `true` | Enable uuid compression | | `enable_vectorized_aggregation` | `BOOLEAN` | `true` | Enable vectorized aggregation for compressed data | | `license` | `STRING` | `TS_LICENSE_DEFAULT` | Determines which features are enabled | | `max_cached_chunks_per_hypertable` | `INTEGER` | `1024` | Maximum number of chunks stored in the cache min: `0`, max: `65536` | | `max_open_chunks_per_insert` | `INTEGER` | `1024` | Maximum number of open chunk tables per insert min: `0`, max: `PG_INT16_MAX` | | `max_tuples_decompressed_per_dml_transaction` | `INTEGER` | `100000` | If the number of tuples exceeds this value, an error will be thrown and transaction rolled back. Setting this to 0 sets this value to unlimited number of tuples decompressed. min: `0`, max: `2147483647` | | `read_legacy_bloom1_v1` | `BOOLEAN` | `false` | These legacy indexes might give false negatives if they were built by the TimescaleDB extension compiled with different build options. | | `restoring` | `BOOLEAN` | `false` | In restoring mode all timescaledb internal hooks are disabled. This mode is required for restoring logical dumps of databases with timescaledb. | | `shutdown_bgw_scheduler` | `BOOLEAN` | `false` | this is for debugging purposes | | `skip_cagg_invalidation` | `BOOLEAN` | `false` | When enabled, DML (INSERT/UPDATE/DELETE/COPY) and DDL (DROP CHUNK, drop\_chunks, TRUNCATE) on hypertables with continuous aggregates will not record invalidation entries for the duration of the session or transaction. The caller must explicitly refresh affected continuous aggregates with force => true after the operation. Intended for bulk-migration tools; use with SET LOCAL inside a transaction. Misuse will leave continuous aggregate state out of sync with the underlying hypertable. | | `skip_scan_run_cost_multiplier` | `REAL` | `1.0` | Default is 1.0 i.e. regularly estimated SkipScan run cost, 0.0 will make SkipScan to have run cost = 0 min: `0.0`, max: `1.0` | | `stats_max_chunks` | `INTEGER` | `TS_STATS_MAX_CHUNKS_DEFAULT, /* default */0, /* min: 0 = disabled */TS_STATS_MAX_CHUNKS_MAX, /* max: 2^18 = 262144 chunks */PGC_SIGHUP` | Must be 0 or a power of 2. Takes effect only after server restart. min: `0`, max: `stats_max_chunks_check_hook` | | `telemetry_level` | `ENUM` | `TELEMETRY_DEFAULT` | Level used to determine which telemetry to send | Version: [2.28.2](https://github.com/timescale/timescaledb/releases/tag/2.28.2) # Configuration parameters > Configure your TimescaleDB instance including settings related to memory, workers, disk writes, and transactional locks By default, TimescaleDB uses the default PostgreSQL server configuration settings. However, in some cases, these settings are not appropriate, especially if you have larger servers that use more hardware resources such as CPU, memory, and storage. This section explains the settings you are most likely to need to adjust. Some of these settings are PostgreSQL settings, and some are TimescaleDB-specific settings. For most changes, you can use the [tuning tool](/docs/deploy/self-hosted/configuration/timescaledb-config) to adjust your configuration. For more advanced configuration settings, or to change settings that aren't included in the `timescaledb-tune` tool, you can manually adjust the `postgresql.conf` configuration file. ## Manual PostgreSQL configuration If you prefer to tune settings yourself, or for settings not covered by `timescaledb-tune`, you can manually configure your installation using the PostgreSQL configuration file. For more information about the PostgreSQL configuration page, see the [PostgreSQL documentation](https://www.postgresql.org/docs/current/config-setting.html). ### Edit the PostgreSQL configuration file The location of the PostgreSQL configuration file depends on your operating system and installation. 1. **Find the location of the config file for your PostgreSQL instance** Connect to your database: ```shell psql -d "postgres://:@:/" ``` Retrieve the database file location from the database internal configuration: ```sql SHOW config_file; ``` PostgreSQL returns the path to your configuration file. For example: ```sql -------------------------------------------- /home/postgres/pgdata/data/postgresql.conf (1 row) ``` 2. **Open the config file, then [edit your PostgreSQL configuration](https://www.postgresql.org/docs/current/config-setting.html)** ```shell vi /home/postgres/pgdata/data/postgresql.conf ``` 3. **Save your updated configuration** When you save the changes you make to the configuration file, the new configuration is not applied immediately. The configuration file is automatically reloaded when the server receives a `SIGHUP` signal. To manually reload the file, use the `pg_ctl` command. ### Set parameters at the command prompt If you don't want to open the configuration file to make changes, you can also set parameters directly from the command prompt, using the `postgres` command. For example: ```sql postgres -c log_connections=yes -c log_destination='syslog' ``` ## Memory settings You can adjust these settings to match your machine's available memory: * `shared_buffers` * `effective_cache_size` * `work_mem` * `maintenance_work_mem` * `max_connections` To make it easier, you can use the [PgTune](https://pgtune.leopard.in.ua/) site to work out what settings to use. Enter your machine details, and select the `data warehouse` DB type to see the suggested parameters. Tips You can adjust these settings with `timescaledb-tune`. ## Worker settings PostgreSQL uses worker pools to provide workers for live queries and background jobs. If you do not configure these settings, your queries and background jobs could run more slowly. Configure these worker settings: * `timescaledb.max_background_workers` * `max_parallel_workers` * `max_worker_processes` ### Background workers TimescaleDB background workers are configured with `timescaledb.max_background_workers`. Each database needs a background worker allocated to schedule jobs. Additional workers run background jobs as required. This setting should be the sum of the total number of databases and the total number of concurrent background workers you want running at any one time. By default, `timescaledb-tune` sets `timescaledb.max_background_workers` to 16. You can change this setting directly, use the `--max-bg-workers` flag, or adjust the `TS_TUNE_MAX_BG_WORKERS` [Docker environment variable](/docs/deploy/self-hosted/configuration/docker-config). ### Parallel workers TimescaleDB parallel workers are configured with `max_parallel_workers`. For larger queries, PostgreSQL automatically uses parallel workers if they are available. Increasing this setting can improve query performance for large queries that trigger the use of parallel workers. By default, this setting corresponds to the number of CPUs available. You can change this parameter directly, by adjusting the `--cpus` flag, or by using the `TS_TUNE_NUM_CPUS` [Docker environment variable](/docs/deploy/self-hosted/configuration/docker-config). ### Worker processes The `max_worker_processes` setting defines the total pool of workers available to both background and parallel workers, as well a small number of built-in PostgreSQL workers. It should be at least the sum of `timescaledb.max_background_workers` and `max_parallel_workers`. Tips You can adjust these settings with `timescaledb-tune`. ## Disk write settings By default, disk writes are performed synchronously, so each transaction must be completed and a success message sent, before the next transaction can begin. You can change this to asynchronous to increase write throughput by setting `synchronous_commit = 'off'`. Note that disabling synchronous commits could result in some committed transactions being lost. To help reduce the risk, do not also change `fsync` setting. For more information about asynchronous commits and disk write speed, see the [PostgreSQL documentation](https://www.postgresql.org/docs/current/wal-async-commit.html). Tips You can adjust these settings in the `postgresql.conf` configuration file. ## Transaction lock settings TimescaleDB relies on table partitioning to scale time-series workloads. A hypertable needs to acquire locks on many chunks during queries, which can exhaust the default limits for the number of allowed locks held. In some cases, you might see a warning like this: ```sql psql: FATAL: out of shared memory HINT: You might need to increase max_locks_per_transaction. ``` To avoid this issue, increase the `max_locks_per_transaction` setting from the default value, which is usually 64. This parameter limits the average number of object locks used by each transaction. Individual transactions can lock more objects as long as the locks of all transactions fit in the lock table. For most workloads, choose a number equal to double the maximum number of chunks you expect to have in a hypertable divided by `max_connections`. This takes into account that the number of locks used by a hypertable query is roughly equal to the number of chunks in the hypertable if you need to access all chunks in a query, or double that number if the query uses an index. You can see how many chunks you currently have using the [`timescaledb_information.hypertables`](/docs/reference/timescaledb/informational-views/hypertables) view. Changing this parameter requires a database restart, so make sure you pick a larger number to allow for some growth. For more information about lock management, see the [PostgreSQL documentation](https://www.postgresql.org/docs/current/runtime-config-locks.html). Tips **Tiger Cloud:** Adjust `max_locks_per_transaction` from Tiger Console under **Database configuration → Advanced parameters**. Search for the parameter, edit the value, and click **Apply changes and restart** (this parameter requires a restart). See [Advanced parameters](/docs/deploy/tiger-cloud/tiger-cloud-aws/configuration/advanced-parameters) for details. **Self-hosted:** Adjust this setting in the `postgresql.conf` configuration file and restart the database. ## TimescaleDB configuration settings Just as you can tune settings in PostgreSQL, TimescaleDB provides a number of configuration settings that may be useful to your specific installation and performance needs. You can set these within the `postgresql.conf` file or as command-line parameters when starting PostgreSQL. ### Query planning and execution ##### `timescaledb.enable_chunkwise_aggregation (bool)` If enabled, aggregations are converted into partial aggregations during query planning. The first part of the aggregation is executed on a per-chunk basis. Then, these partial results are combined and finalized. Splitting aggregations decreases the size of the created hash tables and increases data locality, which speeds up queries. ##### `timescaledb.vectorized_aggregation (bool)` Enables or disables the vectorized optimizations in the query executor. For example, the `sum()` aggregation function on compressed chunks can be optimized in this way. ##### `timescaledb.enable_merge_on_cagg_refresh (bool)` Set to `ON` to dramatically decrease the amount of data written on a continuous aggregate in the presence of a small number of changes, reduce the I/O cost of refreshing a [continuous aggregate](/docs/build/continuous-aggregates/create-a-continuous-aggregate), and generate fewer Write-Ahead Logs (WAL). Only works for continuous aggregates that don't have compression enabled. ### Policies ##### `timescaledb.max_background_workers (int)` Maximum background worker processes allocated to TimescaleDB. Set to at least 1 + the number of databases loaded with the TimescaleDB extension in a PostgreSQL instance. Default value is 16. ### Tiger Cloud service tuning ##### `timescaledb.disable_load (bool)` Disables the loading of the actual extension. ### Administration ##### `timescaledb.restoring (bool)` Sets TimescaleDB in restoring mode. Disabled by default. ##### `timescaledb.license (string)` Changes access to features based on the TimescaleDB license in use. For example, setting `timescaledb.license` to `apache` limits TimescaleDB to features that are implemented under the Apache 2 license. The default value is `timescale`, which allows access to all features. ##### `timescaledb.telemetry_level (enum)` Telemetry settings level. Level used to determine which telemetry to send. Can be set to `off` or `basic`. Defaults to `basic`. ##### `timescaledb.last_tuned (string)` Records last time `timescaledb-tune` ran. ##### `timescaledb.last_tuned_version (string)` Version of `timescaledb-tune` used to tune when it runs. ## Complete list of GUC parameters For a complete list, see the [Grand Unified Configuration (GUC) parameters](/docs/reference/timescaledb/configuration/gucs) reference. # Continuous aggregates overview > TimescaleDB reference for calculating continuous aggregates on your data. Includes SQL functions and views related to creating, altering, and dropping continuous aggregates In modern applications, data usually grows very quickly. This means that aggregating it into useful summaries can become very slow. If you are collecting data very frequently, you might want to aggregate your data into minutes or hours instead. For example, if an IoT device takes temperature readings every second, you might want to find the average temperature for each hour. Every time you run this query, the database needs to scan the entire table and recalculate the average. TimescaleDB makes aggregating data lightning fast, accurate, and easy with continuous aggregates. ![How a continuous aggregate incrementally materializes time-bucketed rollups from a raw hypertable, with a real-time aggregate query reading both pre-computed and recent raw data](/docs/_astro/continuous-aggregates-light.BhW3GFu2_pFsfL.webp) ![How a continuous aggregate incrementally materializes time-bucketed rollups from a raw hypertable, with a real-time aggregate query reading both pre-computed and recent raw data](/docs/_astro/continuous-aggregates-dark.CnXJ1DFs_2kSGiL.webp) Continuous aggregates in TimescaleDB are a kind of hypertable that is refreshed automatically in the background as new data is added, or old data is modified. Changes to your dataset are tracked, and the hypertable behind the continuous aggregate is automatically updated in the background. Continuous aggregates have a much lower maintenance burden than regular PostgreSQL materialized views, because the whole view is not created from scratch on each refresh. This means that you can get on with working your data instead of maintaining your database. Because continuous aggregates are based on hypertables, you can query them in exactly the same way as your other tables. This includes continuous aggregates in the rowstore, compressed into the [columnstore](/docs/learn/columnar-storage/understand-hypercore), or [tiered to object storage](/docs/build/data-management/storage/manage-storage). You can even create [continuous aggregates on top of your continuous aggregates](/docs/learn/continuous-aggregates/hierarchical-continuous-aggregates), for an even more fine-tuned aggregation. [Real-time aggregation](/docs/learn/continuous-aggregates/real-time-aggregates) enables you to combine pre-aggregated data from the materialized view with the most recent raw data. This gives you up-to-date results on every query. In TimescaleDB v2.13 and later, real-time aggregates are **DISABLED** by default. In earlier versions, real-time aggregates are **ENABLED** by default; when you create a continuous aggregate, queries to that view include the results from the most recent raw data. For more information about using continuous aggregates, see the documentation in [Use continuous aggregates](/docs/build/continuous-aggregates/create-a-continuous-aggregate). ## Samples [Section titled “Samples”](#samples) ### Create a continuous aggregate [Section titled “Create a continuous aggregate”](#create-a-continuous-aggregate) Create a continuous aggregate that calculates hourly average temperature: ```sql CREATE MATERIALIZED VIEW conditions_hourly WITH (timescaledb.continuous) AS SELECT time_bucket('1 hour', time) AS bucket, location, AVG(temperature) AS avg_temp, MAX(temperature) AS max_temp, MIN(temperature) AS min_temp FROM conditions GROUP BY bucket, location; ``` ### Add a refresh policy [Section titled “Add a refresh policy”](#add-a-refresh-policy) Automatically refresh the continuous aggregate to keep it up to date: ```sql SELECT add_continuous_aggregate_policy('conditions_hourly', start_offset => INTERVAL '3 hours', end_offset => INTERVAL '1 hour', schedule_interval => INTERVAL '1 hour'); ``` ### Manually refresh a continuous aggregate [Section titled “Manually refresh a continuous aggregate”](#manually-refresh-a-continuous-aggregate) Refresh a specific time range in the continuous aggregate: ```sql CALL refresh_continuous_aggregate('conditions_hourly', '2024-01-01', '2024-02-01'); ``` ### Query a continuous aggregate [Section titled “Query a continuous aggregate”](#query-a-continuous-aggregate) Query the continuous aggregate just like a regular table: ```sql SELECT bucket, location, avg_temp FROM conditions_hourly WHERE bucket >= NOW() - INTERVAL '7 days' AND location = 'office' ORDER BY bucket DESC; ``` ## Available functions [Section titled “Available functions”](#available-functions) ### Create and modify continuous aggregates [Section titled “Create and modify continuous aggregates”](#create-and-modify-continuous-aggregates) * [`CREATE MATERIALIZED VIEW (Continuous Aggregate)`](/docs/reference/timescaledb/continuous-aggregates/create_materialized_view): create a continuous aggregate on a hypertable or another continuous aggregate * [`ALTER MATERIALIZED VIEW (Continuous Aggregate)`](/docs/reference/timescaledb/continuous-aggregates/alter_materialized_view): change an existing continuous aggregate * [`DROP MATERIALIZED VIEW (Continuous Aggregate)`](/docs/reference/timescaledb/continuous-aggregates/drop_materialized_view): drop a continuous aggregate view * [`cagg_migrate()`](/docs/reference/timescaledb/continuous-aggregates/cagg_migrate): migrate a continuous aggregate from the old format to the new format introduced in TimescaleDB 2.7 ### Refresh continuous aggregates [Section titled “Refresh continuous aggregates”](#refresh-continuous-aggregates) * [`refresh_continuous_aggregate()`](/docs/reference/timescaledb/continuous-aggregates/refresh_continuous_aggregate): manually refresh a continuous aggregate ### Manage policies [Section titled “Manage policies”](#manage-policies) * [`add_continuous_aggregate_policy()`](/docs/reference/timescaledb/continuous-aggregates/add_continuous_aggregate_policy): add policy to schedule automatic refresh of a continuous aggregate * [`remove_continuous_aggregate_policy()`](/docs/reference/timescaledb/continuous-aggregates/remove_continuous_aggregate_policy): remove a refresh policy from a continuous aggregate ### Experimental policy management [Section titled “Experimental policy management”](#experimental-policy-management) * [`add_policies()`](/docs/reference/timescaledb/continuous-aggregates/add_policies): add refresh, compression, and data retention policies on a continuous aggregate * [`alter_policies()`](/docs/reference/timescaledb/continuous-aggregates/alter_policies): alter refresh, compression, or data retention policies on a continuous aggregate * [`remove_policies()`](/docs/reference/timescaledb/continuous-aggregates/remove_policies): remove refresh, compression, or data retention policies from a continuous aggregate * [`remove_all_policies()`](/docs/reference/timescaledb/continuous-aggregates/remove_all_policies): remove all policies from a continuous aggregate * [`show_policies()`](/docs/reference/timescaledb/continuous-aggregates/show_policies): show all policies that are currently set on a continuous aggregate # add_continuous_aggregate_policy() > Add policy to schedule automatic refresh of a continuous aggregate Since [1.7.0](https://github.com/timescale/timescaledb/releases/tag/1.7.0) Create a policy that automatically refreshes a continuous aggregate. To view the policies that you set or the policies that already exist, see [informational views](/docs/reference/timescaledb/informational-views/jobs). ## Samples [Section titled “Samples”](#samples) Add a policy that refreshes the last month once an hour, excluding the latest hour from the aggregate. For performance reasons, we recommend that you exclude buckets that see lots of writes: ```sql SELECT add_continuous_aggregate_policy('conditions_summary', start_offset => INTERVAL '1 month', end_offset => INTERVAL '1 hour', schedule_interval => INTERVAL '1 hour'); ``` ## Arguments [Section titled “Arguments”](#arguments) The syntax is: ```sql SELECT add_continuous_aggregate_policy( continuous_aggregate = '', start_offset = , end_offset = , schedule_interval = , if_not_exists = true | false, initial_start = , timezone = '', include_tiered_data = true | false, buckets_per_batch = , max_batches_per_execution = , refresh_newest_first = true | false ); ``` | Name | Type | Default | Required | Description | | --------------------------- | ------------------- | -------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `continuous_aggregate` | REGCLASS | - | ✔ | The continuous aggregate to add the policy for | | `start_offset` | INTERVAL or integer | - | ✔ | Start of the refresh window as an interval relative to the time when the policy is executed. `NULL` is equivalent to `MIN(timestamp)` of the hypertable. | | `end_offset` | INTERVAL or integer | - | ✔ | End of the refresh window as an interval relative to the time when the policy is executed. `NULL` is equivalent to `MAX(timestamp)` of the hypertable. | | `schedule_interval` | INTERVAL | 24 hours | ✔ | Interval between refresh executions in wall-clock time. | | `initial_start` | TIMESTAMPTZ | NULL | - | Time the policy is first run. Defaults to NULL. If omitted, then the schedule interval is the interval between the finish time of the last execution and the next start. If provided, it serves as the origin with respect to which the next\_start is calculated | | `if_not_exists` | BOOLEAN | false | - | Set to `true` to issue a notice instead of an error if the job already exists. | | `timezone` | TEXT | NULL | - | A valid time zone. If you specify `initial_start`, subsequent executions of the refresh policy are aligned on `initial_start`. However, daylight savings time (DST) changes may shift this alignment. If this is an issue you want to mitigate, set `timezone` to a valid time zone. Default is `NULL`, [UTC bucketing](/docs/learn/data-lifecycle/time-buckets/about-time-buckets#timezones) is performed. | | `include_tiered_data` | BOOLEAN | NULL | - | Enable/disable reading tiered data. This setting helps override the current settings for the`timescaledb.enable_tiered_reads` GUC. The default is NULL i.e we use the current setting for `timescaledb.enable_tiered_reads` GUC | | `buckets_per_batch` | INTEGER | 10 | - | Number of buckets to be refreshed by a batch. This value is multiplied by the continuous aggregate bucket width to determine the size of the batch range. Default value is `10`. Values of less than `0` are not allowed. | | `max_batches_per_execution` | INTEGER | 0 | - | Limit the maximum number of batches to run when a policy executes. If some batches remain, they are processed the next time the policy runs. Default value is `0`, for an unlimted number of batches. Values of less than `0` are not allowed. | | `refresh_newest_first` | BOOLEAN | TRUE | - | Control the order of incremental refreshes. Set to `TRUE` to refresh from the newest data to the oldest. Set to `FALSE` for oldest to newest. The default is `TRUE`. | The `start_offset` should be greater than `end_offset`. You must specify the `start_offset` and `end_offset` parameters differently, depending on the type of the time column of the hypertable: * For hypertables with `TIMESTAMP`, `TIMESTAMPTZ`, and `DATE` time columns, set the offset as an `INTERVAL` type. * For hypertables with integer-based timestamps, set the offset as an `INTEGER` type. Tips While setting `end_offset` to `NULL` is possible, it is not recommended. To include the data between `end_offset` and the current time in queries, enable [real-time aggregation](/docs/learn/continuous-aggregates/real-time-aggregates). You can add [concurrent refresh policies](/docs/build/continuous-aggregates/refresh-policies) on each continuous aggregate, as long as the `start_offset` and `end_offset` does not overlap with another policy on the same continuous aggregate. Note Setting `buckets_per_batch` greater than zero means that the refresh window is split in batches of `bucket width` \* `buckets per batch`. For example, a given continuous aggregate with `bucket width` of `1 day` and `buckets_per_batch` of 10 has a batch size of `10 days` to process the refresh. Because each `batch` is an individual transaction, executing a policy in batches make the data visible for the users before the entire job is executed. Batches are processed from the most recent data to the oldest. ## Returns [Section titled “Returns”](#returns) | Column | Type | Description | | -------- | ------- | -------------------------------------------------------------- | | `job_id` | INTEGER | TimescaleDB background job ID created to implement this policy | # add_policies() > Add refresh, compression, and data retention policies on a continuous aggregate Early access [2.10.0](https://github.com/timescale/timescaledb/releases/tag/2.10.0) Add refresh, compression, and data retention policies to a continuous aggregate in one step. The added compression and retention policies apply to the continuous aggregate, *not* to the original hypertable. ```sql timescaledb_experimental.add_policies( relation REGCLASS, if_not_exists BOOL = false, refresh_start_offset "any" = NULL, refresh_end_offset "any" = NULL, compress_after "any" = NULL, drop_after "any" = NULL) ) RETURNS BOOL ``` Note `add_policies()` does not allow the `schedule_interval` for the continuous aggregate to be set, instead using a default value of 1 hour. If you would like to set this add your policies manually (see [`add_continuous_aggregate_policy`](/docs/reference/timescaledb/continuous-aggregates/add_continuous_aggregate_policy)). ## Samples [Section titled “Samples”](#samples) Given a continuous aggregate named `example_continuous_aggregate`, add three policies to it: 1. Regularly refresh the continuous aggregate to materialize data between 1 day and 2 days old. 2. Compress data in the continuous aggregate after 20 days. 3. Drop data in the continuous aggregate after 1 year. ```sql SELECT timescaledb_experimental.add_policies( 'example_continuous_aggregate', refresh_start_offset => '2 days'::interval, refresh_end_offset => '1 day'::interval, compress_after => '20 days'::interval, drop_after => '1 year'::interval ); ``` ## Arguments [Section titled “Arguments”](#arguments) The syntax is: ```sql SELECT timescaledb_experimental.add_policies( relation = '', refresh_start_offset = , refresh_end_offset = , compress_after = , drop_after = ); ``` | Name | Type | Default | Required | Description | | ---------------------- | ----------------------- | ------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | | `relation` | `REGCLASS` | - | ✔ | The continuous aggregate that the policies should be applied to | | `if_not_exists` | `BOOL` | false | - | When true, prints a warning instead of erroring if the continuous aggregate doesn't exist. | | `refresh_start_offset` | `INTERVAL` or `INTEGER` | - | - | The start of the continuous aggregate refresh window, expressed as an offset from the policy run time. | | `refresh_end_offset` | `INTERVAL` or `INTEGER` | - | - | The end of the continuous aggregate refresh window, expressed as an offset from the policy run time. Must be smaller than `refresh_start_offset`. | | `compress_after` | `INTERVAL` or `INTEGER` | - | - | continuous aggregate chunks are compressed if they exclusively contain data older than this interval. | | `drop_after` | `INTERVAL` or `INTEGER` | - | - | continuous aggregate chunks are dropped if they exclusively contain data older than this interval. | For arguments that could be either an `INTERVAL` or an `INTEGER`, use an `INTERVAL` if your time bucket is based on timestamps. Use an `INTEGER` if your time bucket is based on integers. ## Returns [Section titled “Returns”](#returns) Returns `true` if successful. # ALTER MATERIALIZED VIEW (continuous aggregate) > Change an existing continuous aggregate Since [1.3.0](https://github.com/timescale/timescaledb/releases/tag/1.3.0) You use the `ALTER MATERIALIZED VIEW` statement to modify some of the `WITH` clause [options](/docs/reference/timescaledb/continuous-aggregates/create_materialized_view#arguments) for a continuous aggregate view. You can only set the `continuous` and `create_group_indexes` options when you [create a continuous aggregate](/docs/reference/timescaledb/continuous-aggregates/create_materialized_view#arguments). `ALTER MATERIALIZED VIEW` also supports the following clauses on the continuous aggregate view. The [PostgreSQL clauses](https://www.postgresql.org/docs/current/sql-alterview.html) behave as they do for a standard view: * `RENAME TO`: rename the continuous aggregate view * `RENAME [COLUMN]`: rename the continuous aggregate column * `SET SCHEMA`: set the new schema for the continuous aggregate view * `SET TABLESPACE`: move the materialization of the continuous aggregate view to the new tablespace * `OWNER TO`: set a new owner for the continuous aggregate view * `ADD COLUMN GENERATED ALWAYS AS () STORED`: add a new column to the continuous aggregate without recreating it. The expression can be a new aggregate, or a column that already appears in the `GROUP BY` clause of the continuous aggregate definition. New data populates the column as the continuous aggregate refreshes going forward; existing rows remain `NULL` until you force a refresh over the historical range to backfill them. ## Samples [Section titled “Samples”](#samples) * Enable real-time aggregates for a continuous aggregate: ```sql ALTER MATERIALIZED VIEW contagg_view SET (timescaledb.materialized_only = false); ``` * Enable hypercore for a continuous aggregate: Community Since [2.18.0](https://github.com/timescale/timescaledb/releases/tag/2.18.0) ```sql ALTER MATERIALIZED VIEW contagg_view SET ( timescaledb.enable_columnstore = true, timescaledb.segmentby = 'symbol' ); ``` * Rename a column for a continuous aggregate: ```sql ALTER MATERIALIZED VIEW contagg_view RENAME COLUMN old_name TO new_name; ``` * Add a new aggregate to a continuous aggregate without recreating it: ```sql ALTER MATERIALIZED VIEW contagg_view ADD COLUMN max_temp double precision GENERATED ALWAYS AS (max(temperature)) STORED; ``` You can also add a column that already appears in the `GROUP BY` clause of the continuous aggregate definition, even when it is not an aggregate: ```sql ALTER MATERIALIZED VIEW contagg_view ADD COLUMN device_id integer GENERATED ALWAYS AS (device_id) STORED; ``` In both cases, the new column is populated for new data as the continuous aggregate refreshes going forward. Existing rows remain `NULL` until you force a refresh over the historical range to backfill them: ```sql CALL refresh_continuous_aggregate('contagg_view', '2020-01-01', '2020-02-01', force => true); ``` ## Arguments [Section titled “Arguments”](#arguments) The syntax is: ```sql ALTER MATERIALIZED VIEW SET ( timescaledb. = [, ... ] ) ``` | Name | Type | Default | Required | Description | | ------------------------------------------------------------------------- | -------- | ---------------------------------------------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `view_name` | TEXT | - | ✔ | The name of the continuous aggregate view to be altered. | | `timescaledb.materialized_only` | BOOLEAN | `true` | - | Return only materialized data when querying the continuous aggregate view. Set to `false` to enable real-time aggregation. | | `timescaledb.enable_columnstore` | BOOLEAN | `true` | - | Enable columnstore. Since 2.18.0 | | `timescaledb.orderby` | TEXT | Descending order on the time column in `table_name`. | - | Set the order in which items are used in the columnstore. Specified in the same way as an `ORDER BY` clause in a `SELECT` query. Since 2.18.0 | | `timescaledb.segmentby` | TEXT | No segmentation by column. | - | Set the list of columns used to segment data in the columnstore for `table`. An identifier representing the source of the data such as `device_id` or `tags_id` is usually a good candidate. Since 2.18.0 | | `column_name` | TEXT | - | - | Set the name of the column to order by or segment by. | | `timescaledb.compress_chunk_time_interval` | TEXT | - | - | Reduce the total number of compressed/columnstore chunks for `table`. If you set `compress_chunk_time_interval`, compressed/columnstore chunks are merged with the previous adjacent chunk within `chunk_time_interval` whenever possible. These chunks are irreversibly merged. If you call to [decompress](/docs/reference/timescaledb/hypercore/convert_to_rowstore)/[convert\_to\_rowstore](/docs/reference/timescaledb/hypercore/convert_to_rowstore), merged chunks are not split up. You can call `compress_chunk_time_interval` independently of other compression settings; `timescaledb.enable_columnstore` is not required. | | `timescaledb.chunk_interval` (formerly `timescaledb.chunk_time_interval`) | INTERVAL | 10x the original hypertable. | - | Set the chunk interval. Renamed in TimescaleDB V2.20. | ## Returns [Section titled “Returns”](#returns) For standard `ALTER MATERIALIZED VIEW` return behavior, see the [PostgreSQL ALTER MATERIALIZED VIEW documentation](https://www.postgresql.org/docs/current/sql-alterview.html). # alter_policies() > Alter refresh, compression, or data retention policies on a continuous aggregate Early access [2.10.0](https://github.com/timescale/timescaledb/releases/tag/2.10.0) Alter refresh, columnstore, or data retention policies on a continuous aggregate. The altered columnstore and retention policies apply to the continuous aggregate, *not* to the original hypertable. ```sql timescaledb_experimental.alter_policies( relation REGCLASS, if_exists BOOL = false, refresh_start_offset "any" = NULL, refresh_end_offset "any" = NULL, compress_after "any" = NULL, drop_after "any" = NULL ) RETURNS BOOL ``` ## Samples [Section titled “Samples”](#samples) Given a continuous aggregate named `example_continuous_aggregate` with an existing columnstore policy, alter the columnstore policy to compress data older than 16 days: ```sql SELECT timescaledb_experimental.alter_policies( 'continuous_agg_max_mat_date', compress_after => '16 days'::interval ); ``` ## Arguments [Section titled “Arguments”](#arguments) The syntax is: ```sql SELECT timescaledb_experimental.alter_policies( relation = '', refresh_start_offset = , refresh_end_offset = , compress_after = , drop_after = ); ``` | Name | Type | Default | Required | Description | | ---------------------- | ----------------------- | ------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | | `relation` | `REGCLASS` | - | ✔ | The continuous aggregate that you want to alter policies for | | `if_exists` | `BOOL` | false | - | When true, prints a warning instead of erroring if the policy doesn't exist. | | `refresh_start_offset` | `INTERVAL` or `INTEGER` | - | - | The start of the continuous aggregate refresh window, expressed as an offset from the policy run time. | | `refresh_end_offset` | `INTERVAL` or `INTEGER` | - | - | The end of the continuous aggregate refresh window, expressed as an offset from the policy run time. Must be smaller than `refresh_start_offset`. | | `compress_after` | `INTERVAL` or `INTEGER` | - | - | continuous aggregate chunks are compressed into the columnstore if they exclusively contain data older than this interval. | | `drop_after` | `INTERVAL` or `INTEGER` | - | - | continuous aggregate chunks are dropped if they exclusively contain data older than this interval. | For arguments that could be either an `INTERVAL` or an `INTEGER`, use an `INTERVAL` if your time bucket is based on timestamps. Use an `INTEGER` if your time bucket is based on integers. ## Returns [Section titled “Returns”](#returns) Returns `true` if successful. # cagg_migrate() > Migrate a continuous aggregate from the old format to the new format introduced in TimescaleDB 2.7 Sunsetted [2.25.0](https://github.com/timescale/timescaledb/releases/tag/2.25.0)\ Since [2.7.0](https://github.com/timescale/timescaledb/releases/tag/2.7.0) Note This procedure was removed in TimescaleDB 2.25. The underlying migration planning tables were removed in TimescaleDB 2.26. Migrate a continuous aggregate from the old format to the new format introduced in TimescaleDB 2.7. ```sql CALL cagg_migrate ( cagg REGCLASS, override BOOLEAN DEFAULT FALSE, drop_old BOOLEAN DEFAULT FALSE ); ``` TimescaleDB 2.7 introduced a new format for continuous aggregates that improves performance. It also makes continuous aggregates compatible with more types of SQL queries. The new format, also called the finalized format, stores the continuous aggregate data exactly as it appears in the final view. The old format, also called the partial format, stores the data in a partially aggregated state. Use this procedure to migrate continuous aggregates from the old format to the new format. For more information, see the [migration how-to guide](/docs/build/continuous-aggregates/migrate-to-new-form). Warning There are known issues with `cagg_migrate()` in version TimescaleDB 2.8.0. Upgrade to version 2.8.1 or above before using it. ## Arguments [Section titled “Arguments”](#arguments) The syntax is: ```sql CALL cagg_migrate( cagg = '', override = true | false, drop_old = true | false ); ``` | Name | Type | Default | Required | Description | | ---------- | ---------- | ------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `cagg` | `REGCLASS` | - | ✔ | The continuous aggregate to migrate | | `override` | `BOOLEAN` | `false` | - | If false, the old continuous aggregate keeps its name. The new continuous aggregate is named `_new`. If true, the new continuous aggregate gets the old name. The old continuous aggregate is renamed `_old`. | | `drop_old` | `BOOLEAN` | `false` | - | If true, the old continuous aggregate is deleted. Must be used together with `override`. | ## Returns [Section titled “Returns”](#returns) This function returns void. # CREATE MATERIALIZED VIEW (continuous aggregate) > Create a continuous aggregate on a hypertable or another continuous aggregate You use the `CREATE MATERIALIZED VIEW` statement to create continuous aggregates. To learn more, see the [continuous aggregate how-to guides](/docs/build/continuous-aggregates/create-a-continuous-aggregate). The syntax is: ```sql CREATE MATERIALIZED VIEW [ ( column_name [, ...] ) ] WITH ( timescaledb.continuous [, timescaledb.