--- title: Sync data from S3 | Tiger Data Docs description: Sync CSV and Parquet files from an S3 bucket to Tiger Cloud in real time using the source S3 connector --- This guide walks you through syncing CSV and Parquet files from an Amazon S3 bucket to your Tiger Cloud service in real time using the source S3 connector in Tiger Cloud. The connector runs continuously so you can use Tiger Cloud as your analytics database with data constantly synced from S3, without building or operating custom ETL between S3 and Tiger Cloud. **What you get:** The connector syncs from an S3 bucket to a Tiger Cloud service using glob patterns, watches for new files and imports them on a configurable schedule, supports CSV (including GZ and ZIP compression) and Parquet (converted to CSV then processed with timescaledb-parallel-copy), lets you enable a hypertable during file-to-table mapping and add columnstore and continuous aggregates later in the SQL editor, and polls on a default 1-minute interval (configurable with a cron expression). **File ordering:** The connector processes files in **lexicographical order**. It uses the name of the last file processed as a marker and only fetches files that sort later in the alphabet. Files whose names sort earlier than the marker are skipped and never synced (for example, if the marker is at `Elephant`, a file named `Bob` added later is never processed). For large backlogs, use a 1-minute check until caught up. Note The connector only syncs existing and new files. It does not update or delete records in your Tiger Cloud tables when files are updated or deleted in S3. ![Connectors overview in Tiger Console](/docs/_astro/tiger-console-connector-overview.DOnSC8st_1D7nU8.webp) ## Before you start ### Prerequisites To follow the steps on this page: - Create a [Tiger Cloud service](/docs/get-started/quickstart/create-service/index.md). - Ensure access to a standard Amazon S3 bucket containing your data files. **Directory buckets are not supported.** - Configure access credentials for the S3 bucket: - **IAM Role:** Set the trust policy `Principal` to `arn:aws:iam::142548018081:role/timescale-s3-connections` and `ExternalID` to your Tiger Cloud project and service ID in the form `/` (to avoid the confused deputy problem). Grant `s3:GetObject` and `s3:ListBucket`. - **Public anonymous user:** Supported if the bucket is public. Note This feature is currently not supported for Tiger Cloud on Microsoft Azure. ### Limitations - **File naming:** Files must follow lexicographical ordering; names that sort earlier than already-processed files are permanently skipped (for example, after `file_2024_01_15.csv` is processed, `file_2024_01_10.csv` added later is never synced). Use timestamps (for example, `YYYY-MM-DD-HHMMSS`) or sequential names with fixed padding (for example, `file_00001`, `file_00002`). - **CSV:** Max file size 1 GB (contact sales\@tigerdata.com to increase), max row size 2 MB. Compressed formats: GZ, ZIP. Delimiter and “skip header” are configurable in advanced settings. - **Parquet:** Max file size 1 GB, max row size 2 MB. - **Sync iteration:** Up to 100 files per sync iteration; additional checks only fill empty queue slots. ## How to set up S3 sync Follow these steps in [Tiger Console](https://console.cloud.tigerdata.com/dashboard/services) to connect your S3 bucket and start syncing. 1. **Connect to your Tiger Cloud service** In [Tiger Console](https://console.cloud.tigerdata.com/dashboard/services), select the service you want to sync live data into. 2. **Connect the source S3 bucket to the target service** 1. Click `Connectors` → `Amazon S3`. 2. Click the pencil icon and set the name for the new connector. 3. Set `Bucket name` and `Authentication method`, then click `Continue`. For IAM role setup, click `Learn how` in the UI. Tiger Console connects to the source bucket. 4. In `Define files to sync`, choose `File type` and set the `Glob pattern`. Examples: - `/*`, all files in a folder (patterns ending with `/` are treated as `/*`). - `/**`: all files recursively. - `/**/*.csv`, a specific file type. Keep filters toward the end of the pattern; complex filtering can cause S3 list operations to time out. 5. Click the search icon to see the files that match, then click `Continue`. ![Connecting Tiger Cloud to an S3 bucket](/docs/_astro/s3-connector-tiger-console.DDuCo9oV_ZBF0v1.webp) 3. **Map schema and optimize for hypertables** Tiger Console infers the file schema and, when possible, suggests the time column for a hypertable. ![S3 connector table selection in Tiger Console](/docs/_astro/tiger-console-s3-connector-create-tables.CWZ2J2w5_Z1XzU94.webp) - Choose `Create a new table for your data` or `Ingest data to an existing table`. - Set the `Data type` for each column, then click `Continue`. - Configure insert behavior on conflict, then click `Continue`. - Choose the polling interval (for example, minute, hour, or a cron expression). - Click `Start Connector`. Tiger Console starts the connector and shows progress. You now have continuous sync from your S3 bucket to your Tiger Cloud service. ## How to monitor synchronization - **Connector data flow:** Click `Connectors` to see the diagram with connector status and amount of data replicated. - **Per-connector stats:** Click `Connectors` → `Source connectors`, then select your connector. ![S3 connector import details and statistics](/docs/_astro/tiger-console-s3-connector-import-details.DbkXHMp-_1BMtgk.webp) From the connector dashboard you can: - **Search by file name** to find specific imports. - **Filter by status:** All statuses, Cancelled, Failure, In Queue, Paused, Pending Retry, Running, Success. - **Bulk retry** for files in Error status. - View **Lifecycle history** for time spent in each status. - **Refresh every minute** (auto-refresh). ## How to manage the connector - **Pause:** `Connectors` → `Source connectors` → three-dot menu next to your connector → `Pause`. ![Pausing an S3 connector in Tiger Console](/docs/_astro/tiger-console-s3-connector-pause.C5OYyjFX_1JBHaD.webp) - **Edit:** Same table → three-dot menu → `Edit` → `Connector settings`. You must pause the connector before editing. ![Editing S3 connector settings in Tiger Console](/docs/_astro/tiger-console-s3-connector-edit.Dwm-LblS_XvOVP.webp) - **Delete:** Three-dot menu → `Delete` (pause the connector first). ## Summary You now have continuous sync from your S3 bucket to your Tiger Cloud service: new files that match your glob pattern are picked up on the schedule you chose, and you can monitor imports and manage the connector (pause, edit, or delete) from Tiger Console anytime. To extend the pipeline, enable [hypertables](/docs/learn/hypertables/understand-hypertables/index.md), [columnstore](/docs/learn/columnar-storage/understand-hypercore/index.md), or [continuous aggregates](/docs/learn/continuous-aggregates/index.md) on the synced table via the SQL editor. ## Related - [Sync data from PostgreSQL](/docs/migrate/livesync-for-postgresql/index.md): Continuously replicate from a PostgreSQL database instead of S3. - [Upload a file using Tiger Console](/docs/migrate/import-console/index.md): Upload CSV/TSV directly in the web console without a connector. - [Upload a file using the terminal](/docs/migrate/import-terminal/index.md): Load data using the command line (for example, psql, `COPY`). - [Live import from a database](/docs/migrate/live-migration/index.md): Migrate with minimal or no downtime from databases. - [Source S3 connector reference](/docs/integrate/connectors/source/sync-from-s3/index.md): Technical details and capabilities of the S3 connector.