Category: All posts
Aug 01, 2023
Virtual Private Cloud (or VPC) peering enables you to securely access data stored in your cloud database from your existing cloud infrastructure without ever exposing your services to the public internet. This can be incredibly useful if your use case demands maximum security and privacy—VPC peering allows you to replicate (as much as possible) the isolation of a self-hosted deployment.
In this blog post, we walk you through everything you need to know to configure VPC peering between Timescale and AWS, moving from the fundamentals all the way toward advanced use cases. This post aims to help users with little or no networking background get up and running quickly with VPC peering—but even more experienced users may find helpful information here!
As we explained, VPC peering is like a superhero invisibility cloak, allowing you to access your cloud data without the risk of having your services exposed to the public internet.
More specifically, this feature enables you to create a private network “peering” connection between your Amazon VPC(s) and your Timescale VPC(s), making it possible for the machines in the two VPCs to speak to each other directly without going through the wider Internet. The services within your Timescale VPC will only be accessible from your Amazon VPC. By isolating services in such a manner, you gain greater security and control over your database.
Therefore, VPC peering involves configuring two networks: an external cloud providers network (AWS) and the Timescale network. So let’s start by covering the basic networking concepts you need to be familiar with.
In the Timescale Support Team, we often receive support cases from users seeking assistance with VPC configuration. Often, we find that a lack of knowledge about networking fundamentals is at the heart of the issue—let’s fix that by explaining the fundamental components of a network and how they relate to VPCs.
If you’re familiar with these topics, please feel free to jump to the Configuring Your Networks for VPC Peering section.
An IP address is your Internet Protocol Address. In much the same way as your postal address allows other people to find your home, your IP address allows other computers to find your computer on the internet.
An IPv4 address is written in a human-readable format of dotted decimal (base 10) notation e.g., 192.168.0.1. This represents four groups of decimal values separated by a decimal point. Computers interpret this IPv4 address as a series of ones and zeros (binary), and, therefore, this IPv4 address in binary format is represented by four segments of ones and zeros separated by a decimal point (11000000.10101000.00000000.00000001).
Every IP address has two portions that represent a network portion and a host portion. The network ID helps route traffic to your host ID, in the same way as the country and city on your postal address help route packages to your home.
10 Main Street could be anywhere, but 10 Main Street in City (Nairobi), in Country (Kenya), can be found efficiently and securely.
A subnet mask differentiates the network portion and the host portion of your IP address. Without going into too much detail, the subnet mask is a combination of ones and zeros, which, when applied to an IP address, will distinguish the host and network portions. The network bits will be ones, and the host bits will be zeros. Below is an illustration of these two IPv4 address portions together with a subnet mask.
| Network ID | Host ID | ||
IPv4 address | 192 | 168 | 0 | 1 |
Subnet mask | 255 | 255 | 255 | 0 |
| 24 bits | |||
This is a 24-bit subnet mask because, in binary representation, we have eight ones in the first three octets (i.e., 11111111.11111111.11111111.00000000). In Classless Inter-Domain Routing format (or supernetting), this will be presented with a /24 as 192.168.0.0/24.
CIDR breaks away from the standard network classes to expand the standard subnets to accommodate a wider customizable range of network IPv4 addresses. For example, if we modify the /24 subnet mask to /20 subnet mask, this will increase the number of host network IDs from 254 (2^8 -2) to 4,094 (2^12 -2).
In our analogy, if a subnet is seen as a city, we’ll describe a CIDR block as a further way to segment this subnet, so a CIDR block could be seen as a postcode or neighborhood.
Routing is the selection of the best route from one computer (IP Address) to another. Reviewing your table of available routes is like reading a map.
Picture yourself visiting Nairobi National Park in Kenya for the first time. When you land at Jomo Kenyatta International Airport, you take a taxi, and the taxi driver can use different routes to get you to Nairobi National park. Many routes are available, but you want to take the fastest or most efficient path to your destination. You can’t make it to your destination if there's no available route. Similarly, if a suitable route doesn’t exist in your VPC routing table, then your computer (IP address) in one VPC can’t communicate with computers (IP addresses) in the connected VPC.
A security group in AWS is a virtual firewall that helps control inbound and outbound traffic into and out of AWS VPC. A security group can only be linked to one VPC, but you can set up more than one security group in a VPC. For simplicity, we encourage users to create a new AWS security group specific to Timescale VPC peering.
Note: There is a limit of five AWS security groups in a VPC. If you’ve hit this limit in your AWS VPC, you can edit an existing security group instead of creating a new one.
VPC peering involves two networks which, if not carefully designed, can lead to a conflict between the two VPCs network IP addresses—for instance, assuming that your AWS VPC CIDR block is using a network such as 10.0.0.0/16, which has a subnet mask of 255.255.0.0 and a usable host IP range of 10.0.0.1 - 10.0.255.254 within the AWS environment. You must ensure that there is no overlap with this range of IP addresses on your Timescale VPC CIDR block.
The image below illustrates a CIDR overlap error when using AWS VPC CIDR block of 10.0.0.0/16 and Timescale VPC CIDR block of 10.0.0.0/16. The error displayed in red reads, “AWS vpc peering connection request has failed.” The error will not indicate that this is a CIDR overlap; therefore, part of the troubleshooting process whenever the VPC creation errors out should be to check both the AWS and Timescale VPC CIDR blocks to ensure the two CIDR blocks do not overlap.
The best way to avoid overlap is to be sensible with the IP address range you assign to each side of the VPC. Estimate the total number of client devices or servers connecting to the TimescaleDB database now and project future usage based on your company's growth.
We gave an example of a network with a subnet mask of /24, giving us 254 host IP addresses. In most cases, this number of host addresses is enough to serve most organization requirements, but you can also use a lower subnet mask (e.g., /20 to increase the number of host IP addresses).
Using AWS VPC CIDR block 10.0.0.0/16 and a Timescale VPC CIDR block of 192.168.0.0/24, the VPC peering connection will be successful, as illustrated below. A point worth noting is that the peering connection request will be in a “processing” state on the Timescale console until you accept the peering connection request on the AWS end.
Note: The peering connection expires after seven days. If you have not accepted or declined this peering connection request, you will need to initiate the peering connection again from the Timescale end.
The screenshot below illustrates a peering connection request's “processing” state.
1. Proceed to the AWS VPC peering connections page. The new Timescale peering connection request status will be in “pending acceptance” state.
a. Before accepting a new peering connection request, validate it to ensure that it corresponds to your Timescale request.
b. You should check fields such as “requester CIDRs” and “requester region” and make sure that these two fields correspond to your Timescale VPC details.
2. Click on the peering connection ID hyperlink that has a "pcx-" prefix to view more details of this peering connection request.
3. Go to the Actions drop-down menu and select the “Accept request” option.
a. This establishes a VPC peering connection between Timescale and AWS.
b. The status on AWS changes to “Active,” while the status on the Timescale end changes to “Accepted.”
We can now create new Timescale services or migrate existing services into this VPC. However, these services will not be accessible from the public internet or AWS network because we have not yet defined routes to route traffic between AWS and Timescale.
Note: Timescale VPC is region-based, and existing services must be in the same region to allow migration of these services to a VPC. This also applies when you create a new service. Timescale has implemented control measures to prevent users from attaching services to a VPC that is in a different region.
We defined our AWS VPC CIDR block as 10.0.0.0/16, which means that all AWS services (EC2 instances, Lambda functions, bastion host, QuickSight, etc.) hosted in this VPC will have IP addresses within this CIDR block range, and traffic from these services will need a route to navigate to the Timescale VPC.
When an AWS VPC is created, the main route table is created by default. This route table has one route that allows internal traffic to flow between services created within this VPC. We leverage this main route table to define another route that will allow both outbound and inbound traffic to and from Timescale VPC.
On the AWS console, under the VPC page, go to the “Route tables” page. All routes are displayed, which also includes routes for other VPCs within your AWS environment. To narrow down the routes associated with your VPC, you can filter route tables using your AWS VPC ID.
We then click on the main route table (that has a “YES” under the “Main” column) and add our Timescale VPC peering connection. The destination column requires our Timescale VPC CIDR block (192.168.0.0/24), and the target column will be the peering connection ID (that starts with the prefix "pcx-").
This newly added route will simply ensure that any traffic destined to or from the Timescale CIDR block 192.168.0.0/24 will be routed through the VPC peering target connection (pcx-).
So far, we have created a peering connection between Timescale and AWS and defined a route for traffic flow. Still, by default, all Timescale services ports are blocked, and no traffic can flow from AWS to Timescale. We, therefore, need to open port 5432 explicitly.
Why port 5432? All Timescale VPC-peered services are automatically assigned port 5432. The port might be seen as the gate to the house (IP address). You can get super close but can’t get to the door without the gate (port) being open.
How-To: Add a new outbound rule of “type” Custom TCP and “port range” 5432, choose custom “destination,” and add Timescale CIDR block (192.168.0.0/24).
To access Timescale services, we need to connect from within the AWS environment. This includes such services as EC2 instances, a bastion host, Lambda functions, QuickSight, and so on.
Many corporations have VPN services running in their data centers. You can leverage your organization’s VPN to access the Timescale VPC-peered services directly from your client network. This requires your networking team to set up site-to-site VPN tunnels between your on-premise data center and AWS VPC. You can also leverage AWS direct connect services to link on-premise data centers with AWS VPC. In case you are using your home network, you can also set up a VPN tunnel using other free VPN providers, such as OpenVPN, Wireguard, etc.
Now that you’re familiar with the networking concepts necessary to navigate VPC peering and you have your networks ready, let's discuss how to configure three of the most common AWS services, starting with EC2 instances (the most common scenario) to continue with more advanced use cases like Lambda functions and AWS QuickSight.
Start by Creating a new EC2 instance in your AWS VPC (make sure that you use the AWS VPC we used in the above steps), attach the security group we created above, and install PostgreSQL. Below are simplified steps to install PostgreSQL version 13 on an EC2 instance:
sudo yum -y update
sudo tee /etc/yum.repos.d/pgdg.repo<<EOF
[pgdg13]
name=PostgreSQL 13 for RHEL/CentOS 7 - x86_64
baseurl=https://download.postgresql.org/pub/repos/yum/13/redhat/rhel-7-x86_64
enabled=1
gpgcheck=0
EOF
sudo yum makecache
sudo yum -y install postgresql13 postgresql13-server
sudo yum -y install postgresql13 postgresql13-server
sudo systemctl start postgresql-13
sudo systemctl enable postgresql-13
sudo systemctl status postgresql-13
If the service is active, your PostgreSQL client is ready to connect to Timescale VPC-peered service. To establish a connection to the Timescale VPC-peered service, we need a connection string that includes the following components: Hostname, Database name, Username, Port, and Password.
The Timescale service console overview page provides these details except the password. On the Timescale overview page, we have a service URL configured with all these components that we will use to connect to our service.
Grab the service URL and type the following on your EC2 terminal. Replace the service_id and project_id placeholders with your service URL details:
psql -x postgres://tsdbadmin@service_id.project_id.tsdb.cloud.timescale.com:5432/tsdb?sslmode=require
This command will prompt for a password—provide your VPC-peered Timescale service password and press enter. Congratulations! You are now connected to a Timescale VPC-peered service. Take note that sources outside the AWS environment cannot connect to this VPC-peered Timescale service. You can test this scenario using the above psql command on your workstation terminal or other client tools, e.g., pgAdmin, DBeaver, etc.
AWS Lambda service has gained popularity with Timescale customers and is among the most connected AWS services to VPC-peered Timescale services.
Customers connecting to a VPC-peered Timescale service often have issues getting the right VPC settings. This section will highlight the key AWS Lambda VPC settings required to establish connectivity to Timescale services.
AWS Lambda VPC settings
The AWS docs thoroughly define the steps for creating a new Lambda function, and our focus will be to guide you through key Lambda functions' VPC settings. In AWS, create a Lambda function page under Advanced settings, click on the Enable VPC checkbox, select the AWS VPC that we defined above, select Subnet, and finally, the security group we defined above.
For existing Lambda functions that don’t have VPC configured, select the Lambda function name on the functions list page, click on configuration, select VPC from the list and click the edit button to edit VPC settings. This will take you to the VPC settings, as highlighted in the AWS Lambda VPC Settings section.
Testing AWS Lambda functions
Using Python 3.9 as our source code language, we create a simple AWS Lambda function that connects to our VPC-peered Timescale service and fetches some records from a hypertable. To do this, we will use the Psycopg2 PostgreSQL driver to perform our CRUD (create, read, update, and delete) operations. AWS doesn’t have these third-party libraries in-built; therefore, we need to deploy the psycopg2-binary Python package as an AWS Lambda layer.
Deploying psycopg2-binary Python package in AWS Lambda layer
mkdir -p ./python/lib/python3.9/site-packages
cp -r psycopg2 psycopg2_binary-2.9.3.dist-info psycopg2_binary.libs ./python/lib/python3.9/site-packages
zip -r python.zip ./python
Under Layer source, choose Custom layers, and select the layer we created above and the corresponding version.
We should now be able to use the Pyscopg2 library in our Lambda function and query a hypertable hosted in our VPC-peered Timescale service:
Connecting from AWS QuickSight to a publicly accessible (not VPC-peered) Timescale service requires simple steps of creating a PostgreSQL dataset with the following Timescale service details:
One common error you will encounter is when you try to create a QuickSight data source with SSL enabled. Currently, Timescale only has self-signed certificates, while QuickSight does not accept self-signed certificates. To work around this problem, you have to uncheck the Enable SSL checkbox on the QuickSight Create data source page that is checked by default.
Timescale uses the password authentication method SCRAM (Salted Challenge Response Authentication Mechanism) by default, which is more secure than the MD5 (Message-Digest algorithm) method. However, we have a known problem with QuickSight when connecting a username with the SCRAM authentication method to a Timescale database. You will get an “authentication not supported” error using the SCRAM method.
The workaround is to change the Timescale authentication method from SCRAM to MD5. The MD5 method uses a custom, less secure challenge-response mechanism as it provides a potential security gap if one gets access to the server's password hash.
To change the password authentication method on the Timescale service console, navigate to the Operations page, and choose the Reset password option. Authenticate using your service account credentials (credentials linked to your service email and not the database credentials). On the Reset service password prompt, key in the same tsdbadmin database user password (or you can change it if you like), then select MD5 as the authentication type and click on the Reset service password button to save these details.
With these two workarounds, your QuickSight connection will be successfully validated when connecting to public or VPC-peered Timescale services.
Quicksight VPC setup
One of the limitations of setting up a Timescale VPC peer with AWS VPC is that the Timescale service is blocked from public internet access. Therefore, this requires one to use a service hosted within the AWS VPC environment.
QuickSight is a service that is heavily used by Timescale customers and among the many support cases, notwithstanding the SCRAM authentication method and SSL issues highlighted above. We aim to summarize the key steps you can follow to configure QuickSight VPC peering with Timescale VPC-peered service.
First, ensure that QuickSight is running in the same region as the AWS VPC you want to connect to.
On the profile menu (step one above), select Manage QuickSight and then Manage VPC connection. This opens the Manage VPC connections page. If you already have existing QuickSight VPCs defined, these VPCs will be displayed on this page. Click on Add VPC connection, and this page is displayed:
VPC connection name: give a meaningful name to your VPC connection.
VPC ID: use your AWS VPC ID.
Subnet ID: use a subnet with a route table linked to your VPC Internet Gateway.
Security group ID: use the security group we created when peering Timescale service. We just need to modify the inbound rules to include QuickSight IP address range CIDR for our region, as listed on this site. In our example, we are using eu-west-1 (Ireland) region, and therefore, the IP address CIDR to use is 52.210.255.224/27. Below is a snapshot of how to modify our security group inbound rule using the HTTPS protocol.
Data source name: this represents a descriptive name of your data source.
Connection type: use it to select the VPC that we created in step two.
Database server: this is the Timescale service hostname that you will get from the Timescale service console.
Service port: this is, by default, 5432 for Timescale VPC-peered services.
Database name: by default, Timescale creates a database named tsdb.
Username: you can use the default tsdbadmin database user or create a new database user for this connection and grant this new user the required object privileges.
Password: this represents the database username password.
If you’ve read this far, you now have the superpower of configuring your VPC peering, enabling secure access to your cloud data without exposing your services to the public internet. With your new superhero invisibility cloak, you should be able to keep your cloud infrastructure as safe and isolated as a self-hosted deployment.
If you found this information helpful, give VPC peering a try! Everything should go smoothly if you follow the steps outlined in this post, but if you still have any questions, reach out! Our team of qualified and experienced Support engineers is here to help.
If you’re not a Timescale customer yet, you can start a free 30-day trial today and explore all that Timescale has to offer. Every Timescale customer has the right to dedicated support at no extra cost—this includes you during your trial!