Convoy supports exporting events to any S3 compatible cold storage.

Note:

  • The retention policies functionality is available only for licensed users.
  • This feature is currently in beta and disabled by default. To enable it, use the environment variable CONVOY_ENABLE_FEATURE_FLAG=retention-policy or the CLI flag --enable-feature-flag=retention-policy.

​
Configuring Retention

​
JSON Configuration

You can configure read replicas by adding the storage_policy object to your database configuration JSON.

KeyDescription
typeStorage type, either “s3” or “on_prem”
s3.prefixPrefix for S3 storage path
s3.bucketName of the S3 bucket
s3.access_keyAWS access key for S3 authentication
s3.secret_keyAWS secret key for S3 authentication
s3.regionAWS region where the S3 bucket is located
s3.session_tokenTemporary session token for AWS authentication (optional)
s3.endpointCustom endpoint URL for S3-compatible storage
on_prem.pathFile system path for on-premises storage location
using storage policy via JSON configuration
{
	"storage_policy": {
		"type": "s3 | on_prem",
		"s3": {
			"prefix": "<insert-s3-prefix>",
			"bucket": "<insert-s3-bucket>",
			"access_key": "<insert-s3-access-key>",
			"secret_key": "<insert-s3-secret-key>",
			"region": "<insert-s3-region>",
			"session_token": "<insert-s3-session-token>",
			"endpoint": "<insert-s3-endpoint>"
		},
		"on_prem": {
			"path": "<insert-on-prem-path>"
		}
	}
}

​
Environment Variables Configuration

Alternatively, you can supply environment variables to configure the storage policy.

Environment VariableDescription
CONVOY_STORAGE_POLICY_TYPEStorage type, either “s3” or “on_prem”
CONVOY_STORAGE_AWS_PREFIXPrefix for S3 storage path
CONVOY_STORAGE_AWS_BUCKETName of the S3 bucket
CONVOY_STORAGE_AWS_ACCESS_KEYAWS access key for S3 authentication
CONVOY_STORAGE_AWS_SECRET_KEYAWS secret key for S3 authentication
CONVOY_STORAGE_AWS_REGIONAWS region where the S3 bucket is located
CONVOY_STORAGE_AWS_SESSION_TOKENTemporary session token for AWS authentication (optional)
CONVOY_STORAGE_AWS_ENDPOINTCustom endpoint URL for S3-compatible storage
CONVOY_STORAGE_PREM_PATHFile system path for on-premises storage location
using storage policy via environment variable configuration
CONVOY_STORAGE_POLICY_TYPE=''
CONVOY_STORAGE_AWS_PREFIX=''
CONVOY_STORAGE_AWS_BUCKET=''
CONVOY_STORAGE_AWS_ACCESS_KEY=''
CONVOY_STORAGE_AWS_SECRET_KEY=''
CONVOY_STORAGE_AWS_REGION=''
CONVOY_STORAGE_AWS_SESSION_TOKEN=''
CONVOY_STORAGE_AWS_ENDPOINT=''
CONVOY_STORAGE_PREM_PATH=''

​
Retention Using Postgres Partitions

We introduced a new way to manage retention using Postgres-native partitions using go_partman with each table is partitioned by day. Prior to Convoy v25.1.1 the job that ran the retention policy carried out two tasks:

  • Running SELECT queries and uploading webhook event data to S3.
  • Running DELETE queries on the tables.

​
Problems with the old system

​
Concurrency

The problem with the old approach was that both the backup and deletion queries ran in the same job, and any of these tasks could fail for a number of reasons:

  • The backup action could fail because of a network issue, making the retention query not run.
  • The backup action could succeed, but the retention query would time out leading to duplication data the next time the backup action ran.

This approach also had some other issues:

  • There also wasn’t any visibility into what error occurred or why the backup did not work.
  • Running DELETE queries on moderately large Postgres databases triggers AUTOVACUUM and causes disk pressure.

​
Retention Granularity

The previous system also didn’t make it possible to specify different retention periods for different projects. Now it is possible to specify different retention periods per project, and this can be managed from the project settings page.

​
Proposed Solution

With the new system, the backup and delete operations run in different jobs that are triggered at different times.

​
Backups

The backup job backs up events sent on a particular day at the end of that day.

​
Deletion

The retention (deletion) job deletes events sent on a particular day n-days after it was sent; where n is the retention policy duration.

​
Switching to Partition for Retention

The tables that are partitioned are:

  • convoy.delivery_attempts
  • convoy.event_deliveries
  • convoy.events
  • convoy.events_search

Before partitioning (or un-partitioning), the tables be sure to back up the database, or test it on a fresh database with all the organization and project configuration copied over.

​
Partitioning the tables

To switch to managing retention using Postgres partitions, you need to run:

partition all tables
# The retention policy feature flag must be enabled
convoy utils partition --enable-feature-flag=retention-policy
CONVOY_ENABLE_FEATURE_FLAG=retention-policy convoy utils partition

If you are running convoy in a docker container, you can execute this command using docker exec

partitioning using docker exec
docker exec -it convoy-staging /cmd util partition

When the above command is run, it attempts to partition all the tables at the same time. This operation can take a while (10 to 60 minutes) if the tables are moderately large, so you would need to partition them one after the other like this:

partition tables one after the other
convoy utils partition delivery_attempts
convoy utils partition event_deliveries
convoy utils partition events
convoy utils partition events_search

​
Un-partitioning the tables

To switch back from managing retention using Postgres partitions, you can un-partition all of them at the same time by running:

unpartition all tables
convoy utils unpartition

Or you can un-partition them one after the other by running:

unpartition tables one after the other
convoy utils unpartition delivery_attempts
convoy utils unpartition event_deliveries
convoy utils unpartition events
convoy utils unpartition events_search

Was this page helpful?

Read ReplicasConvoy