Skip to content

Commit ed97fa4

Browse files
exAspArkexAspArk
committed
Split docs about Postgres hosting providers
1 parent 3969995 commit ed97fa4

File tree

12 files changed

+434
-334
lines changed

12 files changed

+434
-334
lines changed

‎docs/docs/changelog.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -80,7 +80,7 @@ keywords: ['Bemi Changelog', 'Bemi New Features', 'Postgres Audit Trails', 'Chan
8080
* Allow saving information about a PostgreSQL user who made data changes in the app context
8181
* Integrations
8282
* [Supabase](https://supabase.com/partners/integrations/bemi): new partnership integration
83-
* [GCP Cloud SQL](https://docs.bemi.io/postgresql/source-database#gcp-cloud-sql): describe how to enable logical decoding and connect
83+
* [GCP Cloud SQL](/hosting/gcp): describe how to enable logical decoding and connect
8484
* [PowerSync](https://www.powersync.com): make ingestion worker compatible with a separate PostgreSQL replication
8585

8686
## 2024-04
@@ -110,7 +110,7 @@ keywords: ['Bemi Changelog', 'Bemi New Features', 'Postgres Audit Trails', 'Chan
110110
* Add new helper methods for diffing and sorting changes
111111
* Filter out sensitive logs
112112
* Integrations
113-
* [Render](https://docs.bemi.io/postgresql/source-database#render): create a dedicated integration runbook with their support
114-
* [Neon](https://docs.bemi.io/postgresql/source-database#neon): describe how to integrate by using their new Logical Replication beta feature
115-
* [Supabase](https://docs.bemi.io/postgresql/source-database#supabase): improve IPv6 error handling
113+
* [Render](/hosting/render): create a dedicated integration runbook with their support
114+
* [Neon](/hosting/neon): describe how to integrate by using their new Logical Replication beta feature
115+
* [Supabase](/hosting/supabase): improve IPv6 error handling
116116
* [AWS DMS](https://aws.amazon.com/dms/): make ingestion worker compatible with DMS' logical replication decoding

‎docs/docs/home.md

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -50,11 +50,12 @@ This allows automatically enhancing low-level database changes with application-
5050

5151
## Supported ORMs
5252

53-
#### Node.js
53+
#### JavaScript/TypeScript
5454

5555
* **[Prisma](/orms/prisma)**
56-
* **[Supabase JS](/orms/supabase-js)**
5756
* **[TypeORM](/orms/typeorm)**
57+
* **[Supabase JS](/orms/supabase-js)**
58+
* **[MikroORM](/orms/mikro-orm)**
5859

5960
#### Ruby
6061

‎docs/docs/hosting/aws.md

Lines changed: 124 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,124 @@
1+
---
2+
title: Bemi and AWS RDS Integration - Audit Trail and Data Tracking
3+
sidebar_label: AWS RDS
4+
hide_title: true
5+
description: Learn how to configure your AWS RDS database with Bemi for real-time data tracking using Change Data Capture (CDC). Includes detailed setup instructions for connections and WAL levels.
6+
keywords: [Bemi, AWS RDS, PostgreSQL, Change Data Capture, real-time data tracking, audit trail, WAL, logical replication]
7+
image: 'img/social-card.png'
8+
---
9+
10+
# AWS RDS
11+
12+
## WAL level
13+
14+
At a high level, these are the steps necessary to update the WAL level from `replica` to `logical`
15+
16+
1. Create an RDS parameter group if it doesn’t exist
17+
2. Update `rds.logical_replication` parameter from 0 to 1
18+
3. Apply the parameter group to your RDS instance and restart it
19+
20+
Now let's break down these steps.
21+
22+
Create an RDS parameter group if it doesn’t exist by choose the group family depending on your PostgreSQL version and specifying any name and description:
23+
24+
![](/img/wal_level-aws-1.png)
25+
26+
Edit the created parameter group:
27+
28+
![](/img/wal_level-aws-2.png)
29+
30+
Find and change the `rds.logical_replication` parameter from 0 to 1:
31+
32+
![](/img/wal_level-aws-3.png)
33+
34+
Find and modify your RDS instance by using the parameter group:
35+
36+
![](/img/wal_level-aws-4.png)
37+
38+
Apply the modification by restarting your RDS instance:
39+
40+
![](/img/wal_level-aws-5.png)
41+
42+
If you have a Multi-AZ database cluster and you used a cluster parameter group, you will need to explicitly Reboot the Writer instance (it may take ~ 2 seconds if there is not a lot of activity).
43+
The Reader endpoint will continue to be available without downtime.
44+
45+
![](/img/wal_level-writer-reboot.png)
46+
47+
See the [AWS RDS user guides](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_WorkingWithParamGroups.html) to learn more about parameter groups.
48+
49+
## Connection
50+
51+
You can specify the same regular database credentials you use to connect to PostgreSQL from your code.
52+
And that's it, everything should just work!
53+
54+
## Read-only credentials
55+
56+
Alternatively, you can manually create read-only PostgreSQL database credentials to connect to the primary instance's WAL.
57+
At a high level, you need to run these commands that are safe to execute without any downtime or performance issues:
58+
59+
* `CREATE ROLE` creates a new read-only user for Bemi to read database changes.
60+
* `CREATE PUBLICATION` creates a "channel" that we'll subscribe to and track changes in real-time.
61+
* `REPLICA IDENTITY FULL` enhances records stored in WAL to record the previous state (“before”) in addition to the tracked by default new state (“after”).
62+
63+
```sql
64+
-- Create read-only user
65+
CREATE ROLE [username] WITH LOGIN NOSUPERUSER NOCREATEDB NOCREATEROLE NOREPLICATION PASSWORD '[password]';
66+
-- Grant RDS replication permission
67+
GRANT rds_replication TO [username];
68+
-- Grant SELECT access to existing tables for selective tracking
69+
GRANT SELECT ON ALL TABLES IN SCHEMA public TO [username];
70+
-- Grant SELECT access to new tables created in the future for selective tracking
71+
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON TABLES TO [username];
72+
73+
-- Create "bemi" PUBLICATION to enable logical replication
74+
CREATE PUBLICATION bemi FOR ALL TABLES;
75+
76+
-- Create a procedure to set REPLICA IDENTITY FULL for tables to track the "before" state on DB row changes
77+
CREATE OR REPLACE PROCEDURE _bemi_set_replica_identity() AS $$ DECLARE current_tablename TEXT;
78+
BEGIN
79+
FOR current_tablename IN SELECT tablename FROM pg_tables LEFT JOIN pg_class ON relname = tablename WHERE schemaname = 'public' AND relreplident != 'f' LOOP
80+
EXECUTE format('ALTER TABLE %I REPLICA IDENTITY FULL', current_tablename);
81+
END LOOP;
82+
END $$ LANGUAGE plpgsql;
83+
-- Call the created procedure
84+
CALL _bemi_set_replica_identity();
85+
-- Create a trigger function that calls the created procedure
86+
CREATE OR REPLACE FUNCTION _bemi_set_replica_identity_func() RETURNS event_trigger AS $$
87+
BEGIN CALL _bemi_set_replica_identity(); END $$ LANGUAGE plpgsql;
88+
-- Create a trigger to set REPLICA IDENTITY FULL for all new created tables
89+
CREATE EVENT TRIGGER _bemi_set_replica_identity_trigger ON ddl_command_end WHEN TAG IN ('CREATE TABLE')
90+
EXECUTE FUNCTION _bemi_set_replica_identity_func();
91+
```
92+
93+
## Read-only credentials with manually managed permissions for each table
94+
95+
Run the following queries if you want to isolate read access only to logical replication for certain tables and manage permissions manually
96+
instead of relying on our robust built-in selective tracking manageable through our UI.
97+
98+
```sql
99+
-- Create read-only user
100+
CREATE ROLE [username] WITH LOGIN NOSUPERUSER NOCREATEDB NOCREATEROLE NOREPLICATION PASSWORD '[password]';
101+
-- Grant replication permission to allow using replication slots
102+
GRANT rds_replication TO [username];
103+
104+
-- Create "bemi" PUBLICATION to enable logical replication for selected tables
105+
CREATE PUBLICATION bemi FOR TABLE [table1], [table2];
106+
107+
-- Set REPLICA IDENTITY FULL for tables to track the "before" state on DB row changes
108+
ALTER TABLE [table1] REPLICA IDENTITY FULL;
109+
ALTER TABLE [table2] REPLICA IDENTITY FULL;
110+
```
111+
112+
To enable data change tracking for a new table:
113+
114+
```sql
115+
ALTER PUBLICATION bemi ADD TABLE [table3];
116+
ALTER TABLE [table3] REPLICA IDENTITY FULL;
117+
```
118+
119+
To stop data change tracking for a table:
120+
121+
```sql
122+
ALTER PUBLICATION bemi DROP TABLE [table3];
123+
ALTER TABLE [table3] REPLICA IDENTITY DEFAULT;
124+
```

‎docs/docs/hosting/digitalocean.md

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
---
2+
title: Bemi and DigitalOcean Integration - Audit Trail and Data Tracking
3+
sidebar_label: DigitalOcean
4+
hide_title: true
5+
description: Learn how to configure your DigitalOcean database with Bemi for real-time data tracking using Change Data Capture (CDC). Includes detailed setup instructions for connections and WAL levels.
6+
keywords: [Bemi, DigitalOcean, PostgreSQL, Change Data Capture, real-time data tracking, audit trail, WAL, logical replication]
7+
image: 'img/social-card.png'
8+
---
9+
10+
# DigitalOcean
11+
12+
Navigate to the [DigitalOcean databases](https://cloud.digitalocean.com/databases) tab and specify your database credentials, which can be found in the Connection details:
13+
14+
![](/img/perm-digitalocean.png)

‎docs/docs/hosting/gcp.md

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
---
2+
title: Bemi and GCP Cloud SQL Integration - Audit Trail and Data Tracking
3+
sidebar_label: GCP Cloud SQL
4+
hide_title: true
5+
description: Learn how to configure your GCP Cloud SQL database with Bemi for real-time data tracking using Change Data Capture (CDC). Includes detailed setup instructions for connections and WAL levels.
6+
keywords: [Bemi, GCP Cloud SQL, PostgreSQL, Change Data Capture, real-time data tracking, audit trail, WAL, logical replication]
7+
image: 'img/social-card.png'
8+
---
9+
10+
# GCP Cloud SQL
11+
12+
## WAL level
13+
14+
Logical replication is turned off by default. To turn it on, you have to update the [cloud flag](https://cloud.google.com/sql/docs/postgres/replication/configure-logical-replication#configure-your-postgresql-instance): `cloudsql.logical_decoding` = `on`. This will need a restart of your instance before `SHOW wal_level;` returns `logical`.
15+
16+
## Connection
17+
18+
Run the below command and then you can connect with the same credentials on the Bemi dashboard!
19+
```sql
20+
-- Grant replication permission to allow using replication slots
21+
ALTER USER [user] WITH REPLICATION;
22+
```

‎docs/docs/hosting/neon.md

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
---
2+
title: Bemi and Neon Integration - Audit Trail and Data Tracking
3+
sidebar_label: Neon
4+
hide_title: true
5+
description: Learn how to configure your Neon database with Bemi for real-time data tracking using Change Data Capture (CDC). Includes detailed setup instructions for connections and WAL levels.
6+
keywords: [Bemi, Neon, PostgreSQL, Change Data Capture, real-time data tracking, audit trail, WAL, logical replication]
7+
image: 'img/social-card.png'
8+
---
9+
10+
# Neon
11+
12+
## WAL level
13+
14+
To enable logical replication in [Neon](https://neon.tech/):
15+
16+
1. Select your project in the Neon Console.
17+
2. On the Neon **Dashboard**, select **Settings**.
18+
3. Select **Beta**.
19+
4. Click **Enable** to enable logical replication. This will set the Postgres `wal_level` setting to `logical`.
20+
21+
## Connection
22+
23+
To connect a [Neon](https://neon.tech/docs/guides/bemi) Postgres database, specify your database credentials, which can be found on your Neon project's dashboard:
24+
25+
**Note:** Please use the `Host` name without enabling the "Pooled connection" option.
26+
27+
![](/img/perm-neon.png)
28+
29+
And that's it, everything should just work!
30+
31+
For a detailed setup guide, see [Create an automatic audit trail with Bemi and Neon](https://neon.tech/docs/guides/bemi), in the _Neon documentation_.
32+
33+
## Read-only credentials
34+
35+
Alternatively, you can manually create read-only PostgreSQL database credentials to connect to the primary instance's WAL.
36+
At a high level, you need to run these commands that are safe to execute without any downtime or performance issues:
37+
38+
* `CREATE ROLE` creates a new read-only user for Bemi to read database changes.
39+
* `CREATE PUBLICATION` creates a "channel" that we'll subscribe to and track changes in real-time.
40+
* `REPLICA IDENTITY FULL` enhances records stored in WAL to record the previous state (“before”) in addition to the tracked by default new state (“after”).
41+
42+
```sql
43+
-- Create read-only user with REPLICATION permission
44+
CREATE ROLE [username] WITH LOGIN NOSUPERUSER NOCREATEDB NOCREATEROLE REPLICATION PASSWORD '[password]';
45+
-- Grant SELECT access to tables for selective tracking
46+
GRANT SELECT ON ALL TABLES IN SCHEMA public TO [username];
47+
-- Grant SELECT access to new tables created in the future for selective tracking
48+
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON TABLES TO [username];
49+
50+
-- Create "bemi" PUBLICATION to enable logical replication
51+
CREATE PUBLICATION bemi FOR ALL TABLES;
52+
53+
-- Create a procedure to set REPLICA IDENTITY FULL for tables to track the "before" state on DB row changes
54+
CREATE OR REPLACE PROCEDURE _bemi_set_replica_identity() AS $$ DECLARE current_tablename TEXT;
55+
BEGIN
56+
FOR current_tablename IN SELECT tablename FROM pg_tables LEFT JOIN pg_class ON relname = tablename WHERE schemaname = 'public' AND relreplident != 'f' LOOP
57+
EXECUTE format('ALTER TABLE %I REPLICA IDENTITY FULL', current_tablename);
58+
END LOOP;
59+
END $$ LANGUAGE plpgsql;
60+
-- Call the created procedure
61+
CALL _bemi_set_replica_identity();
62+
```

‎docs/docs/hosting/render.md

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
---
2+
title: Bemi and Render Integration - Audit Trail and Data Tracking
3+
sidebar_label: Render
4+
hide_title: true
5+
description: Learn how to configure your Render database with Bemi for real-time data tracking using Change Data Capture (CDC). Includes detailed setup instructions for connections and WAL levels.
6+
keywords: [Bemi, Render, PostgreSQL, Change Data Capture, real-time data tracking, audit trail, WAL, logical replication]
7+
image: 'img/social-card.png'
8+
---
9+
10+
# Render
11+
12+
## WAL level
13+
14+
Please submit a Render support request, and they'll run a special runbook to set up Bemi:
15+
16+
> In a few words, what can we help you with?
17+
18+
```
19+
Configure database for Bemi
20+
```
21+
22+
> Describe the issue in more detail.
23+
24+
```
25+
- Set "wal_level" to "logical"
26+
- Add "REPLICATION" permission to the database user
27+
- Create "bemi" publication
28+
```
29+
30+
## Connection
31+
32+
To connect a [Render](https://render.com/) database, specify your database credentials that can be found on the Render dashboard:
33+
34+
* Please use the full `Host` name that ends with `.render.com` from the External Database URL section
35+
36+
![](/img/perm-render.png)
37+
38+
*Note that you can't create new credentials with `REPLICATION` permissions in Render.*

‎docs/docs/hosting/self-managed.md

Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
---
2+
title: Bemi and Self-Managed Integration - Audit Trail and Data Tracking
3+
sidebar_label: Self-Managed
4+
hide_title: true
5+
description: Learn how to configure your self-managed database with Bemi for real-time data tracking using Change Data Capture (CDC). Includes detailed setup instructions for connections and WAL levels.
6+
keywords: [Bemi, Self-Managed, PostgreSQL, Change Data Capture, real-time data tracking, audit trail, WAL, logical replication]
7+
image: 'img/social-card.png'
8+
---
9+
10+
# Self-managed PostgreSQL
11+
12+
## WAL level
13+
14+
Run the following SQL command to change the WAL level from `replica` to `logical` and restart your database:
15+
16+
```sql
17+
ALTER SYSTEM SET wal_level = logical;
18+
```
19+
20+
If you have issues in other PostgreSQL hosting environments, please [contact us](https://bemi.io/contact-us), and we will send you detailed instructions on how to set it up.
21+
22+
## Connection
23+
24+
You can specify the same regular database credentials you use to connect to PostgreSQL from your code.
25+
And that's it, everything should just work!
26+
27+
## Read-only credentials
28+
29+
Alternatively, you can manually create read-only PostgreSQL database credentials to connect to the primary instance's WAL.
30+
At a high level, you need to run these commands that are safe to execute without any downtime or performance issues:
31+
32+
* `CREATE ROLE` creates a new read-only user for Bemi to read database changes.
33+
* `CREATE PUBLICATION` creates a "channel" that we'll subscribe to and track changes in real-time.
34+
* `REPLICA IDENTITY FULL` enhances records stored in WAL to record the previous state (“before”) in addition to the tracked by default new state (“after”).
35+
36+
```sql
37+
-- Create read-only user with REPLICATION permission
38+
CREATE ROLE [username] WITH LOGIN NOSUPERUSER NOCREATEDB NOCREATEROLE REPLICATION PASSWORD '[password]';
39+
-- Grant SELECT access to tables for selective tracking
40+
GRANT SELECT ON ALL TABLES IN SCHEMA public TO [username];
41+
-- Grant SELECT access to new tables created in the future for selective tracking
42+
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON TABLES TO [username];
43+
44+
-- Create "bemi" PUBLICATION to enable logical replication
45+
CREATE PUBLICATION bemi FOR ALL TABLES;
46+
47+
-- Create a procedure to set REPLICA IDENTITY FULL for tables to track the "before" state on DB row changes
48+
CREATE OR REPLACE PROCEDURE _bemi_set_replica_identity() AS $$ DECLARE current_tablename TEXT;
49+
BEGIN
50+
FOR current_tablename IN SELECT tablename FROM pg_tables LEFT JOIN pg_class ON relname = tablename WHERE schemaname = 'public' AND relreplident != 'f' LOOP
51+
EXECUTE format('ALTER TABLE %I REPLICA IDENTITY FULL', current_tablename);
52+
END LOOP;
53+
END $$ LANGUAGE plpgsql;
54+
-- Call the created procedure
55+
CALL _bemi_set_replica_identity();
56+
-- Create a trigger function that calls the created procedure
57+
CREATE OR REPLACE FUNCTION _bemi_set_replica_identity_func() RETURNS event_trigger AS $$
58+
BEGIN CALL _bemi_set_replica_identity(); END $$ LANGUAGE plpgsql;
59+
-- Create a trigger to set REPLICA IDENTITY FULL for all new created tables
60+
CREATE EVENT TRIGGER _bemi_set_replica_identity_trigger ON ddl_command_end WHEN TAG IN ('CREATE TABLE')
61+
EXECUTE FUNCTION _bemi_set_replica_identity_func();
62+
```
63+
64+
## Read-only credentials with manually managed permissions for each table
65+
66+
Run the following queries if you want to isolate read access only to logical replication for certain tables and manage permissions manually
67+
instead of relying on our robust built-in selective tracking manageable through our UI.
68+
69+
```sql
70+
-- Create read-only user with REPLICATION permission
71+
CREATE ROLE [username] WITH LOGIN NOSUPERUSER NOCREATEDB NOCREATEROLE REPLICATION PASSWORD '[password]';
72+
73+
-- Create "bemi" PUBLICATION to enable logical replication for selected tables
74+
CREATE PUBLICATION bemi FOR TABLE [table1], [table2];
75+
76+
-- Set REPLICA IDENTITY FULL for tables to track the "before" state on DB row changes
77+
ALTER TABLE [table1] REPLICA IDENTITY FULL;
78+
ALTER TABLE [table2] REPLICA IDENTITY FULL;
79+
```
80+
81+
To enable data change tracking for a new table:
82+
83+
```sql
84+
ALTER PUBLICATION bemi ADD TABLE [table3];
85+
ALTER TABLE [table3] REPLICA IDENTITY FULL;
86+
```
87+
88+
To stop data change tracking for a table:
89+
90+
```sql
91+
ALTER PUBLICATION bemi DROP TABLE [table3];
92+
ALTER TABLE [table3] REPLICA IDENTITY DEFAULT;
93+
```

0 commit comments

Comments
 (0)