Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -69,17 +69,15 @@
CREATE USER clickpipes_user PASSWORD 'some-password';
```

2. Grant the dedicated user permissions on the schema(s) you want to replicate.
2. Grant schema-level, read-only access to the user you created in the previous step. The following example shows permissions for the `public` schema. Repeat these commands for each schema containing tables you want to replicate:

```sql
GRANT USAGE ON SCHEMA "public" TO clickpipes_user;
GRANT SELECT ON ALL TABLES IN SCHEMA "public" TO clickpipes_user;
ALTER DEFAULT PRIVILEGES IN SCHEMA "public" GRANT SELECT ON TABLES TO clickpipes_user;
```

The example above shows permissions for the `public` schema. Repeat the sequence of commands for each schema you want to replicate using ClickPipes.

3. Grant the dedicated user permissions to manage replication:
3. Grant replication privileges to the user:

```sql
ALTER ROLE clickpipes_user REPLICATION;
Expand All @@ -88,7 +86,7 @@
4. Create a [publication](https://www.postgresql.org/docs/current/logical-replication-publication.html) with the tables you want to replicate. We strongly recommend only including the tables you need in the publication to avoid performance overhead.

:::warning
All tables included in the publication must either have a **primary key** defined _or_ have its **replica identity** configured to `FULL`. See the [Postgres FAQs](../faq.md#how-should-i-scope-my-publications-when-setting-up-replication) for guidance on scoping.
Any table included in the publication must either have a **primary key** defined _or_ have its **replica identity** configured to `FULL`. See the [Postgres FAQs](../faq.md#how-should-i-scope-my-publications-when-setting-up-replication) for guidance on scoping.
:::

- To create a publication for specific tables:
Expand All @@ -105,10 +103,10 @@

The `clickpipes` publication will contain the set of change events generated from the specified tables, and will later be used to ingest the replication stream.

## Configure network security {#configure-network-security}
## Configure network access {#configure-network-access}

:::note
ClickPipes does not support Private Service Connect (PSC) connections. If you do not allow public access to your AlloyDB instance, you can [use an SSH tunnel](#configure-network-security) to connect securely. PSC will be supported in the future.
ClickPipes does not support Private Service Connect (PSC) connections. If you do not allow public access to your AlloyDB instance, you can [use an SSH tunnel](#configure-network-access) to connect securely. PSC will be supported in the future.

Check notice on line 109 in docs/integrations/data-ingestion/clickpipes/postgres/source/alloydb.md

View workflow job for this annotation

GitHub Actions / vale

ClickHouse.Contractions

Suggestion: Use 'don't' instead of 'do not'.

Check notice on line 109 in docs/integrations/data-ingestion/clickpipes/postgres/source/alloydb.md

View workflow job for this annotation

GitHub Actions / vale

ClickHouse.Contractions

Suggestion: Use 'doesn't' instead of 'does not'.
:::

Next, you must allow connections to your AlloyDB instance from ClickPipes.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -74,25 +74,39 @@ Connect to your Aurora PostgreSQL writer instance as an admin user and execute t
CREATE USER clickpipes_user PASSWORD 'some-password';
```

2. Grant schema permissions. The following example shows permissions for the `public` schema. Repeat these commands for each schema you want to replicate:
2. Grant schema-level, read-only access to the user you created in the previous step. The following example shows permissions for the `public` schema. Repeat these commands for each schema containing tables you want to replicate:

```sql
GRANT USAGE ON SCHEMA "public" TO clickpipes_user;
GRANT SELECT ON ALL TABLES IN SCHEMA "public" TO clickpipes_user;
ALTER DEFAULT PRIVILEGES IN SCHEMA "public" GRANT SELECT ON TABLES TO clickpipes_user;
```

3. Grant replication privileges:
3. Grant replication privileges to the user:

```sql
GRANT rds_replication TO clickpipes_user;
```

4. Create a publication for replication:
4. Create a [publication](https://www.postgresql.org/docs/current/logical-replication-publication.html) with the tables you want to replicate. We strongly recommend only including the tables you need in the publication to avoid performance overhead.

```sql
CREATE PUBLICATION clickpipes_publication FOR ALL TABLES;
```
:::warning
Any table included in the publication must either have a **primary key** defined _or_ have its **replica identity** configured to `FULL`. See the [Postgres FAQs](../faq.md#how-should-i-scope-my-publications-when-setting-up-replication) for guidance on scoping.
:::

- To create a publication for specific tables:

```sql
CREATE PUBLICATION clickpipes FOR TABLE table_to_replicate, table_to_replicate2;
```

- To create a publication for all tables in a specific schema:

```sql
CREATE PUBLICATION clickpipes FOR TABLES IN SCHEMA "public";
```

The `clickpipes` publication will contain the set of change events generated from the specified tables, and will later be used to ingest the replication stream.

## Configure network access {#configure-network-access}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -37,31 +37,45 @@ ClickPipes supports Postgres version 12 and later.

Connect to your Azure Flexible Server Postgres through the admin user and run the below commands:

1. Create a Postgres user for exclusively ClickPipes.
1. Create a dedicated user for ClickPipes.

```sql
CREATE USER clickpipes_user PASSWORD 'some-password';
```

2. Provide read-only access to the schema from which you are replicating tables to the `clickpipes_user`. Below example shows setting up permissions for the `public` schema. If you want to grant access to multiple schemas, you can run these three commands for each schema.
2. Grant schema-level, read-only access to the user you created in the previous step. The following example shows permissions for the `public` schema. Repeat these commands for each schema containing tables you want to replicate:

```sql
GRANT USAGE ON SCHEMA "public" TO clickpipes_user;
GRANT SELECT ON ALL TABLES IN SCHEMA "public" TO clickpipes_user;
ALTER DEFAULT PRIVILEGES IN SCHEMA "public" GRANT SELECT ON TABLES TO clickpipes_user;
```

3. Grant replication access to this user:
3. Grant replication privileges to the user:

```sql
ALTER ROLE clickpipes_user REPLICATION;
```

4. Create publication that you'll be using for creating the MIRROR (replication) in future.
4. Create a [publication](https://www.postgresql.org/docs/current/logical-replication-publication.html) with the tables you want to replicate. We strongly recommend only including the tables you need in the publication to avoid performance overhead.

```sql
CREATE PUBLICATION clickpipes_publication FOR ALL TABLES;
```
:::warning
Any table included in the publication must either have a **primary key** defined _or_ have its **replica identity** configured to `FULL`. See the [Postgres FAQs](../faq.md#how-should-i-scope-my-publications-when-setting-up-replication) for guidance on scoping.
:::

- To create a publication for specific tables:

```sql
CREATE PUBLICATION clickpipes FOR TABLE table_to_replicate, table_to_replicate2;
```

- To create a publication for all tables in a specific schema:

```sql
CREATE PUBLICATION clickpipes FOR TABLES IN SCHEMA "public";
```

The `clickpipes` publication will contain the set of change events generated from the specified tables, and will later be used to ingest the replication stream.

5. Set `wal_sender_timeout` to 0 for `clickpipes_user`

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,31 +29,45 @@ SHOW max_replication_slots; -- should be 10

Connect to your Crunchy Bridge Postgres through the `postgres` user and run the below commands:

1. Create a Postgres user exclusively for ClickPipes.
1. Create a dedicated user for ClickPipes:

```sql
CREATE USER clickpipes_user PASSWORD 'some-password';
```

2. Grant read-only access to the schema from which you are replicating tables to `clickpipes_user`. Below example shows granting permissions for the `public` schema. If you want to grant access to multiple schemas, you can run these three commands for each schema.
2. Grant schema-level, read-only access to the user you created in the previous step. The following example shows permissions for the `public` schema. Repeat these commands for each schema containing tables you want to replicate:

```sql
GRANT USAGE ON SCHEMA "public" TO clickpipes_user;
GRANT SELECT ON ALL TABLES IN SCHEMA "public" TO clickpipes_user;
ALTER DEFAULT PRIVILEGES IN SCHEMA "public" GRANT SELECT ON TABLES TO clickpipes_user;
```

3. Grant replication access to this user:
3. Grant replication privileges to the user:

```sql
ALTER ROLE clickpipes_user REPLICATION;
```

4. Create publication that you'll be using for creating the MIRROR (replication) in future.
4. Create a [publication](https://www.postgresql.org/docs/current/logical-replication-publication.html) with the tables you want to replicate. We strongly recommend only including the tables you need in the publication to avoid performance overhead.

```sql
CREATE PUBLICATION clickpipes_publication FOR ALL TABLES;
```
:::warning
Any table included in the publication must either have a **primary key** defined _or_ have its **replica identity** configured to `FULL`. See the [Postgres FAQs](../faq.md#how-should-i-scope-my-publications-when-setting-up-replication) for guidance on scoping.
:::

- To create a publication for specific tables:

```sql
CREATE PUBLICATION clickpipes FOR TABLE table_to_replicate, table_to_replicate2;
```

- To create a publication for all tables in a specific schema:

```sql
CREATE PUBLICATION clickpipes FOR TABLES IN SCHEMA "public";
```

The `clickpipes` publication will contain the set of change events generated from the specified tables, and will later be used to ingest the replication stream.

## Safe list ClickPipes IPs {#safe-list-clickpipes-ips}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -54,27 +54,47 @@ ClickPipes supports Postgres version 12 and later.

## Creating a user with permissions and publication {#creating-a-user-with-permissions-and-publication}

Let's create a new user for ClickPipes with the necessary permissions suitable for CDC,
and also create a publication that we'll use for replication.
Connect to your Postgres instance as an admin user and execute the following commands:

For this, you can connect to your Postgres instance and run the following SQL commands:
```sql
CREATE USER clickpipes_user PASSWORD 'clickpipes_password';
GRANT USAGE ON SCHEMA "public" TO clickpipes_user;
GRANT SELECT ON ALL TABLES IN SCHEMA "public" TO clickpipes_user;
ALTER DEFAULT PRIVILEGES IN SCHEMA "public" GRANT SELECT ON TABLES TO clickpipes_user;
1. Create a dedicated user for ClickPipes:

-- Give replication permission to the USER
ALTER USER clickpipes_user REPLICATION;
```sql
CREATE USER clickpipes_user PASSWORD 'some-password';
```

-- Create a publication. We will use this when creating the pipe
CREATE PUBLICATION clickpipes_publication FOR ALL TABLES;
```
:::note
2. Grant schema-level, read-only access to the user you created in the previous step. The following example shows permissions for the `public` schema. Repeat these commands for each schema containing tables you want to replicate:

```sql
GRANT USAGE ON SCHEMA "public" TO clickpipes_user;
GRANT SELECT ON ALL TABLES IN SCHEMA "public" TO clickpipes_user;
ALTER DEFAULT PRIVILEGES IN SCHEMA "public" GRANT SELECT ON TABLES TO clickpipes_user;
```

Make sure to replace `clickpipes_user` and `clickpipes_password` with your desired username and password.
3. Grant replication privileges to the user:

:::
```sql
ALTER ROLE clickpipes_user REPLICATION;
```

4. Create a [publication](https://www.postgresql.org/docs/current/logical-replication-publication.html) with the tables you want to replicate. We strongly recommend only including the tables you need in the publication to avoid performance overhead.

:::warning
Any table included in the publication must either have a **primary key** defined _or_ have its **replica identity** configured to `FULL`. See the [Postgres FAQs](../faq.md#how-should-i-scope-my-publications-when-setting-up-replication) for guidance on scoping.
:::

- To create a publication for specific tables:

```sql
CREATE PUBLICATION clickpipes FOR TABLE table_to_replicate, table_to_replicate2;
```

- To create a publication for all tables in a specific schema:

```sql
CREATE PUBLICATION clickpipes FOR TABLES IN SCHEMA "public";
```

The `clickpipes` publication will contain the set of change events generated from the specified tables, and will later be used to ingest the replication stream.

## Enabling connections in pg_hba.conf to the ClickPipes User {#enabling-connections-in-pg_hbaconf-to-the-clickpipes-user}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -47,31 +47,45 @@ Anything on or after Postgres 12

Connect to your Cloud SQL Postgres through the admin user and run the below commands:

1. Create a Postgres user for exclusively ClickPipes.
1. Create a dedicated user for ClickPipes:

```sql
CREATE USER clickpipes_user PASSWORD 'some-password';
```

2. Provide read-only access to the schema from which you are replicating tables to the `clickpipes_user`. Below example shows setting up permissions for the `public` schema. If you want to grant access to multiple schemas, you can run these three commands for each schema.
2. Grant schema-level, read-only access to the user you created in the previous step. The following example shows permissions for the `public` schema. Repeat these commands for each schema containing tables you want to replicate:

```sql
GRANT USAGE ON SCHEMA "public" TO clickpipes_user;
GRANT SELECT ON ALL TABLES IN SCHEMA "public" TO clickpipes_user;
ALTER DEFAULT PRIVILEGES IN SCHEMA "public" GRANT SELECT ON TABLES TO clickpipes_user;
```

3. Grant replication access to this user:
3. Grant replication privileges to the user:

```sql
ALTER ROLE clickpipes_user REPLICATION;
```

4. Create publication that you'll be using for creating the MIRROR (replication) in future.
4. Create a [publication](https://www.postgresql.org/docs/current/logical-replication-publication.html) with the tables you want to replicate. We strongly recommend only including the tables you need in the publication to avoid performance overhead.

```sql
CREATE PUBLICATION clickpipes_publication FOR ALL TABLES;
```
:::warning
Any table included in the publication must either have a **primary key** defined _or_ have its **replica identity** configured to `FULL`. See the [Postgres FAQs](../faq.md#how-should-i-scope-my-publications-when-setting-up-replication) for guidance on scoping.
:::

- To create a publication for specific tables:

```sql
CREATE PUBLICATION clickpipes FOR TABLE table_to_replicate, table_to_replicate2;
```

- To create a publication for all tables in a specific schema:

```sql
CREATE PUBLICATION clickpipes FOR TABLES IN SCHEMA "public";
```

The `clickpipes` publication will contain the set of change events generated from the specified tables, and will later be used to ingest the replication stream.

[//]: # (TODO Add SSH Tunneling)

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,6 @@ doc_type: 'guide'
keywords: ['clickpipes', 'postgresql', 'cdc', 'data ingestion', 'real-time sync']
---

import neon_commands from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/source/setup/neon-postgres/neon-commands.png'
import neon_enable_replication from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/source/setup/neon-postgres/neon-enable-replication.png'
import neon_enabled_replication from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/source/setup/neon-postgres/neon-enabled-replication.png'
import neon_ip_allow from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/source/setup/neon-postgres/neon-ip-allow.png'
Expand All @@ -21,28 +20,47 @@ Make sure you're signed in to your [Neon console](https://console.neon.tech/app/

## Creating a user with permissions {#creating-a-user-with-permissions}

Let's create a new user for ClickPipes with the necessary permissions suitable for CDC,
and also create a publication that we'll use for replication.
Connect to your Neon instance as an admin user and execute the following commands:

For this, you can head over to the **SQL Editor** tab.
Here, we can run the following SQL commands:
1. Create a dedicated user for ClickPipes:

```sql
CREATE USER clickpipes_user PASSWORD 'clickpipes_password';
GRANT USAGE ON SCHEMA "public" TO clickpipes_user;
GRANT SELECT ON ALL TABLES IN SCHEMA "public" TO clickpipes_user;
ALTER DEFAULT PRIVILEGES IN SCHEMA "public" GRANT SELECT ON TABLES TO clickpipes_user;
```sql
CREATE USER clickpipes_user PASSWORD 'some-password';
```

-- Give replication permission to the USER
ALTER USER clickpipes_user REPLICATION;
2. Grant schema-level, read-only access to the user you created in the previous step. The following example shows permissions for the `public` schema. Repeat these commands for each schema containing tables you want to replicate:

```sql
GRANT USAGE ON SCHEMA "public" TO clickpipes_user;
GRANT SELECT ON ALL TABLES IN SCHEMA "public" TO clickpipes_user;
ALTER DEFAULT PRIVILEGES IN SCHEMA "public" GRANT SELECT ON TABLES TO clickpipes_user;
```

-- Create a publication. We will use this when creating the mirror
CREATE PUBLICATION clickpipes_publication FOR ALL TABLES;
```
3. Grant replication privileges to the user:

```sql
ALTER ROLE clickpipes_user REPLICATION;
```

4. Create a [publication](https://www.postgresql.org/docs/current/logical-replication-publication.html) with the tables you want to replicate. We strongly recommend only including the tables you need in the publication to avoid performance overhead.

:::warning
Any table included in the publication must either have a **primary key** defined _or_ have its **replica identity** configured to `FULL`. See the [Postgres FAQs](../faq.md#how-should-i-scope-my-publications-when-setting-up-replication) for guidance on scoping.
:::

- To create a publication for specific tables:

```sql
CREATE PUBLICATION clickpipes FOR TABLE table_to_replicate, table_to_replicate2;
```

- To create a publication for all tables in a specific schema:

<Image size="lg" img={neon_commands} alt="User and publication commands" border/>
```sql
CREATE PUBLICATION clickpipes FOR TABLES IN SCHEMA "public";
```

Click on **Run** to have a publication and a user ready.
The `clickpipes` publication will contain the set of change events generated from the specified tables, and will later be used to ingest the replication stream.

## Enable logical replication {#enable-logical-replication}
In Neon, you can enable logical replication through the UI. This is necessary for ClickPipes's CDC to replicate data.
Expand Down
Loading