Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 40 additions & 0 deletions docs/enterprise_edition/control_plane/cli.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
---
icon: material/console-line
---

# CLI

The control plane comes with a command-line interface (CLI). It allows you to create users, access tokens, and control other aspects of its operation.

## Commands

The control plane CLI supports the following commands:

| Command | Description |
|-|-|
| [`onboard`](#onboarding) | Create a new user and PgDog deployment. This is typically the first command you need to run when installing the control plane. |
| `migrate` | Run PostgreSQL database migrations. Run this after upgrading your PgDog Enterprise version. |
| `server` | Run the control plane server. This is executed by default when the `control` executable is running. |
| `token` | Create an access token for PgDog to connect to the control plane. |

### Onboarding

The `onboard` command is an all-in-one command to create a PgDog authentication token and a web UI user associated with that token. It's typical to execute this command right after installing the control plane, for example:

```bash
control onboard \
--email demo@pgdog.dev \
--password demopass \
--token 644b527c-b9d6-4fb2-9861-703bad871ec0 \
--name Demo
```

| Argument | Description |
|-|-|
| `email` | The email for the new user. |
| `password` | The password for the new user. |
| `token` | The authentication token, which grants PgDog access to the control plane to upload telemetry. |
| `name` | The name for the deployment. |
| `generate-token` | If `token` is not specified, this will generate a random one and print it to the terminal. |

This command is idempotent: if the user exists already, this will update its password. If the token already exists, the user will be associated to that token. If all of these are already true, no changes will be made.
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ icon: material/console

# Control plane

Multi-node PgDog deployments require synchronization to perform certain tasks, like atomic configuration changes, toggling [maintenance mode](../administration/maintenance_mode.md), [resharding](../features/sharding/resharding/index.md), and more. To make this work, PgDog Enterprise comes with a control plane, an application deployed alongside PgDog, to provide coordination and collect and present system telemetry.
Multi-node PgDog deployments require synchronization to perform certain tasks, like atomic configuration changes, toggling [maintenance mode](../../administration/maintenance_mode.md), [resharding](../../features/sharding/resharding/index.md), and more. To make this work, PgDog Enterprise comes with a control plane, an application deployed alongside PgDog, to provide coordination and collect and present system telemetry.

## How it works

Expand All @@ -16,7 +16,7 @@ The control plane and PgDog processes communicate via the network using HTTP. Th

### Configuration

In order for PgDog to connect to the control plane, it needs to be configured with its endpoint address and an authentication token, both of which are specified in [`pgdog.toml`](../configuration/pgdog.toml/general.md):
In order for PgDog to connect to the control plane, it needs to be configured with its endpoint address and an authentication token, both of which are specified in [`pgdog.toml`](../../configuration/pgdog.toml/general.md):

```toml
[control]
Expand All @@ -26,7 +26,7 @@ token = "cff57e5c-7c4f-4ca0-b81c-c8ed22cf873d"

The authentication token is generated by the control plane and identifies each PgDog deployment. PgDog nodes which are part of the same deployment should use the same token.

For example, if you're using our [Helm chart](../installation.md#kubernetes), you can configure the endpoint and token in `values.yaml` as follows:
For example, if you're using our [Helm chart](../../installation.md#kubernetes), you can configure the endpoint and token in `values.yaml` as follows:

```yaml
control:
Expand All @@ -36,13 +36,13 @@ control:

### Connection flow

The connection to the control plane is initiated by PgDog on startup and happens in the background. Upon connecting, PgDog will send its node identifier (randomly generated, or set in the `NODE_ID` envrionment variable) to register with the control plane, and start uploading telemtry and poll for commands.
The connection to the control plane is initiated by PgDog on startup and happens in the background. Upon connecting, PgDog will send its node identifier (randomly generated, or set in the `NODE_ID` environment variable) to register with the control plane, and start uploading telemetry and poll for commands.

!!! note "Error handling"
Since most PgDog functions (including sharding) are configuration-driven, the control plane connection is **not required**
for PgDog to start and serve queries.

If any error is encounted while communicating with the control plane,
If any error is encountered while communicating with the control plane,
PgDog will continue operating normally, while attempting to reconnect periodically.


Expand All @@ -54,15 +54,15 @@ PgDog transmits the following information to the control plane:

| Telemetry | Description |
|-|-|
| [Metrics](metrics.md) | The same [metrics](../features/metrics.md) as exposed by the Prometheus endpoint (and the admin database), are trasmitted at a much higher frequency, to allow for real-time monitoring. |
| [Active queries](insights/active_queries.md) | Queries that are currently executing through each PgDog node. |
| [Query statistics](insights/statistics.md) | Real-time statistics on each query executed through PgDog, like duration, idle-in-transaction time, and more. |
| [Errors](insights/errors.md) | Recent errors encountered by clients, e.g. query syntax issues. |
| [Query plans](insights/query_plans.md) | Output of `EXPLAIN` for slow and sampled queries, collected by PgDog in the background. |
| [Metrics](../metrics.md) | The same [metrics](../../features/metrics.md) as exposed by the Prometheus endpoint (and the admin database), are transmitted at a much higher frequency, to allow for real-time monitoring. |
| [Active queries](../insights/active_queries.md) | Queries that are currently executing through each PgDog node. |
| [Query statistics](../insights/statistics.md) | Real-time statistics on each query executed through PgDog, like duration, idle-in-transaction time, and more. |
| [Errors](../insights/errors.md) | Recent errors encountered by clients, e.g. query syntax issues. |
| [Query plans](../insights/query_plans.md) | Output of `EXPLAIN` for slow and sampled queries, collected by PgDog in the background. |
| Configuration | Current PgDog settings and database schema. |

#### High availability

The control plane itself is backed by a PostgreSQL database, used for storing historical metrics, query statistics, configuration, and other metadata.

This allows multiple instances of the control plane to be deployed in a high-avaibility setup, since all actions are syncrhonized by PostgreSQL transactions and locks.
This allows multiple instances of the control plane to be deployed in a high-availability setup, since all actions are synchronized by PostgreSQL transactions and locks.
100 changes: 100 additions & 0 deletions docs/enterprise_edition/control_plane/self-hosting.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
---
icon: material/server
---

# Self-hosting

Self-hosting the [control plane](index.md) (not managed by PgDog) is supported and requires a bit of configuration and setup.

## Getting started

The easiest way to run the control plane is using our Docker image. It contains both the backend and the web UI and can be deployed as standalone application, either in Kubernetes or on any machine with Docker installed.

### Dependencies

The control plane has two dependencies:

1. A PostgreSQL database used to store historical metrics, query statistics, users and other metadata
2. A Redis database, used for synchronization and real-time metrics

If you're using our [Helm chart](#kubernetes), both are provided automatically.

### Kubernetes

If you're running PgDog in Kubernetes using our [Helm chart](../../installation.md#kubernetes), you can deploy the control plane into the same cluster using our Enterprise Helm chart:

```
helm repo add pgdogdev-ee https://helm-ee.pgdog.dev
helm install pgdogdev-ee/pgdog-control
```

The following values should be set in `values.yaml`:

| Value | Description |
|-|-|
| `image.tag` | The Docker tag for the control plane image. |
| `ingress.host` | The DNS host for the control plane, e.g., `pgdog.database.internal`. |
| `env` | A key/value mapping of [environment varialbes](#configuration) passed to the control plane application. |

For example:

```yaml
image:
tag: main-ent
ingress:
host: pgdog.database.internal
env:
SESSION_KEY: 1b80a3cc1640a3[...]
```

### Configuration

The control plane is configured via environment variables. The following variables are required for it to work correctly:

| Environment variable | Description |
|-|-|
| `DATABASE_URL` | URL pointing to the Postgres database used for storing control plane data. |
| `SESSION_KEY` | Secret key used to encrypt user session cookies. Can be any value, as long as it's at least 64 bytes. |
| `REDIS_URL` | URL pointing to the Redis database used for synchronization. |
| `FRONTEND_URL` | The URL where the frontend application is hosted. This defaults to `ingress.host` if you're using the Helm chart. |

!!! note "Helm chart"
If you're using the [Helm chart](#kubernetes), these are generated from settings in `values.yaml` and don't need to be configured manually.

#### Session key

The control plane requires a 64 bytes randomly generated session key to encrypt user session cookies. You can generate one with just one line of Python:

=== "Command"
```bash
python3 -c "import secrets,base64; print(base64.b64encode(secrets.token_bytes(64)).decode())"
```
=== "Output"
```
1b80a3cc1640a37b59b7dd591749ebd6532b720712e9ae2c37cb5572828ed5135332595decdadf702f919d5b58099135fbd4344979c2a0e2cf514ff3c6e640ac
```

### Authentication

The control plane web UI suppors two authentication methods:

1. Email and password
2. OAuth2

Password authentication works out of the box and requires no additional setup beyond creating users via the [CLI](cli.md).

For OAuth2, you need to configure each provider, and depending on which provider you choose, different environment variables need to be set as well:

=== "Google"
| Environment variable | Description |
|-|-|
| `GOOGLE_CLIENT_ID` | Google OAuth2 client identifier. You can obtain one by creating an OAuth2 application in the [Google Cloud Console](https://console.cloud.google.com/apis/credentials). |
| `GOOGLE_CLIENT_SECRET` | Google OAuth2 client secret. |
| `GOOGLE_REDIRECT_URL` | OAuth redirect URL. It should be set to the following: `${FRONTEND_URL}/google/oauth/callback`. |

=== "GitHub"
| Environment variable | Description |
|-|-|
| `GITHUB_CLIENT_ID` | GitHub OAuth2 client identifier. You can obtain one by creating an OAuth application in the [Developer Settings](https://github.com/settings/developers) in your GitHub account. |
| `GITHUB_CLIENT_SECRET` | GitHub OAuth2 client secret. |
| `GITHUB_REDIRECT_URL` | OAuth redirect URL. It should be set to the following: `${FRONTEND_URL}/github/oauth/callback`. |
30 changes: 24 additions & 6 deletions docs/enterprise_edition/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,20 +4,42 @@ icon: material/office-building

# PgDog Enterprise

PgDog Enterprise is a version of PgDog that contains additional features for at scale monitoring and deployment of sharded (and unsharded) PostgreSQL databases.
PgDog Enterprise is a version of PgDog that contains additional features for large-scale monitoring and deployment of sharded (and unsharded) PostgreSQL databases.

Unlike PgDog itself, PgDog Enterprise is closed source and available upon the purchase of a license. It comes with a control plane which provides real-time visibility into PgDog's operations and enterprise features.

## Features

| Feature | Description |
|-|-|
| [Control plane](control_plane.md) | Synchronize and monitor multiple PgDog processes. |
| [Control plane](control_plane/index.md) | Synchronize and monitor multiple PgDog processes. |
| [Active queries](insights/active_queries.md) | Real-time view into queries running through PgDog. |
| [Query plans](insights/query_plans.md) | Root cause slow queries and execution anomalies with real-time Postgres query plans, collected in the background. |
| [Real-time metrics](metrics.md) | All PgDog metrics, delivered with second-precision through a dedicated connection. |
| [Query statistics](insights/statistics.md) | Query execution statistics, like duration, idle-in-transaction time, errors, and more. |

## Demo

You can run a demo version of PgDog Enterprise locally with Docker Compose:

```bash
curl -sSL \
https://docs.pgdog.dev/examples/control_plane/docker-compose.yaml \
-o docker-compose.yaml \
&& docker-compose up
```

The demo comes with the control plane, the web UI and PgDog configured as follows:

| Setting | Value |
|-|-|
| Web UI | `http://localhost:8099` |
| Username | `demo@pgdog.dev` |
| Password | `demopass` |
| PgDog | `0.0.0.0:6432` |

For questions about the demo, PgDog Enterprise features, or pricing, [contact us](https://calendly.com/lev-pgdog/30min). PgDog can be deployed on-prem, in your cloud account, or entirely managed by us.

## Roadmap

PgDog Enterprise is new and in active development. A lot of the features we want aren't built yet:
Expand All @@ -27,7 +49,3 @@ PgDog Enterprise is new and in active development. A lot of the features we want
| QoS | Quality of service guarantees, incl. throttling on a per-user/database/query level. |
| AWS RDS integration | Deploy PgDog on top of AWS RDS, without the hassle of Kubernetes or manual configuration. |
| Automatic resharding | Detect hot shards and re-shard data without operator intervention. |

## Get a demo

If you'd like a demo of PgDog EE, [get in touch](https://calendly.com/lev-pgdog/30min) with our sales team. PgDog EE comes with a [cloud](https://cloud.pgdog.dev) deployment managed by us, or can be deployed entirely on-prem.
4 changes: 2 additions & 2 deletions docs/enterprise_edition/insights/active_queries.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ icon: material/play-circle
PgDog Enterprise provides a real-time view into queries currently executing on its PostgreSQL connections. This is accessible in two places:

1. [`SHOW ACTIVE_QUERIES`](#admin-database) admin command
2. [Activity](#dashboard) view in the dashboard
2. [Activity](#web-ui) view in the dashboard

## How it works

Expand Down Expand Up @@ -46,7 +46,7 @@ The following information is available in the running queries view:

### Web UI

If you're running multiple instances of PgDog, active queries from all instances are aggregated and sent to the [control plane](../control_plane.md). They are then made available in the Activity tab, in real-time, with query plans automatically attached for slow queries.
If you're running multiple instances of PgDog, active queries from all instances are aggregated and sent to the [control plane](../control_plane/index.md). They are then made available in the Activity tab, in real-time, with query plans automatically attached for slow queries.

<center>
<img src="/images/ee/activity.png" width="100%" alt="How PgDog works" class="screenshot" />
Expand Down
2 changes: 1 addition & 1 deletion docs/enterprise_edition/insights/errors.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,4 +44,4 @@ max_errors = 100
max_error_age = 300_000 # 5 minutes
```

By default, PgDog will keep up to 100 distinct errors for a maximum of 5 minutes. This data is periodically sent to the [control plane](../control_plane.md), so the history of seen errors is available in the web UI.
By default, PgDog will keep up to 100 distinct errors for a maximum of 5 minutes. This data is periodically sent to the [control plane](../control_plane/index.md), so the history of seen errors is available in the web UI.
2 changes: 1 addition & 1 deletion docs/enterprise_edition/insights/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,4 +17,4 @@ PgDog Enterprise collects the following telemetry:
| [Query statistics](statistics.md) | real time | Query duration, number of rows returned, idle-in-transaction time, errors, and more. |
| [Errors](errors.md) | real time | View into recently encountered query errors, like syntax errors or lock timeouts. |

This data is transmitted to the [control plane](../control_plane.md) in real-time, which makes it available via its web dashboard and HTTP API.
This data is transmitted to the [control plane](../control_plane/index.md) in real-time, which makes it available via its web dashboard and HTTP API.
4 changes: 2 additions & 2 deletions docs/enterprise_edition/insights/query_plans.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,14 @@ icon: material/chart-timeline
---
# Query plans

For any [running query](active_queries.md) exceeding a configurable time threshold, PgDog EE will ask Postgres for a query plan. The query plans are stored in their own view, accessible via two methods:
For any [running query](active_queries.md) exceeding a configurable time threshold, PgDog will ask Postgres for a query plan. The query plans are stored in their own view, accessible via two methods:

1. [`SHOW QUERY_PLANS`](#admin-database) admin command
2. [Activity](active_queries.md#dashboard) view in the dashboard

## How it works

When a [running query](active_queries.md) exceeds a configurable threshold, PgDog EE will ask Postgres for its query plan by sending an `EXPLAIN` command via a dedicated connection. For prepared statements, PgDog automatically provides the parameters sent with the statement by the client.
When a [running query](active_queries.md) exceeds a configurable threshold, PgDog will ask Postgres for its query plan by sending an `EXPLAIN` command via a dedicated connection. For prepared statements, PgDog automatically provides the parameters sent with the statement by the client.

Since `EXPLAIN` itself is very quick, fetching and storing query plans is efficient and doesn't impact database performance. Nonetheless, to avoid planning queries unnecessarily, the plans are stored in an in-memory cache. Old plans are evicted automatically and recomputed.

Expand Down
2 changes: 1 addition & 1 deletion docs/enterprise_edition/insights/statistics.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ All queries are normalized (parameters replaced with `$1`, `$2`, etc.) and group
This data is accessible via two mediums:

1. [Admin database](#admin-database)
2. The Insights page in the web UI of the [control plane](../control_plane.md)
2. The Insights page in the web UI of the [control plane](../control_plane/index.md)

### Admin database

Expand Down
4 changes: 2 additions & 2 deletions docs/enterprise_edition/metrics.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,13 @@ icon: material/speedometer
---
# Real-time metrics

PgDog Enterprise collects and trasmits its own metrics to the [control plane](control_plane.md), at a configurable interval (1s, by default). This provides a real-time view into PgDog internals, without a delay that's typically present in other monitoring solutions.
PgDog Enterprise collects and transmits its own metrics to the [control plane](control_plane/index.md), at a configurable interval (1s, by default). This provides a real-time view into PgDog internals, without a delay that's typically present in other monitoring solutions.

## How it works

Real-time metrics are available in both Open Source and Enterprise versions of PgDog. The [open source metrics](../features/metrics.md) are accessible via an OpenMetrics endpoint or via the admin database.

In PgDog EE, the same metrics are collected and sent via a dedicated connection to the control plane. Since metrics are just numbers, they can be serialized and sent quickly. To deliver second-precision metrics, PgDog EE requires less than 1KB/second of bandwidth and little to no additional CPU or memory.
In PgDog Enterprise, the same metrics are collected and sent via a dedicated connection to the control plane. Since metrics are just numbers, they can be serialized and sent quickly. To deliver second-precision metrics, PgDog requires less than 1KB/second of bandwidth and little to no additional CPU or memory.

### Configuration

Expand Down
Loading
Loading