Skip to content

docs(vault): consolidate HashiCorp setup across charts and lib #171

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Keshoid merged 2 commits into from
May 21, 2026
+265 −87
Merged

docs(vault): consolidate HashiCorp setup across charts and lib #171

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6f45a13
docs(vault): consolidate HashiCorp setup into ton-rust-node vault.md,…
Keshoid May 21, 2026
3902210
refactor(adnl): reduce timeout_connect() comments
Keshoid May 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
80 changes: 35 additions & 45 deletions helm/nodectl/docs/setup.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,11 +52,6 @@ secrets to live.
| **File** | `file://` | Encrypted JSON file on the nodectl PVC, AES-256-GCM under a master key | Single-cluster deployments, simplest setup |
| **HashiCorp Vault** | `hashicorp://` | Remote Vault — Ed25519 keys in Transit engine, blobs in KV v2 | Multi-tenant infra, shared key management, centralised audit |

For URL parameter reference (auth modes, mount paths, prefixes) see the
[secrets-vault README](../../../src/secrets-vault/README.md#vault-url-schemes).
For the broader vault model see also
[Secrets Vault — ton-rust-node chart](../../ton-rust-node/docs/vault.md).

#### File backend

```bash
Expand All @@ -69,64 +64,59 @@ it securely — anyone with the key can decrypt the vault file.

#### HashiCorp Vault backend

Before creating the Secret, prepare HashiCorp Vault: enable the Transit and
KV v2 engines, create a policy that grants the required capabilities, and
either issue a static token or enable Kubernetes auth and bind a role to the
nodectl Pod's ServiceAccount. A worked example for HCP Vault Dedicated lives
in [hcp-vault-setup.md](../../../src/node-control/docs/hcp-vault-setup.md).
Prepare the Vault server (enable the Transit and KV v2 engines, create the policy and — for Kubernetes auth — the role) per [vault.md → HashiCorp Vault backend](../../ton-rust-node/docs/vault.md#hashicorp-vault-backend). The procedure is identical for nodectl and the node chart; only the placeholders differ.

URL format (one of `api_key=...` or `auth=k8s&role=...`):
Use these values when applying the policy and role templates:

```
hashicorp://<vault_address>?<auth_params>&transit_mount=<mount>&transit_prefix=<prefix>&kv_mount=<mount>&kv_prefix=<prefix>
| Placeholder | nodectl value |
|--------------------|-------------------|
| `<TRANSIT_MOUNT>` | `ton-transit` |
| `<TRANSIT_PREFIX>` | `nodectl` |
| `<KV_MOUNT>` | `ton` |
| `<KV_PREFIX>` | `nodectl` |
| `<AUTH_MOUNT>` | `kubernetes` |
| `<ROLE>` | `nodectl` |
| `<SA>` | `nodectl-sa` |

For the full `VAULT_URL` grammar (every accepted query parameter, defaults) see [secrets-vault README](../../../src/secrets-vault/README.md#vault-url-schemes).

##### Create the K8s Secret

Pick one of the two URLs and put it into a `Secret` referenced by `vault.secretName`.

**Static token** — for development or out-of-cluster Vault:

```bash
kubectl create secret generic nodectl-vault \
--from-literal=VAULT_URL='hashicorp://https://vault.example.com:8200?api_key=hvs.xxx&transit_mount=ton-transit&transit_prefix=nodectl&kv_mount=ton&kv_prefix=nodectl'
```

**Static token auth.** Suitable for development or for clusters where you
manage token rotation externally:
**Kubernetes auth** — recommended for in-cluster Vault:

```bash
kubectl create secret generic nodectl-vault \
--from-literal=VAULT_URL='hashicorp://https://vault.example.com:8200?api_key=hvs.xxx&transit_mount=transit&transit_prefix=nodectl&kv_mount=secret&kv_prefix=nodectl'
--from-literal=VAULT_URL='hashicorp://http://vault.vault.svc:8200?auth=k8s&auth_mount=kubernetes&role=nodectl&transit_mount=ton-transit&transit_prefix=nodectl&kv_mount=ton&kv_prefix=nodectl'
```

**Kubernetes auth.** Recommended for production — no long-lived token in a
Secret; the Pod authenticates with its ServiceAccount token. Requires the
chart to attach a ServiceAccount to the Pod and a Vault role bound to that
ServiceAccount.
##### Helm values

In your Helm values:
For Kubernetes auth, the chart must attach the SA bound to the Vault role:

```yaml
vault:
secretName: nodectl-vault

serviceAccount:
enabled: true # chart creates the SA
name: nodectl-app # match the SA name the Vault role is bound to
# OR, to use an existing SA you manage yourself:
enabled: true # chart creates the SA
name: nodectl-sa # must match bound_service_account_names in the Vault role
# OR, to attach an existing SA you manage yourself:
# enabled: false
# name: my-existing-sa
```

Then create the Secret:
##### Migrating from the file backend

```bash
kubectl create secret generic nodectl-vault \
--from-literal=VAULT_URL='hashicorp://http://node-vault.node-vault:8200?auth=k8s&auth_mount=kubernetes&role=nodectl-app&transit_mount=ton-transit&transit_prefix=nodectl&kv_mount=ton&kv_prefix=nodectl'
```

| Query param | Required | Notes |
|-------------|----------|-------|
| `auth` | No | `token` (default) or `k8s` |
| `api_key` | If `auth=token` | Static Vault token |
| `role` | If `auth=k8s` | Vault role bound to the Pod ServiceAccount |
| `auth_mount` | No | Kubernetes auth mount path (default `kubernetes`) |
| `namespace` | No | Vault namespace (HCP / Vault Enterprise) |
| `transit_mount` | No | Mount path of the Transit engine (default `transit`) |
| `transit_prefix` | No | Path prefix inside Transit (e.g. `nodectl`) |
| `kv_mount` | No | Mount path of the KV v2 engine (default `secret`) |
| `kv_prefix` | No | Path prefix inside KV v2 (e.g. `nodectl`) |

> **Already running on file backend?** You can move an existing deployment to
> HashiCorp without losing data — see
> [copy-file-to-hashicorp.md](copy-file-to-hashicorp.md).
If you already run nodectl on the file backend and want to move secrets into HashiCorp Vault without re-generating keys, use the dedicated migration command — see [copy-file-to-hashicorp.md](copy-file-to-hashicorp.md). The target Vault must already be prepared per the steps above.

### Install the chart

Expand Down
197 changes: 188 additions & 9 deletions helm/ton-rust-node/docs/vault.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,16 @@ When configured, private keys (ADNL, control server, liteserver) are stored in a

> Without vault, private keys remain in `config.json` as plaintext. This is acceptable for fullnodes and liteservers. For validators, configuring a vault is recommended.

## Setup
## Backends

| Backend | URL scheme | Where secrets live | Typical use |
|---------|------------|--------------------|-------------|
| **File** | `file://` | Encrypted JSON file on the node's PVC, AES-256-GCM under a master key | Single-cluster deployments, simplest setup |
| **HashiCorp Vault** | `hashicorp://` | Remote Vault — Ed25519 keys in Transit engine, blobs in KV v2 | Multi-tenant infra, shared key management, centralised audit |

For the full `VAULT_URL` grammar (every accepted query parameter, defaults, KV path layout) see the [secrets-vault README](../../../src/secrets-vault/README.md#vault-url-schemes).

## File backend

### 1. Create a Kubernetes Secret

Expand All @@ -33,6 +42,178 @@ vault:
url: "file:///keys/vault.json?master_key=<64-char-hex>"
```

## HashiCorp Vault backend

Ed25519 keys are managed via Vault's **Transit** engine. Blobs and per-secret metadata are stored in a **KV v2** engine. Both the node and nodectl charts can use the same Vault server — they need different prefixes, policies, and roles (described below).

> **Already running on the file backend?** To move an existing node's keys into HashiCorp without re-generating them, follow [Copying Node Secrets from File Storage to HashiCorp Vault](../../../src/secrets-vault/cli/COPY_FILE_TO_HASHICORP.md). Prepare the target Vault per the steps below first, then run the migration.

### VAULT_URL format

```
hashicorp://<vault_address>?<auth>&<vault_config>
```

`<vault_address>` is `host[:port]`, optionally prefixed with `http://` or `https://`. If the scheme is omitted, `https://` is assumed.

**Authentication** — choose one:

| Parameter | Required | Description |
|--------------|--------------------|------------------------------------------------------|
| `auth` | No | `token` (default) or `k8s` |
| `api_key` | If `auth=token` | Static Vault token |
| `role` | If `auth=k8s` | Vault role bound to the Pod ServiceAccount |
| `auth_mount` | No | Kubernetes auth mount path (default `kubernetes`) |
| `jwt_path` | No | ServiceAccount JWT path (default `/var/run/secrets/kubernetes.io/serviceaccount/token`) |

**Vault configuration:**

| Parameter | Default | Description |
|-----------------------|------------|------------------------------------------------------|
| `prefer_local_crypto` | `false` | Cache extractable private keys locally to sign without round-tripping to Transit |
| `transit_mount` | `transit` | Mount path of the Transit secret engine |
| `transit_prefix` | — | Prefix inside Transit — becomes part of every key name. **No `/` allowed** |
| `kv_mount` | `secret` | Mount path of the KV v2 secret engine |
| `kv_prefix` | — | Prefix inside the KV mount. Slashes are allowed |

### KV path layout

The backend stores two kinds of data side-by-side under `kv_prefix`. A Vault policy must cover **both** subtrees:

| Logical store | KV data path | KV metadata path |
|-----------------|----------------------------------------|-------------------------------------------|
| Blobs | `<kv_mount>/data/blobs/<kv_prefix>/*` | `<kv_mount>/metadata/blobs/<kv_prefix>/*` |
| Per-secret meta | `<kv_mount>/data/meta/<kv_prefix>/*` | `<kv_mount>/metadata/meta/<kv_prefix>/*` |

### Preparing the Vault server

The steps below run against your Vault server with a token that has admin rights (typically `root` or an equivalent operator policy). They are **shared between the node and nodectl** — only the placeholders differ per client.

Placeholders used throughout:

| Placeholder | Example value | Source |
|---------------------|--------------------|-------------------------------------------|
| `<TRANSIT_MOUNT>` | `ton-transit` | `transit_mount` in the client's URL |
| `<TRANSIT_PREFIX>` | `validator-0` | `transit_prefix` in the client's URL |
| `<KV_MOUNT>` | `ton` | `kv_mount` in the client's URL |
| `<KV_PREFIX>` | `mainnet/validator-0` | `kv_prefix` in the client's URL |
| `<AUTH_MOUNT>` | `kubernetes` | `auth_mount` in the client's URL |
| `<ROLE>` | `validator-0` | `role` in the client's URL |
| `<SA>` | `validator-0-sa` | Pod ServiceAccount name |

#### 1. Enable the engines

Idempotent — skip if already enabled.

```bash
vault secrets enable -path=<TRANSIT_MOUNT> transit
vault secrets enable -path=<KV_MOUNT> -version=2 kv
```

#### 2. Create the client policy

One policy per client (one per validator, one for nodectl). Each policy is scoped to that client's `<TRANSIT_PREFIX>` and `<KV_PREFIX>` only — nothing more.

```hcl
# Transit: per-prefix key management + crypto
path "<TRANSIT_MOUNT>/keys/<TRANSIT_PREFIX>.*" { capabilities = ["create", "read", "update"] }
path "<TRANSIT_MOUNT>/keys/" { capabilities = ["list"] }
path "<TRANSIT_MOUNT>/sign/<TRANSIT_PREFIX>.*" { capabilities = ["update"] }
path "<TRANSIT_MOUNT>/verify/<TRANSIT_PREFIX>.*" { capabilities = ["update"] }
path "<TRANSIT_MOUNT>/export/signing-key/<TRANSIT_PREFIX>.*" { capabilities = ["read"] }
path "<TRANSIT_MOUNT>/wrapping_key" { capabilities = ["read"] }

# KV v2: blobs + per-secret metadata under the prefix
path "<KV_MOUNT>/data/blobs/<KV_PREFIX>/*" { capabilities = ["create", "read", "update", "delete"] }
path "<KV_MOUNT>/data/meta/<KV_PREFIX>/*" { capabilities = ["create", "read", "update", "delete"] }
path "<KV_MOUNT>/metadata/blobs/<KV_PREFIX>/*" { capabilities = ["read", "list", "delete"] }
path "<KV_MOUNT>/metadata/meta/<KV_PREFIX>/*" { capabilities = ["read", "list", "delete"] }
```

Write it as:

```bash
vault policy write <policy-name> ./<policy-name>.hcl
```

#### 3. Kubernetes auth (skip for static token)

Enable the auth method (idempotent):

```bash
vault auth enable -path=<AUTH_MOUNT> kubernetes
```

If the Vault server runs **inside** the same cluster, it auto-discovers the Kubernetes API. If it runs **outside**, configure it explicitly:

```bash
vault write auth/<AUTH_MOUNT>/config \
kubernetes_host="https://<k8s-api>:6443" \
kubernetes_ca_cert=@/path/to/ca.crt
```

Create one role per client, bound to that client's Pod ServiceAccount and policy:

```bash
vault write auth/<AUTH_MOUNT>/role/<ROLE> \
bound_service_account_names=<SA> \
bound_service_account_namespaces=<your-namespace> \
policies=<ROLE> \
ttl=10m
```

> Per-client isolation requires **one role per client** (one SA per Pod, one policy per role). Sharing a role across multiple SAs means any of those Pods can use any of the policies attached to the role.

### Node deployment

Each validator is its own Helm release of this chart. The chart attaches a ServiceAccount to the Pod — by default named after the release (e.g. release `validator-0` → SA `validator-0-sa` when `serviceAccount.name` is set accordingly). Each validator should have its own Vault prefix, policy, and role.

#### Suggested per-validator values

For validator `i` (e.g. `i = 0`):

| Field | Value |
|------------------|-------------------|
| Helm release | `validator-0` |
| SA | `validator-0-sa` |
| Policy | `validator-0` |
| Role | `validator-0` |
| `transit_prefix` | `validator-0` |
| `kv_prefix` | `mainnet/validator-0` (or `dev/validator-0` per environment) |

Apply the [policy template](#2-create-the-client-policy) and [role template](#3-kubernetes-auth-skip-for-static-token) with these substitutions.

#### VAULT_URL

```
hashicorp://http://vault.vault.svc:8200?auth=k8s&role=validator-0&transit_mount=ton-transit&transit_prefix=validator-0&kv_mount=ton&kv_prefix=mainnet/validator-0
```

#### Create the K8s Secret

```bash
kubectl create secret generic ton-node-vault \
--from-literal=VAULT_URL='hashicorp://http://vault.vault.svc:8200?auth=k8s&role=validator-0&transit_mount=ton-transit&transit_prefix=validator-0&kv_mount=ton&kv_prefix=mainnet/validator-0'
```

For a static token instead of Kubernetes auth, swap the URL:

```bash
kubectl create secret generic ton-node-vault \
--from-literal=VAULT_URL='hashicorp://https://vault.example.com:8200?api_key=hvs.xxx&transit_mount=ton-transit&transit_prefix=validator-0&kv_mount=ton&kv_prefix=mainnet/validator-0'
```

#### Helm values

```yaml
vault:
secretName: ton-node-vault

serviceAccount:
enabled: true
name: validator-0-sa # must match bound_service_account_names in the role
```

## Values reference

| Parameter | Description | Default |
Expand All @@ -43,20 +224,18 @@ vault:

When `vault.secretName` is set, it takes precedence over `vault.url`.

## Vault URL formats

| Backend | URL format |
|---------|------------|
| File | `file:///keys/vault.json?master_key=<64-char-hex>` |

## Troubleshooting

**"vault is not set"** — `VAULT_URL` environment variable is not set. Check that `vault.secretName` or `vault.url` is configured in Helm values and the K8s Secret exists.

**Secret not found in vault** — Keys are not auto-generated (except the master wallet key in nodectl). Create all referenced keys before starting the service.

**`permission denied` on KV writes** — The policy is missing one of the two subtrees (`data/blobs/...` or `data/meta/...`). Both are required, see [KV path layout](#kv-path-layout).

**`permission denied` on Transit `sign`/`export`** — The policy paths must match the `transit_prefix` you put into the URL exactly, with `.*` to cover per-key suffixes (e.g. `validator-0.*`, not `validator-0/*`).

## Important

- Vault is configured **only** via the `VAULT_URL` environment variable. The `secrets_vault_config` field in `config.json` is no longer supported.
- The vault file is stored on the `keys` volume (`/keys`), which has `helm.sh/resource-policy: keep` by default — it survives `helm uninstall`.
- Both the node and nodectl must point to the **same vault** for key management to work correctly.
- The file-backend vault file is stored on the `keys` volume (`/keys`), which has `helm.sh/resource-policy: keep` by default — it survives `helm uninstall`.
- The node and nodectl can share the **same Vault server** but should use **different prefixes, policies, and roles**. They must not point at the same `transit_prefix`/`kv_prefix` — that would let either client clobber the other's keys.
15 changes: 2 additions & 13 deletions src/adnl/src/adnl/client.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -145,19 +145,8 @@ impl AdnlClient {
Ok(Self { crypto, stream })
}

/// Connect to server using async-native TCP connect with a timeout.
///
/// Unlike [`Self::connect`], which performs a synchronous `connect(2)` via
/// `socket2::Socket::connect_timeout` and parks the tokio worker thread
/// until the kernel returns (up to the configured write timeout), this
/// variant uses `tokio::net::TcpStream::connect` wrapped in
/// `tokio::time::timeout`. The runtime worker stays free to drive other
/// futures while the kernel is performing the TCP handshake or waiting on
/// an unresponsive peer.
///
/// The address family of the resulting socket is selected from
/// `config.server_address` (IPv4 or IPv6). The original `SO_LINGER 0`
/// option is preserved.
/// Like [`Self::connect`], but uses `tokio::net::TcpStream::connect` so the
/// runtime worker is not parked while the kernel waits on an unresponsive peer.
pub async fn timeout_connect(config: &AdnlClientConfig) -> Result<Self> {
let connect_timeout = config.timeouts.write();
let tcp = tokio::time::timeout(
Expand Down
Loading
Loading