Skip to content
6 changes: 6 additions & 0 deletions docs/kratos/concepts/security.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -70,3 +70,9 @@ password policy, refer to the [password policy page](../../concepts/password-pol

Ory OAuth2 and OpenID Connect is a certified OAuth2 and OpenID Connect provider. You can read more in the
[OAuth 2.0 security overview](../../hydra/security-architecture) documentation.

## Filesystem sandbox

On Linux 5.13 and later, Ory Identities applies a [Landlock filesystem sandbox](../../security-compliance/landlock-sandbox.mdx) to
limit what the process can read or execute at runtime. Ory Network and OEL deployments sandbox the full `kratos serve` process;
the OSS binary sandboxes the hidden Jsonnet worker so mapper code cannot reach the filesystem under any circumstances.
9 changes: 9 additions & 0 deletions docs/kratos/debug/troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,15 @@ Please add your troubleshooting tricks and other tips to this document, You can

:::

### `permission denied` / `EPERM` reading a config, schema, template, or certificate

If `kratos serve` logs `permission denied` opening a path that is owned by the right user and has the right Unix permissions, the
[Landlock filesystem sandbox](../../security-compliance/landlock-sandbox.mdx) on Ory Network and OEL binaries is most likely
denying the access. Check the startup logs for the `Landlock filesystem sandbox is active` line and the preceding `roPaths` /
`rwDirs` lists, then either add the missing path to `security.landlock.allowed_paths` or temporarily set
`security.landlock.disabled: true` to confirm. The sandbox page has the full troubleshooting walkthrough, including how to read
kernel audit records and use `strace`.

### `400: Bad Request` on self-service flows

Make sure you are starting and finishing the request in one browser. Self-service browser flows need to be executed in the same
Expand Down
8 changes: 8 additions & 0 deletions docs/kratos/emails-sms/01_sending-emails-smtp.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,14 @@ courier:
in: header
```
:::info Self-hosted (Network / OEL)
The `body: file://...` Jsonnet template path is auto-discovered by the
[Landlock filesystem sandbox](../../security-compliance/landlock-sandbox.mdx): Kratos walks the loaded config at startup and
allow-lists every `file://` URI it finds. No `security.landlock.allowed_paths` entry is required.

:::

Here is one example of a Jsonnet body:

```jsonnet
Expand Down
2 changes: 2 additions & 0 deletions docs/kratos/guides/deploy-kratos-example.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -716,3 +716,5 @@ We need to set up secure keys and run Ory Kratos in production mode:
- [Add Two Factor Authentication (2FA) to your App](../mfa/01_overview.mdx)
- [Configure The Password Policy](../../concepts/password-policy.mdx)
- [Integrate using webhooks](../hooks/01_configure-hooks.mdx)
- Review the [Landlock filesystem sandbox](../../security-compliance/landlock-sandbox.mdx) — Ory Network and OEL binaries restrict
`kratos serve` to its config, TLS, schema, courier template, and SQLite database paths at runtime.
27 changes: 27 additions & 0 deletions docs/kratos/guides/https-tls.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -34,3 +34,30 @@ docker run ... \
```

or mount the files using `--mount` and linking to the files.

## Certificate renewal with the filesystem sandbox

Ory Network and Ory Enterprise License (OEL) deployments run with the
[Landlock filesystem sandbox](../../security-compliance/landlock-sandbox.mdx) active. The configured TLS cert and key paths are
allow-listed at startup, including any symlinks resolved at that moment. Landlock rules are irrevocable, and they attach to a
specific inode.

With cert-manager or certbot, a renewal writes a new file and re-points a symlink at it. Whether that "just works" depends on
**how** the cert path is allow-listed:

- **Containing-directory grant — transparent renewals.** Add the cert directory under `security.landlock.allowed_paths` (for
example `/etc/letsencrypt` for certbot, or the cert-manager volume mount). The rule covers every inode underneath, so the
symlink (typically in `live/<domain>/`) and the renewed target (in `archive/<domain>/`) both sit inside the sandbox and reads
keep working — no restart needed.

```yaml
security:
landlock:
allowed_paths:
- /etc/letsencrypt
```

- **Leaf grant — restart required.** If only the cert and key files themselves are allow-listed (the default for
`SERVE_PUBLIC_TLS_CERT_PATH` and friends), the rule attaches to the original target inode. After a renewal the symlink resolves
to a new inode that is not in the allowlist, and the kernel denies reads with `EPERM`. Restart `kratos serve` after every
renewal so the rules re-attach. Wire the restart into your renew hook.
11 changes: 11 additions & 0 deletions docs/kratos/guides/production.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,17 @@ case, the request will fail with an error.
Never expose the Ory Kratos Admin API to the internet unsecured. Always require authorization. A good practice is to not expose
the Admin API at all to the public internet and use a Zero Trust Networking Architecture within your intranet.

### Filesystem sandbox (Ory Network / OEL)

Ory Network and Ory Enterprise License binaries activate a
[Landlock filesystem sandbox](../../security-compliance/landlock-sandbox.mdx) for `kratos serve` on Linux 5.13 and later. The
config files, TLS material, the courier template directory, the SQLite database directory, and every `file://` URI referenced in
the loaded configuration (identity schemas, OIDC mappers, web-hook bodies, courier templates, tokenizer JWKS files, and so on) are
auto-allowed at startup; every other path is denied by the kernel. If you depend on files the auto-discovery does not see — for
example a corporate CA bundle pointed to by `SSL_CERT_FILE`, a JSON Schema `$ref` fragment inside an identity schema body, or a
legacy config field that takes a bare path instead of a `file://` URI — list them under `security.landlock.allowed_paths` before
going to production.

## Scaling

There are no additional requirements for scaling Ory Kratos, just spin up another container!
Original file line number Diff line number Diff line change
Expand Up @@ -945,6 +945,14 @@ You can read more about using the `definitions` and `dependencies` keywords in t
[structuring a complex schema](https://json-schema.org/understanding-json-schema/structuring.html?highlight=definitions) and
[conditional subschemas](https://json-schema.org/understanding-json-schema/reference/conditionals.html#applying-subschemas-conditionally).

:::info Self-hosted

If you split a schema across multiple files with `"$ref": "file:///path/to/fragment.json"`, list each referenced file under
`security.landlock.allowed_paths` so the [Landlock filesystem sandbox](../../security-compliance/landlock-sandbox.mdx) does not
block the read. Only the top-level `identity.schemas[].url` entries are auto-allowed.

:::

Below is an example of how to do this:

```json5
Expand Down
9 changes: 9 additions & 0 deletions docs/kratos/manage-identities/35_best-practices.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,15 @@ identity:
url: file://path/to/user_v0.json
```

:::info Self-hosted

On Ory Network and Ory Enterprise License (OEL), `file://` schema URLs are auto-allowed by the
[Landlock filesystem sandbox](../../security-compliance/landlock-sandbox.mdx). If a schema body uses `"$ref": "file:///..."` to
pull in another local file, that referenced file must be listed under `security.landlock.allowed_paths` — only the top-level
`identity.schemas[].url` entries are auto-discovered.

:::

After a few weeks, you decide that you want to add additional fields or that you need to break compatibility with your current
schema. To do that, add another version of the schema to the configuration and change the `default_schema_id` to use the new
schema:
Expand Down
8 changes: 8 additions & 0 deletions docs/kratos/reference/jsonnet.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,14 @@ allowing you to write code that modifies your identity's data and loads it into

We highly recommend checking out the official [Learning Jsonnet Tutorial](https://jsonnet.org/learning/tutorial.html).

:::note

Jsonnet evaluation runs in a separate worker process that is fully isolated from the filesystem by the
[Landlock sandbox](../../security-compliance/landlock-sandbox.mdx). Jsonnet code can transform claims and produce identity traits,
but it cannot read files from disk under any circumstances.

:::

## Input and output

Jsonnet is a data-templating language that allows you to define identity traits and metadata based on input data from external
Expand Down
8 changes: 8 additions & 0 deletions docs/kratos/self-hosted/01_mail-courier-selfhosted.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,14 @@ directories of the root directory, corresponding to the respective methods for f
and HTML templates are required. The courier uses them as
[alternatives](https://github.com/ory/kratos/blob/871ee0475a27771dd6395aad617f41a22ccc3b9a/courier/courier.go#L205) for fallback.

:::info Filesystem sandbox

On Ory Network and Ory Enterprise License (OEL) deployments, the courier template directory is automatically allow-listed by the
[Landlock filesystem sandbox](../../security-compliance/landlock-sandbox.mdx). Templates are loaded lazily at send time, so an
out-of-tree template path that is not allow-listed will fail with `EPERM`.

:::

:::tip

If you're running multiple instances of Kratos and separate courier job, make sure to provide templates to all instances (both
Expand Down
9 changes: 9 additions & 0 deletions docs/kratos/self-hosted/02_mail-courier-templates.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,15 @@ In self-hosted Ory Kratos instances, you can add custom templates from `http://`
and `html` fields are mandatory when defining the `body` key. All other keys are optional. If the system can't use the custom
templates, it falls back to the built-in templates or to the `template_override_path` (if specified).

:::info Self-hosted (Network / OEL)

The `template_override_path` directory and any `file://` template URIs are automatically allow-listed by the
[Landlock filesystem sandbox](../../security-compliance/landlock-sandbox.mdx) — Kratos walks the loaded config at startup and
allow-lists every `file://` URI it finds, regardless of where on disk the template lives. No `security.landlock.allowed_paths`
entry is required.

:::

```yaml title=path/to/kratos/config.yml
courier:
template_override_path: /conf/courier-template
Expand Down
207 changes: 207 additions & 0 deletions docs/security-compliance/landlock-sandbox.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,207 @@
---
id: landlock-sandbox
title: Landlock filesystem sandbox for Ory Identities
sidebar_label: Landlock filesystem sandbox
---

Ory Identities (Kratos) uses the Linux [Landlock](https://docs.kernel.org/userspace-api/landlock.html) LSM to restrict filesystem
access at runtime. After Kratos starts up and loads everything it needs from disk, the kernel denies any further open or execute
on paths that are not explicitly allow-listed. This narrows the blast radius of bugs, misconfigurations, or supply-chain
compromises that could otherwise read arbitrary files.

The sandbox requires Linux 5.13 or later. On older kernels and non-Linux platforms, Kratos runs without it.

## What is sandboxed

### Jsonnet worker

The hidden `jsonnet` subcommand that Kratos re-execs to evaluate Jsonnet mappers — OIDC claim mappers, courier templates, and
identity-schema transforms — applies an empty Landlock layer at startup. The worker only ever needs the inherited
stdin/stdout/stderr, so every path-based filesystem access from inside the worker is denied by the kernel, even if a Jsonnet
snippet were to bypass the in-process import barrier. Already-open file descriptors keep working, so the parent ↔ worker IPC is
unaffected.

This layer is active on Ory Identities OSS, Network, and OEL and is not configurable: there is no legitimate use case to allow the
Jsonnet VM to read from disk.

### `kratos serve` (Ory Network and OEL)

On Ory Network and OEL, the Landlock sandbox also wraps the main `kratos serve` process. After initialization, the process is
restricted to the files and directories it needs at runtime:

- the configuration files passed via `--config`
- TLS certificates and keys for the public and admin listeners
- SMTP client certificate and key files (`courier.smtp.client_cert_path`, `courier.smtp.client_key_path`, and the equivalents
under `courier.channels[].smtp_config`)
- the courier template directory (courier templates are loaded lazily, at send time)
- **every file referenced from the config via a `file://` URI.** Kratos walks the loaded configuration once at startup and
allow-lists every value that begins with `file://`. This covers identity schemas under `identity.schemas[].url`, OIDC claim
mappers, `web_hook` body templates, courier HTTP body templates, session tokenizer mappers and JWKS files, and any future
`file://` field added to the config schema. Operators do not need to duplicate these paths under
`security.landlock.allowed_paths`.
- the directory containing the SQLite database — covers the database file itself and any `-journal`, `-wal`, `-shm`, or transient
`-mj-XXXXX` siblings SQLite creates next to it
- any paths listed in `security.landlock.allowed_paths`

Auto-discovery matches the `file://` URI form documented in the config schema. A few legacy fields still accept a bare filesystem
path without the `file://` prefix (notably the deprecated form of `web_hook` `body`); those bare paths are not auto-discovered, so
list them under `security.landlock.allowed_paths` or migrate the configuration to the `file://` form.

A small set of system files is allowed by default:

- `/dev/null` — subprocess plumbing
- `/etc/resolv.conf` and `/etc/hosts` — required by Go's pure-Go DNS resolver
- the running Kratos binary itself, with read + execute, so the Jsonnet sandbox can re-exec it as a worker

The system trust store at `/etc/ssl` is **not** allowed. Ory Network and OEL binaries embed Mozilla's CA bundle via
`golang.org/x/crypto/x509roots/fallback` and run with `godebug x509usefallbackroots=1`, so `crypto/x509` never reads the system
store at runtime. Operators who need to trust an additional CA must point `SSL_CERT_FILE` or `SSL_CERT_DIR` at the file or
directory and list it under `security.landlock.allowed_paths`.

All other filesystem access is denied by the kernel after activation. This includes the `/proc` and `/sys` virtual filesystems.

`os.TempDir` (typically `/tmp`) is granted read-write only when SQLite is the configured DSN — SQLite stores some temporary files
there. With CockroachDB, PostgreSQL, or MySQL, `kratos serve` never writes to the temp directory, so it is left out of the
allowlist for a tighter sandbox.

## Configuration

The sandbox is enabled by default. Two options control it:

```yaml
security:
landlock:
# Set to true to opt out completely. Not recommended in production.
disabled: false
# Extra paths to allow. Directories grant access to every file underneath;
# individual files grant access only to themselves.
allowed_paths:
- /etc/kratos/schemas/fragments/address.json
- /etc/ssl/my-corporate-ca.pem
```

### Hot reload

Landlock restrictions are irrevocable for the lifetime of the process. Hot-reloading a config that flips
`security.landlock.disabled` from `false` to `true` does not lift the sandbox — the process must be restarted for the change to
take effect. Other config changes (allowlist entries, courier templates, and so on) reload normally, but newly-introduced paths
are only honoured on the next process start.

Check warning on line 88 in docs/security-compliance/landlock-sandbox.mdx

View workflow job for this annotation

GitHub Actions / misspell

[misspell] docs/security-compliance/landlock-sandbox.mdx#L88

"honoured" is a misspelling of "honored"
Raw output
./docs/security-compliance/landlock-sandbox.mdx:88:9: "honoured" is a misspelling of "honored"

## Symlinks

Symlinks in any configured path — `--config` files, TLS paths, `identity.schemas[].url`, the SQLite DSN,
`security.landlock.allowed_paths`, and so on — are followed by the kernel when the rule is added at startup. The rule attaches to
the inode that the symlink resolves to at that moment. As long as the target does not change, accesses through the symlink keep
working transparently.

Landlock rules are irrevocable, so when a symlink target swaps at runtime — for example when `cert-manager` or `certbot` renews a
certificate by writing a new file and re-pointing the symlink — the rule still references the original target inode. What happens
next depends on the shape of the grant:

- **Leaf grant on a file.** The rule covers only the original target inode. The renewed target is a fresh inode that is not in the
allowlist, and the kernel denies reads through the symlink with `EPERM`. Restart `kratos serve` after the swap so the rules
re-attach to the new inodes; with automated renewal, wire the restart into the renew hook.
- **Grant on a containing directory.** The rule covers every inode underneath. If both the symlink and the renewed target sit
under the granted directory — as they do with certbot (`/etc/letsencrypt/live/<domain>/...` and
`/etc/letsencrypt/archive/<domain>/...`, both under `/etc/letsencrypt`) or a typical cert-manager volume mount — the swap is
transparent and no restart is needed.

To make cert renewals robust, add the cert directory (for example `/etc/letsencrypt`) to `security.landlock.allowed_paths` instead
of relying on the per-file grants Kratos derives from the TLS path configuration.

## Local `$ref` in identity schemas

Auto-discovery walks the loaded **config**, not the JSON bodies that the config points at. `$ref` references inside an identity
schema body that target local files (`"$ref": "file:///path/to/fragment.json"`) are therefore not picked up — the top-level
`identity.schemas[].url` is allowed, but the schema content is not parsed. If a schema relies on a local `$ref`, the referenced
file must be listed under `security.landlock.allowed_paths`, otherwise schema compilation fails because the kernel denies the
read.

```yaml
security:
landlock:
allowed_paths:
- /etc/kratos/schemas/fragments/address.json
identity:
schemas:
- id: customer
url: file:///etc/kratos/schemas/customer.json # auto-allowed
```

…where `customer.json` contains:

```json
{ "$ref": "file:///etc/kratos/schemas/fragments/address.json" }
```

This applies only to schemas that are split across multiple local files via `$ref`. Schemas served over HTTPS, inlined as
`base64://`, or kept in a single file need no extra configuration.

## Troubleshooting

A path that the sandbox does not allow surfaces in the application as `EPERM` ("Operation not permitted") on `open(2)`,
`openat(2)`, or `execve(2)`. Kratos typically logs this as `permission denied` while loading a config file, schema, template, or
TLS material. To distinguish a Landlock denial from a regular Unix permission error, work through the steps below.

### 1. Confirm the sandbox is the cause

Check the Kratos startup logs for:

```
level=info msg="Landlock filesystem sandbox is active."
```

Just before it, two log lines list every path that was added to the allowlist:

```
level=info msg="Landlock: collected roPaths." roPaths=[...]
level=info msg="Landlock: collected rwDirs." rwDirs=[...]
```

If the path that triggered `EPERM` is missing from both lists, Landlock is the cause. As a sanity check, restart with
`security.landlock.disabled: true`: if the error disappears, the denial came from the sandbox.

### 2. Read the kernel audit log

On Linux 6.10 and later, Landlock emits a kernel audit record for every denied access. The record names the syscall, the resolved
path, and the denied access right:

```bash
sudo journalctl -k --since "5 minutes ago" | grep -i landlock
sudo dmesg -T | grep -i landlock
sudo ausearch -m LANDLOCK_DENY -ts recent # auditd-based distros
```

A typical record looks like:

```
audit: type=1334 audit(...): domain=2 op=fs blockers=fs.read_file path="/etc/kratos/schemas/fragments/address.json" dev="vda1" ino=131072
```

The `path=` field is exactly what to add to `security.landlock.allowed_paths`. Older kernels (5.13 – 6.9) do not emit these
records — fall back to step 3 there.

### 3. Trace the syscall directly

When the audit log is unavailable or the path is templated, attach `strace` to the running process and watch for `EPERM` on the
relevant syscalls:

```bash
sudo strace -f -p "$(pgrep -f 'kratos serve')" -e trace=openat,execve -e status=failed
```

Lines ending in `= -1 EPERM (Operation not permitted)` show the exact path the kernel rejected, even when Kratos's own log message
has been swallowed by a wrapper.

### 4. Fix the configuration

Once the offending path is known, add it to the allowlist (a directory grants every file underneath; a file grants only itself),
then restart `kratos serve` — Landlock rules are immutable for the lifetime of the process, so a hot reload will not lift the
denial.

```yaml
security:
landlock:
allowed_paths:
- /etc/kratos/schemas/fragments/address.json
```
Loading