Skip to content

feat: add hand-crafted llms.txt with build-time validation #784

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
crtahlin wants to merge 4 commits into
base: master
Choose a base branch
from
+290 −150
Open

feat: add hand-crafted llms.txt with build-time validation #784

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
0caa22c
fix: improve /rchash endpoint documentation
crtahlin Feb 12, 2026
b62582c
fix: reconfigure docusaurus-plugin-llms for correct llms-full.txt output
crtahlin Mar 17, 2026
4c482dc
feat: add hand-crafted llms.txt and build-time validation (#838)
crtahlin Mar 20, 2026
865e735
fix: point llms.txt OpenAPI link to docs.ethswarm.org/openapi.yaml
crtahlin Mar 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 23 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,29 @@ npm run swizzle docusaurus-lunr-search SearchBar -- --eject --danger
See the documentation for the above command and the plugin at its github repo [here](https://github.com/praveenn77/docusaurus-lunr-search).


## LLM-Friendly Documentation (`llms.txt`)

The site serves two files for AI agents at the root:

- **`/llms.txt`** — Hand-crafted index file (`static/llms.txt`). A curated, categorised list of every documentation page with one-line descriptions. This is the entry point AI agents use to find relevant pages.
- **`/llms-full.txt`** — Auto-generated by `docusaurus-plugin-llms` at build time. Contains the full text of every page concatenated into a single file.

### Keeping `llms.txt` up to date

The validation script `scripts/validate-llms-txt.mjs` runs automatically during `npm run build` (via the `prebuild` hook). It cross-checks `static/llms.txt` against the actual doc files and prints warnings for:

- **Stale links** — a URL in `llms.txt` points to a doc page that no longer exists (renamed/deleted).
- **Missing coverage** — a doc file exists that isn't listed in `llms.txt` (new page added without updating the index).

The script is **informational only** (exit 0) — it won't block the build.

### What to do when warnings appear

1. **Stale link**: Open `static/llms.txt`, find the flagged URL, and either update the path to match the renamed page or remove the entry if the page was deleted.
2. **Missing coverage**: Decide which section the new page belongs in and add a line in `static/llms.txt` following the existing format: `- [Title](https://docs.ethswarm.org/docs/path): One-line description`. If the page is a landing/index page with no unique content, it's fine to leave it out — the warning is expected.

A few pages are intentionally excluded (intro/landing pages that only contain navigation cards). Their warnings are expected and can be ignored.

## Bumping Version

Don't forget to find and replace the version number for the whole of the docs folder.
Expand Down
173 changes: 39 additions & 134 deletions docs/bee/working-with-bee/bee-api.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -315,7 +315,7 @@ From the results we can see that our node's neighborhood size and batch commitme
* `"lastWonRound"` - The last round number in which your node won the redistribution game.
* `"lastPlayedRound"` - The last round number in which your node participating in the redistribution game. If this number matches the number of the current round shown in `"round"`, then your node is participating in the current round.
* `"lastFrozenRound"` - The last round in which your node was frozen.
* `"lastSelectedRound"` - The last round in which your node's neighborhood was selected. Note that it is possible for your node's neighborhood to be selected without your node playing in the redistribution game. This may potentially indicate your node's hardware is not sufficient to calculate the commitment hash fast enough. See [section on the `/rchash` endpoint](#) for more information.
* `"lastSelectedRound"` - The last round in which your node's neighborhood was selected. Note that it is possible for your node's neighborhood to be selected without your node playing in the redistribution game. This may potentially indicate your node's hardware is not sufficient to calculate the commitment hash fast enough. See [section on the `/rchash` endpoint](#rchash) for more information.
* `"lastSampleDuration"` - The time it took for your node to calculate the sample commitment hash in nanoseconds.
* `"block"` - current Gnosis block number
* `"reward"` - The total all-time reward in PLUR earned by your node.
Expand Down Expand Up @@ -472,163 +472,68 @@ From the results we can see that our node's neighborhood size and batch commitme

### _/rchash_

Calling the `/rchash` endpoint triggers the generation of a reserve commitment hash, which is used in the [redistribution game](/docs/concepts/incentives/redistribution-game), and will report the amount of time it took to generate the hash. This is useful for getting a performance benchmark to ensure that your node's hardware is sufficient.
Calling the `/rchash` endpoint triggers the generation of a reserve commitment hash,
which is used in the [redistribution game](/docs/concepts/incentives/redistribution-game),
and will report the amount of time it took to generate the hash. This is useful for
getting a performance benchmark to ensure that your node's hardware is sufficient.

The `/rchash` endpoint has 3 parameters: `depth` and `anchor_01` and `anchor_02`. For both of the anchor parameters, you should use the first 4 digits from your node's overlay address (which you can find from the `/addresses` endpoint). For depth, you should use the current storage depth of your node which you can find using the `/status` endpoint (storage depth is equivalent to the `storageRadius` value returned from `/status`):
The `/rchash` endpoint has 3 parameters: `depth`, `anchor1`, and `anchor2`.
For both anchor parameters, use the first 4 hex digits from your node's overlay
address (which you can find from the `/addresses` endpoint). For depth, use the
current storage depth of your node from the `/status` endpoint (`storageRadius` value):

```
/rchash/{depth}/{anchor_01}/{anchor_02}
```text
/rchash/{depth}/{anchor1}/{anchor2}
```

:::info anchor parameter details

- The anchor parameters must match the prefix bits of the node's overlay address up to at least the current storage depth (with each hex digit equal to 4 bits).
- The anchor parameters must match the prefix bits of the node's overlay address
up to at least the current storage depth (with each hex digit equal to 4 bits).
- The anchor parameters also must have an even number of digits.

Therefore you can use the first four digits of your node's overlay address since it will work for depths up to depth 16, which will not be approached unless the depth increases up to depth 17, which is not likely to happen in the near future. If it does increase to depth 17, then the first 6 overlay digits should be used.
Therefore you can use the first four digits of your node's overlay address since
it will work for depths up to depth 16, which will not be approached unless the
depth increases up to depth 17, which is not likely to happen in the near future.
If it does increase to depth 17, then the first 6 overlay digits should be used.
:::

```bash
sudo curl -sX GET http://localhost:1633/rchash/10/1e20/1e20 | jq
curl -sX GET http://localhost:1633/rchash/10/1e20/1e20 | jq
```

It should not take much longer than 6 minutes at most for results to be returned:
The response includes a `hash`, `proofs` (used by the redistribution smart contract),
and `durationSeconds` which is the benchmark metric. Here is an example of a
successful result:

```bash
```json
{
"hash": "a1d6e1700dff0c5259029c8a58904251855911eb298b45fab4b0c26d4de0fa5f",
"proofs": {
"proof1": {
"proofSegments": [
"00001099542c8958b2a3a0f7803ee0c07a34de91dd34d4567bf16b367686a387",
"75b51d98c9346a9cb2120bd3a10c0febb43a1d900ff1e4247a29e46974cd4e02",
"d3e289937aa2e2a49e27c9b525580002cbcab4e4d4a972a97ef8e272e0bafebb",
"2215b68c492114e3b4f78080f924bf9cdb3b7da9d5f4e811631f50d2ac45af14",
"c115c46c632fcefea88def9a241fb37acaf222bbbd74a4435d3cde48b6210436",
"0eb01ebfc9ed27500cd4dfc979272d1f0913cc9f66540d7e8005811109e1cf2d",
"887c22bd8750d34016ac3c66b5ff102dacdd73f6b014e710b51e8022af9a1968"
],
"proveSegment": "1e09adea90a0935fdcd03b5b8193b95767f2f40308d4ea29323845de1270cdd7",
"proofSegments2": [
"74656e656666656b74000c4469676974616c20636f707900052e63616c6c000d",
"b5d06699c843bc0cc39abd993553865a01c1366174e9e87539ac491dce0a3b77",
"1513dd5a4a3633ecf370ef6f27af99b0f23d73d4773e504314bbf2c690887999",
"a16bdb02905d76eb17965eca8a32f303f0a6129a125c30767a097f80a9e30ded",
"ec01379e1a2ae10fd4e0ee4e4fc4b0d3f6f9ac55038ced386fc7f4294d4bb103",
"456189653d7fdcd3f1ef6b27d00cb8700cd3abb59a1d60b08b61af1d77c74a13",
"4f5384191e60b983ee26b2d42bdcce3fba7041f4cde80b78d808a893d2dc8e79"
],
"proveSegment2": "436f6465706167652039353000054f72626f7400064d50432d4843000c536569",
"chunkSpan": 4096,
"proofSegments3": [
"74656e656666656b74000c4469676974616c20636f707900052e63616c6c000d",
"d99a7bf301af141ce9e1acb909c631975c2f7bc60984bbfb9d4097415d2ea723",
"f77855426ebacca7833d31b559f8ef0f5909849e8604856ef7c2344e536252bd",
"273b259e9fb51852d8bbfe409c648605f60fcdaec7fff4ec20120d6f5898f412",
"7d0e379980bf75f76e9a128eb05fdfb8d9db7da40060eff06629ee48465b3a70",
"1be5155660d125ee125cceaf3ea81313321b9ac36aefb6b7b3b35c4a4c0fd5b2",
"8b1a6990327bfb0738025c57b6f06d7255e49b89a2d5f4bed1aed202816843ae"
],
"postageProof": {
"signature": "d13f85223e1fe47f3689e31471f322d61b10b25231db5319819c8f68548fdc907acf17a0341ba81114d1667693b1e6f35f37c50b4365474baa03f8b233566fa51b",
"postageId": "9af1b37f38d4af75dbee9ba713b95bcc6d03e1c759ac774ec8bd4864e68d2c03",
"index": "1e0900000e13",
"timeStamp": "17ca57481e839b79"
},
"socProof": null
},
"proof2": {
"proofSegments": [
"0000336f98de7901000b056246b5b105611a56a9e45c452fe65288751f90de2e",
"55f65e720d9d53fc7f6d73c87e9a7fd1ad73343b96bd4fc679fdb0d188fe872c",
"7a753e4aa045cc1ce5834a669755bd43f6344ade1890f0080e5c50c23489f098",
"4081e7d623e62c35a17a1167f7f295a26cbf7705a443ed06dfb8fcc416dfc2e4",
"306564763d673ca314aaa072d79ac4b888b69aad80da3ca387aafd2752afce8e",
"0eb01ebfc9ed27500cd4dfc979272d1f0913cc9f66540d7e8005811109e1cf2d",
"887c22bd8750d34016ac3c66b5ff102dacdd73f6b014e710b51e8022af9a1968"
],
"proveSegment": "1e1b2adb768e9d05021c514914292acc56a8273a4f2ea196f50add7c3e5bff45",
"proofSegments2": [
"cf19af651fdc9ae2aab3c240860b11156691c6561e0291633cc00aea719c3db5",
"91afb2735a5f002134c31fbf4f1b0c2b99b47ea1d84da75a3d8da597e1d27034",
"9b9e75f236b13ae250a9cf3ae59655054583166c6f6379871c5f2c9b1fdcdb1d",
"4cad5229b21c2f1ca0e5dcf34e61253709dcef1d11e9346c9daf6d6a7a7d6ea2",
"ca49b26daaaffb849045d55f25f3a05c942f8b5844c71a8fcc2172c1ba5e2f86",
"3bbf39b295aa2a01a8b6de929d85cb3fc375bddeaf70ac0ad9ff17f64bca3892",
"3a8343e42db633b929473ccbd90e36320cf0bb6a19474c8a3a01dbbc9ebc99ad"
],
"proveSegment2": "4cb106406558d74c99ed246e22f73f55abd96f633c1e00025511b98930faafb4",
"chunkSpan": 4096,
"proofSegments3": [
"cf19af651fdc9ae2aab3c240860b11156691c6561e0291633cc00aea719c3db5",
"8d4665ad0c7ba6a24fe285613057e6289c3b341887fedba9a4190d236f62f31e",
"5fdf57b41fb119b29275edcb55a00b7a0ea5119e1ad78dc97ea615236f9112cb",
"32bc2d3b400b6837117f7720af512913d85388cafe365e06f96172531a59622f",
"d393a441a2dcabfe6ae7a5da201badcd1da877f33092e1f823b816fa826879ac",
"557d9f4ca3655f6bc06f4d13b87da1ddc5c2f7b6eb60b043a7a5a989bb4b2b5f",
"f81cf8329547445e4427e91f8682ef05e482c7b8219adc34066bd9942aec6d05"
],
"postageProof": {
"signature": "efae95247cae78db875194de1851a6866456f9cb881079676289bd3b72f27af53d783441bc1992f5439e1815179afb05b1a12c7a86b36b41fd411fa29529f1981c",
"postageId": "922b7387276adeea51df5aeb80f597a62c7c236b4387f7f06ced2883762f8fef",
"index": "1e1b0000006a",
"timeStamp": "17ad450de2b34084"
},
"socProof": null
},
"proofLast": {
"proofSegments": [
"00003dbfed587dc866be862e65ebf4da79b54397c6dee4cefa2754e8e0ad2f37",
"0b6fa766b60f03b9efec530413243318f277adccdd974d316cf7b52b5f072681",
"1629dad7244d3efcfb5bddcc794cbd42eb5d6e81ecd47711a7ef08ea9c80c405",
"4081e7d623e62c35a17a1167f7f295a26cbf7705a443ed06dfb8fcc416dfc2e4",
"306564763d673ca314aaa072d79ac4b888b69aad80da3ca387aafd2752afce8e",
"0eb01ebfc9ed27500cd4dfc979272d1f0913cc9f66540d7e8005811109e1cf2d",
"887c22bd8750d34016ac3c66b5ff102dacdd73f6b014e710b51e8022af9a1968"
],
"proveSegment": "1e2fbcf6d9a79786ce93970536eb4f18ec148cefd2f5e7e15a203f99ad2b1d0a",
"proofSegments2": [
"1a00fb8c44cf0f65d12d2aed8fcde9517684dd7afc6e3a488cc5a882a92e8cbf",
"5936e75eea7818850651a3e50fab30826766f9777c9099b045c1e6454355111c",
"0ed1fbde9703139dcf9aa8c008c64ac8a85fea8fcfb237986c4652844b49c318",
"f1904237d00d889455695fe711b3f4c7a6e0c40b5e85ae66eefcbf34631b9802",
"19a4dcef98735175b96c675fb1ffd5c59a2fa1a3515724d3d3c2849fa2b75672",
"1449ca5c3b4a701dde9a4dd38f8db6216a5fe8c72e8701dd20263622b3375c77",
"1160eaff3f42ad8c90f8d95fdee20daa5f1fece84947e265878be9db543c4651"
],
"proveSegment2": "57b3f11f2fe2ba52d0897d936901db164d04fa5a9684009a17a39aa718b2b1c8",
"chunkSpan": 4096,
"proofSegments3": [
"1a00fb8c44cf0f65d12d2aed8fcde9517684dd7afc6e3a488cc5a882a92e8cbf",
"5f99524dc31e7bf8ca75011c74f739b30bcb855c3ad7e46701d30dc6901cb563",
"0fc1e41481afc47ae70056c780149a50ab9c48630eb44607b9100dfda0ba9dfe",
"e4d7e158b9a446e8f2bad51acf26fc1874013dc23352471ba945fbec0a97d910",
"dddc32e1f2ac86aa4bb711bb6b812afa5d6ca9405d04816e2dcfefd303eb8294",
"e5384c29f838e85b71bc2eb5cb14d59201e8e1997a9779d193a6a9b327637693",
"6f08720c155d4a53fd94ed403fb4476a4c7e6058bf021df23efdfdca1746027d"
],
"postageProof": {
"signature": "8039da66b4b6e45d5559d0d060f968ccfa11b052234b5645d90d396a0069b6ff40aecb62e9c45d13a68dda0b1fd68dee1de575e28f72c3766d64e824e7c9f9211b",
"postageId": "9af1b37f38d4af75dbee9ba713b95bcc6d03e1c759ac774ec8bd4864e68d2c03",
"index": "1e2f000038a3",
"timeStamp": "17dd336464f4f0bc"
},
"socProof": null
}
},
"durationSeconds": 1191.652640831
"proofs": { "proof1": { ... }, "proof2": { ... }, "proofLast": { ... } },
"durationSeconds": 287.52
}
```

If the `Time` value is much longer than 6 minutes then it likely means that the node's hardware performance is not sufficient. In the example above, it took almost 20 minutes to complete, indicating that the hardware is not sufficient. In such cases, consider upgrading to use faster memory or processor.
The `durationSeconds` value should not exceed roughly 6 minutes (360 seconds).

:::warning Slow results
If `durationSeconds` is much longer than 360 seconds (for example, 1191 seconds /
~20 minutes), the node will likely fail to submit proofs in time during the
redistribution game, resulting in missed rewards or freezing. SSD speed and RAM
are typically the main bottlenecks, since the sampler does heavy random I/O
across the reserve. Upgrade to a faster SSD first, then consider more RAM or a
faster processor.
:::

If while running the `/rchash` command there is an evictions related error such as the one below, try running the call to the `/rchash` endpoint again.
If while running the `/rchash` command there is an evictions related error such
as the one below, try running the call to the `/rchash` endpoint again.

```
```text
error: "level"="error" "logger"="node/storageincentives" "msg"="make sample" "error"="sampler: failed creating sample: sampler stopped due to ongoing evictions"
```

While evictions are a normal part of Bee's standard operation, the event of an eviction will interrupt the sampler process.
While evictions are a normal part of Bee's standard operation, the event of an
eviction will interrupt the sampler process.

### _/health_

Expand Down
28 changes: 13 additions & 15 deletions docusaurus.config.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,23 +35,21 @@ export default {
[
'docusaurus-plugin-llms',
{
// Include core developer documentation
include: [
generateLLMsTxt: false,
generateLLMsFullTxt: true,
title: 'Ethereum Swarm Documentation',
description: 'Swarm is a decentralised storage and communication system for a sovereign digital society.',
excludeImports: true,
removeDuplicateHeadings: true,
includeOrder: [
'docs/concepts/**',
'docs/bee/installation/**',
'docs/bee/working-with-bee/**',
'docs/bee/faq.*',
'docs/develop/**',
'docs/api/**',
'docs/learn/technology/**'
'docs/desktop/**',
'docs/references/**',
],
// Exclude non-essential content
exclude: [
'docs/learn/ecosystem/**',
'docs/desktop/**'
],
// Prioritize essential developer content
priority: {
'docs/develop/getting-started': 'high',
'docs/api/': 'high',
'docs/learn/technology/**': 'medium'
}
}
],
[
Expand Down
2 changes: 1 addition & 1 deletion package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
"scripts": {
"docusaurus": "docusaurus",
"start": "docusaurus start",
"prebuild": "node scripts/fetch-awesome-swarm.mjs",
"prebuild": "node scripts/fetch-awesome-swarm.mjs && node scripts/validate-llms-txt.mjs",
"build": "docusaurus build",
"swizzle": "docusaurus swizzle",
"deploy": "docusaurus deploy",
Expand Down
Loading