Add documentation (first pass)

README.md

@@ -1,6 +1,18 @@
 # pulp-rust
 
-A Pulp plugin to support hosting your own Rust package registry.
+A Pulp plugin to support hosting your own Rust/Cargo package registry.
+
+> **Tech Preview**: This project is currently in tech preview. APIs, behaviors, and data models are subject to change, including breaking changes, without prior notice.
+
+## Features
+
+- Use Pulp as a pull-through cache for crates.io or any Cargo sparse registry
+- Host a private Cargo registry for internal crates
+- Implements the [Cargo sparse registry protocol](https://doc.rust-lang.org/cargo/reference/registry-index.html#sparse-index) for compatibility with standard Cargo tooling
+- Download crates on-demand to reduce disk usage
+- Every operation creates a restorable snapshot with Versioned Repositories
+- Host content either locally or on S3/Azure/GCP
+- De-duplication of all saved content
 
 For more information, please see the [documentation](docs/index.md) or the [Pulp project page](https://pulpproject.org/).
 

docs/index.md

@@ -1,12 +1,20 @@
-# Welcome to Pulp 
+# Welcome to Pulp Rust
 
-The `` plugin extends pulpcore to support hosting  packages.
-This plugin is a part of the Pulp Project, and assumes some familiarity with the
-[pulpcore documentation](site:pulpcore/).
+The `rust` plugin extends [pulpcore](site:pulpcore/) to support hosting Rust/Cargo package registries. This plugin is a part of the [Pulp Project](site:/), and assumes some familiarity with the [pulpcore documentation](site:pulpcore/).
 
-If you are just getting started, we recommend:
+!!! warning "Tech Preview"
+    This plugin is currently in **tech preview**. APIs, behaviors, and data models are subject to change, including breaking changes, without prior notice.
 
-- [Getting Started with ](site:pulp_rust/docs/tutorials/getting-started.md),
-  for a starting out with a common use case.
+If you are just getting started, we recommend getting to know the [basic workflows](site:pulp_rust/docs/user/guides/pull-through-cache/).
 
+See the [REST API documentation](site:pulp_rust/restapi/) for detailed endpoint reference.
 
+## Features
+
+- [Use Pulp as a pull-through cache](site:pulp_rust/docs/user/guides/pull-through-cache/) for crates.io or any Cargo sparse registry
+- [Host a private Cargo registry](site:pulp_rust/docs/user/guides/private-registry/) for internal crates
+- Implements the [Cargo sparse registry protocol](https://doc.rust-lang.org/cargo/reference/registry-index.html#sparse-index) for compatibility with standard Cargo tooling
+- Download crates on-demand to reduce disk usage
+- Every operation creates a restorable snapshot with Versioned Repositories
+- Host content either locally or on S3/Azure/GCP
+- De-duplication of all saved content

docs/user/guides/_SUMMARY.md

@@ -0,0 +1,2 @@
+* [Pull-Through Cache](pull-through-cache.md)
+* [Host a Private Registry](private-registry.md)

docs/user/guides/private-registry.md

@@ -0,0 +1,131 @@
+# Host a Private Cargo Registry
+
+This guide walks you through setting up Pulp as a private Cargo registry for hosting internal
+crates. This is useful for organizations that need to distribute proprietary or internal-only
+Rust packages.
+
+!!! note
+    Package publishing support (`cargo publish`) is not yet available but is planned for an
+    upcoming release. In the meantime, content can be uploaded through the Pulp REST API.
+
+## Create a Repository
+
+```bash
+pulp rust repository create --name my-crates
+```
+
+## Create a Distribution
+
+A distribution makes the repository's content available to Cargo over HTTP.
+
+```bash
+pulp rust distribution create \
+    --name my-crates \
+    --base-path my-crates \
+    --repository my-crates
+```
+
+Your private registry is now served at `http://<pulp-host>/pulp/cargo/my-crates/`.
+
+## Configure Cargo
+
+Add the private registry to your Cargo configuration. Create or edit `~/.cargo/config.toml`:
+
+```toml
+[registries.my-crates]
+index = "sparse+http://<pulp-host>/pulp/cargo/my-crates/"
+```
+
+### Using the Private Registry as a Dependency Source
+
+To depend on crates from your private registry, specify the registry in your `Cargo.toml`:
+
+```toml
+[dependencies]
+my-internal-lib = { version = "1.0", registry = "my-crates" }
+```
+
+### Setting the Default Registry
+
+You can set your private registry as the default for `cargo publish` and other registry commands
+so you don't need to pass `--registry` every time:
+
+```toml
+[registry]
+default = "my-crates"
+```
+
+This affects commands like `cargo publish`, `cargo yank`, and `cargo owner`. It does **not**
+change where dependencies are resolved from — that is controlled by source replacement (below).
+
+!!! tip
+    Setting a default registry is recommended for organizations with private crates. Without it,
+    running `cargo publish` without `--registry` will publish to crates.io by default, which could
+    accidentally leak proprietary code to the public registry.
+
+### Replacing crates.io Entirely
+
+If you want all crate lookups to go through your private registry (for example, in an air-gapped
+environment), you can replace the default source:
+
+```toml
+[source.crates-io]
+replace-with = "my-crates"
+
+[source.my-crates]
+registry = "sparse+http://<pulp-host>/pulp/cargo/my-crates/"
+```
+
+This redirects all dependency resolution — including transitive dependencies — through your
+private registry. Any crate not present in the registry will fail to resolve.
+
+## Combining with Pull-Through Caching
+
+If you need both private crates and public crates.io dependencies, we recommend keeping them as
+**separate registries** rather than mixing them into one. This avoids
+[dependency confusion](https://medium.com/@alex.birsan/dependency-confusion-4a5d60fec610) attacks,
+where a malicious package on a public registry could impersonate a private dependency.
+
+```bash
+# Set up a separate pull-through cache for crates.io
+pulp rust remote create --name crates-io --url "sparse+https://index.crates.io/" --policy on_demand
+pulp rust repository create --name crates-io-cache --remote crates-io --retain-repo-versions 1
+pulp rust distribution create \
+    --name crates-io-cache \
+    --base-path crates-io-cache \
+    --repository crates-io-cache \
+    --remote crates-io
+```
+
+Then configure Cargo to use both registries, with crates.io going through the cache and private
+crates resolved from your private registry:
+
+```toml
+[registries.my-crates]
+index = "sparse+http://<pulp-host>/pulp/cargo/my-crates/"
+
+[source.crates-io]
+replace-with = "crates-io-cache"
+
+[source.crates-io-cache]
+registry = "sparse+http://<pulp-host>/pulp/cargo/crates-io-cache/"
+```
+
+```toml
+[dependencies]
+serde = "1.0"                                              # resolved from crates-io-cache
+my-internal-lib = { version = "1.0", registry = "my-crates" }  # resolved from private registry
+```
+
+!!! warning
+    Avoid adding a public remote (such as crates.io) to a private registry's distribution. Mixing
+    public and private packages in a single registry index creates a risk of dependency confusion
+    attacks, where an attacker publishes a crate on the public registry with the same name as one
+    of your private crates.
+
+## Further Reading
+
+- [Cargo registries configuration](https://doc.rust-lang.org/cargo/reference/registries.html) -- configuring alternate registries in Cargo
+- [Cargo source replacement](https://doc.rust-lang.org/cargo/reference/source-replacement.html) -- replacing crates.io with an alternate source
+- [Cargo config reference](https://doc.rust-lang.org/cargo/reference/config.html) -- full reference for `.cargo/config.toml`
+- [Specifying dependencies](https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html#specifying-dependencies-from-other-registries) -- using dependencies from alternate registries

docs/user/guides/pull-through-cache.md

@@ -0,0 +1,109 @@
+# Use Pulp as a Pull-Through Cache
+
+This guide walks you through setting up Pulp as a pull-through cache for an upstream Cargo registry
+such as [crates.io](https://crates.io). Cargo will fetch crates through Pulp, which transparently
+downloads and stores them on first access. Subsequent requests are served directly from Pulp's
+storage backend.
+
+## Create a Remote
+
+A remote tells Pulp where to fetch crates from. The URL should point to the sparse index of
+the upstream registry.
+
+```bash
+pulp rust remote create --name crates-io --url "sparse+https://index.crates.io/" --policy on_demand
+```
+
+The `--policy` flag controls how content is stored:
+
+| Policy      | Behavior                                                                                    |
+|-------------|---------------------------------------------------------------------------------------------|
+| `on_demand` | Downloads and caches crate files on first request. Metadata is saved to the Pulp database.  |
+| `streamed`  | Streams crate files to the client without saving them locally.                              |
+
+For a pull-through cache, `on_demand` is recommended so that crates are retained for future
+requests. `streamed` is primarily useful in cases where you want a proxy in between clients and
+the public registry for e.g. access control - which is not so common in the Rust ecosystem.
+
+## Create a Repository
+
+Repositories store and organize content in Pulp. When using pull-through caching, crates are
+automatically added to the repository as they are downloaded.
+
+```bash
+pulp rust repository create --name crates-io-cache --remote crates-io --retain-repo-versions 1
+```
+
+!!! tip
+    Setting `--retain-repo-versions 1` is recommended for pull-through caches. Each new crate
+    download creates a new repository version, and without this setting the number of versions
+    will grow unboundedly.
+
+## Create a Distribution
+
+A distribution makes the repository available to Cargo over HTTP. The `--base-path` determines the
+URL path where the registry is served.
+
+```bash
+pulp rust distribution create \
+    --name crates-io-cache \
+    --base-path crates-io-cache \
+    --repository crates-io-cache \
+    --remote crates-io
+```
+
+!!! note
+    The remote must be set on both the repository and the distribution. The distribution's remote
+    controls the proxy fallback for the sparse index, while the repository's remote is used for
+    pull-through content storage.
+
+Your registry is now available at `http://<pulp-host>/pulp/cargo/crates-io-cache/`.
+
+## Configure Cargo
+
+To use your Pulp instance as a registry, add it to your Cargo configuration. Create or edit
+`~/.cargo/config.toml`:
+
+```toml
+[registries.pulp]
+index = "sparse+http://<pulp-host>/pulp/cargo/crates-io-cache/"
+
+[source.crates-io]
+replace-with = "pulp"
+
+[source.pulp]
+registry = "pulp"
+```
+
+This tells Cargo to route all crate downloads through Pulp instead of directly to crates.io.
+
+!!! tip
+    You can also set this per-project by placing a `.cargo/config.toml` file in your project root.
+
+Now any `cargo build`, `cargo add`, or `cargo install` command will fetch crates through Pulp:
+
+```bash
+cargo add serde --features derive
+cargo build
+```
+
+The first time a crate is requested, Pulp fetches it from the upstream registry and caches it
+locally. All subsequent requests for the same crate version are served from Pulp's storage backend.
+
+## Verify Cached Content
+
+You can inspect what content has been cached in Pulp:
+
+```bash
+# List cached crates
+pulp rust content list --limit 10
+
+# Check repository versions
+pulp rust repository version list --repository crates-io-cache
+```
+
+## Further Reading
+
+- [Cargo registries configuration](https://doc.rust-lang.org/cargo/reference/registries.html) -- configuring alternate registries in Cargo
+- [Cargo source replacement](https://doc.rust-lang.org/cargo/reference/source-replacement.html) -- replacing crates.io with an alternate source
+- [Cargo config reference](https://doc.rust-lang.org/cargo/reference/config.html) -- full reference for `.cargo/config.toml`