feat(kvp): add store trait and Hyper-V KVP storage crate

[peytonr18]

What this PR introduces

• Adds workspace crate libazureinit-kvp.
• Adds libazureinit-kvp/src/lib.rs, libazureinit-kvp/src/store.rs, and libazureinit-kvp/src/error.rs.
• Adds crate dependencies: libc, tracing. Dev-dependencies: rstest, tempfile.
• This is step 1 of the abstraction — the crate exists and is exercised by its own tests, but is not yet wired into libazureinit. A follow-up PR will replace libazureinit/src/kvp.rs with this implementation.

KVP design

Single concrete type — no trait indirection in the public API.

• KvpPoolStore: file-backed KVP pool store.
• KvpPool: enum identifying one of the five Hyper-V pool indices.
• PoolMode: policy for write-path size limits.
• KvpError: error type.

KvpPoolStore API

Construction:

• KvpPoolStore::new(pool, mode) — uses the default directory /var/lib/hyperv.
• KvpPoolStore::new_in(pool, dir, mode) — uses a caller-supplied directory (file name derived from pool).

Reads / queries (do not enforce mode size limits on the key argument; a Safe-mode store can read keys written in Unsafe mode):

• read(key) -> Result<Option<String>, KvpError> — last-write-wins on duplicate keys.
• entries() -> Result<HashMap<String, String>, KvpError> (deduplicated, last-write-wins).
• dump() -> Result<Vec<(String, String)>, KvpError> (on-disk order, preserves duplicates).
• len() -> Result<usize, KvpError> (on-disk record count, not unique keys).
• is_empty() -> Result<bool, KvpError>
• is_stale() -> Result<bool, KvpError>

Writes (enforce PoolMode key and value size limits):

• insert(key, value) -> Result<(), KvpError> — upsert; overwrites first match in place, removes any further duplicates, and enforces the 1024 unique-key cap for new keys.
• append(key, value) -> Result<(), KvpError> — blind append; no duplicate check, no unique-key cap. Intended for log/telemetry workloads.
• populate(records) -> Result<(), KvpError> — atomic replace-all. Validates the full batch (including the unique-key cap) before locking, then truncates and rewrites under one exclusive lock. Preserves duplicates in iteration order. Inverse of dump.
• delete(key) -> Result<bool, KvpError> — removes all records matching the key; returns true if any were present.
• clear() -> Result<(), KvpError> — truncates the file.
• clear_if_stale() -> Result<(), KvpError> — reads boot time, takes the exclusive lock, then truncates only if the file's mtime is <= boot time. The check happens under the write lock so a concurrent writer can't extend mtime between the staleness check and the truncate.

Metadata accessors:

• pool() -> KvpPool
• mode() -> PoolMode
• path() -> &Path
• max_key_size() -> usize
• max_value_size() -> usize

Pool identity

KvpPool enumerates the five standard Hyper-V KVP pool indices:

Variant	File	Role
External	.kvp_pool_0	Host-to-guest, pushed by host admin
Guest	.kvp_pool_1	Guest-to-host; where cloud-init / azure-init write
Auto	.kvp_pool_2	Guest intrinsics generated by the daemon
AutoExternal	.kvp_pool_3	Host-originated data describing the host
AutoInternal	.kvp_pool_4	Undocumented; no pool file exists

KvpPool::default_dir() returns /var/lib/hyperv. KvpPool::default_path() returns the full default path for a pool.

On-disk format and limits

Hyper-V wire format: fixed-size records, identical across environments.

• key field: 512 bytes (zero-padded UTF-8)
• value field: 2048 bytes (zero-padded UTF-8)
• record size: 2560 bytes
• maximum unique keys: 1024 (enforced by insert and populate, not append)

All sizes are UTF-8 byte counts measured against the Linux on-disk layout. The Windows-side Hyper-V spec expresses limits in UTF-16 code units (256 / 1024), which equal the 512 / 2048-byte fields used by hv_kvp_daemon on Linux.

PoolMode controls write-side size limits:

Mode	Max key	Max value	Notes
Safe	254 bytes	1022 bytes	Recommended for Linux kernel compatibility (2 bytes under the kernel HV_KVP_EXCHANGE_MAX_* maximums to leave room for a NUL terminator on either side of the boundary)
Unsafe	512 bytes	2048 bytes	Full wire-format maximums

Read-side operations do not enforce these limits, so a Safe-mode store can read or delete keys written in Unsafe mode.

Locking: both flock and fcntl OFD

The store takes both flock and fcntl open file description (OFD) locks for every operation:

• Acquire flock first (LOCK_EX for writes, LOCK_SH for reads). If the subsequent fcntl lock fails, the flock is released before returning the error (all-or-nothing acquisition).
• Then fcntl(F_OFD_SETLKW, ...) with F_WRLCK for writes or F_RDLCK for reads, range l_start = 0, l_len = 0 (whole file).
• Linux flock(2) and fcntl(2) byte-range locks live in disjoint kernel domains and do not coordinate with each other. hv_kvp_daemon uses fcntl; cloud-init uses flock(LOCK_EX). Holding both is the only way to coordinate with either.
• OFD fcntl locks (F_OFD_SETLKW) share a domain with classic POSIX fcntl byte-range locks but are per-fd and thread-safe — compatible with hv_kvp_daemon while remaining safe across threads sharing a process.
• Successful locks are released when the file descriptor closes (via Drop on the handle), so there is no explicit unlock in normal paths.

File I/O behavior

Standard file I/O via OpenOptions, read_exact, write_all, set_len, seek, flush.

• open_read_write_create() sets .truncate(false) to avoid implicit truncation on open and .mode(0o600) so newly created pool files are private to the owning user.
• Iteration is handled by an internal KvpPoolIter that holds the file and keeps the lock for its lifetime, and supports in-place value overwrite and swap-remove. The lock is released when the iterator is dropped.

Flows:

• insert — open read+write+create, take exclusive locks, iterate records (collecting unique keys into a set). If the key exists, overwrite the first match in place and remove any further duplicates via swap-with-last + truncate. If the key is new, enforce the 1024 unique-key cap and append at EOF. Flush; locks released on drop.
• append — open read+write+create, take exclusive locks, seek to EOF, write a new record. No duplicate check, no unique-key cap. Flush.
• populate — validate the full batch up front (key/value sizes and unique-key cap), then open read+write+create, take exclusive locks, set_len(0), and write every record. An I/O error mid-write may leave the file partially written.
• delete — open read+write, take exclusive locks, iterate records. For each match, swap with the final record and truncate by one. Flush only if at least one record was removed.
• clear — open read+write, take exclusive locks, set_len(0).
• read / entries / dump — open read-only, take shared locks, decode records.

All missing-file paths return well-defined sentinels (None, false, empty collections, 0, false) instead of erroring.

Malformed-file handling:

• A file whose size is not a multiple of the 2560-byte record size is rejected with an Io error from record_count_from_len on any operation that opens it for iteration or calls len(). Recover by calling clear().
• Records whose key or value fields are not valid UTF-8 surface as Io(InvalidData) from decode_record. Trailing zero bytes in either field are trimmed during decode.

Stale-data handling

• is_stale() compares the pool file's mtime to the system boot time, derived by parsing the btime field of /proc/stat.
• clear_if_stale() is a convenience that clears the file only when stale.
• Stale detection is explicit and caller-controlled — no automatic truncation occurs during construction.

Validation

Key and value validation are implemented as free functions:

• validate_key(key, max) — rejects empty keys, null bytes, and lengths greater than max.
• validate_value(value, max) — rejects null bytes and lengths greater than max.

insert, append, and populate validate using limits derived from the store's PoolMode. read and delete validate the key's content (non-empty, no nulls) but pass usize::MAX for the size cap so they don't reject otherwise-valid keys that were written in a different PoolMode.

Error model

KvpError variants:

• EmptyKey
• KeyContainsNull
• KeyTooLarge { max, actual }
• ValueContainsNull
• ValueTooLarge { max, actual }
• MaxUniqueKeysExceeded { max }
• Io(io::Error)

Implements Display, Error (with source() chaining to the inner io::Error), and From<io::Error>.

Testability: internal SysOps / Handle traits

To exercise error paths that are difficult to trigger with real syscalls (lock failures, mid-write I/O errors, transient filesystem failures, deterministic boot times), the store is implemented against two internal traits:

• SysOps — filesystem opens, path_metadata, boot_time.
• Handle — per-file read, write, seek, set_len, metadata, flock_*, fcntl_*.

OsSysOps / OsHandle are the production implementations and are constructed by default by new / new_in. Tests can substitute a fault-injecting backend via the crate-private KvpPoolStore::with_ops constructor. The public API exposes only the concrete KvpPoolStore (which is Clone + Debug; the inner SysOps is held behind an Arc).

Test coverage

Comprehensive unit tests cover validation, lock interactions, fault injection via the SysOps / Handle traits, concurrent reader/writer scenarios, stale-data detection, and all documented error paths. 100% line coverage on Codecov.

[Copilot]

Pull request overview

Adds a new standalone workspace crate (libazureinit-kvp) that defines a storage abstraction (KvpStore) and a production Hyper-V pool-file implementation (HyperVKvpStore), along with updated technical reference documentation.

Changes:

• Introduces libazureinit-kvp crate with KvpStore and KvpLimits (Hyper-V/Azure presets).
• Implements Hyper-V binary pool-file backend with flock-based concurrency and stale-file truncation support.
• Rewrites doc/libazurekvp.md as a technical reference for the new crate/API.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
Cargo.toml	Adds libazureinit-kvp to the workspace members.
libazureinit-kvp/Cargo.toml	Defines the new crate and its dependencies (fs2, sysinfo).
libazureinit-kvp/src/lib.rs	Defines KvpStore, KvpLimits, and exports HyperVKvpStore plus size constants.
libazureinit-kvp/src/hyperv.rs	Implements Hyper-V pool-file encoding/decoding and the KvpStore backend, plus unit tests.
doc/libazurekvp.md	Updates documentation to describe the new crate’s record format, semantics, truncation behavior, and limits.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[cjp256]

should we use iter_mut() here?

[peytonr18]

I think this is correct as is because delete() opens the file via open_for_read_write() where there isn't a create call and then the function exits with Ok(false) if the file is missing. using iter_mut() would open the file via open_for_read_write_create() and subsequently create an empty pool file as a side effect from trying to delete from it

[cjp256]

we want to take both fcntl and flock locks:

this lock works for hv_kvp_daemon, but cloud-init uses flock(LOCK_EX)

https://github.com/canonical/cloud-init/blob/c7ae6a4cc0dad5a411945e0644dab6926d41a997/cloudinit/reporting/handlers.py#L286

[cjp256]

maybe add a lock_for_reading() and lock_for_writing()

and the semantic here, if needed, perhaps should be read_only: bool, because we want to guarantee the locks for any write operations

[cjp256]

why last index? doesn't this reverse ordering if copying the last record and moving it to current?

[peytonr18]

I added some comments to make the iteration logic clearer here, but let me know if you still have questions/concerns and I can rework the approach!

[cjp256]

    /// Exposes a bug where `remove_current()` reorders duplicate
    /// records via its swap-with-last strategy, silently changing
    /// which duplicate `entries()` reports as last-write-wins.
    ///
    /// Initial on-disk order has `("dup", "C")` as the final record
    /// for key `"dup"`, so `entries()["dup"] == "C"`. After deleting
    /// the unrelated `"middle"` key, the swap moves `("dup", "C")`
    /// into the freed slot ahead of `("dup", "B")`, leaving `"B"` as
    /// the on-disk-last record for `"dup"`.
    #[test]
    fn test_delete_reorders_duplicates_breaking_last_write_wins() {
        let dir = TempDir::new().unwrap();
        let store = safe_store(dir.path());

        store
            .populate(pairs([
                ("dup", "A"),
                ("middle", "m"),
                ("dup", "B"),
                ("other", "x"),
                ("dup", "C"),
            ]))
            .unwrap();

        assert_eq!(
            store.entries().unwrap().get("dup"),
            Some(&"C".to_string()),
            "precondition: last-write-wins picks the final on-disk record",
        );

        assert!(store.delete("middle").unwrap());

        assert_eq!(
            store.entries().unwrap().get("dup"),
            Some(&"C".to_string()),
            "deleting an unrelated key must not change which duplicate \
             entries() reports as the winning value",
        );
    }

    /// Companion to
    /// [`test_delete_reorders_duplicates_breaking_last_write_wins`]
    /// for the other production caller of `remove_current()`: the
    /// duplicate-collapsing branch of [`KvpPoolStore::insert`].
    ///
    /// Initial on-disk order has `("dup", "C")` as the final record
    /// for key `"dup"`, so `entries()["dup"] == "C"`. Inserting
    /// `"ins"` overwrites the first `"ins"` record in place and then
    /// calls `remove_current()` on the second `"ins"`, which swaps
    /// the tail `("dup", "C")` forward of `("dup", "B")`, leaving
    /// `"B"` as the on-disk-last record for `"dup"`.
    #[test]
    fn test_insert_reorders_duplicates_breaking_last_write_wins() {
        let dir = TempDir::new().unwrap();
        let store = safe_store(dir.path());

        store
            .populate(pairs([
                ("ins", "i1"),
                ("ins", "i2"),
                ("dup", "B"),
                ("dup", "C"),
            ]))
            .unwrap();

        assert_eq!(
            store.entries().unwrap().get("dup"),
            Some(&"C".to_string()),
            "precondition: last-write-wins picks the final on-disk record",
        );

        store.insert("ins", "new").unwrap();

        assert_eq!(
            store.entries().unwrap().get("dup"),
            Some(&"C".to_string()),
            "insert collapsing its own duplicates must not change which \
             duplicate of an unrelated key entries() reports as winning",
        );
    }

failures:

---- store::tests::test_delete_reorders_duplicates_breaking_last_write_wins stdout ----

thread 'store::tests::test_delete_reorders_duplicates_breaking_last_write_wins' (1817184) panicked at libazureinit-kvp/src/store.rs:1341:9:
assertion `left == right` failed: deleting an unrelated key must not change which duplicate entries() reports as the winning value
  left: Some("B")
 right: Some("C")
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

---- store::tests::test_insert_reorders_duplicates_breaking_last_write_wins stdout ----

thread 'store::tests::test_insert_reorders_duplicates_breaking_last_write_wins' (1817203) panicked at libazureinit-kvp/src/store.rs:1382:9:
assertion `left == right` failed: insert collapsing its own duplicates must not change which duplicate of an unrelated key entries() reports as winning
  left: Some("B")
 right: Some("C")


failures:
    store::tests::test_delete_reorders_duplicates_breaking_last_write_wins
    store::tests::test_insert_reorders_duplicates_breaking_last_write_wins

test result: FAILED. 100 passed; 2 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.72s

error: test failed, to rerun pass `-p libazureinit-kvp --lib`

[cjp256]

or for a dump-based view

    /// Exposes a bug where `remove_current()` reorders duplicate
    /// records via its swap-with-last strategy, silently changing
    /// which duplicate `entries()` reports as last-write-wins.
    ///
    /// Initial on-disk order has `("dup", "C")` as the final record
    /// for key `"dup"`, so `entries()["dup"] == "C"`. After deleting
    /// the unrelated `"middle"` key, the swap moves `("dup", "C")`
    /// into the freed slot ahead of `("dup", "B")`, leaving `"B"` as
    /// the on-disk-last record for `"dup"`.
    #[test]
    fn test_delete_reorders_duplicates_breaking_last_write_wins() {
        let dir = TempDir::new().unwrap();
        let store = safe_store(dir.path());

        store
            .populate(pairs([
                ("dup", "A"),
                ("middle", "m"),
                ("dup", "B"),
                ("other", "x"),
                ("dup", "C"),
            ]))
            .unwrap();

        assert_eq!(
            store.entries().unwrap().get("dup"),
            Some(&"C".to_string()),
            "precondition: last-write-wins picks the final on-disk record",
        );

        assert!(store.delete("middle").unwrap());

        assert_eq!(
            store.dump().unwrap(),
            pairs([
                ("dup", "A"),
                ("dup", "B"),
                ("other", "x"),
                ("dup", "C"),
            ]),
            "deleting an unrelated key must preserve the relative order \
             of every other record (so last-write-wins still picks C \
             for `dup`)",
        );
        assert_eq!(
            store.entries().unwrap().get("dup"),
            Some(&"C".to_string()),
            "deleting an unrelated key must not change which duplicate \
             entries() reports as the winning value",
        );
    }

    /// Companion to
    /// [`test_delete_reorders_duplicates_breaking_last_write_wins`]
    /// for the other production caller of `remove_current()`: the
    /// duplicate-collapsing branch of [`KvpPoolStore::insert`].
    ///
    /// Initial on-disk order has `("dup", "C")` as the final record
    /// for key `"dup"`, so `entries()["dup"] == "C"`. Inserting
    /// `"ins"` overwrites the first `"ins"` record in place and then
    /// calls `remove_current()` on the second `"ins"`, which swaps
    /// the tail `("dup", "C")` forward of `("dup", "B")`, leaving
    /// `"B"` as the on-disk-last record for `"dup"`.
    #[test]
    fn test_insert_reorders_duplicates_breaking_last_write_wins() {
        let dir = TempDir::new().unwrap();
        let store = safe_store(dir.path());

        store
            .populate(pairs([
                ("ins", "i1"),
                ("ins", "i2"),
                ("dup", "B"),
                ("dup", "C"),
            ]))
            .unwrap();

        assert_eq!(
            store.entries().unwrap().get("dup"),
            Some(&"C".to_string()),
            "precondition: last-write-wins picks the final on-disk record",
        );

        store.insert("ins", "new").unwrap();

        assert_eq!(
            store.dump().unwrap(),
            pairs([("ins", "new"), ("dup", "B"), ("dup", "C")]),
            "collapsing duplicate `ins` records must preserve the \
             relative order of unrelated `dup` records (so last-write-wins \
             still picks C for `dup`)",
        );
        assert_eq!(
            store.entries().unwrap().get("dup"),
            Some(&"C".to_string()),
            "insert collapsing its own duplicates must not change which \
             duplicate of an unrelated key entries() reports as winning",
        );
    }

failures:

---- store::tests::test_delete_reorders_duplicates_breaking_last_write_wins stdout ----

thread 'store::tests::test_delete_reorders_duplicates_breaking_last_write_wins' (1818095) panicked at libazureinit-kvp/src/store.rs:1341:9:
assertion `left == right` failed: deleting an unrelated key must preserve the relative order of every other record (so last-write-wins still picks C for `dup`)
  left: [("dup", "A"), ("dup", "C"), ("dup", "B"), ("other", "x")]
 right: [("dup", "A"), ("dup", "B"), ("other", "x"), ("dup", "C")]
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

---- store::tests::test_insert_reorders_duplicates_breaking_last_write_wins stdout ----

thread 'store::tests::test_insert_reorders_duplicates_breaking_last_write_wins' (1818139) panicked at libazureinit-kvp/src/store.rs:1394:9:
assertion `left == right` failed: collapsing duplicate `ins` records must preserve the relative order of unrelated `dup` records (so last-write-wins still picks C for `dup`)
  left: [("ins", "new"), ("dup", "C"), ("dup", "B")]
 right: [("ins", "new"), ("dup", "B"), ("dup", "C")]

[cjp256]

I think we just have to bite the bullet and redo these to shift everything up (and delete dupe keys in the process). I think if we have to iterate through all entries anyways, the cost isn't really a problem. Callers should avoid introducing duplicate keys anyways so we likely won't hit the worst-case.

[cjp256]

why is this special cased and not handled by iter_mut()? we should be able to see the NotFound error [and ignore it] through that interface too?

[peytonr18]

Good call out - this should be fixed in the most recent commit!

[cjp256]

Great job @peytonr18, thanks!