Parser: implement pstring multi-byte length prefix variants (/B, /H, /L)

[unclesp1d3r]

Summary

Implement multi-byte length-prefix variants for the pstring type: /B (1-byte, default), /H (2-byte little-endian), and /L (4-byte little-endian).

Context

GNU libmagic supports length-prefix width suffixes on the pstring type keyword:

• pstring or pstring/B -- 1-byte length prefix (0-255), already implemented in Parser: implement pstring (Pascal string) type #43
• pstring/H or pstring/h -- 2-byte little-endian length prefix (0-65535)
• pstring/L or pstring/l -- 4-byte little-endian length prefix (0-4294967295)

The current implementation only supports the default 1-byte prefix. Magic files using /H or /L suffixes will fail to parse.

Acceptance Criteria

• Add PStringLengthWidth enum (or equivalent) to represent prefix widths
• Extend TypeKind::PString with a field for the length-prefix width
• Parser recognizes pstring/B, pstring/H, pstring/h, pstring/L, pstring/l suffixes
• Evaluator reads 2-byte and 4-byte little-endian length prefixes
• Bounds checking works correctly for all prefix widths
• Unit tests for each variant
• Update documentation

Impact

LOW -- Multi-byte pstring prefixes are uncommon in real-world magic files, but needed for full GNU libmagic compatibility.

Files to Modify

• src/parser/ast.rs -- Add length-width enum, extend PString variant
• src/parser/types.rs -- Parse /B, /H, /L suffixes after pstring keyword
• src/evaluator/types/string.rs -- Extend read_pstring() for 2-byte and 4-byte prefixes
• src/parser/codegen.rs -- Update serialization for new field

References

• GNU file magic(5) man page: pstring/B, pstring/H, pstring/L
• Parent issue: Parser: implement pstring (Pascal string) type #43 (basic pstring implementation)
• Epic Epic: Type System Expansion #54 (Types)

[traycerai]

Warning

Ticket Assist will be discontinued on May 1, 2026. After that date, the feature will no longer be available. We suggest installing the Traycer IDE Extension for continued AI-powered, spec-driven development. Thank you for choosing Traycer!

Plan

Observations

The codebase has a clean layered architecture: TypeKind enum in ast.rs → keyword parsing in types.rs → grammar dispatch in grammar/mod.rs → evaluation dispatch in evaluator/types/mod.rs → actual reading in evaluator/types/string.rs. There is no existing PString variant in TypeKind — the comment in types.rs line 98 explicitly notes pstring is a planned addition. The codegen.rs serializes TypeKind variants to Rust source for the build-time compiled rules.

Approach

Add a PStringLengthWidth enum and a TypeKind::PString variant, wire it through all layers in order: AST → parser types → grammar → evaluator dispatch → evaluator reader → codegen. This follows the exact same pattern used by TypeKind::String and TypeKind::Date throughout the codebase.

---

Implementation Steps

1. Add PStringLengthWidth enum and TypeKind::PString variant — file:src/parser/ast.rs

• Define a new PStringLengthWidth enum with three variants: Byte (1-byte prefix, default), Short (2-byte little-endian), Long (4-byte little-endian). Derive Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq.
• Add a TypeKind::PString variant with a single field length_width: PStringLengthWidth.
• Extend TypeKind::bit_width() to return None for TypeKind::PString { .. } (same as String).
• Update the #[cfg(test)] serialization test test_type_kind_serialization to include TypeKind::PString variants.

2. Extend type keyword parsing — file:src/parser/types.rs

In parse_type_keyword, add pstring to the string-types alt branch (before string to avoid ambiguity, since pstring starts with p there is no prefix conflict, but keep longest-first ordering):

// String types
alt((
    tag("pstring"),
    tag("string"),
))

In type_keyword_to_kind, add a match arm for "pstring" that returns TypeKind::PString { length_width: PStringLengthWidth::Byte }. The suffix (/B, /H, /L) is not part of the keyword string — it is parsed separately in the grammar (see step 3).

Update the test_roundtrip_all_keywords test to include "pstring".

3. Parse /B, /H, /L suffixes in the grammar — file:src/parser/grammar/mod.rs

The current parse_type_and_operator calls parse_type_keyword and then type_keyword_to_kind. After obtaining the TypeKind, add a suffix-parsing step that runs only when the type is TypeKind::PString:

• After type_keyword_to_kind produces a TypeKind::PString, attempt to consume an optional / followed by one of B, b, H, h, L, l using nom::combinator::opt + nom::bytes::complete::tag.
• Map the suffix character to the appropriate PStringLengthWidth variant:
  • B / b → PStringLengthWidth::Byte (default, same as no suffix)
  • H / h → PStringLengthWidth::Short
  • L / l → PStringLengthWidth::Long
• Replace the length_width field in the returned TypeKind::PString accordingly.
• Update the doc comment on parse_type to list pstring, pstring/B, pstring/H, pstring/L.

4. Add read_pstring to the evaluator — file:src/evaluator/types/string.rs

Add a new public function read_pstring(buffer: &[u8], offset: usize, length_width: PStringLengthWidth) -> Result<Value, TypeReadError>:

• PStringLengthWidth::Byte: Read 1 byte at offset as u8 → len. Bounds-check that offset + 1 + len <= buffer.len(). Return the len bytes starting at offset + 1 as a lossy UTF-8 string.
• PStringLengthWidth::Short: Read 2 bytes at offset as u16::from_le_bytes → len. Bounds-check that offset + 2 + len <= buffer.len(). Return the len bytes starting at offset + 2.
• PStringLengthWidth::Long: Read 4 bytes at offset as u32::from_le_bytes → len. Bounds-check that offset + 4 + len <= buffer.len(). Return the len bytes starting at offset + 4.
• Return TypeReadError::BufferOverrun for any out-of-bounds access.
• Re-export read_pstring from file:src/evaluator/types/mod.rs alongside read_string.

5. Wire TypeKind::PString into the evaluator dispatch — file:src/evaluator/types/mod.rs

In read_typed_value, add a match arm:

TypeKind::PString { length_width } => read_pstring(buffer, offset, *length_width),

Also update coerce_value_to_type — since PString produces Value::String, no coercion is needed; the existing wildcard _ => value.clone() arm already handles it correctly.

6. Update strength calculation — file:src/evaluator/strength.rs

In the match &rule.typ block that computes type strength, add a TypeKind::PString { .. } arm that mirrors the TypeKind::String arm (same base strength of 20, since pstrings are equally specific byte-sequence matches).

7. Update codegen serialization — file:src/parser/codegen.rs

In serialize_type_kind, add a match arm for TypeKind::PString { length_width }:

• Import PStringLengthWidth in the use statement at the top of the file (or use the full path).
• Emit TypeKind::PString { length_width: PStringLengthWidth::Byte } / ::Short / ::Long as appropriate.
• Update the use import line in generate_builtin_rules to include PStringLengthWidth.

8. Update build_helpers.rs — file:src/build_helpers.rs

If build_helpers.rs duplicates any serialize_type_kind logic (the grep shows it has a test_serialize_type_kind_string test), ensure the PString arm is covered there too, consistent with step 7.

---

Data Flow Summary

sequenceDiagram
    participant MF as Magic File Text
    participant TK as parse_type_keyword (types.rs)
    participant GR as parse_type_and_operator (grammar/mod.rs)
    participant AST as TypeKind::PString (ast.rs)
    participant EV as read_typed_value (evaluator/types/mod.rs)
    participant RD as read_pstring (evaluator/types/string.rs)

    MF->>TK: "pstring/H ..."
    TK->>GR: keyword = "pstring", rest = "/H ..."
    GR->>GR: parse optional /B|/H|/L suffix
    GR->>AST: TypeKind::PString { length_width: Short }
    AST->>EV: dispatch on PString variant
    EV->>RD: read_pstring(buffer, offset, Short)
    RD-->>EV: Value::String(content)

Loading

Import In IDE

🤖 Prompt for AI Agents

## Observations

The codebase has a clean layered architecture: `TypeKind` enum in `ast.rs` → keyword parsing in `types.rs` → grammar dispatch in `grammar/mod.rs` → evaluation dispatch in `evaluator/types/mod.rs` → actual reading in `evaluator/types/string.rs`. There is **no existing `PString` variant** in `TypeKind` — the comment in `types.rs` line 98 explicitly notes `pstring` is a planned addition. The `codegen.rs` serializes `TypeKind` variants to Rust source for the build-time compiled rules.

## Approach

Add a `PStringLengthWidth` enum and a `TypeKind::PString` variant, wire it through all layers in order: AST → parser types → grammar → evaluator dispatch → evaluator reader → codegen. This follows the exact same pattern used by `TypeKind::String` and `TypeKind::Date` throughout the codebase.

---

## Implementation Steps

### 1. Add `PStringLengthWidth` enum and `TypeKind::PString` variant — `file:src/parser/ast.rs`

- Define a new `PStringLengthWidth` enum with three variants: `Byte` (1-byte prefix, default), `Short` (2-byte little-endian), `Long` (4-byte little-endian). Derive `Debug`, `Clone`, `Copy`, `Serialize`, `Deserialize`, `PartialEq`, `Eq`.
- Add a `TypeKind::PString` variant with a single field `length_width: PStringLengthWidth`.
- Extend `TypeKind::bit_width()` to return `None` for `TypeKind::PString { .. }` (same as `String`).
- Update the `#[cfg(test)]` serialization test `test_type_kind_serialization` to include `TypeKind::PString` variants.

### 2. Extend type keyword parsing — `file:src/parser/types.rs`

In `parse_type_keyword`, add `pstring` to the string-types `alt` branch (before `string` to avoid ambiguity, since `pstring` starts with `p` there is no prefix conflict, but keep longest-first ordering):

```
// String types
alt((
    tag("pstring"),
    tag("string"),
))
```

In `type_keyword_to_kind`, add a match arm for `"pstring"` that returns `TypeKind::PString { length_width: PStringLengthWidth::Byte }`. The suffix (`/B`, `/H`, `/L`) is **not** part of the keyword string — it is parsed separately in the grammar (see step 3).

Update the `test_roundtrip_all_keywords` test to include `"pstring"`.

### 3. Parse `/B`, `/H`, `/L` suffixes in the grammar — `file:src/parser/grammar/mod.rs`

The current `parse_type_and_operator` calls `parse_type_keyword` and then `type_keyword_to_kind`. After obtaining the `TypeKind`, add a suffix-parsing step that runs only when the type is `TypeKind::PString`:

- After `type_keyword_to_kind` produces a `TypeKind::PString`, attempt to consume an optional `/` followed by one of `B`, `b`, `H`, `h`, `L`, `l` using `nom::combinator::opt` + `nom::bytes::complete::tag`.
- Map the suffix character to the appropriate `PStringLengthWidth` variant:
  - `B` / `b` → `PStringLengthWidth::Byte` (default, same as no suffix)
  - `H` / `h` → `PStringLengthWidth::Short`
  - `L` / `l` → `PStringLengthWidth::Long`
- Replace the `length_width` field in the returned `TypeKind::PString` accordingly.
- Update the doc comment on `parse_type` to list `pstring`, `pstring/B`, `pstring/H`, `pstring/L`.

### 4. Add `read_pstring` to the evaluator — `file:src/evaluator/types/string.rs`

Add a new public function `read_pstring(buffer: &[u8], offset: usize, length_width: PStringLengthWidth) -> Result<Value, TypeReadError>`:

- **`PStringLengthWidth::Byte`**: Read 1 byte at `offset` as `u8` → `len`. Bounds-check that `offset + 1 + len <= buffer.len()`. Return the `len` bytes starting at `offset + 1` as a lossy UTF-8 string.
- **`PStringLengthWidth::Short`**: Read 2 bytes at `offset` as `u16::from_le_bytes` → `len`. Bounds-check that `offset + 2 + len <= buffer.len()`. Return the `len` bytes starting at `offset + 2`.
- **`PStringLengthWidth::Long`**: Read 4 bytes at `offset` as `u32::from_le_bytes` → `len`. Bounds-check that `offset + 4 + len <= buffer.len()`. Return the `len` bytes starting at `offset + 4`.
- Return `TypeReadError::BufferOverrun` for any out-of-bounds access.
- Re-export `read_pstring` from `file:src/evaluator/types/mod.rs` alongside `read_string`.

### 5. Wire `TypeKind::PString` into the evaluator dispatch — `file:src/evaluator/types/mod.rs`

In `read_typed_value`, add a match arm:

```
TypeKind::PString { length_width } => read_pstring(buffer, offset, *length_width),
```

Also update `coerce_value_to_type` — since `PString` produces `Value::String`, no coercion is needed; the existing wildcard `_ => value.clone()` arm already handles it correctly.

### 6. Update strength calculation — `file:src/evaluator/strength.rs`

In the `match &rule.typ` block that computes type strength, add a `TypeKind::PString { .. }` arm that mirrors the `TypeKind::String` arm (same base strength of 20, since pstrings are equally specific byte-sequence matches).

### 7. Update codegen serialization — `file:src/parser/codegen.rs`

In `serialize_type_kind`, add a match arm for `TypeKind::PString { length_width }`:

- Import `PStringLengthWidth` in the use statement at the top of the file (or use the full path).
- Emit `TypeKind::PString { length_width: PStringLengthWidth::Byte }` / `::Short` / `::Long` as appropriate.
- Update the `use` import line in `generate_builtin_rules` to include `PStringLengthWidth`.

### 8. Update `build_helpers.rs` — `file:src/build_helpers.rs`

If `build_helpers.rs` duplicates any `serialize_type_kind` logic (the grep shows it has a `test_serialize_type_kind_string` test), ensure the `PString` arm is covered there too, consistent with step 7.

---

## Data Flow Summary

```mermaid
sequenceDiagram
    participant MF as Magic File Text
    participant TK as parse_type_keyword (types.rs)
    participant GR as parse_type_and_operator (grammar/mod.rs)
    participant AST as TypeKind::PString (ast.rs)
    participant EV as read_typed_value (evaluator/types/mod.rs)
    participant RD as read_pstring (evaluator/types/string.rs)

    MF->>TK: "pstring/H ..."
    TK->>GR: keyword = "pstring", rest = "/H ..."
    GR->>GR: parse optional /B|/H|/L suffix
    GR->>AST: TypeKind::PString { length_width: Short }
    AST->>EV: dispatch on PString variant
    EV->>RD: read_pstring(buffer, offset, Short)
    RD-->>EV: Value::String(content)
```

---

Execution Information

Branch: main
Commit: 8be6924

---

💡 Tips

Supported Commands (Inside Comments)

• Use @traycerai generate to iterate on the previous version of the implementation plan.

Supported Commands (Inside Description)

• Add @traycerai ignore anywhere in the ticket description to prevent this ticket from being processed.
• Add @traycerai branch:<branch-name> anywhere in the ticket description to specify the target branch for the implementation plan.

Community

• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.