Feature/streams

.markdownlint.yaml

@@ -0,0 +1,5 @@
+default: true
+extends: null
+
+MD013: false
+

CHANGELOG.md

@@ -1,4 +1,22 @@
-## 4.0.0
+<!-- markdownlint-disable-file MD025 -->
+
+# 5.0.0
+
+- **BREAKING CHANGES:**
+  - **Stream API**: rewritten architecture — `StreamBinaryReader` now `implements TransactionalReader<List<int>>` with delegation to state
+  - **`NotEnoughDataException`**: message changed from `required $required bytes` to `required $required`
+
+- **New Features:**
+  - **Strings**: added `writeStringFixed` and `readStringFixed` for fixed-width length-prefixed strings.
+  - **Stream API**: added `TransactionalReader<TChunk>` interface for abstracting transactional read model
+  - **Stream API**: added `ChunkedTransactionalState<TChunk>` base class for managing chunk queue and bookmarks
+  - **Stream API**: added `TransactionalStreamTransformer<TMessage, TChunk, TReader>` — generic `StreamTransformer` for parsing streams
+  - **Stream API**: `BinaryStreamTransformer` now extends `TransactionalStreamTransformer`
+  - **BinaryWriter**: improved `writeVarString` documentation with examples
+  - **BinaryReader**: `peekByte()` added bounds check — throws `RangeError`
+  - **Examples**: full restructuring — removed `example/main.dart`, added `example/basic/`, `example/network_streaming/`, `example/file_streaming/`
+
+# 4.0.0
 
 **BREAKING CHANGES:**
 
@@ -20,7 +38,7 @@
 
 **Tests:** Improved all tests
 
-## 3.2.0
+# 3.2.0
 
 **BREAKING CHANGES:**
 
@@ -50,7 +68,7 @@
 
 - Added tests for pool statistics, edge cases takeBytes/reset/release
 
-## 3.1.0
+# 3.1.0
 
 - **feat**: Added `BinaryWriterPool.withWriter()` for safer and more concise object pool usage.
 - **feat**: Added modern API features for a more idiomatic experience:
@@ -68,7 +86,7 @@
 - **docs**: Fixed minor typos and improved documentation for `BinaryWriterPool`.
 - **docs**: Complete README overhaul with a focus on recipes and technical clarity.
 
-## 3.0.0
+# 3.0.0
 
 **Improvements:**
 
@@ -97,12 +115,12 @@
 
 - **fix**: Resolved known issues
 
-## 2.2.0
+# 2.2.0
 
  **test**: Added integration tests for new error handling features
  **deps**: Update internal dependencies to latest versions
 
-## 2.1.0
+# 2.1.0
 
 - **feat**: Added detailed error messages with context (offset, available bytes)
 - **feat**: Added `toBytes()` method in `BinaryWriter` (returns buffer without reset)
@@ -114,16 +132,16 @@
 - **test**: Added new tests for boundary checks and new methods
 - **docs**: Updated documentation with better examples and error handling
 
-## 2.0.0
+# 2.0.0
 
 - Update dependencies
 - sdk: ^3.6.0
 
-## 1.1.1
+# 1.1.1
 
 - fix: warnings
 
-## 1.1.0
+# 1.1.0
 
 - fix: Increased test coverage, providing more comprehensive validation for edge cases.
 - performance: Optimized buffer management to reduce memory reallocations and improve efficiency.
@@ -138,15 +156,15 @@
   - feat: Added `bytesRead` property to monitor the total number of bytes read from the buffer.
   - feat: Introduced `reset` method, allowing users to reset the reading position to the start of the buffer for convenient re-reading.
 
-## 1.0.2
+# 1.0.2
 
 - docs: Updated documentation.
 
-## 1.0.1
+# 1.0.1
 
 - docs: Updated documentation.
 - feat: Added `example` directory with basic usage examples.
 
-## 1.0.0
+# 1.0.0
 
 - Initial release.

README.md

@@ -4,23 +4,23 @@
 [![Tests](https://github.com/pro100andrey/pro_binary/workflows/Tests/badge.svg)](https://github.com/pro100andrey/pro_binary/actions)
 [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)
 
-**High-performance binary serialization for Dart.** Optimized for speed, zero-copy reads, and Protocol Buffers-compatible encoding.
+**High-performance binary serialization and deserialization for Dart.** Optimized for high-frequency network protocols, real-time streaming, and fast local storage. Features zero-copy reads, object pooling, and transactional stream parsing.
 
 ## Key Features
 
-* **Zero-Copy Reads**: Operations return `Uint8List` views without allocation.
-* **One-Pass Strings**: Optimized `writeVarString` with optimistic shift (30% faster).
-* **Smart Buffering**: Exponential growth (×1.5) and object pooling.
-* **Compact Encoding**: VarInt & ZigZag support
-* **Stream Parsing**: `StreamBinaryReader` and `BinaryStreamTransformer` for async data.
-* **Universal**: Supports Native & Web (WASM/JS) with consistent API.
-* **Modern API**: Leverages Dart Extension Types for zero-overhead abstractions.
+* **Extreme Performance:** Built from the ground up for speed. Leverages Dart Extension Types for zero-overhead abstractions and direct memory manipulation.
+* **Zero-Copy Reads:** Deserialization operations return `Uint8List` views instead of allocating new memory arrays, significantly reducing GC (Garbage Collector) pauses.
+* **One-Pass String Encoding:** Features a highly optimized `writeVarString` with optimistic size estimation and native memory shifting. Up to **~30% faster** than standard `utf8.encode`.
+* **Zero-Allocation Object Pooling:** Includes built-in `BinaryWriterPool` to reuse writer instances. Perfect for high-frequency network packets (e.g., game servers, WebSockets).
+* **Compact Encoding:** Native support for VarInt and ZigZag encoding to shrink payload sizes for integers.
+* **Transactional Stream Parsing:** Easily process fragmented asynchronous data chunks using `StreamBinaryReader` with `bookmark()` and `rollback()` capabilities.
+* **Cross-Platform:** 100% pure Dart. Works seamlessly across Native (AOT/JIT) and Web (WASM/JS) with a consistent, predictable API.
 
 ## Installation
 
 ```yaml
 dependencies:
-  pro_binary: ^4.0.0
+  pro_binary: ^5.0.0
 ```
 
 ## Quick Start
@@ -50,6 +50,7 @@ final reader2 = BinaryReader.fromList(bytesList);
 ## Recipes & Patterns
 
 ### 1. Efficient Object Serialization
+
 ```dart
 class User {
   final int id;
@@ -70,41 +71,55 @@ class User {
 ```
 
 ### 2. High-Frequency writes (Pooling)
+
 Avoid GC pressure by reusing writer instances.
 
 **Recommended (Safe & Concise):**
+
 ```dart
 final data = BinaryWriterPool.withWriter((writer) {
   writer.writeUint32(1);
   writer.writeVarString('Dart Rocks!');
-  return writer.toBytes(); // View of the buffer
+
+  // toBytes(): returns a zero-copy VIEW. Use for immediate processing (e.g. socket.add).
+  // takeBytes(): detaches the buffer and RESETS the writer. Safe for returning data.
+  return writer.takeBytes(); 
 });
 ```
 
 **Low-level API:**
+
 ```dart
 final writer = BinaryWriterPool.acquire();
 try {
   writer.writeUint32(1);
   writer.writeVarString('Dart Rocks!');
+
   final data = writer.toBytes();
+  socket.add(data); // Process data BEFORE releasing back to the pool
 } finally {
   BinaryWriterPool.release(writer);
 }
 ```
 
 ### 3. Stream Parsing (Async Binary Messages)
+
 Process binary data arriving in chunks over a stream.
 
 **Custom Transformer:**
+
 ```dart
 class MessageParser extends BinaryStreamTransformer<Message> {
   @override
   Message? parse(StreamBinaryReader reader) {
     // Return null when not enough data yet
-    if (!reader.hasBytes(4)) return null;
+    if (!reader.hasBytes(4))  {
+      return null;
+    }
+
     final id = reader.readUint32();
     final name = reader.readVarString();
+
     return Message(id, name);
   }
 }
@@ -114,6 +129,7 @@ stream.transform(MessageParser()).listen((msg) => print(msg));
 ```
 
 **Manual Chunk Reading:**
+
 ```dart
 final reader = StreamBinaryReader();
 reader.addChunk(chunk1);
@@ -130,6 +146,7 @@ try {
 ```
 
 ### 4. Binary Packets (Manual navigation)
+
 ```dart
 final reader = BinaryReader(bytes);
 final type = reader[0]; // Absolute peek via operator []
@@ -139,24 +156,56 @@ if (reader.hasBytes(4)) {
 }
 ```
 
+## Examples
+
+Explore the [example](example/) directory for complete, runnable projects:
+
+* [Basic Usage](example/basic/): Simple serialization and deserialization.
+* [File Streaming](example/file_streaming/): Reading and writing large binary files using streams.
+* [Network Streaming](example/network_streaming/): Implementing a custom protocol for TCP/Socket data.
+
 ## API Overview
 
-Full API documentation: https://pub.dev/documentation/pro_binary/latest/pro_binary/
+[Full API documentation](https://pub.dev/documentation/pro_binary/latest/pro_binary/)
 
-| Class                            | Description                                                                                                        |
-| -------------------------------- | ------------------------------------------------------------------------------------------------------------------ |
-| **BinaryWriter**                 | Encode data: fixed types, VarInt/ZigZag, strings, bytes. Supports `takeBytes()`, `toBytes()`, `reset()`, `seek()`. |
-| **BinaryReader**                 | Decode data: all fixed/variable types, navigation (`skip`, `seek`, `rewind`, `peek`), `rebind()` for reuse.        |
-| **StreamBinaryReader**           | Async streaming: chunk-based reading with `bookmark`/`rollback`/`commit` transactional model.                      |
-| **BinaryStreamTransformer\<T\>** | Stream parser: extend and implement `parse()` to process binary streams.                                           |
-| **BinaryWriterPool**             | Object pool: `acquire()`/`release()` or `withWriter()` for high-frequency writes.                                  |
-| **getUtf8Length**                | Utility: calculate UTF-8 byte length without encoding.                                                             |
+| Component | Description |
+| --------- | ----------- |
+| **BinaryWriter** | Fast encoder for fixed-width, VarInt/ZigZag, and one-pass strings. Features automatic expansion and pooling. |
+| **BinaryReader** | Zero-copy decoder with advanced navigation (`seek`, `rewind`, `peek`). Optimized for performance. |
+| **StreamBinaryReader** | Handles async data chunks seamlessly with a transactional `bookmark`/`rollback` model for partial data. |
+| **BinaryStreamTransformer** | The easiest way to parse a `Stream<List<int>>` into a stream of typed messages or objects. |
+| **BinaryWriterPool** | Object pool for `BinaryWriter` to eliminate GC pressure during high-frequency write operations. |
+| **getUtf8Length** | High-speed utility to calculate UTF-8 byte length without encoding (O(n) but heavily optimized). |
+| **TransactionalReader** | Base interface for custom transactional readers. Used internally by `StreamBinaryReader`. |
 
 ## Performance
 
 Run benchmarks to see it in action:
+
 ```bash
-dart run benchmark_harness:bench --flavor aot --target test/performance/serialization_bench.dart
+# Serialization (Writer)
+dart run performance/serialization_bench.dart
+
+# Deserialization (Reader)
+dart run performance/deserialization_bench.dart
+
+# String encoding (One-pass vs Two-pass vs Standard)
+dart run performance/strings_bench.dart
+
+# Object Pooling (GC impact mitigation)
+dart run performance/pool_bench.dart
+```
+
+## Testing
+
+The library is heavily tested with over 200+ unit and integration tests.
+
+```bash
+# Run all tests
+dart test
+
+# Run tests with coverage
+dart test --coverage=coverage
 ```
 
 ## License

example/README.md

@@ -0,0 +1,45 @@
+# pro_binary Examples
+
+This directory contains examples demonstrating how to use `pro_binary` effectively in various scenarios.
+
+## 📁 Examples Structure
+
+### 1. [Basic Serialization & Streaming](basic/)
+A self-contained example showing the core API:
+*   **Simple Serialization**: How to encode and decode a class.
+*   **Pool API**: Using `BinaryWriterPool` for high-performance applications.
+*   **Basic Streaming**: Implementing a `BinaryStreamTransformer` to parse objects from fragmented byte streams.
+
+### 2. [Advanced Network Streaming](network_streaming/)
+A multi-file architectural example simulating a real-world IoT/Telemetry protocol:
+*   **Protocol Framing**: Searching for sync bytes (magic bytes).
+*   **Nested Models**: Encoding/decoding complex structures with lists.
+*   **Fragmentation Resilience**: Proving that the parser can reconstruct packets from tiny network chunks.
+
+### 3. [File Streaming (Big Data)](file_streaming/)
+A high-performance example demonstrating how to process large binary files:
+*   **Incremental Processing**: Using `File.openRead()` to process data without loading the entire file into RAM.
+*   **Market Data Simulation**: Parsing 250,000+ trade records (Market Ticks) on-the-fly.
+*   **Memory Efficiency**: Maintaining a constant memory footprint regardless of file size.
+
+---
+
+## 🚀 How to Run
+
+You can run any example directly using the Dart CLI:
+
+```bash
+# Run the basic overview
+dart example/basic/main.dart
+
+# Run the advanced telemetry simulation
+dart example/network_streaming/main.dart
+
+# Run the big data file streaming example
+dart example/file_streaming/main.dart
+```
+
+## 🛠 Best Practices Demonstrated
+*   **Model Design**: Using `factory Model.decode(BinaryReader r)` and `void encode(BinaryWriter w)`.
+*   **Efficiency**: Utilizing `takeBytes()` to reuse writers and `BinaryReader.rebind()` for readers.
+*   **Robustness**: Handling `NotEnoughDataException` for asynchronous data sources.