BWeave (Binary Weave) is a hybrid binary data compression library written in pure TypeScript, designed for high-load real-time applications (WebSocket, RTC, game telemetry, financial flows).
The key idea is to automatically select the most appropriate compression algorithm for each data fragment, ensuring minimal latency with an acceptable compression ratio.
About | Get Started | Internal Structure | Benchmarks and Tests | Contact Me
Unlike general-purpose algorithms (zlib, Brotli), BWeave doesn't use entropy coding (Huffman), allowing it to operate synchronously without micro-delays due to promises and event queues. At the same time, its hybrid approach and pre-trained dictionaries achieve compression close to LZ4 and Snappy, but with better performance on repeating data.
πΉ Lightweight library with zero dependencies;
πΉ Flexible hybrid compression library written in Typescript;
πΉ Extremely fast (great for realtime and IoT applications);
πΉ Production ready with benchmarks;
- Automatic compression selection algorithm;
- Pre-trained dictionaries β 6 ready-made dictionaries (JSON, HTTP, HTML, JS, binary, Protobuf) increase compression by 15β30% without sacrificing speed.
- Parallel compression β splitting into blocks and parallel processing via Promise.all.
- Dictionary caching (dictCache) β remembers the last LZ77 window for compressing successive similar packets.
- CRC32 checksum β optional data integrity check.
From NPM:
npm install bweaveFrom GitHub:
git clone https://github.com/devsdaddy/bweave/bweave.git
cd ./bweave/Basic usage:
import { BWeave, BWeaveUtils } from 'bweave';
const compressor = new BWeave();
const data = BWeaveUtils.textToBytes(JSON.stringify({ id: 123, name: 'Alice' }));
const compressed = compressor.compress(data);
const decompressed = compressor.decompress(compressed);
console.log('Uncompressed size: ', data.length, 'Compressed size:', compressed.length);Pre-trained dictionaries:
import { BWeave, BWeaveUtils, PresetDictionaries } from 'bweave';
const dict = PresetDictionaries.json;
const compressor = new BWeave({
autoMode: false,
forcedMode: BWeaveMode.LZ77,
sharedDict: dict
});
const jsonData = JSON.stringify({
id: 123,
name: 'John Doe',
value: 99.5,
data: { items: [1, 2, 3], total: 6, page: 1, limit: 10, offset: 0 },
});
const data = BWeaveUtils.textToBytes(jsonData);
const compress = compressor.compress(data);Parallel compression:
import { BWeaveCompressorParallel } from 'bweave';
const parallel = new BWeaveCompressorParallel({ autoMode: true });
const bigData = new Uint8Array(1000000); // 1 ΠΠ
const compressed = await parallel.compress(bigData, 32768);
const decompressed = await parallel.decompress(compressed);BWeave contains 6 strategies, each optimized for a specific data type:
| Mode | Purpose | Principle |
|---|---|---|
| LZ77 | Text, JSON, HTML, CSS, Logs | Sliding window (256β16384 bytes) with duplicate phrase search; link is encoded in 2 bytes (distance up to 4095, length up to 18). |
| RLE | Binary repetitions (palettes, masks, monochrome images) | Replacing a series of identical bytes with a "counter + value" pair. |
| DELTA | Monotonic sequences (time series, audio samples, coordinates) | Calculate differences between adjacent bytes, then compress with RLE or LZ77. |
| JSON_WEAVE | JSON with repeating schema | Tokenization of keys (replacement with short integers), then LZ77. |
| DEDUP_WEAVE | Binary protocols with repeating blocks | Fixed-size block deduplication (64 bytes) β replacing duplicates with links. |
| STORE | Incompressible data (random, already compressed) | Direct copying without processing. |
The detector analyzes the first 1024 bytes (or more during test compression) and selects the best mode based on heuristics:
- RLE β if the proportion of repeated bytes is > 60%.
- DELTA β if the proportion of monotonic steps is > 40%.
- JSON_WEAVE β if the JSON scheme provides a better compression result during test compression.
- LZ77 β for texts with entropy < 5.5 bits/byte.
- STORE β in all other cases.
A quick test compression is also implemented (we compress the sample using each algorithm and select the best one by size) β optional, for maximum adaptability.
- The escape byte
0xFEis used to denote the literal0xFE(doubled) and the beginning of a reference. - Window reference: [
0xFE,distHigh,distLow], where distance is12 bits(1β4096) and length is4 bits(3β18). - Dictionary reference: [
0xFE,0xFD,dictHigh,dictLow] β points to a static pretrained dictionary (up to 4096 bytes). - Adaptive window size:
256 bytesfor data<1 KB,4096for1β64 KB,16384for>64 KB.
The library includes six built-in dictionaries (each ~4β8 KB):
- json β Frequent JSON keys and values.
- http β HTTP/1.1 headers.
- html β HTML tags and attributes.
- js β JavaScript keywords.
- binary β Null blocks, byte patterns, pcap/ELF/PE/PNG headers.
- protobuf β Varint and field key / wire type.
The dictionaries are passed when creating the compressor (presetDict: 'json' or sharedDict: myDict). They increase the compression ratio by 15β30% on structured data without sacrificing speed.
Header (4 or 8 bytes):
[ mode (1 byte) | len (3 bytes) ] + [ CRC32 (4 bytes, optional) ]
Where:
- mode β the low-order 7 bits (0β5). The high-order bit is the CRC flag.
- len β the original block length (up to 16 MB).
The header is separated from the compressed data.
Comparison table with popular solutions:
| Parameter | BWeave | zlib | Brotli | Snappy |
|---|---|---|---|---|
| Compression rate, Text | ~ 3.4:1 | ~5.2:1 | ~5.5:1 | ~2.4:1 |
| Compression rate, HTML / JSON | ~ 4.5:1 | ~5.1:1 | ~5.5:1 | ~2.4:1 |
| Compression rate, binary | ~ 12:1 | ~16.1:1 | ~20:1 | ~6.1:1 |
| Compression rate, telemetry | ~ 15:1 | ~20.1:1 | ~21:1 | ~7.1:1 |
| Compression speed, Text | ~480mb/s | ~130mb/s | ~45mb/s | ~650mb/s |
| Compression speed, HTML / JSON | ~210mb/s | ~120mb/s | ~40mb/s | ~600mb/s |
| Compression speed, binary | ~780mb/s | ~110mb/s | ~30mb/s | ~720mb/s |
| Compression speed, telemetry | ~520mb/s | ~110mb/s | ~30mb/s | ~500mb/s |
| Automatic mode selection | β | β | β | β |
| JSON-tokenize | β | β | β | β |
| Block deduplication | β | β | β | β |
| Dictionary Cache | β | β | β | β |
| Parallel Compression | β | β | β | β |
| CRC32 Check | β | β | β | β |
- Socket-based chats / games: Synchronous compression of each message, dictCache for repeating commands, low latency.
- Financial Tickers (FIX/FAST): DELTA mode is ideal for compressing sequences of numbers; RLE is for repeating values.
- Real-time log streaming: JSON_WEAVE reduces the size of keys by 4-6 times; the speed allows processing thousands of messages per second.
- Browser application with WebRTC: Pure JS, works on all platforms without WASM; small library size.
- IoT: RLE and DELTA are effective for byte series and monotonic signals.
- API: Pre-trained dictionaries for JSON provide compression close to zlib, but 4 times faster.
- Text compression is lower than zlib/Brotli due to the lack of entropy encoding. For data storage or transmission over slow links, zlib is better suited.
- For very short messages (<50 bytes), the overhead of the header (4 bytes) can negate the gain.
- Parallel compression requires additional memory and may be excessive for very small packets.
BWeave isn't just another zlib replacement, but a specialized tool for real-time applications where speed and predictability are more important than maximum compression. It offers a unique combination:
- LZ4-like (or sometimes better) performance on text.
- Approaching zlib-like performance on repetitive and structured data (thanks to RLE, DELTA, and dictionaries).
- Advanced features (parallelism, checksums) unavailable in competitors.
- Full compatibility with browsers and Node.js without native modules.
If you need to compress traffic in real time without compromising latency, BWeave is the best choice. If minimizing storage space is more important, use zlib or Brotli. BWeave doesn't replace them, but it occupies its own niche, where no other library offers such a balance.
BWeave compression library is distributed under the MIT license. You can use it however you like. I would appreciate any feedback and suggestions for improvement. Full license text can be found here