Skip to content

Full native ast coverage #19

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
alex-clickhouse merged 2 commits into from
Mar 11, 2026
Merged

Full native ast coverage #19

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
fb73a7c
add support for selecting protocol version in native format
alex-clickhouse Mar 11, 2026
b0bd5b4
asto coverage of the native format to 100%
alex-clickhouse Mar 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ A tool for visualizing ClickHouse RowBinary and Native format data. Features an
## Features

- **Format support**: RowBinary and Native, modular system allows adding more
- **Native protocol version**: Select the Native `client_protocol_version` to inspect revision-specific wire layouts
- **Hex Viewer**: Virtual-scrolling hex display with ASCII column
- **AST Tree**: Collapsible tree view showing decoded structure
- **Interactive Highlighting**: Selecting a node in the tree highlights corresponding bytes in the hex view (and vice versa)
Expand Down Expand Up @@ -73,6 +74,7 @@ Open http://localhost:5173
- Click nodes in the AST tree to highlight bytes
- Click bytes in the hex viewer to select the corresponding node
- Use "Expand All" / "Collapse All" to navigate complex structures
4. When using `Native`, choose a protocol preset to compare legacy HTTP output against newer revisions such as custom serialization, Dynamic/JSON v2, replicated, and nullable sparse encodings

## Example Queries

Expand All @@ -91,11 +93,24 @@ SELECT 42::Dynamic
SELECT '{"user": {"id": 123}}'::JSON(`user.id` UInt32)
```

## Native Protocol Versions

The `Native` format toolbar exposes upstream protocol milestones from `0` through `54483`. This controls the `client_protocol_version` request parameter and the local decoder behavior, so the explorer can parse:

- legacy HTTP Native blocks without `BlockInfo` (`0`)
- per-column serialization metadata (`54454+`)
- sparse and replicated serialization kinds (`54465+`, `54482+`)
- Dynamic/JSON v2 Native layouts (`54473+`)
- nullable sparse serialization (`54483`)

See [docs/native-protocol-versions.md](docs/native-protocol-versions.md) for the revision-by-revision reference, and [docs/nativespec.md](docs/nativespec.md) for the Native layout details.

## Tech Stack

- React + TypeScript + Vite
- Zustand (state management)
- react-window (virtualized hex viewer)
- react-resizable-panels (split pane layout)
- Electron (desktop app, optional)
- Vitest + testcontainers (integration testing)
- Playwright (e2e testing)
238 changes: 9 additions & 229 deletions src/components/AstTree/AstTree.tsx
View file
Open in desktop

Large diffs are not rendered by default.

145 changes: 10 additions & 135 deletions src/components/HexViewer/HexViewer.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -118,69 +118,8 @@ function buildHighlightMap(
}

// Handle blocks for Native format
parsedData.blocks?.forEach((block, blockIndex) => {
const metadataColor = '#ce93d8'; // Purple for metadata

// Check for block header metadata section (the parent "Header" item)
const blockHeaderId = `block-${blockIndex}-header`;
if (activeNodeId === blockHeaderId || hoveredNodeId === blockHeaderId) {
const isActive = activeNodeId === blockHeaderId;
for (let i = block.header.byteRange.start; i < block.header.byteRange.end; i++) {
const existing = map.get(i);
if (!existing || isActive || !existing.isActive) {
map.set(i, { color: metadataColor, isActive, isHovered: !isActive });
}
}
}

// Check for individual block header items (numColumns, numRows)
const numColsId = `block-${blockIndex}-numcols`;
const numRowsId = `block-${blockIndex}-numrows`;

if (activeNodeId === numColsId || hoveredNodeId === numColsId) {
const isActive = activeNodeId === numColsId;
for (let i = block.header.numColumnsRange.start; i < block.header.numColumnsRange.end; i++) {
const existing = map.get(i);
if (!existing || isActive || !existing.isActive) {
map.set(i, { color: metadataColor, isActive, isHovered: !isActive });
}
}
}

if (activeNodeId === numRowsId || hoveredNodeId === numRowsId) {
const isActive = activeNodeId === numRowsId;
for (let i = block.header.numRowsRange.start; i < block.header.numRowsRange.end; i++) {
const existing = map.get(i);
if (!existing || isActive || !existing.isActive) {
map.set(i, { color: metadataColor, isActive, isHovered: !isActive });
}
}
}

const blockInfoId = `block-${blockIndex}-blockinfo`;
if (block.header.blockInfo && (activeNodeId === blockInfoId || hoveredNodeId === blockInfoId)) {
const isActive = activeNodeId === blockInfoId;
for (let i = block.header.blockInfo.byteRange.start; i < block.header.blockInfo.byteRange.end; i++) {
const existing = map.get(i);
if (!existing || isActive || !existing.isActive) {
map.set(i, { color: metadataColor, isActive, isHovered: !isActive });
}
}
}

block.header.blockInfo?.fields.forEach((field) => {
const fieldId = `block-${blockIndex}-blockinfo-field-${field.fieldNumber}`;
if (activeNodeId === fieldId || hoveredNodeId === fieldId) {
const isActive = activeNodeId === fieldId;
for (let i = field.byteRange.start; i < field.byteRange.end; i++) {
const existing = map.get(i);
if (!existing || isActive || !existing.isActive) {
map.set(i, { color: metadataColor, isActive, isHovered: !isActive });
}
}
}
});

parsedData.blocks?.forEach((block) => {
visitNode(block.header.astNode, 0);
block.columns.forEach((col) => {
// Check if the column itself is active/hovered
const isColActive = col.id === activeNodeId;
Expand All @@ -196,82 +135,15 @@ function buildHighlightMap(
}
}
}

// Check for column metadata section (name + type together)
const colMetaId = `${col.id}-meta`;
const colNameId = `${col.id}-name`;
const colTypeId = `${col.id}-type`;

if (activeNodeId === colMetaId || hoveredNodeId === colMetaId) {
const isActive = activeNodeId === colMetaId;
for (let i = col.metadataByteRange.start; i < col.metadataByteRange.end; i++) {
const existing = map.get(i);
if (!existing || isActive || !existing.isActive) {
map.set(i, { color: metadataColor, isActive, isHovered: !isActive });
}
}
}

if (activeNodeId === colNameId || hoveredNodeId === colNameId) {
const isActive = activeNodeId === colNameId;
for (let i = col.nameByteRange.start; i < col.nameByteRange.end; i++) {
const existing = map.get(i);
if (!existing || isActive || !existing.isActive) {
map.set(i, { color: metadataColor, isActive, isHovered: !isActive });
}
}
}

if (activeNodeId === colTypeId || hoveredNodeId === colTypeId) {
const isActive = activeNodeId === colTypeId;
for (let i = col.typeByteRange.start; i < col.typeByteRange.end; i++) {
const existing = map.get(i);
if (!existing || isActive || !existing.isActive) {
map.set(i, { color: metadataColor, isActive, isHovered: !isActive });
}
}
}

if (col.serializationInfo) {
const serializationId = `${col.id}-serialization`;
if (activeNodeId === serializationId || hoveredNodeId === serializationId) {
const isActive = activeNodeId === serializationId;
for (let i = col.serializationInfo.byteRange.start; i < col.serializationInfo.byteRange.end; i++) {
const existing = map.get(i);
if (!existing || isActive || !existing.isActive) {
map.set(i, { color: metadataColor, isActive, isHovered: !isActive });
}
}
}

const hasCustomId = `${col.id}-serialization-has-custom`;
if (activeNodeId === hasCustomId || hoveredNodeId === hasCustomId) {
const isActive = activeNodeId === hasCustomId;
for (let i = col.serializationInfo.hasCustomRange.start; i < col.serializationInfo.hasCustomRange.end; i++) {
const existing = map.get(i);
if (!existing || isActive || !existing.isActive) {
map.set(i, { color: metadataColor, isActive, isHovered: !isActive });
}
}
}

const kindsId = `${col.id}-serialization-kinds`;
if (col.serializationInfo.kindStackRange && (activeNodeId === kindsId || hoveredNodeId === kindsId)) {
const isActive = activeNodeId === kindsId;
for (let i = col.serializationInfo.kindStackRange.start; i < col.serializationInfo.kindStackRange.end; i++) {
const existing = map.get(i);
if (!existing || isActive || !existing.isActive) {
map.set(i, { color: metadataColor, isActive, isHovered: !isActive });
}
}
}
}

visitNode(col.metadataNode, 0);
col.dataPrefixNodes.forEach((node) => visitNode(node, 0));
// Also visit individual values
col.values.forEach((node) => visitNode(node, 0));
});
});

parsedData.trailingNodes?.forEach((node) => visitNode(node, 0));

return map;
}

Expand Down Expand Up @@ -430,12 +302,15 @@ export function HexViewer() {
parsedData.rows?.forEach((row) => {
row.values.forEach((node) => visitNode(node, 0));
});
// TODO: Handle blocks for Native format
parsedData.blocks?.forEach((block) => {
visitNode(block.header.astNode, 0);
block.columns.forEach((col) => {
visitNode(col.metadataNode, 0);
col.dataPrefixNodes.forEach((node) => visitNode(node, 0));
col.values.forEach((node) => visitNode(node, 0));
});
});
parsedData.trailingNodes?.forEach((node) => visitNode(node, 0));

if (deepestNode) {
setActiveNode((deepestNode as AstNode).id);
Expand Down
140 changes: 79 additions & 61 deletions src/core/decoder/coverage.integration.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,63 @@ import {
formatUncoveredRanges,
} from './test-helpers';
import { SMOKE_TEST_CASES } from './smoke-cases';
import { NATIVE_PROTOCOL_PRESETS } from '../types/native-protocol';

interface NativeCoverageMatrixCase {
name: string;
query: string;
settings?: Record<string, string | number>;
}

const NATIVE_COVERAGE_MATRIX_CASES: NativeCoverageMatrixCase[] = [
{
name: 'simple UInt8 column',
query: 'SELECT number::UInt8 AS val FROM numbers(3)',
},
{
name: 'multiple columns baseline',
query: "SELECT 42::UInt32 as int_col, 'hello'::String as str_col, true::Bool as bool_col, 3.14::Float64 as float_col",
},
{
name: 'Array integers',
query: 'SELECT [1, 2, 3]::Array(UInt32) as val',
},
{
name: 'Tuple simple',
query: "SELECT (42, 'hello')::Tuple(UInt32, String) as val",
},
{
name: 'Map with entries',
query: "SELECT map('a', 1, 'b', 2)::Map(String, UInt32) as val",
},
{
name: 'LowCardinality compatibility',
query: 'SELECT toLowCardinality(toString(number % 2)) AS val FROM numbers(4)',
settings: { allow_suspicious_low_cardinality_types: 1 },
},
{
name: 'AggregateFunction compatibility',
query: 'SELECT avgState(number) AS val FROM numbers(10)',
},
{
name: 'serialization metadata gate',
query: 'SELECT if(number = 5, 1, 0)::UInt8 AS sparse_val FROM numbers(10)',
},
{
name: 'Nullable serialization metadata gate',
query: 'SELECT if(number = 5, 42, NULL)::Nullable(UInt8) AS sparse_nullable FROM numbers(10)',
},
{
name: 'Dynamic serialization version gate',
query: 'SELECT 42::Dynamic AS val',
settings: { allow_experimental_dynamic_type: 1 },
},
{
name: 'JSON dynamic-path serialization version gate',
query: `SELECT '{"ip":"127.0.0.1","name":"test"}'::JSON(ip IPv4) AS val`,
settings: { allow_experimental_json_type: 1 },
},
];

/**
* Byte coverage tests - verify that the AST leaf nodes cover all bytes in the data
Expand All @@ -25,19 +82,6 @@ describe('Byte Coverage Tests', () => {
await ctx.stop();
});

// Test a representative subset of cases for coverage (used by Native format)
const coverageTestCases = SMOKE_TEST_CASES.filter(c =>
// Focus on diverse type categories
c.name.includes('UInt8') ||
c.name.includes('String basic') ||
c.name.includes('Array integers') ||
c.name.includes('Tuple simple') ||
c.name.includes('Map with entries') ||
c.name.includes('Nullable non-null') ||
c.name.includes('Multiple columns') ||
c.name.includes('IntervalSecond')
);

describe('RowBinary Format', () => {
it.each(SMOKE_TEST_CASES)(
'$name - byte coverage',
Expand All @@ -59,53 +103,27 @@ describe('Byte Coverage Tests', () => {
});

describe('Native Format', () => {
it.each(coverageTestCases)(
'$name - byte coverage',
async ({ query, settings, skipNative }) => {
if (skipNative) return;

const data = await ctx.queryNative(query, settings);
const parsed = decodeNative(data);
const coverage = analyzeByteRange(parsed, data.length);

if (!coverage.isComplete) {
const details = formatUncoveredRanges(coverage, data);
console.log(`[Native] ${query}\n${details}`);
}

// Native format has block headers that may not be fully covered
expect(coverage.coveragePercent).toBeGreaterThan(70);
},
);
});

describe('Full Coverage Sanity Checks', () => {
it('simple UInt8 value has reasonable coverage (RowBinary)', async () => {
const data = await ctx.queryRowBinary('SELECT 42::UInt8 as val');
const parsed = decodeRowBinary(data);
const coverage = analyzeByteRange(parsed, data.length);

// Should cover most of the data
expect(coverage.coveragePercent).toBeGreaterThan(50);

// Log uncovered if any
if (!coverage.isComplete) {
console.log('Uncovered ranges:', coverage.uncoveredRanges);
}
});

it('simple UInt8 value has reasonable coverage (Native)', async () => {
const data = await ctx.queryNative('SELECT 42::UInt8 as val');
const parsed = decodeNative(data);
const coverage = analyzeByteRange(parsed, data.length);

// Should cover most of the data
expect(coverage.coveragePercent).toBeGreaterThan(50);

// Log uncovered if any
if (!coverage.isComplete) {
console.log('Uncovered ranges:', coverage.uncoveredRanges);
}
});
for (const testCase of NATIVE_COVERAGE_MATRIX_CASES) {
describe(testCase.name, () => {
it.each(NATIVE_PROTOCOL_PRESETS.map((preset) => preset.value))(
'revision %s - byte coverage',
async (revision) => {
const data = await ctx.queryNative(testCase.query, {
...(testCase.settings ?? {}),
client_protocol_version: revision,
});
const parsed = decodeNative(data, revision);
const coverage = analyzeByteRange(parsed, data.length);

if (!coverage.isComplete) {
const details = formatUncoveredRanges(coverage, data);
console.log(`[Native r${revision}] ${testCase.query}\n${details}`);
}

expect(coverage.isComplete).toBe(true);
},
);
});
}
});
}, 300000);
Loading
Loading