Skip to content

Add support for POCO inserts#273

Merged
alex-clickhouse merged 9 commits intomainfrom
poco-inserts
Apr 13, 2026
Merged

Add support for POCO inserts#273
alex-clickhouse merged 9 commits intomainfrom
poco-inserts

Conversation

@alex-clickhouse
Copy link
Copy Markdown
Collaborator

@alex-clickhouse alex-clickhouse commented Apr 1, 2026

Right now the binary insert requires users to provide object[][]. This PR lets them use POCOs instead.

  • New InsertBinaryAsync<T>() method for inserting.
  • Two new property attributes: [ClickHouseColumn(Name, Type)] for setting column name/type, [ClickHouseNotMapped] for exclusion. If all properties are covered, we skip the schema probe.
  • Type registration required via RegisterBinaryInstertType<T>(). This saves the serialization info and compiles property accessors into Func<T, object> delegates to avoid reflection when we're reading the properties in the serialization loop.
  • Internally, extracted shared insert prep functionality into PrepareInsertAsync().
  • Added new example..

New benchmark added to compare object[][] and POCO serialization. Current results:

Method Count Mean Error StdDev Ratio RatioSD Gen0 Gen1 Gen2 Allocated Alloc Ratio
ObjectArray 100000 47.17 ms 3.338 ms 7.185 ms 1.02 0.22 2000.0000 1000.0000 - 13.76 MB 1.00
Poco 100000 51.21 ms 5.293 ms 11.839 ms 1.11 0.31 2000.0000 1000.0000 - 13 MB 0.94
ObjectArray 500000 198.00 ms 3.673 ms 7.828 ms 1.00 0.06 12000.0000 7000.0000 2000.0000 66.72 MB 1.00
Poco 500000 177.00 ms 2.992 ms 6.178 ms 0.90 0.05 12000.0000 7000.0000 2000.0000 62.91 MB 0.94

Related docs PR: ClickHouse/clickhouse-docs#5919

@alex-clickhouse alex-clickhouse requested a review from mzitnik as a code owner April 1, 2026 07:27
Copilot AI review requested due to automatic review settings April 1, 2026 07:27
@codecov
Copy link
Copy Markdown

codecov bot commented Apr 1, 2026

Codecov Report

❌ Patch coverage is 89.84772% with 20 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
ClickHouse.Driver/ClickHouseClient.cs 82.92% 5 Missing and 9 partials ⚠️
...ouse.Driver/Copy/Serializer/PocoBatchSerializer.cs 86.20% 3 Missing and 1 partial ⚠️
.../Serializer/PocoRowBinaryWithDefaultsSerializer.cs 71.42% 1 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR extends ClickHouseClient’s binary insert pipeline to accept strongly-typed POCOs (IEnumerable<T>) via a new InsertBinaryAsync<T> overload, using upfront type registration to precompute property→column mappings and compiled getters, with optional attribute-based column naming/type hints to reduce or eliminate schema probing.

Changes:

  • Add POCO binary insert API (InsertBinaryAsync<T>) + registration (RegisterBinaryInsertType<T>) with [ClickHouseColumn] / [ClickHouseNotMapped] attributes.
  • Introduce POCO-specific batching/serialization components mirroring the existing object[] bulk insert path.
  • Add examples, benchmarks, and extensive NUnit coverage for POCO insert behavior (mapping, exclusion, defaults, schema-probe skipping).

Reviewed changes

Copilot reviewed 21 out of 21 changed files in this pull request and generated 8 comments.

Show a summary per file
File Description
RELEASENOTES.md Document POCO binary insert feature and attribute-based mapping/schema probe skip.
CHANGELOG.md Mirror release notes entry for the new POCO insert feature.
examples/README.md Add the new POCO insert example to the index.
examples/Program.cs Wire the new example into the example runner.
examples/Insert/Insert_009_PocoInsert.cs New end-to-end example demonstrating POCO insert + attributes + explicit types.
ClickHouse.Driver/InsertOptions.cs Add internal helper to apply column type maps derived from POCO registration.
ClickHouse.Driver/IClickHouseClient.cs Add new public POCO insert + registration APIs on the interface.
ClickHouse.Driver/ClickHouseClient.cs Implement POCO insert pipeline; factor shared insert setup into PrepareInsertAsync().
ClickHouse.Driver/Attributes/ClickHouseColumnAttribute.cs New attribute for column name/type mapping.
ClickHouse.Driver/Attributes/ClickHouseNotMappedAttribute.cs New attribute to exclude properties from mapping.
ClickHouse.Driver/Copy/BinaryInsertTypeRegistry.cs New registry building mappings + compiled getters for POCO inserts.
ClickHouse.Driver/Copy/BinaryInsertTypeMapping.cs New mapping model for registered POCOs (properties/getters/optional ColumnTypes).
ClickHouse.Driver/Copy/BinaryInsertPropertyInfo.cs New cached property metadata model.
ClickHouse.Driver/Copy/PocoBatch.cs New batch container for pooled POCO rows.
ClickHouse.Driver/Copy/Serializer/IPocoRowSerializer.cs New row-serializer abstraction for POCO rows.
ClickHouse.Driver/Copy/Serializer/PocoRowBinarySerializer.cs RowBinary POCO row serializer.
ClickHouse.Driver/Copy/Serializer/PocoRowBinaryWithDefaultsSerializer.cs RowBinaryWithDefaults POCO row serializer with DBDefault handling.
ClickHouse.Driver/Copy/Serializer/PocoBatchSerializer.cs Batch serializer for POCO rows (gzip + query header + binary rows).
ClickHouse.Driver.Tests/Copy/InsertBinaryPocoTests.cs New integration tests validating mapping, exclusions, batching, schema-probe behavior.
ClickHouse.Driver.Tests/Copy/BinaryInsertTypeRegistryTests.cs New unit tests for registry validation/duplicate columns/no mappable properties.
ClickHouse.Driver.Benchmark/PocoInsertBenchmark.cs New benchmark comparing object[] vs POCO insert performance.

Comment thread ClickHouse.Driver/IClickHouseClient.cs
Comment thread ClickHouse.Driver/IClickHouseClient.cs
Comment thread ClickHouse.Driver/Copy/BinaryInsertTypeRegistry.cs
Comment thread ClickHouse.Driver/Copy/BinaryInsertTypeRegistry.cs
Comment thread ClickHouse.Driver/Copy/BinaryInsertTypeRegistry.cs
Comment thread ClickHouse.Driver/Copy/BinaryInsertTypeRegistry.cs Outdated
Comment thread ClickHouse.Driver/Copy/PocoBatch.cs Outdated
Comment thread ClickHouse.Driver.Benchmark/PocoInsertBenchmark.cs Outdated
namespace ClickHouse.Driver.Tests.Copy;

[TestFixture]
public class BinaryInsertTypeRegistryTests
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should add a few more edge cases

  • two columns with the same name
  • column with reserved name

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

two columns with the same name

Already covered by PocoWithDuplicateColumnNames.

/// <param name="stream">The output stream (typically a recyclable memory stream).</param>
public void Serialize<T>(PocoBatch<T> batch, Func<T, object>[] getters, Stream stream)
{
using var gzipStream = new BufferedStream(new GZipStream(stream, CompressionLevel.Fastest, true), 256 * 1024);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i tihnk we should have some of the params of GZipStream configerable

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good but let's leave that for another PR. I added a story for it.

@alex-clickhouse alex-clickhouse merged commit f3b86db into main Apr 13, 2026
16 checks passed
@alex-clickhouse alex-clickhouse deleted the poco-inserts branch April 13, 2026 13:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants