adapter: Optimize COPY FROM STDIN, parallelize it, use constant memory#35036
Open
def- wants to merge 7 commits intoMaterializeInc:mainfrom
Open
adapter: Optimize COPY FROM STDIN, parallelize it, use constant memory#35036def- wants to merge 7 commits intoMaterializeInc:mainfrom
def- wants to merge 7 commits intoMaterializeInc:mainfrom
Conversation
Previously failed: materialize=> alter system set max_copy_from_size=15000000000; ERROR: parameter "max_copy_from_size" requires a "unsigned integer" value
Pre-merge checklist
|
4e1af8e to
76a31a3
Compare
86fc97d to
d5b109c
Compare
d5b109c to
0dd54e9
Compare
0dd54e9 to
862818c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tried with 100 million rows in https://github.com/def-/ClickBench/tree/pr-materialize/materialize. Before this PR it took 101 min, would go OoM if the files were not split, and have query errors because of results being too large. With this PR the 100 million row ingestion runs in 4:38 min on my dev server (8 cores), should scale pretty linearly with the number of cores. For reference
COPY WITH FREEZEin PostgreSQL takes 20 min.Example test run of the new benchmark in CI with 10 million rows: https://buildkite.com/materialize/release-qualification/builds/1089#019c68b4-4b3f-424f-b3c4-05518e95f1a0
Run in environmentd spec sheet doesn't scale well in Cloud (should be investigated!), but scales well locally:
Fixes: https://github.com/MaterializeInc/database-issues/issues/7674
Motivation:
COPY FROM STDINhas been slow for workload replay and testing with large amounts of data in general, also been annoying in https://github.com/MaterializeInc/database-issues/issues/7674 and recently https://materializeinc.slack.com/archives/C08A62E0751/p1770835109967349