Skip to content

chore: bump to 0.4.0-dev, update README benchmarks #48

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
cigrainger merged 1 commit into from
Mar 31, 2026
Merged

chore: bump to 0.4.0-dev, update README benchmarks #48

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 12 additions & 10 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,16 +24,18 @@ Dux.from_parquet("s3://data/sales/**/*.parquet")

## Performance

Dux pipelines compile to SQL and execute inside DuckDB — no data crosses into Elixir until you materialise. On a 10M-row dataset (Apple M3 Max, 36GB):

| Operation | Dux | Explorer (Polars) | Ratio |
|-----------|-----|-------------------|-------|
| Filter (10M rows) | 41ms | 13ms | 3.1x |
| Mutate (10M rows) | ~40ms | ~14ms | ~3x |
| Group + Summarise | ~12ms | ~21ms | **0.6x** |
| Memory per compute | 5-10 KB | 5-10 KB | ~same |

Dux is within 3x of Polars for single-node operations and **faster for aggregations** (DuckDB's columnar engine). The gap narrows further at scale — Dux can distribute across machines while Polars is single-node.
Dux pipelines compile to SQL and execute inside DuckDB — no data crosses into Elixir until you materialise. On a 10M-row dataset (Apple M4 Max, 128GB):

| Operation | Dux | Explorer (Polars) | Winner |
|-----------|-----|-------------------|--------|
| Filter (lazy) | 24ms | 59ms | **Dux 2.5x faster** |
| Filter (eager) | 45ms | 53ms | **Dux 1.2x faster** |
| Mutate (eager) | 17ms | 28ms | **Dux 1.6x faster** |
| Group + Summarise (lazy) | 40ms | 63ms | **Dux 1.6x faster** |
| Group + Summarise (eager) | 81ms | 88ms | **Dux 1.1x faster** |
| Memory per compute | 11-15 KB | 8-9 KB | Explorer ~1.5x less |

Dux is **faster than Explorer/Polars** on every operation at 10M rows. The lazy path (view-based `compute/1`) is particularly fast since no data is copied — DuckDB executes the full pipeline when results are read. And Dux can distribute across machines while Polars is single-node.

## Design

Expand Down
2 changes: 1 addition & 1 deletion mix.exs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
defmodule Dux.MixProject do
use Mix.Project

@version "0.3.0"
@version "0.4.0-dev"
@source_url "https://github.com/elixir-dux/dux"

def project do
Expand Down
Loading