ci: add swap during build, use tpchgen-cli (#1443)

* restrict number of rustc jobs during build stage

* temporarily run release build on PR

* Change optimization setting for substrait

* Add swap during release build

* Remove temporary checks to build in PR

* Try using tpchgen-cli for test files. commit answers

* taplo fmt

* do not run rat on data files

* ci needs ./ in path

* add no-project to uv run

* Temporary debug lines to figure out what is happening in CI

* filter null value during aggregation instead now that apache/datafusion#21011 is closed

Author: timsaucer

.github/workflows/build.yml

@@ -226,6 +226,19 @@ jobs:
         with:
           enable-cache: true
 
+      - name: Add extra swap for release build
+        if: inputs.build_mode == 'release'
+        run: |
+          set -euxo pipefail
+          sudo swapoff -a || true
+          sudo rm -f /swapfile
+          sudo fallocate -l 16G /swapfile || sudo dd if=/dev/zero of=/swapfile bs=1M count=16384
+          sudo chmod 600 /swapfile
+          sudo mkswap /swapfile
+          sudo swapon /swapfile
+          free -h
+          swapon --show
+
       - name: Build (release mode)
         uses: PyO3/maturin-action@v1
         if: inputs.build_mode == 'release'

.github/workflows/test.yml

@@ -101,21 +101,14 @@ jobs:
           cd examples/datafusion-ffi-example
           uv run --no-project pytest python/tests/_test*.py
 
-      - name: Cache the generated dataset
-        id: cache-tpch-dataset
-        uses: actions/cache@v5
-        with:
-          path: benchmarks/tpch/data
-          key: tpch-data-2.18.0
-
-      - name: Run dbgen to create 1 Gb dataset
-        if: ${{ steps.cache-tpch-dataset.outputs.cache-hit != 'true' }}
+      - name: Run tpchgen-cli to create 1 Gb dataset
         run: |
-          cd benchmarks/tpch
-          RUN_IN_CI=TRUE ./tpch-gen.sh 1
+          mkdir examples/tpch/data
+          cd examples/tpch/data
+          uv pip install tpchgen-cli
+          uv run --no-project tpchgen-cli -s 1 --format=parquet
 
       - name: Run TPC-H examples
         run: |
           cd examples/tpch
-          uv run --no-project python convert_data_to_parquet.py
           uv run --no-project pytest _tests.py

dev/release/rat_exclude_files.txt

@@ -47,4 +47,5 @@ benchmarks/tpch/queries/q*.sql
 benchmarks/tpch/create_tables.sql
 .cargo/config.toml
 **/.cargo/config.toml
-uv.lock
+uv.lock
+examples/tpch/answers_sf1/*.tbl

examples/tpch/_tests.py

@@ -115,7 +115,7 @@ def test_tpch_query_vs_answer_file(query_code: str, answer_file: str) -> None:
         get_answer_file(answer_file),
         schema=read_schema,
         delimiter="|",
-        file_extension=".out",
+        file_extension=".tbl",
     )
 
     df_expected = df_expected.select(*expected_selections)

examples/tpch/answers_sf1/q1.tbl

@@ -0,0 +1,5 @@
+l|l|sum_qty                                  |sum_base_price                           |sum_disc_price                           |sum_charge                               |avg_qty                                  |avg_price                                |avg_disc                                 |count_order           
+A|F|37734107.00|56586554400.73|53758257134.87|55909065222.83|25.52|38273.13|0.05|               1478493
+N|F|991417.00|1487504710.38|1413082168.05|1469649223.19|25.52|38284.47|0.05|                 38854
+N|O|74476040.00|111701729697.74|106118230307.61|110367043872.50|25.50|38249.12|0.05|               2920374
+R|F|37719753.00|56568041380.90|53741292684.60|55889619119.83|25.51|38250.85|0.05|               1478870

examples/tpch/answers_sf1/q10.tbl

@@ -0,0 +1,21 @@
+c_custkey           |c_name                   |revenue                                  |c_acctbal     |n_name                   |c_address                               |c_phone        |c_comment                                                                                                            
+               57040|Customer#000057040       |734235.25|632.87|JAPAN                    |Eioyzjf4pp                              |22-895-641-3466|sits. slyly regular requests sleep alongside of the regular inst                                                     
+              143347|Customer#000143347       |721002.69|2557.47|EGYPT                    |1aReFYv,Kw4                             |14-742-935-3718|ggle carefully enticing requests. final deposits use bold, bold pinto beans. ironic, idle re                         
+               60838|Customer#000060838       |679127.31|2454.77|BRAZIL                   |64EaJ5vMAHWJlBOxJklpNc2RJiWE            |12-913-494-9813| need to boost against the slyly regular account                                                                     
+              101998|Customer#000101998       |637029.57|3790.89|UNITED KINGDOM           |01c9CILnNtfOQYmZj                       |33-593-865-6378|ress foxes wake slyly after the bold excuses. ironic platelets are furiously carefully bold theodolites              
+              125341|Customer#000125341       |633508.09|4983.51|GERMANY                  |S29ODD6bceU8QSuuEJznkNaK                |17-582-695-5962|arefully even depths. blithely even excuses sleep furiously. foxes use except the dependencies. ca                   
+               25501|Customer#000025501       |620269.78|7725.04|ETHIOPIA                 |  W556MXuoiaYCCZamJI,Rn0B4ACUGdkQ8DZ    |15-874-808-6793|he pending instructions wake carefully at the pinto beans. regular, final instructions along the slyly fina          
+              115831|Customer#000115831       |596423.87|5098.10|FRANCE                   |rFeBbEEyk dl ne7zV5fDrmiq1oK09wV7pxqCgIc|16-715-386-3788|l somas sleep. furiously final deposits wake blithely regular pinto b                                                
+               84223|Customer#000084223       |594998.02|528.65|UNITED KINGDOM           |nAVZCs6BaWap rrM27N 2qBnzc5WBauxbA      |33-442-824-8191| slyly final deposits haggle regular, pending dependencies. pending escapades wake                                   
+               54289|Customer#000054289       |585603.39|5583.02|IRAN                     |vXCxoCsU0Bad5JQI ,oobkZ                 |20-834-292-4707|ely special foxes are quickly finally ironic p                                                                       
+               39922|Customer#000039922       |584878.11|7321.11|GERMANY                  |Zgy4s50l2GKN4pLDPBU8m342gIw6R           |17-147-757-8036|y final requests. furiously final foxes cajole blithely special platelets. f                                         
+                6226|Customer#000006226       |576783.76|2230.09|UNITED KINGDOM           |8gPu8,NPGkfyQQ0hcIYUGPIBWc,ybP5g,       |33-657-701-3391|ending platelets along the express deposits cajole carefully final                                                   
+                 922|Customer#000000922       |576767.53|3869.25|GERMANY                  |Az9RFaut7NkPnc5zSD2PwHgVwr4jRzq         |17-945-916-9648|luffily fluffy deposits. packages c                                                                                  
+              147946|Customer#000147946       |576455.13|2030.13|ALGERIA                  |iANyZHjqhyy7Ajah0pTrYyhJ                |10-886-956-3143|ithely ironic deposits haggle blithely ironic requests. quickly regu                                                 
+              115640|Customer#000115640       |569341.19|6436.10|ARGENTINA                |Vtgfia9qI 7EpHgecU1X                    |11-411-543-4901|ost slyly along the patterns; pinto be                                                                               
+               73606|Customer#000073606       |568656.86|1785.67|JAPAN                    |xuR0Tro5yChDfOCrjkd2ol                  |22-437-653-6966|he furiously regular ideas. slowly                                                                                   
+              110246|Customer#000110246       |566842.98|7763.35|VIETNAM                  |7KzflgX MDOq7sOkI                       |31-943-426-9837|egular deposits serve blithely above the fl                                                                          
+              142549|Customer#000142549       |563537.24|5085.99|INDONESIA                |ChqEoK43OysjdHbtKCp6dKqjNyvvi9          |19-955-562-2398|sleep pending courts. ironic deposits against the carefully unusual platelets cajole carefully express accounts.     
+              146149|Customer#000146149       |557254.99|1791.55|ROMANIA                  |s87fvzFQpU                              |29-744-164-6487| of the slyly silent accounts. quickly final accounts across the                                                     
+               52528|Customer#000052528       |556397.35|551.79|ARGENTINA                |NFztyTOR10UOJ                           |11-208-192-3205| deposits hinder. blithely pending asymptotes breach slyly regular re                                                
+               23431|Customer#000023431       |554269.54|3381.86|ROMANIA                  |HgiV0phqhaIa9aydNoIlb                   |29-915-458-2654|nusual, even instructions: furiously stealthy n