Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 33 additions & 10 deletions .github/workflows/publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ jobs:

- name: Install dependencies
run: |
dnf install -y openblas-devel openssl-devel perl-IPC-Cmd
dnf install -y openssl-devel perl-IPC-Cmd
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --no-modify-path
/opt/python/cp312-cp312/bin/pip install maturin

Expand Down Expand Up @@ -54,24 +54,48 @@ jobs:
- name: Install Rust
uses: dtolnay/rust-toolchain@stable

- name: Install OpenBLAS
run: brew install openblas

- name: Install maturin
run: pip install maturin

- name: Build wheel
run: |
export OPENBLAS_DIR=$(brew --prefix openblas)
export PKG_CONFIG_PATH=$(brew --prefix openblas)/lib/pkgconfig
maturin build --release --out dist --features extension-module
run: maturin build --release --out dist --features extension-module

- name: Upload wheels
uses: actions/upload-artifact@v4
with:
name: wheels-macos-arm64-py${{ matrix.python-version }}
path: dist/*.whl

# Build wheels on Windows
build-windows:
name: Build Windows wheels
runs-on: windows-latest
strategy:
matrix:
python-version: ['3.9', '3.10', '3.11', '3.12']
steps:
- uses: actions/checkout@v4

- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}

- name: Install Rust
uses: dtolnay/rust-toolchain@stable

- name: Install maturin
run: pip install maturin

- name: Build wheel
run: maturin build --release --out dist --features extension-module

- name: Upload wheels
uses: actions/upload-artifact@v4
with:
name: wheels-windows-py${{ matrix.python-version }}
path: dist/*.whl

# Build source distribution
build-sdist:
name: Build source distribution
Expand All @@ -97,10 +121,9 @@ jobs:
path: dist/*.tar.gz

# Publish to PyPI
# Windows and macOS x86_64 users install from sdist and get pure Python fallback
publish:
name: Publish to PyPI
needs: [build-linux, build-macos-arm, build-sdist]
needs: [build-linux, build-macos-arm, build-windows, build-sdist]
runs-on: ubuntu-latest
environment: pypi
permissions:
Expand Down
87 changes: 57 additions & 30 deletions .github/workflows/rust-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,19 +22,23 @@ env:
CARGO_TERM_COLOR: always

jobs:
# Run Rust unit tests
# Run Rust unit tests on all platforms
rust-tests:
name: Rust Unit Tests
runs-on: ubuntu-latest
name: Rust Unit Tests (${{ matrix.os }})
runs-on: ${{ matrix.os }}
env:
PYO3_USE_ABI3_FORWARD_COMPATIBILITY: 1
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, macos-latest, windows-latest]

steps:
- uses: actions/checkout@v4

- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable

- name: Install OpenBLAS
run: sudo apt-get update && sudo apt-get install -y libopenblas-dev

- name: Run Rust tests
working-directory: rust
run: cargo test --verbose
Expand All @@ -46,8 +50,7 @@ jobs:
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, macos-latest]
# Windows excluded due to Intel MKL build complexity
os: [ubuntu-latest, macos-latest, windows-latest]

steps:
- uses: actions/checkout@v4
Expand All @@ -57,20 +60,6 @@ jobs:
with:
python-version: '3.11'

- name: Install OpenBLAS (Ubuntu)
if: matrix.os == 'ubuntu-latest'
run: sudo apt-get update && sudo apt-get install -y libopenblas-dev

- name: Install OpenBLAS (macOS)
if: matrix.os == 'macos-latest'
run: brew install openblas

- name: Set OpenBLAS paths (macOS)
if: matrix.os == 'macos-latest'
run: |
echo "OPENBLAS_DIR=$(brew --prefix openblas)" >> $GITHUB_ENV
echo "PKG_CONFIG_PATH=$(brew --prefix openblas)/lib/pkgconfig" >> $GITHUB_ENV

- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable

Expand All @@ -82,28 +71,66 @@ jobs:
pip install maturin
maturin build --release -o dist
echo "=== Built wheels ==="
ls -la dist/
# --no-index ensures we install from local wheel, not PyPI
pip install --no-index --find-links=dist diff-diff
ls -la dist/ || dir dist
shell: bash

- name: Install wheel (Unix)
if: runner.os != 'Windows'
run: pip install --no-index --find-links=dist diff-diff

- name: Verify Rust backend is available
# Run from /tmp to avoid source directory shadowing installed package
- name: Install wheel (Windows)
if: runner.os == 'Windows'
run: |
$wheel = Get-ChildItem dist/*.whl | Select-Object -First 1
pip install $wheel.FullName
shell: pwsh

- name: Verify Rust backend is available (Unix)
if: runner.os != 'Windows'
working-directory: /tmp
run: |
python -c "import diff_diff; print('Location:', diff_diff.__file__)"
python -c "from diff_diff import HAS_RUST_BACKEND; print('HAS_RUST_BACKEND:', HAS_RUST_BACKEND); assert HAS_RUST_BACKEND, 'Rust backend not available'"

- name: Copy tests to isolated location
- name: Verify Rust backend is available (Windows)
if: runner.os == 'Windows'
working-directory: ${{ runner.temp }}
run: |
python -c "import diff_diff; print('Location:', diff_diff.__file__)"
python -c "from diff_diff import HAS_RUST_BACKEND; print('HAS_RUST_BACKEND:', HAS_RUST_BACKEND); assert HAS_RUST_BACKEND, 'Rust backend not available'"

- name: Copy tests to isolated location (Unix)
if: runner.os != 'Windows'
run: cp -r tests /tmp/tests

- name: Run Rust backend tests
- name: Copy tests to isolated location (Windows)
if: runner.os == 'Windows'
run: Copy-Item -Recurse tests $env:RUNNER_TEMP\tests
shell: pwsh

- name: Run Rust backend tests (Unix)
if: runner.os != 'Windows'
working-directory: /tmp
run: pytest tests/test_rust_backend.py -v

- name: Run tests with Rust backend
- name: Run Rust backend tests (Windows)
if: runner.os == 'Windows'
working-directory: ${{ runner.temp }}
run: pytest tests/test_rust_backend.py -v

- name: Run tests with Rust backend (Unix)
if: runner.os != 'Windows'
working-directory: /tmp
run: DIFF_DIFF_BACKEND=rust pytest tests/ -x -q

- name: Run tests with Rust backend (Windows)
if: runner.os == 'Windows'
working-directory: ${{ runner.temp }}
run: |
$env:DIFF_DIFF_BACKEND="rust"
pytest tests/ -x -q
shell: pwsh

# Test pure Python fallback (without Rust extension)
python-fallback:
name: Pure Python Fallback
Expand Down
42 changes: 7 additions & 35 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,9 +40,6 @@ maturin develop
# Build with release optimizations
maturin develop --release

# Run Rust unit tests
cd rust && cargo test

# Force pure Python mode (disable Rust backend)
DIFF_DIFF_BACKEND=python pytest

Expand All @@ -53,35 +50,9 @@ DIFF_DIFF_BACKEND=rust pytest
pytest tests/test_rust_backend.py -v
```

#### Troubleshooting Rust Tests (PyO3 Linking)

If `cargo test` fails with `library 'pythonX.Y' not found`, PyO3 cannot find the Python library. This commonly happens on macOS when using the system Python (which lacks development headers in expected locations).

**Solution**: Use a Python environment with proper library paths (e.g., conda, Homebrew, or pyenv):

```bash
# Using miniconda (example path - adjust for your system)
cd rust
PYO3_PYTHON=/path/to/miniconda3/bin/python3 \
DYLD_LIBRARY_PATH="/path/to/miniconda3/lib" \
cargo test

# Using Homebrew Python
PYO3_PYTHON=/opt/homebrew/bin/python3 \
DYLD_LIBRARY_PATH="/opt/homebrew/lib" \
cargo test
```

**Environment variables:**
- `PYO3_PYTHON`: Path to Python interpreter with development headers
- `DYLD_LIBRARY_PATH` (macOS) / `LD_LIBRARY_PATH` (Linux): Path to `libpythonX.Y.dylib`/`.so`

**Verification**: All 22 Rust tests should pass, including bootstrap weight tests:
```
test bootstrap::tests::test_webb_variance_approx_correct ... ok
test bootstrap::tests::test_webb_values_correct ... ok
test bootstrap::tests::test_webb_mean_approx_zero ... ok
```
**Note**: As of v2.2.0, the Rust backend uses the pure-Rust `faer` library for linear algebra,
eliminating external BLAS/LAPACK dependencies. This enables Windows wheel builds and simplifies
cross-platform compilation - no OpenBLAS or Intel MKL installation required.

## Architecture

Expand Down Expand Up @@ -173,16 +144,17 @@ test bootstrap::tests::test_webb_mean_approx_zero ... ok
- Exports `HAS_RUST_BACKEND` flag and Rust function references
- Other modules import from here to avoid circular imports with `__init__.py`

- **`rust/`** - Optional Rust backend for accelerated computation (v2.0.0):
- **`rust/`** - Optional Rust backend for accelerated computation (v2.0.0+):
- **`rust/src/lib.rs`** - PyO3 module definition, exports Python bindings
- **`rust/src/bootstrap.rs`** - Parallel bootstrap weight generation (Rademacher, Mammen, Webb)
- **`rust/src/linalg.rs`** - OLS solver and cluster-robust variance estimation
- **`rust/src/linalg.rs`** - OLS solver (SVD-based) and cluster-robust variance estimation
- **`rust/src/weights.rs`** - Synthetic control weights and simplex projection
- **`rust/src/trop.rs`** - TROP estimator acceleration:
- `compute_unit_distance_matrix()` - Parallel pairwise RMSE distance computation (4-8x speedup)
- `loocv_grid_search()` - Parallel LOOCV across tuning parameters (10-50x speedup)
- `bootstrap_trop_variance()` - Parallel bootstrap variance estimation (5-15x speedup)
- Uses ndarray-linalg with OpenBLAS (Linux/macOS) or Intel MKL (Windows)
- Uses pure-Rust `faer` library for linear algebra (no external BLAS/LAPACK dependencies)
- Cross-platform: builds on Linux, macOS, and Windows without additional setup
- Provides 4-8x speedup for SyntheticDiD, 5-20x speedup for TROP

- **`diff_diff/results.py`** - Dataclass containers for estimation results:
Expand Down
Loading