Minimizing Learner–Expert Asymmetry in End-to-End Driving

Website  ·  Docs  ·  Dataset  ·  Model  ·  NAVSIM Model  ·  Supplementary  ·  Paper

An open-source end-to-end driving stack for CARLA, achieving state-of-the-art
closed-loop performance across all major Leaderboard 2.0 benchmarks.

---

Table of Contents

• Table of Contents
• Updates
• Quick Start CARLA Leaderboard
  • 1. Environment initialization
  • 2. Install dependencies
  • 3. Download checkpoints
  • 4. Setup VSCode/PyCharm
  • 5. Evaluate model
  • 6. Infraction Analysis Webapp
• CARLA Training
• CARLA Data Collection
• CARLA 123D Data Collection
• CARLA Benchmarking
• CaRL Agent Evaluation
• NAVSIM Training and Evaluation
• Project Structure
• Common Issues
• Beyond CARLA: Cross-Benchmark Deployment
• Further Documentation
• Acknowledgements
• Citation
• License

---

Updates

Date	Content
26.03.21	Added evaluation support for the RL planner CaRL, see instructions.
26.03.18	Deactivated creeping heuristic. Set sensor_agent_creeping=True in config_closed_loop to re-enable.
26.02.25	LEAD is accepted to CVPR 2026!
26.02.25	NAVSIM extension released. Code and instructions available. Supplementary data coming soon.
26.02.02	Preliminary support for 123D. See instructions.
26.01.18	Deactivated Kalman filter. Set use_kalman_filter=True in config_closed_loop to re-enable.
26.01.13	CARLA dataset and training documentation released.
26.01.05	Deactivated stop-sign heuristic. Set slower_for_stop_sign=True in config_closed_loop to re-enable.
26.01.05	RoutePlanner bug fix — fixed an index error causing crashes at end of routes in Town13.
25.12.24	Initial release — paper, checkpoints, expert driver, and inference code.

---

Quick Start CARLA Leaderboard

1. Environment initialization

Clone the repository and register the project root:

git clone https://github.com/autonomousvision/lead.git
cd lead

# Set project root variable
echo -e "export LEAD_PROJECT_ROOT=$(pwd)" >> ~/.bashrc

# Activate project's hook
echo "source $(pwd)/scripts/main.sh" >> ~/.bashrc

# Reload shell config
source ~/.bashrc

Verify that ~/.bashrc reflects these paths correctly.

2. Install dependencies

We use Miniconda, conda-lock, and uv:

# Create conda environment
pip install conda-lock && conda-lock install -n lead conda-lock.yml

# Activate conda environment
conda activate lead

# Install dependencies and setup git hooks
pip install uv && uv pip install -r requirements.txt && uv pip install -e .

# Install other tools needed for development
conda install -c conda-forge ffmpeg parallel tree gcc zip unzip

# Optional: activate git hooks
pre-commit install

Set up CARLA:

# Download and setup CARLA at 3rd_party/CARLA_0915
bash scripts/setup_carla.sh

# Or symlink your pre-installed CARLA
ln -s /your/carla/path 3rd_party/CARLA_0915

3. Download checkpoints

Pre-trained checkpoints are hosted on HuggingFace. To reproduce the published results, enable the Kalman filter, stop-sign, and creeping heuristics. Performance without these heuristics (fully end-to-end) should be comparable to performance with them.

Variant	Bench2Drive	Longest6 v2	Town13	Checkpoint
Full TransFuser V6	95	62	5.24	Link
ResNet34 (60M params)	94	57	5.01	Link
+ Rear camera	95	53	TBD	Link
− Radar	94	52	TBD	Link
Vision only	91	43	TBD	Link
Town13 held out	93	52	3.52	Link

Download the checkpoints:

# Download one checkpoint for testing
bash scripts/download_one_checkpoint.sh

# Download all checkpoints
git clone https://huggingface.co/ln2697/tfv6 outputs/checkpoints
cd outputs/checkpoints
git lfs pull

4. Setup VSCode/PyCharm

VSCode — install recommended extensions when prompted. Debugging works out of the box.

PyCharm — add the CARLA Python API 3rd_party/CARLA_0915/PythonAPI/carla to your interpreter paths via Settings → Python → Interpreter → Show All → Show Interpreter Paths.

5. Evaluate model

Verify your setup with a single route:

# Start driving environment
bash scripts/start_carla.sh

# Run policy on one route
python lead/leaderboard_wrapper.py \
  --checkpoint outputs/checkpoints/tfv6_resnet34 \
  --routes data/benchmark_routes/bench2drive/23687.xml \
  --bench2drive

Driving logs are saved to outputs/local_evaluation/<route_id>/:

Output	Description
*_debug.mp4	Debug visualization video
*_demo.mp4	Demo video
*_grid.mp4	Grid visualization video
*_input.mp4	Raw input video
alpasim_metric_log.json	AlpaSim metric log
checkpoint_endpoint.json	Checkpoint endpoint metadata
infractions.json	Detected infractions
metric_info.json	Evaluation metrics
debug_images/	Per-frame debug visualizations
demo_images/	Per-frame demo images
grid_images/	Per-frame grid visualizations
input_images/	Per-frame raw inputs
input_log/	Input log data

6. Infraction Analysis Webapp

Launch the interactive infraction dashboard to analyze driving failures — especially useful for Longest6 or Town13 where iterating over evaluation logs is time-consuming:

python lead/infraction_webapp/app.py

Navigate to http://localhost:5000 and point it at outputs/local_evaluation.

Tip

The app supports browser bookmarking to jump directly to a specific timestamp.

---

CARLA Training

Download the dataset from HuggingFace:

# Download all routes
git clone https://huggingface.co/datasets/ln2697/lead_carla data/carla_leaderboard2/zip
cd data/carla_leaderboard2/zip
git lfs pull

# Or download a single route for testing
bash scripts/download_one_route.sh

# Unzip the routes
bash scripts/unzip_routes.sh

# Build data cache
python scripts/build_cache.py

Perception pretraining. Logs and checkpoints are saved to outputs/local_training/pretrain:

# Single GPU
python3 lead/training/train.py \
  logdir=outputs/local_training/pretrain

# Distributed Data Parallel
bash scripts/pretrain_ddp.sh

Planning post-training. Logs and checkpoints are saved to outputs/local_training/posttrain:

# Single GPU
python3 lead/training/train.py \
  logdir=outputs/local_training/posttrain \
  load_file=outputs/local_training/pretrain/model_0030.pth \
  use_planning_decoder=true

# Distributed Data Parallel
bash scripts/posttrain_ddp.sh

Tip

1. For distributed training on SLURM, see the SLURM training docs.
2. For a complete workflow (pretrain → posttrain → eval), see this example.
3. For detailed documentation, see the training guide.

---

CARLA Data Collection

With CARLA running, collect data for a single route via Python (recommended for debugging):

python lead/leaderboard_wrapper.py \
  --expert \
  --routes data/data_routes/lead/noScenarios/short_route.xml

Or via bash (recommended for flexibility):

bash scripts/eval_expert.sh

Collected data is saved to outputs/expert_evaluation/ with the following structure:

Directory	Content
bboxes/	3D bounding boxes per frame
depth/	Compressed and quantized depth maps
depth_perturbated/	Depth from perturbated ego state
hdmap/	Ego-centric rasterized HD map
hdmap_perturbated/	HD map aligned to perturbated ego pose
lidar/	LiDAR point clouds
metas/	Per-frame metadata and ego state
radar/	Radar detections
radar_perturbated/	Radar from perturbated ego state
rgb/	RGB images
rgb_perturbated/	RGB from perturbated ego state
semantics/	Semantic segmentation maps
semantics_perturbated/	Semantics from perturbated ego state
results.json	Route-level summary and evaluation metadata

Tip

1. To configure camera/lidar/radar calibration, see config_base.py and config_expert.py.
2. For large-scale collection on SLURM, see the data collection docs.
3. The Jupyter notebooks provide visualization examples.

---

CARLA 123D Data Collection

With CARLA running, collect data in 123D format via Python:

export LEAD_EXPERT_CONFIG="target_dataset=6 \
  py123d_data_format=true \
  use_radars=false \
  lidar_stack_size=2 \
  save_only_non_ground_lidar=false \
  save_lidar_only_inside_bev=false"

python -u $LEAD_PROJECT_ROOT/lead/leaderboard_wrapper.py \
    --expert \
    --py123d \
    --routes data/data_routes/50x38_Town12/ParkingCrossingPedestrian/3250_1.xml

Or via bash:

bash scripts/eval_expert_123d.sh

Output in 123D format is saved to data/carla_leaderboard2_py123d/:

Directory	Content
logs/train/*.arrow	Per-route driving logs in Arrow format
logs/train/*.json	Per-route metadata
maps/carla/*.arrow	Map data in Arrow format

Tip

This feature is experimental. Change PY123D_DATA_ROOT in scripts/main.sh to set the output directory.

---

CARLA Benchmarking

With CARLA running, evaluate on any benchmark via Python:

python lead/leaderboard_wrapper.py \
  --checkpoint outputs/checkpoints/tfv6_resnet34 \
  --routes <ROUTE_FILE> \
  [--bench2drive]

Benchmark	Route file	Extra flag
Bench2Drive	data/benchmark_routes/bench2drive/23687.xml	--bench2drive
Longest6 v2	data/benchmark_routes/longest6/00.xml	—
Town13	data/benchmark_routes/Town13/0.xml	—

Or via bash:

bash scripts/eval_bench2drive.sh   # Bench2Drive
bash scripts/eval_longest6.sh      # Longest6 v2
bash scripts/eval_town13.sh        # Town13

Results are saved to outputs/local_evaluation/ with videos, infractions, and metrics.

Tip

1. See the evaluation docs for details.
2. For distributed evaluation, see the SLURM evaluation docs.
3. Our SLURM wrapper supports WandB for reproducible benchmarking.

---

CaRL Agent Evaluation

With CARLA running, evaluate the CaRL agent via Python:

CUBLAS_WORKSPACE_CONFIG=:4096:8 \
python lead/leaderboard_wrapper.py \
  --checkpoint outputs/checkpoints/CaRL \
  --routes data/benchmark_routes/bench2drive/24240.xml \
  --carl-agent \
  --bench2drive \
  --timeout 900

Or via bash:

bash scripts/eval_carl.sh

The results are in outputs/local_evaluation/<route_id>/.

Tip

With small code changes, you can also integrate CaRL into LEAD's expert-driving pipeline as a hybrid expert policy.

---

NAVSIM Training and Evaluation

Setup. Install navtrain and navtest splits following navsimv1.1/docs/install.md, then install the navhard split following navsimv2.2/docs/install.md.

Training. Run perception pretraining (script) followed by planning post-training (script). We use one seed for pretraining and three seeds for post-training to estimate performance variance.

Evaluation. Run evaluation on navtest and navhard.

---

Project Structure

The project is organized into the following top-level directories. See the full documentation for a detailed breakdown.

Directory	Purpose
lead/	Main package — model architecture, training, inference, expert driver
3rd_party/	Third-party dependencies (CARLA, benchmarks, evaluation tools)
data/	Route definitions. Sensor data will be stored here, too.
scripts/	Utility scripts for data processing, training, and evaluation
outputs/	Checkpoints, evaluation results, and visualizations
notebooks/	Jupyter notebooks for data inspection and analysis
slurm/	SLURM job scripts for large-scale experiments

---

Common Issues

Symptom	Fix
Stale or corrupted data errors	Delete and rebuild the training cache / buckets
Simulator hangs or is unresponsive	Restart the CARLA simulator
Route or evaluation failures	Restart the leaderboard
Need to reset the map without restarting CARL	Run scripts/reset_carla_world.py (much faster on large maps)

---

Beyond CARLA: Cross-Benchmark Deployment

The LEAD pipeline and TFv6 models serve as reference implementations across multiple E2E driving platforms:

Platform	Model	Highlight
Waymo E2E Driving Challenge	DiffusionLTF	2nd place in the inaugural vision-based E2E driving challenge
NAVSIM v1	LTFv6	+3 PDMS over Latent TransFuser baseline on navtest
NAVSIM v2	LTFv6	+6 EPMDS over Latent TransFuser baseline on navhard
NVIDIA AlpaSim	TransFuserModel	Official baseline policy for closed-loop simulation

---

Further Documentation

For a deeper dive, visit the full documentation site:

Data Collection  ·  Training  ·  Evaluation.

The documentation will be updated regularly.

---

Acknowledgements

This project builds on the shoulders of excellent open-source work. Special thanks to carla_garage for the foundational codebase.

PDM-Lite  ·  Leaderboard  ·  Scenario Runner  ·  NAVSIM  ·  Waymo Open Dataset
SimLingo  ·  PlanT2  ·  Bench2Drive Leaderboard  ·  Bench2Drive  ·  CaRL

Long Nguyen led development of the project. Kashyap Chitta, Bernhard Jaeger, and Andreas Geiger contributed through technical discussion and advisory feedback. Daniel Dauner provided guidance with NAVSIM.

---

Citation

If you find this work useful, please consider giving this repository a star and citing our paper:

@inproceedings{Nguyen2026CVPR,
	author = {Long Nguyen and Micha Fauth and Bernhard Jaeger and Daniel Dauner and Maximilian Igl and Andreas Geiger and Kashyap Chitta},
	title = {LEAD: Minimizing Learner-Expert Asymmetry in End-to-End Driving},
	booktitle = {Conference on Computer Vision and Pattern Recognition (CVPR)},
	year = {2026},
}

License

This project is released under the MIT License. See LICENSE for details.