Skip to content

Adds mapf benchmark tools to this repo#41

Open
arjo129 wants to merge 7 commits intomainfrom
arjoc/feat/benchmark-infra
Open

Adds mapf benchmark tools to this repo#41
arjo129 wants to merge 7 commits intomainfrom
arjoc/feat/benchmark-infra

Conversation

@arjo129
Copy link
Copy Markdown
Member

@arjo129 arjo129 commented Apr 7, 2026

New feature implementation

Implemented feature

This adds the standard MAPF benchmarks as infrastructure.
The key idea is if we have new techniques that are faster like #40 . we should be able to test their improvement rapidly.

Implementation description

We added a new crate which converts the mapf benchmark into a negotiation Scenario. We also have a python script that automatically downloads the relevant benchmarks and runs them. To use it run:

scripts/benchmark.py --timeout <seconds> --max-scenarios <number of scenarios> --maps <which maps to use>

GenAI Use

We follow OSRA's policy on GenAI tools

  • I used a GenAI tool in this PR.
  • I did not use GenAI

Generated-by: Gemini-CLI

@mxgrey mxgrey added this to PMC Board Apr 7, 2026
@github-project-automation github-project-automation bot moved this to Inbox in PMC Board Apr 7, 2026
arjo129 and others added 6 commits April 7, 2026 05:39
I think the root cause of #27 is this. I'm not sure why it was not being triggered earlier,
but seems like we are accessing an 8 bit bitfield but limiting the index to `u8::MAX`instead of 7.

Signed-off-by: Arjo Chakravarty <arjo129@gmail.com>
Signed-off-by: Arjo Chakravarty <arjoc@intrinsic.ai>
Signed-off-by: Arjo Chakravarty <arjoc@intrinsic.ai>
Signed-off-by: Luca Della Vedova <lucadv@intrinsic.ai>
Signed-off-by: Arjo Chakravarty <arjoc@intrinsic.ai>
Signed-off-by: Michael X. Grey <greyxmike@gmail.com>
Signed-off-by: Arjo Chakravarty <arjoc@intrinsic.ai>
Signed-off-by: Michael X. Grey <mxgrey@intrinsic.ai>
Co-authored-by: Michael X. Grey <mxgrey@intrinsic.ai>
Signed-off-by: Arjo Chakravarty <arjoc@intrinsic.ai>
The project now includes:
- mapf-bench crate: A tool for running multi-agent negotiation benchmarks on Moving AI grid maps.
- scripts/benchmark.py: An automated script that downloads maps/scenarios, runs benchmarks for 2-50 agents, and generates reports with strict timeouts.
- .gitignore update: The cache/ directory used for benchmarking data is now ignored.

Generated-by: Gemini-CLI
Signed-off-by: Arjo Chakravarty <arjoc@intrinsic.ai>
@arjo129 arjo129 force-pushed the arjoc/feat/benchmark-infra branch from d5ed740 to 04e67d5 Compare April 7, 2026 05:39
Signed-off-by: Arjo Chakravarty <arjoc@intrinsic.ai>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Inbox

Development

Successfully merging this pull request may close these issues.

4 participants