[Refactor] Habitat Refactor & Support Distributed VLN-PE Evaluation #168

kew6688 · 2025-11-17T13:00:56Z

PR Summary

Refactored Habitat integration into the InternNav framework for improved readability and reusability.
- Maintained a unified evaluation script for all InternNav benchmarks with the same config format and starter workflow.
- Created a reusable Distributed Evaluator that encapsulates distribution logic for multiple benchmarks.
- Added a Habitat Env wrapper that handles reset/step, episode assignment, and episode reloading inside the worker, significantly streamlining the evaluation pipeline.
Refactored VLNPE evaluation flow:
- Added an option to bypass the agent server for models that cannot process batched observations, splitting evaluation per distributed rank.
- Now VlnMultiEvaluator inherited from DistributedEvaluator, the same to Habitat. Ensured each Isaac Sim simulator and model instance is bound to a dedicated GPU, aligning with the Habitat evaluator’s design and enabling deployment on both Aliyun DLC and Slurm.
- refactor new episode loader and generate all episodes under internutopia env. Keep the evaluator logic clean and align with Habitat evaluation workflow.

Distributed Eval Time

Using 16 nodes, each with 1× RTX 4090 GPU, 8 CPUs, and 60 GB RAM.

Model Variant	GPUs Used	Previous Runtime (Single 4090)	Distributed Runtime (16×4090)	Speedup
InternVLA-N1 Flash	1 GPU → 16 GPUs	~13.5 hours	~1 hour	13× faster
InternVLA-N1 PE`	1 GPU → 16 GPUs	~21 hours	~1.6 hour	13× faster

Performance after Refactor

Re-ran InternNav benchmarks (Internutopia VLNPE + Habitat VLNCE) three times post-refactor, matching previous performance.

Model	Dataset/Benchmark	NE	OS	SR	SPL
`InternVLA-N1`	Habitat R2R	4.88	62.2	57.0	52.4
`InternVLA-N1`	Flash	4.17	67.2	59.8	53.9
`InternVLA-N1`	PE	4.87	55.7	50.0	42.9
`RDP`	Flash	7.11	41.7	24.3	17.4
`RDP`	PE	6.73	38.0	26.3	18.6

…nd agent created

…aac sim gpu setting

…d on lmdb path + rank; remove TODOs and unused files

yuqiang-yang · 2025-11-21T13:00:29Z

scripts/eval/eval.py

+        from internnav.configs.evaluator.vln_default_config import get_config
+
+        evaluator_cfg = get_config(evaluator_cfg)
+    elif evaluator_cfg.eval_type == 'habitat_vln':


这个是干什么的，habitat vln

这个flag没有用到，我去掉

yuqiang-yang · 2025-11-21T13:02:22Z

scripts/eval/eval.py

        default='scripts/eval/configs/h1_rdp_cfg.py',
        help='eval config file path, e.g. scripts/eval/configs/h1_cma_cfg.py',
    )
+    parser.add_argument('--port', type=int, default=None)


我看server里面写了默认的host和port，这里也写一下会不会好一些，方便跑

好的，已在AgentConfig里面加了默认的host和port

scripts/eval/bash/eval_vln_distributed.sh

scripts/eval/bash/eval_dual_system.sh

internnav/env/utils/episode_loader/dataset_utils.py

internnav/evaluator/__init__.py

internnav/evaluator/distributed_base.py

…tics#176) * [file] Update 3d printing files for camera of Unitree Go2. * [Doc] Update readme for 3d printing models.

internnav/evaluator/utils/dataset.py

internnav/habitat_extensions/habitat_default_evaluator.py

internnav/habitat_extensions/habitat_vln_evaluator.py

This reverts commit 7a23b72, reversing changes made to ea73075.

* update vln yaml; fix import agent * update habitat, using evaluator and config; env and agent is WIP * add distributed_base evaluator * Habitat env applied, distributed evaluator applied; clean evaluator and agent created * fix observation issues * update new register name; tiny fix on style * latest tested * delete temp agent; rename default evaluator for habitat * update slurm bash * update readme * fix init dist print * fix eval config; fix local rank to rank * update init distributed mode if condition * update dist for dlc * fix bug in evaluator * update distributed vln multi, episode loader * cma tested * cma tested; episode loader; torchrun; local agent tested; TODO fix isaac sim gpu setting * add vlnpe distributed script * add grscene; add new result write and resumble load data feature based on lmdb path + rank; remove TODOs and unused files * fix bugs in evaluator and dataset for distributed; n1 and rdp tested * fix comm log concurrency issue, create file with exist true * fix progress log mkdir race condition * fix comments * polish existing configs and bash * update bash align with doc * fix CI test * rename habitat_extensions * fix comments * remove useless line * [file] Update 3D Printing Files for Camera of Unitree Go2 (InternRobotics#176) * [file] Update 3d printing files for camera of Unitree Go2. * [Doc] Update readme for 3d printing models. * fix visualize image size; fix unused comment code; fixed evaluator name * bump to version v0.0.2 * update vlnmulti to VLN; update habitatVln to HabitatVLN * :Revert "Merge branch 'main' into vlnpe_refactor" This reverts commit 7a23b72, reversing changes made to ea73075. --------- Co-authored-by: Yuqiang Yang <49871992+yuqiang-yang@users.noreply.github.com>

kew6688 added 20 commits November 3, 2025 08:11

update vln yaml; fix import agent

0112c69

update habitat, using evaluator and config; env and agent is WIP

05ea2a3

add distributed_base evaluator

9902401

Habitat env applied, distributed evaluator applied; clean evaluator a…

0d00014

…nd agent created

fix observation issues

7e25e72

update new register name; tiny fix on style

2b0eb8b

latest tested

b414ba3

delete temp agent; rename default evaluator for habitat

99adf73

update slurm bash

75b38a7

merge to main

dcf7ee5

update readme

08bb9c3

fix init dist print

cde84b3

fix eval config; fix local rank to rank

c89723d

update init distributed mode if condition

7836276

update dist for dlc

dac13e1

fix bug in evaluator

d8734c7

update distributed vln multi, episode loader

bfe3616

cma tested

43957f6

cma tested; episode loader; torchrun; local agent tested; TODO fix is…

a9ca15d

…aac sim gpu setting

add vlnpe distributed script

6e93ba0

kew6688 changed the base branch from dev to main November 17, 2025 13:01

kew6688 changed the title ~~[Feat] Habitat refactor & Distributed VLNPE Evaluator for internvla-n1~~ [Fix] Habitat refactor & Distributed VLNPE Evaluator for internvla-n1 Nov 17, 2025

kew6688 changed the title ~~[Fix] Habitat refactor & Distributed VLNPE Evaluator for internvla-n1~~ [Fix] Habitat Refactor & Distributed VLNPE Evaluator for internvla-n1 Nov 17, 2025

kew6688 added 2 commits November 18, 2025 06:55

add grscene; add new result write and resumble load data feature base…

b4af0ce

…d on lmdb path + rank; remove TODOs and unused files

fix bugs in evaluator and dataset for distributed; n1 and rdp tested

ae33f09

kew6688 changed the base branch from main to dev November 19, 2025 12:55

fix comm log concurrency issue, create file with exist true

53258b1

kew6688 marked this pull request as ready for review November 21, 2025 05:33

fix progress log mkdir race condition

50b1f24

kew6688 changed the title ~~[Fix] Habitat Refactor & Distributed VLNPE Evaluator for internvla-n1~~ [Fix] Habitat Refactor & Distributed VLNPE Refactor Nov 21, 2025

yuqiang-yang approved these changes Nov 24, 2025

View reviewed changes

kew6688 added 3 commits November 25, 2025 09:43

fix comments

8665b38

polish existing configs and bash

7d5daa6

update bash align with doc

4f67d9b

kew6688 requested a review from Tai-Wang November 25, 2025 11:07

fix CI test

d7e12e9