Skip to content

Speed up MIB compilation with IndexedFileReader#22953

Open
FlorianVeaux wants to merge 5 commits intomasterfrom
flo/speed-up-mib-compilation-v2
Open

Speed up MIB compilation with IndexedFileReader#22953
FlorianVeaux wants to merge 5 commits intomasterfrom
flo/speed-up-mib-compilation-v2

Conversation

@FlorianVeaux
Copy link
Member

@FlorianVeaux FlorianVeaux commented Mar 18, 2026

Summary

  • Replace pysmi's FileReader with a custom _IndexedFileReader that indexes the source directory once at construction, then does O(1) dict lookups instead of scanning the filesystem on every MIB lookup
  • Profiling showed FileReader.get_subdirs() consumed 89% of compilation time — 1.7M stat() calls for just 20 MIBs, repeated for every single getData() call across 11,436 MIBs
  • _IndexedFileReader delegates to pysmi's get_mib_variants() for fuzzy matching so behavior is identical to the stock reader

Result: compilation of 11,436 MIBs goes from ~60 min → ~8 min

Test plan

  • Run invoke generate-traps-db on the full 11,436 MIB corpus
  • Diff output dd_traps_db.json against committed version — zero missing MIBs, identical coverage
  • Verified with cProfile that get_subdirs is no longer in the hot path

🤖 Generated with Claude Code

Replace pysmi's FileReader with an IndexedFileReader that indexes the
source directory once at construction, then does O(1) dict lookups
instead of scanning the filesystem on every MIB lookup.

Profiling showed FileReader.get_subdirs() consumed 89% of compilation
time (1.7M stat() calls for 20 MIBs). This change brings compilation
of 11,436 MIBs from ~45 min down to ~5 min.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@github-actions
Copy link
Contributor

⚠️ Recommendation: Add qa/skip-qa label

This PR does not modify any files shipped with the agent.

To help streamline the release process, please consider adding the qa/skip-qa label if these changes do not require QA testing.

@FlorianVeaux FlorianVeaux added the qa/skip-qa Automatically skip this PR for the next QA label Mar 18, 2026
Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 03e583fbb1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@codecov
Copy link

codecov bot commented Mar 18, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.05%. Comparing base (f410c11) to head (3ef21db).
⚠️ Report is 3 commits behind head on master.

Additional details and impacted files
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant