Speed up MIB compilation with IndexedFileReader#22953
Speed up MIB compilation with IndexedFileReader#22953FlorianVeaux wants to merge 5 commits intomasterfrom
Conversation
Replace pysmi's FileReader with an IndexedFileReader that indexes the source directory once at construction, then does O(1) dict lookups instead of scanning the filesystem on every MIB lookup. Profiling showed FileReader.get_subdirs() consumed 89% of compilation time (1.7M stat() calls for 20 MIBs). This change brings compilation of 11,436 MIBs from ~45 min down to ~5 min. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
This PR does not modify any files shipped with the agent. To help streamline the release process, please consider adding the |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 03e583fbb1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
datadog_checks_dev/datadog_checks/dev/tooling/commands/meta/snmp/generate_traps_db.py
Show resolved
Hide resolved
datadog_checks_dev/datadog_checks/dev/tooling/commands/meta/snmp/generate_traps_db.py
Outdated
Show resolved
Hide resolved
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files🚀 New features to boost your workflow:
|
Summary
FileReaderwith a custom_IndexedFileReaderthat indexes the source directory once at construction, then does O(1) dict lookups instead of scanning the filesystem on every MIB lookupFileReader.get_subdirs()consumed 89% of compilation time — 1.7Mstat()calls for just 20 MIBs, repeated for every singlegetData()call across 11,436 MIBs_IndexedFileReaderdelegates to pysmi'sget_mib_variants()for fuzzy matching so behavior is identical to the stock readerResult: compilation of 11,436 MIBs goes from ~60 min → ~8 min
Test plan
invoke generate-traps-dbon the full 11,436 MIB corpusdd_traps_db.jsonagainst committed version — zero missing MIBs, identical coverageget_subdirsis no longer in the hot path🤖 Generated with Claude Code