fix(app-router): stream generated metadata for non-html bots#1585
Open
NathanDrake2406 wants to merge 5 commits into
Open
fix(app-router): stream generated metadata for non-html bots#1585NathanDrake2406 wants to merge 5 commits into
NathanDrake2406 wants to merge 5 commits into
Conversation
Dynamic App Router metadata was always folded into the route head. That diverged from Next.js for normal browser-like requests, where generated metadata is sent through a body outlet while html-limited bots still receive blocking head metadata. Track whether the matched route uses generateMetadata, thread next.config htmlLimitedBots into the generated RSC entry, and choose head versus body placement at page element construction.
commit: |
Generated metadata must keep the default html-limited bot list when serialized config contains a falsy regex source. Empty config strings previously produced an empty regex, which treated every user agent as blocking and moved generated metadata back into the head. Add regression coverage for the default Twitterbot path and the streaming body serializer, then match Next.js by falling back to the default bot regex for any falsy htmlLimitedBots value.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR aligns vinext App Router metadata streaming behavior with Next.js by conditionally emitting generateMetadata-driven metadata via a hidden body outlet for streaming-capable user agents, while keeping metadata head-blocking for html-limited bots (default list + configurable htmlLimitedBots).
Changes:
- Adds
hasDynamicMetadatato App Router head resolution to detectgenerateMetadataparticipation (including parallel routes). - Threads
htmlLimitedBotsfromnext.configthrough the generated RSC entry into page element construction, and uses UA matching to choose"head"vs"body"metadata placement. - Introduces a small metadata-to-HTML serializer and adds unit tests covering default bots, configured bots, and falsy-config fallback behavior.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/next-config.test.ts | Updates resolved config expectations to include htmlLimitedBots. |
| tests/features.test.ts | Adds coverage for renderMetadataToHtml serialization behavior. |
| tests/app-page-route-wiring.test.ts | Adds routing-level assertions for head vs hidden-body metadata placement across UA/config cases. |
| tests/app-page-head.test.ts | Adds coverage for hasDynamicMetadata detection. |
| packages/vinext/src/shims/metadata.tsx | Adds renderMetadataToHtml with HTML escaping + element serialization. |
| packages/vinext/src/server/streaming-metadata.ts | Introduces UA-based shouldServeStreamingMetadata helper with default html-limited bot list. |
| packages/vinext/src/server/app-page-route-wiring.tsx | Adds metadataPlacement support and renders body metadata outlet using inert HTML. |
| packages/vinext/src/server/app-page-head.ts | Computes and returns hasDynamicMetadata from layouts/pages/parallel routes. |
| packages/vinext/src/server/app-page-element-builder.ts | Chooses metadata placement based on hasDynamicMetadata + UA matching + htmlLimitedBots. |
| packages/vinext/src/index.ts | Threads htmlLimitedBots into the App Router config passed to entry generation. |
| packages/vinext/src/entries/app-rsc-entry.ts | Embeds serialized htmlLimitedBots into the generated RSC entry and passes it into page building. |
| packages/vinext/src/config/next-config.ts | Adds htmlLimitedBots to config types and resolves it to a serialized regex source string. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Invalid serialized htmlLimitedBots values could reach request handling and throw while building the bot matcher. That puts a config error on the request path and recompiles the same matcher for repeated requests. Validate serialized regex sources while resolving next.config and reuse compiled bot matchers in the streaming metadata helper. The focused tests cover invalid config, falsy fallback, and matcher reuse.
Config resolution had several local assertions around experimental options, Turbopack aliases, output mode, and Sass options. Those assertions made the boundary wider than necessary in the code adjacent to htmlLimitedBots handling. Read optional records, strings, arrays, and body-size inputs through typed narrowing helpers so the resolved config path keeps the same semantics without spreading unchecked asserted values.
Config resolution imported the streaming metadata server module to validate htmlLimitedBots. In RSC-backed App Router integration tests, that cross-layer import can pull server runtime modules through config resolution and surface rsc reference-validation errors. Move the shared html-limited bot matcher cache to a neutral utils module and import it from config and the server streaming metadata helper. This keeps config validation independent from the server runtime graph while preserving the cached matcher behavior.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Upstream deploy-suite status
test/e2e/app-dir/metadata-streaming/metadata-streaming-customized-rule.test.tsrun-nextjs-deploy-suite.shtest/e2e/app-dir/metadata-streaming/metadata-streaming.test.tsOverview
generateMetadataparticipation during head resolution, then choose head or hidden body placement from the request user agent.app-page-head.ts,app-page-element-builder.ts,app-page-route-wiring.tsx,metadata.tsx,streaming-metadata.tsWhy
Generated metadata is not always a head-blocking contract in Next.js. For streaming-capable requests, Next renders generated metadata through a hidden body outlet so the initial document can stream without waiting on metadata. For html-limited bots, Next blocks and emits metadata in the head because those crawlers require head-visible tags.
generateMetadata.hasDynamicMetadatato the resolved head result.htmlLimitedBotsthrough config and generated RSC entry into page element construction.<title>and<meta>unless body metadata is emitted as inert HTML.What changed
generateMetadata, normal UAgenerateMetadata, default html-limited botTwitterbotcoverage pins head-blocking metadata for the default bot list.generateMetadata, configured html-limited bothtmlLimitedBotskeeps generated metadata in the head.htmlLimitedBotsMaintainer review path
packages/vinext/src/server/app-page-head.ts:hasDynamicMetadatais derived from layout, page, and active parallel route modules.packages/vinext/src/server/app-page-element-builder.ts: request UA plushtmlLimitedBotsdecides body versus head placement.packages/vinext/src/server/app-page-route-wiring.tsxandpackages/vinext/src/shims/metadata.tsx: body metadata is emitted without letting React hoist it back into the head.packages/vinext/src/config/next-config.ts,packages/vinext/src/entries/app-rsc-entry.ts,packages/vinext/src/index.ts: config threading only.Validation
vp test run tests/app-page-route-wiring.test.ts -t "falls back to the default html-limited bot list": failed before thehtmlLimitedBots || DEFAULTfix, then passed.vp test run tests/streaming-metadata.test.ts tests/next-config.test.ts -t "streaming metadata bot matching|resolveNextConfig htmlLimitedBots": 6/6 selected tests passed.vp test run tests/app-page-head.test.ts tests/app-page-route-wiring.test.ts tests/features.test.ts tests/entry-templates.test.ts tests/next-config.test.ts tests/streaming-metadata.test.ts: 496/496 passed.vp check: formatting, lint, and typecheck passed with no warnings.vp env exec --node 24 ./scripts/run-nextjs-deploy-suite.sh /Users/nathan/Projects/vinext/.refs/nextjs-v16.2.6 --retries 0 -c 2 --debug test/e2e/app-dir/metadata-streaming/metadata-streaming.test.ts test/e2e/app-dir/metadata-streaming/metadata-streaming-customized-rule.test.ts: customized-rule file passed 2/2; broader metadata-streaming file is intentionally not claimed green in this PR.Non-goals
metadata-streaming.test.tsfile.notFound()andredirect()status or boundary behavior from metadata navigation cases.References
shouldServeStreamingMetadatahtmlLimitedBotshtmlLimitedBotsnormalizing to the default bot regex.serveStreamingMetadatafrom the request UA and config.htmlLimitedBotsregression.