Skip to content

fix: filter unlisted pages from llms.txt when sitemap fallback is used#1401

Open
prajakta128 wants to merge 2 commits into
flare-foundation:mainfrom
prajakta128:fix/unlisted-paths-not-filtered
Open

fix: filter unlisted pages from llms.txt when sitemap fallback is used#1401
prajakta128 wants to merge 2 commits into
flare-foundation:mainfrom
prajakta128:fix/unlisted-paths-not-filtered

Conversation

@prajakta128
Copy link
Copy Markdown

This PR fixes a bug where pages marked unlisted: true in frontmatter were never filtered out of generated llms.txt files when the sitemap fallback path was used.

The issue was caused because buildReplacementsAndUnlisted() correctly returned both replacements and unlistedPaths, but main() only extracted replacements. As a result, the computed unlistedPaths set was discarded before it could be used during pruning.

Problematic code:

const { replacements } = buildReplacementsAndUnlisted();

Fixed code:

const { replacements, unlistedPaths } =
  buildReplacementsAndUnlisted();

This bug is silent in production environments where sitemap.xml exists, but it becomes active in local development and partial CI builds that rely on the globalData.json fallback path.

Changes included:

  • Threaded unlistedPaths through main()
  • Updated pruneAndMarkdownify() signature to accept unlistedPaths
  • Added explicit unlistedPaths.has(route) filtering logic
  • Updated processLlmsTxt() signature and call sites to pass unlistedPaths

No new dependencies added. No behavior change when sitemap.xml is present.

@prajakta128 prajakta128 marked this pull request as draft May 22, 2026 18:45
@prajakta128 prajakta128 marked this pull request as ready for review May 22, 2026 18:46
@prajakta128
Copy link
Copy Markdown
Author

Hi,

I have submitted PR #1401 for review. This fix addresses
a bug where unlisted pages were not being filtered from
llms.txt when the sitemap fallback path was used.

Could you please:

  1. Approve the pending workflow to allow the CI checks to run
  2. Review and approve the pull request

Summary of changes:

  • Threaded unlistedPaths through axis()
  • Updated processMarkdownly() to accept unlistedPaths
  • Added explicit unlistedPaths.has(route) filtering logic
  • Updated process.Init() signature to pass unlistedPaths

No new dependencies have been added. There is no behavior
change when sitemap.xml is present.

PR: #1401

Please let me know if you have any questions or require
any changes.

Thank you.

@prajakta128 prajakta128 mentioned this pull request May 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant