Don't index sitemap in opensearch

[JaspreetSinghChahal]

In stormcrawler, I want to fetch and parse sitemaps to discover new urls but don't want to index it in opensearch.

I can't exclude it in regex as new urls won't be discovered then.

[rzo1]

The sitemap parser bolt will set a metadata key isSitemap and use filter keys in the indexer's configuration to remove them from indexing: indexer.md.filter

[JaspreetSinghChahal]

It seems to be taking only single key. How to do it for more keys, example "isSitemap=false", "isFeed=false"?

[rzo1]

@JaspreetSinghChahal There’s a mismatch between the current implementation and the documentation. From my perspective, it makes more sense to align the implementation with what the documentation specifies. Therefore, it would be useful to allow this parameter to accept a YAML list instead of just a single key-value.

Any implementation should remain backward compatible, still supporting the previous configuration format. Would you be interested in submitting a PR with the fix and a test?

[JaspreetSinghChahal]

I fixed this for me.

For PR I have a doubt , I needed to exclude documents with isFeed=true or isSitemap=true in metadata. That meant documents with this value as false or not present should be included. It is in contradiction to current implementation . If I make it inclusive
i.e say isFeed=false then documents were this field is no present is also excluded.

[rzo1]

Yes, you are right. The current implementation is a bit weird in that sense.

From my POV, filtering is exclusion-based, not inclusion-based, i.e. “exclude documents matching these conditions”.

That means:

• If a document has key=value matching the filter, it should be filtered out
• If the key is not present, the document should not be filtered
• Multiple filters should be supported (as the docs claim) - guess it would be or between multiple filters.

For each filter entry:

• If key not present -> ignore
• If key present and value matches -> reject document

In that sense, I think, that we should fix it (or introduce a new config option to be backward compatbile).