feat: filter out docs#297

Draft

forshev wants to merge 20 commits intomainfrom

290-filter-out-docs

Contributor

forshev commented Dec 12, 2025 •

edited

Loading

Description

This is PoC of "delete imitation"

Fixes #290

I have read and followed all requirements in CONTRIBUTING.md;
I used LLM/AI assistance to make this pull request;

If you have used LLM/AI assistance please provide model name and full prompt:

Model: {{model-name}}
Prompt: {{prompt}}

forshev marked this pull request as draft

December 12, 2025 13:33

codecov-commenter commented Dec 12, 2025 •

edited

Loading

Codecov Report

❌ Patch coverage is 55.70891% with 353 lines in your changes missing coverage. Please review.
✅ Project coverage is 70.87%. Comparing base (77fae2c) to head (fc34615).

Files with missing lines	Patch %	Lines
docsfilter/docs_filter.go	21.42%	134 Missing and 9 partials ⚠️
docsfilter/loader.go	44.44%	19 Missing and 16 partials ⚠️
docsfilter/encoding.go	78.80%	17 Missing and 15 partials ⚠️
util/fs.go	38.77%	17 Missing and 13 partials ⚠️
frac/active_index.go	63.04%	15 Missing and 2 partials ⚠️
docsfilter/filter.go	0.00%	16 Missing ⚠️
cmd/seq-db/seq-db.go	0.00%	14 Missing ⚠️
docsfilter/merged_iterator.go	81.48%	7 Missing and 3 partials ⚠️
frac/sealed_index.go	70.96%	8 Missing and 1 partial ⚠️
docsfilter/iterator_asc.go	84.00%	5 Missing and 3 partials ⚠️
... and 8 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #297      +/-   ##
==========================================
- Coverage   71.59%   70.87%   -0.73%     
==========================================
  Files         204      213       +9     
  Lines       14770    15501     +731     
==========================================
+ Hits        10575    10986     +411     
- Misses       3441     3708     +267     
- Partials      754      807      +53

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

cheb0 reviewed

View reviewed changes

docsfilter/metrics.go Outdated

+              	diskUsage = promauto.NewGaugeVec(prometheus.GaugeOpts{
+              		Namespace: "seq_db_store",
+              		Subsystem: "filters",
+              		Name:      "disk_usage_bytes_total",

Member

cheb0 Dec 15, 2025

According to company standards, only counters now can (and must) have _total suffix. Gauges can not end with _total

forshev force-pushed the 290-filter-out-docs branch from 6b1e6c5 to f98406a Compare

December 26, 2025 11:24

forshev force-pushed the 290-filter-out-docs branch from 35c810c to 7baeaef Compare

January 21, 2026 09:05

Contributor

github-actions bot commented Jan 21, 2026

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table

Name	Previous	Current	Ratio	Verdict
`FindSequence_Random/medium-4`	`467eb3`	`61c22e`
	`9583.66 MB/s`	`7535.90 MB/s`	`0.79`	`🔴`
	`106.80 ns/op`	`135.90 ns/op`	`1.27`	`🔴`
`FindSequence_Random/small-4`	`467eb3`	`61c22e`
	`5128.78 MB/s`	`4585.96 MB/s`	`0.89`	`🔴`
	`49.91 ns/op`	`55.82 ns/op`	`1.12`	`🔴`
`Indexer-4`	`467eb3`	`61c22e`
	`265877562.00 ns/op`	`296289980.00 ns/op`	`1.11`	`🔴`

forshev force-pushed the 290-filter-out-docs branch from 7baeaef to 5e040ab Compare

January 21, 2026 09:10

Contributor

github-actions bot commented Jan 21, 2026

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table

Name	Previous	Current	Ratio	Verdict
`FindSequence_Random/small-4`	`467eb3`	`1045a0`
	`5128.78 MB/s`	`3914.73 MB/s`	`0.76`	`🔴`
	`49.91 ns/op`	`65.39 ns/op`	`1.31`	`🔴`
`GenerateDocs-4`	`467eb3`	`1045a0`
	`975.70 ns/op`	`1179.00 ns/op`	`1.21`	`🔴`

forshev force-pushed the 290-filter-out-docs branch from 5e040ab to 5fb5af0 Compare

January 21, 2026 12:43

Contributor

github-actions bot commented Jan 21, 2026

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table

Name	Previous	Current	Ratio	Verdict
`ESBulk-4`	`abfd3f`	`5035be`
	`378.00 B/op`	`444.00 B/op`	`1.17`	`🔴`
`FindSequence_Random/medium-4`	`abfd3f`	`5035be`
	`12756.46 MB/s`	`9068.94 MB/s`	`0.71`	`🔴`
	`80.27 ns/op`	`112.90 ns/op`	`1.41`	`🔴`
`FindSequence_Random/small-4`	`abfd3f`	`5035be`
	`6511.28 MB/s`	`4151.19 MB/s`	`0.64`	`🔴`
	`39.32 ns/op`	`61.67 ns/op`	`1.57`	`🔴`

forshev force-pushed the 290-filter-out-docs branch from 5fb5af0 to 277508f Compare

January 23, 2026 09:59

Contributor

github-actions bot commented Jan 23, 2026

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table

Name	Previous	Current	Ratio	Verdict
`Bitmask-4`	`32489f`	`90d653`
	`0.00 B/op`	`3.00 B/op`	`NaN`	`🔴`
`FindSequence_Random/medium-4`	`32489f`	`90d653`
	`9785.69 MB/s`	`8062.37 MB/s`	`0.82`	`🔴`
	`104.60 ns/op`	`127.00 ns/op`	`1.21`	`🔴`
`GenerateDocs-4`	`32489f`	`90d653`
	`958.10 ns/op`	`1143.00 ns/op`	`1.19`	`🔴`

forshev force-pushed the 290-filter-out-docs branch from 4846a3e to 3deb507 Compare

February 3, 2026 11:10

Contributor

github-actions bot commented Feb 3, 2026

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table

Name	Previous	Current	Ratio	Verdict
`FindSequence_Random/small-4`	`174405`	`7d5d8f`
	`10280.15 MB/s`	`4422.80 MB/s`	`0.43`	`🔴`
	`24.90 ns/op`	`57.88 ns/op`	`2.32`	`🔴`
`FindSequence_Random/tiny-4`	`174405`	`7d5d8f`
	`2491.86 MB/s`	`2183.17 MB/s`	`0.88`	`🔴`
	`25.68 ns/op`	`29.32 ns/op`	`1.14`	`🔴`
`GenerateDocs-4`	`174405`	`7d5d8f`
	`952.20 ns/op`	`1126.00 ns/op`	`1.18`	`🔴`
`ProcessDocuments-4`	`174405`	`7d5d8f`
	`381.00 B/op`	`33066.00 B/op`	`86.79`	`🔴`
	`4.00 allocs/op`	`5.00 allocs/op`	`1.25`	`🔴`

forshev force-pushed the 290-filter-out-docs branch from 3deb507 to 6b2cfb2 Compare

February 3, 2026 14:03

Contributor

github-actions bot commented Feb 3, 2026

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table

Name	Previous	Current	Ratio	Verdict
`FindSequence_Random/small-4`	`174405`	`4712c5`
	`10280.15 MB/s`	`4097.76 MB/s`	`0.40`	`🔴`
	`24.90 ns/op`	`62.47 ns/op`	`2.51`	`🔴`

forshev added 14 commits

February 5, 2026 11:11


          feat(config): allow multiple filtering requests

8b8ca04


          feat(config): allow multiple filtering requests

4b10175


          feat(config): allow multiple filtering requests

f3b1f61


          feat: remove imitation PoC - write path

ce32cf4


          feat(docsfilter): store lids in binary format

a33d824


          feat(docsfilter): store lids in binary format

ad64838


          feat(docsfilter): store lids in binary format

53875cf


          feat(docsfilter): method to read filtered lids

b7c98b6


          feat(docsfilter): filter ids on search

a78ec41


          feat(docsfilter): filter ids on fetch for sealed fracs

774d552


          feat(docsfilter): support filtering on active fraction

a41d959


          feat(docsfilter): refresh frac tobstones after sealing

a320cd2


          fix: fix rebase

53fe8ee


          fix: fix tests

58a1b42

forshev added 5 commits

February 5, 2026 11:11


          chore(docsfilter): organize tombstone files in blocks

ccde368


          chore(docsfilter): compress lids blocks in tombstone files

dcc53ef


          chore(docsfilter): handle errors

12f0274


          chore(docsfilter): add iterator to lazy load tombstones

f85bbec


          chore(docsfilter): merge multiple iterators into one

37173f0

forshev force-pushed the 290-filter-out-docs branch from 6b2cfb2 to 37173f0 Compare

February 5, 2026 06:11

Contributor

github-actions bot commented Feb 5, 2026

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table

Name	Previous	Current	Ratio	Verdict
`FindSequence_Random/large-4`	`77fae2`	`f8591d`
	`13987.92 MB/s`	`12148.77 MB/s`	`0.87`	`🔴`
	`1171.00 ns/op`	`1349.00 ns/op`	`1.15`	`🔴`
`FindSequence_Random/small-4`	`77fae2`	`f8591d`
	`5061.19 MB/s`	`4319.71 MB/s`	`0.85`	`🔴`
	`50.58 ns/op`	`59.26 ns/op`	`1.17`	`🔴`
`GenerateDocs-4`	`77fae2`	`f8591d`
	`949.50 ns/op`	`1106.00 ns/op`	`1.16`	`🔴`

Contributor

github-actions bot commented Feb 5, 2026

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table

Name	Previous	Current	Ratio	Verdict
`FindSequence_Random/tiny-4`	`77fae2`	`ef9cdd`
	`2850.02 MB/s`	`2399.97 MB/s`	`0.84`	`🔴`
	`22.46 ns/op`	`26.67 ns/op`	`1.19`	`🔴`


          chore(docsfilter): use iterators in search and fetch

fc34615

forshev force-pushed the 290-filter-out-docs branch from 1e19ca1 to fc34615 Compare

February 5, 2026 14:33

Contributor

github-actions bot commented Feb 5, 2026

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table

Name	Previous	Current	Ratio	Verdict
`FindSequence_Random/medium-4`	`77fae2`	`32ec58`
	`11769.73 MB/s`	`10062.12 MB/s`	`0.85`	`🔴`
	`87.00 ns/op`	`101.80 ns/op`	`1.17`	`🔴`
`FindSequence_Random/tiny-4`	`77fae2`	`32ec58`
	`2850.02 MB/s`	`2247.37 MB/s`	`0.79`	`🔴`
	`22.46 ns/op`	`28.48 ns/op`	`1.27`	`🔴`
`GenerateDocs-4`	`77fae2`	`32ec58`
	`949.50 ns/op`	`1120.00 ns/op`	`1.18`	`🔴`

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet