MB-65018 add custom_filter/custom_score query nodes with context-driven callback hooks by maneuvertomars · Pull Request #2289 · blevesearch/bleve

maneuvertomars · 2026-02-22T06:30:41Z

This PR adds two new query node types in Bleve: custom_filter and custom_score, both wrapping an inner query.
Execution is embedding-driven via request-scoped context hooks (CustomFilterFactory / CustomScoreFactory) so Bleve remains engine-agnostic.
custom_filter applies per-hit keep/drop logic through FilteringSearcher; custom_score applies per-hit score mutation through ScoreMutatingSearcher.

…en callback hooks

coveralls · 2026-02-22T06:34:01Z

coverage: 52.405% (-0.3%) from 52.691%
when pulling de161f1 on MB-65018
into f7b13b9 on master.

docs/custom_query_hooks.md

CascadingRadium · 2026-02-24T05:59:41Z

search/query/custom_filter.go

+	childSearcher, err := q.QueryVal.Searcher(ctx, i, m, options)
+	if err != nil {
+		return nil, err
+	}
+
+	// Resolve the request-scoped callback builder injected by the embedder.
+	factory, _ := ctx.Value(CustomFilterContextKey).(CustomFilterFactory)
+	if factory == nil {
+		return nil, fmt.Errorf("no custom filter factory registered in context")
+	}
+
+	// Build the per-hit filter callback from query-provided source/params/fields.
+	filterFunc, err := factory(q.Source, q.Params, q.Fields)
+	if err != nil {


I would argue that passing a callback function in the context object is very flaky, and difficult for library users to add directly to their application.

Also if no FilterHook is defined we should not error out i feel and just do unfiltered search.

Why not try this?

func (q *CustomFilterQuery) Searcher(ctx context.Context, i index.IndexReader, m mapping.IndexMapping, options search.SearcherOptions) (search.Searcher, error) { // Build the inner searcher first; custom filtering wraps its output. childSearcher, err := q.QueryVal.Searcher(ctx, i, m, options) if err != nil { return nil, err } // Check if the factor hook is registered if FilterHook == nil { // No filtering hook registered => no filtering to be done // Just return the child searcher itself return childSearcher, nil } filterFunc, err := filterFactory(q.Source, q.Params, q.Fields) if err != nil { return nil, err } }

I think we should fail fast here rather than silently bypass custom_filter/custom_score. If a query explicitly contains a custom node and no hook is registered, returning the child searcher would change query semantics and produce incorrect results. The context-carried hook is intentional so the execution runtime stays request-scoped and embedder-defined, while Bleve remains generic.

Also fair point on standalone Bleve use case. I’ve addressed that by adding helper APIs (WithCustomFilterFactory(...), WithCustomScoreFactory(...), WithCustomFactories(...)) so users don’t have to use raw context keys directly.

ctx := query.WithCustomFilterFactory(context.Background(), filterFactory) res, err := index.SearchInContext(ctx, req)

CascadingRadium · 2026-02-24T06:02:54Z

search/query/custom_filter.go

+
+type CustomFilterQuery struct {
+	// QueryVal is the child query whose candidate matches are filtered.
+	QueryVal Query `json:"query"`


Query as the param name perhaps?

CascadingRadium · 2026-02-24T06:03:58Z

search/query/custom_filter.go

+	}
+	if q.Source == "" {
+		return fmt.Errorf("custom filter query must have source")
+	}


Is a nil Fields param value valid?

Yes, nil/empty Fields is valid by design. It means no stored fields are loaded into doc.fields, I clarified that in the field comments.

CascadingRadium · 2026-02-24T06:05:03Z

search/query/custom_filter.go

+	// Params carries caller-provided values passed as the second UDF argument.
+	Params map[string]interface{} `json:"params,omitempty"`
+	// Source carries embedding-defined callback source that travels with the query.
+	Source string `json:"source,omitempty"`


Since a nil or empty source is invalid we should not have omitempty

CascadingRadium · 2026-02-24T06:14:07Z

search/query/custom_filter.go

+}
+
+func (q *CustomFilterQuery) MarshalJSON() ([]byte, error) {
+	inner := map[string]interface{}{


to allow for easier extensibility of the struct, we can have a local copy of the struct here right?

type inner struct { Query Query `json:"query"` Fields []string `json:"fields,omitempty"` Params map[string]interface{} `json:"params,omitempty"` Source string `json:"source"` } return util.MarshalJSON(struct { CustomFilter inner `json:"custom_filter"` }{ CustomFilter: inner{ Query: q.QueryVal, Fields: q.Fields, Params: q.Params, Source: q.Source, }, })

Makes sense. I switched MarshalJSON() to use a typed local wrapper so the JSON shape stays explicit and easier to extend.

CascadingRadium · 2026-02-24T06:45:42Z

search/query/custom_hooks.go

+
+// CustomScoreFactory lets the embedding application provide request-scoped
+// score callbacks created from query-provided source/params/fields.
+type CustomScoreFactory func(source string, params map[string]interface{}, fields []string) (searcher.ScoreFunc, error)


Instead of having these types and having library users pass in a function in context, consider this.

var ( filterFactory CustomFilterFactory scoreFactory CustomScoreFactory ) func SetFilterFactory(f CustomFilterFactory) { filterFactory = f } func SetScoreFactory(f CustomScoreFactory) { scoreFactory = f } // usage from a bleve API POV ```go bleve.SetFilterFactory(f) bleve.SetFilterFactory(f) // can avoid context overhead entirely

That would make the hook registration package-global in Bleve, whereas the current design keeps the execution hook request-scoped and explicit at search time. I’d prefer avoiding Bleve-level global mutable state here; the context handoff is small, aligns with existing search-time context usage, and keeps the embedder control per request.

CascadingRadium · 2026-02-24T06:50:46Z

search/query/custom_score.go

+	}
+
+	// Resolve the request-scoped callback builder injected by the embedder.
+	factory, _ := ctx.Value(CustomScoreContextKey).(CustomScoreFactory)


ditto, same as before - can avoid context ops by a simple global var init.

Same reasoning here. I’d prefer keeping this request-scoped rather than introducing Bleve-level global mutable hooks. The context lookup cost is negligible relative to search/execution, and the current design keeps the embedder control explicit per request.

abhinavdangeti · 2026-03-18T20:17:20Z

search/query/custom_filter.go

+	// Params carries caller-provided values passed as the second UDF argument.
+	Params map[string]interface{} `json:"params,omitempty"`
+	// Source carries embedding-defined callback source that travels with the query.
+	Source string `json:"source"`


Let's move all this in-line commentary to before declaring the struct (like we do with the rest of the query constructs).

Sure Abhi, it makes sense.

abhinavdangeti · 2026-03-18T22:27:10Z

search/query/custom_score.go

+	// Params carries caller-provided values passed as the second UDF argument.
+	Params map[string]interface{} `json:"params,omitempty"`
+	// Source carries embedding-defined callback source that travels with the query.
+	Source string `json:"source"`


abhinavdangeti · 2026-03-18T22:29:50Z

@maneuvertomars Let's make sure you scrub the proposed code once again for any nil handling, especially around pointer dereferencing.

abhinavdangeti

@maneuvertomars Let's do a quick walk through tomorrow.

abhinavdangeti · 2026-03-24T20:46:34Z

search_test.go

 	req = NewSearchRequest(q)
 	req.Explain = true
-	req.Fields = []string{"title"}
+	req.Fields = []string{"*"}


Why changes to this test case, seems unrelated - please undo.

abhinavdangeti · 2026-03-24T20:48:28Z

search/query/custom_hooks.go

+// Context keys used by CustomFilterQuery/CustomScoreQuery to retrieve
+// request-scoped hooks from the embedding application (e.g. cbft).
+type customFilterKey string
+type customScoreKey string


You don't need two different typecasts for string. Given these aren't exported and you're not really using them anywere - just make the constants below of type strings.

abhinavdangeti

Please wait for review from @CascadingRadium as well.

capemox · 2026-03-30T08:32:38Z

search/query/custom_filter.go

+	}
+
+	// Resolve the request-scoped callback builder injected by the embedder.
+	factory := customFilterFactoryFromContext(ctx)


Explicitly extract the callback from the function, this is unnecessary.

capemox · 2026-03-30T08:35:51Z

search/query/custom_hooks.go

+)
+
+const (
+	CustomFilterContextKey string = "custom_filter"


Nit: you can let these be untyped strings.

capemox · 2026-03-30T10:00:06Z

search/query/custom_hooks.go

+
+// CustomFilterFactory lets the embedding application provide request-scoped
+// filter callbacks created from query-provided source/params/fields.
+type CustomFilterFactory func(source string, params map[string]interface{}, fields []string) (searcher.FilterFunc, error)


Why do you need a factory wrapper around your searcher.FilterFunc? For a user using bleve only, why would they need source, params and fields?

The factory is useful for embedders(cbft in our case) that want to derive a per-hit callback once from query-provided source/params/fields during Searcher() construction. To keep standalone Bleve usage simple, I added direct helpers as well: WithCustomFilterFunc, WithCustomScoreFunc, and WithCustomFuncs, so a Bleve-only user can register plain per-hit callbacks without needing to care about the builder path.
For example:

ctx := bleveQuery.WithCustomFuncs(context.Background(), func(_ *search.SearchContext, d *search.DocumentMatch) bool { return d.Score > 0 }, func(_ *search.SearchContext, d *search.DocumentMatch) float64 { return d.Score + 1 }, ) res, err := index.SearchInContext(ctx, req)

Hmm, CustomFilterFactory and CustomScoreFactory are still cbft specific. You can instead situate these types in cbft only, and call the factory there giving you FilterFunc or ScoreFunc. You can then just pass these down as part of the context to bleve.

Basically right now, you're doing:
query comes in -> get the factory callback, put in the query's context -> forward query to bleve -> bleve executes the factory with the query source, params and fields to get the filter/score func -> use the filter/score func against each hit

Instead, why not do this:
query comes in -> get the factory callback, execute it with the params required (on cbft) -> get the filter/scorer func, put in the query's context -> forward to bleve -> bleve will execute the filter/scorer.

This will make bleve completely oblivious to cbft specific functionality.

capemox · 2026-03-30T10:00:14Z

search/query/custom_hooks.go

+
+// CustomScoreFactory lets the embedding application provide request-scoped
+// score callbacks created from query-provided source/params/fields.
+type CustomScoreFactory func(source string, params map[string]interface{}, fields []string) (searcher.ScoreFunc, error)


Same reasoning as above.

capemox · 2026-03-30T10:01:49Z

search/query/custom_hooks.go

+
+// WithCustomFactories returns a context carrying both request-scoped
+// custom query factories for search execution.
+func WithCustomFactories(ctx context.Context, filterFactory CustomFilterFactory,


Factory methods are not needed IMO, users can just directly register the callbacks in the context.

Users can register directly in context, but I’m keeping the helpers because they make the supported contract explicit and avoid repeating the context-key plumbing at each call site.
Also, standalone Bleve users now have a direct callback path via WithCustomFilterFunc / WithCustomScoreFunc / WithCustomFuncs, while embedders that need one-time callback construction from source / params / fields can still use the factory-based helpers.

capemox · 2026-03-30T10:07:09Z

search/searcher/search_score_mutator.go

+}
+
+// ScoreFunc defines a function which can mutate document scores
+type ScoreFunc func(sctx *search.SearchContext, d *search.DocumentMatch) float64


ScoreFunc is a callback that's created outside the index, but SearchContext is an internal variable that has access to the index reader, collector and document match pool, all of which can potentially be misused. I don't think access to this struct should be given.

Looks like FilterFunc which was implemented as part of boolean queries follows this signature as well, but that was used internally only. Since these callbacks are registered outside bleve I still think this might be a bad idea. @CascadingRadium what do you think?

capemox · 2026-03-30T11:12:30Z

@maneuvertomars please add functions to create new custom_filter/custom_score queries in query.go in bleve's root, and also add end to end tests in search_test.go to validate it works with some non-trivial custom filters and custom scorers.

capemox · 2026-03-31T09:43:46Z

search/query/custom_filter.go

+	Fields []string               `json:"fields,omitempty"`
+	Params map[string]interface{} `json:"params,omitempty"`
+	Source string                 `json:"source"`


For a user using only bleve, these additional query args don't make much sense and are cbft specific

capemox · 2026-03-31T09:44:28Z

search/query/custom_score.go

+	Fields []string               `json:"fields,omitempty"`
+	Params map[string]interface{} `json:"params,omitempty"`
+	Source string                 `json:"source"`


abhinavdangeti · 2026-04-01T19:34:52Z

docs/custom_query_hooks.md

+`CustomFilterQuery` contains:
+
+- a child query
+- a bound `searcher.FilterFunc`


What does bound here and below supposed to indicate?

abhinavdangeti · 2026-04-01T19:36:39Z

docs/custom_query_hooks.md

+
+```go
+filterQuery := bleve.NewCustomFilterQuery(childQuery)
+scoreQuery := bleve.NewCustomScoreQuery(childQuery)


Looks like here and elsewhere you need to update with how a user is meant to pass in filters and payloads.

abhinavdangeti · 2026-04-01T19:38:55Z

search/query/custom_filter.go

+}
+
+func (q *CustomFilterQuery) UnmarshalJSON(data []byte) error {
+	child, payload, err := unmarshalCustomQueryPayload(data, "custom_filter")


This is bound to fail if the payload does not comply to map[string]map[string]json.RawMessage - which is going to be pretty bad UX.

abhinavdangeti · 2026-04-01T19:39:16Z

search/query/custom_score.go

+}
+
+func (q *CustomScoreQuery) UnmarshalJSON(data []byte) error {
+	child, payload, err := unmarshalCustomQueryPayload(data, "custom_score")


abhinavdangeti · 2026-04-01T19:41:35Z

search/query/custom_payload.go

+			return nil, nil, err
+		}
+	}
+


I'm confused at what we're doing here - why this query attribute within the payload, if there's one outside already?

abhinavdangeti · 2026-04-01T19:44:08Z

search/query/query.go

 }

+var CustomFilterQueryParser func([]byte) (Query, error)
+var CustomScoreQueryParser func([]byte) (Query, error)


Declare each of these alongside their respective query types - and add sufficient commentary around how these can be updated only during init and with examples of how bleve users can use these.

abhinavdangeti · 2026-04-01T19:48:56Z

search/searcher/search_score_mutator.go

+
+// ScoreMutatingSearcher wraps any other searcher, but mutates the score
+// of any Next/Advance call using the supplied ScoreFunc
+type ScoreMutatingSearcher struct {


Refactor to CustomScoreSearcher.

abhinavdangeti · 2026-04-01T19:49:50Z

query.go

+// child query and binding the provided per-hit score callback.
+func NewCustomScoreQueryWithScorer(child query.Query, score searcher.ScoreFunc) *query.CustomScoreQuery {
+	return query.NewCustomScoreQueryWithScorer(child, score)
+}


Let's drop all these convenience redirectors for now.

abhinavdangeti · 2026-04-01T19:50:36Z

search/query/custom_filter.go

+	}
+}
+
+func NewCustomFilterQueryWithFilterAndPayload(query Query, filter searcher.FilterFunc, payload map[string]interface{}) *CustomFilterQuery {


Drop this, and make NewCustomFilterQueryWithFilter handle the payload as well - you'll want to accommodate nil payload situations.

abhinavdangeti · 2026-04-01T19:50:54Z

search/query/custom_score.go

+	}
+}
+
+func NewCustomScoreQueryWithScorerAndPayload(query Query, score searcher.ScoreFunc, payload map[string]interface{}) *CustomScoreQuery {


Drop this, and make NewCustomScoreQueryWithScorer handle the payload as well - you'll want to accommodate nil payload situations.

MB-65018 add custom_filter/custom_score query nodes with context-driv…

39964ff

…en callback hooks

maneuvertomars requested review from CascadingRadium, Likith101, Thejas-bhat, abhinavdangeti and capemox February 22, 2026 06:31

CascadingRadium added this to Script Queries Feb 23, 2026

github-project-automation bot moved this to Todo in Script Queries Feb 23, 2026

Added a md file for context

289490f

CascadingRadium requested changes Feb 24, 2026

View reviewed changes

abhinavdangeti added this to the v2.6.0 milestone Feb 24, 2026

maneuvertomars added 2 commits March 9, 2026 20:01

Refine custom query hook API and JSON handling

6aea7ea

Add helper APIs for custom query hook context wiring for sole bleve user

5e08a5a

abhinavdangeti reviewed Mar 18, 2026

View reviewed changes

added nil handling around pointer dereferencing

22ea158

abhinavdangeti requested changes Mar 24, 2026

View reviewed changes

maneuvertomars added 3 commits March 25, 2026 19:20

Updated plain string type for custom keys

1f4dc6a

Removed redundant test

80c4822

Removed redundant change in test file

c03f095

abhinavdangeti previously approved these changes Mar 27, 2026

View reviewed changes

capemox requested changes Mar 30, 2026

View reviewed changes

Add end to end coverage and root constructors for custom queries

2e38bc1

maneuvertomars dismissed abhinavdangeti’s stale review via 2e38bc1 March 30, 2026 14:07

capemox reviewed Mar 31, 2026

View reviewed changes

Use interface to carry payload

de161f1

abhinavdangeti requested changes Apr 1, 2026

View reviewed changes

Conversation

maneuvertomars commented Feb 22, 2026

Uh oh!

coveralls commented Feb 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maneuvertomars Mar 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maneuvertomars Mar 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

abhinavdangeti commented Mar 18, 2026

Uh oh!

abhinavdangeti left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

abhinavdangeti left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maneuvertomars Mar 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

capemox commented Mar 30, 2026

coveralls commented Feb 22, 2026 •

edited

Loading

maneuvertomars Mar 12, 2026 •

edited

Loading

maneuvertomars Mar 9, 2026 •

edited

Loading

maneuvertomars Mar 30, 2026 •

edited

Loading