Skip to content

sparseembedding.md

Latest commit

 

History

History
689 lines (488 loc) · 26.8 KB

sparseembedding.md

File metadata and controls

689 lines (488 loc) · 26.8 KB

sparseembedding

import "github.com/Aleph-Alpha/std/v1/sparseembedding"

Package sparseembedding provides primitives to interact with the openapi HTTP API.

Code generated by github.com/oapi-codegen/oapi-codegen/v2 version v2.5.1 DO NOT EDIT.

Package sparseembedding provides a type-safe HTTP client for interacting with the BM25 sparse embedding API service.

Overview

The sparseembedding package offers a client generated from an OpenAPI specification for computing BM25 sparse embeddings. BM25 (Best Matching 25) is a ranking function used to estimate the relevance of documents to a given search query, producing sparse vector embeddings suitable for semantic search and information retrieval tasks.

Basic Usage

Create a new client:

client, err := sparseembedding.NewClient("https://api.example.com")
if err != nil {
	log.Fatal("Failed to create client", err, nil)
}

For easier response handling, use ClientWithResponses:

clientWithResponses, err := sparseembedding.NewClientWithResponses("https://api.example.com")
if err != nil {
	log.Fatal("Failed to create client", err, nil)
}

Generate BM25 embedding:

ctx := context.Background()
request := sparseembedding.BM25EmbedRequest{
	Text:     "This is a sample text for embedding",
	Language: sparseembedding.English, // Optional: omit for auto-detection
	AverageWordCount: func() *int {
		count := 256 // Optional: default is 256
		return &count
	}(),
}

response, err := clientWithResponses.EmbedBm25EmbedBm25PostWithResponse(ctx, request)
if err != nil {
	log.Error("Failed to generate embedding", err, nil)
	return
}

if response.StatusCode() == 200 && response.JSON200 != nil {
	embedding := response.JSON200
	// Use the sparse embedding
	// embedding.Indices contains the token indices
	// embedding.Values contains the corresponding BM25 scores
}

Client Options

The client supports various configuration options:

Custom HTTP client:

httpClient := &http.Client{
	Timeout: 30 * time.Second,
}
client, err := sparseembedding.NewClient(
	"https://api.example.com",
	sparseembedding.WithHTTPClient(httpClient),
)

Request editor for authentication:

client, err := sparseembedding.NewClient(
	"https://api.example.com",
	sparseembedding.WithRequestEditorFn(func(ctx context.Context, req *http.Request) error {
		req.Header.Set("Authorization", "Bearer your-token")
		return nil
	}),
)

Supported Languages

The BM25 embedding service supports the following languages:

  • Arabic, Basque, Catalan, Danish, Dutch, English, Finnish, French, German, Greek, Hungarian, Indonesian, Italian, Norwegian, Portuguese, Romanian, Russian, Spanish, Swedish, Turkish

If no language is specified, the service will automatically detect the language of the input text.

Sparse Embedding Format

The BM25 embedding is returned as a sparse vector with two arrays:

  • Indices: Array of integers representing token indices
  • Values: Array of float32 values representing BM25 scores for each token

This sparse representation is memory-efficient as it only stores non-zero values, making it suitable for large vocabulary spaces.

Average Word Count

The average_word_count parameter represents the average word count after stemming and removal of stopwords. Since there's no precise way to calculate this beforehand, a good estimate is approximately 60% of the original word count. If not specified, the default value is 256.

Error Handling

The client returns structured error responses:

  • 200 OK: Successful response with SparseEmbedding in JSON200
  • 422 Unprocessable Entity: Validation error with details in JSON422

Code Generation

The client code is automatically generated from the OpenAPI specification using oapi-codegen. To regenerate the client after updating api.yml:

cd v1/sparseembedding
go generate

This will regenerate client.gen.go based on the OpenAPI specification in api.yml.

Thread Safety

The Client struct is safe for concurrent use by multiple goroutines. Each HTTP request is independent and does not modify shared state.

Index

func NewEmbedBm25EmbedBm25PostRequest

func NewEmbedBm25EmbedBm25PostRequest(server string, body EmbedBm25EmbedBm25PostJSONRequestBody) (*http.Request, error)

NewEmbedBm25EmbedBm25PostRequest calls the generic EmbedBm25EmbedBm25Post builder with application/json body

func NewEmbedBm25EmbedBm25PostRequestWithBody

func NewEmbedBm25EmbedBm25PostRequestWithBody(server string, contentType string, body io.Reader) (*http.Request, error)

NewEmbedBm25EmbedBm25PostRequestWithBody generates requests for EmbedBm25EmbedBm25Post with any type of body

func NewRootGetRequest

func NewRootGetRequest(server string) (*http.Request, error)

NewRootGetRequest generates requests for RootGet

type BM25EmbedRequest

BM25EmbedRequest defines model for BM25EmbedRequest.

type BM25EmbedRequest struct {
    AverageWordCount *int `json:"average_word_count,omitempty"`

    // Language Supported languages.
    Language *Language `json:"language,omitempty"`
    Text     string    `json:"text"`
}

type Client

Client which conforms to the OpenAPI3 specification for this service.

type Client struct {
    // The endpoint of the server conforming to this interface, with scheme,
    // https://api.deepmap.com for example. This can contain a path relative
    // to the server, such as https://api.deepmap.com/dev-test, and all the
    // paths in the swagger spec will be appended to the server.
    Server string

    // Doer for performing requests, typically a *http.Client with any
    // customized settings, such as certificate chains.
    Client HttpRequestDoer

    // A list of callbacks for modifying requests which are generated before sending over
    // the network.
    RequestEditors []RequestEditorFn
}

func NewClient

func NewClient(server string, opts ...ClientOption) (*Client, error)

Creates a new Client, with reasonable defaults

func (*Client) EmbedBm25EmbedBm25Post

func (c *Client) EmbedBm25EmbedBm25Post(ctx context.Context, body EmbedBm25EmbedBm25PostJSONRequestBody, reqEditors ...RequestEditorFn) (*http.Response, error)

func (*Client) EmbedBm25EmbedBm25PostWithBody

func (c *Client) EmbedBm25EmbedBm25PostWithBody(ctx context.Context, contentType string, body io.Reader, reqEditors ...RequestEditorFn) (*http.Response, error)

func (*Client) RootGet

func (c *Client) RootGet(ctx context.Context, reqEditors ...RequestEditorFn) (*http.Response, error)

type ClientInterface

The interface specification for the client above.

type ClientInterface interface {
    // RootGet request
    RootGet(ctx context.Context, reqEditors ...RequestEditorFn) (*http.Response, error)

    // EmbedBm25EmbedBm25PostWithBody request with any body
    EmbedBm25EmbedBm25PostWithBody(ctx context.Context, contentType string, body io.Reader, reqEditors ...RequestEditorFn) (*http.Response, error)

    EmbedBm25EmbedBm25Post(ctx context.Context, body EmbedBm25EmbedBm25PostJSONRequestBody, reqEditors ...RequestEditorFn) (*http.Response, error)
}

type ClientOption

ClientOption allows setting custom parameters during construction

type ClientOption func(*Client) error

func WithBaseURL

func WithBaseURL(baseURL string) ClientOption

WithBaseURL overrides the baseURL.

func WithHTTPClient

func WithHTTPClient(doer HttpRequestDoer) ClientOption

WithHTTPClient allows overriding the default Doer, which is automatically created using http.Client. This is useful for tests.

func WithRequestEditorFn

func WithRequestEditorFn(fn RequestEditorFn) ClientOption

WithRequestEditorFn allows setting up a callback function, which will be called right before sending the request. This can be used to mutate the request.

type ClientWithResponses

ClientWithResponses builds on ClientInterface to offer response payloads

type ClientWithResponses struct {
    ClientInterface
}

func NewClientWithResponses

func NewClientWithResponses(server string, opts ...ClientOption) (*ClientWithResponses, error)

NewClientWithResponses creates a new ClientWithResponses, which wraps Client with return type handling

func (*ClientWithResponses) EmbedBm25EmbedBm25PostWithBodyWithResponse

func (c *ClientWithResponses) EmbedBm25EmbedBm25PostWithBodyWithResponse(ctx context.Context, contentType string, body io.Reader, reqEditors ...RequestEditorFn) (*EmbedBm25EmbedBm25PostResponse, error)

EmbedBm25EmbedBm25PostWithBodyWithResponse request with arbitrary body returning *EmbedBm25EmbedBm25PostResponse

func (*ClientWithResponses) EmbedBm25EmbedBm25PostWithResponse

func (c *ClientWithResponses) EmbedBm25EmbedBm25PostWithResponse(ctx context.Context, body EmbedBm25EmbedBm25PostJSONRequestBody, reqEditors ...RequestEditorFn) (*EmbedBm25EmbedBm25PostResponse, error)

func (*ClientWithResponses) RootGetWithResponse

func (c *ClientWithResponses) RootGetWithResponse(ctx context.Context, reqEditors ...RequestEditorFn) (*RootGetResponse, error)

RootGetWithResponse request returning *RootGetResponse

type ClientWithResponsesInterface

ClientWithResponsesInterface is the interface specification for the client with responses above.

type ClientWithResponsesInterface interface {
    // RootGetWithResponse request
    RootGetWithResponse(ctx context.Context, reqEditors ...RequestEditorFn) (*RootGetResponse, error)

    // EmbedBm25EmbedBm25PostWithBodyWithResponse request with any body
    EmbedBm25EmbedBm25PostWithBodyWithResponse(ctx context.Context, contentType string, body io.Reader, reqEditors ...RequestEditorFn) (*EmbedBm25EmbedBm25PostResponse, error)

    EmbedBm25EmbedBm25PostWithResponse(ctx context.Context, body EmbedBm25EmbedBm25PostJSONRequestBody, reqEditors ...RequestEditorFn) (*EmbedBm25EmbedBm25PostResponse, error)
}

type EmbedBm25EmbedBm25PostJSONRequestBody

EmbedBm25EmbedBm25PostJSONRequestBody defines body for EmbedBm25EmbedBm25Post for application/json ContentType.

type EmbedBm25EmbedBm25PostJSONRequestBody = BM25EmbedRequest

type EmbedBm25EmbedBm25PostResponse

type EmbedBm25EmbedBm25PostResponse struct {
    Body         []byte
    HTTPResponse *http.Response
    JSON200      *SparseEmbedding
    JSON422      *HTTPValidationError
}

func ParseEmbedBm25EmbedBm25PostResponse

func ParseEmbedBm25EmbedBm25PostResponse(rsp *http.Response) (*EmbedBm25EmbedBm25PostResponse, error)

ParseEmbedBm25EmbedBm25PostResponse parses an HTTP response from a EmbedBm25EmbedBm25PostWithResponse call

func (EmbedBm25EmbedBm25PostResponse) Status

func (r EmbedBm25EmbedBm25PostResponse) Status() string

Status returns HTTPResponse.Status

func (EmbedBm25EmbedBm25PostResponse) StatusCode

func (r EmbedBm25EmbedBm25PostResponse) StatusCode() int

StatusCode returns HTTPResponse.StatusCode

type HTTPValidationError

HTTPValidationError defines model for HTTPValidationError.

type HTTPValidationError struct {
    Detail *[]ValidationError `json:"detail,omitempty"`
}

type HttpRequestDoer

Doer performs HTTP requests.

The standard http.Client implements this interface.

type HttpRequestDoer interface {
    Do(req *http.Request) (*http.Response, error)
}

type Language

Language Supported languages.

type Language string

Defines values for Language.

const (
    Arabic     Language = "arabic"
    Basque     Language = "basque"
    Catalan    Language = "catalan"
    Danish     Language = "danish"
    Dutch      Language = "dutch"
    English    Language = "english"
    Finnish    Language = "finnish"
    French     Language = "french"
    German     Language = "german"
    Greek      Language = "greek"
    Hungarian  Language = "hungarian"
    Indonesian Language = "indonesian"
    Italian    Language = "italian"
    Norwegian  Language = "norwegian"
    Portuguese Language = "portuguese"
    Romanian   Language = "romanian"
    Russian    Language = "russian"
    Spanish    Language = "spanish"
    Swedish    Language = "swedish"
    Turkish    Language = "turkish"
)

type RequestEditorFn

RequestEditorFn is the function signature for the RequestEditor callback function

type RequestEditorFn func(ctx context.Context, req *http.Request) error

type RootGetResponse

type RootGetResponse struct {
    Body         []byte
    HTTPResponse *http.Response
}

func ParseRootGetResponse

func ParseRootGetResponse(rsp *http.Response) (*RootGetResponse, error)

ParseRootGetResponse parses an HTTP response from a RootGetWithResponse call

func (RootGetResponse) Status

func (r RootGetResponse) Status() string

Status returns HTTPResponse.Status

func (RootGetResponse) StatusCode

func (r RootGetResponse) StatusCode() int

StatusCode returns HTTPResponse.StatusCode

type SparseEmbedding

SparseEmbedding defines model for SparseEmbedding.

type SparseEmbedding struct {
    Indices []int     `json:"indices"`
    Values  []float32 `json:"values"`
}

type ValidationError

ValidationError defines model for ValidationError.

type ValidationError struct {
    Loc  []ValidationErrorLocInner `json:"loc"`
    Msg  string                    `json:"msg"`
    Type string                    `json:"type"`
}

type ValidationErrorLocInner

ValidationErrorLocInner defines model for ValidationError_loc_inner.

type ValidationErrorLocInner struct {
    // contains filtered or unexported fields
}

func (ValidationErrorLocInner) AsValidationErrorLocInner0

func (t ValidationErrorLocInner) AsValidationErrorLocInner0() (ValidationErrorLocInner0, error)

AsValidationErrorLocInner0 returns the union data inside the ValidationErrorLocInner as a ValidationErrorLocInner0

func (ValidationErrorLocInner) AsValidationErrorLocInner1

func (t ValidationErrorLocInner) AsValidationErrorLocInner1() (ValidationErrorLocInner1, error)

AsValidationErrorLocInner1 returns the union data inside the ValidationErrorLocInner as a ValidationErrorLocInner1

func (*ValidationErrorLocInner) FromValidationErrorLocInner0

func (t *ValidationErrorLocInner) FromValidationErrorLocInner0(v ValidationErrorLocInner0) error

FromValidationErrorLocInner0 overwrites any union data inside the ValidationErrorLocInner as the provided ValidationErrorLocInner0

func (*ValidationErrorLocInner) FromValidationErrorLocInner1

func (t *ValidationErrorLocInner) FromValidationErrorLocInner1(v ValidationErrorLocInner1) error

FromValidationErrorLocInner1 overwrites any union data inside the ValidationErrorLocInner as the provided ValidationErrorLocInner1

func (ValidationErrorLocInner) MarshalJSON

func (t ValidationErrorLocInner) MarshalJSON() ([]byte, error)

func (*ValidationErrorLocInner) MergeValidationErrorLocInner0

func (t *ValidationErrorLocInner) MergeValidationErrorLocInner0(v ValidationErrorLocInner0) error

MergeValidationErrorLocInner0 performs a merge with any union data inside the ValidationErrorLocInner, using the provided ValidationErrorLocInner0

func (*ValidationErrorLocInner) MergeValidationErrorLocInner1

func (t *ValidationErrorLocInner) MergeValidationErrorLocInner1(v ValidationErrorLocInner1) error

MergeValidationErrorLocInner1 performs a merge with any union data inside the ValidationErrorLocInner, using the provided ValidationErrorLocInner1

func (*ValidationErrorLocInner) UnmarshalJSON

func (t *ValidationErrorLocInner) UnmarshalJSON(b []byte) error

type ValidationErrorLocInner0

ValidationErrorLocInner0 defines model for .

type ValidationErrorLocInner0 = string

type ValidationErrorLocInner1

ValidationErrorLocInner1 defines model for .

type ValidationErrorLocInner1 = int

Generated by gomarkdoc