Skip to content

feat: add pg_tiktoken_c - pure C tiktoken for PostgreSQL, 1700x faster than pg_tiktoken #18

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
fasdwcx wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat: add pg_tiktoken_c - pure C tiktoken for PostgreSQL, 1700x faster than pg_tiktoken #18

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions data/pigsty.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ id,name,alias,category,url,license,tags,version,repo,lang,utility,lead,has_solib
1850,smlar,smlar,RAG,https://github.com/jirutka/smlar,PostgreSQL,{nil-lic},1.0,PIGSTY,C,f,t,t,t,f,,t,,"{17,16,15,14,13}",,1.0,PIGSTY,smlar_$v*,"{17,16,15,14,13}",,1.0,PIGSTY,postgresql-$v-smlar,,"{17,16,15,14,13}",,Effective similarity search,高效的相似度搜索函数,"fix math.abs, gist pointer problem"
1860,pg_summarize,pg_summarize,RAG,https://github.com/HexaCluster/pg_summarize,PostgreSQL,{pgrx},0.0.1,PIGSTY,Rust,f,t,t,t,f,f,f,,"{17,16,15,14,13}",,0.0.1,PIGSTY,pg_summarize_$v,"{17,16,15,14,13}",,0.0.1,PIGSTY,postgresql-$v-pg-summarize,,"{17,16,15,14,13}",,Text Summarization using LLMs. Built using pgrx,使用LLM对文本字段进行总结,pgrx=0.12.4
1870,pg_tiktoken,pg_tiktoken,RAG,https://github.com/kelvich/pg_tiktoken,Apache-2.0,{pgrx},0.0.1,PIGSTY,Rust,f,t,t,t,f,f,f,,"{17,16,15,14,13}",,0.0.1,PIGSTY,pg_tiktoken_$v,"{17,16,15,14,13}",,0.0.1,PIGSTY,postgresql-$v-pg-tiktoken,,"{17,16,15,14,13}",,tiktoken tokenizer for use with OpenAI models in postgres,在PostgreSQL中计算OpenAI使用的Token数,pgrx=0.12.6
1875,pg_tiktoken_c,pg_tiktoken_c,RAG,https://github.com/relytcloud/pg_tiktoken_c,Apache-2.0,,1.1,NONE,C,f,t,t,t,f,f,f,,"{17,16,15,14,13}",,,,,,,,,,,,tiktoken tokenizer for PostgreSQL in pure C, 1700x faster than pg_tiktoken (Rust/pgrx),纯C实现的PostgreSQL tiktoken分词器，比Rust版本快1700倍，支持RAG文档切分与Token计数,
1880,pg4ml,pg4ml,RAG,https://gitee.com/guotiecheng/plpgsql_pg4ml,AGPL-3.0,,2.0,PIGSTY,C,f,t,f,t,f,t,t,,"{17,16,15,14,13}","{plpgsql,tablefunc,cube,plpython3u}",2.0,PIGSTY,pg4ml_$v,"{17,16,15,14,13}",,2.0,PIGSTY,postgresql-$v-pg4ml,,"{17,16,15,14,13}",,Machine learning framework for PostgreSQL,PG4ML是一个机器学习框架,require python3
1890,pgml,pgml,RAG,https://github.com/postgresml/postgresml,MIT,{pgrx},2.10.0,PIGSTY,Rust,f,t,t,t,t,f,f,{pgml},"{17,16,15,14}",,2.10.0,PIGSTY,pgml_$v,"{17,16,15,14,13}",,2.10.0,PIGSTY,postgresql-$v-pgml,,"{17,16,15,14}",,Run AL/ML workloads with SQL interface,PostgresML：用SQL运行机器学习算法并训练模型,pgrx=0.12.9
2100,pg_search,pg_search,FTS,https://github.com/paradedb/paradedb/tree/dev/pg_search,AGPL-3.0,{pgrx},0.16.0,PIGSTY,Rust,f,t,t,t,f,t,f,{paradedb},"{17,16,15,14}",,0.15.18,PIGSTY,pg_search_$v,"{17,16,15,14}",,0.16.0,PIGSTY,postgresql-$v-pg-search,,"{17,16,15,14}",,Full text search for PostgreSQL using BM25,ParadeDB BM25算法全文检索插件，ES全文检索,pgrx=0.14.1 0.15.19+ has broken libicu on el systems
Expand Down