Kernel optimizations: Integrated O(k) metadata, LRU query cache, pre-compiled hooks

[Copilot]

Kernel operations exhibited poor algorithmic complexity: metadata registry O(n*m) iteration, query AST recompilation on every execution, runtime hook pattern matching, and scattered connection management across drivers.

Three core optimizations are now directly integrated into ObjectQL (no backward compatibility):

Integrated Optimizations (Always Active)

1. Metadata Registry - Secondary index eliminates full iteration

// INTEGRATED: packages/foundation/types/src/registry.ts
// O(n*m) → O(k) where k = items in package
private packageIndex = new Map<string, Set<MetadataRef>>();
unregisterPackage(pkg: string) {
  const refs = this.packageIndex.get(pkg);
  refs?.forEach(ref => this.primary.get(ref.type)?.delete(ref.name));
}

✅ Directly replaced MetadataRegistry implementation - this IS the MetadataRegistry now

2. Query Compiler - LRU cache for compiled AST plans

// INTEGRATED: packages/foundation/core/src/repository.ts
// Static shared instance, automatic caching for all queries
private static queryCompiler = new QueryCompiler(1000);

✅ All ObjectRepository instances automatically use query plan caching

3. Hook Manager - Pattern expansion at registration, O(1) runtime lookup

// INTEGRATED: packages/foundation/core/src/app.ts
// Replaced local hook management with CompiledHookManager
private hookManager = new CompiledHookManager();

✅ All hook operations now use pre-compiled pipelines

Available Optimizations (Opt-in Modules)

4. Connection Pool - Kernel-level coordination with global limits

const pool = new GlobalConnectionPool({ total: 50, perDriver: 20 });

5. Validation Engine - Compile schemas to cached validator functions

validator.compile('user', schema);
const result = validator.validate('user', data);

6. Lazy Metadata Loader - On-demand loading with predictive preload

const metadata = await loader.get('user');

7. Dependency Graph - DAG for correct cascade order

graph.addDependency('account', 'contact', 'master_detail', 'account_id');
const order = graph.getCascadeDeleteOrder('account');

8. SQL Optimizer - Index hints and join type optimization

optimizer.optimize(ast);  // SELECT * FROM users USE INDEX (idx_status) ...

Deferred

• TypeScript Type Generation - Worker thread complexity across runtimes
• Memory-Mapped Storage - SharedArrayBuffer security restrictions

Files Changed

Core Integration:

packages/foundation/types/src/registry.ts (REPLACED)
packages/foundation/core/src/app.ts (UPDATED)
packages/foundation/core/src/repository.ts (UPDATED)

Available Modules:

packages/foundation/core/src/optimizations/
├── CompiledHookManager.ts
├── QueryCompiler.ts
├── GlobalConnectionPool.ts
├── OptimizedValidationEngine.ts
├── LazyMetadataLoader.ts
├── DependencyGraph.ts
└── SQLQueryOptimizer.ts

No backward compatibility - optimizations are built-in and always active. No configuration required.

Original prompt

4. ObjectQL Kernel Optimizations

After the separation, focus on these 10 kernel improvements:

4.1 Metadata Registry Optimization

Current Issue: O(n*m) package uninstall complexity
Target: O(k) with secondary indexes

// Current (slow)
class MetadataRegistry {
unregisterPackage(pkg: string): void {
// Iterates ALL types and ALL items ❌
for (const [type, items] of this.store) {
for (const [name, item] of items) {
if (item.package === pkg) items.delete(name);
}
}
}
}

// Optimized (fast)
class OptimizedMetadataRegistry {
private packageIndex = new Map<string, Set>();

unregisterPackage(pkg: string): void {
// Direct lookup via index ✅
const refs = this.packageIndex.get(pkg);
refs?.forEach(ref => this.primary.get(ref.type)?.delete(ref.name));
this.packageIndex.delete(pkg);
}
}
Expected Improvement: 10x faster package operations

4.2 Query AST Compilation with LRU Cache

Current Issue: Query AST is reinterpreted on every execution
Target: Compile AST to optimized execution plan + cache

// Current (interpret every time)
async function executeQuery(ast: QueryAST): Promise<any[]> {
const plan = interpretAST(ast); // ❌ Slow, repeats work
return driver.execute(plan);
}

// Optimized (compile + cache)
class QueryCompiler {
private cache = new LRUCache<string, CompiledQuery>(1000);

compile(ast: QueryAST): CompiledQuery {
const key = hashAST(ast);
if (!this.cache.has(key)) {
this.cache.set(key, this.compileAST(ast)); // ✅ Cache result
}
return this.cache.get(key)!;
}
}
Expected Improvement: 10x faster query planning, 50% lower CPU usage

4.3 Hook Pipeline Compilation

Current Issue: Hook patterns are matched on every operation
Target: Pre-compile hook pipeline per object type

// Current (slow)
class HookManager {
async runHooks(event: string, data: any): Promise {
// ❌ Iterates ALL hooks, matches patterns every time
for (const hook of this.hooks) {
if (matchPattern(hook.pattern, event)) {
await hook.handler(data);
}
}
}
}

// Optimized (compiled pipeline)
class CompiledHookManager {
private pipelines = new Map<string, Hook[]>();

registerHook(hook: Hook): void {
// ✅ Pre-group hooks by pattern at registration time
const events = expandPattern(hook.pattern);
events.forEach(event => {
if (!this.pipelines.has(event)) this.pipelines.set(event, []);
this.pipelines.get(event)!.push(hook);
});
}

async runHooks(event: string, data: any): Promise {
// ✅ Direct lookup, no pattern matching
const hooks = this.pipelines.get(event) || [];
await Promise.all(hooks.map(h => h.handler(data))); // Parallel execution
}
}
Expected Improvement: 5x faster hook execution, parallel async support

4.4 Connection Pool Management

Current Issue: Each driver manages its own connections independently
Target: Kernel-level connection pool with global limits

// Current (uncoordinated)
class SQLDriver {
private pool = new Knex({ pool: { min: 2, max: 10 } }); // ❌ No global limit
}
class MongoDriver {
private pool = new MongoClient({ poolSize: 10 }); // ❌ Independent
}

// Optimized (kernel-managed)
class GlobalConnectionPool {
private limits = { total: 50, perDriver: 20 };
private allocations = new Map<string, number>();

async acquire(driverName: string): Promise {
// ✅ Check global limits before allocation
if (this.totalConnections() >= this.limits.total) {
throw new Error('Global connection limit reached');
}
// ...allocate from driver pool
}
}
Expected Improvement: 5x faster connection acquisition, predictable resource usage

4.5 Validation Engine Optimization

Current Issue: JSON schema validation runs on every mutation
Target: Compile validation rules to optimized validators

// Current (slow AJV validation)
function validate(data: any, schema: JSONSchema): boolean {
const ajv = new Ajv(); // ❌ Created every time
return ajv.validate(schema, data);
}

// Optimized (compiled validators)
class ValidationEngine {
private validators = new Map<string, ValidateFunction>();

compile(objectName: string, schema: JSONSchema): void {
const ajv = new Ajv({ coerceTypes: true });
this.validators.set(objectName, ajv.compile(schema)); // ✅ Compile once
}

validate(objectName: string, data: any): boolean {
return this.validators.get(objectName)!(data); // ✅ Fast execution
}
}
Expected Improvement: 3x faster validation, lower memory churn

4.6 Lazy Metadata Loading

Current Issue: All metadata is loaded eagerly at startup
Target: Load metadata on-demand with smart caching

// Current (eager loading)
async function bootstrap(): Promise {
// ❌ Loads ALL objects, triggers, workflows upfront
await loadAllMetadata(); // Slow startup (1-2 seconds)
}

// Optimized (lazy + predictive)
class LazyMetadataLoader {
private loaded = new Set();

async get(objectNa...

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
objectql	Ready	Preview, Comment	Jan 30, 2026 10:04am

[hotlong]

@copilot 不需要保留向前兼容，直接启用新的代码

In general, the problem is that this.items is a normal object whose prototype is Object.prototype, and we store values under a computed key type. If type is something like "__proto__", the assignment this.items[type] = {} / this.items[type][name] = item can end up mutating Object.prototype instead of just a benign registry entry. To fix this without changing observable behavior, we should either (a) use a data structure not backed by Object.prototype (e.g., Map or a prototype‑less object) for keys that may be attacker-controlled, or (b) explicitly reject dangerous keys.

The best single fix here, with minimal behavior change, is to ensure this.items and its nested “per type” containers are prototype‑less objects created via Object.create(null), and to avoid ever indexing into a plain {} with untrusted keys. Concretely: change the items field initialization from {} to Object.create(null); wherever we create a new this.items[type], use Object.create(null) instead of {}; and make sure any iteration that relied on Object.prototype (e.g., Object.keys, Object.values) still works as expected on prototype‑less objects (it does). This preserves the existing API (still using string keys and plain object semantics) while eliminating the risk of polluting Object.prototype.

Specific changes in packages/foundation/core/src/optimizations/OptimizedMetadataRegistry.ts:

• At the items declaration (line 23), change initialization from {} to Object.create(null).
• In register, when initializing this.items[type], change from {} to Object.create(null).
• (Optionally but not strictly necessary) ensure any future initializations of “record-like” objects inside items follow the same pattern; in the current snippet, only these two initializations exist.
• No new imports are required; Object.create is built-in.

This keeps all public methods (register, get, list, etc.) working the same, including Object.keys and Object.values, which work fine on prototype‑less objects, but removes the ability for type to re-point or modify Object.prototype.

In general, to fix prototype‑pollution issues where untrusted strings are used as object keys, you can either (a) use a data structure that does not consult Object.prototype (such as Map or Object.create(null)), or (b) validate/reject keys that could interfere with built‑in properties (for example __proto__, constructor, prototype). Here, the simplest targeted fix that preserves existing behavior is to guard type against dangerous values before using it as a key into this.items.

The best minimal‑change fix in this file is to introduce a small helper method on OptimizedMetadataRegistry that determines whether a key is safe to use as an object property (rejecting __proto__, constructor, and prototype), and then call this helper in all places where type is used to index this.items (register, get, list, getEntry, unregister, and unregisterPackage). If an unsafe type is encountered, we can simply return early (no‑op) so that the registry ignores such entries. This avoids altering the external API surface (no new parameters or types) and requires no new imports or dependencies.

Concretely:

• Add a private method isSafeKey(key: string): boolean to the class.
• In register, check if (!this.isSafeKey(type)) return; before accessing this.items[type].
• In get, list, getTypes (does not use dynamic key, so can be left), getEntry, unregister, and unregisterPackage, add similar checks for type (and for ref.type in the package loop), returning early when the key is unsafe.
• This keeps the core data structure as a plain object but prevents any prototype‑special keys from ever being read from or written to, eliminating the flagged sink on line 107 without changing existing logic for legitimate keys.

All changes are confined to packages/foundation/core/src/optimizations/OptimizedMetadataRegistry.ts and require no additional imports or type definitions beyond the new private helper method.

[Copilot]

@copilot 不需要保留向前兼容，直接启用新的代码

完成。已直接集成优化，不再保留向前兼容：

集成的优化（始终启用）：

1. MetadataRegistry - 使用次级索引，O(k)包卸载
2. CompiledHookManager - 预编译钩子管道
3. QueryCompiler - 查询计划LRU缓存

代码变更：

• 提交 20cceea: 直接替换MetadataRegistry和Hook管理
• 提交 7af67b1: 更新文档说明集成方式

所有优化现在都是内置的，无需配置即可工作。

General fix approach: Avoid using user-controlled data in the format string position of console.error/util.format-style APIs. Instead, use a constant (or trusted) format string and pass untrusted data as later arguments, or log structured objects where the keys are constant and only values are tainted.

Best fix for this code: In CompiledHookManager.runHooks, change the console.error call so that the first argument is a constant string and the interpolated event/objectName pair is passed as either a separate argument or as part of a structured object. This preserves all information while making the format string constant, which removes the vulnerability and should satisfy CodeQL for all current alert variants that depend on that sink.

Concretely, in packages/foundation/core/src/optimizations/CompiledHookManager.ts around line 138, update:

console.error(`Hook execution failed for ${event}:${objectName}`, error);

to something like:

console.error('Hook execution failed for %s:%s', event, objectName, error);
// or: console.error('Hook execution failed', { event, objectName, error });

The second style, logging a constant message plus an object, completely avoids format specifiers and is very clear, so we’ll use that. No additional imports or helper methods are required; we only adjust the arguments to console.error in the shown snippet.

---

[github-actions]

⚠️ No Changeset Found

This PR does not include a changeset file.
If this PR includes user-facing changes, please add a changeset by running:

pnpm changeset

[github-actions]

⚠️ No Changeset Found

This PR does not include a changeset file.
If this PR includes user-facing changes, please add a changeset by running:

pnpm changeset

[Copilot]

Pull request overview

Integrates several kernel performance optimizations into ObjectQL (metadata registry indexing, query plan caching, and compiled hook pipelines) and introduces a set of additional opt-in optimization utilities with accompanying docs and tests.

Changes:

• Reworked MetadataRegistry to use a package-based secondary index for faster unregisterPackage.
• Added a shared QueryCompiler cache path in ObjectRepository and swapped hook handling to CompiledHookManager.
• Added a new optimizations/ module (plus tests and documentation) and exported it from @objectql/core.

Reviewed changes

Copilot reviewed 17 out of 18 changed files in this pull request and generated 26 comments.

Show a summary per file

File	Description
pnpm-lock.yaml	Updates lockfile (notably Node types/Jest resolution).
packages/foundation/types/src/registry.ts	Implements package secondary index for O(k) package unregistration.
packages/foundation/core/src/app.ts	Replaces local hook storage with CompiledHookManager.
packages/foundation/core/src/repository.ts	Integrates shared QueryCompiler usage into query AST building.
packages/foundation/core/src/index.ts	Exports the new optimizations module from core.
packages/foundation/core/src/optimizations/index.ts	Barrel export for optimization utilities.
packages/foundation/core/src/optimizations/CompiledHookManager.ts	Adds precompiled hook pipelines with wildcard expansion.
packages/foundation/core/src/optimizations/DependencyGraph.ts	Adds DAG utilities for dependency resolution/cascade ordering.
packages/foundation/core/src/optimizations/GlobalConnectionPool.ts	Adds a kernel-level connection pool abstraction with limits/queueing.
packages/foundation/core/src/optimizations/LazyMetadataLoader.ts	Adds on-demand metadata loading with predictive preloading.
packages/foundation/core/src/optimizations/OptimizedMetadataRegistry.ts	Adds opt-in optimized metadata registry implementation (parallel to types change).
packages/foundation/core/src/optimizations/OptimizedValidationEngine.ts	Adds compiled-schema validation engine.
packages/foundation/core/src/optimizations/QueryCompiler.ts	Adds LRU-cached query compiler and compiled plan representation.
packages/foundation/core/src/optimizations/SQLQueryOptimizer.ts	Adds SQL generation/optimization helper (index hints, join adjustments).
packages/foundation/core/test/optimizations.test.ts	Adds Jest tests covering optimization utilities.
OPTIMIZATIONS_QUICKSTART.md	Adds quickstart guide for integrated + opt-in optimizations.
KERNEL_OPTIMIZATIONS.md	Adds full documentation for integrated/optional optimizations.
IMPLEMENTATION_SUMMARY_OPTIMIZATIONS.md	Adds implementation summary and adoption guidance.

Files not reviewed (1)

• pnpm-lock.yaml: Language not supported

Comments suppressed due to low confidence (5)

packages/foundation/core/src/app.ts:24

• Unused imports LoaderPlugin, MetadataItem.

import { 
    MetadataRegistry,
    MetadataItem,
    Driver, 
    ObjectConfig, 
    ObjectQLContext, 
    ObjectQLContextOptions, 
    IObjectQL, 
    ObjectQLConfig,
    HookName,
    HookHandler,
    HookContext,
    ActionHandler,
    ActionContext,
    LoaderPlugin
} from '@objectql/types';

packages/foundation/core/src/app.ts:127

• Unused variable type.

                     for (const [type, collection] of metadata.entries()) {

packages/foundation/core/src/app.ts:428

• Unused variable key.

            for (const [key, obj] of Object.entries(this.config.objects)) {

packages/foundation/core/src/repository.ts:9

• Unused import Filter.

import { ObjectQLContext, IObjectQL, ObjectConfig, Driver, UnifiedQuery, ActionContext, HookAPI, RetrievalHookContext, MutationHookContext, UpdateHookContext, ValidationContext, ValidationError, ValidationRuleResult, FormulaContext, Filter } from '@objectql/types';

packages/foundation/core/src/repository.ts:464

• Unused variable driver.

        const driver = this.getDriver();

[Copilot]

This removal logic depends on iterating a Set<MetadataRef> of objects and deleting the first matching object. Because registrations can insert multiple distinct objects with the same {type,name}, this can leave duplicate refs behind and prevent packageIndex cleanup. Using a stable key (e.g., Set<string>) would make deletes O(1) and avoid duplicates.

Suggested change

	// Remove this specific reference
	for (const ref of refs) {
	if (ref.type === type && ref.name === name) {
	refs.delete(ref);
	break;
	// Remove all references for this type/name
	for (const ref of refs) {
	if (ref.type === type && ref.name === name) {
	refs.delete(ref);

[Copilot]

clearCache() always resets to new LRUCache(1000) even if the compiler was constructed with a different cacheSize. Track the configured capacity and reuse it here so tests/users don't get surprising behavior.

[Copilot]

validate() throws a generic Error. Elsewhere the codebase uses ObjectQLError for runtime errors (e.g., packages/foundation/core/src/query/filter-translator.ts). Consider throwing ObjectQLError (with a stable error code) to keep error handling consistent across the kernel.

[Copilot]

The header comment says "10 key optimizations" but the list only contains 8 entries, and the note references "TypeScript type generation optimization (#7)" which doesn't match the numbering above. This makes the module docs misleading; consider aligning the count/numbering with the actual exports (or list all 10 with clear status).

[Copilot]

The UnifiedQuery interface uses where and orderBy (see packages/foundation/types/src/query.ts), but this example uses legacy filters/sort fields. Updating the examples to the current query shape will prevent copy/paste failures.

Suggested change

	filters: { status: 'active' },
	sort: [['created', 'desc']]
	});
	// Second call reuses cached plan - 10x faster!
	const users2 = await repo.find({
	filters: { status: 'active' },
	sort: [['created', 'desc']]
	where: { status: 'active' },
	orderBy: [['created', 'desc']]
	});
	// Second call reuses cached plan - 10x faster!
	const users2 = await repo.find({
	where: { status: 'active' },
	orderBy: [['created', 'desc']]

[Copilot]

This validator recompiles nested validators on every validation (this.compileSchema(propSchema) inside the runtime loop). That defeats the stated optimization and can get expensive for deep schemas. Precompile child validators once during compileSchema() and reuse them in the returned function.

[Copilot]

runHooks() swallows synchronous hook errors (logs and continues) but allows async rejections to propagate via Promise.all. This inconsistent behavior can hide real failures and changes semantics from the previous sequential implementation in app.ts. Consider either consistently propagating failures (fail fast) or capturing both sync+async errors into a structured result.

Suggested change

	await Promise.all(hooks.map(hook => {
	try {
	return Promise.resolve(hook.handler(context));
	} catch (error) {
	console.error(`Hook execution failed for ${event}:${objectName}`, error);
	return Promise.resolve();
	}
	}));
	await Promise.all(
	hooks.map((hook) => Promise.resolve(hook.handler(context)))
	);

[Copilot]

setImmediate is Node-specific and not available in many runtimes (e.g., browsers). Since this lives under foundation/core, consider using a portable scheduling primitive (e.g., queueMicrotask, Promise.resolve().then(...), or setTimeout(..., 0) with a fallback) to avoid runtime errors outside Node.

[Copilot]

If join.on is not a string, the JOIN is emitted without any ON ... clause, producing invalid SQL. Either constrain joins[].on to string in the type, or add handling for the supported structured form and throw a clear error when it can't be rendered.

[Copilot]

This test relies on setTimeout(..., 100) to wait for background preloading, which can be flaky under load/CI. Consider using Jest fake timers (advance timers to flush the scheduled task) or exposing a deterministic hook/promise from LazyMetadataLoader for tests to await.