`split` config

[ShaMan123]

Description 📝

I think that the split functionality can be cleaned up as follows:

• add console or an equivalent interface to deps, alongside fs, path, making it optional
• adding a config object that has control over the hardcoded values used by the splitter (e.g. ELEMENT_TYPES), making fix(ifc-splitter): add IFCMECHANICALFASTENER to elements types #179 redundant

related #174

I am willing to PR

Validations ✅

• Read the docs.
• Check that there isn't already an issue that requests the same feature to avoid creating a duplicate.

[agviegas]

Hey @ShaMan123, both of these are good ideas. Yes please, a PR would be welcome.

Some thoughts on the shape:

For the logger injection, I'd make it accept any Partial<Console>-style subset rather than the full Console interface, so people can pass a minimal shim:

interface IfcSplitterLogger {
  log?: (...args: unknown[]) => void;
  time?: (label: string) => void;
  timeEnd?: (label: string) => void;
}

interface IfcSplitterDeps {
  fs: IfcSplitterFs;
  path: IfcSplitterPath;
  logger?: IfcSplitterLogger; // defaults to a no-op
}

Defaulting to no-op is nicer than defaulting to console because (a) server deployments often don't want the library yelling into stdout during routine work, and (b) users who want logs can pass console directly.

For the config object, I'd go with a single optional arg that shallow-merges onto the defaults, so users only override what they care about:

interface IfcSplitterConfig {
  elementTypes?: Set<string>;
  spatialTypes?: Set<string>;
  // Per-type arg index override for the list of related entities.
  // Default covers IFCRELAGGREGATES (5), IFCRELCONNECTSWITHREALIZINGELEMENTS (7),
  // IFCPRESENTATIONLAYERASSIGNMENT (2), all others (4).
  listIdxByType?: (type: string) => number | undefined;
}

Exposing listIdxByType as an override (not just elementTypes) would future-proof us against the same kind of bug as #174 for other non-standard IFC exports.

On #179 being redundant: agreed, adding IFCMECHANICALFASTENER as a hardcoded entry was a narrow band-aid. The proper fix is the config object, but I'd still leave the default list inclusive of common types (including IFCMECHANICALFASTENER) so most users don't need to configure anything.

Looking forward to the PR. Let me know if you want to sketch the API first before the full implementation.

[ShaMan123]

Excellent, this is what I had in mind.

[ShaMan123]

I want to broaden the discussion regarding IfcSplitterDeps before jumping back to implementation detail.
Why are fs and path abstracted to begin with? Is there a plan to somehow support splitting in the browser? If not why should it be abstracted?
There are other ways to abstract, e.g. creating a specialized class that holds all the API's needs, extending an event emitter (instead of using console API callbacks). That might be better in the long run if supporting non node.js environments is the goal.
What is the goal?
I think that since this is a game changer (at least for me) the API should be robust and simple.

[ShaMan123]

Another thought related to #164
I think it is crucial that the split function returns a Map/fragments 1:1 index: localId => split id.
Doing something trivial like using highlight/outliner with some custom code needs this.
I am using it as follows:

  splitToModelIdMap(
    originalModelId: string,
    ids: number[] | Set<number>
  ): OBC.ModelIdMap {
    const modelIndex = this.getIndex(originalModelId);
    if (!modelIndex) return {};
    const out: OBC.ModelIdMap = {};
    ids.forEach((id) => {
      const chunk = modelIndex.getChunk(id);
      if (chunk < 0) return;
      (out[resolveModel(modelId, chunk)] ||= new Set()).add(id);
    });
    return out;
  }

[ShaMan123]

Also I would like to discuss best practices of indexing when splitting files.
This will provide a robust solution to all needs.
Currently I am splitting a large IFC and iterating over it to collect information that I wish to index.
The package should expose methods that allow this.

[agviegas]

Three good threads here, taking each in order.

On fs / path abstraction. The intent is exactly what you suspected: keep the door open to non-Node backends. Splitting in the browser is plausible (web-ifc + OPFS, or in-memory for unit tests), and an opaque IfcSplitterDeps lets us swap storage without changing the splitter's logic. That said, your point about console being a poor fit for that pattern is fair. Two options:

1. Keep the deps shape additive: fs, path, plus a logger that defaults to no-op. Logger is for diagnostics; it doesn't pretend to be a Node runtime in disguise.
2. Lift logging out of the deps object entirely and onto an EventEmitter-style API on the splitter itself: splitter.onProgress.add(...), splitter.onItemProcessed.add(...). That would match how the rest of the engine surfaces lifecycle events (Hoverer, Highlighter, Views, etc.) and avoids the "is this a console or a logger" awkwardness.

I lean towards (2) because it's more idiomatic with the rest of the components codebase and lets consumers subscribe to specific events instead of capturing log strings. Happy with either; what's your preference?

On returning a localId → splitId map. Agreed, this is essential. Without it, every consumer is forced to recompute it from the splits, which defeats the speed advantage of splitting in the first place. I'd return it from split() as part of a structured result:

interface SplitResult {
  splits: Array<{ id: string; path: string }>;
  localIdToSplit: Map<number, string>;
  // (or expose it as an index in #164's terms; see below)
}

Even better, this maps cleanly onto the index primitive we're discussing in #164: the split operation can write a built-in index named "splits" into each split file (or into a small companion file) where the key is the localId and the value is the split id. Then consumers reach for it via model.getIndexEntry("splits", localId) and don't need to thread a Map through their app. Same data, surfaced through an API that already has lazy inverse lookups, so "give me all items in split X" becomes free.

On indexing while splitting. The split pass already iterates the entire model, so it's the right moment to populate indexes. I'd add a config hook so users can declare indexes to compute:

interface IfcSplitterConfig {
  // ...elementTypes, spatialTypes, listIdxByType from before
  indexes?: Array<{
    name: string;
    extract: (item: SplitItem) => { key: number; value: unknown } | null;
  }>;
}

The splitter handles the iteration and storage; the user supplies just the extraction logic. Built-in defaults could include the localId → splitId index above; user-declared indexes ride in on top. Output goes into the split files using the #164 schema, so reading is model.getIndexEntry(name, key) regardless of whether the index came from the splitter or was added later via the edit API.

This connects #180 and #164 cleanly: indexes are the primitive, splits are one of several producers that can populate them.

So three concrete decisions to make before the PR:

1. Logging via Partial<Console>-shaped logger or via EventEmitter-style events on the splitter? I'd take events.
2. Return value: structured SplitResult with maps inline, or write everything as named indexes consumed via getIndexEntry? I'd take the index path, since it composes with Support custom user-defined indexes for frag schema and querying #164.
3. Index hooks: declarative indexes config in IfcSplitterConfig, or a lower-level onItem callback? I'd take declarative.

What do you think?

[ShaMan123]

1. EventEmitter. I would abstract fs and path deps as well to be specialized interfaces as well. e.g. openFile, readLine, streamSomething etc. keeping in mind browser future proofing.
2. Indexes sound good to me because of the benefits however at that point in time the files are ifc format so where will the index be stored and how will the consumer embed it into a frag file? Is this process an overkill? A simple map might be best. Not sure.
3. I think both are valuable. Consider threadGroups ThreadGroups: Prioritize visual requests over data requests engine_components#706: I want to extract all PSets and assemblies to a separate split so that I can load it into a data threadGroup that won't slow down my visuals. onItem is befitting for this case. In general, I believe the consumer will want control over splitting once the feature is broadly adopted so a low level entry point should be available.

[ShaMan123]

Thought about (2) again. Considering splitToModelIdMap I would like fragments to manage this under the hood, abstracting it away from the consumer. This way I could use the existing API as is regardless of any splitting done, making the migration path seamless.
This means that internally the API should use the split index to reroute requests to the correct fragment model and surface back a flat response to the consumer.
This makes me lean towards returning an index, since the consumer will not need to manually access the splits, the API covers that.
split will become what it should be, a performance optimization, and a good one, since it won't require any logic/thought model changes.

Another thought related to #164 I think it is crucial that the split function returns a Map/fragments 1:1 index: localId => split id. Doing something trivial like using highlight/outliner with some custom code needs this. I am using it as follows:

  splitToModelIdMap(
    originalModelId: string,
    ids: number[] | Set<number>
  ): OBC.ModelIdMap {
    const modelIndex = this.getIndex(originalModelId);
    if (!modelIndex) return {};
    const out: OBC.ModelIdMap = {};
    ids.forEach((id) => {
      const chunk = modelIndex.getChunk(id);
      if (chunk < 0) return;
      (out[resolveModel(modelId, chunk)] ||= new Set()).add(id);
    });
    return out;
  }

[agviegas]

Three good directions. Aligning on each.

1. EventEmitter + specialized IO interfaces. Agreed on both. Replace the raw fs / path shape with a small, semantic IO surface so the splitter doesn't reach into Node primitives:

interface IfcSplitterIO {
  openRead(path: string): Promise<IfcReadStream>;   // sequential reads
  openWrite(path: string): Promise<IfcWriteStream>; // chunked writes
  exists?(path: string): Promise<boolean>;
  mkdir?(path: string): Promise<void>;
  join(...parts: string[]): string;
}

We ship a nodeIO() helper that fulfills it via node:fs/promises and node:path for the common case. A future opfsIO() or memoryIO() slots in without touching the splitter. Logging moves to events on the splitter (onProgress, onItemProcessed, etc.) so consumers subscribe explicitly and we don't have a "what does the logger know about side effects" question.

2. Indexes as the routing primitive, splits transparent. This is the cleanest framing and I'd take it. The mental model becomes:

• split() produces N split .frag files plus a small manifest file. The manifest carries the localId → splitId index using Support custom user-defined indexes for frag schema and querying #164's schema, plus paths for each split.
• The user calls fragments.loadSplit(manifestBytes, { ... }). Behind the scenes the manager loads each split, registers them with their assigned threadGroup if the manifest declares one per split, and stores the index.
• Public API stays the same. model.highlight([ids...]) looks up each id in the index, dispatches to the correct split's worker, joins the results.
• The consumer never threads splitId through their app.

Given the index has to be loaded into the runtime at startup anyway for routing, putting it in a manifest file (also a .frag, just one with no geometry, only the index) is a natural fit. Not overkill: this is the entire point of the abstraction. A loose Map<number, string> would only work if every consumer held it explicitly, which is exactly the friction we want to remove.

So split() returns:

interface SplitResult {
  manifest: { id: string; path: string };
  splits: Array<{ id: string; path: string; threadGroup?: string }>;
}

And the manifest's contents are (mostly) the indexes. splitToModelIdMap becomes an internal detail of fragments.loadSplit rather than user-facing code.

3. Both declarative and onItem. Yeah, you're right. Declarative covers the common "give me an index of names" case in five lines, and onItem is the escape hatch for "I want to extract psets to a separate split that goes on its own thread group". Two layers, each easy to understand:

interface IfcSplitterConfig {
  // ...existing element/spatial config
  indexes?: Array<{
    name: string;
    extract: (item: SplitItem) => { key: number; value: unknown } | null;
  }>;
  onItem?: (item: SplitItem) => void; // low-level escape hatch
  splits?: {
    [splitName: string]: {
      filter: (item: SplitItem) => boolean;
      threadGroup?: string; // routes loads into this group when fragments loads the split
    };
  };
}

That last splits block is the piece that ties #180 to #706: declare what goes into which split AND which thread group it lands on at load time. The data-only split for psets/assemblies you described becomes:

const result = await splitter.split(ifcBytes, {
  splits: {
    geometry: { filter: (i) => isGeometry(i) },
    data: { filter: (i) => isPsetOrAssembly(i), threadGroup: "data" },
  },
});

fragments.init(workerUrl, { threadGroups: { data: 1 } });
await fragments.loadSplit(result.manifest.path);

// regular API, transparently routed:
await model.highlight([id1, id2]);

Sound right? If so I'd go ahead and:

1. Land the index primitive in Support custom user-defined indexes for frag schema and querying #164.
2. Land threadGroups (already in main).
3. Land the splitter API on top, integrated with both.

We can do them in that order so each layer is testable independently and the splitter isn't blocked on schema work.

[ShaMan123]

I love this so much and am actually excited to see it land.
Naming of loadSplit is a bit confusing. But that is polish not for now.

Do you want me to join the effort somehow?
If so be explicit and detailed.

[ShaMan123]

I was thinking about IfcSplitterConfig#splits
filter is great but in order to limit the size of the split the consumer will need to store an array and count length and at some point increment a split counter.
I think we can offer a better solution.

  splits?: {
      splitBy: (item: SplitItem) => string | null; // threadGroup id, null for default
      maxSize?: number // defaulting to something
  };

Using splitBy will define the thread groups, maxSize will define the split max size. The abstracts splitting even further, coupling it with threading in terms of thought model.
It is also more efficient, since the splitter will cal splitBy once on each SplitItem and it is easier to use with more involved filtering/splitBy business logic on the consumer side (a single entry point allowing a switch case).

What do you think?

[agviegas]

Sorry for the lag, back on this.

On splitBy + maxSize. Sold. One callback per item beats N filters, and it falls out naturally as a switch on the consumer side. Updated config:

interface IfcSplitterConfig {
  // ...elementTypes, spatialTypes, listIdxByType, indexes, onItem
  splits?: {
    splitBy: (item: SplitItem) => string | null; // thread-group id, null = default
    maxSize?: number; // default: something sensible, e.g. 50_000 elements
  };
}

Two clarifications so we're aligned:

• splitBy returns a thread-group label, not a split id. Splitter opens as many physical splits per label as maxSize requires. So splitBy: () => "data" with 200k items and maxSize: 50_000 gives you 4 splits, all tagged data.
• null falls back to the default thread group (today's round-robin pool).
• maxSize is item count, not bytes. Easier to reason about, and matches how the manifest indexes will be sized.

Sound right?

On joining. Yes please. The most useful slice, and the one that unblocks everything else, is the IO + events refactor of the current splitter. Stable foundation first, new config and indexes on top.

As a single PR:

1. Replace IfcSplitterDeps.fs / IfcSplitterDeps.path with a semantic IfcSplitterIO:

   interface IfcSplitterIO {
     openRead(path: string): Promise<IfcReadStream>;
     openWrite(path: string): Promise<IfcWriteStream>;
     exists?(path: string): Promise<boolean>;
     mkdir?(path: string): Promise<void>;
     join(...parts: string[]): string;
     dirname(p: string): string;
     basename(p: string): string;
   }

   Stream shape is up to you, just keep it minimal. Async-first signatures even if the Node impl is sync under the hood, so OPFS slots in later without a rewrite.

2. Ship a nodeIO() helper from a separate entry point so the main bundle doesn't pull in node:fs. Update test.ts to import it from there.

3. Replace every console.* call with events on the splitter:

   • progress: { stage, elapsedMs }
   • groupResolved: { group, elementCount, totalIds }
   • groupWritten: { group, path, bytes }
   • warning: { message, context? }

   Current logs map onto these 1:1, no behavior lost. Anyone who wants stdout writes a one-liner subscriber.

4. Convert split and extract into a class (IfcSplitter) so the events have a host. Keep the free functions as thin wrappers this release, deprecate next.

5. process.exit(1) on missing input becomes a throw. We're a library, the host decides.

Out of scope here, follow-up PRs in order:

• New IfcSplitterConfig (elementTypes, spatialTypes, listIdxByType, indexes, onItem, splits.splitBy/maxSize)
• Manifest format and fragments.loadSplit(manifestPath)
• localId → splitId index, depends on Support custom user-defined indexes for frag schema and querying #164 landing first

I'll take #164 (the indexes primitive) in parallel so it's ready when you're ready to wire indexes and the manifest. No blocking either way: your PR is a refactor on the existing shape, mine stages behind a separate API.

If anything in 1 to 5 feels off, or you want the streams API pinned down further before you start, ping me here. Otherwise go for it, and thanks.

[ShaMan123]

Great!
I will get to it in the upcoming days and ping you if I need clarification.
It is a pleasure contributing in this manner.