Relax strict 1:1 verification in DatasetBatchManager.update_records (Support 1:N generation)

[JaoMarcos]

Priority Level

Medium (Nice to have)

Is your feature request related to a problem? Please describe.

I'm working on a plugin and ran into an issue with the strict verification in the DatasetBatchManager.update_records function. Currently, it enforces that the number of incoming records matches the current buffer size.

DataDesigner/packages/data-designer-engine/src/data_designer/engine/dataset_builders/utils/dataset_batch_manager.py

Line 194 in 184348a

def update_records(self, records: list[dict]) -> None:

The Use Case I need to support cases where a single input record produces multiple output records (1:N), essentially "exploding" the dataframe.

The main driver for this is cost and efficiency with LLMs. For complex prompts with large input contexts, if I need multiple variations (e.g., "Generate 5 variations of X"), it is significantly cheaper and faster to ask the model to generate all 5 in a single API call rather than making 5 separate calls with the same large input.

Generating them in a single pass also often improves quality/variance, as the model has "in-context" awareness of the other variations it is generating, preventing duplicates.

Describe the solution you'd like

add a flag (e.g., strict_mapping=False) to update_records?

Describe alternatives you've considered

No response

Additional context

No response

[andreatgretel]

hey @JaoMarcos, thanks for the detailed write-up and for building plugins on top of Data Designer!

this is definitely something we've been thinking about. 1:N generation (or "exploding" as you called it) is on our roadmap -
we're actually experimenting with it right now on a feature branch.
your use case around batching multiple variations in a single llm call makes a lot of sense for cost. for diversity though, we've found that using seeds (different input contexts per call) often gives better variance than generating multiple outputs in one pass - the model tends to stay in a similar "mode" within a single generation. something to consider depending on your use case.
we'll keep you posted as we make progress. in the meantime, if you want to try things out early, feel free to check out the andreatgretel/feat/allow-resize branch, as well as draft PR #286, and let us know how it goes!

p.s.: you might also be interested in the CustomColumn feature that we just merged, as it's similar to the plugin you're writing - the plugins remains more flexible, though!