Skip to content

feat: Allow variable fields on shadow blocks#9695

Draft
B-head wants to merge 8 commits intoRaspberryPiFoundation:mainfrom
B-head:test/shadow-variable-fields-spec
Draft

feat: Allow variable fields on shadow blocks#9695
B-head wants to merge 8 commits intoRaspberryPiFoundation:mainfrom
B-head:test/shadow-variable-fields-spec

Conversation

@B-head
Copy link
Copy Markdown

@B-head B-head commented Apr 9, 2026

Summary

Allows field_variable on shadow blocks. Specifically, this PR permits variable_get to be placed as a shadow block, and adjusts the connection / serialization / variable cascade paths so that variable fields on shadow blocks stay in sync with the workspace's variable map.

Motivation

In a personal project, I'm defining blocks that operate on objects and that perform higher-order list operations (map / filter / reduce style blocks). Having to specify a variable explicitly every time I use one of these blocks is cumbersome. I'd like to provide blocks that come with a default variable reference preset, so users can just drop them in. Today I work around this by inserting a non-shadow variable getter, but it would be more natural UX-wise if the preset were a shadow that the user could simply replace with another variable.

For context, my own use case involves combining this with @mit-app-inventor/blockly-block-lexical-variables. However, the changes in this PR work with stock Blockly's field_variable in the same way, so this is not a plugin-specific feature. That said, I have not yet verified whether this PR alone is sufficient to make the lexical-variables combination work end-to-end as intended; additional adjustments may be required.

Why a draft?

I judged that ad-hoc patching was not the right way to fix this. The code in this PR is also pure AI-generated code that I have not yet reviewed in detail myself.

The patches in this PR work around several existing issues in the codebase, which makes them fragile and likely to break under future changes. The right fix would be a more fundamental design change, but most of those changes would be breaking. Rather than push that decision unilaterally, I'd like to report the issues I uncovered while building this experimental implementation and ask for guidance on how to proceed.

For transparency: the issue analysis below was put together through a dialogue with Claude Code. I have verified the substance of the findings myself, but some of the wording is left as-is from the AI.

What this PR implements

Just allowing variable fields on shadow blocks isn't enough on its own. To make a variable field on a shadow block behave equivalently to a non-shadow variable_get, the following two consistency requirements had to be addressed:

  • Cascade on variable deletion: when a variable referenced by a variable field inside a shadow block is deleted via Workspace.deleteVariable, the shadow's reference must also be cleaned up so that the workspace stays consistent.
  • Undo/redo for changes inside shadow blocks: changes to a variable field on a shadow block must be properly undoable / redoable.

To make these work, the PR routes around the existing event system / serialization / connection paths in several places. The structural reasons why those workarounds were necessary are documented in the Known issues section below.

Known issues

This section is context for reviewers, not a list of fixes proposed by this PR. For each item, I'd like your judgement on whether it should be (a) folded into this PR's scope, (b) split out into a separate issue, or (c) marked WONTFIX.

1. Shadow lifecycle is not a first-class undo/redo operation

Symptom: Creating and destroying a shadow block does fire BlockCreate / BlockDelete events, but events_block_create.ts:48-50 and events_block_delete.ts:54-57 force recordUndo = false whenever isShadow() is true, so these events never end up on the undo stack. Field edits inside a shadow are similarly not recorded. In effect, the shadow lifecycle relies entirely on "side effects of the parent's events plus implicit respawning along the connect/disconnect path."

Impact on this PR: Because shadow blocks are now allowed to carry variable fields, there is no way to undo a variable-id change inside a shadow, and the cascade for variable deletion has no choice but to ride on the parent block's events.

Suggested upstream fix: Drop the forced recordUndo = false on shadow events and promote the shadow lifecycle to a first-class undoable operation. However, this is tightly coupled to the meaning of ConnectionState.shadow (see section 2 below), so it can't really be fixed in isolation. I think of it as an RFC-level item for a major version (v14).

2. ConnectionState.shadow conflates template and live snapshot

Symptom: serialization/blocks.ts:322 saveConnection calls getShadowState(true), which serializes the live shadow block's current values. On load, that serialized state is written back into shadowState_ via setShadowStateInternal in connection.ts, and then re-serialized again by serializeShadow at L622/L628/L632. After even a single save/load round trip, the original template is lost and the most recently saved live state becomes the new template. As a result, "the shadow template right after dropping from the flyout" and "the shadow template after a save/load cycle" no longer mean the same thing.

Impact on this PR: During a cascade, parent.shadowState may still be a template, or it may already be a live snapshot, depending on the calling context. This forced us to introduce a Case 1 (template != live) / Case 2 (template == live) split and an isSelfTemplate flag.

Suggested upstream fix: Split the ConnectionState schema into shadowTemplate and shadowOverride. This is a serialization-output change and therefore clearly breaking. It really should be solved together with section 1.

3. Connection.disconnectInternal implicitly respawns shadow blocks

Symptom: Around connection.ts:311, when a non-shadow child is disconnected, the parent silently runs respawnShadow_(). This is a side effect that's invisible to API consumers, and it triggers a chain like disconnect → respawn → getOrCreateVariablePackage → recreates a variable that was about to be deleted.

Workaround in this PR: Introduced a Workspace.suppressShadowRespawn flag and wrapped the entire cascade dispose loop with it to suppress respawning.

Suggested upstream fix: Add a {respawnShadow?: boolean} option to disconnect() so the side effect becomes opt-in at the call site. Changing the default would be feat!:; rewriting only internal callers while keeping the surface API compatible could potentially go in as a non-breaking feat:. If section 1 is being addressed, the cleanest path is to replace the implicit respawn with an event-driven mechanism instead.

4. getOrCreateVariablePackage's lookup-or-create semantics

Symptom: variables.ts:691-702 returns an existing variable or creates one if missing. This is convenient when loading variables, but in order-sensitive contexts like a cascade, the result depends on whether you call it before or after the deletion has happened. In this PR, we had to split the cascade into two passes to make this work.

Suggested upstream fix: Add an explicit "always create" API such as createVariableSameId(workspace, id, name, type). Since this leaves the existing getOrCreateVariablePackage untouched, it's an additive change and could land as a feat: in v13. The conditions are that it should still fire a VarCreate event like the existing createVariable, and that its behavior when a variable with the same id already exists should be explicitly defined (throwing seems safest).

5. getVariableUsesById returns the same block multiple times

Symptom: variables.ts:852-867 returns the same block multiple times if a single block contains multiple fields referencing the same variable. This PR works around it with a local Set<string> dedupe inside deleteVariable, but Blockly.Variables.deleteVariable at variables.ts:879 uses uses.length directly as the count shown in the DELETE_VARIABLE_CONFIRMATION dialog. That makes it a user-visible bug that double-counts the same block. The uses.length > 1 threshold check is also off, so there are cases where a confirmation dialog appears even though only a single block uses the variable.

Suggested upstream fix: Make getVariableUsesById itself dedupe by block id. The internal callers (VariableMap.deleteVariable at variable_map.ts:308 is already defended by an isDeadOrDying() guard; the deprecated wrappers just forward) are unaffected, and the dialog bug above gets fixed as a bonus. This should be doable as a fix: in v13. Since it's independent of this PR, it could also be split out as a separate PR.

B-head and others added 8 commits April 10, 2026 04:18
The generators and compile webdriver scripts built `file://` URLs by
concatenating `__dirname` directly. On Windows `__dirname` contains
backslashes, which yields a malformed URL like
`file://S:\source\blockly\.../tests/compile/index.html` that Chrome
refuses to load — making the `generators` and `advanced_compile_in_browser`
test steps fail.

Route both URLs through the existing `posixPath()` helper in
`scripts/helpers.js`, matching the pattern already used in
`tests/mocha/webdriver.js`. The helper is a no-op on POSIX so this
keeps existing Linux/macOS behaviour unchanged.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
`eslint-plugin-prettier` inherits Prettier's `endOfLine: 'lf'` default,
so any source file checked out with CRLF on a Windows machine that has
`core.autocrlf=true` fails lint with `Delete \`␍\`` errors. Setting
`* text=auto eol=lf` makes git always check out detected text files
with LF regardless of the user's local autocrlf setting, removing the
class of failures at its source.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The renamings test step ran the `.mjs` file directly via execSync,
relying on its `#!/usr/bin/env node` shebang. Windows `cmd.exe` does
not honor shebangs and `.mjs` has no default file association, so the
step failed with "is not recognized as an internal or external command".
Prefix the invocation with `node` so it works on every platform.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
`tests/node/node_modules/blockly-test` is a git symlink (mode 120000)
pointing at `../../../dist`. With `core.symlinks` disabled (the default
on Windows) git checks it out as a 13-byte text file containing the
link target literally, so `import 'blockly-test'` in
`tests/node/run_node_test.mjs` fails with `ERR_MODULE_NOT_FOUND` and
the `node` test step never runs.

Add an `ensureBlocklyTestLink` helper that runs immediately before the
node test step. It detects the broken state via `lstatSync`, replaces
the file with a junction (which works on Windows without admin rights
and is treated as a regular symlink on POSIX), then best-effort marks
the path `skip-worktree` so the resulting "deleted symlink" diff stays
out of `git status`. Unexpected failures of the skip-worktree step are
surfaced as a warning rather than silently swallowed.

The helper is idempotent and a no-op on Linux/macOS, where the git
checkout already produces a usable symlink.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Drop the throw in FieldVariable.setSourceBlock that prevented a
FieldVariable from attaching to a shadow block. This is the minimum
precondition for using variables_get as a shadow under any value
input; without it the field cannot even be constructed in a shadow
context.

This commit intentionally does NOT touch the variable deletion cascade,
respawn machinery, or any other related code path. Those concerns are
exercised by the accompanying test suite (tests/mocha/shadow_variable_fields_test.js)
and will be addressed by a follow-up redesign.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add "Shadow Variables" and "Shadow Variables (typed)" toolbox options to
tests/playground.html that mirror the block types defined inline in
tests/mocha/shadow_variable_fields_test.js (shadow_var_parent,
shadow_var_wrapper, shadow_var_two_field) plus a small statement-style
scaffold (shadow_var_host), so an operator can reach every scenario the
spec exercises from the playground UI:

  - shadow_var_parent + shadow variables_get
      basic / Case 1 / Case 2 / rename / delete cascade / save-load
  - shadow_var_parent + shadow shadow_var_wrapper + shadow variables_get
      nested shadow
  - shadow_var_parent + shadow shadow_var_two_field
      mixed Case 1 / Case 2
  - shadow_var_host + shadow_var_parent + shadow variables_get
      shadow_var_parent attached to a real statement parent

The typed variant routes the same shapes through variables_get_dynamic
and the dynamic Variables category, exercising the FieldVariable shadow
patch on the type-enforcing onchange path.

The block types are defined inline via Blockly.defineBlocksWithJsonArray
in the existing module script of playground.html; no new files are
introduced.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add tests/mocha/shadow_variable_fields_test.js (and register it in
tests/mocha/index.html) as an executable specification for the
"shadow variables_get" feature.

The suite encodes the quality requirements that were distilled from
the design discussion driving this work:

  RaspberryPiFoundation#1 Public API side-effect freedom: attaching a shadow variables_get
     does not fire VarCreate or change getAllVariables().
  RaspberryPiFoundation#2 Symmetric behavior: shadow and non-shadow variables_get on the
     same id share the same VariableModel; rename and code generation
     flow through the same path; serialization round-trips preserve
     the field id.
  RaspberryPiFoundation#3 Variable delete cascade processes shadow uses (no special
     skipping by isShadow()).
  RaspberryPiFoundation#4 Undo restores both the variable and the shadow display, with
     the shadow block id preserved across delete/undo cycles.
  RaspberryPiFoundation#5 Case 2 (parent template references the deleted variable, reached
     by save / load) does not crash, throw, or leave the variable map
     inconsistent.
  RaspberryPiFoundation#6 deleteVariable does not throw for inputs the patch enables.

Coverage areas (29 active tests + 4 acceptance criteria as test.skip):

  - basic shadow attachment and identity sharing with non-shadow
  - no spurious VarCreate / getAllVariables churn at attachment time
  - Case 1 cascade reset (live shadow value differs from template)
  - Case 2 cascade behavior (parent template references the deleted
    variable, reached via save / load round-trip)
  - symmetry with non-shadow variables_get for rename / cascade /
    code generation / save-load
  - rename propagation and undo
  - load auto-creates a missing variable (parity with non-shadow)
  - deletion of an unrelated variable leaves the shadow alone
  - chained deleteVariable calls without intermediate undo
  - BlockChange event recording for cascade resets
  - nested deleteVariable from a change listener
  - workspace.clear() with shadow variables_get blocks
  - non-default variable type round-trip
  - mixed Case 1 / Case 2 fields in one shadow
  - nested shadow blocks (shadow inside shadow inside parent)

The "acceptance criteria" suite collects four `test.skip` tests that
encode the contract the implementation must meet. They appear as
pending in mocha output and serve as the explicit acceptance gate
for the redesign work that will follow this spec branch.

Many of the active tests fail today (most fail with "shadow child
must be attached: expected null to exist", because the default
delete cascade disposes shadow uses without preserving them). The
failing tests are the red side of the spec; making them green is
the redesign session's charter.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
VariableMap.deleteVariable now keeps shadow variable_get blocks attached
at the same parent input by replaying each shadow's parent connection
shadowState through getOrCreateVariablePackage. The helper either
returns an existing live variable (template references a different,
still-live variable — Case 1) or re-creates a same-id same-name
variable (template references the variable being deleted — Case 2),
and setValue rebinds the field to the result. The two cases run on
opposite sides of the variable map removal so the resulting event group
unwinds cleanly:

  (a) Case 1 setValue replay (BlockChange events)
  (b) non-shadow uses dispose, with workspace.suppressShadowRespawn set
      so Connection.disconnectInternal does not auto-spawn a fresh
      shadow from a stale shadowState that still references the
      doomed variable
  (c) variable map removal + VarDelete fire
  (d) Case 2 setValue replay (re-creates the variable, fires VarCreate
      for the fresh VariableModel)

Undo unwinds in reverse:

  - Case 2 VarCreate is undone (the regenerated VariableModel goes
    away — the surviving field still holds a reference to it but the
    block id is preserved)
  - VarDelete is undone (the original variable comes back)
  - non-shadow dispose events undo (their BlockCreate events restore
    the disposed blocks)
  - Case 1 BlockChange events undo (the field validator now finds the
    restored variable on the workspace, so setValue accepts the old id)

The cascade also dedupes uses by block id (a single block with two
matching fields would otherwise appear twice in the raw uses list)
and falls back to the non-shadow dispose path for shadows that cannot
be classified (no reachable parent, no stored shadowState, no matching
field).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@github-actions github-actions bot added the PR: feature Adds a feature label Apr 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

PR: feature Adds a feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant