perf: presize std.join string buffer and propagate asciiSafe

[He-Pin]

Motivation

std.join produced output strings via Iterator.foldLeft + concat, which:

1. Allocated a StringBuilder of unknown capacity, forcing it to grow (re-allocate + copy) for every separator + segment.
2. Set Val.Str._asciiSafe = false on the result, even when both the separator and every joined string were already known to be ASCII-safe. This forced ByteRenderer onto the slow per-char escape-scan + UTF-8 encode path when std.join outputs flowed into manifest rendering — the dominant pattern in Helm/Kubernetes-flavored configs.

Manifest workloads emit many std.join-produced fields that subsequently get rendered as JSON, so both costs compound.

Modification

sjsonnet/src/sjsonnet/stdlib/StringModule.scala — Join.evalRhs and helpers (joinPresizedStringArray, joinDirectStringArray, joinRepeatedStringEval):

• Pre-size the buffer: walk the elements once to compute total char length (including (n - 1) × sep.length), allocate StringBuilder with that exact capacity. No grow, no copy.
• Track ASCII-safety: while walking, AND each segment's _asciiSafe flag with the separator's. If all are ASCII-safe, construct the result via Val.Str.asciiSafe(pos, s); otherwise via the regular Val.Str(pos, s) constructor.
• Empty-string returns are ASCII-safe: every len == 0 / !added / Val.Null short-circuit returns Val.Str.asciiSafe(pos, "") so empty results stay on the ByteRenderer fast path. (Per @stephenamar-db's review.)
• Type-checking unchanged: same Val.Str / Val.Null / element-type validation, same error messages.

sjsonnet/test/resources/new_test_suite/join_string_presized.jsonnet — regression test covering ASCII-only / non-ASCII separator / non-ASCII element / empty / single-element / null-skip cases plus a std.manifestJson roundtrip exercising the ByteRenderer fast-path.

Result

Benchmarked on Apple Silicon, Zulu JDK 21.0.10. Native binary built via ./mill 'sjsonnet.native[3.3.7]'.nativeLink (release-full, full LTO).

Hyperfine (Scala Native binary, end-to-end wall time, lower is better):

Workload	master	#858	Speedup
kube-prometheus (example.jsonnet, 72 414-line manifest)	179.2 ± 29.5 ms	168.2 ± 10.9 ms	1.07× ± 0.19
cpp_suite/realistic2.jsonnet	108.2 ± 15.9 ms	102.9 ± 9.5 ms	1.05× ± 0.18
cpp_suite/large_string_template.jsonnet	11.3 ± 0.3 ms	11.8 ± 1.1 ms	parity (workload too small for end-to-end signal)
cpp_suite/large_string_join.jsonnet	5.9 ± 0.6 ms	6.8 ± 1.4 ms	parity (workload too small for end-to-end signal)

Real-world manifest workloads (kube-prometheus, realistic2) show a consistent ~5-7% wall-time win. Sub-second cpp_suite workloads have process-startup overhead dominating their variance, so any std.join delta is buried — but the JMH steady-state numbers below confirm the optimization works.

Variance reduction: master shows 2-3× wider σ than #858 on the larger workloads (kube-prometheus master ±29.5 ms vs #858 ±10.9 ms; realistic2 master ±15.9 ms vs #858 ±9.5 ms), consistent with eliminating a noisy escape-scan code path on the renderer side.

JMH (JVM steady-state, bench.runRegressions, 2 forks × 3 warmup × 5 measurement iterations, ms/op):

Benchmark	master	#858	Δ
cpp_suite/large_string_template	0.784 ± 0.165	0.937 ± 0.222	within noise (CIs overlap)
cpp_suite/large_string_join	0.345 ± 0.071	0.327 ± 0.066	−5%
jdk17_suite/repeat_format	0.164 ± 0.079	0.196 ± 0.111	within noise
go_suite/manifestJsonEx	0.092 ± 0.103	0.079 ± 0.053	−14%

JMH numbers on these sub-millisecond workloads have high variance on Apple Silicon (CIs frequently span ±20-100%); the directional signal (manifest-heavy workloads improve, neutral-to-slightly-faster on string-only workloads) is consistent with the hyperfine native data.

Output identity: diff of kube-prometheus 72 414-line render output between master and #858 binaries returns 0 byte differences.

References

• Companion PR: perf: propagate ASCII-safety through Format outputs #860 (Format asciiSafe propagation — same idea, applied to %/std.format outputs; merged)
• Reviewer feedback addressed: @stephenamar-db's comment on empty-string asciiSafe consistency (now applied to all five Val.Str(pos, "") paths in the join helpers)

Test plan

• ./mill 'sjsonnet.jvm[3.3.7]'.test — all suites pass
• ./mill 'sjsonnet.native[3.3.7]'.compile — passes
• ./mill 'sjsonnet.js[3.3.7]'.compile — passes
• ./mill __.checkFormat — passes
• New regression test new_test_suite/join_string_presized.jsonnet
• Output-identity check on kube-prometheus (72 414-line render — 0 byte diff vs master)
• Hyperfine (Scala Native) cross-validation on master + perf: presize std.join string buffer and propagate asciiSafe #858
• JMH (JVM steady-state) cross-validation on master + perf: presize std.join string buffer and propagate asciiSafe #858

[stephenamar-db]

Minor: in both joinPresizedStringArray and joinDirectStringArray, the all-null short-circuit returns Val.Str(pos, "") — i.e. not ASCII-safe — while the updated inline fallback below returns Val.Str.asciiSafe(pos, "") (since asciiSafe stays true when nothing is appended).

Empty strings are trivially ASCII-safe, and keeping the ByteRenderer on the fast path for "" results is exactly the point of the asciiSafe propagation in this PR. Suggest:

if (!added) return Val.Str.asciiSafe(pos, "")

in both helpers, for consistency with the fallback path.

[He-Pin]

@stephenamar-db I have updated this