Test suite performance: per-suite breakdown and reordering opportunities

[jnasbyupgrade]

Per-suite timing (Linux container, PostgreSQL available)

Total: ~32s

Suite	Time	Tests	ms/test
sequential (00–04)	11.1s	62	179
doc (standard)	1.7s	9	189
make-test	1.9s	10	190
multi-extension	1.3s	14	93
pgtle-install	1.6s	7	229
gitattributes	1.2s	3	400
dist-clean	1.1s	9	122
make-results	0.9s	8	113
update-setup-files	0.7s	8	88
control-errors	0.9s	12	75

Foundation runs 3× (initial doc env, recursion check, main run) at ~0s/2.3s/3.6s — the second and third are the interesting ones; the third is building the real foundation, the second is a pre-check.

The sequential suite (11.1s)

Dominated by 04-pgtle.bats. Within that file the order is already logical, but three tests contain mandatory sleeps totalling 5 seconds:

Test	Sleep	Why
requires field becomes ARRAY when present	sleep 2; touch; sleep 1 = 3s	Ensure control file mtime > output file mtime on coarse-grained filesystems
control file change triggers rebuild	sleep 1 = 1s	Same reason
SQL file change triggers rebuild	sleep 1 = 1s	Same reason

The sleep 2 in the first test is the redundant one — reducing to sleep 1; touch pattern (like the other two tests) would save 2s. New timestamp infrastructure in the other branch will address this more elegantly.

Remaining time in the sequential suite (~6s) is genuine work: git operations, make dist, make pgtle invocations.

Independent suite overhead

Each independent suite pays ~0.5s for env creation (rsync from foundation). With 8 independent suites that's ~4s of overhead.

Reordering opportunities

Reviewed each slow suite for redundant expensive operations:

gitattributes (1.2s for 3 tests — worst ms/test): Env setup + two make dist calls dominate. Tests are inherently sequential (test 1 expects failure, test 2 commits .gitattributes and expects success, test 3 inspects the archive). No reordering wins. These tests could potentially be folded into 02-dist.bats to avoid the separate env cost (~0.5s), but that changes the independent/sequential architecture.

make-test (1.9s): Contains a skip_if_no_postgres path and a make test call. Already tight.

pgtle-install (1.6s): Two make pgtle + make run-pgtle pairs (tests 3 and 6) — the second is needed to test the empty-upgrade-file scenario. No reordering wins.

doc (1.7s): Test 6 (make docclean removes documentation) opens with run make html to ensure docs exist, but docs already exist from test 4 + test 5. The make html is a no-op when files are current (make checks timestamps), so savings are negligible.

multi-extension (1.3s): Only one make all call (test 9). Tests 1–8 and 10–14 are pure file/content checks. Already tight.

update-setup-files (0.7s): setup() does git reset --hard + git clean -fd before every test. Already efficient.

Summary

The biggest wins, in order:

1. 5s of sleeps in 04-pgtle — 3s reducible now (sleep 2 → sleep 1), remainder addressed by new timestamp infrastructure (other branch)
2. ~4s of independent suite env creation overhead — architectural cost, hard to eliminate without merging suites
3. gitattributes ms/test ratio — 3 tests that each pay for make dist; could be merged into 02-dist.bats to save ~0.5s env setup
4. Everything else is already reasonably tight

Notes on timing methodology

~/bin/timestamp was updated to show total_elapsed +delta: line, making suite boundaries trivial to read:

   9.393s +   0.098s: # Creating sequential environment...
  20.541s +   0.074s: # Creating control-errors environment...

Sequential suite elapsed = 20.541 - 9.393 = 11.1s

[jnasbyupgrade]

Laptop timing (macOS, anti-virus overhead)

Ran concurrent-make-test.bats with --timing on a MacBook:

	Time
Total wall clock	11.4s
Test 1: repos have different REGRESS_DBNAME	624ms
Test 2: single-extension --dbname count	263ms
Test 3: multi-extension --dbname count	259ms
Test 4: concurrent make test succeeds	1,202ms
Actual test time	~2.3s
setup_file overhead	~9s

The setup builds two environments sequentially, each doing git subtree add + sleep 1 + setup.sh. On this laptop that's ~4.5s per environment vs ~0.5s on the Linux container.