📖 Docs: English · Español · Português
A free and open-source implementation compatible with Advantage Database Server (ADS), discontinued by SAP.
The goal is to provide a drop-in replacement for the Advantage Client Engine (ace32.dll / ace64.dll / libace.so) so existing applications — particularly Harbour/Clipper apps using contrib/rddads — keep working without recompilation.
- Independent implementation. OpenADS is an independent open-source project. It is not affiliated with, sponsored by, or endorsed by SAP SE. "Advantage Database Server", "ADS", and any related marks, logos, and product names are the property of their respective owners and are referenced here solely to describe compatibility — their use does not imply any affiliation or endorsement.
- No SAP-owned binaries required. The OpenADS DLL is a
drop-in replacement; running an application against
ace32.dll/ace64.dll/libace.soproduced by this project does not require any DLL,.so, or other binary owned by SAP. The only runtime dependencies are the host operating system's standard libraries (e.g.KERNEL32.dlland the Microsoft Visual C++ / Universal CRT runtime on Windows;libc/libstdc++on Linux). - Provenance — clean-room. This codebase is written from
publicly observable behavior of the original Advantage Client
Engine and from the public Harbour
contrib/rddadssource (which is the call site OpenADS targets). It is not derived from leaked internal manuals or from disassembly / reverse engineering of SAP-owned binaries that would violate the Advantage SDK / ACE EULA. The implementation has been generated by an AI assistant (Anthropic Claude) under direct human supervision; every milestone is reviewed, tested, and committed by a human maintainer. - Purpose — non-commercial preservation. OpenADS is a
community-driven open-source project pursued without economic
benefit to its maintainers. Its only goal is to provide a
compatibility path for legacy applications affected by SAP's
discontinuation of Advantage Database Server, so the existing
Harbour / Clipper code base that depends on the ACE entry points
can keep running. The project does not sell, license, sublicense,
or otherwise monetise the software; redistribution is permitted
by the Apache License 2.0 (see
LICENSE) but the upstream maintainers receive no fee for the work. - No warranty, no support contract. OpenADS is provided AS IS, without warranty of any kind, express or implied (see the Apache License 2.0 §7 and §8). There is no service-level agreement, no commercial support channel, and no representation of fitness for any particular purpose.
- Downstream responsibility. Users who deploy OpenADS as a drop-in replacement for the Advantage Client Engine remain solely responsible for ensuring that their own use of related third-party tooling (Harbour, Clipper, original Advantage installations, application code, fixtures) complies with whatever licenses, EULAs, or service agreements apply to those components. The project ships no SAP-owned binary and asserts no permission on behalf of any third-party rightsholder.
Harbour rddtst.prg: 442 / 442 PASS (100%) (2026-05-08). The
canonical Harbour RDD compatibility harness (harbour/tests/rddtest/ rddtst.prg, the same suite that DBFCDX is benchmarked against)
runs end-to-end through contrib/rddads linked to OpenADS'
ace64.dll with zero failures on Windows MSVC64. Validated
fixture (tools/harbour_patch/adscl52_ads.prg) is regenerated by
rddmktst against the ADS RDD itself, so the inlined expected
values reflect ADS-engine behaviour rather than DBFCDX's. For
context: native DBFCDX scores 369 / 442 against its own DBFCDX-
generated baseline; OpenADS clears every line of the regenerated
ADS-flavoured baseline. The session that closed the last gap is
recorded across 28 incremental commits ending at 28be1be.
0.3.6 released (2026-05-06). Phase 2 TCP server is feature-
complete for v0.3.x — tools/serverd/openads_serverd exposes the
engine over an OpenADS-native wire protocol;
AdsConnect60("tcp://host:port/dir", ...) opens the same
ace64.dll against a remote server. The wire surface now covers
read (M12.4), write (M12.6), SQL exec (M12.7),
Reindex (M12.8), credential auth (M12.9), ACE error-code
propagation (M12.10), batched row fetch (M12.11), the
tls:// URI reservation (M12.12 stub), and a transport
abstraction layer (M12.13) that lets a future TLS transport plug
in without touching the server / client business logic. Real TLS
(vendored mbedtls / platform-native) is the v0.4.0 milestone.
docs/wire-protocol.md is the formal spec of every opcode +
payload format so non-C++ clients (Python, Go, Rust, Harbour AEP)
can talk to an OpenADS server without reading the C++ source.
Cross-platform CI is back to green on all three runners
(ubuntu-24.04 / ninja-clang, macos-14 / default,
windows-2022 / msvc-x64).
Release timeline:
| Tag | Date | Highlights |
|---|---|---|
| v1.0.0-rc15 | 2026-05-09 | Field classifier maps '+' (VFP autoincrement column) to DbfFieldType::Integer. Until this fix the dBASE Level 7 autoincrement type byte fell through to Unknown, so any DBF carrying a '+' column failed schema parsing even though the underlying storage is the 4-byte integer the engine already supports as DbfFieldType::Integer. One-line addition in src/drivers/dbf_common.cpp::classify_field. |
| v1.0.0-rc14 | 2026-05-08 | Run as a system service. openads_serverd.exe doubles as a Windows Service — --install-service [extra-flags…] writes a SERVICE_AUTO_START registration with the SCM (binPath bakes the trailing flags), sc start openads_serverd boots it, --uninstall-service drops it, the SCM dispatcher path drives the same g_running graceful-stop flag interactive Ctrl+C uses. Bundled tools/scripts/systemd/openads-serverd.service (User=openads, ProtectSystem=strict, NoNewPrivileges, RestrictAddressFamilies, Restart=on-failure) and tools/scripts/launchd/com.openads.serverd.plist (KeepAlive on Crashed; logs to /var/log/openads-serverd.{out,err}.log) cover Linux + macOS. New "Running as a system service" section in the README walks through all three flavours. main() refactored: argv parsing → parse_args(), bind-and-loop → run_server(args, console); SCM and interactive paths share one body. |
| v1.0.0-rc13 | 2026-05-08 | M-AOF.6 — production-CDX auto-open in AdsOpenTable. rddads / ADS convention: opening <base>.dbf auto-binds the sibling <base>.cdx so every tag in it is navigable on the Table without an explicit AdsOpenIndex60 call. Until rc13 OpenADS skipped this step, so Studio's per-request short-lived AbiSession re-opened the DBF on every /rows fetch but never picked up a CDX created in a prior session — AdsGetAOFOptLevel therefore reported NONE forever even after CREATE INDEX. Studio gains a one-click ▶ Demo button that walks the entire AOF / Rushmore story end-to-end (peek row 1 → install AOF → trigger suggested CREATE INDEX → re-apply → land at FULL), plus per-field "💡 Create index on <field>" hint chips when the cond doesn't reach FULL. openads_serverd --version is now driven from git describe --tags --always --dirty so the binary actually reports its real release tag (the previous hard-coded 1.0.0-rc1 had drifted twelve releases behind reality). Bench TAG values switched to AAAA/BBBB/…/ZZZZ so a user typing TAG='AAAA' in Studio matches what the column actually stores. |
| v1.0.0-rc12 | 2026-05-08 | M-AOF.5 — sparse-bitmap Skip / GoTop (O(M) walk through the visible set). The AOF bitmap installed by AdsSetAOF is now flattened into a sorted recno_sequence_ on the table; Skip / GoTop / GoBottom therefore O(1)-jump per step over only the matching records instead of decoding every recno asking the predicate. On the same 100k-row synthetic dataset, aof_between_idx_walk (3-tag visible set) drops from 321 ms (rc11, OptLevel=FULL but O(N) walk) to 24 ms — a 13.4× total speedup vs the original full-scan baseline; the eq-walk case (~3.8% selectivity) is now bounded only by the per-visible-record load_record_ decode cost (~80 µs × 3848 records = ~310 ms floor). AdsClearAOF drops both the predicate and the sequence in lockstep. Pinned by 367 / 367 tests, 43 424 / 43 424 assertions; nothing previously-tested broke. |
| v1.0.0-rc11 | 2026-05-08 | M-AOF.1..4 — first working slice of Advantage Optimized Filters (AOF) — Rushmore-style query optimisation. AdsSetAOF now actually parses + evaluates the cond, installs a per-record bitmap as a filter predicate that Skip / GoTop honour, and routes individual leaves through CDX / NTX range scans whenever an open index's key expression matches the leaf's field. AdsGetAOFOptLevel reports ADS_OPTIMIZED_FULL / PART / NONE based on per-leaf coverage. V1 grammar: <field> OP <literal> (= == != <> # < <= > >=), BETWEEN, IN, AND / OR / NOT, parens; both Clipper-style (.AND., .T.) and SQL-style keywords; index-accelerated leaves on character / memo fields with bare-field-name index expressions. tools/bench/openads_bench carries three AOF workloads — on a 100 000-row synthetic DBF, AdsSetAOF("TAG='AAAA'") is 1.83× faster with a TAG index installed (OptLevel=FULL) than without (OptLevel=NONE). Pinned by 367 / 367 tests, 43 424 / 43 424 assertions; nothing previously-tested broke. M-AOF.5 (sparse-bitmap Skip → O(M) walk through visible set) is the next step and is what unlocks the textbook 10-100× total speedup. |
| v1.0.0-rc10 | 2026-05-08 | studio.web.0.17 — Studio's SPA header now carries a deployment-mode badge: 🏠 LocalServer (green) when the console is hosted in-process by ace64.dll / ace32.dll (the rc9 in-DLL Studio), 🌐 Remote Server (blue) when hosted by openads_serverd. Hover surfaces the active data directory. Driven by a new mode field on /api/health ("localserver" when the HttpConsole was started without a backing wire-server pointer, "remote-server" otherwise) so two side-by-side Studio tabs are trivial to tell apart at a glance. Failure path is silent — the badge stays hidden if /api/health is unreachable, so reverse-proxy deployments that strip unknown fields keep working. |
| v1.0.0-rc9 | 2026-05-08 | Embedded Studio in ace64.dll / ace32.dll. Three new OpenADS-only entry points (AdsStudioStart(port, data_dir) / AdsStudioStop() / AdsStudioPort(*port)) plus an OPENADS_STUDIO_PORT=<N> (OPENADS_STUDIO_DATA, OPENADS_STUDIO_HOST) env-var auto-start hook driven from DllMain on Windows / a constructor attribute on POSIX. LocalServer apps (Harbour / X# / Clipper) loading the OpenADS DLL directly now get the same single-page Studio web console + REST surface in their own process — no openads_serverd.exe daemon required. The Studio HTTP target is built into the DLL only when -DOPENADS_WITH_HTTP=ON; without that flag the three entry points are still exported but return AE_FUNCTION_NOT_AVAILABLE so callers can detect the build flavour at runtime. Auto-start path is silent on bind failure so a host loading the DLL on a port-busy box doesn't crash; explicit AdsStudioStart() returns AE_INTERNAL_ERROR instead. Default bind host is 127.0.0.1 to keep the console off the LAN unless explicitly opted in. |
| v1.0.0-rc8 | 2026-05-08 | XSharp-feedback round (Robert van der Hulst): release ZIP now bundles both ace64.dll (x64) and ace32.dll (x86) via the new tools/scripts/build_release_windows.bat two-arch driver; vendored mbedtls 3.6.2 is forced statically linked (USE_STATIC_MBEDTLS_LIBRARY=ON, BUILD_SHARED_LIBS=OFF) so the shipped DLL / openads_serverd.exe has zero runtime libssl/libcrypto dependency; openads_serverd --port <N> already accepted, the bind-failure path now prints an explicit "port 6262 is the SAP Advantage Database Server default" hint when the operator hits a clash with a running ADS service; tools/bench/README.md documents that openads_bench synthesises its three-column DBF (ID N(8,0), TAG C(4), AMT N(8,2)) on every run from a fixed seed — no shipped fixture, fully reproducible. |
| v1.0.0-rc7 | 2026-05-08 | Harbour rddtst.prg: 442 / 442 PASS via contrib/rddads against OpenADS ace64.dll (vs 369 / 442 for native DBFCDX). 28 commits in the session: rddads default-connection fallback, ORDSETFOCUS by CDX-insertion order, CDX FOR-clause filter + sibling-tag resync on CREATE INDEX, SET DELETED skip-deleted in goto/skip walks, AfterEndKey / Between cursor states, empty-key DBSEEK quirk, DOUBLEKEY field-width formatting, soft-seek-key-below-first → EoF, DESCEND walk-to-last-ASC, INDEX inserts deleted rows, key_size sub-tag-header rewrite on re-CREATE, plus the patched tools/harbour_patch/rddads-compat.patch (rddads-side fBof Limbo carve-out, bFindLast retry guarded by u16Found, ASC-only Brian-Hays trick) and the regenerated tools/harbour_patch/adscl52_ads.prg ADS-baseline fixture. |
| v1.0.0-rc6 | 2026-05-07 | studio.web.0.16: schema view now renders single-letter dBASE field-type labels (C / N / D / L / M / B / P / I / T / @ / +) instead of the raw ADS_* numeric code; /api/tables/<t>/schema adds a type_name field alongside the existing type integer. |
| v1.0.0-rc5 | 2026-05-07 | studio.web.0.15: Studio's /api/tables/<t>/rows endpoint no longer 500s on binary memo content — non-UTF-8 cells now return a {"_b64": "<base64 preview>", "_size": N, "_truncated": bool} object (1 KB cap) so multi-MB ZIP / image / encrypted blobs render cleanly; memo cells also stop being silently truncated at the previous 4 KB read buffer. Validated against the same 1.0 GB FPT used by openads_memo_stress. Includes Windows x86 (32-bit) -Werror fix in tx_log_test.cpp (uintmax_t → size_t cast). |
| v1.0.0-rc4 | 2026-05-07 | M(cdx-split) full multi-level B+tree CDX leaf+branch split (closes the M3.5 single-leaf limitation); compact-leaf encoder gains entry/suffix overlap check; structure-tag leaf widens its recno field to 32 bits so multi-tag bags >64 KB no longer truncate sub-tag header offsets. AdsGetField memo-read truncation fix — read path no longer caps at 65 534 bytes, so memos and binary blobs (e.g. ZIP archives stored in M fields) round-trip bit-perfect at any size; pinned by tests/unit/abi_memo_large_test.cpp (200 KB) and exercised by openads_memo_stress (1.0 GB FPT, 12 000 rows × 2 fields × cycling lengths up to 256 KB, 0 mismatches). Cross-platform -Werror cleanup: builds clean on Windows MSVC, macOS AppleClang, and Ubuntu 24.04 GCC 13 simultaneously (346 / 346 unit tests pass on all three). New stress / verification tools: openads_memo_stress, openads_memo_zip (ZIP-in-memo round-trip with python -m zipfile -t validation), 200 K-row CDX multi-bag scenarios via openads_stress. SQL DDL CREATE INDEX defaults back to .cdx. |
| v1.0.0-rc3 | 2026-05-06 | CI build timeout 45 → 60 minutes (mbedtls FetchContent on macOS). |
| v0.3.6 | 2026-05-06 | M12.12 tls:// URI reservation. |
| v0.3.5 | 2026-05-06 | M12.10 ACE error-code propagation; M12.11 batched fetch. |
| v0.3.4 | 2026-05-06 | M12.6 remote write; M12.7 remote SQL; M12.8 remote Reindex; M12.9 server auth; RemoteConnection dtor fix. |
| v0.3.3 | 2026-05-06 | M12.3 server skeleton; M12.4 remote read; M12.5 dual-mode DLL; openads_bench; cross-platform clang -Werror clean-up. |
| v0.3.0 | 2026-05-05 | 42 M10.x SQL milestones + 3 M11.x non-SQL (V/Q types, AEP host, OpenADS-encrypted DBF). Full Harbour-reachable Ads* surface MISS-free. |
| v0.2.0 | 2026-05-04 | 27 M9.x milestones on top of 0.1.0; full Ads* ABI unlocked. |
| v0.1.0 | earlier | M0–M8.11 — local DBF / CDX / NTX / FPT / DBT engine. |
A real Harbour application, compiled against the standard
contrib/rddads static library, opens a DBF, walks records, runs
dbSeek, swaps focus across multiple CDX tags, runs ARIES
transactions with rollback semantics, reads and writes memo
fields, creates tables / indexes dynamically, packs and
zaps tables, reindexes from a compound key expression, and
reopens to verify durability — every call lands on OpenADS'
ace64.dll with no Harbour rebuild. See
tests/harbour_smoke/README.md for the captured output of every
M8.x / M9.x milestone.
$ smoke.exe
OpenADS smoke test
ACE DLL reports: 0.0a
Schema:
1 NAME C len=10 dec=0
2 AGE N len= 3 dec=0
3 ACTIVE L len= 1 dec=0
4 BORN D len= 8 dec=0
5 NOTES M len=10 dec=0
Walk via UPPER_NAME (compound key)
Append "delta" → recno 4
Reopen + Seek 'DELTA' (upper) : Found=T RecNo=4 NAME=[delta]
Done.
Building OpenADS produces ace64.dll (or ace32.dll on x86; on
POSIX, libace.so / libace.dylib):
git clone https://github.com/FiveTechSoft/OpenADS
cd OpenADS
cmake --preset default
cmake --build build/default --config Release
ctest --test-dir build/default --output-on-failure -C ReleaseDrop the resulting ace64.dll (under
build/default/src/Release/) onto a Harbour app's PATH ahead of
any SAP-shipped copy and the standard contrib/rddads calls land
on OpenADS without recompiling Harbour.
OpenADS ships a single-page web admin console (Studio) that lists the connection's tables, shows their schema, runs ad-hoc SQL, and inspects records — including memo / binary fields. Studio comes in two flavours:
- Remote-server mode:
openads_serverd.exe --http-port 8080 --data c:\app\dataexposes Studio alongside the TCP wire server. - LocalServer mode: the DLL itself spins up Studio in the
caller's process, so a Harbour / X# / Clipper app loading
ace64.dll(orace32.dll) directly gets the same UI without launching any extra daemon.
Two ways to enable LocalServer Studio:
-
Programmatic — call the OpenADS-only extension entry points:
AdsStudioStart(8080, (UNSIGNED8*)"c:\\app\\data"); // bind 127.0.0.1:8080 /* ... ShellExecute("http://localhost:8080") ... */ AdsStudioStop();
AdsStudioPort(&port)reports the bound port.AdsStudioStartreturnsAE_SUCCESSon success,AE_INTERNAL_ERRORif the bind / listen fails (e.g. another process already holds the port), orAE_FUNCTION_NOT_AVAILABLEif OpenADS was built without-DOPENADS_WITH_HTTP=ON. -
Environment-driven auto-start — set
OPENADS_STUDIO_PORT=<port>before launching the host app and the DLL boots Studio automatically when it loads:set OPENADS_STUDIO_PORT=8080 set OPENADS_STUDIO_DATA=C:\app\data :: default = "." set OPENADS_STUDIO_HOST=127.0.0.1 :: default = 127.0.0.1 start MyApp.exe
The auto-start hook runs from
DllMain DLL_PROCESS_ATTACHon Windows and a constructor attribute on POSIX. WithoutOPENADS_STUDIO_PORTthe hook is a no-op — the DLL will not bind any port unless the host explicitly asks for it.
Locking. Studio opens tables read-only via short-lived ABI connections, so tables your app holds in EXCLUSIVE mode show up as "table busy" in the browser until the app releases the lock; shared opens coexist fine.
Default bind host is 127.0.0.1, not 0.0.0.0 — the console is
local-only out of the box. Set OPENADS_STUDIO_HOST=0.0.0.0 (or
pass an explicit host through a wrapper) when the deployment
genuinely needs LAN visibility, and pair it with
openads_serverd --http-user user:pass for HTTP Basic auth.
openads_serverd knows how to register itself as a Windows
Service and ships ready-to-drop unit files for the systemd /
launchd equivalents on Linux / macOS, so a typical
deployment never has to keep a console window open.
Windows. From an elevated cmd / PowerShell:
:: register the service (auto-start at boot). The flags after
:: --install-service are baked into the registered binPath:
openads_serverd.exe --install-service --port 6262 --http-port 6263 --data C:\app\data
:: start it via SCM
sc start openads_serverd
:: tail logs (the SCM redirects stdout to the Application event log)
:: or check the running state:
sc query openads_serverd
:: drop the registration when you no longer need it:
openads_serverd.exe --uninstall-serviceThe --service flag exists for the SCM dispatcher only; running
openads_serverd.exe --service interactively now prints an
explicit explanation rather than hanging.
Linux (systemd). Drop the bundled unit and enable:
sudo install -m 644 tools/scripts/systemd/openads-serverd.service \
/etc/systemd/system/openads-serverd.service
sudo systemctl daemon-reload
sudo systemctl enable --now openads-serverd
sudo journalctl -u openads-serverd -fThe bundled unit drops privileges (User=openads,
ProtectSystem=strict, NoNewPrivileges, restricted address
families) and Restart=on-failure. Tweak the --data / --port
flags inside the unit to match your deployment.
macOS (launchd). Drop the bundled plist into
/Library/LaunchDaemons/ for a system-wide service or
~/Library/LaunchAgents/ for a per-user one:
sudo install -m 644 tools/scripts/launchd/com.openads.serverd.plist \
/Library/LaunchDaemons/com.openads.serverd.plist
sudo launchctl bootstrap system \
/Library/LaunchDaemons/com.openads.serverd.plist
sudo launchctl kickstart -kp system/com.openads.serverd
tail -f /var/log/openads-serverd.{out,err}.logStop / unload via sudo launchctl kill SIGTERM system/com.openads.serverd and sudo launchctl bootout system /Library/LaunchDaemons/com.openads.serverd.plist.
OpenADS ships a working AOF (Advantage Optimized Filters) layer
modelled on the Rushmore query-optimisation technique used by
FoxPro and the original SAP Advantage Database Server. The idea
is simple: instead of walking every record evaluating the filter
expression against decoded field values, OpenADS builds a
per-record bitmap once, with each <field> OP <literal> leaf
served by an existing CDX / NTX index range scan when one is
available, and Skip / GoTop only visit records whose bit is set.
Minimal Harbour-flavoured C example:
ADSHANDLE hT, hIdx;
UNSIGNED8 dbf[] = "people.dbf";
UNSIGNED8 cdx[] = "people.cdx";
UNSIGNED8 nm[] = "LASTNAME_IDX";
UNSIGNED8 expr[] = "LASTNAME";
UNSIGNED8 cond[] = "LASTNAME = 'SMITH'";
AdsOpenTable(hConn, dbf, NULL, ADS_CDX, ADS_ANSI, 0, 0, 0, &hT);
/* Build an index on the field used by the AOF leaf. Without it, */
/* AdsSetAOF still works correctly (full-scan bitmap path) but no */
/* range-scan speedup; AdsGetAOFOptLevel will report NONE. */
AdsCreateIndex61(hT, cdx, nm, expr, NULL, NULL, 0, 0, &hIdx);
/* Install the AOF. Skip / GoTop now walk only matching records. */
AdsSetAOF(hT, cond, 0);
UNSIGNED16 lvl = 0;
AdsGetAOFOptLevel(hT, &lvl, NULL, NULL);
/* lvl == ADS_OPTIMIZED_FULL when every leaf served by an index, */
/* PART when some leaves served + others scanned, NONE otherwise. */
AdsGotoTop(hT);
UNSIGNED16 eof = 0;
while (AdsAtEOF(hT, &eof) == 0 && eof == 0) {
/* ... process visible record ... */
AdsSkip(hT, 1);
}
AdsClearAOF(hT);
AdsCloseTable(hT);V1 grammar (the AOF cond passed to AdsSetAOF):
<field> OP <literal> OP in { = == != <> # < <= > >= }
<field> BETWEEN a AND b
<field> IN ( v1, v2, ... )
expr AND expr also `.AND.` (Clipper)
expr OR expr also `.OR.`
NOT expr also `.NOT.` and `!`
( expr )
Identifiers and keywords are case-insensitive; both Clipper-style
(.T., .AND.) and SQL-style keywords are accepted because
rddads forwards either depending on the call site. Anything
outside the documented grammar (function calls, arithmetic, LIKE)
returns AE_PARSE_ERROR so the caller can fall back to a manual
walk; OpenADS does not silently swallow an unsupported expression.
Index-accelerated leaves in V1 cover character / memo fields with
a bare-field-name index expression. Numeric / date / logical
fields, and indexes whose expression is UPPER(field) /
DTOC(field) / a compound expression, still produce a correct
bitmap via the per-record fallback — they just don't count as
"served by index" in the OptLevel report.
Measured speedup (tools/bench/openads_bench, 100 000 synthetic
rows, Windows MSVC x64 Release, M-AOF.5):
| AOF workload | med (ms) | OptLevel | Speedup |
|---|---|---|---|
AdsSetAOF("TAG='AAAA'"), no TAG index |
586 | NONE | 1.0× |
AdsSetAOF("TAG='AAAA'"), TAG indexed |
323 | FULL | 1.83× |
AdsSetAOF("TAG BETWEEN 'AAAA' AND 'CCCC'"), TAG indexed |
24 | FULL | 24.4× |
Two things drive the speedup:
AdsSetAOFitself becomes a range scan over the matching index instead of a full record decode + AST eval per row. That alone is the rc11 (M-AOF.4) gain.Skip/GoTopwalk only the matching records (sparse-bitmap navigation, M-AOF.5) instead of iterating every recno asking the predicate. That's the rc12 gain that flips the textbook Rushmore window on for selective filters.
The eq-walk case at 1.83× is now bounded by the per-visible-record
load_record_ decode cost (~80 µs × 3848 matching records ≈ 310
ms floor); applications that don't touch the matched data — for
example COUNT(*) over the AOF set, or dbSeek-style point
lookups — hit the full Rushmore speedup window on top of that.
tools/bench/openads_bench generates a synthetic 100 000-row DBF
(ID N(8,0), TAG C(4), AMT N(8,2)) and times a fixed set of
SQL workloads through the public ABI (AdsExecuteSQLDirect).
Median of 5 repeats per workload, all-Release builds:
| Workload (median ms) | Windows MSVC | Linux clang -O3 | macOS AppleClang |
|---|---|---|---|
| stage 100 k-row DBF | 63.5 | 57.9 | 34.0 |
SELECT COUNT(*) |
297.7 | 42.0 | 103.9 |
WHERE TAG = 'AAAA' |
303.7 | 48.3 | 108.4 |
SUM/AVG/MIN/MAX(AMT) |
374.3 | 120.5 | 136.1 |
GROUP BY TAG |
321.9 | 58.6 | 120.9 |
ORDER BY AMT LIMIT 10 |
668.0 | 165.4 | 260.5 |
DISTINCT TAG |
598.4 | 95.2 | 213.4 |
BETWEEN 100 AND 500 |
314.1 | 63.7 | 114.4 |
Same source tree, same workload, three OS / toolchain combinations.
Linux clang -O3 wins every SQL workload — roughly 7× faster than
MSVC Release on the full-table count, 4× on the heavier ORDER BY.
macOS Intel sits between the two. Run it on your own hardware:
cmake --build build/default --target openads_bench
./build/default/tools/bench/openads_bench --rows 100000 --repeats 5 --csvA working TCP server (tools/serverd/openads_serverd) + dual-mode
client (tcp://host:port/dir URI in AdsConnect60) is already in
the tree (M12.x, Phase 2). Cross-host smoke: 13-op session
(Hello → Connect → OpenTable → GetRecordCount → GotoTop → GetField → Skip → Disconnect) round-trips end-to-end through a
Windows-side client → remote Linux server over an SSH-forwarded
TCP channel; ~9 ms server-side per op, the rest is real WAN RTT.
- DBF read/write — C / N / L / D / M columns, positional and
by-name field access,
APPEND BLANK, per-field assignment, deletion flag, durable flush, dynamic table creation (AdsCreateTableparses rddads'NAME,Type,Len,Dec;…field-def syntax). - DBF maintenance —
AdsZapTableempties a DBF + clears every bound index in lockstep;AdsPackTablecompacts deleted records out of the DBF (Clipper semantics: leaves indexes stale, caller follows up withAdsReindex);AdsReindexrebuilds every bound index from the current records using each tag's expression. - CDX index — full FoxPro compound layout (file header at offset
0 holding the structure tag, sub-tag headers, per-tag B+tree),
multi-tag-per-file API (
add_tag/open_named/list_tags), Harbour-equivalent compact-leaf bit-pack (bBitsderived from key length, mirroringhb_cdxPageLeafInitSpace), full multi-level B+tree leaf+branch split with separator promotion and new-root creation (M(cdx-split), 0.3.7 — stress-tested at 200 000 rows × 2 tags in one bag), branch descent (BE child pointers),dbSeekexact + soft,dbGoTop/dbSkipwalks, auto-sync on every mutation across all bound tags (active + parked extras), dynamic creation viaAdsCreateIndex61. - NTX index — Clipper layout, multi-level B+tree split (M9.10
closed the M3.7 single-level limitation), cache-based in-order
traversal for
next/prevover multi-level trees, dynamic creation viaAdsCreateIndex61, multi-file binding (M9.14 — apps can bundle several.ntxfiles on a singleUSEand swap focus between them without losing the parked tags' write sync). - Compound key expressions —
UPPER(field),LOWER(field),LTRIM/RTRIM/ALLTRIM,STR(n)/STR(n,len)/STR(n,len,dec),DTOS(date),SUBSTR(s,start[,len]), string concatenation with+.UPPER/LOWER/SUBSTRwalk UTF-8 codepoints (ASCII + Latin-1 supplement case map, includingÿ↔Ÿ); the evaluator runs at index sync time, so anINDEX ON UPPER(name)tag normalises every key as the app writes records, including non-ASCII rows produced byAdsSetStringW. - Full-text search —
AdsCreateFTSIndexwrites a clean-room# OpenADS FTS v0text file (sorted token + recno-list per row);AdsFTSSearchand the SQLCONTAINS(<col>, '<query>')predicate intersect per-token recno lists with AND semantics. Tokeniser honoursulMinWordLen/ulMaxWordLen, custom delimiter + noise-word arrays, and an English stop-word seed. - WAL recovery — append-only log with CRC-32C records,
group-commit primitive (
sync_to(lsn)), per-record LSN, idempotent recovery via theopenads.lsnmapsidecar. - Transactions (TPS) —
AdsBeginTransaction/AdsCommitTransaction/AdsRollbackTransaction, in-memory ordered op log + named savepoints, multi-table commit, rollback marks appended records as deleted (Clipper convention) and writes back before-images for modified rows. - AES-128 / AES-256 ECB — vendored tiny-AES-c (Unlicense), validated against FIPS-197 + NIST SP 800-38A.
- Memo (DBT / FPT) — read + write round-trip;
Connection::open_tableauto-attaches the right memo store based on the DBF signature (0x03→ no memo,0x30→ CDX with FPT memo). FPT blocks carry an explicit type tag (Text / Picture / Object), so the same field can hold either text memos orADS_BINARY/ADS_IMAGEpayloads with embedded NULs. - Data Dictionary —
.addalias resolution;Connection::open(<.add>)resolves member tables on everyAdsOpenTable. - Locking — OS byte-range locks with the same ranges as the
original ACE so installs can coexist during migration. Lock
acquires are non-blocking (
LockFileEx LOCKFILE_FAIL_IMMEDIATELYon Windows,fcntl F_SETLKon POSIX) and re-attempt up to a configurable retry budget (AdsSetLockCycle/AdsSetLockRetryCount, defaults 100 ms / 10 retries).
- 231
Ads*exports — every entry point Harbourc:\harbour\lib\win\msvc64\rddads.libreferences is resolvable through OpenADS' DLL. Real implementations for ~ 130 of them; the remainder are local-mode silent-success (theAdsMg*server-management surface, theAdsDD*advanced-DD CRUD surface, and theCache*/Set*/Refresh*/Customize*Harbour-side preferences — all returnAE_SUCCESSand either zero-fill the caller's struct or report empty / quiescent state, so apps that only inspect the return code keep running). No exports hard-fail withAE_FUNCTION_NOT_AVAILABLEat the function level any more. Specific ADD-only branches (e.g.AdsRestructureTable'spucDeleteFields/pucChangeFieldsarguments) still surface that error code at the argument level with a clear comment pointing at the 0.3.x deferral. The split is documented inline insrc/abi/ace_stubs.cpp. - 6 legacy CRT shims —
_dclass,_dsign,_wfsopen,_getch,_kbhit,_eofre-exported fromace64.dllso apps built against Harbour's prebuilt MSVC2013-era libs link without rebuilding Harbour itself.
- DML.
INSERT INTO <t> (cols) VALUES (lits)(M10.5),UPDATE <t> SET col=lit, … [WHERE …](M10.7), andDELETE FROM <t> [WHERE …](M10.7) all flow throughAdsExecuteSQLDirect; the dispatcher peeks at the leading keyword and routes to the right path. - SELECT.
SELECT * FROM <table> [WHERE <expr>] [ORDER BY <col> [ASC|DESC]]where<expr>is a full boolean tree built fromAND/OR/NOT/ parens (M10.3) over leaves that are either an infix comparison<col> op <lit>(six operators:=,!=/<>,<,>,<=,>=; literal can be a string'…'or a numeric42/3.14) orCONTAINS(<col>, '<query>')against a prebuilt.ftsfile. ORDER BY (M10.6) materialises matching recnos and sorts by the column's typed value. CONTAINS captures a precomputed recno-set so the FTS lookup runs once per query, not per row. Projection lists, joins, aggregates, subqueries,CREATE TABLE/CREATE INDEXarrive in later 0.3.x milestones.
- 271 doctest cases / 4360 assertions passing on Windows / MSVC Release.
- Harbour smoke harness producing a runnable
smoke.exethat drives the full read + write + index + multi-tag + transaction + memo + compound-expression path throughrddads.liband OpenADS'ace64.dll.
OpenADS ships in three rough phases. Each row links to the milestone
tag that lands the work; partial milestones become done once their
follow-ups merge.
Validated against c:\harbour\contrib\rddads.lib end-to-end through
tests/harbour_smoke/smoke.prg.
| Tag | Milestone |
|---|---|
m0-done |
Project scaffolding, build, doctest harness |
m1-done |
ABI handle registry + minimal C entry points |
m2-done |
DBF reader (header / fields / records) |
m3-done |
CDX + NTX index drivers (M3 baseline) |
m3.5-done |
CDX leaf bit-pack matches FoxPro on disk |
m3.6..3.10 |
Reviewer-flagged compat fixes; CDX compound layout; CDX multi-tag API; NTX cache traversal |
m4-partial |
DBF write path + memo (DBT / FPT) + AES-128/256 |
m5..5.5 |
WAL with savepoints, group commit, idempotent recovery via openads.lsnmap |
m6-partial |
Data Dictionary .add alias resolution |
m7.x-partial |
Minimal SQL (SELECT * FROM ... [WHERE ...]) |
m8.0..8.2 |
DLL build (ace64.dll/ace32.dll); rddads link validation; first end-to-end smoke (AdsVersion) |
m8.3 |
Harbour walks a real DBF |
m8.4 |
ACE field-type constants verified empirically |
m8.5 |
Multi-field DBF (C/N/L/D) end-to-end |
m8.6 |
dbSeek through CDX |
m8.7..8.8 |
Write path (dbAppend + FIELD-> :=); active index auto-sync |
m8.9 |
Multi-tag CDX with OrdSetFocus |
m8.10 |
Transactions: Begin/Commit/Rollback |
m8.11 |
Memo M-fields (FPT) round-trip |
0.1.0 |
Final 0.1.0 release |
| Tag | Milestone |
|---|---|
m9.1-done |
Compound CDX expressions evaluator (UPPER, STR, concat, ...) |
m9.2-done |
Stub batch reorganised into real / no-op / missing |
m9.3-done |
Compound expressions validated end-to-end via Harbour |
m9.4-done |
AdsGotoRecord + table/file metadata real impls |
m9.5-done |
AdsCreateTable (rddads field-def parser → DBF on disk) |
m9.6-done |
AdsRefreshRecord + AdsExtractKey |
m9.7-done |
AdsCreateIndex61 with compound-expression support |
m9.8-done |
AdsZapTable + AdsPackTable |
m9.9-done |
AdsReindex — rebuild every bound index from current records |
m9.10-done |
NTX multi-level B+tree split (closes M3.7 limit) |
m9.11-done |
AdsCopyTable / AdsCopyTableContents / AdsConvertTable |
m9.12-done |
AdsFindFirstTable / AdsFindNextTable / AdsFindClose (* / ? glob, case-insensitive, returns AE_NO_FILE_FOUND when exhausted) |
m9.13-done |
AdsGetBinaryLength / AdsGetBinary / AdsSetBinary + real AdsGetMemoDataType (FPT block-type tag round-trip; ADS_BINARY → Object, ADS_IMAGE → Picture, text → Text; offset-based chunked reads) |
m9.14-done |
NTX multi-tag binding — multiple .ntx files coexist on one Table (AdsOpenIndex / AdsCreateIndex61 / legacy AdsCreateIndex are all additive; same-path reopen refreshes; AdsCloseIndex releases extra views without disturbing the active order) |
m9.15-done |
Real AdsGetServerName / AdsGetServerTime — local host name + ISO date / HH:MM:SS time + ms-of-day, plus the 6-arg AdsGetServerTime shape rddads' ADSGETSERVERTIME actually expects (the previous 2-arg stub left rddads' on-stack date/time bufs uninitialised). Also fixes a latent index-binding leak: AdsCloseTable / AdsCloseAllTables / AdsDisconnect now purge the global binding map so a future Table allocation at the same heap address can't inherit stale entries. |
m9.16-done |
Chunked AdsSetBinary — per-(table, field) accumulator lets rddads deliver an oversized ADS_BINARY / ADS_IMAGE payload across several calls (ulOffset != 0, ulBytes < ulTotalBytes); the field only lands in the memo store once every byte arrives. Mid-write chunks that would run past the announced total fail; pending state is dropped at table teardown. |
m9.17-done |
Unicode *W variants — AdsSetStringW / AdsGetStringW / AdsGetFieldW. UTF-16LE ↔ UTF-8 transcoding at the ABI boundary; field names accept both UTF-16 NUL-terminated strings and ADSFIELD(n)-style numeric indices (low-pointer encoded). Engine continues to store byte sequences without a fixed codepage assumption. |
m9.18-done |
Lock retry / cycle policy — AdsSetLockCycle / AdsGetLockCycle / AdsSetLockRetryCount / AdsGetLockRetryCount (ms between attempts + retry count, defaults 100 ms / 10 retries). AdsLockTable / AdsLockRecord switched to non-blocking byte-range acquires (LockMgr::try_lock_* / LockFileEx LOCKFILE_FAIL_IMMEDIATELY / fcntl F_SETLK) and re-attempt up to the configured budget before reporting AE_LOCKED. |
m9.19-done |
AdsCreateFTSIndex — clean-room OpenADS-native .fts inverted-index file (UTF-8 text: # OpenADS FTS v0 header + sorted <token>\t<recno1>,<recno2>,... rows). Tokeniser respects ulMinWordLen / ulMaxWordLen, custom delimiter / noise-word arrays, plus a default ASCII delimiter set and an English stop-word seed. Search-side functions remain a follow-up milestone. |
m9.20-done |
AdsAddCustomKey / AdsDeleteCustomKey — manual-mode (key, recno) injection on the current record. Each call evaluates the index's expression against the positioned row and routes through the existing IIndex::insert / IIndex::erase paths, matching rddads' DBOI_KEYADD / DBOI_KEYDELETE call sites. |
m9.21-done |
FTS search side — AdsFTSSearch(hConn, pucFile, pucQuery, paRecnos, *pulCount) loads the .fts file, tokenises the query, intersects per-token recno lists (AND semantics), and writes the hit list into paRecnos with truncation reporting. SQL gains a CONTAINS(<col>, '<query>') predicate that lowers to a precomputed recno set captured in the row predicate, so the FTS lookup runs once per query instead of per row. |
m9.22-done |
UTF-8 codepoint-aware index-expression evaluator — UPPER, LOWER, SUBSTR walk codepoints instead of bytes. ASCII + Latin-1 supplement (incl. ÿ↔Ÿ) case map cleanly; codepoints outside that range pass through. INDEX ON UPPER(name) over a UTF-8 column now produces stable keys for non-ASCII rows. Bare-field indexes still byte-identical (existing CDX / NTX files round-trip unchanged). |
m9.23-done |
Misc MISS fillers — real AdsGetLongLong, AdsSetFieldRaw, AdsVerifySQL, AdsFailedTransactionRecovery, AdsGetAllLocks, AdsSkipUnique. |
m9.24-done |
Local-mode AdsMg* surface (15 calls). Synthetic mgmt handle, struct-shaped queries zero-fill caller buffers, list-shaped queries report empty count; apps see "everything quiescent" instead of AE_FUNCTION_NOT_AVAILABLE. |
m9.25-done |
Local-mode AdsDD* CRUD surface (14 calls). All 14 accept silently / zero-fill. 0.3.x will replace these no-ops with real persistence in the OpenADS DD format. |
m9.26-done |
AdsRestructureTable (ADD-fields path) — rewrites the DBF with the original schema + pucAddFields appended, copies every record's old-field bytes verbatim and blank-pads the new fields. The MISS list is now empty. |
m9.27-done |
CI matrix portability fix — legacy_crt_shims.cpp is now a no-op on POSIX builds; the engine compiles cleanly on Linux + macOS through .github/workflows/ci.yml (windows-2022 / ubuntu-22.04 / macos-13). |
v0.2.0 |
Final 0.2.0 release (2026-05-04) |
- Linux Harbour smoke install. The engine builds + tests cleanly
on Linux + macOS through the CI matrix; what's still missing is a
Linux Harbour install so the smoke harness (
smoke.exe→ace64.dll) can run end-to-end on a non-Windows runner. usPageSizehonoured beyond NTX/CDX.AdsCreateIndex61/AdsCreateFTSIndexaccept ausPageSizeargument. NTX (1024) and FoxPro CDX (512) are fixed-size by their on-disk format — matching the behaviour of the original ACE — so OpenADS records the value but doesn't change the layout. Variable page sizes will land alongside the proprietary ADI driver in 0.3.x.
OpenADS will only adopt the proprietary formats once a clean-room
compatibility specification is available — written from publicly
observable behaviour and from the Harbour contrib/rddads source,
not from disassembly of SAP-owned binaries or from any material
whose use is restricted by the Advantage SDK / ACE EULA.
| Tag | Milestone |
|---|---|
m10.1-done |
Real OpenADS-native DD persistence — replaces M9.25 silent-success no-ops with round-trip storage in the v1 text format (USER / MEMBER / LINK / INDEX / RI / DBPROP / USERPROP rows); Connection::open(<.add>) callers now see CRUD writes survive across reopens. |
m10.2-done |
VFP I (4-byte int32) / Y (8-byte currency, money * 10000) / B (8-byte IEEE-754 double) field decode + encode round-trip. |
m10.3-done |
SQL WHERE — full boolean expression tree (OR / NOT / parens) on top of the M9.21 CONTAINS predicate; numeric literals (AGE >= 18) and string literals coexist. |
m10.4-done |
AdsRestructureTable DELETE-fields path + real AdsSetIndexDirection (active order walks reverse on demand). The function-level Ads* surface is now MISS-free; CHANGE-fields still deferred. |
m10.5-done |
SQL INSERT INTO <t> (cols) VALUES (lits) — string + numeric literals; AdsExecuteSQLDirect dispatches by leading keyword. |
m10.6-done |
SQL `ORDER BY |
m10.7-done |
SQL UPDATE <t> SET col=lit, … [WHERE …] + DELETE FROM <t> [WHERE …] — bulk row mutation through AdsExecuteSQLDirect. DELETE follows Clipper convention (rows marked, not removed; AdsPackTable evicts later). |
m10.8-done |
SQL projection lists — SELECT col1, col2 FROM …; cursor reports only the projected columns (in the listed order) via AdsGetNumFields / AdsGetFieldName / AdsGetFieldType / AdsGetFieldLength / AdsGetFieldDecimals / ADSFIELD(n). |
m10.9-done |
SQL DDL — CREATE TABLE <name> (<col> <Type> [(<len>[,<dec>])], …) + CREATE INDEX <tag> ON <table> (<expr>) [DESCENDING] [UNIQUE]. Both lower into the engine's existing AdsCreateTable / AdsCreateIndex61 entry points. |
m10.10-done |
SQL aggregates — COUNT(*) / COUNT(col) / SUM / AVG / MIN / MAX(col). The cursor is a synthetic 1-row temp DBF (one C(30) column per aggregate). |
m10.11-done |
VFP autoinc fields — descriptor offsets 18 (flags) / 19-22 (counter LE) / 23 (step). AppendRecord pre-fills the field with the current counter and persists next = counter + step back to disk. Counter survives close + reopen. |
m10.12-done |
AdsRestructureTable CHANGE-fields — same-type length / decimals overrides for existing columns. Type conversion still surfaces AE_FUNCTION_NOT_AVAILABLE (clean-room policy). |
m10.13-done |
SQL INNER JOIN <b> ON <l_col> = <r_col> — parser. |
m10.14-done |
SQL INNER JOIN executor — hash on right column, materialise merged rows into temp DBF cursor; right-side fields prefixed R_. Combos with WHERE / ORDER BY / aggregates over the joined cursor land in a follow-up. |
m10.15-done |
SQL <col> IN (…) — literal list IN ('a', 'b', …) and subquery IN (SELECT col FROM t). Matching set captured at compile time so per-row check is O(1). |
m10.16-done |
SQL LEFT [OUTER] JOIN — left rows survive without a right-side match (right fields blank-pad); bare JOIN keyword treated as INNER per SQL convention. |
m10.17-done |
SQL EXISTS (SELECT … FROM t) — uncorrelated presence test, materialised once at compile time. NOT EXISTS falls out of the existing NOT-tree. |
m10.18-done |
SQL scalar subquery — <col> op (SELECT <col> FROM <t>). Inner subquery's first projected value lands in the cmp's literal slot at compile time; outer-column type drives string vs numeric compare semantics. |
m10.19-done |
SQL aggregate scalar subquery — <col> op (SELECT MAX(x)/MIN/SUM/AVG/COUNT FROM t). The single aggregate lands in the cmp's number slot at compile time. |
m10.20-done |
SQL JOIN combined with WHERE / ORDER BY in a single statement. Outer clauses compile against the merged cursor's schema (left names verbatim, right names R_<orig>). |
m10.21-done |
SQL RIGHT [OUTER] JOIN — every right row survives, blank left when no match. Hash built on LEFT column; merged cursor schema stays direction-agnostic. |
m10.22-done |
SQL FULL [OUTER] JOIN — union of LEFT + RIGHT semantics; left walk emits matched + LEFT-style fillers and tracks which right recnos were matched, then a follow-up scan emits unmatched right rows with a blank left filler. |
m10.23-done |
SQL JOIN + aggregate combo — COUNT/SUM/AVG/MIN/MAX over the merged JOIN cursor; aggregate walk runs after the M10.20 outer WHERE, materialises a 1-row temp DBF with C(30) per aggregate. |
m10.24-done |
SQL correlated EXISTS subquery — EXISTS (SELECT … FROM b WHERE b.x = a.y) re-evaluates per outer row; outer-column refs in the subquery WHERE bind to the live outer cursor. |
m10.25-done |
SQL GROUP BY <col>[, <col>…] [HAVING <agg> op num] — multi-row aggregate cursor (one row per group), preserves source column type byte + length, HAVING filters on a projection-aligned aggregate slot. |
m10.26-done |
SQL UNION / UNION ALL — N-way merge of SELECT * FROM t [WHERE ...] members; first member's schema drives the merged cursor; UNION dedups by raw record bytes, UNION ALL keeps duplicates. |
m10.27-done |
SQL UNION + member projection — UNION members may now carry a projection list (SELECT col1, col2 FROM t); first member's projection drives the merged DBF schema (preserving raw_type / length / decimals) and subsequent members align positionally. |
m10.28-done |
SQL UNION + ORDER BY — last UNION member may carry an ORDER BY clause; the sort applies to the merged result. Numeric vs lex compare follows the merged column type. |
m10.29-done |
SQL correlated scalar subquery — <col> op (SELECT <col_or_agg> FROM b WHERE b.x = a.y) re-evaluates per outer row, binding outer columns into the inner WHERE (M10.24 binding pattern reused). |
m10.30-done |
SQL HAVING tree — boolean AND / OR / NOT / parens over aggregate comparisons; e.g. HAVING COUNT(*) > 1 AND SUM(amt) > 50. |
m10.31-done |
SQL DISTINCT — SELECT DISTINCT [cols|*] FROM ... dedups by projected columns; first occurrence wins, applied post-WHERE / post-ORDER-BY. |
m10.32-done |
SQL LIMIT N [OFFSET M] — slices the post-clause traversal sequence (skip M, take N). |
m10.33-done |
SQL BETWEEN / LIKE — <col> BETWEEN <lit1> AND <lit2> (inclusive, numeric or lexicographic) and <col> LIKE '<pattern>' with SQL % (any sequence) and _ (single char) wildcards. |
m10.34-done |
SQL GROUP BY across JOIN — SELECT <agg>(...) FROM a JOIN b ON ... GROUP BY ... produces one row per group of joined rows, mirroring the M10.25 grouped path on the merged cursor. |
m10.35-done |
SQL correlated IN subquery — <col> IN (SELECT col FROM b WHERE b.x = a.y) re-evaluates per outer row when the inner WHERE references an outer column. |
m10.37-done |
SQL multi-column ORDER BY — ORDER BY a [ASC|DESC], b [ASC|DESC] ... cascades on ties; per-column direction independent; numeric vs lex driven by column type. |
m11.1-done |
VFP V (Varchar) + Q (Varbinary) field types — decode trims trailing NUL pad to recover the meaningful prefix; encode NUL-pads the unused tail so reads recover the original length. |
m11.3-done |
TPS — AdsReleaseSavepoint drops a savepoint marker without rolling back; nested AdsBeginTransaction / AdsCommitTransaction increment / decrement depth (only the outermost commit flushes); inner ROLLBACK still aborts every nesting level. |
m10.38-done |
SQL CASE WHEN <cond> THEN <lit> [WHEN ...] [ELSE <lit>] END [AS alias] in projection — first matching branch wins; result lands in a C(30) cell of the materialised result cursor. Conditions may use Cmp / AND / OR / NOT / BETWEEN / LIKE. |
m10.36-done |
SQL UNION + complex members — UNION members may now carry JOIN, GROUP BY, aggregates, CASE, DISTINCT, LIMIT. Dispatcher recurses into AdsExecuteSQLDirect per member (state mutex is recursive); each member's cursor schema drives merge alignment via cursor_projections. |
m11.4-done |
AEP host — CREATE PROCEDURE <name> AS '<dll>::<sym>' registers an external function on a connection; EXECUTE PROCEDURE <name>(args) calls it through platform/dll (LoadLibrary / dlopen) and returns its result string in a 1-row C(255) RESULT cursor. Clean-room ABI: int proc(const char* args, char* out_buf, size_t out_cap); not byte-compatible with the legacy ADS Extended Procedure protocol. |
m11.2-done |
OpenADS-encrypted DBF (AES-256-CTR) — header byte 0xC3 marks an OpenADS-encrypted variant; record bodies XOR with an AES-256-CTR keystream keyed off the connection password (AdsSetEncryptionPassword). AdsEncryptTable upgrades a plain DBF in place. Schema stays plaintext. Not byte-compatible with proprietary SAP ADS encrypted .adt files — that format's byte boundary remains undocumented. |
m10.39-done |
SQL scalar string fns — UPPER / LOWER / LEN / TRIM / LTRIM / RTRIM over a single column in projection, optional AS alias. Materialised result cell is C(N). |
m10.40-done |
SQL arithmetic in projection — <col> {+,-,*,/} <num_or_col> evaluated as doubles, formatted with %g into a C(30) cell. |
m10.41-done |
SQL INSERT INTO t (cols) SELECT ... — inner SELECT may use any engine clause; rows copy positionally into the target via the existing append path. |
m10.42-done |
SQL CREATE TABLE t AS SELECT ... — derives the new table's schema from the inner cursor's projection, creates the DBF, bulk-copies rows. |
m10.43-done |
SQL multi-arg scalar fns — SUBSTR(col, start, [len]), CONCAT(a, b, ...), REPLACE(col, old, new). Each arg may be column / string / numeric literal. |
m10.45-done |
SQL date arithmetic on YYYYMMDD strings — DATEDIFF(a, b) (days), DATEADD(col, n) (returns YYYYMMDD). |
m10.46-done |
SQL derived tables — FROM (SELECT ...) [AS alias]. Inner SELECT runs through the recursive dispatcher; outer WHERE / ORDER BY / projection apply on top of its cursor. |
m10.48-done |
SQL Common Table Expression — WITH name AS (SELECT ...) SELECT ... FROM name. Rewritten in parse_select to a derived-table form by inline-substituting name with (<inner>). |
m10.44-done |
SQL <col> IS [NOT] NULL — first checks the VFP NULL bitmap (M11.6); falls back to a trim-trailing-space check that treats all-blank cells as NULL for non-bitmap-bearing DBFs. |
m10.47-done |
SQL ROW_NUMBER() OVER (...) in projection — first cut emits the 1-based position in the materialised result cursor; PARTITION BY / ORDER BY inside the OVER are parsed but ignored. |
m11.6-done |
VFP NULL bitmap — parse_dbf_fields detects flags-byte bit 1 + assigns each nullable field a null_bit ordinal; Table::is_field_null reads _NullFlags system column and tests the bit. |
m12.1-done |
Phase 2 wire-protocol skeleton — network/wire.{h,cpp} defines a length-prefixed frame (4 BE bytes + opcode + payload) with an Opcode enum reserved for Hello / Connect / OpenTable / ExecuteSQL / Fetch / Disconnect / Error. No socket layer yet. |
m11.7-done |
String collation — AdsSetCollation(hConn, BINARY|NOCASE). Connection-scoped flag stamps onto every WHERE Cmp; NOCASE lowercases ASCII A-Z before compare across Eq/Ne/Lt/Gt/Le/Ge/Between/Like. |
m11.8-done |
OEM (CP437) ↔ UTF-8 conversion — AdsConvertOemToAnsi / AdsConvertAnsiToOem round-trip text through a new engine/codepage module carrying the public IBM CP437 upper-half table. |
m10.49-done |
SQL window OVER (PARTITION BY + ORDER BY) — pre-computes per-row values into a side map; partitions group by concatenated key, ORDER BY drives within-partition order. |
m10.50-done |
SQL RANK() / DENSE_RANK() — join ROW_NUMBER in the window family; RANK skips on ties, DENSE_RANK doesn't. Tie detection via the ORDER BY key. |
m10.51-done |
SQL qualified column refs — <alias>.<col> in WHERE / projection / JOIN ON; read_identifier consumes the suffix and resolves the bare column name. |
m10.52-done |
SQL multi-row INSERT INTO t (cols) VALUES (...), (...), ... — appends one record per tuple. |
m10.53-done |
SQL NULLIF(a,b) / COALESCE(a,b,...) / IFNULL(expr,default) — null-handling helpers in projection (empty string = NULL by convention). |
m10.54-done |
SQL aggregate <agg>(...) FILTER (WHERE <expr>) — per-slot row filter; CountStar with FILTER uses the filtered count, others always do. |
m12.2-done |
Phase 2 TCP socket layer — network/socket.{h,_win32.cpp,_posix.cpp}. listen / accept / connect / send / recv / close + ephemeral-port binding. Win32 links ws2_32. |
m12.3-done |
Phase 2 server skeleton — network/server spawns an accept thread + per-connection session thread. Dispatches Hello → HelloAck (version), Connect → ConnectAck (data_dir echo), Disconnect → close, others → Error frame. recv_exact / read_frame / write_frame helpers exposed for M12.4 / M12.5. |
m12.4-done |
Phase 2 remote read-only table ops — server proxies OpenTable / GotoTop / Skip / GetField / GetRecordCount / AtEOF / CloseTable. Per-session engine::Connection + 32-bit table-id map. |
m12.5-done |
Phase 2 dual-mode DLL — AdsConnect60("tcp://host:port/<dir>", ...) opens a wire client; AdsOpenTable / AdsGotoTop / AdsSkip / AdsGetField / AdsGetRecordCount / AdsAtEOF / AdsCloseTable / AdsDisconnect each branch on remote-handle kind. ace64.dll is now drop-in for either local or TCP-server mode. |
m12.6-done |
Phase 2 remote write surface — server proxies AppendBlank / SetField / DeleteRecord / RecallRecord / GotoRecord / FlushTable; AdsAppendRecord / AdsSetString / AdsDeleteRecord / AdsRecallRecord / AdsGotoRecord / AdsWriteRecord each gate a remote branch. SetField wire payload is [u32 tid][u16 name_len][name][value]; engine stores via Table::set_field(idx, std::string). Round-trip verified by reopening the same DBF locally and confirming the new record + value hit disk. |
m12.7-done |
Phase 2 remote SQL exec — AdsCreateSQLStatement on a remote connection allocates a wire-aware statement; AdsExecuteSQLDirect ships the SQL through a new ExecuteSQL/ExecuteSQLAck opcode pair. Server lazily opens a parallel ABI connection on first use, runs the query through AdsExecuteSQLDirect, and wraps the returned cursor handle in cursor_tbls so the M12.4 read ops (GotoTop / Skip / GetField / GetRecordCount / AtEOF / CloseTable) serve it through the same wire. Returns cursor table-id (0 = no cursor for INSERT / UPDATE / DELETE / DDL). |
m12.8-done |
Phase 2 remote AdsReindex — new Reindex/ReindexAck opcode pair routes through engine::Table::reindex() on the per-session connection. CREATE INDEX is already covered by M12.7's ExecuteSQL DDL path. |
m12.9-done |
Phase 2 server credential auth — Connect frame payload extended to [u16 dlen][dir][u16 ulen][user][u16 plen][password]; Server::add_credential registers a user/password pair; empty map = back-compat dev mode (any client passes). AdsConnect60 forwards pucUser / pucPwd through the wire. RemoteConnection now self-disconnects on drop so a failed connect doesn't leave the server-side session blocked in read_frame(). |
m12.10-done |
Phase 2 ACE error-code propagation — Error frame payload now starts with [u32 LE ace_code][message]; server folds either a literal ACE code or a code pulled from a sub-result's util::Error into every Error frame. Client parses the prefix back into util::Error.code so the Ads* return value surfaces the real ACE code (AE_LOGIN_FAILED 7077, AE_NO_FILE_FOUND 5018, AE_PARSE_ERROR 7200, …) instead of the legacy generic AE_INTERNAL_ERROR 5000. |
m12.11-done |
Phase 2 batch row fetch — Fetch/FetchAck opcode pair (0x32 / 0x33, reserved in M12.1). Request: [u32 tid][u32 max_rows][u8 ncols][per col: u8 nlen, name]. Reply: [u32 nrows][u8 ncols][per row, per col: u16 vlen, val]. Walks up to max_rows from the cursor's current position; works for both engine handles (returned by OpenTable) and SQL-cursor handles (returned by ExecuteSQL). RemoteConnection::fetch_batch exposes the surface as vector<vector<string>>; collapses per-row Skip+GetField round-trip count by N for WAN-latency callers. |
m12.12-done |
Phase 2 real TLS — vendored mbedtls 3.6 LTS (Apache 2.0) via OPENADS_WITH_TLS CMake option (OFF by default). When enabled, AdsConnect60("tls://host:port/<dir>", ...) opens a real MbedTlsTransport instead of returning AE_FUNCTION_NOT_AVAILABLE. connect_tls(host, port, TlsConfig) returns a unique_ptr<ITransport> ready for read_frame / write_frame. Server-side TLS termination is queued for v1.0.x (mbedtls 3.6 doesn't expose a supported way to adopt an externally-accepted fd; deployments today should front the server with a TLS proxy if needed). |
m12.13-done |
Phase 2 transport abstraction — network/transport.h defines a polymorphic ITransport surface (send / recv / close / valid); PlainTransport is the only concrete impl today. read_frame / write_frame carry both Socket& and ITransport& overloads so existing call sites stay one-liner while the v0.4.0 TlsTransport plugs in without touching the server / client business logic. |
| docs (M12) | docs/wire-protocol.md — formal spec of every opcode + payload format, error-frame layout, transport notes, and versioning policy. Reference for non-C++ clients (Python, Go, Rust, Harbour AEP) that want to talk to an OpenADS server without reading the C++ source. |
- M11.5 ADT (proprietary table format) — depends on a clean-room specification of the ADT on-disk layout. None is publicly available; the only known route is reverse- engineering SAP-owned binaries, which violates the project's clean-room policy and the Advantage SDK / ACE EULA. Not implementable until a clean-room spec exists.
- VFP NULL-bitmap extension — the
0x32autoinc / null-flag header byte stays deferred. Autoinc (M10.11) and V / Q types (M11.1) are real today. - ADM memo format — pairs with ADT, same gating as ADT.
- ADI index format — proprietary B+tree variant; same gating.
- Phase 2 server hardening (M12.6+) — remote write ops
(Insert / Update / Delete via wire), remote SQL exec
(
AdsExecuteSQLDirectover the wire), remote index ops (CreateIndex/Reindex), authentication + connection multiplexing, full ACE error-code mapping over the wire, large-payload streaming (memo blobs / SQL result sets). Read-only Open / Goto / Skip / GetField / GetRecordCount / AtEOF / CloseTable already round-trip through the M12.4 server and the M12.5 dual-mode client. - More SQL — ORDER BY across DISTINCT, multi-column ORDER BY inside UNION, richer projection expressions on top of CASE, arithmetic and string concat (already partial via M10.39 / M10.40 / M10.43).
- Server reuses the same engine; clients speak the original ACE
remote protocol so a single
ace64.dllbuild can act as either a local DLL or a TCP client to a remote OpenADS server. - Network framing + auth + connection multiplexing.
- Compatibility-test matrix against real Advantage 11.x + 12.x installations.
| Topic | Decision |
|---|---|
| Operation mode | LOCAL only (no remote server). Phase 2 will add a TCP server reusing the same engine. |
| Table formats | ADT + CDX + NTX + VFP (all four ADS-supported types). |
| Memo / index | ADM / FPT / DBT (memo) · ADI / CDX / NTX (index). |
| ABI compatibility | Identical C ABI to ACE; applications do not recompile. |
| Validation frontend | c:\harbour\contrib\rddads, unmodified. |
| SQL | Full Advantage SQL dialect (parser + planner + executor + xBase UDFs + AEP host for external stored procedures). |
| Concurrency | OS byte-range locking with ranges identical to ACE — coexistence with original ACE installations during migration. |
Data Dictionary (.add) |
Full support: member tables, users/groups/permissions, RI, views, procedures, links, validations, defaults. |
| Encryption | AES-128 / AES-256 (ADS 9+ format). The legacy proprietary cipher is out of scope for phase 1. |
| Transactions (TPS) | Multi-table ACID, savepoints, crash recovery via write-ahead log. |
| Platforms | Windows (x86 + x64), Linux, macOS, BSD. |
| Language / build | C++17 with extern "C" external ABI. CMake + GitHub Actions. |
| i18n | OEM ↔ ANSI ↔ UTF-8 ↔ UTF-16 (the API's *W variants). |
| License | Apache License 2.0. |
┌──────────────────────────────────────────────────────────────────┐
│ Harbour application (no recompilation) │
│ REQUEST ADS / dbUseArea( .T., "ADS", ... ) │
└────────────────────────┬─────────────────────────────────────────┘
│ Clipper RDD API
┌────────────────────────▼─────────────────────────────────────────┐
│ rddads.lib (contrib/rddads — untouched) │
│ ads1.c / adsfunc.c / adsx.c / adsmgmnt.c │
└────────────────────────┬─────────────────────────────────────────┘
│ ACE C ABI (~230 Ads* entry points)
│ ace.h headers
═════════════════════════╪══════════════════════════════════════════
│ ▼ OPENADS REPLACES HERE
┌────────────────────────▼─────────────────────────────────────────┐
│ L1 — ABI Layer (libace32.dll / libace64.dll / libace.so) │
│ extern "C" wrappers → ACE handle translation → C++ engine │
│ Error code mapping (ACE codes ↔ engine errors) │
│ OEM / ANSI / UTF-8 / UTF-16 translation │
└────────────────────────┬─────────────────────────────────────────┘
│ internal C++ API (RAII handles)
┌────────────────────────▼─────────────────────────────────────────┐
│ L2 — Session / Connection Layer │
│ Connection (local path or Data Dictionary) │
│ Statement (prepared SQL cursor) │
│ HandleRegistry (ADSHANDLE → object pointer, thread-safe) │
└────────────────────────┬─────────────────────────────────────────┘
│
┌────────────────────────▼─────────────────────────────────────────┐
│ L3 — SQL Engine │
│ Lexer → Parser (AST) → Resolver → Planner → Executor │
│ DD-aware Catalog, xBase UDFs │
│ AEP host (stored procedures as .dll/.so plugins) │
└────────────────────────┬─────────────────────────────────────────┘
│
┌────────────────────────▼─────────────────────────────────────────┐
│ L4 — Engine Core (transport-agnostic) │
│ Table / Index / MemoStore / Cursor / Filter (AOF) │
│ LockMgr (OS byte-range, ACE-compatible ranges) │
│ TxLog (multi-table WAL ACID + savepoints + crash recovery) │
│ Catalog (DD .add reader/writer) │
│ PageCache / BufferMgr │
└────────────────────────┬─────────────────────────────────────────┘
│ Driver trait (open / read / write page)
┌────────────────┼────────────────┬───────────────┐
▼ ▼ ▼ ▼
┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐
│AdtDriver│ │CdxDriver│ │NtxDriver│ │VfpDriver│
│.adt+.adm│ │.dbf+.cdx│ │.dbf+.ntx│ │.dbf+.fpt│
│ +.adi │ │ +.fpt │ │ +.dbt │ │ +.cdx │
└────┬────┘ └────┬────┘ └────┬────┘ └────┬────┘
└────────────────┴────────────────┴────────────────┘
│
┌────────────────────────────────▼─────────────────────────────────┐
│ L5 — OS Abstraction (Platform) │
│ File I/O · mmap · byte-range locks · paths · time · threads │
│ Win32 and POSIX implementations │
└──────────────────────────────────────────────────────────────────┘
- L1 is the only module with C ABI. Everything else is internal C++17.
- L4's driver trait is the extension point: each table format is a swappable driver.
- The remote server in phase 2 will simply be another L1 frontend (TCP transport layer); L2 through L5 are reused as is.
- The SQL engine (L3) consumes L4 only through
CursorandCatalog; it has no knowledge of file formats. - The LockMgr preserves the exact byte-range semantics of ACE, allowing coexistence with original ACE installations on the same files during migration.
OpenADS/
├── CMakeLists.txt # top-level build, presets per platform
├── CMakePresets.json
├── LICENSE # Apache License 2.0
├── NOTICE # attribution + trademark notice
├── README.md
├── docs/
│ ├── architecture.md
│ ├── ace-coverage.md # entry-point status table (~230 fns)
│ ├── adt-format.md # ADT/ADM/ADI on-disk spec
│ ├── lock-ranges.md # ACE-compat byte-range table
│ ├── tx-log.md # WAL format + recovery protocol
│ └── sql-grammar.md # Advantage SQL EBNF + diffs
│
├── third_party/ # vendored deps
│ ├── tinyaes/ # AES-128/256 (Unlicense)
│ ├── utf8.h/ # UTF conversion (Unlicense)
│ ├── doctest/ # unit test framework (MIT)
│ └── ace-headers/ # OpenADS-authored ace.h compatibility surface
│
├── include/openads/ # public C++ headers (consumed by L1)
│ ├── engine.h
│ ├── connection.h
│ ├── table.h
│ ├── cursor.h
│ ├── catalog.h
│ └── error.h
│
├── src/
│ ├── abi/ # L1 — ACE C ABI shim
│ │ ├── ace_exports.def # Windows DLL export list
│ │ ├── ace_exports.cpp # ~230 extern "C" thunks
│ │ ├── handle_registry.cpp # ADSHANDLE ↔ object map
│ │ ├── error_map.cpp # ACE error codes
│ │ ├── charset.cpp # OEM/ANSI/UTF conversion
│ │ └── version.cpp # AdsGetVersion / AdsGetServerName
│ │
│ ├── session/ # L2
│ │ ├── connection.cpp
│ │ ├── statement.cpp
│ │ └── globals.cpp # AdsSetDefault / AdsSetFileType / etc.
│ │
│ ├── sql/ # L3
│ │ ├── lex/lexer.cpp
│ │ ├── parse/parser.cpp # recursive-descent
│ │ ├── parse/ast.h
│ │ ├── resolve/resolver.cpp # name binding, type check
│ │ ├── plan/planner.cpp # logical → physical
│ │ ├── plan/optimizer.cpp # predicate pushdown, index selection
│ │ ├── exec/executor.cpp # iterator pipeline
│ │ ├── exec/operators/ # scan / filter / sort / agg / join / ...
│ │ ├── func/scalar.cpp # xBase UDFs (LEFT/SUBSTR/CTOD/...)
│ │ ├── func/aggregate.cpp
│ │ └── aep/host.cpp # AEP plugin loader (.dll / .so)
│ │
│ ├── engine/ # L4 — core
│ │ ├── table.cpp
│ │ ├── cursor.cpp
│ │ ├── filter_aof.cpp # Advantage Optimized Filter
│ │ ├── lock_mgr.cpp # OS byte-range, ACE-compat ranges
│ │ ├── tx_log.cpp # WAL writer
│ │ ├── tx_recover.cpp # crash recovery
│ │ ├── savepoint.cpp
│ │ ├── catalog_dd.cpp # .add reader / writer
│ │ ├── page_cache.cpp
│ │ ├── buffer_mgr.cpp
│ │ └── encryption.cpp # AES-128 / 256
│ │
│ ├── drivers/ # L4 — format drivers
│ │ ├── driver_trait.h # abstract interface
│ │ ├── adt/
│ │ │ ├── adt_table.cpp # .adt header + records
│ │ │ ├── adi_index.cpp # .adi B+tree
│ │ │ ├── adm_memo.cpp # .adm blob store
│ │ │ └── field_types.cpp # autoinc / GUID / modtime / timestamp / ...
│ │ ├── cdx/
│ │ │ ├── dbf_table.cpp
│ │ │ ├── cdx_index.cpp # FoxPro CDX compact index
│ │ │ └── fpt_memo.cpp
│ │ ├── ntx/
│ │ │ ├── dbf_table.cpp # shared with cdx via dbf_common
│ │ │ ├── ntx_index.cpp # Clipper NTX
│ │ │ └── dbt_memo.cpp
│ │ ├── vfp/
│ │ │ ├── vfp_table.cpp # DBF v0x30 + nullable + autoinc
│ │ │ ├── cdx_index.cpp # symlink to ../cdx
│ │ │ └── fpt_memo.cpp
│ │ └── dbf_common.cpp # shared DBF header logic
│ │
│ ├── platform/ # L5 — OS abstraction
│ │ ├── file.h
│ │ ├── file_win32.cpp
│ │ ├── file_posix.cpp
│ │ ├── lock.h
│ │ ├── lock_win32.cpp # LockFileEx
│ │ ├── lock_posix.cpp # fcntl F_SETLK
│ │ ├── mmap.cpp
│ │ ├── path.cpp # case-insensitive matching on unix
│ │ ├── time.cpp
│ │ └── thread.cpp
│ │
│ └── util/
│ ├── span.h
│ ├── result.h # error-or-value
│ └── log.cpp
│
├── tests/
│ ├── unit/ # doctest, per-module
│ │ ├── adt_table_test.cpp
│ │ ├── cdx_index_test.cpp
│ │ ├── lock_mgr_test.cpp
│ │ ├── tx_log_test.cpp
│ │ ├── sql_parser_test.cpp
│ │ └── ...
│ ├── integration/ # full ABI roundtrip
│ │ ├── harbour_smoke.prg # runs against rddads
│ │ ├── byte_compat/ # diff vs reference ACE-produced files
│ │ └── conformance/ # ACE entry-point matrix
│ └── fixtures/ # canonical .adt / .dbf / .cdx samples
│
├── tools/
│ ├── adt-dump/ # CLI: hex-dump ADT structure
│ ├── cdx-dump/
│ ├── tx-replay/ # WAL replay / inspect
│ └── ace-trace/ # log every ACE call (debug shim)
│
├── benchmarks/
│ └── micro/ # paged read, lock contention, SQL
│
└── .github/workflows/
├── ci.yml # matrix Win / Linux / macOS / BSD
├── compat.yml # nightly run vs Harbour rddads tests
└── release.yml # tagged DLL / SO artifacts
src/abi/ace_exports.cppis the only contact point with the C ABI. Astatic constexprtable maps each ACE entry point to its internal handler. Optionally generated by a script fromace.h.src/drivers/driver_trait.hdefines the minimum interface:open / close / read_record / write_record / seek / scan / index_create / index_seek / lock_range. Each driver is roughly 3–5 files.src/engine/lock_mgr.cppcentralises lock ranges — the single source of truth for ACE coexistence.src/engine/tx_log.cppandtx_recover.cppare driver-independent: the WAL log records(driver_id, page_no, before_image, after_image)as opaque payloads.src/sql/is driver-independent and operates against an abstractCursor. SQL tests do not require real drivers (mock cursor).src/platform/is the only OS dependency. Engine tests use a platform mock to inject I/O failures.tools/is the debugging artillery — critical for byte-level diffs against original ACE.
Harbour PRG
USE clientes VIA "ADSCDX" SHARED
│
▼
rddads.lib :: hb_adsOpen() [contrib/rddads/ads1.c]
AdsConnect60( "C:\data", ..., &hConn ) ← OpenADS L1 entry
AdsOpenTable( hConn, "clientes.dbf", ← OpenADS L1 entry
ADS_CDX, ADS_DEFAULT,
ADS_NONE, ADS_DEFAULT,
ADS_DEFAULT, &hTbl )
│
▼ L1 ace_exports.cpp
extern "C" AdsConnect60(...)
→ openads::Connection::open( path, ... )
│
▼ L2 session/connection.cpp
Connection ctor
→ resolves path, registers handle, returns ADSHANDLE via HandleRegistry
│
▼ back to L1
extern "C" AdsOpenTable(...)
→ conn->open_table( "clientes.dbf", FormatHint::Cdx, OpenMode::Shared )
│
▼ L2 → L4 engine/table.cpp
Table::open()
1. DriverFactory::detect_or_force(path, hint) → CdxDriver
2. CdxDriver::open()
├─ platform::file_open("clientes.dbf", RW)
├─ read DBF header (32 bytes + field descriptors)
├─ platform::file_open("clientes.cdx", RW) if it exists
├─ CdxIndex::load_root_pages()
└─ FptMemo::open("clientes.fpt") if memo fields
3. LockMgr::acquire_open_share() (byte-range header lock, ACE range)
4. PageCache::register(file_handles)
5. TxLog::register_table(table_id) (no-op outside a transaction)
6. returns Table*
│
▼
HandleRegistry::register(Table*) → ADSHANDLE
│
▼
return SUCCESS to rddads → returned to PRG
PRG
AdsCreateSQLStatement( hConn, &hStmt )
AdsExecuteSQLDirect( hStmt, "SELECT...", &hCursor )
│
▼ L1
extern "C" AdsExecuteSQLDirect(hStmt, sql, &cursor)
→ stmt->execute_direct(sql)
│
▼ L2 session/statement.cpp → L3
Statement::execute_direct(sql)
┌──────────── L3 sql/ pipeline ───────────────────────────┐
│ Lexer → tokens │
│ Parser → AST (SelectStmt) │
│ Resolver → bind "clientes" via Catalog → Table* │
│ bind columns saldo, nombre → ColumnRef │
│ type-check predicates │
│ Planner → LogicalPlan: │
│ Sort(nombre) │
│ └─ Filter(saldo > 1000) │
│ └─ Scan(clientes) │
│ Optimizer → predicate pushdown, index selection: │
│ if index on (saldo) → IndexRangeScan │
│ else → SeqScan + Filter │
│ if index on (nombre) → drop Sort │
│ Executor → builds operator tree (iterator pipeline): │
│ SortOp │
│ └─ FilterOp │
│ └─ TableScanOp(Cursor over Table) │
└──────────────────────────────────────────────────────────┘
│
▼ Cursor handed back
HandleRegistry::register(Cursor*) → ADSHANDLE returned as hCursor
│
▼
PRG fetches rows via AdsGotoTop / AdsGetRecord / AdsSkip
→ each call drives one Executor::next() through L4 PageCache
→ AdsGetField → field decode (xBase types or ADT extended types,
depending on driver)
AdsBeginTransaction(hConn)
→ TxLog::begin(tx_id) (write BEGIN record)
→ LockMgr::tx_attach(tx_id)
AdsLockRecord(hTblA, 42)
→ LockMgr::lock_record(tx_id, A, 42) (escalates byte-range lock)
AdsUpdateRecord(hTblA, ...)
→ Table::update_record():
1. Capture before-image of pages dirtied
2. Apply change in PageCache
3. TxLog::log_update(tx_id, A, page_no, before, after)
AdsAppendRecord(hTblB, ...)
→ similar, writes a log record for table B
AdsCommitTransaction(hConn)
→ TxLog::commit(tx_id) (write COMMIT, fsync log)
→ PageCache::flush_tx_pages(tx_id)
→ LockMgr::tx_release(tx_id)
AdsRollbackTransaction(hConn) [alternative]
→ TxLog::rollback_walk(tx_id):
reverse-iterate log, restore before-images via PageCache
→ LockMgr::tx_release(tx_id)
TxRecover::run()
scan TxLog tail
for each tx without COMMIT → roll back (apply before-images)
for each tx with COMMIT but pages unflushed → roll forward
truncate log
AdsLocking(ON | OFF) selects the byte-range scheme:
| Mode | When to use | Coexistence |
|---|---|---|
| Proprietary (default) | ADS-only deployments | ADS-specific ranges, fastest |
Compatible (ADS_COMPATIBLE_LOCKING) |
Coexistence with Clipper / FoxPro / Harbour DBF* RDDs over the same files |
Standard Clipper / FoxPro ranges |
OpenADS supports both modes. The rddads frontend selects via AdsSetServerType(ADS_LOCAL_SERVER) combined with AdsLocking(ON | OFF).
Three hierarchical levels:
- File lock (exclusive) — locks the whole table.
AdsLockTable/flockopen modeEXCLUSIVE. - Record lock — shared for reads, exclusive for writes.
AdsLockRecord(recno). - Header lock — first byte of the file header, taken only during writes that mutate the header (append, pack, zap, recno recalculation).
Rules:
- Open SHARED → acquires a shared header lock over the header range.
- Append / Pack / Zap → exclusive header lock plus an exclusive file-equivalent lock (Compatible mode uses byte
LOCKOFFSET_FILE; Proprietary uses an internal ADS range). - Update record → record lock required (RDD enforcement, not OS).
- Read record → no lock required in
READ COMMITTED;READ UNCOMMITTEDskips even versioning.
DBF + NTX (Clipper scheme):
FILE LOCK : offset 1_000_000_000 size 1
RECORD LOCK : offset 1_000_000_001 + recno size 1
DBF + CDX (FoxPro scheme — descending):
FILE LOCK : offset 0x7FFFFFFE size 1
RECORD LOCK : offset 0x7FFFFFFE - recno size 1
DBF + VFP CDX (FoxPro VFP — same as CDX but offset 0x40000000-1):
FILE LOCK : offset 0x3FFFFFFE size 1
RECORD LOCK : offset 0x3FFFFFFE - recno size 1
ADT proprietary:
FILE LOCK : offset 0x80000000_00000000 size 0x10000 (64-bit)
RECORD LOCK : offset 0x80000000_00000000 + (recno << 16) size 1
ADS-specific. The ranges are derived from captures of original ACE running over an instrumented filesystem and pinned in docs/lock-ranges.md. For phase 1 they are a constant table, validated by tools/ace-trace.
class LockMgr {
public:
// OS-level byte-range — inter-process coexistence
Result<LockToken> lock_file_excl(FileHandle& fh, LockingMode mode);
Result<LockToken> lock_record (FileHandle& fh, LockingMode mode,
uint64_t recno, LockType, Timeout);
Result<> unlock (LockToken);
// Tx-scoped — lifetime tied to TxLog::tx_id
Result<> lock_for_tx (tx_id_t, FileHandle& fh,
uint64_t recno, LockType);
void release_tx (tx_id_t);
// Intra-process re-entry
bool already_held (FileHandle& fh, uint64_t recno) const;
};Notes:
- Intra-process re-entrancy. A connection that already holds a record lock does not call the OS again; only a local counter is incremented.
- Timeouts. Per-connection (
AdsSetWaitTime-equivalent), default 0 (fail fast). - Deadlock detection. Cycle search over the
tx_id → resource → tx_idgraph. On detection, the youngest transaction is aborted (ADS does not document this precisely; OpenADS picks the youngest as victim). - Errors.
AE_LOCKED(5012) andAE_LOCK_FAILED(5013), mapped from LockMgr return codes.
Inside an AdsBeginTransaction block, locks become tx-scoped: they are not released by AdsUnlockRecord, only by AdsCommitTransaction / AdsRollbackTransaction. This is required to preserve isolation guarantees.
Outside any transaction (autocommit), unlock releases immediately.
lock_mgr_test.cpp— re-entrancy, timeouts, deadlock detection.tools/ace-tracerunning real applications against original ACE → captures range logs → diffs against OpenADS.- Multi-process conformance test: two OpenADS processes plus one original-ACE process operating on the same CDX table in Compatible mode — must complete without corruption.
ARIES-lite. Page-level physiological logging. Multi-table atomicity. Nestable savepoints. Deterministic crash recovery.
A single log shared per data directory (or per Data Dictionary if an .add file exists):
<data-dir>/openads.txlog ← active log
<data-dir>/openads.txlog.<n> ← rotated archives (truncated post-checkpoint)
<data-dir>/openads.checkpoint ← latest CP record (LSN, dirty page table)
One log per DD avoids cross-DD coordination. Applications that do not use a DD but open tables in the same directory automatically share the log (detected by Connection::open).
struct LogRecord {
uint64_t lsn; // monotonic, unique
uint64_t prev_lsn; // previous record in this tx (chain for undo)
uint64_t tx_id;
uint16_t type; // BEGIN | UPDATE | INSERT | DELETE | ...
uint16_t driver_id; // adt | cdx | ntx | vfp | memo | index
uint32_t table_id; // assigned at first touch by tx
uint64_t page_no;
uint16_t before_len;
uint16_t after_len;
uint8_t payload[]; // before_image || after_image
uint32_t crc32c; // record integrity
};
Record types:
| Type | Semantics |
|---|---|
BEGIN |
tx started, includes timestamp |
UPDATE |
physiological page update (before + after image) |
INSERT |
new record append (after only; undo = decrement recno) |
DELETE |
logical delete (after only; undo = clear deleted flag) |
INDEX_OP |
B+tree mutation (page split / merge / insert / delete key) |
MEMO_ALLOC / MEMO_FREE |
blob lifecycle in .adm / .fpt / .dbt |
SAVEPOINT |
named marker; prev_lsn chains here on partial rollback |
CLR |
compensation log record (written during undo, redo-only) |
COMMIT |
tx end ok, fsync barrier |
ABORT |
tx end rollback; all CLRs already written |
CHECKPOINT_BEGIN / CHECKPOINT_END |
stable point, dirty page table snapshot |
CLRs (compensation log records) make undo idempotent: a crash during rollback simply re-executes the CLRs without duplicating work (CLRs are redo-only, never undo).
AdsCommitTransaction(hConn)
→ TxLog::append(COMMIT_PENDING) [in memory]
→ TxLog::group_commit_barrier() [waits up to 10ms or N tx]
└─ writev() batched COMMIT records
└─ fsync(log_fd)
→ tx becomes visible
10 ms / 64 tx, whichever first (configurable). Reduces fsync × N to a single fsync.
AdsCreateSavepoint(hConn, "sp1")
→ TxLog::append(SAVEPOINT name=sp1 lsn=L1)
→ conn->savepoint_stack.push("sp1", L1)
AdsRollbackTransaction80(hConn, "sp1") // partial
→ walk back log from current_lsn to L1, write CLRs for each record
→ discard savepoints above sp1
// tx still active
The classic AdsRollbackTransaction() performs a full rollback to BEGIN.
Three ARIES-lite phases:
TxRecover::run() {
// 1. ANALYSIS
scan log forward from the last CHECKPOINT_BEGIN
build active_tx_table (tx_id → last_lsn)
build dirty_page_table (page_id → first_lsn that dirtied it)
// 2. REDO
scan log forward from min(dirty_page_table.first_lsn)
for each UPDATE / INDEX_OP / INSERT / DELETE / CLR:
if page.lsn < record.lsn: // not yet applied
apply after_image to page
page.lsn = record.lsn
// 3. UNDO (loser txs only — no COMMIT seen)
for each tx in active_tx_table sorted by last_lsn DESC:
walk prev_lsn chain
for each non-CLR record:
apply before_image
write CLR(undo_next_lsn = record.prev_lsn)
write ABORT record
}
Determinism: re-entering recovery N times produces the same final state (idempotent via CLRs).
Each page (DBF, ADT, CDX, NTX, ADI, memo) carries an 8-byte LSN at the end of its page footer. Cost: 8 bytes per page. Required to skip already-applied redo work.
Compatible mode exception. DBF / CDX / NTX pages in Compatible mode do not carry an inline LSN footer (it would break byte compatibility with Clipper / FoxPro applications). Workaround: in Compatible mode TxLog keeps a separate lsn_table (overlay file .lsnmap) instead of inlining the LSN. The cost is an extra page of I/O per commit, which is acceptable.
A background thread:
- Runs every 30 s or every 1000 transactions (configurable).
- Writes
CHECKPOINT_BEGINwith a snapshot ofactive_tx_tableanddirty_page_table. - Flushes dirty pages with
lsn ≤ checkpoint_lsn. - Writes
CHECKPOINT_END. - Truncates
openads.txlog.<n>archives older than the checkpoint.
| Situation | Action |
|---|---|
| Crash with pending tx | Recovery rollback |
| CRC failure on a log record | Stop recovery at the last valid record, log a warning |
| Missing log file at startup | Assume clean shutdown (legacy ACE behaviour) |
| Page LSN > log tail LSN | Log corruption → halt, requires manual intervention |
| Out of space during commit | Force partial fsync → mark log full → reject new tx until space is freed |
tx_log_test.cpp— unit: write / read / CRC / replay.tx_recover_test.cpp— inject a crash at every LSN boundary, verify consistency.tools/tx-replay <log>— human-readable dump and dry-run replay.- Conformance: simulate
AdsBegin / Update / Crashwithkill -9mid-write, verify a consistent restart.
SQL string
▼ Lexer (DFA, ~150 keywords, xBase + ANSI SQL)
tokens
▼ Parser (recursive descent, Pratt for expressions)
AST
▼ Resolver (name binding, type check, * expansion, qualification)
Bound AST
▼ Planner (logical plan: relational algebra tree)
LogicalPlan
▼ Optimizer (rules + cost model)
PhysicalPlan
▼ Executor (Volcano: open / next / close iterators)
Result rows / row count
Hand-written DFA. Recognises:
- Case-insensitive keywords (~150).
- Identifiers:
[A-Za-z_][A-Za-z0-9_]*, plus delimited[name]and"name". - Literals: int, float, string (ANSI plus
''escape), date{^YYYY-MM-DD}, boolean.T./.F.(xBase). - Operators: ANSI plus xBase
=,==,!=,#,$(substring contains),||. - Parameters:
?positional,:namenamed.
Output: stream of Token { kind, lexeme, line, col }. Errors carry source position.
Recursive descent plus a Pratt parser for expressions (ANSI precedence plus xBase $ and ||).
EBNF grammar in docs/sql-grammar.md. Key productions:
SelectStmt := WithClause? "SELECT" SelectList FromClause? WhereClause?
GroupByClause? HavingClause? OrderByClause?
LimitClause? (CompoundOp SelectStmt)?
FromClause := "FROM" TableRef ("," TableRef)*
TableRef := QualifiedName ("AS"? Alias)?
| "(" SelectStmt ")" "AS"? Alias -- derived
| TableRef JoinType TableRef "ON" Expr -- join
JoinType := "INNER" | "LEFT" "OUTER"? | "RIGHT" "OUTER"? |
"FULL" "OUTER"? | "CROSS"
CompoundOp := "UNION" "ALL"? | "INTERSECT" | "EXCEPT"
Expr := PrimaryExpr (InfixOp Expr)* -- Pratt
PrimaryExpr := Literal | ColumnRef | FuncCall | CaseExpr |
"(" Expr ")" | SubQuery | Parameter |
"CAST" "(" Expr "AS" TypeName ")"
CaseExpr := "CASE" Expr? ("WHEN" Expr "THEN" Expr)+
("ELSE" Expr)? "END"The AST is built from sum types (std::variant) per category. Visitor pattern for passes.
- Name binding. Tables are looked up via
Catalog(DD-aware when the connection has a DD). Columns are searched in the current scope, including CTEs, derived tables, andJOIN USING. *expansion.SELECT *becomes a list ofColumnRef;t.*expands only columns oft.- Type check. Arithmetic on numeric, concatenation on character, comparison on compatible types. xBase coercions are permissive (
numeric↔stringimplicit conversion is configurable). - Aggregate detection. Marks expressions as aggregate or scalar; validates
HAVINGvsWHERE. - Subquery scope. Correlated references are annotated with
outer_scope_depth. - Errors.
AE_PARSE_ERROR(7200),AE_COLUMN_NOT_FOUND(5063), and so on.
Generates a relational algebra tree of Volcano nodes:
LogicalNode = Scan(table)
| Filter(child, pred)
| Project(child, exprs)
| Sort(child, keys)
| TopN(child, k)
| Aggregate(child, group_keys, aggs)
| Distinct(child)
| Join(left, right, kind, pred)
| Union(left, right, all?)
| CTE(name, child)
| Insert / Update / Delete (table, source, set, pred)
Canonical construction first, no optimisation.
Rule and cost passes:
| Pass | Type | Description |
|---|---|---|
predicate_pushdown |
rule | Pushes Filter below Join / Project when column refs allow. |
column_pruning |
rule | Project drops columns not used upstream. |
constant_folding |
rule | Evaluates constant expressions. |
predicate_simplify |
rule | x AND TRUE → x, NOT NOT x → x, etc. |
index_selection |
cost | Matches Filter predicates against available indexes → IndexScan. |
join_order |
cost | Selinger-style DP up to 8 tables, greedy beyond. |
join_method |
cost | NLJ vs HashJoin vs MergeJoin based on cardinality and ordering. |
sort_avoidance |
rule | If the Sort key is a prefix of the IndexScan order, drop the Sort. |
aggregate_pushdown |
rule | Pre-aggregates locally when group keys are a subset of the partition. |
topn_pushdown |
rule | LIMIT k paired with index order becomes early-stop IndexScan. |
Cost model:
- Row-count estimation uses per-table statistics in
Catalog(recno plus simple equi-width index histograms). - I/O cost is page reads (CdxIndex 8 KB, ADT 4 KB typical).
- CPU cost is
row_count × per-op constant.
Volcano iterators. Each next() returns Optional<Row>.
Physical operators:
| Operator | Description |
|---|---|
SeqScanOp |
Iterates a Cursor over a Table, reading via L4 Table::scan(). |
IndexScanOp |
Cursor follows Index::seek_range(). |
IndexLookupOp |
Nested-loop join inner side via index seek. |
FilterOp |
Predicate evaluation, drops false rows. |
ProjectOp |
Per-row expression evaluation. |
SortOp |
External merge sort, runs in <data-dir>/openads.tmp.<N>. |
TopNOp |
Min-heap with k slots. |
HashAggregateOp |
Hash-table aggregation. |
StreamAggregateOp |
Input already ordered by group keys. |
DistinctOp |
Hash set. |
NestedLoopJoinOp |
Outer × inner (with index seek when available). |
HashJoinOp |
Builds a hash on the smaller side, probes. |
MergeJoinOp |
Both sides ordered (after sort_avoidance). |
UnionOp / UnionAllOp |
Concat plus optional dedupe. |
CTEScanOp |
Reuses a materialised CTE result. |
InsertOp / UpdateOp / DeleteOp |
DML, writes via Table API and TxLog. |
External sort: runs up to mem_budget (default 64 MB), spills to tempfiles, K-way merge.
Hash join build: if the hash table exceeds the budget, falls back to Grace Hash (partition plus spill).
Registered in src/sql/func/scalar.cpp. Subset (~80 functions):
String : LEFT, RIGHT, SUBSTR, AT, RAT, LTRIM, RTRIM, ALLTRIM, PADL, PADR,
PADC, REPL, SPACE, UPPER, LOWER, STUFF, STRTRAN, LIKE, MATCH
Date : CTOD, DTOS, DTOC, STOD, DAY, MONTH, YEAR, DOW, CMONTH, CDOW,
DATE, TIME, NOW, DATEADD, DATEDIFF
Numeric: STR, VAL, INT, ROUND, MOD, ABS, MAX, MIN, EXP, LOG, SQRT,
SIGN, FLOOR, CEILING
Logic : IIF, IsNULL, COALESCE, NULLIF, CASE
Type : CAST, CONVERT, EMPTY, TYPE
Misc : RECNO, RECCOUNT, DELETED, FOUND, EOF, BOF, LASTREC,
FIELDNAME, FIELDPOS, FCOUNT
Aggregates: COUNT, SUM, AVG, MIN, MAX, STDDEV, VARIANCE, plus xBase TOTAL.
The UDF registry allows AEP plugins to add custom functions.
Advantage Extended Procedures are .dll / .so plugins exposing a C ABI. Loading and invocation:
// AEP plugin entry (mirrored from the ADS spec):
extern "C" UNSIGNED32 GetProcInfo(...);
extern "C" UNSIGNED32 InvokeProc(IInvokeContext*);
// IInvokeContext = ABI exposed by OpenADS to the plugin:
struct IInvokeContext {
UNSIGNED32 (*GetInputRowFieldCount)(...);
UNSIGNED32 (*GetInputRowField)(...);
UNSIGNED32 (*WriteOutputRow)(...);
UNSIGNED32 (*OpenTable)(...);
// ... ~30 functions
};AepHost (src/sql/aep/host.cpp):
- Resolves
dll_name + entryfrom DD propertiesADS_DD_PROC_DLL_*. - Lazy
dlopen/LoadLibraryon first call. - Marshals input / output arguments through a stable ABI.
- Sandboxing is optional; the plugin runs in-process (matching original ACE — no sandbox).
SQL invocation: EXECUTE PROCEDURE my_proc(:p1) causes the planner to emit AepCallOp.
Triggers (BEFORE / AFTER INSERT / UPDATE / DELETE) are AEP plugins fired by Table::write_record during DML.
AdsPrepareSQL(hStmt, sql)
→ parse + resolve + plan + optimize, caches the PhysicalPlan
→ parameter binding deferred
AdsExecuteSQL(hStmt)
→ binds parameters → executes → returns Cursor
Cursor types:
- Forward-only (default):
next()only. - Scrollable (
AdsCacheRecords ON): materialised in a tempfile, supportsAdsGotoRecord(n)andAdsSkip(-N).
PlanCache (LRU, key = SQL hash + schema_version) avoids re-planning on repeats.
AdsGetField(hCursor, fieldName, ...) reads the current row of the cursor. The cursor keeps a pointer to the row buffer (zero-copy on scans, copy when computed).
Mapped to existing ACE codes:
7200AE_PARSE_ERROR7201AE_INVALID_SQL_TOKEN5063AE_COLUMN_NOT_FOUND5066AE_TABLE_NOT_FOUND7041AE_TYPE_MISMATCH7042AE_DIVISION_BY_ZERO- and so on.
sql_lexer_test.cpp,sql_parser_test.cpp— grammar coverage.sql_resolver_test.cpp— name-binding edge cases.sql_planner_test.cpp— golden-file plans for canonical queries.sql_optimizer_test.cpp— verifies that rules fired (snapshot of the post-optimizer plan).sql_executor_test.cpp— end-to-end against fixtures with expected results.aep_host_test.cpp— a sample.dll/.soplugin returning fixed rows.sql_conformance/— Advantage SQL test suite derived from official ADS documentation samples.
Internal C++ code uses Result<T> (an expected-style type in src/util/result.h) so error paths are explicit and exceptions are reserved for true programming bugs (assert-equivalent contract violations).
template<class T> using Result = std::expected<T, Error>;
struct Error {
int32_t code; // ACE error code (e.g. 5012)
int32_t sub_code; // OS errno / GetLastError when applicable
std::string message; // formatted, localised
std::string context; // file, table, recno, sql snippet
};Errors propagate via early-return; no exception unwinding through L4.
The L1 ABI returns the canonical UNSIGNED32 ACE error code on every call. OpenADS reproduces the documented ranges so existing apps reading AdsGetLastError see identical numbers:
| Range | Family | Examples |
|---|---|---|
| 4000–4999 | Transport / connection | 4001 AE_NETWORK_ERROR, 4097 AE_INVALID_CONNECTION_HANDLE |
| 5000–5999 | Engine / locking / records | 5012 AE_LOCKED, 5036 AE_NO_CONNECTION, 5063 AE_COLUMN_NOT_FOUND, 5066 AE_TABLE_NOT_FOUND |
| 6000–6999 | Index / order | 6105 AE_INDEX_NOT_FOUND, 6106 AE_INDEX_CORRUPT |
| 7000–7999 | SQL / parser / type | 7041 AE_TYPE_MISMATCH, 7200 AE_PARSE_ERROR, 7201 AE_INVALID_SQL_TOKEN |
A canonical table lives in src/abi/error_codes.h, generated from the documented ACE constants. Anything not yet implemented returns 5004 AE_FUNCTION_NOT_AVAILABLE rather than a fabricated code, so apps and rddads can degrade gracefully.
ACE keeps a per-thread last-error slot. OpenADS replicates this:
thread_local Error g_last_error;
extern "C" UNSIGNED32 AdsGetLastError(UNSIGNED32* pulErr,
UNSIGNED8* pucBuf,
UNSIGNED16* pusBufLen) {
*pulErr = g_last_error.code;
copy_text(pucBuf, pusBufLen, g_last_error.message);
return AE_SUCCESS;
}Every L1 thunk updates the slot before returning. Successful calls clear it.
Default English. Optional catalogues for Spanish and Portuguese (large legacy ADS user bases). Selected via AdsSetLocalizedMessages or env OPENADS_LOCALE=es. Catalogues live in src/abi/messages_<locale>.cpp, lookup by code.
Three levels controlled by env:
| Var | Effect |
|---|---|
OPENADS_LOG=info |
Connection open / close, table open, SQL executed (truncated) |
OPENADS_LOG=debug |
Plus index seeks, locks acquired, tx boundaries |
OPENADS_LOG=trace |
Every L1 entry / exit with arguments and return code |
OPENADS_LOG_FILE=<path> |
Redirect log; default stderr |
tools/ace-trace is a separate shim that intercepts every Ads* call and writes a structured trace; useful for diffing against original ACE behaviour.
| Boundary | Strategy |
|---|---|
| L5 (OS) errors | Map errno / GetLastError into Error.sub_code, surface a 4xxx or 5xxx ACE code |
| L4 corruption (CRC / LSN mismatch) | Halt access to the affected file, return 5103 AE_TABLE_CORRUPTED, log details |
| L3 SQL errors | Return 7xxx range, no abort of the connection |
| L2 invalid handle | 4097 AE_INVALID_CONNECTION_HANDLE, no crash |
| L1 panic (assert) | Last-resort: log and abort() only on contract violations during debug builds; release builds convert to 5000 AE_INTERNAL_ERROR |
| Layer | Tools | Coverage target |
|---|---|---|
| Unit | doctest, run on every commit | ≥ 85 % per module (engine, drivers, sql, lock, tx) |
| Integration (in-process) | doctest, real files | Full driver matrix (ADT / CDX / NTX / VFP) × open / write / index / memo / tx |
| ABI conformance | C harness invoking L1 entry points | All ~230 ACE entry points exercised at least once |
| Harbour smoke | harbour_smoke.prg linked against rddads and OpenADS DLL |
tests/datad.prg, tests/manage.prg from c:\harbour\contrib\rddads\tests\ plus custom xBase scenarios |
| Byte compatibility | tools/adt-dump, tools/cdx-dump diff vs ACE-produced fixtures |
All write paths produce byte-identical output |
| Multi-process | Two OpenADS plus optional original ACE on shared files | No corruption, lock semantics match |
| Fuzzing | libFuzzer over lexer / parser / log replay / driver readers | Zero crashes / UB after N hours |
| Benchmarks | google-benchmark micro-suite | No regression vs previous tag |
| Recovery | Crash-injection harness (kill -9 between LSN boundaries) |
Recovery converges deterministically |
GitHub Actions:
- Compilers: MSVC 2022 (x86 / x64), GCC 11+, Clang 14+, MinGW-w64.
- OS: Windows 2022, Ubuntu 22.04, macOS 13, FreeBSD 14 (via cross / VM).
- Sanitisers: ASan, UBSan, TSan on Linux Clang job.
- Nightly: full Harbour rddads test suite plus byte-compat job against pinned ACE-produced fixtures.
| Milestone | Deliverable |
|---|---|
| M0 — skeleton | CMake + L5 platform + util + doctest harness. Builds on Win / Linux / macOS. |
| M1 — DBF read | dbf_common + CDX driver read-only, no index. AdsConnect60 / AdsOpenTable / AdsGotoTop / AdsSkip / AdsGetField work over a CDX-typed DBF. |
| M2 — DBF write + lock | Append / update / delete + LockMgr Compatible mode. NTX driver. Single-process integrity tests. |
| M3 — indexes | CDX read / write, NTX read / write, ADI scaffolding. Seek, scope, AOF basics. |
| M4 — ADT + memo | ADT driver full, .adm / .fpt / .dbt memo stores. VFP driver. Encryption AES-128 / 256. |
| M5 — TPS | TxLog WAL + recovery, savepoints, multi-table atomicity, group commit. Compatible-mode .lsnmap overlay. |
| M6 — DD | .add reader / writer, users / groups / RI / views / procs metadata, AdsConnect60 to a DD. |
| M7 — SQL | Lexer / parser / resolver / planner / optimizer / executor. xBase UDFs. AEP host. Triggers. |
| M8 — Conformance | Full Harbour tests/datad.prg and tests/manage.prg green. Byte-compat job green. Multi-process green. First tagged release 0.1.0. |
Phase 2 (post-1.0): TCP server reusing L2-L5, wire-protocol design, replication, AIS / HTTP gateways. Out of scope for this document.
Phase 1 is broken into nine independently shippable milestones (M0–M8). Each milestone gets its own implementation plan under docs/superpowers/plans/, written in TDD bite-sized form so any contributor can pick it up.
| Milestone | Plan | Status |
|---|---|---|
| M0 — Skeleton | 2026-05-03-openads-m0-skeleton.md |
Done. CMake project, L5 platform layer (file / lock / mmap / path / time / thread), util/Result<T> / Span<T> / Log, doctest harness (27 cases / 77 assertions), GitHub Actions matrix (Windows / Linux / macOS). |
| M1 — DBF read (CDX) | 2026-05-03-openads-m1-dbf-read.md |
Done. Read-only DBF (ADS_CDX typed) via AdsConnect60 / AdsOpenTable / AdsGotoTop / AdsSkip / AdsGetField and friends. No memo (M4), no index (M3), no write (M2). |
| M2 — DBF write + LockMgr | 2026-05-03-openads-m2-dbf-write-lock.md |
Done. Append / update / delete on CDX- and NTX-typed DBFs, LockMgr Compatible-mode byte ranges (NTX 1_000_000_000, CDX 0x7FFFFFFE - recno), single-process integrity tests. No pack / zap (M3), no memo (M4), no TPS (M5). |
| M3 — Indexes | 2026-05-03-openads-m3-indexes.md |
Partial — round-trips OpenADS-produced files only. NTX header + leaf read+write+create works against indexes that OpenADS itself wrote. Multi-leaf NTX split, branch descent, and FoxPro CDX byte-compat are blocked by issues tracked in docs/known-issues.md. Fixes land in M3.6. Order + Scope on Table, 15 ACE entry points, AOF/Pack/Zap stubs are all in place. |
| M3.5 — CDX index | (extends M3) | Partial — non-standard byte layout. A working compact-leaf encoder/decoder using a hardcoded 24/8/8-bit split. Round-trips OpenADS-produced .cdx files; does NOT match FoxPro byte layout (bit widths must derive from keylen, tag directory must use the compound structure tag). M3.6 replaces this with a real FoxPro-equivalent encoder driven by Harbour hb_cdxPageLeafInitSpace. See docs/known-issues.md items 1-3. |
| M3.6 — Real index byte-compat | (in flight) | Partial. Done: CDX leaf encoder now uses Harbour-equivalent compute_layout() modelled on hb_cdxPageLeafInitSpace (bBits derived from key length; for keylen=4 the result is 18/3/3 bits packed in 3 bytes). Tightened AdsOpenIndex / AdsCreateIndex lifecycle (prior bindings cleared before set_order). AdsCreateIndex now skips deleted records. NTX unique and descending flags round-trip through create/reopen. Pending: CDX compound structure-tag directory, CDX big-endian branch descent at the right offset, NTX multi-level split, soft-seek past-end fix. See docs/known-issues.md. |
| M4 — ADT + memo + VFP + AES | 2026-05-03-openads-m4-adt-memo-vfp-aes.md |
Partial. Done: AES-128 / AES-256 ECB via vendored tiny-AES-c, validated against FIPS-197 Appendix B (AES-128) and NIST SP 800-38A F.1.5/F.1.6 (AES-256) test vectors. DBT memo real (dBase III/Clipper, 512-byte blocks, 0x1A 0x1A terminator, multi-block walk). FPT memo real (FoxPro/VFP, big-endian header, 8-byte block headers, configurable block size 64/512). Table::attach_memo routes M-type field reads/writes through the memo store; Connection::open_table auto-attaches .dbt / .fpt siblings when M-fields are present. ABI thunks: AdsGetMemoLength, AdsGetMemoDataType, AdsBinaryToFile, AdsFileToBinary are live. AdsGetLastAutoinc returns 0 stub. Encryption ABI (AdsEnableEncryption / AdsEncryptTable / AdsEncryptRecord / etc.) returns AE_FUNCTION_NOT_AVAILABLE until a clean-room specification of the ADS record-level encryption layout is available (the AES primitive itself is ready). Pending: ADT format (proprietary; clean-room spec required), VFP driver autoinc/NULL bitmap extensions, ADM memo (same gating as ADT), AES record-encryption boundary on Table. |
| M5 — TPS / WAL | TBD | Tx + WAL + crash recovery + savepoints landed. ABI: AdsBeginTransaction, AdsCommitTransaction, AdsRollbackTransaction, AdsInTransaction, AdsCreateSavepoint, AdsRollbackTransaction80. Each tx event writes a record to openads.txlog in the data dir (BEGIN / UPDATE / COMMIT / ABORT, CRC-32C protected). UPDATE records carry the table relative path + before/after images. Connection::open runs recovery: any tx without COMMIT or ABORT is replayed by writing back before-images and appending ABORT, then the log is truncated. Savepoints are an in-memory ordered-op log layered on top of the before-image map; AdsRollbackTransaction80 with a savepoint name does a partial rollback, with nullptr it falls back to a full rollback. Smoke tests cover crash mid-tx + recovery and partial rollback through a savepoint. Pending: group commit (batched fsync), page-LSN tracking with .lsnmap overlay for Compatible mode, savepoint persistence in WAL. |
| M6 — Data Dictionary | TBD | Alias resolution landed (OpenADS-native DD format). engine::DataDict is a UTF-8 text file (# OpenADS Data Dictionary v0 + TABLE alias=path lines) created in the data dir. Connection::open accepts either a directory path or a .add path; in the latter case it loads the DD and auto-resolves aliases passed to AdsOpenTable. ABI: AdsDDCreate, AdsDDAddTable, AdsDDRemoveTable. Smoke covers create -> add -> open-by-alias -> reopen. Pending: proprietary .add binary format (deferred until a clean-room spec is available), users / groups / permissions, RI rules, views, stored procedures, validation expressions, default values. |
| M7 — SQL engine | TBD | M7.5 landed (SELECT * + AND-joined WHERE with all six operators). engine::sql::parse_select parses SELECT * FROM <table> [WHERE <cmp> [AND <cmp>]*] where each <cmp> is <col> <op> '<literal>' and <op> is one of =, !=, <>, <, >, <=, >=. Table gained a RowPredicate slot; goto_top / skip automatically advance past non-matching records when a filter is attached. ABI: AdsExecuteSQLDirect lowers each WHERE term into (field_index, op, literal) and the closure short-circuits the AND. Projection lists, OR / NOT / parens, numeric literals, ORDER BY, joins, aggregates, subqueries, and UDFs return AE_PARSE_ERROR. Pending: full Advantage SQL grammar (lexer + AST + planner + executor), xBase UDFs (LEFT, SUBSTR, CTOD, ...), AEP host for stored procedures, triggers, INSERT / UPDATE / DELETE / CREATE TABLE. |
| M8 — Conformance + 0.1.0 | TBD | Full Harbour tests/datad.prg and tests/manage.prg green, byte-compat job green, multi-process scenario green, first tagged release. |
- 135 doctest cases / 1820 assertions passing on Windows / MSVC 2022 Release.
- ~80 ACE entry points wired (read / write / lock / index / scope / memo / encryption / autoinc / transaction / savepoint / data dictionary / SQL).
- Persistent WAL with crash recovery is byte-identical for OpenADS-produced files.
- Live tags:
m0-done,m1-done,m2-done,m3-done,m3.5-done,m3.6-partial,m3.7-partial,m3.7-closed,m3.8-partial,m3.9-partial,m3.10-partial,m4-partial,m5-partial,m5.1-partial,m5.2-partial,m5.3-partial,m5.4-partial,m5.5-partial,m6-partial,m7.1-partial,m7.2-partial,m7.3-partial,m7.4-partial,m7.5-partial,m8.0-partial,m8.1-partial,m8.2-done,m8.3-done,m8.4-done,m8.5-done,m8.6-done,m8.7-partial,m8.8-done,m8.9-done,m8.10-done,m8.11-done,0.1.0-rc1,0.1.0. - Drop-in DLL:
ace64.dll(Win x64) andace32.dll(Win x86) build from theopenads_aceSHARED target, exporting 226Ads*entry points plus 6 legacy MSVC2013-era CRT shims (_dclass,_dsign,_wfsopen,_getch,_kbhit,_eof) referenced by Harbour's prebuiltmsvc64libs. 80 of theAds*are real implementations (M0–M7); the rest are M8.1 stubs that returnAE_FUNCTION_NOT_AVAILABLE(5004). - End-to-end Harbour validation (M8.3–M8.11):
tests/harbour_smoke/smoke.prgexercises the full read + write + index + multi-tag focus + transactions + memo path throughrddads.liband OpenADS'ace64.dll. Seetests/harbour_smoke/README.mdfor captured outputs.
- Brainstorm the milestone briefly against the spec above to surface anything that changed since the original design was written.
- Write its implementation plan into
docs/superpowers/plans/YYYY-MM-DD-openads-mN-<topic>.mdusing the same TDD bite-sized template as M0. - Execute the plan task by task. Each task is
red → green → commitand lands one focused change. - When the milestone is done, mark it green in the table above, push, and tag the head commit
mN-donefor traceability.
Execute M0 using the saved plan. Two execution paths:
- Subagent-driven (recommended). Dispatch a fresh subagent per task with two-stage review between tasks. See
superpowers:subagent-driven-development. - Inline. Walk the plan in the current session with checkpoints. See
superpowers:executing-plans.
git clone https://github.com/FiveTechSoft/OpenADS.git
cd OpenADS
cmake --preset default
cmake --build build/default
ctest --preset default --output-on-failure
Other presets: debug, msvc-x64, ninja-clang — see CMakePresets.json.
Apache License 2.0. See LICENSE for the full text and
NOTICE for attribution metadata.
Bundled third-party components keep their own licenses:
third_party/tinyaes/— tiny-AES-c by kokke, released into the public domain under the Unlicense (https://unlicense.org/).third_party/doctest/— doctest by onqtam, released under the MIT License.third_party/utf8.h/— utf8.h by sheredom, released under the Unlicense.