Added support for first sync

marcobambini · marcobambini · commit 46339f77c8a5 · 2026-04-29T08:35:32.000+02:00
diff --git a/docs/internal/schema-migrations.md b/docs/internal/schema-migrations.md
@@ -46,25 +46,56 @@ SELECT cloudsync_alter_apply();
 ```
 
 `cloudsync_alter_apply()` applies the queued migration locally and stores the
-generated payload in `cloudsync_pending_migration`. After that, an authorized
-client uploads it with:
+generated payload in `cloudsync_pending_migration`. On the next
+`cloudsync_network_sync()` or `cloudsync_network_send_changes()`, the network
+layer uploads every pending schema migration before it sends row data:
 
 ```sql
-SELECT cloudsync_network_migration_upload();
 SELECT cloudsync_network_sync();
 ```
 
-The zero-argument upload form uploads the next pending local migration and marks
-it uploaded only after the backend returns valid JSON. The one-argument form is
-still available for custom backends or tests:
+The zero-argument upload form is still available when an application wants to
+publish schema separately from row data. It uploads the next pending local
+migration and marks it uploaded only after the backend returns an accepted
+response. The one-argument form is available for custom backends or tests:
 
 ```sql
 SELECT cloudsync_network_migration_upload(:json_payload);
 ```
 
-While a local migration is pending upload, `cloudsync_network_send_changes()`
-returns an error instead of sending row changes. This prevents data produced
-with a new local schema from reaching a server that has not accepted that schema.
+If a pending migration upload fails, row changes are not sent. This prevents
+data produced with a new local schema from reaching a server that has not
+accepted that schema.
+
+## Initial Schema Sync
+
+The first version of a database is distributed with the same migration protocol
+used for later schema changes. There is no separate bootstrap format.
+
+Client-to-cloud first sync:
+
+1. The client must have `database_id` configured and must sync with an API key
+   that is allowed to initiate schema migrations.
+2. The app defines the first schema with `cloudsync_alter_create_table()`,
+   `cloudsync_alter_add_column()`, `cloudsync_alter_augment_table()`, and any
+   optional commands such as `cloudsync_alter_set_block_lww()`.
+3. `cloudsync_alter_apply()` creates the local tables, records the schema epoch,
+   and stores a pending upload in `cloudsync_pending_migration`.
+4. The app may insert initial data.
+5. `cloudsync_network_sync()` uploads the pending schema migration first. The
+   backend applies it to the cloud database, records it in the per-`database_id`
+   schema log, and only then can row data be uploaded.
+
+Cloud-to-client first sync:
+
+1. The backend applies and records the initial migration for the `database_id`.
+2. A new SQLite client only needs `database_id` and a valid API key.
+3. `cloudsync_network_sync()` detects that the local database has no augmented
+   tables, calls the schema check endpoint, applies the first migration locally,
+   and then continues with normal row download.
+
+An empty client that has no local schema and no server-side migration simply
+returns an empty sync result.
 
 ## Declarative API
 
@@ -259,7 +290,7 @@ Client-originated migration:
 1. Application queues operations with `cloudsync_alter_*`.
 2. Application calls `cloudsync_alter_apply()`.
 3. Extension applies the migration locally and writes `cloudsync_pending_migration`.
-4. Application or `cloudsync_network_sync()` uploads the pending migration.
+4. `cloudsync_network_sync()` uploads the pending migration before row changes; applications may call `cloudsync_network_migration_upload()` explicitly when they want a separate schema publish step.
 5. Backend authorizes the API key, applies the payload to the cloud database,
    records the migration, and returns success.
 6. Client sends row changes after the pending migration is uploaded.
@@ -286,7 +317,7 @@ Empty client first sync:
 - A migration id is idempotent through `cloudsync_migrations`.
 - Explicit hash guards are enforced when present.
 - Raw SQL runs inside the same savepoint as portable operations.
-- Row changes are not uploaded while `cloudsync_pending_migration` contains an unuploaded migration.
+- Row changes are uploaded only after all pending local schema migrations have been accepted by the backend.
 - V2 migrations should be blocked by the backend when stale/offline clients may still upload incompatible old-epoch payloads, unless the backend has an explicit rejection or translation policy.
 
 ## Tests
diff --git a/examples/schema-migrations/README.md b/examples/schema-migrations/README.md
@@ -21,16 +21,27 @@ SELECT cloudsync_alter_add_column('notes', 'updated_at', 'timestamp', false, '19
 SELECT cloudsync_alter_augment_table('notes', 'CLS', 1);
 SELECT cloudsync_alter_set_block_lww('notes', 'body', char(10));
 SELECT cloudsync_alter_apply();
-SELECT cloudsync_network_migration_upload();
 SELECT cloudsync_network_sync();
 ```
 
-`cloudsync_alter_preview()` can be used before `cloudsync_alter_apply()` to inspect the generated payload. After apply, the payload is saved in `cloudsync_pending_migration`; `cloudsync_network_migration_upload()` uploads the next pending migration and marks it uploaded on success.
+`cloudsync_alter_preview()` can be used before `cloudsync_alter_apply()` to inspect the generated payload. After apply, the payload is saved in `cloudsync_pending_migration`; `cloudsync_network_sync()` uploads pending migrations before it sends row changes. `cloudsync_network_migration_upload()` is still available when schema should be published separately from data.
 
 The backend should authorize the API key, apply the payload to the cloud database, append it to the schema migration log for the `database_id`, and distribute it to other clients through `schema/check` or `schema/download`.
 
 `client-to-server.sql` contains the same flow as an executable SQLite example. `client-to-server-v1.json` is the manual JSON equivalent for backend tests or custom tooling; application code should normally let `cloudsync_alter_apply()` generate that payload.
 
+## Initial Database Sync
+
+The first version of a database uses the same flow. A schema-capable client
+queues the first `createTable` migration, calls `cloudsync_alter_apply()`,
+optionally inserts initial rows, and then calls `cloudsync_network_sync()`.
+The schema upload is accepted by the backend before any row payload is sent.
+
+For cloud-to-client bootstrap, the backend records the initial migration first.
+A new SQLite client with only `database_id` and an API key calls
+`cloudsync_network_sync()`; the client downloads the first schema migration,
+creates/augments the tables, and then downloads data normally.
+
 ## Server-Originated V2 Migration
 
 The backend applies and records a payload such as `server-to-client-v2.json`, then a client can update itself before receiving data:
diff --git a/examples/schema-migrations/client-to-server.sql b/examples/schema-migrations/client-to-server.sql
@@ -2,7 +2,7 @@
 --
 -- Run this on an authorized SQLite client. The extension generates the JSON
 -- migration payload, applies it locally, stores it in cloudsync_pending_migration,
--- and uploads it with cloudsync_network_migration_upload().
+-- and uploads it automatically before row data on the next cloudsync_network_sync().
 
 SELECT cloudsync_alter_create_table('notes');
 SELECT cloudsync_alter_add_column('notes', 'id', 'text', false);
@@ -17,5 +17,11 @@ SELECT cloudsync_alter_set_block_lww('notes', 'body', char(10));
 SELECT cloudsync_alter_preview();
 
 SELECT cloudsync_alter_apply();
-SELECT cloudsync_network_migration_upload();
+
+-- Optional initial data can be inserted here. cloudsync_network_sync() uploads
+-- the pending schema migration first and sends row data only after the backend
+-- accepts that schema.
+INSERT INTO notes (id, title, body, updated_at)
+VALUES (cloudsync_uuid(), 'First note', 'Created before first sync', '1970-01-01T00:00:00Z');
+
 SELECT cloudsync_network_sync();
diff --git a/src/network/network.c b/src/network/network.c
@@ -77,6 +77,7 @@ typedef struct {
 } network_read_data;
 
 static int cloudsync_network_migration_check_internal(sqlite3_context *context, char **result_json, char **err_out);
+static int cloudsync_network_migration_upload_next_pending(sqlite3_context *context, char **result_json, char **err_out);
 static bool network_migration_apply_error(const char *message);
 static void network_result_to_sqlite_error(sqlite3_context *context, NETWORK_RESULT res, const char *default_error_message);
 network_data *cloudsync_network_data(sqlite3_context *context);
@@ -979,71 +980,124 @@ void cloudsync_network_migration_download(sqlite3_context *context, int argc, sq
 }
 
 void cloudsync_network_migration_upload(sqlite3_context *context, int argc, sqlite3_value **argv) {
-    cloudsync_context *data = (cloudsync_context *)sqlite3_user_data(context);
     network_data *netdata = cloudsync_network_data(context);
     if (!netdata || !netdata->schema_upload_endpoint) {
         sqlite3_result_error(context, "Unable to retrieve CloudSync schema migration upload endpoint.", -1);
         return;
     }
 
-    bool from_pending = false;
-    char *pending_id = NULL;
-    char *pending_payload = NULL;
     const char *payload = NULL;
 
     if (argc == 0) {
-        pending_id = cloudsync_pending_migration_next_id(data);
-        if (!pending_id) {
-            sqlite3_result_error(context, "No pending schema migration to upload.", -1);
+        char *result = NULL;
+        int rc = cloudsync_network_migration_upload_next_pending(context, &result, NULL);
+        if (rc == SQLITE_OK && result) sqlite3_result_text(context, result, -1, cloudsync_memory_free);
+        else if (rc == SQLITE_OK) sqlite3_result_text(context, "{\"status\":\"uploaded\"}", -1, SQLITE_TRANSIENT);
+    } else {
+        payload = (const char *)sqlite3_value_text(argv[0]);
+
+        if (!payload || payload[0] == '\0') {
+            sqlite3_result_error(context, "cloudsync_network_migration_upload expects a JSON text payload.", -1);
             return;
         }
-        pending_payload = cloudsync_pending_migration_payload(data, pending_id);
-        if (!pending_payload) {
-            sqlite3_result_error(context, "Unable to load pending schema migration payload.", -1);
-            cloudsync_memory_free(pending_id);
+        if (!json_is_valid_root_object(payload, strlen(payload))) {
+            sqlite3_result_error(context, "cloudsync_network_migration_upload expects a valid JSON object payload.", -1);
             return;
         }
-        payload = pending_payload;
-        from_pending = true;
-    } else {
-        payload = (const char *)sqlite3_value_text(argv[0]);
+
+        NETWORK_RESULT res = network_receive_buffer(netdata, netdata->schema_upload_endpoint, netdata->authentication, true, true, (char *)payload, CLOUDSYNC_HEADER_SQLITECLOUD);
+        if (network_validate_json_response(context, &res, "CloudSync schema migration upload endpoint", NULL)) {
+            char *upload_error = network_migration_upload_error_message(&res);
+            if (res.code == CLOUDSYNC_NETWORK_ERROR) {
+                network_set_sqlite_result(context, &res);
+            } else if (upload_error) {
+                sqlite3_result_error(context, upload_error, -1);
+            } else {
+                network_set_sqlite_result(context, &res);
+            }
+            if (upload_error) cloudsync_memory_free(upload_error);
+        }
+        network_result_cleanup(&res);
     }
+}
 
-    if (!payload || payload[0] == '\0') {
-        sqlite3_result_error(context, "cloudsync_network_migration_upload expects a JSON text payload.", -1);
-        if (pending_id) cloudsync_memory_free(pending_id);
-        if (pending_payload) cloudsync_memory_free(pending_payload);
-        return;
+static int cloudsync_network_migration_upload_next_pending(sqlite3_context *context, char **result_json, char **err_out) {
+    if (result_json) *result_json = NULL;
+    if (err_out) *err_out = NULL;
+
+    cloudsync_context *data = (cloudsync_context *)sqlite3_user_data(context);
+    network_data *netdata = cloudsync_network_data(context);
+    if (!netdata || !netdata->schema_upload_endpoint) {
+        const char *message = "Unable to retrieve CloudSync schema migration upload endpoint.";
+        if (err_out) *err_out = cloudsync_string_dup(message);
+        else sqlite3_result_error(context, message, -1);
+        return SQLITE_ERROR;
     }
-    if (!json_is_valid_root_object(payload, strlen(payload))) {
-        sqlite3_result_error(context, "cloudsync_network_migration_upload expects a valid JSON object payload.", -1);
-        if (pending_id) cloudsync_memory_free(pending_id);
-        if (pending_payload) cloudsync_memory_free(pending_payload);
-        return;
+
+    char *pending_id = cloudsync_pending_migration_next_id(data);
+    if (!pending_id) {
+        const char *message = "No pending schema migration to upload.";
+        if (err_out) *err_out = cloudsync_string_dup(message);
+        else sqlite3_result_error(context, message, -1);
+        return SQLITE_ERROR;
     }
 
-    NETWORK_RESULT res = network_receive_buffer(netdata, netdata->schema_upload_endpoint, netdata->authentication, true, true, (char *)payload, CLOUDSYNC_HEADER_SQLITECLOUD);
-    if (network_validate_json_response(context, &res, "CloudSync schema migration upload endpoint", NULL)) {
-        int rc = DBRES_OK;
-        char *upload_error = network_migration_upload_error_message(&res);
-        if (res.code == CLOUDSYNC_NETWORK_ERROR) {
-            network_set_sqlite_result(context, &res);
-        } else if (upload_error) {
-            sqlite3_result_error(context, upload_error, -1);
+    char *pending_payload = cloudsync_pending_migration_payload(data, pending_id);
+    if (!pending_payload) {
+        const char *message = "Unable to load pending schema migration payload.";
+        if (err_out) *err_out = cloudsync_string_dup(message);
+        else sqlite3_result_error(context, message, -1);
+        cloudsync_memory_free(pending_id);
+        return SQLITE_ERROR;
+    }
+
+    int rc = SQLITE_ERROR;
+    if (!json_is_valid_root_object(pending_payload, strlen(pending_payload))) {
+        const char *message = "cloudsync_network_migration_upload expects a valid JSON object payload.";
+        if (err_out) *err_out = cloudsync_string_dup(message);
+        else sqlite3_result_error(context, message, -1);
+        goto cleanup;
+    }
+
+    NETWORK_RESULT res = network_receive_buffer(netdata, netdata->schema_upload_endpoint, netdata->authentication, true, true, pending_payload, CLOUDSYNC_HEADER_SQLITECLOUD);
+    if (!network_validate_json_response(context, &res, "CloudSync schema migration upload endpoint", err_out)) {
+        network_result_cleanup(&res);
+        goto cleanup;
+    }
+
+    char *upload_error = network_migration_upload_error_message(&res);
+    if (res.code == CLOUDSYNC_NETWORK_ERROR) {
+        if (err_out) *err_out = res.buffer ? cloudsync_string_dup(res.buffer) : cloudsync_string_dup("CloudSync schema migration upload failed.");
+        else network_set_sqlite_result(context, &res);
+    } else if (upload_error) {
+        if (err_out) *err_out = cloudsync_string_dup(upload_error);
+        else sqlite3_result_error(context, upload_error, -1);
+    } else {
+        int db_rc = cloudsync_pending_migration_mark_uploaded(data, pending_id);
+        if (db_rc == DBRES_OK) {
+            if (result_json && res.code == CLOUDSYNC_NETWORK_BUFFER && res.buffer) {
+                *result_json = cloudsync_memory_zeroalloc(res.blen + 1);
+                if (*result_json) memcpy(*result_json, res.buffer, res.blen);
+                else rc = SQLITE_NOMEM;
+            }
+            if (rc != SQLITE_NOMEM) rc = SQLITE_OK;
+            else if (!err_out) sqlite3_result_error_code(context, SQLITE_NOMEM);
+            if (!result_json && !err_out) network_set_sqlite_result(context, &res);
         } else {
-            if (from_pending) rc = cloudsync_pending_migration_mark_uploaded(data, pending_id);
-            if (rc == DBRES_OK) {
-                network_set_sqlite_result(context, &res);
-            } else {
+            if (err_out) *err_out = cloudsync_string_dup(cloudsync_errmsg(data));
+            else {
                 sqlite3_result_error(context, cloudsync_errmsg(data), -1);
-                sqlite3_result_error_code(context, rc);
+                sqlite3_result_error_code(context, db_rc);
             }
         }
-        if (upload_error) cloudsync_memory_free(upload_error);
     }
+    if (upload_error) cloudsync_memory_free(upload_error);
     network_result_cleanup(&res);
+
+cleanup:
     if (pending_id) cloudsync_memory_free(pending_id);
     if (pending_payload) cloudsync_memory_free(pending_payload);
+    return rc;
 }
 
 int network_extract_query_param (const char *query, const char *key, char *output, size_t output_size) {
@@ -1417,9 +1471,20 @@ int cloudsync_network_send_changes_internal (sqlite3_context *context, int argc,
         }
     }
 
-    if (cloudsync_pending_migration_count(data) > 0) {
-        sqlite3_result_error(context, "A pending schema migration must be uploaded before sending row changes.", -1);
-        return SQLITE_ERROR;
+    while (cloudsync_pending_migration_count(data) > 0) {
+        char *migration_result = NULL;
+        char *migration_err = NULL;
+        int mrc = cloudsync_network_migration_upload_next_pending(context, &migration_result, &migration_err);
+        if (migration_result) cloudsync_memory_free(migration_result);
+        if (migration_err) {
+            sqlite3_result_error(context, migration_err, -1);
+            cloudsync_memory_free(migration_err);
+            return SQLITE_ERROR;
+        }
+        if (mrc != SQLITE_OK) {
+            if (mrc == SQLITE_NOMEM) sqlite3_result_error_code(context, SQLITE_NOMEM);
+            return mrc;
+        }
     }
     
     // retrieve payload
diff --git a/test/integration.c b/test/integration.c
@@ -894,7 +894,9 @@ static int test_mock_migration_upload_error_keeps_pending(void) {
     if (rc != SQLITE_OK) goto cleanup;
     rc = db_exec(db, "INSERT INTO upload_notes (id) VALUES ('u1');");
     if (rc != SQLITE_OK) goto cleanup;
-    rc = expect_sql_error_contains(db, "SELECT cloudsync_network_send_changes();", "pending schema migration");
+    rc = expect_sql_error_contains(db, "SELECT cloudsync_network_send_changes();", "missing schema api key");
+    if (rc != SQLITE_OK) goto cleanup;
+    rc = db_expect_int(db, "SELECT count(*) FROM cloudsync_pending_migration WHERE uploaded_at IS NULL;", 1);
 
 cleanup:
     if (db) {
@@ -932,7 +934,9 @@ static int test_mock_migration_upload_missing_status_keeps_pending(void) {
     if (rc != SQLITE_OK) goto cleanup;
     rc = db_exec(db, "INSERT INTO upload_missing_status_notes (id) VALUES ('u1');");
     if (rc != SQLITE_OK) goto cleanup;
-    rc = expect_sql_error_contains(db, "SELECT cloudsync_network_send_changes();", "pending schema migration");
+    rc = expect_sql_error_contains(db, "SELECT cloudsync_network_send_changes();", "accepted status");
+    if (rc != SQLITE_OK) goto cleanup;
+    rc = db_expect_int(db, "SELECT count(*) FROM cloudsync_pending_migration WHERE uploaded_at IS NULL;", 1);
 
 cleanup:
     if (db) {
diff --git a/test/postgresql/52_schema_migrations.sql b/test/postgresql/52_schema_migrations.sql
diff --git a/test/unit.c b/test/unit.c
