FOC Devnet scenarios run report (frontier)

[FilOzzy]

The frontier scenarios run failed ❌.
See the workflow run for details.

Scenarios Tests

Description	Data
Type	frontier
Date	24-May-2026 07:10:26 GMT +0
Status	PASS ✅:4, FAIL 🟥:1, Total:5
CI run	https://github.com/FilOzone/foc-devnet/actions/runs/26353992164

Versions info

foc-devnet version: not available

Tests summary

✅ test_containers - PASS (0s)

[ OK ] container foc-20260524T0655_IckyBoo-lotus is running
[ OK ] container foc-20260524T0655_IckyBoo-lotus-miner is running
[ OK ] container foc-20260524T0655_IckyBoo-curio-1 is running
[ OK ] container foc-20260524T0655_IckyBoo-curio-2 is running

✅ test_basic_balances - PASS (0s)

[ OK ] cast is installed
[ OK ] at least one user exists
[ OK ] USER_1 FIL balance > 0
[ OK ] USER_1 USDFC balance > 0
[ OK ] USER_2 FIL balance > 0
[ OK ] USER_2 USDFC balance > 0
[ OK ] USER_3 FIL balance > 0
[ OK ] USER_3 USDFC balance > 0

✅ test_storage_e2e - PASS (97s)

[ OK ] git is installed
[ OK ] node is installed
[ OK ] pnpm is installed
[ OK ] clone synapse-sdk
[ OK ] checkout master HEAD
[INFO] synapse-sdk commit: e699e604f2c765100144efeed326181db17ee706
[ OK ] pnpm install
[ OK ] pnpm build
[INFO] Creating random file (20971520 bytes)
[ OK ] random_file created with exact size 20971520 bytes
[INFO] Running Synapse SDK storage e2e script against devnet
[INFO] Loading devnet info from: /home/runner/.foc-devnet/state/latest/devnet-info.json
Devnet run: 20260524T0655_IckyBoo
Using user: USER_1 (0x47cc9101fd026fc112d7fadf6b3c9df5be7d4a8c)
SPs available: 2
=== Synapse SDK Storage E2E Example ===

Reading file...
  random_file (20 MB)

--- Initializing Synapse SDK ---
Network: FOC DevNet
RPC URL: http://localhost:5701/rpc/v1
Wallet address: 0x47CC9101fD026fC112D7FADf6B3c9DF5bE7D4A8c
Synapse instance created

--- Checking Balances ---
FIL balance: 999.9999693014 FIL
USDFC balance: 999.999969 USDFC

--- Preparing Account ---
Estimated costs:
  Per epoch (30s): 0.000001 USDFC
  Per month: 0.048000 USDFC
  Deposit needed: 0.000000 USDFC
  Ready: true
Account already ready

--- Uploading ---
Uploading random_file (20 MB) via stream...

  Selected SP 1 (0x446339aE7245e3cd1FeD701b685C196C69af695e)
  Selected SP 2 (0xC60bc6fb60901c2d28ba6e4F8AD4915Bd2106497)
  Upload progress: 10 MB (50.0%)
  Upload progress: 20 MB (100.0%)
  Stored on SP 1: bafkzcibfqcapabiuuwbw4j3fajxirwml7a4bkrhop6gipo3u47drjeusofmnyitg3yaq
  Pulling to SP 2: bafkzcibfqcapabiuuwbw4j3fajxirwml7a4bkrhop6gipo3u47drjeusofmnyitg3yaq (pending)
  Pulling to SP 2: bafkzcibfqcapabiuuwbw4j3fajxirwml7a4bkrhop6gipo3u47drjeusofmnyitg3yaq (pending)
  Pulling to SP 2: bafkzcibfqcapabiuuwbw4j3fajxirwml7a4bkrhop6gipo3u47drjeusofmnyitg3yaq (inProgress)
  Pulling to SP 2: bafkzcibfqcapabiuuwbw4j3fajxirwml7a4bkrhop6gipo3u47drjeusofmnyitg3yaq (complete)
  Copied to SP 2: bafkzcibfqcapabiuuwbw4j3fajxirwml7a4bkrhop6gipo3u47drjeusofmnyitg3yaq
  Pieces committed on SP 2, tx: 0xfc29b2517571b4c4e8a18cc070a22748586904ed2ec69829788b10e427d55474
    bafkzcibfqcapabiuuwbw4j3fajxirwml7a4bkrhop6gipo3u47drjeusofmnyitg3yaq
  Pieces committed on SP 1, tx: 0xac02fb844876639edb51238ce40fc879c749c46cf555557045f1fe869853a9e0
    bafkzcibfqcapabiuuwbw4j3fajxirwml7a4bkrhop6gipo3u47drjeusofmnyitg3yaq
  Data set 2 confirmed on SP 1
    bafkzcibfqcapabiuuwbw4j3fajxirwml7a4bkrhop6gipo3u47drjeusofmnyitg3yaq -> pieceId 0
  Data set 1 confirmed on SP 2
    bafkzcibfqcapabiuuwbw4j3fajxirwml7a4bkrhop6gipo3u47drjeusofmnyitg3yaq -> pieceId 0

--- Upload Summary ---

File: random_file
  PieceCID: bafkzcibfqcapabiuuwbw4j3fajxirwml7a4bkrhop6gipo3u47drjeusofmnyitg3yaq
  Size: 20 MB
  [Primary]   Provider 1 - pieceId: 0, dataSetId: 2
  [Secondary] Provider 2 - pieceId: 0, dataSetId: 1

--- Downloading and Verifying ---

Downloading bafkzcibfqcapabiuuwbw4j3fajxirwml7a4bkrhop6gipo3u47drjeusofmnyitg3yaq...
  VERIFIED: 20 MB matches original

=== SUCCESS: All files uploaded, replicated, and verified ===
The service provider(s) will periodically prove they still have your data.
You are being charged based on the storage size and duration.
[ OK ] NETWORK=devnet node utils/example-storage-e2e.js random_file

✅ test_multi_copy_upload - PASS (82s)

[ OK ] node is installed
[ OK ] npm is installed
[ OK ] npm init
[ OK ] pin filecoin-pin dependencies
[ OK ] npm install
[INFO] Patched @filoz/synapse-core streaming upload (stripped 1 Content-Length line(s))
[INFO] Creating random file (20971520 bytes)
[ OK ] random_file created with exact size 20971520 bytes
[INFO] Running filecoin-pin multi-copy upload script against devnet
[INFO] filecoin-pin add attempt 1 (default multi-copy)
[INFO] Verified retrieval URL 1: http://host.docker.internal:5721/ipfs/bafybeicscvnp3dhanfl3yrgdavs35em6i4ycrkon5gvoggfnn6gfqdfj4q
[INFO] Verified retrieval URL 2: http://host.docker.internal:5725/ipfs/bafybeicscvnp3dhanfl3yrgdavs35em6i4ycrkon5gvoggfnn6gfqdfj4q

❌ test_caching_subsystem - FAIL (128s)

[FAIL] upload 80MB piece (exit=1) Error: Failed to commit on primary provider 1 (http://host.docker.internal:5721) - data is stored but not on-chain

Details: StorageContext commit failed: Failed to commit pieces on-chain - Failed to add pieces.

Details: Service Provider PDP
chain: read tcp 172.18.0.5:45476->172.18.0.2:1234: i/o timeout

Caused by: StorageContext commit failed: Failed to commit pieces on-chain - Failed to add pieces.

Details: Service Provider PDP
chain: read tcp 172.18.0.5:45476->172.18.0.2:1234: i/o timeout

CommitError: Failed to commit on primary provider 1 (http://host.docker.internal:5721) - data is stored but not on-chain

Details: StorageContext commit failed: Failed to commit pieces on-chain - Failed to add pieces.

Details: Service Provider PDP
chain: read tcp 172.18.0.5:45476->172.18.0.2:1234: i/o timeout

    at StorageManager.upload (file:///tmp/synapse-sdk-cache-c6eor2uo/synapse-sdk/packages/synapse-sdk/src/storage/manager.ts:323:13)
    at process.processTicksAndRejections (node:internal/process/task_queues:104:5)
    at async main (file:///tmp/synapse-sdk-cache-c6eor2uo/synapse-sdk/utils/example-storage-e2e.js:119:20) {
  cause: Error: StorageContext commit failed: Failed to commit pieces on-chain - Failed to add pieces.
  
  Details: Service Provider PDP
  chain: read tcp 172.18.0.5:45476->172.18.0.2:1234: i/o timeout
  
      at createError (file:///tmp/synapse-sdk-cache-c6eor2uo/synapse-sdk/packages/synapse-sdk/src/utils/errors.ts:15:18)
      at StorageContext.commit (file:///tmp/synapse-sdk-cache-c6eor2uo/synapse-sdk/packages/synapse-sdk/src/storage/context.ts:883:13)
      at process.processTicksAndRejections (node:internal/process/task_queues:104:5)
      at async file:///tmp/synapse-sdk-cache-c6eor2uo/synapse-sdk/packages/synapse-sdk/src/storage/manager.ts:256:22
      ... 2 lines matching cause stack trace ...
      at async main (file:///tmp/synapse-sdk-cache-c6eor2uo/synapse-sdk/utils/example-storage-e2e.js:119:20) {
    [cause]: AddPiecesError: Failed to add pieces.
    
    Details: Service Provider PDP
    chain: read tcp 172.18.0.5:45476->172.18.0.2:1234: i/o timeout
    
        at addPiecesApiRequest (file:///tmp/synapse-sdk-cache-c6eor2uo/synapse-sdk/packages/synapse-core/dist/src/sp/add-pieces.js:28:19)
        at process.processTicksAndRejections (node:internal/process/task_queues:104:5)
        at async StorageContext.commit (file:///tmp/synapse-sdk-cache-c6eor2uo/synapse-sdk/packages/synapse-sdk/src/storage/context.ts:840:33)
        at async file:///tmp/synapse-sdk-cache-c6eor2uo/synapse-sdk/packages/synapse-sdk/src/storage/manager.ts:256:22
        at async Promise.allSettled (index 0)
        at async StorageManager.upload (file:///tmp/synapse-sdk-cache-c6eor2uo/synapse-sdk/packages/synapse-sdk/src/storage/manager.ts:265:27)
        at async main (file:///tmp/synapse-sdk-cache-c6eor2uo/synapse-sdk/utils/example-storage-e2e.js:119:20) {
      cause: undefined,
      details: 'Service Provider PDP\n' +
        'chain: read tcp 172.18.0.5:45476->172.18.0.2:1234: i/o timeout\n',
      shortMessage: 'Failed to add pieces.',
      Symbol(synapse-error): true
    }
  },
  details: 'StorageContext commit failed: Failed to commit pieces on-chain - Failed to add pieces.\n' +
    '\n' +
    'Details: Service Provider PDP\n' +
    'chain: read tcp 172.18.0.5:45476->172.18.0.2:1234: i/o timeout\n',
  shortMessage: 'Failed to commit on primary provider 1 (http://host.docker.internal:5721) - data is stored but not on-chain',
  providerId: '1',
  endpoint: 'http://host.docker.internal:5721',
  Symbol(synapse-error): true
}
[ OK ] git is installed
[ OK ] node is installed
[ OK ] pnpm is installed
[INFO] Run index: 1, seeds: small=43, large=86
[INFO] cqlsh version: cqlsh 6.2.0
[INFO] Yugabyte cassandra port: localhost:5705
[INFO] Initial row count = 0
[ OK ] clone synapse-sdk
[ OK ] checkout master HEAD
[INFO] synapse-sdk commit: e699e604f2c765100144efeed326181db17ee706
[ OK ] pnpm install
[ OK ] pnpm build
[INFO] Uploading 20MB piece (below 32MB threshold)
[INFO] Loading devnet info from: /home/runner/.foc-devnet/state/latest/devnet-info.json
Devnet run: 20260524T0655_IckyBoo
Using user: USER_1 (0x47cc9101fd026fc112d7fadf6b3c9df5be7d4a8c)
SPs available: 2
=== Synapse SDK Storage E2E Example ===

Reading file...
  small_20mb (20 MB)

--- Initializing Synapse SDK ---
Network: FOC DevNet
RPC URL: http://localhost:5701/rpc/v1
Wallet address: 0x47CC9101fD026fC112D7FADf6B3c9DF5bE7D4A8c
Synapse instance created

--- Checking Balances ---
FIL balance: 999.9999657343666 FIL
USDFC balance: 999.999966 USDFC

--- Preparing Account ---
Estimated costs:
  Per epoch (30s): 0.000001 USDFC
  Per month: 0.048000 USDFC
  Deposit needed: 0.000000 USDFC
  Ready: true
Account already ready

--- Uploading ---
Uploading small_20mb (20 MB) via stream...

  Selected SP 1 (0x446339aE7245e3cd1FeD701b685C196C69af695e)
  Creating new data set: 2
  Selected SP 2 (0xC60bc6fb60901c2d28ba6e4F8AD4915Bd2106497)
  Creating new data set: 1
  Upload progress: 10 MB (50.0%)
  Upload progress: 20 MB (100.0%)
  Stored on SP 1: bafkzcibfqcapabiuoe2wvpyktx3aunlchjyqizqov5lhagoi4ycwnq344ql6k4gctmwa
  Pulling to SP 2: bafkzcibfqcapabiuoe2wvpyktx3aunlchjyqizqov5lhagoi4ycwnq344ql6k4gctmwa (pending)
  Pulling to SP 2: bafkzcibfqcapabiuoe2wvpyktx3aunlchjyqizqov5lhagoi4ycwnq344ql6k4gctmwa (pending)
  Pulling to SP 2: bafkzcibfqcapabiuoe2wvpyktx3aunlchjyqizqov5lhagoi4ycwnq344ql6k4gctmwa (inProgress)
  Pulling to SP 2: bafkzcibfqcapabiuoe2wvpyktx3aunlchjyqizqov5lhagoi4ycwnq344ql6k4gctmwa (complete)
  Copied to SP 2: bafkzcibfqcapabiuoe2wvpyktx3aunlchjyqizqov5lhagoi4ycwnq344ql6k4gctmwa
  Pieces committed on SP 1, tx: 0x77c7a122d0f7f4cc5de0cf82a7af381c818083f7c42c509747820fb9d2949b30
    bafkzcibfqcapabiuoe2wvpyktx3aunlchjyqizqov5lhagoi4ycwnq344ql6k4gctmwa
  Pieces committed on SP 2, tx: 0xc9a2181b4e0117b918551398dab0a24704523ba9db27b3676d4689325757769b
    bafkzcibfqcapabiuoe2wvpyktx3aunlchjyqizqov5lhagoi4ycwnq344ql6k4gctmwa
  Data set 2 confirmed on SP 1
    bafkzcibfqcapabiuoe2wvpyktx3aunlchjyqizqov5lhagoi4ycwnq344ql6k4gctmwa -> pieceId 1
  Data set 1 confirmed on SP 2
    bafkzcibfqcapabiuoe2wvpyktx3aunlchjyqizqov5lhagoi4ycwnq344ql6k4gctmwa -> pieceId 1

--- Upload Summary ---

File: small_20mb
  PieceCID: bafkzcibfqcapabiuoe2wvpyktx3aunlchjyqizqov5lhagoi4ycwnq344ql6k4gctmwa
  Size: 20 MB
  [Primary]   Provider 1 - pieceId: 1, dataSetId: 2
  [Secondary] Provider 2 - pieceId: 1, dataSetId: 1

--- Downloading and Verifying ---

Downloading bafkzcibfqcapabiuoe2wvpyktx3aunlchjyqizqov5lhagoi4ycwnq344ql6k4gctmwa...
  VERIFIED: 20 MB matches original

=== SUCCESS: All files uploaded, replicated, and verified ===
The service provider(s) will periodically prove they still have your data.
You are being charged based on the storage size and duration.
[ OK ] upload 20MB piece
[INFO] Waiting 10s for caching tasks
[INFO] row_count after 'upload 20MB piece' = 0
[ OK ] cache rows count should not increase
[INFO] Uploading 80MB piece (above 32MB threshold)

[Action Run link](https://github.com/FilOzone/foc-devnet/actions/runs/26353992164)

[rjan90]

I dug through the failing nightly run and the scenario artifacts.

The failure is isolated to the frontier matrix, specifically test_caching_subsystem during the 80MB upload path. The 20MB upload passed and correctly left the cache row count at 0, but the 80MB upload failed after the data had already reached the service provider:

Failed to commit on primary provider 1 ... data is stored but not on-chain
Failed to add pieces.
chain: read tcp 172.18.0.5:45476->172.18.0.2:1234: i/o timeout

So the immediate failure appears to be Curio/PDP SP 1 timing out while talking to Lotus during the on-chain AddPieces/commit step. Based on the logs I also saw that we are using the pdpv0-branch in Curio, we probably should update that to main now?

[rjan90]

I manually dispatched CI (Nightly) on #115. Run: https://github.com/FilOzone/foc-devnet/actions/runs/26440135402

Both stability and frontier passed. The frontier job used --curio gitbranch:main --filecoin-services gitbranch:main, and test_caching_subsystem passed, including the 80MB upload path that failed in this issue. Not sure if this is flaky, or that the #115 PR fixed it, but for now I think we can close this issue ticket, and see if it fails again on tha main-branch of Curio