filecoin-pin in CI as a testing scenario for multi-copy upload + Synapse testing

[rvagg]

Part of FilOzone/filecoin-services#357; not yet solved by #90 (Edit: we have scenarios/test_storage_e2e.py now so it is supported, sorry, this is an extra layer that we need to do)

Devnet support got added in filecoin-pin in 0.18 in filecoin-project/filecoin-pin#354, so it should be mostly a matter of making some test data in the form of files, then either installing filecoin-pin or just running npx -y filecoin-pin@v0.18 add --network devnet /path/to/data substituting 0.18 for some other version or just @latest. Potentially also we can test a git clone and npm install then node dist/cli.js add --network devnet /path/to/data and get the same effect if we want to test the master branch.

[BigLep]

@rvagg : given @galargh is going to take a stab at this, I want to make sure we are clear about the flow:

• Start devnet
• filecoin-pin add test-data
• wait for add to finish
• confirm test-data retrievability and integrity using the retrieval URLs returned by the add command

In terms of versions to use, I assume for curio and fwss we'll do this against "stability" and "frontier" per https://github.com/FilOzone/foc-devnet/blob/main/.github/workflows/ci_nightly.ym

In terms of the version of filecoin pin, I assume we'll just the HEAD in main? That appears to be what we're doing for Synapse in https://github.com/FilOzone/foc-devnet/blob/main/scenarios/synapse.py

[rvagg]

We'll get back IPFS urls from add and those won't work, in fact they don't even show up I think since --skip-ipni-verification is implicit in devnet. But there's a few things we can try here, some simple, some complex. We either need the root IPFS CID from the filecoin-pin add output or we need to know it separately. We either use the same fixed CAR each time, we know the roots and the blocks and can fetch those, or we parse the add output to find the root CID.
Once we have that, we can probe retrievals and we have options there too. We could:

• Use [lassie](https://github.com/filecoin-project/lassie) to do the retrieval, that's heavy-weight but would test the full standard retrieval path, I think you can construct a multiaddr from the SP addresses in devnet and give it --providers to point it there. Let me know if you want to get in the weeds with this one.
• Download the full CAR from the /ipfs/ retrieval endpoint, /ipfs/bafy...?format=car&dag-scope=all and it'll regenerate a properly ordered CAR for you, it won't be the same order as filecoin-pin builds the CAR but you can verify all the blocks are in it. You could use go-car to do something like car ls <downloaded> to make sure it has all the CIDs you expect (if you know), or just car inspect --full <downloaded> to do a full verification (re-hash of CIDs) and that command will also tell you the root CID is present (although it doesn't tell you that all the CIDs you wanted are there).
• Download the original CAR from the piece endpoint, /piece/baga... with the PieceCID of the upload, that's the raw upload CAR. You can re-hash that with CommP and verify that it's the right piece (Synapse has utilities to help with this, or https://github.com/filecoin-project/go-fil-commp-hashhash/tree/master/cmd/stream-commp is an efficient way of making a CommP from a piece (though it'll give you PieceCIDv1 and we use v2 [conversion is possible, but if you go this route I could tack on a PieceCIDv2 output to that command pretty easily too). You could do a car ls of the verified piece download and a car inspect --full of the /ipfs/bafy downloaded CAR and a car ls of that and compare the list of CIDs that are supposed to be in it and you'd verify that the SP can do both piece and IPFS download and it serves it in proper order.

This is all probably a bit too complicated, and we could iterate toward it and the MVP might just be to download the single root block from /ipfs/ using ?format=raw - that'll just give bytes of that block and we could verify that the bytes match the CID and that's good enough. Verifying a CID using a Python script is a bit iffy, although there is a new Python team that's been reviving the IPFS/Multiformats/IPLD libraries so maybe directly, or just install some Go or, more likely, JavaScript utility to do it, js-multiformats ('multiformats' in npm) does basic CID hashing but you need some basic scripting to make a utility to do it. AI could easily do it if you point it that way.

Sorry, that's a whole bunch of deep detail, hopefully you can find some meaningful MVP in here. The main point is - you can't do global content routing, we don't involve IPNI here, so you're just testing that the SP can serve IPFS content via the /ipfs/ endpoint in some way. The Synapse test scenario already does basic piece upload and piece retrieval testing already, this would test IPFS.

[rvagg]

I forgot I ran this in the background and meant to post the output. This is what a stdout capture of a filecoin-pin add looks like when you --skip-ipni-validation (implicit for devnet):

Filecoin Pin Add

✓ Directory validated

✓ Connected to Filecoin - Calibration testnet

✓ Minimum payment setup verified (~0.066 USDFC required)

✓ Directory packed with root CID: bafybeiadkvpy4zncq76uks7axzlucydudrfen34pi455rqbsx5ugww5hxy

✓ IPFS content loaded (1.5 MiB)

✓ Payment capacity verified for 1.5 MiB

✓ [Primary] Stored on provider 5

✓ [Secondary] Stored on provider 9

✓ [Secondary] Piece added to Data Set (unconfirmed on-chain)

  Tx: https://calibration.filfox.info/en/message/0xf551d3f2ada3019afa04d88e350fd1848de25bdedf82bfd1350ced98c957b380
✓ [Primary] Piece added to Data Set (unconfirmed on-chain)

  Tx: https://calibration.filfox.info/en/message/0xcc1b765bc4a036ee9f71360fe819e37edeb0b101952f6d28d12625d65b47b6b3
✓ [Primary] Piece added to Data Set (confirmed on-chain)

✓ [Secondary] Piece added to Data Set (confirmed on-chain)

━━━ Add Complete ━━━

Network: Filecoin - Calibration testnet

Add Details
  File: dist/
  Size: 1.5 MiB
  Root CID: bafybeiadkvpy4zncq76uks7axzlucydudrfen34pi455rqbsx5ugww5hxy

Filecoin Storage
  Piece CID: bafkzcibeskrb6ehq4sxcu76dpxyj3dsw7r5c7a7sbfbowvidguzj3aivrqvjzmd6hy
  Piece Size: 1.5 MiB
  Explorer: https://pdp.vxb.ai/calibration/piece/bafkzcibeskrb6ehq4sxcu76dpxyj3dsw7r5c7a7sbfbowvidguzj3aivrqvjzmd6hy

Copies
  [Primary] Provider 5
    Data Set ID: 51
    Piece ID: 25
    Retrieval URL: https://warp.lotus.dedyn.io/piece/bafkzcibeskrb6ehq4sxcu76dpxyj3dsw7r5c7a7sbfbowvidguzj3aivrqvjzmd6hy
  [Secondary] Provider 9
    Data Set ID: 11871
    Piece ID: 12
    Retrieval URL: https://calib.ezpdpz.net/piece/bafkzcibeskrb6ehq4sxcu76dpxyj3dsw7r5c7a7sbfbowvidguzj3aivrqvjzmd6hy
Add completed successfully

Grep that for Root CID: and Piece CID: and you've got the two important pieces. The URLs are useful too, replace /piece/ with /ipfs/ and the Root CID in there and you have the IPFS retrieval endpoints for the two SPs.

[BigLep]

Thanks for all the info/context @rvagg!

MVP might just be to download the single root block from /ipfs/ using ?format=raw

I agree with this. Lets keep it simple currently.

Reminder (mostly for myself) as to why we're adding filecoin-pin into the devnet:

• Ensure SPs index the data stored via Filecoin Pin. (We don't test remote advertisement announcement to IPNI, but we do confirm that the SP is indexing the piece locally)
• Ensure SP software handles /ipfs retrieval. Specifically, we want to confirm we can retrieve the corresponding IPLD block(s) via IPFS endpoints (2 SPs → 2 copies → 2 retrievals).