sqd deploy from within a monorepo

[mlenser]

Hi,

We have a monorepo where our indexer is a package of many. sqd deploy expects a package-lock.json (or the equivalent for yarn/pnpm) within the package, but the lock file is at the root of the monorepo. We've worked around this a bit by forcing our indexer to generate a package-lock.json via npm i --omit=dev --legacy-peer-deps --install-strategy=nested --workspaces=false --package-lock-only.

But that presents its own challenges as we cannot import from any other packages within the monorepo and we have to continuously update the package-lock.json just for sqd deploy.

Is there a recommended path here? We just switched from yarn to pnpm at the monorepo root if it matters.

[mo4islona]

Hey!
Could you please clarify how you deploy squid - are you deploying squid from the package folder rather than from the monorepo root folder? Is the Squid manifest also in that folder?

We run npm ci or add --frozen-lockfile flag for pnpm/yarn builds to ensure that you will have he same list of deps as you expect. Obviously this won't work if the lockfile is missing

[mlenser]

We're running sqd deploy from within the indexer package. The manifests are also in the indexer package.

We run npm ci or add --frozen-lockfile flag for pnpm/yarn builds to ensure that you will have he same list of deps as you expect. Obviously this won't work if the lockfile is missing

Ya, that's why we have to add this fake package-lock.json for the indexer package which also prevents indexer from being able to import from other packages.

[mlenser]

Hi @mo4islona gentle reminder. Is there a pattern to make squid-cli work in a monorepo without the hacks and to allow it to import from local packages?

[mo4islona]

Hey @mlenser! Sorry for the delay.

But that presents its own challenges as we cannot import from any other packages within the monorepo and we have to continuously update the package-lock.json just for sqd deploy.

You can run the postintsall script so that it will automatically trigger now when you make changes to the root package.json.
Have you tried copying the lock file to the squid folder?

"scripts": {
    "postinstall": "cp yarn.lock packages/indexer/yarn.lock"
}

In general, I see 2 options to solve the issue:

• CLI have to search and find the lockfile in the above directories and copy it to the squid folder. That is quite hard to implement/support, but is the right way, I believe.

• Just allow (probably, via special option in manifest) to skip lockfile and install from package.json, but that could cause inconsistency of installed libraries to expected.

WDYT?

[mlenser]

Copying the pnpm-lock.yaml (same for yarn.lock) from the root of the monorepo to the squid package won't work as we'll get a ERR_PNPM_LOCKFILE_CONFIG_MISMATCH error:

Step 11/24 : ADD assets assets
---> 04e023b2b072
Step 12/24 : ADD db db
---> db13bf7fb06a
Step 13/24 : ADD schema.graphql .
---> d678c2a1c7f7
Step 14/24 : ADD src src
---> 3f974c1a08c9
Step 15/24 : RUN pnpm install --frozen-lockfile
---> Running in cae545c34c40
ERR_PNPM_LOCKFILE_CONFIG_MISMATCH  Cannot proceed with the frozen installation. The current "overrides" configuration doesn't match the value found in the lockfile

Update your lockfile using "pnpm install --no-frozen-lockfile"
The command '/bin/sh -c pnpm install --frozen-lockfile' returned a non-zero code: 1
◷ Building the squid... 
 ›   Error: ❌ An error occurred while building the squid
 ›   ------
 ›   Please report to Discord https://discord.gg/KRvRcBdhEE or SquidDevs 
 ›   https://t.me/HydraDevs
 ›   Deploy: 31917
 ›   Squid: OMITTED_REPO_NAME-ci
 ›   Version: v1
 ELIFECYCLE  Command failed with exit code 2.
 ```

🤔 

[mo4islona]

Got it.
So for now we can only automate lock file generation with postintsall and npm i --omit=dev --legacy-peer-deps .....
and wait until we add support for monorepo.

Are you using pnpm and not rush, right?

[mlenser]

We're currently using pnpm at the root and npm inside the indexer package. We have pnpm workspace dependencies inside devDependencies of indexer which we strip out before generating the package-lock.json as npm can't handle workspaces.

Our generate-lock-file script:

pnpm package-json:remove-workspaces && npm i --omit=dev --legacy-peer-deps --install-strategy=nested --workspaces=false --package-lock-only && prettier --log-level warn --write package-lock.json && pnpm package-json:add-workspaces

This allows it to function as the devDependencies aren't used by squid, but it presents 2 problems as discussed above:

1. Having to generate the lock file (tolerable)
2. Not being able to import from other packages (this is the problem)

I'm trying to solve for the second issue at least, but I'm not seeing a way to make pnpm generate a lock file just for indexer like we do with npm. It all boils down to squid here though.

[mo4islona]

So, we are preparing a new release with a feature allowing customization of the install command.
I'll provide instructions how to update when ready

[mo4islona]

So

1. Please update CLI to 2.8.4
   npm i -g @subsquid/cli@2.8.4
2. Remove all lock files from squid directory
3. Add custom install command to build section

build:
  # ... other options
  install:
    cmd: 
      - pnpm
      - install

This will allow you to install deps without a lockfile. But be aware of their possible inconsistency.

[mlenser]

That sounds very promising, thank you!

Upon testing it our squid has been building for 1 hour and 20 mins when it previously took around 5-30 mins. I assume it has failed.

sqd deploy -ru -m ./squid-ci.yaml -o OMITTED_ORG_NAME .
Squid directory: /home/runner/work/OMITTED_REPO_NAME/services/indexer
Build directory: /home/runner/work/OMITTED_REPO_NAME/services/indexer/builds
Manifest: ./squid-ci.yaml
◷ Compressing the squid to OMITTED_REPO_NAME-ci-8017521824-v1.tar.gz ... 85 file(s) ✔️
adding .dockerignore
adding .eslintignore
adding .eslintrc.js
adding .gitignore
adding .gitpod.yml
adding .prettierignore
adding .prettierrc.js
adding .turbo
-- ignoring Dockerfile
adding LICENSE
adding README.md
adding abi
adding assets
-- ignoring builds
adding commands.json
adding db
-- ignoring lib
-- ignoring node_modules
adding package.json
adding renovate.json
adding schema.graphql
adding scripts
adding squid-ci.yaml
adding squid-mainnet.yaml
adding squid-sepolia.yaml
adding src
adding tsconfig.json
adding tsconfig.lint.json
adding .turbo/turbo-build.log
adding assets/README.MD
adding db/migrations
adding db/migrations/1708437055647-Data.js
adding db/migrations/17086OMITTED_PRIVATE_HASH 305309-Data.js
adding src/abi/multicall.ts
adding src/model/generated
adding src/model/index.ts
◷ Uploading OMITTED_PRIVATE_NAME-8017521824-v1.tar.gz... ✔️
🦑 Releasing the squid from local folder
◷ Preparing the squid... ✔️
Downloading squid archive from https://storage.googleapis.com/subsquid-rest-api-tmp-upload/tmp/b0a8bd69-9c5f-430b-be13-d024a56eb664.tar.gz...
Extracting archive...
Extracted 86 files from archive, hash OMITTED_PRIVATE_HASH
◷ Building the squid...
◷ You may now detach from the build process by pressing Ctrl + C. The Squid deployment will continue uninterrupted.
◷ The new squid will be available as soon as the deployment is complete.
◷ Building the squid...
◷ Building the squid...
◷ Building the squid...
forever and ever

[mo4islona]

Well, your build just hanged on our builder. I'm investigating

[mlenser]

Update: I let one build run for 1 hour & 22 minutes. It built the squid forever and then got to

Step 1/13 : FROM node:20-alpine as squid
---> df6a39829ab5
Step 2/13 : WORKDIR /squid
---> Using cache
---> 19764c93dbed
Step 3/13 : RUN echo -e "loglevel=silent\nupdate-notifier=false" > /squid/.npmrc
---> Using cache
---> b2c27c3cb284
Step 4/13 : RUN corepack enable && npm i -g @subsquid/commands && mv $(which squid-commands) /usr/local/bin/sqd
---> Using cache
---> 936572e31741
Step 5/13 : ADD assets assets
---> Using cache
---> 432c1a1f02eb
Step 6/13 : ADD db db
---> Using cache
---> 1908813900c2
Step 7/13 : ADD commands.json commands.json
---> Using cache
---> d08a3d064fad
Step 8/13 : ADD schema.graphql schema.graphql
---> Using cache
---> fb3d08a67db1
Step 9/13 : ADD src src
---> Using cache
---> 8652bab1923e
Step 10/13 : ADD package.json package.json
---> Using cache
---> acb824fc9858
Step 11/13 : ADD tsconfig.json tsconfig.json
---> Using cache
---> 24494db6158d
Step 12/13 : RUN pnpm install
---> Running in a16beb0af720
The command '/bin/sh -c pnpm install' returned a non-zero code: 137
◷ Building the squid... 
 ›   Error: ❌ An error occurred while building the squid
 ›   ------
 ›   Please report to Discord https://discord.gg/KRvRcBdhEE or SquidDevs 
 ›   https://t.me/HydraDevs
 ›   Deploy: 32148
 ›   Squid: OMITTED_REPO_NAME-ci-8018486937
 ›   Version: v1
 ELIFECYCLE  Command failed with exit code 2.
 ›   Error: Nonexistent flag: -o
 ›   See more help with --help

[mo4islona]

Update: I let one build run for 1 hour & 22 minutes. It built the squid forever and then got to

Thats fine for now. I'll come back when issue will be fixed

[mo4islona]

I assume the problem is that the squid directory cannot be built with pnpm but somehow it doesn't stop the build itself

pnpm install
ERR_PNPM_WORKSPACE_PKG_NOT_FOUND  In : "eslint-config-astaria@workspace:*" is in the dependencies but no package named "eslint-config-astaria" is present in the workspace

This error happened while installing a direct dependency of /Users/user/Downloads/b0a8bd69-9c5f-430b-be13-d024a56eb664

Packages found in the workspace:

[mo4islona]

But, in general, you are importing internal packages thats missing in the archive

"contracts-internal": "workspace:*",
"rewards": "workspace:*",

In that case, I think, you should try to upload whole monorepo and modify run commands instead of install

build:
  node_version: 20
  # package_manager: pnpm - BTW no need to specify it explicitly, it will understand by the pnpm lockfile in the root
  # do not modify install / build command
deploy:
   processor:
      cmd: 
        - cd
        - packages/indexer
        - &&
        - node
        - lib/main
   api:
     cmd: 
      - cd
      - packages/indexer
      - &&
      - npx
      - squid-graphql-server
      - '--dumb-cache'
      - in-memory
      - '--dumb-cache-ttl'
      - '1000'
      - '--dumb-cache-size'
      - '100'
      - '--dumb-cache-max-age'
      - '1000'