Skip to content

Shallow and concurrent git clones#10254

Merged
mergify[bot] merged 2 commits into
haskell:masterfrom
alt-romes:wip/romes/cabal-shallow-mr
Nov 17, 2024
Merged

Shallow and concurrent git clones#10254
mergify[bot] merged 2 commits into
haskell:masterfrom
alt-romes:wip/romes/cabal-shallow-mr

Conversation

@alt-romes
Copy link
Copy Markdown
Collaborator

@alt-romes alt-romes commented Aug 12, 2024 •
edited
Loading

Please read Github PR Conventions and then fill in one of these two templates.

Template Α: This PR modifies behaviour or interface

Include the following checklist in your PR:

  • Patches conform to the coding conventions.
  • Any changes that could be relevant to users have been recorded in the changelog.
  • The documentation has been updated, if necessary.
  • Manual QA notes have been included.
  • Tests have been added. (Ask for help if you don’t know how to write them! Ask for an exemption if tests are too complex for too little coverage!)

Manual QA: Create or use a package with multiple source-repository-package dependencies, cabal clean, then cabal build. You should see two repositories start being cloned at the same time, and then see other repositories being cloned concurrently as jobs are finished. The reason why only two repositories get cloned in parallel is that the limit of asynchronous downloads is == 2, to mimic asyncFetchPackages.

Questions: The maximum two-job cap on cloning concurrency seems too low, I think it would be best if it could be configured (as a follow up MR?)

@alt-romes alt-romes force-pushed the wip/romes/cabal-shallow-mr branch 3 times, most recently from 844e1bb to fd373ad Compare August 12, 2024 11:07
@alt-romes alt-romes changed the title Wip/romes/cabal shallow mr Shallow and concurrent git clones Aug 12, 2024
@mpickering
Copy link
Copy Markdown
Collaborator

mpickering commented Aug 16, 2024

This change also affects cabal get -s, that needs documenting if it's intentional.

@alt-romes alt-romes force-pushed the wip/romes/cabal-shallow-mr branch from fd373ad to eb4f43c Compare August 16, 2024 13:51
@alt-romes alt-romes force-pushed the wip/romes/cabal-shallow-mr branch 7 times, most recently from 0416a52 to 85b2a5c Compare November 8, 2024 10:35
@alt-romes
Copy link
Copy Markdown
Collaborator Author

alt-romes commented Nov 9, 2024

This is ready

ulysses4ever
ulysses4ever approved these changes Nov 10, 2024
View reviewed changes
Copy link
Copy Markdown
Collaborator

@ulysses4ever ulysses4ever left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

Comment thread cabal-install/src/Distribution/Client/JobControl.hs
Comment thread cabal-install/src/Distribution/Client/ProjectConfig.hs Outdated
Comment thread cabal-install/src/Distribution/Client/RebuildMonad.hs
Comment thread cabal-install/src/Distribution/Client/VCS.hs Outdated
Comment thread cabal-install/src/Distribution/Client/ProjectConfig.hs Outdated
@alt-romes alt-romes force-pushed the wip/romes/cabal-shallow-mr branch from 85b2a5c to 0d9629a Compare November 11, 2024 10:42
@alt-romes
Copy link
Copy Markdown
Collaborator Author

alt-romes commented Nov 11, 2024

I've addressed your review @ulysses4ever. Merging this now!

@alt-romes alt-romes self-assigned this Nov 11, 2024
@alt-romes alt-romes added the merge me Tell Mergify Bot to merge label Nov 11, 2024
@mergify mergify Bot added the ready and waiting Mergify is waiting out the cooldown period label Nov 11, 2024
@alt-romes alt-romes mentioned this pull request Nov 11, 2024
Closed
@ulysses4ever ulysses4ever linked an issue Nov 11, 2024 that may be closed by this pull request
source-repository-package generates fat clones #7264
Closed
@mergify mergify Bot added the merge delay passed Applied (usually by Mergify) when PR approved and received no updates for 2 days label Nov 13, 2024
@ulysses4ever
Copy link
Copy Markdown
Collaborator

ulysses4ever commented Nov 16, 2024

@alt-romes can you resolve the conflicts?

@ulysses4ever ulysses4ever mentioned this pull request Nov 16, 2024
Closed
5 tasks
@alt-romes
source-repository: Use git shallow clones
b6c28ee 
Cloning the entire repository for the purpose of compiling packages
specified in source-repository-packages is wasted effort. To read and
compile the package, we need only the HEAD of the repository, thus a
shallow clone is sufficient.

Note that this doesn't change the behaviour of `cabal get -s` which
still does a full clone (--depth=1 is only used in vcsSyncRepo, not in
vcsCloneRepo)

Fixes haskell#7264
@alt-romes
Synchronize VCS repos concurrently
c89ab54 
Cloning/synchronising VCS repos can be unnecessarily slow if done
serially. By synchronizing the repos concurrently we make much better
use of time.

Introduces rerunConcurrentlyIfChanged, a Rebuild monad function that
runs, from multiple actions, the actions that need rebuilding concurrently.
@alt-romes alt-romes force-pushed the wip/romes/cabal-shallow-mr branch from 0d9629a to c89ab54 Compare November 16, 2024 21:19
@alt-romes alt-romes added merge me Tell Mergify Bot to merge and removed merge me Tell Mergify Bot to merge labels Nov 16, 2024
@mergify mergify Bot merged commit c3a9dd7 into haskell:master Nov 17, 2024
@9999years
Copy link
Copy Markdown
Collaborator

9999years commented Nov 22, 2024

Hi, I'm working on the VCS test suite and (while running it in verbose mode) I noticed this:

Running: git -c 'protocol.file.allow=always' clone '--depth=1' --no-checkout /private/var/folders/z5/fclwwdms3r1gq4k4p3pkvvc00000gn/T/vcstest-89726/src /private/var/folders/z5/fclwwdms3r1gq4k4p3pkvvc00000gn/T/vcstest-89726/dest1
Cloning into '/private/var/folders/z5/fclwwdms3r1gq4k4p3pkvvc00000gn/T/vcstest-89726/dest1'...
warning: --depth is ignored in local clones; use file:// instead.

As a result, I'm not sure this feature is actually being tested.

When I patched the test suite to use the file:// URLs as suggested by the warning, I found a failure:

Running: git --version
Running: git -c 'protocol.file.allow=always' init
Initialized empty Git repository in /private/var/folders/z5/fclwwdms3r1gq4k4p3pkvvc00000gn/T/vcstest-60316/src/.git/
Running: git -c 'protocol.file.allow=always' add file/C
Running: git -c 'protocol.file.allow=always' -c 'user.name=A' -c 'user.email=a@example.com' commit --all '--message=a patch' '--author=A <a@example.com>'
[master (root-commit) 569c51c] a patch
 1 file changed, 1 insertion(+)
 create mode 100644 file/C
Running: git -c 'protocol.file.allow=always' log '--format=%H' -1
Running: git -c 'protocol.file.allow=always' tag --force --no-sign tag_XFUS

--------------------------------------------------------------------------------

Running: git -c 'protocol.file.allow=always' clone '--depth=1' --no-checkout 'file:///private/var/folders/z5/fclwwdms3r1gq4k4p3pkvvc00000gn/T/vcstest-60316/src' /private/var/folders/z5/fclwwdms3r1gq4k4p3pkvvc00000gn/T/vcstest-60316/dest1
Cloning into '/private/var/folders/z5/fclwwdms3r1gq4k4p3pkvvc00000gn/T/vcstest-60316/dest1'...
remote: Enumerating objects: 4, done.
remote: Counting objects: 100% (4/4), done.
remote: Total 4 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
Receiving objects: 100% (4/4), done.
Running: git -c 'protocol.file.allow=always' submodule deinit --force --all
Running: git -c 'protocol.file.allow=always' fetch origin tag_XFUS
From file:///private/var/folders/z5/fclwwdms3r1gq4k4p3pkvvc00000gn/T/vcstest-60316/src
 * tag               tag_XFUS   -> FETCH_HEAD
Running: /Users/wiggles/.nix-profile/bin/git -c 'protocol.file.allow=always' reset --hard FETCH_HEAD --
HEAD is now at 569c51c a patch
Running: git -c 'protocol.file.allow=always' clean -ffxdq
Running: git -c 'protocol.file.allow=always' clone '--depth=1' --no-checkout 'file:///private/var/folders/z5/fclwwdms3r1gq4k4p3pkvvc00000gn/T/vcstest-60316/src' /private/var/folders/z5/fclwwdms3r1gq4k4p3pkvvc00000gn/T/vcstest-60316/dest2 --reference /private/var/folders/z5/fclwwdms3r1gq4k4p3pkvvc00000gn/T/vcstest-60316/dest1
Cloning into '/private/var/folders/z5/fclwwdms3r1gq4k4p3pkvvc00000gn/T/vcstest-60316/dest2'...
fatal: reference repository '/private/var/folders/z5/fclwwdms3r1gq4k4p3pkvvc00000gn/T/vcstest-60316/dest1' is shallow

@9999years 9999years mentioned this pull request Nov 22, 2024
Merged
2 tasks
@9999years
Copy link
Copy Markdown
Collaborator

9999years commented Nov 22, 2024

I have a number of PRs to make the VCS tests easier to work with open right now; I'll see if I can work on this when I get them merged:

@9999years 9999years mentioned this pull request Dec 3, 2024
Open
@ffaf1 ffaf1 mentioned this pull request Jan 29, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@geekosaur geekosaur geekosaur approved these changes

@ulysses4ever ulysses4ever ulysses4ever approved these changes

@fgaz fgaz Awaiting requested review from fgaz

Assignees

@alt-romes alt-romes

Labels

merge delay passed Applied (usually by Mergify) when PR approved and received no updates for 2 days merge me Tell Mergify Bot to merge ready and waiting Mergify is waiting out the cooldown period

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

source-repository-package generates fat clones

5 participants

@alt-romes @mpickering @ulysses4ever @9999years @geekosaur