only clone a subdirectory of a git repo

[jgphpc]

Can ReFrame support sparse git clones ?

• https://stackoverflow.com/questions/600079/how-do-i-clone-a-subdirectory-only-of-a-git-repository

For now, I do it with a hook (I could also use a fixture but that's not the point):

import os
import reframe as rfm
import reframe.utility.sanity as sn


@rfm.simple_test
class git_clone_test(rfm.CompileOnlyRegressionTest):
    repo = variable(
        str, value='https://github.com/eth-cscs/alps-gh200-reproducers.git')
    sourcesdir = None
    build_system = 'SingleSource'
    sourcepath = 'hello.c'
    valid_systems = ['*']
    valid_prog_environs = ['*']

    @run_before('compile')
    def prepare_clone(self):
        self.prebuild_cmds = [
            f'git clone --no-checkout --depth=1 {self.repo}',
            'cd alps-gh200-reproducers',
            'git config core.sparseCheckout true',
            'echo intranode-pinned-host-comm > .git/info/sparse-checkout',
            'git checkout main',
            'cd ..'
        ]

    @sanity_function
    def validate_test(self):
        return sn.assert_not_found(r'error', self.stdout)

Ideally, I'd like to use:

    sourcesdir = 'https://github.com/eth-cscs/alps-gh200-reproducers.git:intranode-pinned-host-comm'

[victorusu]

@jgphpc , what do you think about this syntax?

import os
import reframe as rfm
import reframe.utility.sanity as sn


@rfm.simple_test
class git_clone_test(rfm.CompileOnlyRegressionTest):
    sourcesdir = {
        'https://github.com/eth-cscs/alps-gh200-reproducers.git' : {
            'files' : ['intranode-pinned-host-comm']
        }
    }
    build_system = 'SingleSource'
    sourcepath = 'intranode-pinned-host-comm/intranode_pinned_host_comm.cpp'
    valid_systems = ['*']
    valid_prog_environs = ['*']

    @sanity_function
    def validate_test(self):
        return sn.assert_not_found(r'error', self.stdout)

This will download all files in the intranode-pinned-host-comm directory. Alternatively, if only the file is required, you can download just the file using:

...
@rfm.simple_test
class git_clone_test(rfm.CompileOnlyRegressionTest):
    sourcesdir = {
        'https://github.com/eth-cscs/alps-gh200-reproducers.git' : {
            'files' : ['intranode-pinned-host-comm/intranode_pinned_host_comm.cpp']
        }
    }
    ...

You can try the proposed syntax using the branch sparse_checkout from my fork.
Let me know what you think

[victorusu]

The way I have implemented, I think it requires Git version greater than 2.25.0. Because I emit git sparse-checkout commands. Something similar to

git clone --no-checkout --depth=1 ...
git sparse-checkout set --no-cone ${file list}
git checkout

[jgphpc]

you could support extra flags too:

'extra': ['--depth=1']

but lgtm as it is, thanks.

[victorusu]

@jgphpc, I changed the syntax slightly.
What do you think about this now?

@rfm.simple_test
class git_clone_test(rfm.CompileOnlyRegressionTest):
    sourcesdir = {
         'url': 'https://github.com/eth-cscs/alps-gh200-reproducers.git',
         'files' : ['intranode-pinned-host-comm'],
         'opts' : ['--no-optional-locks', '--no-advice', '--no-pager']
    }
    build_system = 'SingleSource'
    sourcepath = 'intranode-pinned-host-comm/intranode_pinned_host_comm.cpp'
    valid_systems = ['*']
    valid_prog_environs = ['*']

    @sanity_function
    def validate_test(self):
        return sn.assert_not_found(r'error', self.stdout)

Please note that the options are passed to all git clone commands emitted by ReFrame to that particular test. They do not affect the git sparse-checkout and the git checkout commands.