Feature request: the ability to package and attach additional files to gears function

[AlexMikhalev]

I have tried to build rather small NLP pipeline https://github.com/applied-knowledge-systems/the-pattern-platform and I struggle to improve code quality using standard python coding techniques. For example:

• Currently, all path/URLs and ports are hardcoded which will quickly become technical dept in any longer-term deployment, I created config.py with ini parser and corresponding conf dir, but I couldn't find a way to attach it to submission to gears function. May be pulling from private repo (passed via requirements.txt) will work? Need help here.
• Edges To Graph may be a good candidate to rgsync connector, again I could not figure out a way to submit my own connector without creating public repo.

[MeirShpilraien]

@AlexMikhalev you can put your connector code on some local path on the same machine where Redis running and give this path as a requirement, RedisGears will find it there and install it in its virtual environment...

[MeirShpilraien]

But regardless I agree this is an important feature, but I think it should be supported as part of RedisGears and not gears-cli.

[AlexMikhalev]

@MeirShpilraien thank you for the suggestion, can you give a short example of your first suggestion?

cat config.sh 
PROTECTED_MODE=no
PORT=30000
NODES=$(cat /proc/cpuinfo| grep cores | wc -l)
REPLICAS=1
ADDITIONAL_OPTIONS="--loadmodule /home/alex/infrastructure/RedisGears/bin/linux-x64-release/redisgears.so  PythonHomeDir /home/alex/infrastructure/RedisGears/bin/linux-x64-release/python3_1.0.3 CreateVenv 1 DownloadDeps 1"

Does it mean gears will pickup anything put into PythonHomeDir?
I thought about putting feature request into RedsGears first, but then I assumed it's supported and only needs packaging - export-requirements produces zip file with some blob and manifest, may be same can be applied to project code - zip and manifest to run.

[MeirShpilraien]

@AlexMikhalev RedisGears creates a virtual environment for each shard and installs the given requirement there, Assuming you some python dependency under /path/to/python_dep/ that can be installed using pip install /path/to/python_dep/, Then specify this path as a requireement will cause RedisGears to install it in its virtual environment from this local path. So something like this should work:

gears-cli run <path/to/my/script> REQUIREEMENTS </path/to/python_dep/>

You can also create a requirement file and put /path/to/python_dep/ there.

I agree this is not the best solution and we should allow just sending an archive that contains all the code, but this is a workaround for now.

Let me know if you have any issues with this.

[AlexMikhalev]

@MeirShpilraien thank you, I will give it a try. At the moment I rewrite scripts to work differently - combined multiple gears into one script and use different filter/flatmap/map and another foreach.

[AlexMikhalev]

In the meantime I found an (interim) workaround:

    import httpimport
    with httpimport.remote_repo(['stop_words'], "https://raw.githubusercontent.com/explosion/spaCy/master/spacy/lang/en/"):
        import stop_words
    from stop_words import STOP_WORDS

will use STOP_WORDS without install of whole spacy.
will work for github gists as well.