Multiple concurrent writers (and readers) with shared NFS mount

[matthewcummings]

I'm seeing an issue with cachier where I'm getting Bad File Descriptor errors. Just want to make sure I'm not completely abusing it here, I have multiple writers (and readers) accessing a shared cache directory via NFS.

Am I way off the mark here for thinking this would ever work correctly?

[shaypal5]

Hmmm. Honestly no idea.

Isn't NFS should technically make the software layer "feel" as if it's just accessing the file system? I really lack the deep OS/kernel/filesystem knowledge required to answer this.

Each cachier wrapper gets its own cache file (when using the pickle core) and just acquires and releases a lock to read/write to the file. Specifically, since by cachier also holds the cache in-memory, you can have it not reload the cache from file on every call (with @cachier(pickle_reload=False)), which doesn't effect relevancy of results if you run single threaded, but can mean different threads might get stale results if you run multi-threaded.

Also, the separate_files=True option makes the cache use a separate file for each argument set, so each function uses numerous files. Better for larger results, and might help in your case, if indeed different readers and writers use different argument sets to the same function. Might.

But then, Bad File Descriptor doesn't sound like it has to do with locking. On the other hand, I don't know. :)

Do you get it sporadically, or consistently?

[matthewcummings]

Consistently, it could be a bug in my code too. Mostly I wanted to confirm that this should work.
Thank you

[shaypal5]

Cool. Please let me know how it goes. It will help me help you if I understand the use case better; e.g. do all readers and writers use the same function or not, etc.

[matthewcummings]

Will do, thank you.

[shaypal5]

Any news on that? @matthewcummings

[shaypal5]

Note to all interested parties:
There is a possibly related issue here:
#128

And if portalocker is indeed the culprit then this might be the relevant issue on their repository:
wolph/portalocker#92