Creating a subdirectory for each neuron becomes an issue with larger models #8
Description
At least on linux, the filesystem isn't really built for 10000s of subdirectories.
You run into issues like the ls command taking 10 seconds to run or taking minutes to get the size of the directory.
This is a pretty big problem when running this on large models is already a large undertaking.
I would suggest one of two solutions
- A flat file structure where all graph files are in the same directory (maybe one subdirectory for each type of graph file) in order to minimize the number of subdirectories. I'm not actually sure this would fix the problems I mentioned. This should be tested.
- Allow grouping the graphs into single files, e.g. tar balls. This could be configured by the user. Graphs could be grouped by model, layer og just by 512 neurons or something.