Folder structure

[EstherPlomp]

Share your Folder Structure based on the CookieCutter assignment or manual folder creation:

[EstherSpruit]

I don't understand this assignment. I managed to do the cookie-cutter stuff, but now I have empty files in some folder. If I start copying my normal folders and document into this, then I will have to store everything twice right? This is not the intention, right?

I already have my own folders for my documents, should I then share this? but how can I share this? I looked at the tree command, I couldn't find how to do this with windows. Should I then just use a screen shot of Windows Verkenner, but then the actual documents are not visible

(PS the link to "information about hidden folders on WIndows" does not work.)

[EstherPlomp]

Hi @EstherSpruit: This shared screenshot is perfect! The idea is to share the folder structure, not necessarily all the files: you'll share your file naming convention through assignment 2 (als het goed is :)).

The intention is indeed not to have the files stored in two locations: you can have a look at the cookiecutter template structure to see if that would be helpful for you, and if so, you can move your files into this folder structure format. If you don't find it helpful, I would leave it as it is! Cookiecutter can be helpful when you have a lot of students working on similar projects and you want them to use the same folder structures. In that case it can help to set up your own template and use that instead of reinventing the wheel per student.

Thanks for the broken link comment: I'll check it out!

Some feedback on your folder structure:

• Be careful with using special signs such as & and , in folder names - this doesn't always go well in file transfers or different operating systems
• I don't think you're working with analysis scripts, otherwise I would say that it is better to not have spaces in file names - spaces are not machine readable and scripts can run into problems. Again, if you're not using scripts/code I wouldn't worry too much about it.
• I think most of your file names are very self explanatory, which is great! There are a couple of inconsistencies though, for example, you don't use capitals/lower letters consistently everywhere, and I spot some spelling mistakes? (experimentes should probably experiments? klein stud should probably be klein stuk? adn -> and?) I would also suggest to find a common structure for things such as klein stuk, you now use klein stuk, stukje. From what I read here this could be the same, but if it is not maybe the names should be more different.
• Why have you gathered SAS and IT and Irradiation plans all in the same folder? It seems like these are three different topics?
• For your MEP/BEP projects: one student also has the topic behind their names listed, and then the other one not - is there a reason for that distinction?
• I am now curious what jab ref means? :) Is that all your literature? And if so, why is the main folder not called Literature?

You don't have to answer all of these questions: these are mainly for your own reflection and to see if it makes sense to update your folder structure or not. Although I would now like to know what jab ref means :)

[EstherSpruit]

Thank you for the comments. I had another look at my folders and used your suggestions for improvements :).
Although I do still have to make a better distinction between raw data, analysis, and outcome.

Jabref is indeed my literature, (named after the program I am using to organize my literature). I started with a folder named literature trying to organize the literature myself. Then tried Jabref (and some other programs that did not work). I continued with Jabref and did not change the folder name. (My computer did not allow me to change the name because it said it was in use, even though I closed the program. Furthermore, I was afraid, that the program and/or the links to the pdf-files would stop working if I changed the document or folder name...) However, now I changed the name to literature (and everything still works :) ).

[EstherPlomp]

Ahh! Thanks for explaining! Glad to hear that changing the folder name did not break your workflow! Happy long weekend!

[abermejillo]

Hi! I checked and my folder configuration is quite similar to the one in the post:

Following Esther's example, I share a screenshot of my folder structure:

The main difference is that in the folder of the project, you always find a power point that explains things about the project, and analyzed data, and so forth.
What I am missing and I am working on right now is a more proper saving of the data such that all the metadata is properly added, and the measurement code itself forces me to include that metadata. Also, it includes through a GitHub repository a reference to a copy of the code that was used to measure that data, which I think it's pretty handy. I am doing this in collaboration with a new startup called Balthazar which is developing a project to handle these things.

For my PhD I made this folder structure (which is fully backed up in OneDrive):

[EstherPlomp]

Thanks for sharing @abermejillo! This looks very nice and organised! I hope you can still get something out of the course ;)

measurement code itself forces me to include that metadata. Also, it includes through a GitHub repository a reference to a copy of the code that was used to measure that data,

That sounds very helpful! I'm curious whether this will eventually be wider accessible?

And a couple of small notes/questions on your PhD folder:

• What is C&C? (& is one of the special characters by the way!). It is fine if this is clear to you since you probably don't collaborate with anyone on these folders
• What made you decide between A1, A2, A3 and then not have this numbering in the other folders?

[abermejillo]

C&C is the Culture and Community committee. So it's nothing all that important and I don't share it with anyone.

The A1. A2. A3. is basically because I added them latter, but I wanted them in the same priority as what I had for but didn't want to change all the rest. Should I change it?

[EstherPlomp]

Thanks for elaborating!

• Sounds like something important, but if you indeed don't share it with anyone the name is good: as long as it makes sense to you!
• That is a good intermediate solution! I wouldn't change it if it works for you - perhaps at some point when more folders are added and there are loads of sub letters it may be time for a reorganisation, but otherwise it sounds like a good fix for now!

[irisforkink]

My current structure does not distinguish between research lines, but between topic (e.g. lab work, design etc.):

Inside the folder I try to keep structure by adding numbers/experiment date/description:

Looking at the proposed structure I think that it might be useful to organise by research line, but this is also difficult because I do have some areas of overlap. I do think that it might help with keeping the overview because now it's mixed, for me it is very clear but it might be confusing if someone else want to find data. I'll try to re-organise soon and see if that's more effective.

[EstherPlomp]

Thanks for sharing @irisforkink! You can indeed give it a test round and see what works better for you!

No feedback from my side on the current structure: this looks super well organised, well done!

[lkresik]

Hi everyone,

Here is my folder structure. I divide the data into Projects>Experiment Types>Experiment data files/folders starting with initials and number of experiments (LKXXX).

[EstherPlomp]

Thanks for sharing @lkresik! It looks very clear! I do see some inconsistent usage of spaces (LK006_K562 vs LK018 K562), so do keep an eye on that! Spaces are not machine readable, so if you use any of those folders in analysis scripts you might run into troubles.

The folder under TEM has a very large name: do you need all of that information in the folder/directory name?

[demiteach]

within my project data, this is my file organization:

Everything is backed up in ondedrive.

Sadly, I cannot show everything that is in the folders due to confidentiality. However, I will follow the file organization for experiments that was given in the video (1: Setup, 2: raw data; 3 processed data; 4: output).

Also, all presentations, written files, etc, will begin with YYMMDD. this makes organization of folders easier.

Furthermore, the contents of the 'Writing' and 'Experiments' folder will be regularly uploaded to the U:drive, in order for colaborators to be able to trace back the data.

Hope my data organization is clear in this way.

[EstherPlomp]

Really clear, thanks for sharing!

And indeed - better to not share things in that case, as this is a public platform!

[kmertiri]

Hi, my folder structure looks like this. Inside the folders sit all the versions of my data.

[EstherPlomp]

That looks organised!

A bit of a probing question: How sure are you that the project will run between 2023-2024? Are all your project organised per year? It might make more sense to give your project a name rather than a time span as projects may get delayed. This is up to you of course!

[RiteshDas2000]

I believe my folder structure conforms with the requirements, the habit of relying on temp trash (backups of code that are obsolete but I feel anxious to delete for a long time) is something I am working on eliminating. While the contents under NiPS3 dispersion paper may seem haphazard from the first screenshot. They are spatially separated as below:

As I write modular codes that can have interchangeable modules, I often need to open different codes (there's only so much space on my monitor), I often find it easier to separate spatially my most similar codes in the same folder. Codes with significant differences are of course kept separately.

[EstherPlomp]

Interesting to see how you use the spatial location of files as a way to differentiate between codes! Does that change a lot when new files/codes are added?

I think it looks clear, and I think your file names might be a bit too long since not everything is displayed. If that doesn't bother you it is of course fine, but sometimes file transfers can go wrong if file names are too long (or contain special characters).

[RiteshDas2000]

Hi @EstherPlomp, thanks for the reply. The spatial structure does not change much when new files are added, as new files often replace old ones. I try not to have a lot of files in one folder.

The names are indeed long but to explain the reason for that, I once again have to mention that I open different codes all the time. Having descriptive names makes it easier to pull up files directly from the search engine instead of having to go through multiple folders. Your advice of adding dates would also make things much easier for me to find this way.

[anish8194]

The structure is based on the pptx I have made. All files are fully backed up in OneDrive.
COMSOL and MAtlan scripts are for simulations and they will have their own output files.
Experimental contains the experimental data.
GitLAb repository is the Github/GitLab synced file for upload and download
Presentations contains meeting notes and all my presentations for supervisor, Work Package and external consortium meetings.
docs contains all PhD related documents that I need to keep
Course contains all GS courses I have taken or am currently taking.

[EstherPlomp]

Nice overview! Thanks for sharing.

I wonder what the backup folder contains? If those are the same files that are already backed up in OneDrive in the cloud it is a bit superfluous to have another copy of the data in the same storage location - another location, such as the U:/Project drive makes more sense. In case something happens to OneDrive, all your files will still be affected, whereas if you have your back up in another location you will still have access to back up files.

There is a bit of inconsistency going in your folder naming: some of it starts with a capital and then others not. There is also an inconsistency with use of spaces, for example, for year-1 you don't have spaces, but with 'pre starting material - basic introduction' you do have a lot of spaces (also this folder name is also a bit long). No major issues, but especially when using scripts it may get inconvenient to use spaces and inconsistent capitalisation in folder directories.

[anish8194]

Thanks for you comments! That was a good catch. Yes, if I have the backup also backed up in OneDrive it would not help so what I am doing is using this backup as a sort of temporary hold which I will regularly keep updating in the sharefolder of our dept.

Yes that is true. I will name these folders more accurately and consistently so as to make retrieval easier in the future.

[Nico7575]

On my own laptop I have this structure:

In GitHub I have instead this structure for the codes:

[EstherPlomp]

Looks clear from what I can see here: although HTGR and ROM are a bit of a mystery for me. As long as you know what these folders are for it is of course not an issue. I suppose you use papers for your own work and publications for existing literature?

[linghangdelft]

My folder structure is shown below (it's mainly categorized based on the purpose and file types):

[EstherPlomp]

Thanks for sharing! I wonder how your data folder is set up? Do you separate between raw and processed data?

[BhaveshSChavan]

My data is stored based on the following folder structure. My literature studies and personal notes are structured as per time (for me it is more to see how I am progressing as well)

[EstherPlomp]

Looks very clear, thanks for sharing! You could consider removing the 'my's in front of folder names, unless you also have these folders for other people's files?

For meeting presentation files or folders for conferences: if you add the date in front of the name you could use that as a chronological sorter as well!

[rjhvanderkolk]

For my C:/ -> nameofpaper/typeofdata/datawithtimestamp
for my U:/ (my backup drive)-> complete copy of everything from C:\ +data not directly produced for a project in seperate folders u/typeofdata/datewithtimestamp+approriate name

examples : U:\KavliNanoLab\kolk\AFM\sic paper (iman)\75C-ICP(2-9-2022)
C:\Users\rjhvanderkolk\Desktop\wafer scale optics paper\ellipsometry\R2 a-SiC middle (17-4-2024)

[EstherPlomp]

Looks very clear! Do take care of using spaces if you're using these folders/directories for analysis scripts - spaces may get you in trouble there as they are not machine readable.

If you place the date in the YYYYMMDD format instead of DD-M-YYYY you can also chronologically order your files (but that is of course up to your personal preferences!)

[LonghengZhang]

This is the data folder in our experimental computer. It looks a bit chaotic but the only things that matters it the final folder that we store the data set. Since the data are backed up to the M drive, we can still process the data on our own computer, and we can have our own folder for useful data and other analysis.

[EstherPlomp]

That is a lot indeed! I wonder why some of the folders are ordered chronologically and others per element? Perhaps you don't even know as this is quite a while ago and probably someone else's decision :)

[mmsavic]

Hi everyone,

This is the folder structure I've using for a while now. I find that it works well for me as I can keep track of data fairly easily. I am not allowed to keep Philips code and data for pulse programming anywhere else apart from their VM, so that's why there's none in this folder arrangement.

[EstherPlomp]

Fantastic, thanks for sharing! And indeed - that part of your data should not be shared here 👍

[oonimabisht]

I have put in a screenshot for my folder from my group drive. Everything is backed up in onedrive. I usually save my data in DDMMYYYY format as well as I keep a note/logbook of my weekly meetings so that I am aware of my progress.

[EstherPlomp]

Thanks for sharing! It looks very clear overall!

Pending on whether you use directories for analysis scripts or not, it can be unhandy to have spaces in them. I also see a special character in one of your folder names: &. This may also cause problems, for example, when transferring data.

And another question based on your screenshot: Why is it Phantom sketch (006)? What happened to the previous five versions?

[DaphneCette]

Here is my folder structure, all files are backed up in the drive.