General refactoring and cleanup of the repo

[gianlucatruda]

TODO:

• Update requirements.txt (see PR Refactoring of project structure and support scripts #22)
• Improved documentation in data_loader and preprocessing
• Add a proper .gitignore file (see PR Refactoring of project structure and support scripts #22)
• Structure the first level of project directories (see PR Refactoring of project structure and support scripts #22)
• Remove video_data files from LFS and store as a .zip on Dropbox (see PR Refactoring of project structure and support scripts #22)

[gianlucatruda]

Proposed project structure to make the repo more modular and accessible:

Spo2_evaluation/
├── README.md
├── evaluation
│   ├── README.md
│   ├── Spo2_evaluation.py
│   ├── blant_altman.py
│   ├── test.py
│   └── test_normal.py
├── misc
│   ├── VideoDataAcquisition.java
│   └── fetch_data.py
├── modelling
│   ├── NN_models
│   ├── healthwatcher
│   ├── lamonaca_and_nemcova
│   └── wang_2017
├── preprocessed_data
│   ├── README.md
│   ├── nemcova_data
│   └── sample_data
├── preprocessing
│   ├── README.md
│   ├── data_loader.py
│   ├── structure.py
│   └── utils.py
└── requirements.txt

[MalcolmMielle]

I think we should remove the videos inside the video_data folder from the repo entirely and instead add a bunch of loading scripts in Python so we don't increase the size of the repo too much with videos.

I see multiple solution scripts we could make available:

• Local load of the dataset from Dropbox or other provider
• Batch load of a subset of a dataset from an online source so that the user doesn't have to download all the data in one go.

[gianlucatruda]

I think we should remove the videos inside the video_data folder from the repo entirely and instead add a bunch of loading scripts in Python so we don't increase the size of the repo too much with videos.

I agree! I'll have a look at ways we could request, unzip, and arrange the data via a simple script.

[gianlucatruda]

I've started reorganising the project in the refactoring branch. PR #22

[LittlePea13]

Maybe add the NN models in the modelling folder?

[gianlucatruda]

Maybe add the NN models in the modelling folder?

@LittlePea13 Yes! Thanks! They weren't yet merged when I was doing this initially, but I've added them now.

[gianlucatruda]

@MalcolmMielle I've added a script for fetching and unzipping the data files

https://github.com/CoVital-Project/Spo2_evaluation/blob/refactoring/misc/fetch_data.py