diff --git a/Experiments/results/figures/Description.md b/Experiments/results/figures/Description.md new file mode 100644 index 0000000000000000000000000000000000000000..7ddc3e1b4640c0e56ed3024964156c4d221a001a --- /dev/null +++ b/Experiments/results/figures/Description.md @@ -0,0 +1 @@ +This folder will contain all images that are produced by the scripts in `Experiments`. diff --git a/README.md b/README.md index 11bb840651f4a26d5b4fc09d78555994dc92f5b5..4a1166aa7f739a5b18c9d847ba489375d2477738 100644 --- a/README.md +++ b/README.md @@ -1,10 +1,10 @@ -# Errors-in-Variables for deep learning: rethinking aleatoric uncertainty - supplementary material +# Aleatoric uncertainty for Errors-in-Variables models in deep regression -This directory lists the source code for the article `Errors-in-Variables for deep learning: rethinking aleatoric uncertainty`. +This directory lists the source code for the article `Aleatoric uncertainty for Errors-in-Variables models in deep regression`. ## Requirements -The software used to produce the results from the preprint was written in [Python 3](https://www.python.org/). If not already installed, the easiest way to set up Python is usually via [Anaconda](https://www.anaconda.com/). To use the software, the installation of some additional packages is required. This is discussed below. To avoid any global impacts on the Python install, especially if the system interpreter is used, it might be preferable to do the following in a virtual environment, either in [Anaconda](https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html) or by using the [venv](https://docs.python.org/3/tutorial/venv.html) module. The Python version used for the results in the preprint is 3.9.7. +The software used to produce the results from the article was written in [Python 3](https://www.python.org/). If not already installed, the easiest way to set up Python is usually via [Anaconda](https://www.anaconda.com/). To use the software, the installation of some additional packages is required. This is discussed below. To avoid any global impacts on the Python install, especially if the system interpreter is used, it might be preferable to do the following in a virtual environment, either in [Anaconda](https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html) or by using the [venv](https://docs.python.org/3/tutorial/venv.html) module. The Python version used for the results in the preprint is 3.9.7. ### Installing additional packages (except PyTorch) The Python packages to use this software, except for PyTorch which we will discuss below, can be installed by using the file `requirements.txt` @@ -22,7 +22,7 @@ There is an according `requirements_without_versions.txt` file that does not ins ### Installing PyTorch and using the GPU Much of the source code uses the Python library PyTorch. The best way to install PyTorch will depend on individual requirements such as the availability of a GPU and the CUDA version. The corresponding command can be generated on the PyTorch [website](https://pytorch.org/) -This software can be run with or without a GPU. If a GPU is available and PyTorch is set up to use it, it will be used by the scripts described in the *Training and pre-trained models* Section. As the training was done using a GPU (Tesla K80), using the CPU instead might lead to [different results](https://pytorch.org/docs/stable/notes/randomness.html). +This software can be run with or without a GPU. If a GPU is available and PyTorch is set up to use it, it will be used by the scripts described in the *Training and pre-trained models* Section. As the training was done using a GPU (Tesla K80 and V100), using the CPU instead might lead to [different results](https://pytorch.org/docs/stable/notes/randomness.html). The results in the preprint were created using the CUDA version 11.1. @@ -33,10 +33,40 @@ To enhance the portability of the software, all source code that does not direct pip install EIVPackage/ ``` -Installing this package will make 3 modules available to the python environment: `EIVArchitectures` (for building EiV Models), `EIVTrainingRoutines` (containing a general training framework), `EIVGeneral` (containing a single module needed for repeated sampling). +Installing this package will make 4 modules available to the python environment: `EIVArchitectures` (for building EiV Models), `EIVTrainingRoutines` (containing a general training framework), `EIVGeneral` and `EIVData`. -## Missing -+ Tell to create folders, like `Experiments/results`, `Experiments/results/figures/` + +## Running scripts + +All scripts are designed to run inside the `Experiments` folder. + +### Training + +For training the following two scripts can be used + ++ `train_eiv.py`: to train an EiV model. ++ `train_noneiv.py`: to train a non-EiV model + +For training they use the JSON-configurations in `configurations` according to the provided `--data` keyword, e.g. + +``` +python train_eiv --data sine +``` + +The keyword to `--data` should be the *short dataname*, cf. the JSON configuration files in `configurations`. + + +### Evaluation + +For evaluation of the trained networks the following scripts can be used + ++ `create_tabular.py` ++ `evaluate_metrics.py` ++ `plot_coverage_vs_q.py` ++ `plot_deviation_scatter.py` ++ `plot_prediction.py` ++ `plot_summary.py` ++ `plot_variety_of_coverage_plots.py` ## Contributing diff --git a/requirements.txt b/requirements.txt index 7d2426d016e90ca866a95f5afc547dead7905222..55a2643590e4fef99eb74add8ca716524f7167e5 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,7 +1,5 @@ -matplotlib==3.3.4 -numpy==1.20.2 -pandas==1.2.4 -scipy==1.6.3 -sklearn==0.0 -jupyter==1.0.0 -tqdm==4.59.0 +matplotlib==3.4.3 +numpy==1.21.2 +pandas==1.3.4 +scipy==1.7.1 +tqdm==4.62.3