From a413ecf972ba6c34aba91948d8897842dc45cc87 Mon Sep 17 00:00:00 2001 From: Joerg Martin <joerg.martin@ptb.de> Date: Thu, 17 Feb 2022 15:53:04 +0100 Subject: [PATCH] Updated README.md --- Experiments/results/figures/Description.md | 1 + README.md | 44 ++++++++++++++++++---- requirements.txt | 12 +++--- 3 files changed, 43 insertions(+), 14 deletions(-) create mode 100644 Experiments/results/figures/Description.md diff --git a/Experiments/results/figures/Description.md b/Experiments/results/figures/Description.md new file mode 100644 index 0000000..7ddc3e1 --- /dev/null +++ b/Experiments/results/figures/Description.md @@ -0,0 +1 @@ +This folder will contain all images that are produced by the scripts in `Experiments`. diff --git a/README.md b/README.md index 11bb840..4a1166a 100644 --- a/README.md +++ b/README.md @@ -1,10 +1,10 @@ -# Errors-in-Variables for deep learning: rethinking aleatoric uncertainty - supplementary material +# Aleatoric uncertainty for Errors-in-Variables models in deep regression -This directory lists the source code for the article `Errors-in-Variables for deep learning: rethinking aleatoric uncertainty`. +This directory lists the source code for the article `Aleatoric uncertainty for Errors-in-Variables models in deep regression`. ## Requirements -The software used to produce the results from the preprint was written in [Python 3](https://www.python.org/). If not already installed, the easiest way to set up Python is usually via [Anaconda](https://www.anaconda.com/). To use the software, the installation of some additional packages is required. This is discussed below. To avoid any global impacts on the Python install, especially if the system interpreter is used, it might be preferable to do the following in a virtual environment, either in [Anaconda](https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html) or by using the [venv](https://docs.python.org/3/tutorial/venv.html) module. The Python version used for the results in the preprint is 3.9.7. +The software used to produce the results from the article was written in [Python 3](https://www.python.org/). If not already installed, the easiest way to set up Python is usually via [Anaconda](https://www.anaconda.com/). To use the software, the installation of some additional packages is required. This is discussed below. To avoid any global impacts on the Python install, especially if the system interpreter is used, it might be preferable to do the following in a virtual environment, either in [Anaconda](https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html) or by using the [venv](https://docs.python.org/3/tutorial/venv.html) module. The Python version used for the results in the preprint is 3.9.7. ### Installing additional packages (except PyTorch) The Python packages to use this software, except for PyTorch which we will discuss below, can be installed by using the file `requirements.txt` @@ -22,7 +22,7 @@ There is an according `requirements_without_versions.txt` file that does not ins ### Installing PyTorch and using the GPU Much of the source code uses the Python library PyTorch. The best way to install PyTorch will depend on individual requirements such as the availability of a GPU and the CUDA version. The corresponding command can be generated on the PyTorch [website](https://pytorch.org/) -This software can be run with or without a GPU. If a GPU is available and PyTorch is set up to use it, it will be used by the scripts described in the *Training and pre-trained models* Section. As the training was done using a GPU (Tesla K80), using the CPU instead might lead to [different results](https://pytorch.org/docs/stable/notes/randomness.html). +This software can be run with or without a GPU. If a GPU is available and PyTorch is set up to use it, it will be used by the scripts described in the *Training and pre-trained models* Section. As the training was done using a GPU (Tesla K80 and V100), using the CPU instead might lead to [different results](https://pytorch.org/docs/stable/notes/randomness.html). The results in the preprint were created using the CUDA version 11.1. @@ -33,10 +33,40 @@ To enhance the portability of the software, all source code that does not direct pip install EIVPackage/ ``` -Installing this package will make 3 modules available to the python environment: `EIVArchitectures` (for building EiV Models), `EIVTrainingRoutines` (containing a general training framework), `EIVGeneral` (containing a single module needed for repeated sampling). +Installing this package will make 4 modules available to the python environment: `EIVArchitectures` (for building EiV Models), `EIVTrainingRoutines` (containing a general training framework), `EIVGeneral` and `EIVData`. -## Missing -+ Tell to create folders, like `Experiments/results`, `Experiments/results/figures/` + +## Running scripts + +All scripts are designed to run inside the `Experiments` folder. + +### Training + +For training the following two scripts can be used + ++ `train_eiv.py`: to train an EiV model. ++ `train_noneiv.py`: to train a non-EiV model + +For training they use the JSON-configurations in `configurations` according to the provided `--data` keyword, e.g. + +``` +python train_eiv --data sine +``` + +The keyword to `--data` should be the *short dataname*, cf. the JSON configuration files in `configurations`. + + +### Evaluation + +For evaluation of the trained networks the following scripts can be used + ++ `create_tabular.py` ++ `evaluate_metrics.py` ++ `plot_coverage_vs_q.py` ++ `plot_deviation_scatter.py` ++ `plot_prediction.py` ++ `plot_summary.py` ++ `plot_variety_of_coverage_plots.py` ## Contributing diff --git a/requirements.txt b/requirements.txt index 7d2426d..55a2643 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,7 +1,5 @@ -matplotlib==3.3.4 -numpy==1.20.2 -pandas==1.2.4 -scipy==1.6.3 -sklearn==0.0 -jupyter==1.0.0 -tqdm==4.59.0 +matplotlib==3.4.3 +numpy==1.21.2 +pandas==1.3.4 +scipy==1.7.1 +tqdm==4.62.3 -- GitLab