# Errors-in-Variables for deep learning: rethinking aleatoric uncertainty - supplementary material
This directory lists the source code for the article `Errors-in-Variables for deep learning: rethinking aleatoric uncertainty`.
## Requirements
The software used to produce the results from the preprint was written in [Python 3](https://www.python.org/). If not already installed, the easiest way to set up Python is usually via [Anaconda](https://www.anaconda.com/). To use the software, the installation of some additional packages is required. This is discussed below. To avoid any global impacts on the Python install, especially if the system interpreter is used, it might be preferable to do the following in a virtual environment, either in [Anaconda](https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html) or by using the [venv](https://docs.python.org/3/tutorial/venv.html) module. The Python version used for the results in the preprint is 3.8.5.
### Installing additional packages (except PyTorch)
The Python packages to use this software, except for PyTorch which we will discuss below, can be installed by using the file `requirements.txt`
```
pip install -r requirements.txt
```
When using Anaconda, make sure that `python` is installed in the virtual environment. If not, use `conda install python` or `conda install python=3.8.5` before running the `pip` command.
**When running into version issues**:
There is an according `requirements_without_versions.txt` file that does not insist on the versions from the preprint. In case of a version conflict, e.g. with pre-installed packages, this file can be used instead. When still running into problems, the packages listed in `requirements_without_versions.txt` should be installed one after the other.
### Installing PyTorch and using the GPU
Much of the source code uses the Python library PyTorch. The best way to install PyTorch will depend on individual requirements such as the availability of a GPU and the CUDA version. The corresponding command can be generated on the PyTorch [website](https://pytorch.org/)
This software can be run with or without a GPU. If a GPU is available and PyTorch is set up to use it, it will be used by the scripts described in the *Training and pre-trained models* Section. As the training was done using a GPU (Tesla K80), using the CPU instead might lead to [different results](https://pytorch.org/docs/stable/notes/randomness.html).
The results in the preprint were created using the CUDA version 11.1.
### Installing the Errors-in-Variables Package
To enhance the portability of the software, all source code that does not directly perform data loading, training or evaluation was bundled in a package that is contained in the directory `EIVPackage`. It can be installed by
```
pip install EIVPackage/
```
Installing this package will make 3 modules available to the python environment: `EIVArchitectures` (for building EiV Models), `EIVTrainingRoutines` (containing a general training framework), `EIVGeneral` (containing a single module needed for repeated sampling).
## Training and pre-trained models
To avoid the need for retraining the models, we provide the ready trained network parameters for download under the following [link](https://drive.google.com/file/d/1O_5uudTLbvw_bviK1YSTbWJGPXueP_Uf/view?usp=sharing)
```
https://drive.google.com/
file/d/1O_5uudTLbvw_bviK1YSTbWJGPXueP_Uf/view?usp=sharing
```
Clicking "Download" on this site will start downloading a zipped folder `saved_networks_copy.zip`. Copy the content of the unzipped folder into
`Experiments/saved_networks`. The reminder of this section can then be skipped
### General comments and time required
The preprint contains results for 4 different datasets: data that follow a noisy Mexican hat, a modulated 5D polynomial (multinomial), a dataset about [wine quality](https://archive.ics.uci.edu/ml/datasets/wine+quality) and the famous Boston Housing dataset. The source code contains different training scripts for each dataset and for EiV and non-EiV models. For the Mexican hat example there are two training scripts for each model.
While training of a single network takes something around an hour (for the multinomial) and a couple of minutes (for all other datasets), the scripts below loop over different Deming factors (for the EiV models), random seeds and noise levels (for the Mexican hat and multinomial) so that their execution takes substantially longer. For all datasets except the multinomial this amounts to a computational time of around a day (depending on the available resources) and for the multinomial dataset to a couple of days (around 4, again depending on the available resources). The non-EiV scrips will run substantially faster as they do not loop over Deming factors and since their algorithm is faster by a factor of around 2 for the settings used here.
### Starting the training
All training scripts, together with the scripts to load the data, are contained in the folder `Experiments`. With the packages from above installed, the training can be started by running within `Experiments`
```
python <name-of-training-script>
```
where `<name-of-training-script>` should be replaced with one of the following:
+ *Mexican hat dataset*: `train_eiv_mexican.py` (EiV) and \
`train_noneiv_mexican.py` (non-EiV). There are also two versions that do not loop over `std_x` and only use 0.07 (used for Figure 1): `train_eiv_mexican_fixed_std_x.py` (EiV) \
and `train_noneiv_mexican_fixed_std_x.py` (non-EiV).
+ *Multinomial dataset*: `train_eiv_multinomial.py` (EiV) and \
`train_noneiv_multinomial.py` (non-EiV).
+ *Wine quality dataset*: `train_eiv_wine.py` (EiV) and \
`train_noneiv_wine.py` (non-EiV).
+ *Boston Housing dataset*: `train_eiv_housing.py` (EiV) and \
`train_noneiv_housing.py` (non-EiV).
## Evaluation
The trained models are evaluated using the 4 [Jupyter](https://jupyter.org/) Notebooks contained within `Experiments`
+ `evaluate_mexican.ipynb` for the Mexican hat dataset
+ `evaluate_multinomial.ipynb` for the multinomial dataset (needs around 1h 45min for execution)
+ `evaluate_wine.ipynb` for the wine quality dataset
+ `evaluate_housing.ipynb` for the Boston Housing dataset
To start `jupyter` in a browser run within `Experiments`
```
jupyter notebook
```
and click, in the opening tab, on the notebook you want to execute. Further instructions are given in the headers of the notebooks.
All notebooks will run by default on the CPU. To use the GPU (if available) for the computations in a notebook, set the flag `use_gpu` in the second cell to `True`.
## Results
All results contained in the preprint are produced by the Jupyter notebooks mentioned in the Section *Evaluation*. Plots are displayed in the notebooks and will, in addition, be saved within the folder `Experiments/saved_images`. The contents of Table 1 in the preprint, that is \
\
\
\
\
arise from running `evaluate_mexican.ipynb` (for the *Mexican hat* columns) and `evaluate_multinomial.ipynb` (for the *multinomial* columns) with `std_x` equal to 0.05, 0.07 and 0.10 (see instructions within the header of the notebooks).
## Contributing
Will be completed upon publication. The code will be made publically available on a repository under a BSD-like license.