Errors-in-Variables for deep learning: rethinking aleatoric uncertainty - supplementary material

This directory lists the source code for the article Errors-in-Variables for deep learning: rethinking aleatoric uncertainty.

Requirements

The software used to produce the results from the preprint was written in Python 3. If not already installed, the easiest way to set up Python is usually via Anaconda. To use the software, the installation of some additional packages is required. This is discussed below. To avoid any global impacts on the Python install, especially if the system interpreter is used, it might be preferable to do the following in a virtual environment, either in Anaconda or by using the venv module. The Python version used for the results in the preprint is 3.8.5.

Installing additional packages (except PyTorch)

The Python packages to use this software, except for PyTorch which we will discuss below, can be installed by using the file requirements.txt

pip install -r requirements.txt

When using Anaconda, make sure that python is installed in the virtual environment. If not, use conda install python or conda install python=3.8.5 before running the pip command.

When running into version issues:

There is an according requirements_without_versions.txt file that does not insist on the versions from the preprint. In case of a version conflict, e.g. with pre-installed packages, this file can be used instead. When still running into problems, the packages listed in requirements_without_versions.txt should be installed one after the other.

Installing PyTorch and using the GPU

Much of the source code uses the Python library PyTorch. The best way to install PyTorch will depend on individual requirements such as the availability of a GPU and the CUDA version. The corresponding command can be generated on the PyTorch website

This software can be run with or without a GPU. If a GPU is available and PyTorch is set up to use it, it will be used by the scripts described in the Training and pre-trained models Section. As the training was done using a GPU (Tesla K80), using the CPU instead might lead to different results.

The results in the preprint were created using the CUDA version 11.1.

Installing the Errors-in-Variables Package

To enhance the portability of the software, all source code that does not directly perform data loading, training or evaluation was bundled in a package that is contained in the directory EIVPackage. It can be installed by

pip install EIVPackage/

Installing this package will make 3 modules available to the python environment: EIVArchitectures (for building EiV Models), EIVTrainingRoutines (containing a general training framework), EIVGeneral (containing a single module needed for repeated sampling).

Training and pre-trained models

To avoid the need for retraining the models, we provide the ready trained network parameters for download under the following link

https://drive.google.com/
file/d/1O_5uudTLbvw_bviK1YSTbWJGPXueP_Uf/view?usp=sharing

Clicking "Download" on this site will start downloading a zipped folder saved_networks_copy.zip. Copy the content of the unzipped folder into Experiments/saved_networks. The reminder of this section can then be skipped

General comments and time required

The preprint contains results for 4 different datasets: data that follow a noisy Mexican hat, a modulated 5D polynomial (multinomial), a dataset about wine quality and the famous Boston Housing dataset. The source code contains different training scripts for each dataset and for EiV and non-EiV models. For the Mexican hat example there are two training scripts for each model.

While training of a single network takes something around an hour (for the multinomial) and a couple of minutes (for all other datasets), the scripts below loop over different Deming factors (for the EiV models), random seeds and noise levels (for the Mexican hat and multinomial) so that their execution takes substantially longer. For all datasets except the multinomial this amounts to a computational time of around a day (depending on the available resources) and for the multinomial dataset to a couple of days (around 4, again depending on the available resources). The non-EiV scrips will run substantially faster as they do not loop over Deming factors and since their algorithm is faster by a factor of around 2 for the settings used here.

Starting the training

All training scripts, together with the scripts to load the data, are contained in the folder Experiments. With the packages from above installed, the training can be started by running within Experiments

python <name-of-training-script>

where <name-of-training-script> should be replaced with one of the following:

• Mexican hat dataset: train_eiv_mexican.py (EiV) and
  train_noneiv_mexican.py (non-EiV). There are also two versions that do not loop over std_x and only use 0.07 (used for Figure 1): train_eiv_mexican_fixed_std_x.py (EiV)
  and train_noneiv_mexican_fixed_std_x.py (non-EiV).
• Multinomial dataset: train_eiv_multinomial.py (EiV) and
  train_noneiv_multinomial.py (non-EiV).
• Wine quality dataset: train_eiv_wine.py (EiV) and
  train_noneiv_wine.py (non-EiV).
• Boston Housing dataset: train_eiv_housing.py (EiV) and
  train_noneiv_housing.py (non-EiV).

Evaluation

The trained models are evaluated using the 4 Jupyter Notebooks contained within Experiments

• evaluate_mexican.ipynb for the Mexican hat dataset
• evaluate_multinomial.ipynb for the multinomial dataset (needs around 1h 45min for execution)
• evaluate_wine.ipynb for the wine quality dataset
• evaluate_housing.ipynb for the Boston Housing dataset

To start jupyter in a browser run within Experiments

jupyter notebook

and click, in the opening tab, on the notebook you want to execute. Further instructions are given in the headers of the notebooks.

All notebooks will run by default on the CPU. To use the GPU (if available) for the computations in a notebook, set the flag use_gpu in the second cell to True.

Results

All results contained in the preprint are produced by the Jupyter notebooks mentioned in the Section Evaluation. Plots are displayed in the notebooks and will, in addition, be saved within the folder Experiments/saved_images. The contents of Table 1 in the preprint, that is
\

\

arise from running evaluate_mexican.ipynb (for the Mexican hat columns) and evaluate_multinomial.ipynb (for the multinomial columns) with std_x equal to 0.05, 0.07 and 0.10 (see instructions within the header of the notebooks).

Contributing

Will be completed upon publication. The code will be made publically available on a repository under a BSD-like license.