Welcome to this Workshop !

This workshop is brought to you by PTB's IT department (Q.4)
I work for the HPC department at PTB (Q.45)
I am a physicist (QCD), data analyst (DNA), software architect (HPC)

Landscape

Presenter Notes

Overall Course Goal

Look at general programming concepts ...
... all in the view of Python
modularity, testing, documentation, GitLab integration
exercises will be around creating step by step one big project
... and will cover all aspects of the course concepts

Presenter Notes

Course Material

Available at: https://itgit.bs.ptb.de/python-course/python-advanced-material
Regularly updated
You can use git for updating

Presenter Notes

Recap of previous course

Short recapitulation with small new additions

classes
data structures
numpy
beginners' course material: https://itgit.bs.ptb.de/python-course/python-beginners-course-material
Python Doc: https://docs.python.org/3/reference/

Presenter Notes

Reminder - Classes

Python is intrinsically "object oriented" (everything is an object)
(multiple) inheritance
setup of instance with __init__ (constructor)
parent initialization with super

Presenter Notes

Reminder - Classes

 1 class Vehicle:
 2     def __init__(self):
 3         self._capacity = None
 4         self._num_passengers = None
 5 
 6     def is_full(self):
 7         if self._num_passengers == self._capacity:
 8             return True
 9         else:
10             return False
11 
12 class Car(Vehicle):
13     def __init__(self):
14         super(Car, self).__init__()  # cf. in a second ...
15         self._capacity = 5

Presenter Notes

Reminder Data Structures

list e.g. [1, 2, 4]
dict e.g. {"name": "Katrin", "age": 35}
tuple e.g. (1, 4) # point in 2d plane (x, y)-coords
set e.g. {1, 15, 42}
one more: namedtuple (next slide)

Presenter Notes

Extension for `tuples`: `namedtuple`

1 from collections import namedtuple
2 Person = namedtuple('Person', ["name", "age"])
3 florian = Person(name="Florian", age=40)   # equally: florian = Person("Florian", 40)
4 florian.name
5 >>> Florian
6 florian[0]
7 >>> Florian
8 florian.age
9 >>> 40

Presenter Notes

Reminder Numpy

Doc: https://numpy.org/doc/stable/

 1 import numpy as np
 2 >>> a = np.arange(10)**3
 3 >>> a
 4 array([  0,   1,   8,  27,  64, 125, 216, 343, 512, 729])
 5 >>> a[2]
 6 8
 7 >>> a[2:5]
 8 array([ 8, 27, 64])
 9 # equivalent to a[0:6:2] = 1000;
10 # from start to position 6, exclusive, set every 2nd element to 1000
11 >>> a[:6:2] = 1000
12 >>> a
13 array([1000,    1, 1000,   27, 1000,  125,  216,  343,  512,  729])
14 >>> a[::-1]  # reversed a
15 array([ 729,  512,  343,  216,  125, 1000,   27, 1000,    1, 1000])

Presenter Notes

Modularity

Landscape

Presenter Notes

Why Modularity?

Goals:

Split software into small, weakly dependent units (modules)
Separation of Concerns
Make software manageable and maintainable
"Divide and Conquer" principle
Make the software understandable using mind-sized chunks
Abstraction can be incorporated at ease
Reusability
Concurrent execution

Presenter Notes

Development Cost

Generally, you would want to minimize this, I guess
Cost per module vs. Integration Cost → In real life: case dependent optimum

Landscape

Presenter Notes

Modularity in Python

Packages & Libraries
- "the ecosystem", pypi.org, conda, pip, etc.
Package with Modules my_module.py
- structure on file system level
Classes class MyClass
- (multiple) Inheritance, Interfaces, Traits, Method Specialization
Functions def my_add(x, y)
- may be also local
Additionally: Decorators (to come...), Lambdas, etc.

Presenter Notes

Packaging I

basis of a good packaging is a good file structure
separate off the tests
have a README that explains the package
if Open Source: Provide the license
separate infrastructure (e.g. documentation bootstrapping)

Presenter Notes

Packaging I Example

A very basic sample package 'my-package' could look like this

```
<Package my-package>
    ├── mypackage
    │   └── __init__.py
    ├── docs
    ├── LICENSE
    ├── README.rst
    ├── setup.py  ( later ... )
    └── tests
        └── __init__.py
```

Optional: requirements.txt for development dependencies (or in setup.py)

Presenter Notes

Our Project - A Bitmap Filtering Package

bitmap: simple 2d structure of pixel information of an image
implement several different "filters", which modify the pixel information
integrate testing and documentation
auto-execute testing and doc creation with GitLab
make the package be installable (through pip)

Presenter Notes

Hands On

Exercise Day-1-Ex-1

Tasks: Create a package for "bitmap-filter"

Use above discussed sample file and directory structure and create a barebone package
Add a license of your choice and link it in your README
git commit your created package
From now on: always commit changes to your repository after exercises and changes

Presenter Notes

DRY - Don't Repeat Yourself

Landscape

Every piece of knowledge must have a single, unambiguous, authoritative representation within a system

-- A. Hunt & D.Thomas in The pragmatic Programmer

Presenter Notes

DRY - Motivation

Why?

difficult to change consistently duplicated code (logic)
therefore hard to maintain
sneak in of slight changes makes it even worse
almost impossible to refactor if it happened (hope you have good test coverage!)

Further in a Team: - Gives rise to unmotivated local adaptions of repeated code -> clutter

Thus: - Already the first <Ctrl>-<v> usually is the bad one! - Early (but not premature!) abstractions

Presenter Notes

Example 1 - DRY?

1 def check_area(b):
2     if (1 / 2 * 3.14 * b * b) < 0.2:
3         return True
4     elif (1 / 2 * 3.14 * b * b) < 0.8:
5         return False
6     elif (1 / 2 * 3.14 * b * b) < 2.0:
7         return True
8     else:
9         return False

Presenter Notes

This code is not DRY.

Example 2 - DRY?

 1 class CsvValidation:
 2 
 3     def validate_product(self, product : dict):
 4         if 'color' not in product:
 5             raise RuntimeError(
 6                 'Import fail: the product attribute color is missing')
 7 
 8         if 'size' not in product:
 9             raise RuntimeError(
10                 'Import fail: the product attribute size is missing')
11 
12         if 'type' not in product:
13             raise RuntimeError(
14                 'Import fail: the product attribute type is missing')

Presenter Notes

This code is DRY. But there is some code duplication.

Example 3 - DRY?

1 class CsvValidation:
2 
3     def validate_product(self, product : dict):
4         for property in ['color', 'size', 'type']:
5             if property not in product:
6                 raise RuntimeError(
7                     'Import fail: the product attribute {} is missing'.format(property))

Presenter Notes

This code is DRY. No duplications - but more difficult to understand! There is a tradeoff.

Example 4 - DRY?

 1 class ProductInterface:
 2 
 3     def display_price(self):
 4         raise NotImplementedError(
 5                 "You forgot to implement the displayPrice method")
 6 
 7 class PlasticDuck(ProductInterface):
 8 
 9     def __init__(self, price):
10         self._price = price
11 
12     def display_price(self):
13         print("The price of this plastic"
14               "duck is {} euros!".format(self._price))
15 
16 plast_duck = PlasticDuck(2)
17 plast_duck.displayPrice()

Think about the fact that the word 'price' appears 9! times

Presenter Notes

This code is DRY.

DRY Summary

Try to keep your code DRY
Note that there is a tradeoff
Don't make your code WET since this would mean:
- Write Everything Twice
- We Edit Terribly
- Waste Everyone's Time

Presenter Notes

Hands On

Exercise Day-1-Ex-2: Copy day-1/exercise-2.py

Tasks: Read a bitmap image

Refactor the sequential unstructured code into a class Bitmap. Follow the provided skeleton of the class and separate the concerns
Dry the code - no hard-coded constants
Read width, height and byte offset of pixel array from the input bitmap
Install the Pillow package
Create a main.py with the test code
Hint: bitmap internal structure: https://en.wikipedia.org/wiki/BMP_file_format

Presenter Notes

Interfaces

In object-oriented programming, an interface or protocol type is a data type describing a set of method signatures, the implementations of which may be provided by multiple classes that are otherwise not necessarily related to each other. A class which provides the methods listed in a protocol is said to adopt the protocol, or to implement the interface.

-- Wikipedia article: "Interface (object-oriented programming)"

Presenter Notes

Interfaces

Landscape

Presenter Notes

Interfaces in Python

A simple approach:

 1 class Interface(object):
 2 
 3     def method1(self):
 4         raise NotImplementedError("this is not implemented") # -> provoke runtime error
 5 
 6 class Concrete(Interface):
 7 
 8     def method1(self):
 9         print("method 1 is implemented here")
10         # ...

Presenter Notes

Interfaces in Python with `abc`

abc == "Abstract base class"

A better approach assuring implementation before any execution starts:

 1 import abc
 2 
 3 class Interface(object, metaclass=abc.ABCMeta):
 4 
 5     @abc.abstractmethod
 6     def method1(self):
 7         pass
 8 
 9 class Concrete(Interface):
10 
11     def method1(self):
12         print("method 1 is implemented here")
13         # ...

Presenter Notes

Hands On

Exercise Day-1-Ex-3: Copy day-1/exercise-3.py and `day-1/PTBcolor.bmp`

Tasks:

Use namedtuple to create a simple interface for a "pixel" (holding the information of red, green and blue color value) in class Bitmap
Write an abc-based interface class for a 2d addressable object in another file addressable2d.py
Make class Bitmap implement this interface, check that you cannot instantiate the class if not all methods of the interface are implemented
Hint: Note the lines that have been indicated to be changed

Presenter Notes

Decorators

Decorators in general:

mixin pattern for functions
you may modify the function externally
define your own my_decorator and use syntax sugar @my_decorator to call

Presenter Notes

Function Decorators

 1 def smart_divide(func):
 2     def inner(a, b):
 3         print("I am going to divide", a, "and", b)
 4         if b == 0:
 5             raise ZeroDivisionError("Whoops! cannot divide")
 6 
 7         return func(a, b)
 8     return inner
 9 
10 @smart_divide
11 def divide(a, b):
12     print(a/b)

Presenter Notes

Special method decorator: `@staticmethod`

announces static, i.e. it can be called "on the class" and without an instance
useful for methods that:
- are conceptually linked with class
- but: do not depend on object state
- are useful for external use
Example: The method get_num_wheels of a class Car may be static
Counter-Example: The method get_num_broken_wheels may not be static

Presenter Notes

Classes : @staticmethod

 1 class Person:
 2     def __init__(self, age):
 3         self.age = age
 4 
 5     @staticmethod
 6     def age_is_adult(age_in):
 7         return age_in > 18
 8 
 9     def is_adult(self):
10         return self.age_is_adult(self.age)
11 
12 
13 res = Person.age_is_adult(12)
14 print('Is person adult: ', res)
15 
16 me = Person(35)
17 print('I am adult: ', me.is_adult())

Presenter Notes

Special method decorator: `@classmethod`

returns an instance of the class
can be viewed (used) as alternative constructors
callable also from within the class itself

Presenter Notes

Classes : `@classmethod`

 1 import date
 2 class Person():
 3     species='homo_sapiens' # This is class variable
 4     def __init__(self, age):
 5         self.age = age
 6 
 7     @classmethod
 8     def create_with_birth_year(cls, birth_year):
 9         return cls(date.today().year - birth_year)
10 
11     @classmethod
12     def print_species(cls):
13         print('species: {}'.format(cls.species))
14 
15 class Tutor(Person):
16     pass
17 
18 me = Tutor.create_with_birth_year(1982)
19 print(type(me))

Presenter Notes

Hands On

Exercise Day-1-Ex-4:

Tasks:

Add a @classmethod constructor to your Bitmap class that creates an empty bitmap from input width and height. The pixel array of correct dimension shall be created and pixels shall be initialized to color "white"
Add another @classmethod constructor to your Bitmap class that creates a copy from another input instance of class Bitmap

Table of Contents	t
Exposé	ESC
Presenter View	p
Source Files	s
Slide Numbers	n
Toggle screen blanking	b
Show/hide next slide	c
Notes	2
Help	h

Short recapitulation with small new additions

Exercise Day-1-Ex-1

Exercise Day-1-Ex-2: Copy day-1/exercise-2.py

Exercise Day-1-Ex-3: Copy day-1/exercise-3.py and day-1/PTBcolor.bmp

Exercise Day-1-Ex-4:

Table of Contents

Help

Exercise Day-1-Ex-3: Copy day-1/exercise-3.py and `day-1/PTBcolor.bmp`