Welcome to this Workshop !

  • This workshop is brought to you by PTB's IT department (Q.4)
  • I work for the HPC department at PTB (Q.45)
  • I am a physicist (QCD), data analyst (DNA), software architect (HPC)

Landscape

Presenter Notes

Overall Course Goal

  • Look at general programming concepts ...
  • ... all in the view of Python
  • modularity, testing, documentation, GitLab integration
  • exercises will be around creating step by step one big project
  • ... and will cover all aspects of the course concepts

Presenter Notes

Course Material

  • Available at: https://itgit.bs.ptb.de/python-course/python-advanced-material
  • Regularly updated
  • You can use git for updating

Presenter Notes

Recap of previous course

Short recapitulation with small new additions

  • classes
  • data structures
  • numpy

  • beginners' course material: https://itgit.bs.ptb.de/python-course/python-beginners-course-material

  • Python Doc: https://docs.python.org/3/reference/

Presenter Notes

Reminder - Classes

  • Python is intrinsically "object oriented" (everything is an object)
  • (multiple) inheritance
  • setup of instance with __init__ (constructor)
  • parent initialization with super

Presenter Notes

Reminder - Classes

 1 class Vehicle:
 2     def __init__(self):
 3         self._capacity = None
 4         self._num_passengers = None
 5 
 6     def is_full(self):
 7         if self._num_passengers == self._capacity:
 8             return True
 9         else:
10             return False
11 
12 class Car(Vehicle):
13     def __init__(self):
14         super(Car, self).__init__()  # cf. in a second ...
15         self._capacity = 5

Presenter Notes

Reminder Data Structures

  • list e.g. [1, 2, 4]
  • dict e.g. {"name": "Katrin", "age": 35}
  • tuple e.g. (1, 4) # point in 2d plane (x, y)-coords
  • set e.g. {1, 15, 42}
  • one more: namedtuple (next slide)

Presenter Notes

Extension for tuples: namedtuple

1 from collections import namedtuple
2 Person = namedtuple('Person', ["name", "age"])
3 florian = Person(name="Florian", age=40)   # equally: florian = Person("Florian", 40)
4 florian.name
5 >>> Florian
6 florian[0]
7 >>> Florian
8 florian.age
9 >>> 40

Presenter Notes

Reminder Numpy

Doc: https://numpy.org/doc/stable/

 1 import numpy as np
 2 >>> a = np.arange(10)**3
 3 >>> a
 4 array([  0,   1,   8,  27,  64, 125, 216, 343, 512, 729])
 5 >>> a[2]
 6 8
 7 >>> a[2:5]
 8 array([ 8, 27, 64])
 9 # equivalent to a[0:6:2] = 1000;
10 # from start to position 6, exclusive, set every 2nd element to 1000
11 >>> a[:6:2] = 1000
12 >>> a
13 array([1000,    1, 1000,   27, 1000,  125,  216,  343,  512,  729])
14 >>> a[::-1]  # reversed a
15 array([ 729,  512,  343,  216,  125, 1000,   27, 1000,    1, 1000])

Presenter Notes

Modularity

Landscape

Presenter Notes

Why Modularity?

Goals:

  • Split software into small, weakly dependent units (modules)
  • Separation of Concerns
  • Make software manageable and maintainable
  • "Divide and Conquer" principle
  • Make the software understandable using mind-sized chunks
  • Abstraction can be incorporated at ease
  • Reusability
  • Concurrent execution

Presenter Notes

Development Cost

  • Generally, you would want to minimize this, I guess
  • Cost per module vs. Integration Cost → In real life: case dependent optimum

Landscape

Presenter Notes

Modularity in Python

  • Packages & Libraries
    • "the ecosystem", pypi.org, conda, pip, etc.
  • Package with Modules my_module.py
    • structure on file system level
  • Classes class MyClass
    • (multiple) Inheritance, Interfaces, Traits, Method Specialization
  • Functions def my_add(x, y)
    • may be also local
  • Additionally: Decorators (to come...), Lambdas, etc.

Presenter Notes

Packaging I

  • basis of a good packaging is a good file structure
  • separate off the tests
  • have a README that explains the package
  • if Open Source: Provide the license
  • separate infrastructure (e.g. documentation bootstrapping)

Presenter Notes

Packaging I Example

A very basic sample package 'my-package' could look like this

```
<Package my-package>
    ├── mypackage
    │   └── __init__.py
    ├── docs
    ├── LICENSE
    ├── README.rst
    ├── setup.py  ( later ... )
    └── tests
        └── __init__.py
```

Optional: requirements.txt for development dependencies (or in setup.py)

Presenter Notes

Our Project - A Bitmap Filtering Package

  • bitmap: simple 2d structure of pixel information of an image
  • implement several different "filters", which modify the pixel information
  • integrate testing and documentation
  • auto-execute testing and doc creation with GitLab
  • make the package be installable (through pip)

Presenter Notes

Hands On

Exercise Day-1-Ex-1

Tasks: Create a package for "bitmap-filter"

  • Use above discussed sample file and directory structure and create a barebone package
  • Add a license of your choice and link it in your README
  • git commit your created package
  • From now on: always commit changes to your repository after exercises and changes

Presenter Notes

DRY - Don't Repeat Yourself

Landscape

Every piece of knowledge must have a single, unambiguous, authoritative representation within a system

-- A. Hunt & D.Thomas in The pragmatic Programmer

Presenter Notes

DRY - Motivation

Why?

  • difficult to change consistently duplicated code (logic)
  • therefore hard to maintain
  • sneak in of slight changes makes it even worse
  • almost impossible to refactor if it happened (hope you have good test coverage!)

Further in a Team: - Gives rise to unmotivated local adaptions of repeated code -> clutter

Thus: - Already the first <Ctrl>-<v> usually is the bad one! - Early (but not premature!) abstractions

Presenter Notes

Example 1 - DRY?

1 def check_area(b):
2     if (1 / 2 * 3.14 * b * b) < 0.2:
3         return True
4     elif (1 / 2 * 3.14 * b * b) < 0.8:
5         return False
6     elif (1 / 2 * 3.14 * b * b) < 2.0:
7         return True
8     else:
9         return False

Presenter Notes

This code is not DRY.

Example 2 - DRY?

 1 class CsvValidation:
 2 
 3     def validate_product(self, product : dict):
 4         if 'color' not in product:
 5             raise RuntimeError(
 6                 'Import fail: the product attribute color is missing')
 7 
 8         if 'size' not in product:
 9             raise RuntimeError(
10                 'Import fail: the product attribute size is missing')
11 
12         if 'type' not in product:
13             raise RuntimeError(
14                 'Import fail: the product attribute type is missing')

Presenter Notes

This code is DRY. But there is some code duplication.

Example 3 - DRY?

1 class CsvValidation:
2 
3     def validate_product(self, product : dict):
4         for property in ['color', 'size', 'type']:
5             if property not in product:
6                 raise RuntimeError(
7                     'Import fail: the product attribute {} is missing'.format(property))

Presenter Notes

This code is DRY. No duplications - but more difficult to understand! There is a tradeoff.

Example 4 - DRY?

 1 class ProductInterface:
 2 
 3     def display_price(self):
 4         raise NotImplementedError(
 5                 "You forgot to implement the displayPrice method")
 6 
 7 class PlasticDuck(ProductInterface):
 8 
 9     def __init__(self, price):
10         self._price = price
11 
12     def display_price(self):
13         print("The price of this plastic"
14               "duck is {} euros!".format(self._price))
15 
16 plast_duck = PlasticDuck(2)
17 plast_duck.displayPrice()

Think about the fact that the word 'price' appears 9! times

Presenter Notes

This code is DRY.

DRY Summary

  • Try to keep your code DRY
  • Note that there is a tradeoff
  • Don't make your code WET since this would mean:
    • Write Everything Twice
    • We Edit Terribly
    • Waste Everyone's Time

Presenter Notes

Hands On

Exercise Day-1-Ex-2: Copy day-1/exercise-2.py

Tasks: Read a bitmap image

  • Refactor the sequential unstructured code into a class Bitmap. Follow the provided skeleton of the class and separate the concerns
  • Dry the code - no hard-coded constants
  • Read width, height and byte offset of pixel array from the input bitmap
  • Install the Pillow package
  • Create a main.py with the test code
  • Hint: bitmap internal structure: https://en.wikipedia.org/wiki/BMP_file_format

Presenter Notes

Interfaces

In object-oriented programming, an interface or protocol type is a data type describing a set of method signatures, the implementations of which may be provided by multiple classes that are otherwise not necessarily related to each other. A class which provides the methods listed in a protocol is said to adopt the protocol, or to implement the interface.

-- Wikipedia article: "Interface (object-oriented programming)"

Presenter Notes

Interfaces

Landscape

Presenter Notes

Interfaces in Python

A simple approach:

 1 class Interface(object):
 2 
 3     def method1(self):
 4         raise NotImplementedError("this is not implemented") # -> provoke runtime error
 5 
 6 class Concrete(Interface):
 7 
 8     def method1(self):
 9         print("method 1 is implemented here")
10         # ...

Presenter Notes

Interfaces in Python with abc

abc == "Abstract base class"

A better approach assuring implementation before any execution starts:

 1 import abc
 2 
 3 class Interface(object, metaclass=abc.ABCMeta):
 4 
 5     @abc.abstractmethod
 6     def method1(self):
 7         pass
 8 
 9 class Concrete(Interface):
10 
11     def method1(self):
12         print("method 1 is implemented here")
13         # ...

Presenter Notes

Hands On

Exercise Day-1-Ex-3: Copy day-1/exercise-3.py and day-1/PTBcolor.bmp

Tasks:

  • Use namedtuple to create a simple interface for a "pixel" (holding the information of red, green and blue color value) in class Bitmap
  • Write an abc-based interface class for a 2d addressable object in another file addressable2d.py
  • Make class Bitmap implement this interface, check that you cannot instantiate the class if not all methods of the interface are implemented
  • Hint: Note the lines that have been indicated to be changed

Presenter Notes

Decorators

Decorators in general:

  • mixin pattern for functions
  • you may modify the function externally
  • define your own my_decorator and use syntax sugar @my_decorator to call

Presenter Notes

Function Decorators

 1 def smart_divide(func):
 2     def inner(a, b):
 3         print("I am going to divide", a, "and", b)
 4         if b == 0:
 5             raise ZeroDivisionError("Whoops! cannot divide")
 6 
 7         return func(a, b)
 8     return inner
 9 
10 @smart_divide
11 def divide(a, b):
12     print(a/b)

Presenter Notes

Special method decorator: @staticmethod

  • announces static, i.e. it can be called "on the class" and without an instance
  • useful for methods that:
    • are conceptually linked with class
    • but: do not depend on object state
    • are useful for external use
  • Example: The method get_num_wheels of a class Car may be static
  • Counter-Example: The method get_num_broken_wheels may not be static

Presenter Notes

Classes : @staticmethod

 1 class Person:
 2     def __init__(self, age):
 3         self.age = age
 4 
 5     @staticmethod
 6     def age_is_adult(age_in):
 7         return age_in > 18
 8 
 9     def is_adult(self):
10         return self.age_is_adult(self.age)
11 
12 
13 res = Person.age_is_adult(12)
14 print('Is person adult: ', res)
15 
16 me = Person(35)
17 print('I am adult: ', me.is_adult())

Presenter Notes

Special method decorator: @classmethod

  • returns an instance of the class
  • can be viewed (used) as alternative constructors
  • callable also from within the class itself

Presenter Notes

Classes : @classmethod

 1 import date
 2 class Person():
 3     species='homo_sapiens' # This is class variable
 4     def __init__(self, age):
 5         self.age = age
 6 
 7     @classmethod
 8     def create_with_birth_year(cls, birth_year):
 9         return cls(date.today().year - birth_year)
10 
11     @classmethod
12     def print_species(cls):
13         print('species: {}'.format(cls.species))
14 
15 class Tutor(Person):
16     pass
17 
18 me = Tutor.create_with_birth_year(1982)
19 print(type(me))

Presenter Notes

Hands On

Exercise Day-1-Ex-4:

Tasks:

  • Add a @classmethod constructor to your Bitmap class that creates an empty bitmap from input width and height. The pixel array of correct dimension shall be created and pixels shall be initialized to color "white"
  • Add another @classmethod constructor to your Bitmap class that creates a copy from another input instance of class Bitmap

Presenter Notes

Day-1 Recap

  • Modularity
  • Interfaces in Python
  • Package structure
  • DRY

Thanks!

Presenter Notes