Table of Contents

  • Day 1: Adaptability & Extendibility

    * Modularity in python
    * DRY
    * SOLID
    * Design Patterns
    
  • Day 2: Testing, Documentation & Readability

    * Packaging
    * Unit Tests
    * Doctest
    * Sphinx
    * GitLab Integration
    
  • Day 3: Performance & Scalability

    * Introduction
    * Profiling & Benchmarking
    * Data, Do's and Dont's
    * Ahmdal's law
    * Parallelisation with `joblib` and `mpi4py`
    

Presenter Notes

A Python Package

  • Use your code easily in other projects, environments etc. without copying code
  • Pass your code to others
  • Later: Git(-Lab, -Hub) workflows: automated testing, docs generation etc.
  • Manage dependencies
  • Publish (pypi.org, anaconda.org etc.)

Presenter Notes

Basic Structure

A very basic sample package 'bitmap-filter' could look like this

```
<Package bitmap-filter>
    ├── bitmapfilter
    │   └── __init__.py
    ├── docs
    ├── LICENSE
    ├── README.rst
    ├── setup.py
    └── tests
        └── __init__.py
```

Optional: requirements.txt for development dependencies (or in setup.py)

Presenter Notes

Purpose of Files

  • LICENSE: Most important file, how can others use your code
  • README.rst (may also be in Markdown format):
    • Purpose of the package
    • How to setup / install
    • How to contribute
    • Doc or where is the doc
  • setup.py
  • Separate folders for source, test and doc files

Presenter Notes

setup.py

 1 from setuptools import setup
 2 
 3 setup(
 4     name='bitmap-filter',
 5     version='0.1.0',
 6     description='A sample project for the Python programming course',
 7     packages=['bitmap-filter'],
 8     cmdclass={},
 9     test_suite="tests",
10     install_requires=[],
11     tests_require=[],
12     extras_require={},
13 )

Presenter Notes

GitLab, Kickstart

Landscape

Presenter Notes

Hands On

Exercise Day-2/exercise-1.md

Please pull from the course materials repository:

https://itgit.bs.ptb.de/burger03/python-course-material

Presenter Notes

Testing

Presenter Notes

Testing Theory

  • Test hierarchy

    • Acceptance
    • System
    • Integration
    • Unit
  • Often split teams: Development ↔ Testing

Presenter Notes

Unit Tests

Why:

  • Guarantees (gives confidence in) proper functioning of your modules & classes
  • Detects regressions (stuff that used to work and stopped working)
  • Are helpful when developing (cf. Test Driven Development)
  • Are indispensable for larger refactoring
  • Some tests is better than no tests at all
  • Show how to use the unit

Presenter Notes

Unit Tests

How I:

  • Test scenarios
  • Test public interfaces
  • One thing a time
  • Easy to understand
  • Self-contained
  • Run fast
  • One-Click

Presenter Notes

Unit Tests

How II:

  • Good naming scheme (test modules, test cases, tests)
  • Use mock data
  • Be deterministic (fix randomness)
  • Personal opinion: Also test your (private) core algorithms
  • Publish the Good
    • pass locally before push
    • must pass before merge

Presenter Notes

Python unittest I

Linksofficial page      good tutorial

 1 # module: mymodule_test.py
 2 import unittest
 3 
 4 class MyModuleTest(unittest.TestCase):
 5     def test_one(self):
 6         self.assertEqual(2+2, 4, "The simple addition failed")
 7 
 8     def test_two(self):
 9         self.assertLessEqual(2, 3, "2 should be <= 3")
10         self.assertLessEqual(3, 3, "3 should as well be <= 3")

If you want to execute tests stand-alone from CLI, add:

1 if __name__ == '__main__':
2     unittest.main()

Presenter Notes

Python unittest II

  • Powerful and flexible
  • Fixtures: setUp, tearDown and setUpClass, tearDownClass
  • Organize your tests into one or several unittest.TestSuite
  • Easily skip tests via decorators, e.g. @unittest.skipIf
  • Run tests from CLI: python -m unittest tests/test_something.py
  • Run easily from PyCharm (→ right click on file and look for "Run 'Python tests in ...'")
  • Easily discover tests into a test suite: python -m unittest discover -p <TEST_PATTERN>

Presenter Notes

Python unittest III

Add to setup.py:

1 from setuptools import setup
2 
3 setup(
4     #...
5     test_suite="tests",  # folder containing the unit tests
6     #...
7 )

Presenter Notes

Python doctest I

Example:

 1 def my_add(a: int, b: int) -> int:
 2     """
 3     Adds two integer numbers.
 4 
 5     :param a: first value
 6     :param b: second value
 7     :return: int
 8 
 9     Example:
10 
11     >>> my_add(1, 2)
12     3
13     >>> my_add(3, 4)
14     7
15     """
16     return a + b

Presenter Notes

Short Excursion: Doc Blocks

1 def fun() -> int:
2     """Just an example for a single line doc string."""
  • single- or multi-line docstrings
  • docstring should be descriptive
  • must be kept in sync with code
  • give fast understanding of what the unit is doing (functionality, interface and return values)
  • help(fun) shows docstring in Python Console
  • Several docblock formats (Epytext, reST, Google, Numpydoc) c.f. this SO discussion
  • I use reST here

Presenter Notes

Python doctest II

Pros:

  • Test is near to the actual code tested
  • Shows how to use the unit
  • Code examples shown in compiled Doc (cf. Sphinx, later)
  • You can run these from PyCharm, too
  • Tests can be included into unittest.TestSuite

Cons:

  • Not suitable for complex tests
  • No setup / preparation step

Presenter Notes

Python doctest III

Adding doctests to your unittest.TestSuite

Doctest's unittest API

Just add this snippet to your_module.py or your_module_test.py:

1 import unittest
2 import doctest
3 import yourmodule
4 
5 
6 def load_tests(loader, tests, ignore):
7     tests.addTests(doctest.DocTestSuite(yourmodule))
8     return tests

Presenter Notes

GitLab

  • MUCH more than just a graphical frontend to Git
  • GitLab@PTB → See this Wiki-Page (by B. Ludwig)
  • Acts as:

    • Dev Discussion Platform
    • Knowledge Base (Wiki)
    • Bug / Issue Tracker
    • Workflow Executor (CI/CD, API, Webhooks, Integrations)
    • Analytics
  • Integrations, Plugins (Jira)

  • Open Source

Presenter Notes

GitLab, Typical Workflow

  • Create an "Issue" (e.g. Feature Request, Bug Report, etc.)
  • Discussion is open ...
  • Implement Feature and Tests on another branch
  • Watch test outcome
  • Create a "Merge Request"
  • Maintainer and others review and discuss code changes, possibly re-iterate
  • Maintainer merges branch

Presenter Notes

GitLab, CI / CD

CI/CD is a method to frequently deliver apps to customers by introducing automation into the stages of app development. The main concepts attributed to CI/CD are continuous integration, continuous delivery, and continuous deployment. CI/CD is a solution to the problems integrating new code can cause for development and operations teams (AKA "integration hell").

-- Redhat Web Page link

  • GitLab CI / CD Pipelines based on docker containers
  • Execution through "GitLab Runners"
  • After each git push and also scheduled
  • Can be used for: Building, Tests, Creating Deliverables (e.g. containers, bundles, binaries)
  • Definition through .gitlab-ci.yml in YAML syntax

Presenter Notes

GitLab .gitlab-ci.yml

default:
  image: <DOCKER_IMAGE_NAME>

stages:
  - test
  # further stages go here

test:
  stage: test
  script:
    - python3 -m venv venv
    # further setup and tests go here
  cache:
    paths:
      - venv
  • GitLab provides you a linter under Project → CI/CD → CI Lint (upper right corner)

Presenter Notes

More CI/CD GitLab Tasks

  • Compiling / Building
  • Linting
  • Deploying "GitLab pages"
  • Deployable containers

Presenter Notes

Hands On

Exercise Day-2/exercise-2.md

Please pull from the course materials repository:

https://itgit.bs.ptb.de/burger03/python-course-material

Presenter Notes

Documentation

Why?:

  • There are always at least 2 people working on some given code:
    • You
    • You (in 6 monthes)
  • Others want to use your code
  • Others may want to contribute to your code

https://www.writethedocs.org/guide/writing/beginners-guide-to-docs/

Presenter Notes

Documentation

Landscape

Presenter Notes

Documentation

How?:

  • Keep in mind your audiences: Users ↔ Developers
  • Be concise and simple, DRY!
  • What:
    • Which problem is solved
    • Examples: How to use
    • Link to code and issue trackers
    • FAQs
    • Contributing guide
    • How to get support
    • Installation instructions

Presenter Notes

Code Comments ...

Based on this SO blog

  • do not duplicate code
  • are no excuse for unclear code
  • unclear → possible code issue
  • should clear confusion, not create it
  • explain unidiomatic code
  • provide links to original (copied code)
  • link external references where most helpful
  • when fixing bugs
  • mark incomplete implementations

Presenter Notes

Sphinx Doc I

  • the "Python Documentation Generator"
  • multiple output formats (HTML, pdf etc.)
  • hierarchical structure (toc tree)
  • code highlighting
  • cross-referencing
  • highly flexible
  • math formulas

Presenter Notes

Sphinx Doc II

  • Installation via pip
  • Easy setup due to provided quickstart executable (just answer the questions)
  • Further configuration: conf.py: configure extensions here (cf. exercise)
  • index.rst: entry point and overall layout
  • Possible alternative to Sphinx: Doxygen (not python centric!)

Presenter Notes

Hands On

Exercise Day-2/exercise-3.md

Please pull from the course materials repository:

https://itgit.bs.ptb.de/burger03/python-course-material

Presenter Notes

Day-2 Recap

  • Created a Python package
  • Testing
  • Sphinx documentation
  • Automation with GitLab

Thanks!

Presenter Notes