Developer documentation for the DIANNA project

This chapter lists the tools and practices we typically use during development. Most of our choices are motivated in the NL eScience Center Guide, specifically the Python chapter and our software guide checklist. If you’re considering contributing to DIANNA, please have a look and be sure to let us know (e.g. via an issue) if you have any questions.

Development install

# Create a virtual environment, e.g. with
python3 -m venv env

# activate virtual environment
source env/bin/activate

# make sure to have a recent version of pip and setuptools
python3 -m pip install --upgrade pip setuptools

# (from the project root directory)
# install dianna as an editable package and install development dependencies
python3 -m pip install --no-cache-dir --editable .[dev]

It’s also possible to use conda for maintaining virtual environments; in that case you can still install DIANNA and dependencies with pip.

Dependencies and Package management

DIANNA aims to support all Python 3 minor versions that are still actively maintained, currently:

Add or remove Python versions based on availability of dependencies in all versions. See the guide for more information about Python versions.

When adding new dependencies, make sure to do so as follows:

• Runtime dependencies should be added to setup.cfg in the install_requires list under [options].

• Development dependencies should be added to setup.cfg in one of the lists under [options.extras_require].

Testing and code coverage

• Tests should be put in the tests folder.

• Take a look at the existing tests and add your own meaningful tests (file: test_my_module.py) when you add a feature.

• The testing framework used is PyTest

• The project uses GitHub action workflows to automatically run tests on GitHub infrastructure against multiple Python versions

  • Workflows can be found in .github/workflows

• Relevant section in the guide

Running the tests

There are two ways to run tests.

The first way requires an activated virtual environment with the development tools installed run the following from the root directory of this repository:

pytest -v

The second is to use tox, which must be installed separately (e.g. with pip install tox), but then builds the necessary virtual environments itself by simply running:

tox

Testing with tox allows for keeping the testing environment separate from your development environment. The development environment will typically accumulate (old) packages during development that interfere with testing; this problem is avoided by testing with tox.

Testing the dashboard

The dashboard workflow can be tested using playwright.

Setup:

pip install pytest-playwright
playwright install chromium

To run the dashboard tests:

pytest -v --dashboard

To help with developing the dashboard tests, you can use the playwright code generator:

playwright codegen http://localhost:8501

Running linters locally

For linting and import sorting we use ruff, and to autoformat the code we use yapf Running the linters requires pre-commit.

# staged files only
pre-commit

# all files
pre-commit run --all-files

You can enable automatic linting and code formatting with pre-commit on each commit by enabling the git hook, like so:

pre-commit install

We also check linting errors in a GitHub Actions CI workflow.

Documentation

• Documentation should be put in the docs/ directory in the repository.

• We use Restructured Text (reST) and Google style docstrings.

  • Restructured Text (reST) primer

  • Google style docstring examples.

• The documentation is set up with the ReadTheDocs Sphinx theme.

  • Check out its configuration options.

• AutoAPI is used to generate documentation for the package Python objects.

• .readthedocs.yaml is the ReadTheDocs configuration file. When ReadTheDocs is building the documentation this package and its development dependencies are installed so the API reference can be rendered.

• Relevant section in the guide

Generating documentation

cd docs
make html

The documentation will be in docs/_build/html

If you do not have make use

sphinx-build -b html docs docs/_build/html

To find undocumented Python objects you can run

cd docs
make coverage
cat _build/coverage/python.txt

We also check for undocumented functionality in a GitHub Actions CI workflow.

To test snippets in documentation run

cd docs
make doctest

Versioning

Bumping the version across all files is done with bumpversion, e.g.

bumpversion major
bumpversion minor
bumpversion patch

Making a release

This section describes how to make a release in 4 steps:

1. Verify that the information in CITATION.cff is correct.

2. Make sure the version has been updated.

3. Run the unit tests with pytest -v or tox.

4. If applicable: list non-Python files that should be included in the distribution in MANIFEST.in.

5. Make a release on GitHub. This will trigger the release workflow, which will build and upload DIANNA as a package to PyPI. It will also trigger Zenodo into making a snapshot of the repository and sticking a DOI on it. In this project the habit is to use the release notes that can be auto-generated by Github.

Note that the build is uploaded to both pypi and test-pypi. If you trigger the workflow manually, it’s only uploaded to test-pypi, which can be useful for testing.