Test and Cover Your Code Today!. A hands-on guide for adding a… | by Itay Bittan | Dec, 2023

I’ll start with a confession: as a software engineer, I hated and avoided writing tests for years. I came across so many dusty projects — some with no tests at all, others that had tests but those had never run as part of the CI/CD pipelines, and the last ones included really poor test coverage.

On one hand, there are plenty of reasons why we should write tests but on the other, there are more excuses (or “justifications”) why we skip those.

While aiming to be a professional, I was always jealous of those shiny 100% test coverage open-source repositories and dreamed about that in my day-to-day repositories out there. Four years ago, while grappling with myself about that matter, I found diff-cover, a great open-source project with a simple mission — cover your own changes with tests. Here is how the authors describe it:

Diff coverage is the percentage of new or modified lines that are covered by tests. This provides a clear and achievable standard for code review: If you touch a line of code, that line should be covered. Code coverage is every developer’s responsibility!

In a nutshell, diff-cover assumes that you are using Git and running a coverage tool. Using git it’s easy to get your modified line numbers and compare them with the uncovered line numbers of your favorite coverage tool. Almost all of the coverage tools can yield a unified and generic XML format, no matter what your code language is (Python, JavaScript, etc.)

To sum up, the process I have been doing until now as part of the CI was:

1. Run all tests with a coverage tool, using the pytest and pytest-cov packages:

py.test -o junit_family=xunit2 --junitxml result.xml -xv --ff --cov-config=.coveragerc --cov=<my_package> --cov-report=xml --cov-report=term <tests_package>

(note that it will create coverage.xml and result.xml report files).

2. Run diff-cover tool, using the diff-cover package:

diff-cover coverage.xml --compare-branch=origin/master

which will print something like the following output:

-------------
Diff Coverage
Diff: origin/master...HEAD, staged and unstaged changes
-------------
my_package/connections/snowflake_client.py (100%)
my_package/logic/celery_tasks/top_score_task.py (100%)
my_package/queries/build_algorithm_studio_dataframes.py (100%)
-------------
Total:   16 lines
Missing: 0 lines
Coverage: 100%
-------------

As you can see from the output above, I’ve made changes in 3 different files, and each of them is fully covered (I had to add some new tests and other existing tests already covered some of my changes).

Seeing that Diff Coverage report on each PR (Pull Request) has made everyone addicted to achieving that 100%. We want to prove that we are responsible for our changes and can cover them, instead of being perceived as losers and getting a low percentage. Moreover, as a side effect, we have experienced smaller, incremental changes in PRs, which is another best practice. That’s because everyone thinks twice before adding redundant lines of code now.

After using this methodology for a few years now, we see a constant increase in the overall coverage percentages of our repositories. As a result, there has been an increase in our production stability as well.

New GitHub Action

A few months ago, my talented colleague Asaf Gallea decided to leverage this success into a simpler yet more powerful new GitHub Action. This Action applies the same idea as diff-cover and also generates a friendly report as a comment in your Pull Request, providing links to uncovered lines in case you missed something. The new action also allows you to set a minimal coverage threshold (default to 80%) — otherwise, the status check will fail and you won’t be able to merge your changes:

In the image above we see the GitHub Action report example. There is a minimal threshold of 95%, there were 20 lines changed in this Pull Request, with 18 lines covered by tests, and two lines, 505–506 are not covered. Since we achieved only 90% coverage for the changed files, the status check failed and it’s impossible to merge this into the master branch.

Note that this report says nothing about the total coverage of the repository. It might be low (60%), and yet, any new change has to pass 95% so eventually, the total coverage will increase.

Setup tests-coverage-report action in your repository

That’s it! Now let’s add this Action to your repository within a few steps. I will assume that it’s a Python project but you can add it to projects in different programming languages as well.

In the repository’s root folder, create the .github/workflows folders if it does not yet exist. Now, within the workflows folder let’s create a new file called test.yml with the following content:

# This workflow will install Python dependencies, run tests check the coverage
name: Test with coverage
on:
pull_request:
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/setup-python@v3
with:
python-version: 3.10
- name: Install Dependencies
run: |
python -m pip install --upgrade pip
pip install pytest pytest-cov
- name: Test with pytest
run: py.test -o junit_family=xunit2 --junitxml result.xml -v --ff --cov=<my_package> --cov-report=xml --cov-report=term <my_tests>
- name: Coverage Report
if: always()
uses: aGallea/tests-coverage-report@1.3.1
with:
min-coverage-percentage: '100'
fail-under-coverage-percentage: 'true'
cobertura-path: ./coverage.xml
junit-path: ./result.xml

make sure to replace the <my_package> and <my_tests> above with your package name and the tests folder accordingly.

That’s it! if you open a new Pull Request to add this file, the action should be fired automatically and you’ll see the report:

Note that since there are no changes in the package files (source files), there are no coverage details to present. The image above was taken from my Pull Request while adding the test coverage action into one of my public repositories.

You have just completed the first action (double meaning) that will change your life and make you a better developer. Our generation is addicted to likes, claps, upvotes, and geeks like us to show off our professionalism with a 100% coverage report as well. We would love to get feedback, suggestions, and feature requests for this action that can enhance your testing experience and motivation.