I really like to write tests and have a nice set up so that my test can run often and fast. Of course a considerable amount of time goes into writing code in a way so that it is (automatically) testable as well as provisioning and configuring test environments. However overall we save a lot of time by using the “test driven development” paradigm in a clever way. In this post I will clarify some points which often lead to confusion and misunderstanding.
One often is confronted with the question on why all the hoop jumping is necessary to enable the automated testing. “Would we not be faster without all these test?” a question from stakeholders might be. Or “We really love all these tests but now we have this deadline, so can we just not not write them just this time to be faster?”.
What both questions reveal is the ignorance of the fact, that there is no possibility to “not test”. It is just a question of “Who runs the tests? A human or a machine?”.
It is very important to understand, that code is always tested. If one does not write tests, the code has to be tested by hand, which usually means that the developer will write some code and then somehow “use the app” - usually by filling in forms and clicking buttons - in order to verify that the addition of their local changes to the app make the app behave in a different and expected way. So there is never “untested code”. Untested code would effectively be cargo cult. So the important aspect of testing is not if it is executed at all, but the cost to re-run the tests. One can measure this cost in development time. If the developer tests the changes by hand, these costs are extremely high! Using automated testing via scripts can push these costs (to re-run the tests) to close to zero. That is the main benefit of test driven development: Because tests are usually re-run very often, automated tests promise an insane reduction in development costs.
Additionally one should consider the following: If the costs of re-running a test are very high, teams often just resort to not re-running the tests or doing so rarely: If performing all the tests to verify that the app behaves as defined in the specifications means 5 QA-engineers clicking buttons in the app for 3 days straight, nobody is going to do this on a regular basis.
This effectively means that there will be no regression tests. This is then why, if the functionality breaks, nobody will see the regression. Once discovered by the users, fixing this regression might introduce one or more new regressions, which the developers will not notice.
Automated testing is not an additional exercise but decreases the time spent on and increases the quality of the result of software development.
As you probably know, there are so called “unit tests” and “integration tests” in software development. I have my problems with these terms because in practice, these terms describe two points at the edges of a continuous interval.
A unit test usually means that the code is tested very much in isolation. This is possible for simple pure functions that transform data. As soon as there are side effects things become more complicated. One can solve this by using dependency injection with some kind of spy functionality to see that the code would at least attempt to trigger the side effect. This is however very tedious and often tests implementation rather than functionality. As the components become more complex and use more dependencies, mocking these dependencies (the side effects) becomes ever more complicated and error prone.
On the other end of the spectrum, there are the integration tests. In theory this involves deploying the app with a complete production-like back end and then only testing behavior as a user would experience it, say by putting data into forms, clicking buttons, etc. Writing such tests is often very complex and since it takes time, expensive. If requirements change during development, the adjustment of the tests can sometimes take more time than writing the code to implement the functionality.
This is why I like to approach testing by just forgetting about these terms and choosing a sweet spot between the two extremes for the job at hand. This could mean while I have a stack with a database, an api and a front end, it might be good for the current situation to make a test that uses the database and the api, but not the front end. I could start the database and the api and then call one endpoint of the api to then check that the result of another endpoint has changed.
In my experience, blockchain-based code is especially hard to test. This is due to the fact that this kind of application usually involves a lot of parts:
In a follow up post I
will show an example of how you can use
miniwasm in order to automatically
and cheaply test crucial parts of this stack without wasting endless amounts of
time and money on manual testing.
