This is the second part of my short series on my experiences with Elm while reimplementing Carna.io. Check out my part one for some background info.

Fuzz testing in Elm

I will assume that you are familiar with the general concept of fuzz testing aka generative testing aka property based testing. If you are not familiar with it yet, you can get a nice intro from the elm-test readme or in case you have access to frontend-masters please take a look at the fuzz testing part of Richard’s Course. It is short and simple and is enough to get you started.

Fuzz testing all the time

Now that the background should be clear. There is only one thing I want to mention in this post: Use fuzz testing, ideally in every project! Even if you think it is not worth to add it to a certain project. I don’t mean use it for everything but prefer it over unit testing as the goto tool and you will quickly be surprised by its value.

My experience in Elm

I added tests and fuzz testing to Carna just to get used to it. Then surprisingly my tests failed for an unexpected input. You can find the actual test that failed here. My tests are really rudimentary but they still found multiple bugs that where quiet interesting.

Unexpected String.toInt behavior

In Carna I am asking for numeric values like weight or height and parse them to Int. (Now they are Float, but that is not important here) Thus the value is represented as a Result String Int. That is also what is tested at the line above. Then my tests failed for the input “+”. The reason:

> String.toInt "+"
Ok NaN : Result.Result String Int

I think I would have never thought about using “+” or “-“ as a input value that would cause an Ok result. Especially I would not have thought that the state Ok NaN was even possible.

What I was expected is something that happens when you use String.toFloat

> String.toFloat "+"
Err "could not convert string '+' to a Float" : Result.Result String Float

This bug was not severe because toString NaN just resulted in my input fields having the content “NaN” if you enter “+” which is not is better than crashing but not what I wanted.

Fixing toFloat

I asked about the behavior in the elm slack and got quick and helpful feedback there as well as on twitter. So thanks to the great, friendly and helpful community. ❤️❤️❤️

I learned that the current state is related to the current behavior of toInt and toFloat in Javascript, but more importantly that this is already known and will be fixed in Elm 0.19, which is great news!

Further issues

This was only the first issue in a list of bugs my fuzz tests found. Afterwards I switch to regex matches instead of relying on toInt never creating invalid Ok states, and I forget to allow decimal points for floats, because I used the same pattern for Int and Float validation, but my fuzz tests found that quickly. And there were more, but I think my point is clear.

Combine with test watch mode

I realized that elm-test watch mode is great to have a higher number of generated tests cases. By default 100 inputs are generated per fuzz-invocation. You can change that value with the --fuzz option of elm-test. However this can quickly become annoying if you are executing your tests manually and wait for the result. However in CI many people already use higher values. But you can also increase the number locally when combining it with the recently added watch mode that you can start with --watch. Because the tests start immediately after saving the file so they already run when you switch to the console (if you do that at all). I use this line here for local development:

elm-test --watch --fuzz 10000

Since the numbers the fuzzers are biased towards values that are typical error cases higher values helped my to find more edge cases quicker. For more details take a look at elm-test Fuzz.frequency.

I guess if you have way more fuzz tests you might go down to 2000 or 1000 but this is still much more than 100 and it did not annoy me at all yet.

Summary

I really think everybody should consider adding fuzz testing to their Elm projects. Even in my case of a private side project it was very valuable because there are not only edge cases that you are not aware of, but you might also find bugs in libraries or tools you use. This was for example the case for riak which had a high priority bug discovered via generative testing. In addition I also recommend this article from basho for further reading and do not forget to increase your --fuzz value in case you use watch mode and on CI.