Fuzzy and exhaustive testing#343
Open
stefano-zanotti-88 wants to merge 8 commits intocharlesnicholson:mainfrom
Open
Fuzzy and exhaustive testing#343stefano-zanotti-88 wants to merge 8 commits intocharlesnicholson:mainfrom
stefano-zanotti-88 wants to merge 8 commits intocharlesnicholson:mainfrom
Conversation
Sync with original repo
…on default-setting until after we have fetched the dynamic precision
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds a python script that can be used to generate random test cases, to possibly uncover bugs in obscure corners of the specs or NPF implementation.
The script generates a .c test file that can be compiled as is (it only needs nanoprintf.h, and C11 support).
It only exercises well-formed format strings and variadic arguments. That is, it also covers some IB or UB, but only of the type that can be sensibly handled and can be expected to be somewhat consistent across different printf implementations. It does not test NULL strings, NUL characters, wrong argument types, ... all things that are much more likely to cause a crash than some sort of sensible output string.
The output of the script is on stdout, it can then be redirected to a file.
The testing is "uniformly random", which means that it leans quite heavily towards "unreasonable" format strings, much more complicated than what would be found in real usage. This seems quite good to exercise parts of the library that are not well tested by the manual test cases.
The script is meant to be run manually, as well as the resulting .c file -- you need to specify a whole set of NPF options inside the script. However, maybe you might consider making the test automatic on multiple configurations, though I'm afraid it would cause many false positives (due to differences with sys printf), and also non-reproducible issues in the pre-submit checks, given the random nature of the tests. It seems better to run this test manually, once in a while, to look for new issues, rather than having it run in the presubmit checks, given its random nature.
Then, there is a script to generate "exhaustive" test cases, which means all possible combinations of specifiers and options (flags, width, precision, length modifiers), all tested with a handful of edge case values (like 0, INT_MIN, INT_MAX).
This needs #340 to have a chance of all-passing tests.
Otherwise, the precision-and-0-flag issue will raise complaints.
Support for %a %e %g (#315, #325) is not strictly needed, as those specifiers can be excluded from this test.
I wanted to let you have a look at this before you get to work on paland.cc, even though this PR will need some cleanup after #340 (and others) are merged.