== Testing Graphite2 ==

During development Graphite2 is regularly checked against it's test suite 
of over a 100 test cases.  This happens automatically on every check-in to the 
default branch thanks to our continuous build server for Windows and Linux. 
Prior to each major release it's tested using an automated fuzzing system, 
which checks for robustness in the face of corrupted font files.  This set of 
fuzz tests uses valgrind to check for rogue memory accesses and runs several 
thousand tests on each of the 4 major test fonts and takes considerably longer 
to run.
We also have a growing suite of fuzz test regressions culled from logs 
generated by the above fuzz test system, which can be run with a make command. 
These are intended to be run before bug fix release and other point releases.

=== Running the tests ===

==== Running the standard tests ====
The standard test suite, the same one run on every checkin to the graphite2 
project repository, can be run with a simple:
----
make test
----

==== Runnging the fuzztest regressions ====
----
make fuzztest
----
These should be run before point and bug fix releases.

==== Running the full fuzz test script ====
----
full-fuzz-test.sh script [fuzztest options]
----
This script exercises graphite over 4 scripts Myanmar, Devangari, extended 
Latin and Arabic using 4 fonts and text in the Myanmar, Nepalese, Yoroba and 
Arabic languages.  It uses the `fuzzcomparerender` script to fuzz every byte of 
each font with a random value and enforce generous enough runtime and memory 
resource limits to detect infinite loops or memory leaks. 
A successfull run of this script will produce four empty log files. Passing 
`--valgrind` to the script this is passed down to the `fuzztest` program which 
will run the test program with valgrind, this increases the runtime of script 
considerably.  Normally the script can run all four tests within 24 hours, 
fully loading our 4 core hyperthreaded Xeon/Core i7 system.  Using the valgrind 
option this takes approximately a week on the same system.

==== Running fuzzcomparerender ====
----
tests/fuzzcomparerender <font name> <text name> [-r] [fuzztest options]
----
The `font name` must be the basename of a font file under `tests/fonts` and the
`text name` must be the basename of a text file from `tests/texts`.  The `-r` 
option if present is passed to comparerenderer and tells it the text is 
right-to-left.  Any further options are passed to the fuzztest program, this is
typically one of `--valgrind` or `--input`, or less frequently `--passes`, 
`--jobs` or `--status`. See `tests/fuzztest --help` for more information.
This script runs `fuzztest` so that it corrupts every byte of the required 
TrueType and Graphite specific tables with a random value, but excludes 
OpenType and AAT related tables. It also imposes a runtime limit of 10 seconds 
(most test should compete in a fraction of a second) and a memory limit 200MiB,
again a normal run should only use a tiny fraction of that.
If the `comparerenderer` test segfaults, exceeds those limits or returns an 
error value it is logged to a file named on the following pattern: 
`fuzzfont-<font name>-<text name>.log`, which is written to the script's 
current directory.

==== Running fuzztest ====
----
tests/fuzztest --font=font [options] -- <test harness>
----
A multiprocess fuzz test framework that generates corrupt fonts to run a user 
supplied test harness against.  This will check each byte in every table of a 
TTF or subset of it's tables if specified, by overwriting with a random or user
specified value. Using the `--input` option it can also re-run tests using 
previous output logs as input, it will ignore any line that doesn't match the 
format of a fuzz line generated by the program, so such input fuzzes can be well
annotated.  It is this facility that is used to drive the `make fuzztest` 
regression check.
By default this will try to ensure there is always one test harness running on 
each OS reported core at all times, the `--jobs` option can be used to limit 
this if need to limit the load.
Unless told otherwise the program will display a status line inidcating the 
percentage of the tests run so far, the rate of testing and an estimated time 
and date for completion.  Once the status has updated 4-5 times the estimate 
usually settles down to a frequently accurate estimate.

=== Adding fuzz tests ===

Fuzz regression test files are simply copies of the log generated by the 
`fuzzcomparerender` script.  Once you have a log that generates a test case, you 
can edit it to produce a more targeted set of fuzz lines (a log can generate 
several different bugs).  You should try to produce a minimal set of fuzz lines 
for that particular test case, however the more fuzz lines that casue the same 
bug the better.

The test case should be placed as follows:
----
tests/fuzz-tests/<font name>/<text name>/<bug description>.fuzz
----
where:

font name::
    The name of a font, minus the .ttf extension from tests/fonts.
    
text name:: 
    The name of a text file, minus .txt extension from tests/texts.
    
bug description::
    A short description of the bug, this cannot contain any ":" characters or 
    other characters forbidden by Windows or Unix filename schemes. For 
    `class::member` references use an `_` e.g. `class_member`.

