118 lines
3.6 KiB
Plaintext
118 lines
3.6 KiB
Plaintext
|
===============
|
||
|
C coverage tool
|
||
|
===============
|
||
|
|
||
|
This directory contains a tool to generate C code-coverage reports
|
||
|
using valgrind's callgrind tool.
|
||
|
|
||
|
Prerequisites
|
||
|
-------------
|
||
|
|
||
|
* `Valgrind <http://www.valgrind.org/>`_ (3.5.0 tested, earlier
|
||
|
versions may work)
|
||
|
|
||
|
* `Pygments <http://www.pygments.org/>`_ (0.11 or later required)
|
||
|
|
||
|
C code-coverage
|
||
|
---------------
|
||
|
|
||
|
Generating C code coverage reports requires two steps:
|
||
|
|
||
|
* Collecting coverage results (from valgrind)
|
||
|
|
||
|
* Generating a report from one or more sets of results
|
||
|
|
||
|
For most cases, it is good enough to do::
|
||
|
|
||
|
> c_coverage_collect.sh python -c "import numpy; numpy.test()"
|
||
|
> c_coverage_report.py callgrind.out.pid
|
||
|
|
||
|
which will run all of the NumPy unit tests, create a directory called
|
||
|
`coverage` and place the coverage results there.
|
||
|
|
||
|
In a more advanced scenario, you may wish to run individual unit tests
|
||
|
(since running under valgrind slows things down) and combine multiple
|
||
|
results files together in a single report.
|
||
|
|
||
|
Collecting results
|
||
|
``````````````````
|
||
|
|
||
|
To collect coverage results, you merely run the python interpreter
|
||
|
under valgrind's callgrind tool. The `c_coverage_collect.sh` helper
|
||
|
script will pass all of the required arguments to valgrind.
|
||
|
|
||
|
For example, in typical usage, you may want to run all of the NumPy
|
||
|
unit tests::
|
||
|
|
||
|
> c_coverage_collect.sh python -c "import numpy; numpy.test()"
|
||
|
|
||
|
This will output a file ``callgrind.out.pid`` containing the results of
|
||
|
the run, where ``pid`` is the process id of the run.
|
||
|
|
||
|
Generating a report
|
||
|
```````````````````
|
||
|
|
||
|
To generate a report, you pass the ``callgrind.out.pid`` output file to
|
||
|
the `c_coverage_report.py` script::
|
||
|
|
||
|
> c_coverage_report.py callgrind.out.pid
|
||
|
|
||
|
To combine multiple results files together, simply list them on the
|
||
|
commandline or use wildcards::
|
||
|
|
||
|
> c_coverage_report.py callgrind.out.*
|
||
|
|
||
|
Options
|
||
|
'''''''
|
||
|
|
||
|
* ``--directory``: Specify a different output directory
|
||
|
|
||
|
* ``--pattern``: Specify a regex pattern to match for source files.
|
||
|
The default is `numpy`, so it will only include source files whose
|
||
|
path contains the string `numpy`. If, for instance, you wanted to
|
||
|
include all source files covered (that are available on your
|
||
|
system), pass ``--pattern=.``.
|
||
|
|
||
|
* ``--format``: Specify the output format(s) to generate. May be
|
||
|
either ``text`` or ``html``. If ``--format`` is not provided,
|
||
|
both formats will be output.
|
||
|
|
||
|
Reading a report
|
||
|
----------------
|
||
|
|
||
|
The C code coverage report is a flat directory of files, containing
|
||
|
text and/or html files. The files are named based on their path in
|
||
|
the original source tree with slashes converted to underscores.
|
||
|
|
||
|
Text reports
|
||
|
````````````
|
||
|
|
||
|
The text reports add a prefix to each line of source code:
|
||
|
|
||
|
- '>' indicates the line of code was run
|
||
|
|
||
|
- '!' indicates the line of code was not run
|
||
|
|
||
|
HTML reports
|
||
|
````````````
|
||
|
|
||
|
The HTML report highlights the code that was run in green.
|
||
|
|
||
|
The HTML report has special support for the "generated" functions in
|
||
|
NumPy. Each run line of code also contains a number in square
|
||
|
brackets indicating the number of different generated functions the
|
||
|
line was run in. Hovering the mouse over the line will display a list
|
||
|
of the versions of the function in which the line was run. These
|
||
|
numbers can be used to see if a particular line was run in all
|
||
|
versions of the function.
|
||
|
|
||
|
Caveats
|
||
|
-------
|
||
|
|
||
|
The coverage results occasionally misses lines that clearly must have
|
||
|
been run. This usually can be traced back to the compiler optimizer
|
||
|
removing lines because they are tautologically impossible or to
|
||
|
combine lines together. Compiling NumPy without optimizations helps,
|
||
|
but not completely. Even despite this flaw, this tool is still
|
||
|
helpful in identifying large missed blocks or functions.
|