Unit-testing usage examples in README.rst

Automatic testing of code examples in README is an area that is often neglected. I’d like to share some experience I got when was integrating doctests from README.rst of paka.cmark into existing unittest-based test discovery.

Let’s say there are following files:

somelib-project/
    README.rst
    somelib/
        ...
    tests/
        test_readme.py

To run tests from tests/ dir following command is executed from inside somelib-project/ dir:

$ python3 -m unittest discover --start-directory tests/

Inside test_readme.py file doctest examples are parsed from README.rst and then are exposed via load_tests protocol:

import doctest
import unittest


def load_tests(loader, tests, pattern):
    suite = doctest.DocFileSuite("../README.rst")
    tests.addTests(suite)
    return tests

This might be enough in general. Now for the specifics. Tests and code of paka.cmark have to support both Python 2.7 and 3.5 with single source. As paka.cmark.to_html accepts and returns unicode in 2.7 and str in 3, use of repr in doctests will fail for one version of language, and succeed for other. That is, not using “u” string prefix will cause failure under 2.7:

>>> cmark.to_html(u"Hello,\n*World*!")
'<p>Hello, <em>World</em>!</p>\n'

Failed example:
    cmark.to_html(u"Hello,\n*World*!")
Expected:
    '<p>Hello, <em>World</em>!</p>\n'
Got:
    u'<p>Hello, <em>World</em>!</p>\n'

And using “u” string prefix will result in fail under 3.5 (but tests will pass for 2.7):

>>> cmark.to_html(u"Hello,\n*World*!")
u'<p>Hello, <em>World</em>!</p>\n'

Failed example:
    cmark.to_html(u"Hello,\n*World*!")
Expected:
    u'<p>Hello, <em>World</em>!</p>\n'
Got:
    '<p>Hello, <em>World</em>!</p>\n'

I use a “hack”—wrap each call with print:

>>> print(cmark.to_html(u"Hello,\n*World*!"))
<p>Hello, <em>World</em>!</p>

But just wrapping them is not enough, as resulting HTML has newline at the end:

Failed example:
    print(cmark.to_html(u"Hello,\n*World*!"))
Expected:
    <p>Hello, <em>World</em>!</p>
Got:
    <p>Hello, <em>World</em>!</p>
    <BLANKLINE>

With doctest directive that problem is solved:

>>> print(cmark.to_html(u"Hello,\n*World*!"))  # doctest: +NORMALIZE_WHITESPACE
<p>Hello, <em>World</em>!</p>

If you don’t want to add such directives to each example individually, you can use optionflags argument of doctest.DocFileSuite to pass doctest.NORMALIZE_WHITESPACE (and thus use that directive for all examples):

import doctest
import unittest


def load_tests(loader, tests, pattern):
    suite = doctest.DocFileSuite(
        "../README.rst", optionflags=doctest.NORMALIZE_WHITESPACE)
    tests.addTests(suite)
    return tests