Running CPython under Memcheck

Recently I’ve been doing some things that involve use of ctypes. Running CPython under Memcheck (part of Valgrind) is known to have some quirks. Nevertheless, it is still useful for my particular case: finding memory leaks in a program that uses ctypes to call malloc but sometimes does not call free. Here’s a contrived example. It is a Python script (test.py) that allows introduction of deliberate memory leak:

import ctypes
import argparse


libc = ctypes.CDLL("libc.so.6")


def allocate_and_maybe_free(must_free):
    mem = libc.malloc(ctypes.c_size_t(1234))
    if must_free:
        libc.free(mem)


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("-f", "--no-free", default=False, action="store_true")
    args = parser.parse_args()
    allocate_and_maybe_free(not args.no_free)

When --no-free command-line argument is passed, args.no_free becomes True, so must_free (not args.no_free) in allocate_and_maybe_free becomes False, therefore libc.free(mem) is not called.

To use Valgrind (and its tools, like Memcheck) with CPython (3.6 in my case), debug-enabled build of the latter is needed:

$ ./configure --prefix /home/user/build --with-pydebug --with-valgrind
$ make install

Three options are passed above:

• --prefix is usual installation prefix
• --with-pydebug is recommended option for building debug-enabled interpreter
• --with-valgrind makes interpreter automatically disable pymalloc memory allocator when running under Valgrind

After building & installation successfully finishes, python3 binary is installed into /home/user/build/bin/ directory (due to --prefix /home/user/build). Now let’s see what happens when memory is allocated and then correctly freed in test.py:

$ valgrind --leak-check=full --show-possibly-lost=no --show-reachable=no ./build/bin/python3 test.py
...
==...== LEAK SUMMARY:
==...==    definitely lost: 0 bytes in 0 blocks
==...==    indirectly lost: 0 bytes in 0 blocks
==...==      possibly lost: 1,993,410 bytes in 9,066 blocks
==...==    still reachable: 6,179 bytes in 18 blocks
==...==         suppressed: 0 bytes in 0 blocks
...

Memcheck reports no “definitely lost” memory. There is “possibly lost” & “still reachable”, but that’s CPython. But if deliberate memory leak is introduced by passing --no-free to test.py, Memcheck complains about “definitely lost” memory:

$ valgrind --leak-check=full --show-possibly-lost=no --show-reachable=no ./build/bin/python3 test.py --no-free
...
==...== 1,234 bytes in 1 blocks are definitely lost in loss record 22 of 34
==...==    at 0x...: malloc (in /usr/lib/valgrind/vgpreload_memcheck-....so)
...
==...==    by 0x...: ffi_call (in /usr/lib/.../libffi...)
...
==...==    by 0x...: _PyFunction_FastCall (ceval.c:4891)
==...==
==...== LEAK SUMMARY:
==...==    definitely lost: 1,234 bytes in 1 blocks
==...==    indirectly lost: 0 bytes in 0 blocks
==...==      possibly lost: 1,993,277 bytes in 9,065 blocks
==...==    still reachable: 6,179 bytes in 18 blocks
==...==         suppressed: 0 bytes in 0 blocks
...

As you can see, “definitely lost” are 1234 bytes allocated by libc.malloc(ctypes.c_size_t(1234)) that due to --no-free are not freed. So, Memcheck may help catch memory leaks in Python programs under CPython.