Benchmark

=========

The following is the result of a synthetic benchmark comparing both compilation

time and module size of pybind11 against Boost.Python. A detailed report about a

Boost.Python to pybind11 conversion of a real project is available here: [#f1]_.

.. [#f1] http://graylab.jhu.edu/RosettaCon2016/PyRosetta-4.pdf

Setup

-----

A python script (see the ``docs/benchmark.py`` file) was used to generate a set

of files with dummy classes whose count increases for each successive benchmark

(between 1 and 2048 classes in powers of two). Each class has four methods with

a randomly generated signature with a return value and four arguments. (There

was no particular reason for this setup other than the desire to generate many

unique function signatures whose count could be controlled in a simple way.)

Here is an example of the binding code for one class:

.. code-block:: cpp

    ...

    class cl034 {

    public:

        cl279 *fn_000(cl084 *, cl057 *, cl065 *, cl042 *);

        cl025 *fn_001(cl098 *, cl262 *, cl414 *, cl121 *);

        cl085 *fn_002(cl445 *, cl297 *, cl145 *, cl421 *);

        cl470 *fn_003(cl200 *, cl323 *, cl332 *, cl492 *);

    };

    ...

    PYBIND11_MODULE(example, m) {

        ...

        py::class_<cl034>(m, "cl034")

            .def("fn_000", &cl034::fn_000)

            .def("fn_001", &cl034::fn_001)

            .def("fn_002", &cl034::fn_002)

            .def("fn_003", &cl034::fn_003)

        ...

    }

The Boost.Python version looks almost identical except that a return value

policy had to be specified as an argument to ``def()``. For both libraries,

compilation was done with

.. code-block:: bash

    Apple LLVM version 7.0.2 (clang-700.1.81)

and the following compilation flags

.. code-block:: bash

    g++ -Os -shared -rdynamic -undefined dynamic_lookup -fvisibility=hidden -std=c++14

Compilation time

----------------

The following log-log plot shows how the compilation time grows for an

increasing number of class and function declarations. pybind11 includes many

fewer headers, which initially leads to shorter compilation times, but the

performance is ultimately fairly similar (pybind11 is 19.8 seconds faster for

the largest largest file with 2048 classes and a total of 8192 methods -- a

modest **1.2x** speedup relative to Boost.Python, which required 116.35

seconds).

.. only:: not latex

    .. image:: pybind11_vs_boost_python1.svg

.. only:: latex

    .. image:: pybind11_vs_boost_python1.png

Module size

-----------

Differences between the two libraries become much more pronounced when

considering the file size of the generated Python plugin: for the largest file,

the binary generated by Boost.Python required 16.8 MiB, which was **2.17

times** / **9.1 megabytes** larger than the output generated by pybind11. For

very small inputs, Boost.Python has an edge in the plot below -- however, note

that it stores many definitions in an external library, whose size was not

included here, hence the comparison is slightly shifted in Boost.Python's

favor.

.. only:: not latex

    .. image:: pybind11_vs_boost_python2.svg

.. only:: latex

    .. image:: pybind11_vs_boost_python2.png