Upgrade guide

#############

This is a companion guide to the :doc:`changelog`. While the changelog briefly

lists all of the new features, improvements and bug fixes, this upgrade guide

focuses only the subset which directly impacts your experience when upgrading

to a new version. But it goes into more detail. This includes things like

deprecated APIs and their replacements, build system changes, general code

modernization and other useful information.

v2.2

====

Deprecation of the ``PYBIND11_PLUGIN`` macro

--------------------------------------------

``PYBIND11_MODULE`` is now the preferred way to create module entry points.

The old macro emits a compile-time deprecation warning.

.. code-block:: cpp

    // old

    PYBIND11_PLUGIN(example) {

        py::module m("example", "documentation string");

        m.def("add", [](int a, int b) { return a + b; });

        return m.ptr();

    }

    // new

    PYBIND11_MODULE(example, m) {

        m.doc() = "documentation string"; // optional

        m.def("add", [](int a, int b) { return a + b; });

    }

New API for defining custom constructors and pickling functions

---------------------------------------------------------------

The old placement-new custom constructors have been deprecated. The new approach

uses ``py::init()`` and factory functions to greatly improve type safety.

Placement-new can be called accidentally with an incompatible type (without any

compiler errors or warnings), or it can initialize the same object multiple times

if not careful with the Python-side ``__init__`` calls. The new-style custom

constructors prevent such mistakes. See :ref:`custom_constructors` for details.

.. code-block:: cpp

    // old -- deprecated (runtime warning shown only in debug mode)

    py::class<Foo>(m, "Foo")

        .def("__init__", [](Foo &self, ...) {

            new (&self) Foo(...); // uses placement-new

        });

    // new

    py::class<Foo>(m, "Foo")

        .def(py::init([](...) { // Note: no `self` argument

            return new Foo(...); // return by raw pointer

            // or: return std::make_unique<Foo>(...); // return by holder

            // or: return Foo(...); // return by value (move constructor)

        }));

Mirroring the custom constructor changes, ``py::pickle()`` is now the preferred

way to get and set object state. See :ref:`pickling` for details.

.. code-block:: cpp

    // old -- deprecated (runtime warning shown only in debug mode)

    py::class<Foo>(m, "Foo")

        ...

        .def("__getstate__", [](const Foo &self) {

            return py::make_tuple(self.value1(), self.value2(), ...);

        })

        .def("__setstate__", [](Foo &self, py::tuple t) {

            new (&self) Foo(t[0].cast<std::string>(), ...);

        });

    // new

    py::class<Foo>(m, "Foo")

        ...

        .def(py::pickle(

            [](const Foo &self) { // __getstate__

                return py::make_tuple(f.value1(), f.value2(), ...); // unchanged

            },

            [](py::tuple t) { // __setstate__, note: no `self` argument

                return new Foo(t[0].cast<std::string>(), ...);

                // or: return std::make_unique<Foo>(...); // return by holder

                // or: return Foo(...); // return by value (move constructor)

            }

        ));

For both the constructors and pickling, warnings are shown at module

initialization time (on import, not when the functions are called).

They're only visible when compiled in debug mode. Sample warning:

.. code-block:: none

    pybind11-bound class 'mymodule.Foo' is using an old-style placement-new '__init__'

    which has been deprecated. See the upgrade guide in pybind11's docs.

Stricter enforcement of hidden symbol visibility for pybind11 modules

---------------------------------------------------------------------

pybind11 now tries to actively enforce hidden symbol visibility for modules.

If you're using either one of pybind11's :doc:`CMake or Python build systems

<compiling>` (the two example repositories) and you haven't been exporting any

symbols, there's nothing to be concerned about. All the changes have been done

transparently in the background. If you were building manually or relied on

specific default visibility, read on.

Setting default symbol visibility to *hidden* has always been recommended for

pybind11 (see :ref:`faq:symhidden`). On Linux and macOS, hidden symbol

visibility (in conjunction with the ``strip`` utility) yields much smaller

module binaries. `CPython's extension docs`_ also recommend hiding symbols

by default, with the goal of avoiding symbol name clashes between modules.

Starting with v2.2, pybind11 enforces this more strictly: (1) by declaring

all symbols inside the ``pybind11`` namespace as hidden and (2) by including

the ``-fvisibility=hidden`` flag on Linux and macOS (only for extension

modules, not for embedding the interpreter).

.. _CPython's extension docs: https://docs.python.org/3/extending/extending.html#providing-a-c-api-for-an-extension-module

The namespace-scope hidden visibility is done automatically in pybind11's

headers and it's generally transparent to users. It ensures that:

* Modules compiled with different pybind11 versions don't clash with each other.

* Some new features, like ``py::module_local`` bindings, can work as intended.

The ``-fvisibility=hidden`` flag applies the same visibility to user bindings

outside of the ``pybind11`` namespace. It's now set automatic by pybind11's

CMake and Python build systems, but this needs to be done manually by users

of other build systems. Adding this flag:

* Minimizes the chances of symbol conflicts between modules. E.g. if two

  unrelated modules were statically linked to different (ABI-incompatible)

  versions of the same third-party library, a symbol clash would be likely

  (and would end with unpredictable results).

* Produces smaller binaries on Linux and macOS, as pointed out previously.

Within pybind11's CMake build system, ``pybind11_add_module`` has always been

setting the ``-fvisibility=hidden`` flag in release mode. From now on, it's

being applied unconditionally, even in debug mode and it can no longer be opted

out of with the ``NO_EXTRAS`` option. The ``pybind11::module`` target now also

adds this flag to it's interface. The ``pybind11::embed`` target is unchanged.

The most significant change here is for the ``pybind11::module`` target. If you

were previously relying on default visibility, i.e. if your Python module was

doubling as a shared library with dependents, you'll need to either export

symbols manually (recommended for cross-platform libraries) or factor out the

shared library (and have the Python module link to it like the other

dependents). As a temporary workaround, you can also restore default visibility

using the CMake code below, but this is not recommended in the long run:

.. code-block:: cmake

    target_link_libraries(mymodule PRIVATE pybind11::module)

    add_library(restore_default_visibility INTERFACE)

    target_compile_options(restore_default_visibility INTERFACE -fvisibility=default)

    target_link_libraries(mymodule PRIVATE restore_default_visibility)

Local STL container bindings

----------------------------

Previous pybind11 versions could only bind types globally -- all pybind11

modules, even unrelated ones, would have access to the same exported types.

However, this would also result in a conflict if two modules exported the

same C++ type, which is especially problematic for very common types, e.g.

``std::vector<int>``. :ref:`module_local` were added to resolve this (see

that section for a complete usage guide).

``py::class_`` still defaults to global bindings (because these types are

usually unique across modules), however in order to avoid clashes of opaque

types, ``py::bind_vector`` and ``py::bind_map`` will now bind STL containers

as ``py::module_local`` if their elements are: builtins (``int``, ``float``,

etc.), not bound using ``py::class_``, or bound as ``py::module_local``. For

example, this change allows multiple modules to bind ``std::vector<int>``

without causing conflicts. See :ref:`stl_bind` for more details.

When upgrading to this version, if you have multiple modules which depend on

a single global binding of an STL container, note that all modules can still

accept foreign  ``py::module_local`` types in the direction of Python-to-C++.

The locality only affects the C++-to-Python direction. If this is needed in

multiple modules, you'll need to either:

* Add a copy of the same STL binding to all of the modules which need it.

* Restore the global status of that single binding by marking it

  ``py::module_local(false)``.

The latter is an easy workaround, but in the long run it would be best to

localize all common type bindings in order to avoid conflicts with

third-party modules.

Negative strides for Python buffer objects and numpy arrays

-----------------------------------------------------------

Support for negative strides required changing the integer type from unsigned

to signed in the interfaces of ``py::buffer_info`` and ``py::array``. If you

have compiler warnings enabled, you may notice some new conversion warnings

after upgrading. These can be resolved using ``static_cast``.

Deprecation of some ``py::object`` APIs

---------------------------------------

To compare ``py::object`` instances by pointer, you should now use

``obj1.is(obj2)`` which is equivalent to ``obj1 is obj2`` in Python.

Previously, pybind11 used ``operator==`` for this (``obj1 == obj2``), but

that could be confusing and is now deprecated (so that it can eventually

be replaced with proper rich object comparison in a future release).

For classes which inherit from ``py::object``, ``borrowed`` and ``stolen``

were previously available as protected constructor tags. Now the types

should be used directly instead: ``borrowed_t{}`` and ``stolen_t{}``

(`#771 <https://github.com/pybind/pybind11/pull/771>`_).

Stricter compile-time error checking

------------------------------------

Some error checks have been moved from run time to compile time. Notably,

automatic conversion of ``std::shared_ptr<T>`` is not possible when ``T`` is

not directly registered with ``py::class_<T>`` (e.g. ``std::shared_ptr<int>``

or ``std::shared_ptr<std::vector<T>>`` are not automatically convertible).

Attempting to bind a function with such arguments now results in a compile-time

error instead of waiting to fail at run time.

``py::init<...>()`` constructor definitions are also stricter and now prevent

bindings which could cause unexpected behavior:

.. code-block:: cpp

    struct Example {

        Example(int &);

    };

    py::class_<Example>(m, "Example")

        .def(py::init<int &>()); // OK, exact match

        // .def(py::init<int>()); // compile-time error, mismatch

A non-``const`` lvalue reference is not allowed to bind to an rvalue. However,

note that a constructor taking ``const T &`` can still be registered using

``py::init<T>()`` because a ``const`` lvalue reference can bind to an rvalue.

v2.1

====

Minimum compiler versions are enforced at compile time

------------------------------------------------------

The minimums also apply to v2.0 but the check is now explicit and a compile-time

error is raised if the compiler does not meet the requirements:

* GCC >= 4.8

* clang >= 3.3 (appleclang >= 5.0)

* MSVC >= 2015u3

* Intel C++ >= 15.0

The ``py::metaclass`` attribute is not required for static properties

---------------------------------------------------------------------

Binding classes with static properties is now possible by default. The

zero-parameter version of ``py::metaclass()`` is deprecated. However, a new

one-parameter ``py::metaclass(python_type)`` version was added for rare

cases when a custom metaclass is needed to override pybind11's default.

.. code-block:: cpp

    // old -- emits a deprecation warning

    py::class_<Foo>(m, "Foo", py::metaclass())

        .def_property_readonly_static("foo", ...);

    // new -- static properties work without the attribute

    py::class_<Foo>(m, "Foo")

        .def_property_readonly_static("foo", ...);

    // new -- advanced feature, override pybind11's default metaclass

    py::class_<Bar>(m, "Bar", py::metaclass(custom_python_type))

        ...

v2.0

====

Breaking changes in ``py::class_``

----------------------------------

These changes were necessary to make type definitions in pybind11

future-proof, to support PyPy via its ``cpyext`` mechanism (`#527

<https://github.com/pybind/pybind11/pull/527>`_), and to improve efficiency

(`rev. 86d825 <https://github.com/pybind/pybind11/commit/86d825>`_).

1. Declarations of types that provide access via the buffer protocol must

   now include the ``py::buffer_protocol()`` annotation as an argument to

   the ``py::class_`` constructor.

   .. code-block:: cpp

       py::class_<Matrix>("Matrix", py::buffer_protocol())

           .def(py::init<...>())

           .def_buffer(...);

2. Classes which include static properties (e.g. ``def_readwrite_static()``)

   must now include the ``py::metaclass()`` attribute. Note: this requirement

   has since been removed in v2.1. If you're upgrading from 1.x, it's

   recommended to skip directly to v2.1 or newer.

3. This version of pybind11 uses a redesigned mechanism for instantiating

   trampoline classes that are used to override virtual methods from within

   Python. This led to the following user-visible syntax change:

   .. code-block:: cpp

       // old v1.x syntax

       py::class_<TrampolineClass>("MyClass")

           .alias<MyClass>()

           ...

       // new v2.x syntax

       py::class_<MyClass, TrampolineClass>("MyClass")

           ...

   Importantly, both the original and the trampoline class are now specified

   as arguments to the ``py::class_`` template, and the ``alias<..>()`` call

   is gone. The new scheme has zero overhead in cases when Python doesn't

   override any functions of the underlying C++ class.

   `rev. 86d825 <https://github.com/pybind/pybind11/commit/86d825>`_.

   The class type must be the first template argument given to ``py::class_``

   while the trampoline can be mixed in arbitrary order with other arguments

   (see the following section).

Deprecation of the ``py::base<T>()`` attribute

----------------------------------------------

``py::base<T>()`` was deprecated in favor of specifying ``T`` as a template

argument to ``py::class_``. This new syntax also supports multiple inheritance.

Note that, while the type being exported must be the first argument in the

``py::class_<Class, ...>`` template, the order of the following types (bases,

holder and/or trampoline) is not important.

.. code-block:: cpp

    // old v1.x

    py::class_<Derived>("Derived", py::base<Base>());

    // new v2.x

    py::class_<Derived, Base>("Derived");

    // new -- multiple inheritance

    py::class_<Derived, Base1, Base2>("Derived");

    // new -- apart from `Derived` the argument order can be arbitrary

    py::class_<Derived, Base1, Holder, Base2, Trampoline>("Derived");

Out-of-the-box support for ``std::shared_ptr``

----------------------------------------------

The relevant type caster is now built in, so it's no longer necessary to

include a declaration of the form:

.. code-block:: cpp

    PYBIND11_DECLARE_HOLDER_TYPE(T, std::shared_ptr<T>)

Continuing to do so won’t cause an error or even a deprecation warning,

but it's completely redundant.

Deprecation of a few ``py::object`` APIs

----------------------------------------

All of the old-style calls emit deprecation warnings.

+---------------------------------------+---------------------------------------------+

|  Old syntax                           |  New syntax                                 |

+=======================================+=============================================+

| ``obj.call(args...)``                 | ``obj(args...)``                            |

+---------------------------------------+---------------------------------------------+

| ``obj.str()``                         | ``py::str(obj)``                            |

+---------------------------------------+---------------------------------------------+

| ``auto l = py::list(obj); l.check()`` | ``py::isinstance<py::list>(obj)``           |

+---------------------------------------+---------------------------------------------+

| ``py::object(ptr, true)``             | ``py::reinterpret_borrow<py::object>(ptr)`` |

+---------------------------------------+---------------------------------------------+

| ``py::object(ptr, false)``            | ``py::reinterpret_steal<py::object>(ptr)``  |

+---------------------------------------+---------------------------------------------+

| ``if (obj.attr("foo"))``              | ``if (py::hasattr(obj, "foo"))``            |

+---------------------------------------+---------------------------------------------+

| ``if (obj["bar"])``                   | ``if (obj.contains("bar"))``                |

+---------------------------------------+---------------------------------------------+