Passing Python strings to C++

When a Python str is passed from Python to a C++ function that accepts std::string or char * as arguments, pybind11 will encode the Python string to UTF-8. All Python str can be encoded in UTF-8, so this operation does not fail.

The C++ language is encoding agnostic. It is the responsibility of the programmer to track encodings. It's often easiest to simply use UTF-8 everywhere.

m.def("utf8_test",
    [](const std::string &s) {
        cout << "utf-8 is icing on the cake.\n";
        cout << s;
    }
);
m.def("utf8_charptr",
    [](const char *s) {
        cout << "My favorite food is\n";
        cout << s;
    }
);

>>> utf8_test('🎂')
utf-8 is icing on the cake.
🎂

>>> utf8_charptr('🍕')
My favorite food is
🍕

The results are the same whether the C++ function accepts arguments by value or reference, and whether or not const is used.

Returning C++ strings to Python

When a C++ function returns a std::string or char* to a Python caller, pybind11 will assume that the string is valid UTF-8 and will decode it to a native Python str, using the same API as Python uses to perform bytes.decode('utf-8'). If this implicit conversion fails, pybind11 will raise a UnicodeDecodeError.

m.def("std_string_return",
    []() {
        return std::string("This string needs to be UTF-8 encoded");
    }
);

>>> isinstance(example.std_string_return(), str)
True

Because UTF-8 is inclusive of pure ASCII, there is never any issue with returning a pure ASCII string to Python. If there is any possibility that the string is not pure ASCII, it is necessary to ensure the encoding is valid UTF-8.

// This uses the Python C API to convert Latin-1 to Unicode
m.def("str_output",
    []() {
        std::string s = "Send your r\xe9sum\xe9 to Alice in HR"; // Latin-1
        py::str py_s = PyUnicode_DecodeLatin1(s.data(), s.length());
        return py_s;
    }
);

>>> str_output()
'Send your résumé to Alice in HR'

m.def("return_bytes",
    []() {
        std::string s("\xba\xd0\xba\xd0");  // Not valid UTF-8
        return py::bytes(s);  // Return the data without transcoding
    }
);

>>> example.return_bytes()
b'\xba\xd0\xba\xd0'

m.def("asymmetry",
    [](std::string s) {  // Accepts str or bytes from Python
        return s;  // Looks harmless, but implicitly converts to str
    }
);

>>> isinstance(example.asymmetry(b"have some bytes"), str)
True

>>> example.asymmetry(b"\xba\xd0\xba\xd0")  # invalid utf-8 as bytes
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xba in position 0: invalid start byte

Wide character strings

When a Python str is passed to a C++ function expecting std::wstring, wchar_t*, std::u16string or std::u32string, the str will be encoded to UTF-16 or UTF-32 depending on how the C++ compiler implements each type, in the platform's native endianness. When strings of these types are returned, they are assumed to contain valid UTF-16 or UTF-32, and will be decoded to Python str.

#define UNICODE
#include <windows.h>

m.def("set_window_text",
    [](HWND hwnd, std::wstring s) {
        // Call SetWindowText with null-terminated UTF-16 string
        ::SetWindowText(hwnd, s.c_str());
    }
);
m.def("get_window_text",
    [](HWND hwnd) {
        const int buffer_size = ::GetWindowTextLength(hwnd) + 1;
        auto buffer = std::make_unique< wchar_t[] >(buffer_size);

        ::GetWindowText(hwnd, buffer.data(), buffer_size);

        std::wstring text(buffer.get());

        // wstring will be converted to Python str
        return text;
    }
);

Strings in multibyte encodings such as Shift-JIS must transcoded to a UTF-8/16/32 before being returned to Python.

Character literals

C++ functions that accept character literals as input will receive the first character of a Python str as their input. If the string is longer than one Unicode character, trailing characters will be ignored.

When a character literal is returned from C++ (such as a char or a wchar_t), it will be converted to a str that represents the single character.

m.def("pass_char", [](char c) { return c; });
m.def("pass_wchar", [](wchar_t w) { return w; });

>>> example.pass_char('A')
'A'

While C++ will cast integers to character types (char c = 0x65;), pybind11 does not convert Python integers to characters implicitly. The Python function chr() can be used to convert integers to characters.

>>> example.pass_char(0x65)
TypeError

>>> example.pass_char(chr(0x65))
'A'

If the desire is to work with an 8-bit integer, use int8_t or uint8_t as the argument type.

>>> example.pass_wchar('é')
'é'

>>> combining_e_acute = 'e' + '\u0301'

>>> combining_e_acute
'é'

>>> combining_e_acute == 'é'
False

>>> example.pass_wchar(combining_e_acute)
'e'

>>> example.pass_wchar(unicodedata.normalize('NFC', combining_e_acute))
'é'

C++17 string views

C++17 string views are automatically supported when compiling in C++17 mode. They follow the same rules for encoding and decoding as the corresponding STL string type (for example, a std::u16string_view argument will be passed UTF-16-encoded data, and a returned std::string_view will be decoded as UTF-8).

References

• The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)
• C++ - Using STL Strings at Win32 API Boundaries