csomh / source-git / rpm

Forked from source-git/rpm 4 years ago
Clone

420bed In Python 3, return all our string data as surrogate-escaped utf-8 strings

17 files Authored by Panu Matilainen 4 years ago, Committed by Hunor Csomortáni 4 years ago,
    In Python 3, return all our string data as surrogate-escaped utf-8 strings
    
    In the almost ten years of rpm sort of supporting Python 3 bindings, quite
    obviously nobody has actually tried to use them. There's a major mismatch
    between what the header API outputs (bytes) and what all the other APIs
    accept (strings), resulting in hysterical TypeErrors all over the place,
    including but not limited to labelCompare() (RhBug:1631292). Also a huge
    number of other places have been returning strings and silently assuming
    utf-8 through use of Py_BuildValue("s", ...), which will just irrevocably
    fail when non-utf8 data is encountered.
    
    The politically Python 3-correct solution would be declaring all our data
    as bytes with unspecified encoding - that's exactly what it historically is.
    However doing so would by definition break every single rpm script people
    have developed on Python 2. And when 99% of the rpm content in the world
    actually is utf-8 encoded even if it doesn't say so (and in recent times
    packages even advertise themselves as utf-8 encoded), the bytes-only route
    seems a wee bit too draconian, even to this grumpy old fella.
    
    Instead, route all our string returns through a single helper macro
    which on Python 2 just does what we always did, but in Python 3 converts
    the data to surrogate-escaped utf-8 strings. This makes stuff "just work"
    out of the box pretty much everywhere even with Python 3 (including
    our own test-suite!), while still allowing to handle the non-utf8 case.
    Handling the non-utf8 case is a bit more uglier but still possible,
    which is exactly how you want corner-cases to be. There might be some
    uses for retrieving raw byte data from the header, but worrying about
    such an API is a case for some other rainy day, for now we mostly only
    care that stuff works again.
    
    Also add test-cases for mixed data source labelCompare() and
    non-utf8 insert to + retrieve from header.
    
        
file modified
+1 -1
file modified
+4 -4
file modified
+3 -3
file modified
+12 -12
file modified
+13 -13
file modified
+1 -1
file modified
+1 -1
file modified
+1 -1
file modified
+4 -4
file modified
+1 -1
file modified
+7 -0
file modified
+1 -1
file modified
+8 -8
file modified
+6 -5
file modified
+4 -4
file modified
+1 -0
file modified
+34 -0