# Universal Types with BER/DER Decoder and DER Encoder

The *asn1crypto* library is a combination of universal type classes that

implement BER/DER decoding and DER encoding, a PEM encoder and decoder, and a

number of pre-built cryptographic type classes. This document covers the

universal type classes.

For a general overview of ASN.1 as used in cryptography, please see

[A Layman's Guide to a Subset of ASN.1, BER, and DER](http://luca.ntop.org/Teaching/Appunti/asn1.html).

This page contains the following sections:

 - [Universal Types](#universal-types)

 - [Basic Usage](#basic-usage)

 - [Sequence](#sequence)

 - [Set](#set)

 - [SequenceOf](#sequenceof)

 - [SetOf](#setof)

 - [Integer](#integer)

 - [Enumerated](#enumerated)

 - [ObjectIdentifier](#objectidentifier)

 - [BitString](#bitstring)

 - [Strings](#strings)

 - [UTCTime](#utctime)

 - [GeneralizedTime](#generalizedtime)

 - [Choice](#choice)

 - [Any](#any)

 - [Specification via OID](#specification-via-oid)

 - [Explicit and Implicit Tagging](#explicit-and-implicit-tagging)

## Universal Types

For general purpose ASN.1 parsing, the `asn1crypto.core` module is used. It

contains the following classes, that parse, represent and serialize all of the

ASN.1 universal types:

| Class              | Native Type                            | Implementation Notes                 |

| ------------------ | -------------------------------------- | ------------------------------------ |

| `Boolean`          | `bool`                                 |                                      |

| `Integer`          | `int`                                  | may be `long` on Python 2            |

| `BitString`        | `tuple` of `int` or `set` of `unicode` | `set` used if `_map` present         |

| `OctetString`      | `bytes` (`str`)                        |                                      |

| `Null`             | `None`                                 |                                      |

| `ObjectIdentifier` | `str` (`unicode`)                      | string is dotted integer format      |

| `ObjectDescriptor` |                                        | no native conversion                 |

| `InstanceOf`       |                                        | no native conversion                 |

| `Real`             |                                        | no native conversion                 |

| `Enumerated`       | `str` (`unicode`)                      | `_map` must be set                   |

| `UTF8String`       | `str` (`unicode`)                      |                                      |

| `RelativeOid`      | `str` (`unicode`)                      | string is dotted integer format      |

| `Sequence`         | `OrderedDict`                          |                                      |

| `SequenceOf`       | `list`                                 |                                      |

| `Set`              | `OrderedDict`                          |                                      |

| `SetOf`            | `list`                                 |                                      |

| `EmbeddedPdv`      | `OrderedDict`                          | no named field parsing               |

| `NumericString`    | `str` (`unicode`)                      | no charset limitations               |

| `PrintableString`  | `str` (`unicode`)                      | no charset limitations               |

| `TeletexString`    | `str` (`unicode`)                      |                                      |

| `VideotexString`   | `bytes` (`str`)                        | no unicode conversion                |

| `IA5String`        | `str` (`unicode`)                      |                                      |

| `UTCTime`          | `datetime.datetime`                    |                                      |

| `GeneralizedTime`  | `datetime.datetime`                    | treated as UTC when no timezone      |

| `GraphicString`    | `str` (`unicode`)                      | unicode conversion as latin1         |

| `VisibleString`    | `str` (`unicode`)                      | no charset limitations               |

| `GeneralString`    | `str` (`unicode`)                      | unicode conversion as latin1         |

| `UniversalString`  | `str` (`unicode`)                      |                                      |

| `CharacterString`  | `str` (`unicode`)                      | unicode conversion as latin1         |

| `BMPString`        | `str` (`unicode`)                      |                                      |

For *Native Type*, the Python 3 type is listed first, with the Python 2 type

in parentheses.

As mentioned next to some of the types, value parsing may not be implemented

for types not currently used in cryptography (such as `ObjectDescriptor`,

`InstanceOf` and `Real`). Additionally some of the string classes don't

enforce character set limitations, and for some string types that accept all

different encodings, the default encoding is set to latin1.

In addition, there are a few overridden types where various specifications use

a `BitString` or `OctetString` type to represent a different type. These

include:

| Class                | Native Type         | Implementation Notes            |

| -------------------- | ------------------- | ------------------------------- |

| `OctetBitString`     | `bytes` (`str`)     |                                 |

| `IntegerBitString`   | `int`               | may be `long` on Python 2       |

| `IntegerOctetString` | `int`               | may be `long` on Python 2       |

For situations where the DER encoded bytes from one type is embedded in another,

the `ParsableOctetString` and `ParsableOctetBitString` classes exist. These

function the same as `OctetString` and `OctetBitString`, however they also

have an attribute `.parsed` and a method `.parse()` that allows for

parsing the content as ASN.1 structures.

All of these overrides can be used with the `cast()` method to convert between

them. The only requirement is that the class being casted to has the same tag

as the original class. No re-encoding is done, rather the contents are simply

re-interpreted.

```python

from asn1crypto.core import BitString, OctetBitString, IntegerBitString

bit = BitString({

    0, 0, 0, 0, 0, 0, 0, 1,

    0, 0, 0, 0, 0, 0, 1, 0,

})

# Will print (0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0)

print(bit.native)

octet = bit.cast(OctetBitString)

# Will print b'\x01\x02'

print(octet.native)

i = bit.cast(IntegerBitString)

# Will print 258

print(i.native)

```

## Basic Usage

All of the universal types implement four methods, a class method `.load()` and

the instance methods `.dump()`, `.copy()` and `.debug()`.

`.load()` accepts a byte string of DER or BER encoded data and returns an

object of the class it was called on. `.dump()` returns the serialization of

an object into DER encoding.

```python

from asn1crypto.core import Sequence

parsed = Sequence.load(der_byte_string)

serialized = parsed.dump()

```

By default, *asn1crypto* tries to be efficient and caches serialized data for

better performance. If the input data is possibly BER encoded, but the output

must be DER encoded, the `force` parameter may be used with `.dump()`.

```python

from asn1crypto.core import Sequence

parsed = Sequence.load(der_byte_string)

der_serialized = parsed.dump(force=True)

```

The `.copy()` method creates a deep copy of an object, allowing child fields to

be modified without affecting the original.

```python

from asn1crypto.core import Sequence

seq1 = Sequence.load(der_byte_string)

seq2 = seq1.copy()

seq2[0] = seq1[0] + 1

if seq1[0] != seq2[0]:

    print('Copies have distinct contents')

```

The `.debug()` method is available to help in situations where interaction with

another ASN.1 serializer or parsing is not functioning as expected. Calling

this method will print a tree structure with information about the header bytes,

class, method, tag, special tagging, content bytes, native Python value, child

fields and any sub-parsed values.

```python

from asn1crypto.core import Sequence

parsed = Sequence.load(der_byte_string)

parsed.debug()

```

In addition to the available methods, every instance has a `.native` property

that converts the data into a native Python data type.

```python

import pprint

from asn1crypto.core import Sequence

parsed = Sequence.load(der_byte_string)

pprint(parsed.native)

```

## Sequence

One of the core structures when dealing with ASN.1 is the Sequence type. The

`Sequence` class can handle field with universal data types, however in most

situations the `_fields` property will need to be set with the expected

definition of each field in the Sequence.

### Configuration

The `_fields` property must be set to a `list` of 2-3 element `tuple`s. The

first element in the tuple must be a unicode string of the field name. The

second must be a type class - either a universal type, or a custom type. The

third, and optional, element is a `dict` with parameters to pass to the type

class for things like default values, marking the field as optional, or

implicit/explicit tagging.

```python

from asn1crypto.core import Sequence, Integer, OctetString, IA5String

class MySequence(Sequence):

    _fields = [

        ('field_one', Integer),

        ('field_two', OctetString),

        ('field_three', IA5String, {'optional': True}),

    ]

```

Implicit and explicit tagging will be covered in more detail later, however

the following are options that can be set for each field type class:

 - `{'default: 1}` sets the field's default value to `1`, allowing it to be

   omitted from the serialized form

 - `{'optional': True}` set the field to be optional, allowing it to be

   omitted

### Usage

To access values of the sequence, use dict-like access via `[]` and use the

name of the field:

```python

seq = MySequence.load(der_byte_string)

print(seq['field_two'].native)

```

The values of fields can be set by assigning via `[]`. If the value assigned is

of the correct type class, it will be used as-is. If the value is not of the

correct type class, a new instance of that type class will be created and the

value will be passed to the constructor.

```python

seq = MySequence.load(der_byte_string)

# These statements will result in the same state

seq['field_one'] = Integer(5)

seq['field_one'] = 5

```

When fields are complex types such as `Sequence` or `SequenceOf`, there is no

way to construct the value out of a native Python data type.

### Optional Fields

When a field is configured via the `optional` parameter, not present in the

`Sequence`, but accessed, the `VOID` object will be returned. This is an object

that is serialized to an empty byte string and returns `None` when `.native` is

accessed.

## Set

The `Set` class is configured in the same was as `Sequence`, however it allows

serialized fields to be in any order, per the ASN.1 standard.

```python

from asn1crypto.core import Set, Integer, OctetString, IA5String

class MySet(Set):

    _fields = [

        ('field_one', Integer),

        ('field_two', OctetString),

        ('field_three', IA5String, {'optional': True}),

    ]

```

## SequenceOf

The `SequenceOf` class is used to allow for zero or more instances of a type.

The class uses the `_child_spec` property to define the instance class type.

```python

from asn1crypto.core import SequenceOf, Integer

class Integers(SequenceOf):

    _child_spec = Integer

```

Values in the `SequenceOf` can be accessed via `[]` with an integer key. The

length of the `SequenceOf` is determined via `len()`.

```python

values = Integers.load(der_byte_string)

for i in range(0, len(values)):

    print(values[i].native)

```

## SetOf

The `SetOf` class is an exact duplicate of `SequenceOf`. According to the ASN.1

standard, the difference is that a `SequenceOf` is explicitly ordered, however

`SetOf` may be in any order. This is an equivalent comparison of a Python `list`

and `set`.

```python

from asn1crypto.core import SetOf, Integer

class Integers(SetOf):

    _child_spec = Integer

```

## Integer

The `Integer` class allows values to be *named*. An `Integer` with named values

may contain any integer, however special values with named will be represented

as those names when `.native` is called.

Named values are configured via the `_map` property, which must be a `dict`

with the keys being integers and the values being unicode strings.

```python

from asn1crypto.core import Integer

class Version(Integer):

    _map = {

        1: 'v1',

        2: 'v2',

    }

# Will print: "v1"

print(Version(1).native)

# Will print: 4

print(Version(4).native)

```

## Enumerated

The `Enumerated` class is almost identical to `Integer`, however only values in

the `_map` property are valid.

```python

from asn1crypto.core import Enumerated

class Version(Enumerated):

    _map = {

        1: 'v1',

        2: 'v2',

    }

# Will print: "v1"

print(Version(1).native)

# Will raise a ValueError exception

print(Version(4).native)

```

## ObjectIdentifier

The `ObjectIdentifier` class represents values of the ASN.1 type of the same

name. `ObjectIdentifier` instances are converted to a unicode string in a

dotted-integer format when `.native` is accessed.

While this standard conversion is a reasonable baseline, in most situations

it will be more maintainable to map the OID strings to a unicode string

containing a description of what the OID repesents.

The mapping of OID strings to name strings is configured via the `_map`

property, which is a `dict` object with keys being unicode OID string and the

values being a unicode string.

The `.dotted` attribute will always return a unicode string of the dotted

integer form of the OID.

The class methods `.map()` and `.unmap()` will convert a dotted integer unicode

string to the user-friendly name, and vice-versa.

```python

from asn1crypto.core import ObjectIdentifier

class MyType(ObjectIdentifier):

    _map = {

        '1.8.2.1.23': 'value_name',

        '1.8.2.1.24': 'other_value',

    }

# Will print: "value_name"

print(MyType('1.8.2.1.23').native)

# Will print: "1.8.2.1.23"

print(MyType('1.8.2.1.23').dotted)

# Will print: "1.8.2.1.25"

print(MyType('1.8.2.1.25').native)

# Will print "value_name"

print(MyType.map('1.8.2.1.23'))

# Will print "1.8.2.1.23"

print(MyType.unmap('value_name'))

```

## BitString

When no `_map` is set for a `BitString` class, the native representation is a

`tuple` of `int`s (being either `1` or `0`).

```python

from asn1crypto.core import BitString

b1 = BitString((1, 0, 1))

```

Additionally, it is possible to set the `_map` property to a dict where the

keys are bit indexes and the values are unicode string names. This allows

checking the value of a given bit by item access, and the native representation

becomes a `set` of unicode strings.

```python

from asn1crypto.core import BitString

class MyFlags(BitString):

    _map = {

        0: 'edit',

        1: 'delete',

        2: 'manage_users',

    }

permissions = MyFlags({'edit', 'delete'})

# This will be printed

if permissions['edit'] and permissions['delete']:

    print('Can edit and delete')

# This will not

if 'manage_users' in permissions.native:

    print('Is admin')

```

## Strings

ASN.1 contains quite a number of string types:

| Type              | Standard Encoding                 | Implementation Encoding | Notes                                                                     |

| ----------------- | --------------------------------- | ----------------------- | ------------------------------------------------------------------------- |

| `UTF8String`      | UTF-8                             | UTF-8                   |                                                                           |

| `NumericString`   | ASCII `[0-9 ]`                    | ISO 8859-1              | The implementation is a superset of supported characters                  |

| `PrintableString` | ASCII `[a-zA-Z0-9 '()+,\\-./:=?]` | ISO 8859-1              | The implementation is a superset of supported characters                  |

| `TeletexString`   | ITU T.61                          | Custom                  | The implementation is based off of https://en.wikipedia.org/wiki/ITU_T.61 |

| `VideotexString`  | *?*                               | *None*                  | This has no set encoding, and it not used in cryptography                 |

| `IA5String`       | ITU T.50 (very similar to ASCII)  | ISO 8859-1              | The implementation is a superset of supported characters                  |

| `GraphicString`   | *                                 | ISO 8859-1              | This has not set encoding, but seems to often contain ISO 8859-1          |

| `VisibleString`   | ASCII (printable)                 | ISO 8859-1              | The implementation is a superset of supported characters                  |

| `GeneralString`   | *                                 | ISO 8859-1              | This has not set encoding, but seems to often contain ISO 8859-1          |

| `UniversalString` | UTF-32                            | UTF-32                  |                                                                           |

| `CharacterString` | *                                 | ISO 8859-1              | This has not set encoding, but seems to often contain ISO 8859-1          |

| `BMPString`       | UTF-16                            | UTF-16                  |                                                                           |

As noted in the table above, many of the implementations are supersets of the

supported characters. This simplifies parsing, but puts the onus of using valid

characters on the developer. However, in general `UTF8String`, `BMPString` or

`UniversalString` should be preferred when a choice is given.

All string types other than `VideotexString` are created from unicode strings.

```python

from asn1crypto.core import IA5String

print(IA5String('Testing!').native)

```

## UTCTime

The class `UTCTime` accepts a unicode string in one of the formats:

 - `%y%m%d%H%MZ`

 - `%y%m%d%H%M%SZ`

 - `%y%m%d%H%M%z`

 - `%y%m%d%H%M%S%z`

or a `datetime.datetime` instance. See the

[Python datetime strptime() reference](https://docs.python.org/3/library/datetime.html#strftime-and-strptime-behavior)

for details of the formats.

When `.native` is accessed, it returns a `datetime.datetime` object with a

`tzinfo` of `asn1crypto.util.timezone.utc`.

## GeneralizedTime

The class `GeneralizedTime` accepts a unicode string in one of the formats:

 - `%Y%m%d%H`

 - `%Y%m%d%H%M`

 - `%Y%m%d%H%M%S`

 - `%Y%m%d%H%M%S.%f`

 - `%Y%m%d%HZ`

 - `%Y%m%d%H%MZ`

 - `%Y%m%d%H%M%SZ`

 - `%Y%m%d%H%M%S.%fZ`

 - `%Y%m%d%H%z`

 - `%Y%m%d%H%M%z`

 - `%Y%m%d%H%M%S%z`

 - `%Y%m%d%H%M%S.%f%z`

or a `datetime.datetime` instance. See the

[Python datetime strptime() reference](https://docs.python.org/3/library/datetime.html#strftime-and-strptime-behavior)

for details of the formats.

When `.native` is accessed, it returns a `datetime.datetime` object with a

`tzinfo` of `asn1crypto.util.timezone.utc`. For formats where the time has a

timezone offset is specified (`[+-]\d{4}`), the time is converted to UTC. For

times without a timezone, the time is assumed to be in UTC.

## Choice

The `Choice` class allows handling ASN.1 Choice structures. The `_alternatives`

property must be set to a `list` containing 2-3 element `tuple`s. The first

element in the tuple is the alternative name. The second element is the type

class for the alternative. The, optional, third element is a `dict` of

parameters to pass to the type class constructor. This is used primarily for

implicit and explicit tagging.

```python

from asn1crypto.core import Choice, Integer, OctetString, IA5String

class MyChoice(Choice):

    _alternatives = [

        ('option_one', Integer),

        ('option_two', OctetString),

        ('option_three', IA5String),

    ]

```

`Choice` objects has two extra properties, `.name` and `.chosen`. The `.name`

property contains the name of the chosen alternative. The `.chosen` property

contains the instance of the chosen type class.

```python

parsed = MyChoice.load(der_bytes)

print(parsed.name)

print(type(parsed.chosen))

```

The `.native` property and `.dump()` method work as with the universal type

classes. Under the hood they just proxy the calls to the `.chosen` object.

## Any

The `Any` class implements the ASN.1 Any type, which allows any data type. By

default objects of this class do not perform any parsing. However, the

`.parse()` instance method allows parsing the contents of the `Any` object,

either into a universal type, or to a specification pass in via the `spec`

parameter.

This type is not used as a top-level structure, but instead allows `Sequence`

and `Set` objects to accept varying contents, usually based on some sort of

`ObjectIdentifier`.

```python

from asn1crypto.core import Sequence, ObjectIdentifier, Any, Integer, OctetString

class MySequence(Sequence):

    _fields = [

        ('type', ObjectIdentifier),

        ('value', Any),

    ]

```

## Specification via OID

Throughout the usage of ASN.1 in cryptography, a pattern is present where an

`ObjectIdenfitier` is used to determine what specification should be used to

interpret another field in a `Sequence`. Usually the other field is an instance

of `Any`, however occasionally it is an `OctetString` or `OctetBitString`.

*asn1crypto* provides the `_oid_pair` and `_oid_specs` properties of the

`Sequence` class to allow handling these situations.

The `_oid_pair` is a tuple with two unicode string elements. The first is the

name of the field that is an `ObjectIdentifier` and the second if the name of

the field that has a variable specification based on the first field. *In

situations where the value field should be an `OctetString` or `OctetBitString`,

`ParsableOctetString` and `ParsableOctetBitString` will need to be used instead

to allow for the sub-parsing of the contents.*

The `_oid_specs` property is a `dict` object with `ObjectIdentifier` values as

the keys (either dotted or mapped notation) and a type class as the value. When

the first field in `_oid_pair` has a value equal to one of the keys in

`_oid_specs`, then the corresponding type class will be used as the

specification for the second field of `_oid_pair`.

```python

from asn1crypto.core import Sequence, ObjectIdentifier, Any, OctetString, Integer

class MyId(ObjectIdentifier):

    _map = {

        '1.2.3.4': 'initialization_vector',

        '1.2.3.5': 'iterations',

    }

class MySequence(Sequence):

    _fields = [

        ('type', MyId),

        ('value', Any),

    ]

    _oid_pair = ('type', 'value')

    _oid_specs = {

        'initialization_vector': OctetString,

        'iterations': Integer,

    }

```

## Explicit and Implicit Tagging

When working with `Sequence`, `Set` and `Choice` it is often necessary to

disambiguate between fields because of a number of factors:

 - In `Sequence` the presence of an optional field must be determined by tag number

 - In `Set`, each field must have a different tag number since they can be in any order

 - In `Choice`, each alternative must have a different tag number to determine which is present

The universal types all have unique tag numbers. However, if a `Sequence`, `Set`

or `Choice` has more than one field with the same universal type, tagging allows

a way to keep the semantics of the original type, but with a different tag

number.

Implicit tagging simply changes the tag number of a type to a different value.

However, Explicit tagging wraps the existing type in another tag with the

specified tag number.

In general, most situations allow for implicit tagging, with the notable

exception than a field that is a `Choice` type must always be explicitly tagged.

Otherwise, using implicit tagging would modify the tag of the chosen

alternative, breaking the mechanism by which `Choice` works.

Here is an example of implicit and explicit tagging where explicit tagging on

the `Sequence` allows a `Choice` type field to be optional, and where implicit

tagging in the `Choice` structure allows disambiguating between two string of

the same type.

```python

from asn1crypto.core import Sequence, Choice, IA5String, UTCTime, ObjectIdentifier

class Person(Choice):

    _alternatives = [

        ('name', IA5String),

        ('email', IA5String, {'implicit': 0}),

    ]

class Record(Sequence):

    _fields = [

        ('id', ObjectIdentifier),

        ('created', UTCTime),

        ('creator', Person, {'explicit': 0, 'optional': True}),

    ]

```

As is shown above, the keys `implicit` and `explicit` are used for tagging,

and are passed to a type class constructor via the optional third element of

a field or alternative tuple. Both parameters may be an integer tag number, or

a 2-element tuple of string class name and integer tag.

If a tagging value needs its tagging changed, the `.untag()` method can be used

to create a copy of the object without explicit/implicit tagging. The `.retag()`

method can be used to change the tagging. This method accepts one parameter, a

dict with either or both of the keys `implicit` and `explicit`.

```python

person = Person(name='email', value='will@wbond.net')

# Will display True

print(person.implicit)

# Will display False

print(person.untag().implicit)

# Will display 0

print(person.tag)

# Will display 1

print(person.retag({'implicit': 1}).tag)

```