==============================

How to read the source of lxml

==============================

:Author:

  Stefan Behnel

.. meta::

  :description: How to read and work on the source code of lxml

  :keywords: lxml, XML, Cython, source code, develop, comprehend, understand

This document describes how to read the source code of lxml_ and how

to start working on it.  You might also be interested in the companion

document that describes `how to build lxml from sources`_.

.. _lxml: http://lxml.de/

.. _`how to build lxml from sources`: build.html

.. _`ReStructured Text`: http://docutils.sourceforge.net/rst.html

.. _epydoc: http://epydoc.sourceforge.net/

.. _docutils: http://docutils.sourceforge.net/

.. _`C-level API`: capi.html

.. contents::

..

   1  What is Cython?

   2  Where to start?

     2.1  Concepts

     2.2  The documentation

   3  lxml.etree

   4  lxml.objectify

   5  lxml.html

What is Cython?

===============

.. _Cython: http://cython.org/

.. _Pyrex: http://www.cosc.canterbury.ac.nz/~greg/python/Pyrex/

Cython_ is the language that lxml is written in.  It is a very

Python-like language that was specifically designed for writing Python

extension modules.

The reason why Cython (or actually its predecessor Pyrex_ at the time)

was chosen as an implementation language for lxml, is that it makes it

very easy to interface with both the Python world and external C code.

Cython generates all the necessary glue code for the Python API,

including Python types, calling conventions and reference counting.

On the other side of the table, calling into C code is not more than

declaring the signature of the function and maybe some variables as

being C types, pointers or structs, and then calling it.  The rest of

the code is just plain Python code.

The Cython language is so close to Python that the Cython compiler can

actually compile many, many Python programs to C without major

modifications.  But the real speed gains of a C compilation come from

type annotations that were added to the language and that allow Cython

to generate very efficient C code.

Even if you are not familiar with Cython, you should keep in mind that

a slow implementation of a feature is better than none.  So, if you

want to contribute and have an idea what code you want to write, feel

free to start with a pure Python implementation.  Chances are, if you

get the change officially accepted and integrated, others will take

the time to optimise it so that it runs fast in Cython.

Where to start?

===============

First of all, read `how to build lxml from sources`_ to learn how to

retrieve the source code from the GitHub repository and how to

build it.  The source code lives in the subdirectory ``src`` of the

checkout.

The main extension modules in lxml are ``lxml.etree`` and

``lxml.objectify``.  All main modules have the file extension

``.pyx``, which shows the descendence from Pyrex.  As usual in Python,

the main files start with a short description and a couple of imports.

Cython distinguishes between the run-time ``import`` statement (as

known from Python) and the compile-time ``cimport`` statement, which

imports C declarations, either from external libraries or from other

Cython modules.

Concepts

--------

lxml's tree API is based on proxy objects.  That means, every Element

object (or rather ``_Element`` object) is a proxy for a libxml2 node

structure.  The class declaration is (mainly)::

    cdef class _Element:

        cdef _Document _doc

        cdef xmlNode* _c_node

It is a naming convention that C variables and C level class members

that are passed into libxml2 start with a prefixed ``c_`` (commonly

libxml2 struct pointers), and that C level class members are prefixed

with an underscore.  So you will often see names like ``c_doc`` for an

``xmlDoc*`` variable (or ``c_node`` for an ``xmlNode*``), or the above

``_c_node`` for a class member that points to an ``xmlNode`` struct

(or ``_c_doc`` for an ``xmlDoc*``).

It is important to know that every proxy in lxml has a factory

function that properly sets up C level members.  Proxy objects must

*never* be instantiated outside of that factory.  For example, to

instantiate an _Element object or its subclasses, you must always call

its factory function::

    cdef xmlNode* c_node

    cdef _Document doc

    cdef _Element element

    ...

    element = _elementFactory(doc, c_node)

A good place to see how this factory is used are the Element methods

``getparent()``, ``getnext()`` and ``getprevious()``.

The documentation

-----------------

An important part of lxml is the documentation that lives in the

``doc`` directory.  It describes a large part of the API and comprises

a lot of example code in the form of doctests.

The documentation is written in the `ReStructured Text`_ format, a

very powerful text markup language that looks almost like plain text.

It is part of the docutils_ package.

The project web site of lxml_ is completely generated from these text

documents.  Even the side menu is just collected from the table of

contents that the ReST processor writes into each HTML page.

Obviously, we use lxml for this.

The easiest way to generate the HTML pages is by calling::

    make html

This will call the script ``doc/mkhtml.py`` to run the ReST processor

on the files.  After generating an HTML page the script parses it back

in to build the side menu, and injects the complete menu into each

page at the very end.

Running the ``make`` command will also generate the API documentation

if you have epydoc_ installed.  The epydoc package will import and

introspect the extension modules and also introspect and parse the

Python modules of lxml.  The aggregated information will then be

written out into an HTML documentation site.

lxml.etree

==========

The main module, ``lxml.etree``, is in the file `lxml.etree.pyx

<https://github.com/lxml/lxml/blob/master/src/lxml/lxml.etree.pyx>`_.  It

implements the main functions and types of the ElementTree API, as

well as all the factory functions for proxies.  It is the best place

to start if you want to find out how a specific feature is

implemented.

At the very end of the file, it contains a series of ``include``

statements that merge the rest of the implementation into the

generated C code.  Yes, you read right: no importing, no source file

namespacing, just plain good old include and a huge C code result of

more than 100,000 lines that we throw right into the C compiler.

The main include files are:

apihelpers.pxi

    Private C helper functions.  Except for the factory functions,

    most of the little functions that are used all over the place are

    defined here.  This includes things like reading out the text

    content of a libxml2 tree node, checking input from the API level,

    creating a new Element node or handling attribute values.  If you

    want to work on the lxml code, you should keep these functions in

    the back of your head, as they will definitely make your life

    easier.

classlookup.pxi

    Element class lookup mechanisms.  The main API and engines for

    those who want to define custom Element classes and inject them

    into lxml.

docloader.pxi

    Support for custom document loaders.  Base class and registry for

    custom document resolvers.

extensions.pxi

    Infrastructure for extension functions in XPath/XSLT, including

    XPath value conversion and function registration.

iterparse.pxi

    Incremental XML parsing.  An iterator class that builds iterparse

    events while parsing.

nsclasses.pxi

    Namespace implementation and registry.  The registry and engine

    for Element classes that use the ElementNamespaceClassLookup

    scheme.

parser.pxi

    Parsers for XML and HTML.  This is the main parser engine.  It's

    the reason why you can parse a document from various sources in

    two lines of Python code.  It's definitely not the right place to

    start reading lxml's source code.

parsertarget.pxi

    An ElementTree compatible parser target implementation based on

    the SAX2 interface of libxml2.

proxy.pxi

    Very low-level functions for memory allocation/deallocation

    and Element proxy handling.  Ignoring this for the beginning

    will safe your head from exploding.

public-api.pxi

    The set of C functions that are exported to other extension

    modules at the C level.  For example, ``lxml.objectify`` makes use

    of these.  See the `C-level API` documentation.

readonlytree.pxi

    A separate read-only implementation of the Element API.  This is

    used in places where non-intrusive access to a tree is required,

    such as the ``PythonElementClassLookup`` or XSLT extension

    elements.

saxparser.pxi

    SAX-like parser interfaces as known from ElementTree's TreeBuilder.

serializer.pxi

    XML output functions.  Basically everything that creates byte

    sequences from XML trees.

xinclude.pxi

    XInclude support.

xmlerror.pxi

    Error log handling.  All error messages that libxml2 generates

    internally walk through the code in this file to end up in lxml's

    Python level error logs.

    At the end of the file, you will find a long list of named error

    codes.  It is generated from the libxml2 HTML documentation (using

    lxml, of course).  See the script ``update-error-constants.py``

    for this.

xmlid.pxi

    XMLID and IDDict, a dictionary-like way to find Elements by their

    XML-ID attribute.

xpath.pxi

    XPath evaluators.

xslt.pxi

    XSL transformations, including the ``XSLT`` class, document lookup

    handling and access control.

The different schema languages (DTD, RelaxNG, XML Schema and

Schematron) are implemented in the following include files:

* dtd.pxi

* relaxng.pxi

* schematron.pxi

* xmlschema.pxi

Python modules

==============

The ``lxml`` package also contains a number of pure Python modules:

builder.py

    The E-factory and the ElementBuilder class.  These provide a

    simple interface to XML tree generation.

cssselect.py

    A CSS selector implementation based on XPath.  The main class is

    called ``CSSSelector``.

doctestcompare.py

    A relaxed comparison scheme for XML/HTML markup in doctest.

ElementInclude.py

    XInclude-like document inclusion, compatible with ElementTree.

_elementpath.py

    XPath-like path language, compatible with ElementTree.

sax.py

    SAX2 compatible interfaces to copy lxml trees from/to SAX compatible

    tools.

usedoctest.py

    Wrapper module for ``doctestcompare.py`` that simplifies its usage

    from inside a doctest.

lxml.objectify

==============

A Cython implemented extension module that uses the public C-API of

lxml.etree.  It provides a Python object-like interface to XML trees.

The implementation resides in the file `lxml.objectify.pyx

<https://github.com/lxml/lxml/blob/master/src/lxml/lxml.objectify.pyx>`_.

lxml.html

=========

A specialised toolkit for HTML handling, based on lxml.etree.  This is

implemented in pure Python.