Design document for the unified scheme data

===========================================

How are things connected?

-------------------------

The unified scheme takes all its data from the build.info files seen

throughout the source tree.  These files hold the minimum information

needed to build end product files from diverse sources.  See the

section on build.info files below.

From the information in build.info files, Configure builds up an

information database as a hash table called %unified_info, which is

stored in configdata.pm, found at the top of the build tree (which may

or may not be the same as the source tree).

Configurations/common.tmpl uses the data from %unified_info to

generate the rules for building end product files as well as

intermediary files with the help of a few functions found in the

build-file templates.  See the section on build-file templates further

down for more information.

build.info files

----------------

As mentioned earlier, build.info files are meant to hold the minimum

information needed to build output files, and therefore only (with a

few possible exceptions [1]) have information about end products (such

as scripts, library files and programs) and source files (such as C

files, C header files, assembler files, etc).  Intermediate files such

as object files are rarely directly referred to in build.info files (and

when they are, it's always with the file name extension .o), they are

inferred by Configure.  By the same rule of minimalism, end product

file name extensions (such as .so, .a, .exe, etc) are never mentioned

in build.info.  Their file name extensions will be inferred by the

build-file templates, adapted for the platform they are meant for (see

sections on %unified_info and build-file templates further down).

The variables PROGRAMS, LIBS, ENGINES and SCRIPTS are used to declare

end products.  There are variants for them with '_NO_INST' as suffix

(PROGRAM_NO_INST etc) to specify end products that shouldn't get

installed.

The variables SOURCE, DEPEND and INCLUDE are indexed by a produced

file, and their values are the source used to produce that particular

produced file, extra dependencies, and include directories needed.

All their values in all the build.info throughout the source tree are

collected together and form a set of programs, libraries, engines and

scripts to be produced, source files, dependencies, etc etc etc.

Let's have a pretend example, a very limited contraption of OpenSSL,

composed of the program 'apps/openssl', the libraries 'libssl' and

'libcrypto', an engine 'engines/ossltest' and their sources and

dependencies.

    # build.info

    LIBS=libcrypto libssl

    INCLUDE[libcrypto]=include

    INCLUDE[libssl]=include

    DEPEND[libssl]=libcrypto

This is the top directory build.info file, and it tells us that two

libraries are to be built, the include directory 'include/' shall be

used throughout when building anything that will end up in each

library, and that the library 'libssl' depend on the library

'libcrypto' to function properly.

    # apps/build.info

    PROGRAMS=openssl

    SOURCE[openssl]=openssl.c

    INCLUDE[openssl]=.. ../include

    DEPEND[openssl]=../libssl

This is the build.info file in 'apps/', one may notice that all file

paths mentioned are relative to the directory the build.info file is

located in.  This one tells us that there's a program to be built

called 'apps/openssl' (the file name extension will depend on the

platform and is therefore not mentioned in the build.info file).  It's

built from one source file, 'apps/openssl.c', and building it requires

the use of '.' and 'include' include directories (both are declared

from the point of view of the 'apps/' directory), and that the program

depends on the library 'libssl' to function properly.

    # crypto/build.info

    LIBS=../libcrypto

    SOURCE[../libcrypto]=aes.c evp.c cversion.c

    DEPEND[cversion.o]=buildinf.h

    GENERATE[buildinf.h]=../util/mkbuildinf.pl "$(CC) $(CFLAGS)" "$(PLATFORM)"

    DEPEND[buildinf.h]=../Makefile

    DEPEND[../util/mkbuildinf.pl]=../util/Foo.pm

This is the build.info file in 'crypto', and it tells us a little more

about what's needed to produce 'libcrypto'.  LIBS is used again to

declare that 'libcrypto' is to be produced.  This declaration is

really unnecessary as it's already mentioned in the top build.info

file, but can make the info file easier to understand.  This is to

show that duplicate information isn't an issue.

This build.info file informs us that 'libcrypto' is built from a few

source files, 'crypto/aes.c', 'crypto/evp.c' and 'crypto/cversion.c'.

It also shows us that building the object file inferred from

'crypto/cversion.c' depends on 'crypto/buildinf.h'.  Finally, it

also shows the possibility to declare how some files are generated

using some script, in this case a perl script, and how such scripts

can be declared to depend on other files, in this case a perl module.

Two things are worth an extra note:

'DEPEND[cversion.o]' mentions an object file.  DEPEND indexes is the

only location where it's valid to mention them

Lines in 'BEGINRAW'..'ENDRAW' sections must always mention files as

seen from the top directory, no exception.

    # ssl/build.info

    LIBS=../libssl

    SOURCE[../libssl]=tls.c

This is the build.info file in 'ssl/', and it tells us that the

library 'libssl' is built from the source file 'ssl/tls.c'.

    # engines/build.info

    ENGINES=dasync

    SOURCE[dasync]=e_dasync.c

    DEPEND[dasync]=../libcrypto

    INCLUDE[dasync]=../include

    ENGINES_NO_INST=ossltest

    SOURCE[ossltest]=e_ossltest.c

    DEPEND[ossltest]=../libcrypto.a

    INCLUDE[ossltest]=../include

This is the build.info file in 'engines/', telling us that two engines

called 'engines/dasync' and 'engines/ossltest' shall be built, that

dasync's source is 'engines/e_dasync.c' and ossltest's source is

'engines/e_ossltest.c' and that the include directory 'include/' may

be used when building anything that will be part of these engines.

Also, both engines depend on the library 'libcrypto' to function

properly.  ossltest is explicitly linked with the static variant of

the library 'libcrypto'.  Finally, only dasync is being installed, as

ossltest is only for internal testing.

When Configure digests these build.info files, the accumulated

information comes down to this:

    LIBS=libcrypto libssl

    SOURCE[libcrypto]=crypto/aes.c crypto/evp.c crypto/cversion.c

    DEPEND[crypto/cversion.o]=crypto/buildinf.h

    INCLUDE[libcrypto]=include

    SOURCE[libssl]=ssl/tls.c

    INCLUDE[libssl]=include

    DEPEND[libssl]=libcrypto

    PROGRAMS=apps/openssl

    SOURCE[apps/openssl]=apps/openssl.c

    INCLUDE[apps/openssl]=. include

    DEPEND[apps/openssl]=libssl

    ENGINES=engines/dasync

    SOURCE[engines/dasync]=engines/e_dasync.c

    DEPEND[engines/dasync]=libcrypto

    INCLUDE[engines/dasync]=include

    ENGINES_NO_INST=engines/ossltest

    SOURCE[engines/ossltest]=engines/e_ossltest.c

    DEPEND[engines/ossltest]=libcrypto.a

    INCLUDE[engines/ossltest]=include

    GENERATE[crypto/buildinf.h]=util/mkbuildinf.pl "$(CC) $(CFLAGS)" "$(PLATFORM)"

    DEPEND[crypto/buildinf.h]=Makefile

    DEPEND[util/mkbuildinf.pl]=util/Foo.pm

A few notes worth mentioning:

LIBS may be used to declare routine libraries only.

PROGRAMS may be used to declare programs only.

ENGINES may be used to declare engines only.

The indexes for SOURCE must only be end product files, such as

libraries, programs or engines.  The values of SOURCE variables must

only be source files (possibly generated).

INCLUDE and DEPEND shows a relationship between different files

(usually produced files) or between files and directories, such as a

program depending on a library, or between an object file and some

extra source file.

When Configure processes the build.info files, it will take it as

truth without question, and will therefore perform very few checks.

If the build tree is separate from the source tree, it will assume

that all built files and up in the build directory and that all source

files are to be found in the source tree, if they can be found there.

Configure will assume that source files that can't be found in the

source tree (such as 'crypto/bildinf.h' in the example above) are

generated and will be found in the build tree.

The %unified_info database

--------------------------

The information in all the build.info get digested by Configure and

collected into the %unified_info database, divided into the following

indexes:

  depends   => a hash table containing 'file' => [ 'dependency' ... ]

               pairs.  These are directly inferred from the DEPEND

               variables in build.info files.

  engines   => a list of engines.  These are directly inferred from

               the ENGINES variable in build.info files.

  generate  => a hash table containing 'file' => [ 'generator' ... ]

               pairs.  These are directly inferred from the GENERATE

               variables in build.info files.

  includes  => a hash table containing 'file' => [ 'include' ... ]

               pairs.  These are directly inferred from the INCLUDE

               variables in build.info files.

  install   => a hash table containing 'type' => [ 'file' ... ] pairs.

               The types are 'programs', 'libraries', 'engines' and

               'scripts', and the array of files list the files of

               that type that should be installed.

  libraries => a list of libraries.  These are directly inferred from

               the LIBS variable in build.info files.

  programs  => a list of programs.  These are directly inferred from

               the PROGRAMS variable in build.info files.

  rawlines  => a list of build-file lines.  These are a direct copy of

               the BEGINRAW..ENDRAW lines in build.info files.  Note:

               only the BEGINRAW..ENDRAW section for the current

               platform are copied, the rest are ignored.

  scripts   => a list of scripts.  There are directly inferred from

               the SCRIPTS variable in build.info files.

  sources   => a hash table containing 'file' => [ 'sourcefile' ... ]

               pairs.  These are indirectly inferred from the SOURCE

               variables in build.info files.  Object files are

               mentioned in this hash table, with source files from

               SOURCE variables, and AS source files for programs and

               libraries.

  shared_sources =>

               a hash table just like 'sources', but only as source

               files (object files) for building shared libraries.

As an example, here is how the build.info files example from the

section above would be digested into a %unified_info table:

    our %unified_info = (

        "depends" =>

            {

                "apps/openssl" =>

                    [

                        "libssl",

                    ],

                "crypto/buildinf.h" =>

                    [

                        "Makefile",

                    ],

                "crypto/cversion.o" =>

                    [

                        "crypto/buildinf.h",

                    ],

                "engines/dasync" =>

                    [

                        "libcrypto",

                    ],

                "engines/ossltest" =>

                    [

                        "libcrypto.a",

                    ],

                "libssl" =>

                    [

                        "libcrypto",

                    ],

                "util/mkbuildinf.pl" =>

                    [

                        "util/Foo.pm",

                    ],

            },

        "engines" =>

            [

                "engines/dasync",

                "engines/ossltest",

            ],

        "generate" =>

            {

                "crypto/buildinf.h" =>

                    [

                        "util/mkbuildinf.pl",

                        "\"\$(CC)",

                        "\$(CFLAGS)\"",

                        "\"$(PLATFORM)\"",

                    ],

            },

        "includes" =>

            {

                "apps/openssl" =>

                    [

                        ".",

                        "include",

                    ],

                "engines/ossltest" =>

                    [

                        "include"

                    ],

                "libcrypto" =>

                    [

                        "include",

                    ],

                "libssl" =>

                    [

                        "include",

                    ],

                "util/mkbuildinf.pl" =>

                    [

                        "util",

                    ],

            }

        "install" =>

            {

                "engines" =>

                    [

                        "engines/dasync",

                    ],

                "libraries" =>

                    [

                        "libcrypto",

                        "libssl",

                    ],

                "programs" =>

                    [

                        "apps/openssl",

                    ],

           },

        "libraries" =>

            [

                "libcrypto",

                "libssl",

            ],

        "programs" =>

            [

                "apps/openssl",

            ],

        "rawlines" =>

            [

            ],

        "sources" =>

            {

                "apps/openssl" =>

                    [

                        "apps/openssl.o",

                    ],

                "apps/openssl.o" =>

                    [

                        "apps/openssl.c",

                    ],

                "crypto/aes.o" =>

                    [

                        "crypto/aes.c",

                    ],

                "crypto/cversion.o" =>

                    [

                        "crypto/cversion.c",

                    ],

                "crypto/evp.o" =>

                    [

                        "crypto/evp.c",

                    ],

                "engines/e_dasync.o" =>

                    [

                        "engines/e_dasync.c",

                    ],

                "engines/dasync" =>

                    [

                        "engines/e_dasync.o",

                    ],

                "engines/e_ossltest.o" =>

                    [

                        "engines/e_ossltest.c",

                    ],

                "engines/ossltest" =>

                    [

                        "engines/e_ossltest.o",

                    ],

                "libcrypto" =>

                    [

                        "crypto/aes.c",

                        "crypto/cversion.c",

                        "crypto/evp.c",

                    ],

                "libssl" =>

                    [

                        "ssl/tls.c",

                    ],

                "ssl/tls.o" =>

                    [

                        "ssl/tls.c",

                    ],

            },

    );

As can be seen, everything in %unified_info is fairly simple suggest

of information.  Still, it tells us that to build all programs, we

must build 'apps/openssl', and to build the latter, we will need to

build all its sources ('apps/openssl.o' in this case) and all the

other things it depends on (such as 'libssl').  All those dependencies

need to be built as well, using the same logic, so to build 'libssl',

we need to build 'ssl/tls.o' as well as 'libcrypto', and to build the

latter...

Build-file templates

--------------------

Build-file templates are essentially build-files (such as Makefile on

Unix) with perl code fragments mixed in.  Those perl code fragment

will generate all the configuration dependent data, including all the

rules needed to build end product files and intermediary files alike.

At a minimum, there must be a perl code fragment that defines a set of

functions that are used to generates specific build-file rules, to

build static libraries from object files, to build shared libraries

from static libraries, to programs from object files and libraries,

etc.

    generatesrc - function that produces build file lines to generate

                  a source file from some input.

                  It's called like this:

                        generatesrc(src => "PATH/TO/tobegenerated",

                                    generator => [ "generatingfile", ... ]

                                    generator_incs => [ "INCL/PATH", ... ]

                                    generator_deps => [ "dep1", ... ]

                                    incs => [ "INCL/PATH", ... ],

                                    deps => [ "dep1", ... ],

                                    intent => one of "libs", "dso", "bin" );

                  'src' has the name of the file to be generated.

                  'generator' is the command or part of command to

                  generate the file, of which the first item is

                  expected to be the file to generate from.

                  generatesrc() is expected to analyse and figure out

                  exactly how to apply that file and how to capture

                  the result.  'generator_incs' and 'generator_deps'

                  are include directories and files that the generator

                  file itself depends on.  'incs' and 'deps' are

                  include directories and files that are used if $(CC)

                  is used as an intermediary step when generating the

                  end product (the file indicated by 'src').  'intent'

                  indicates what the generated file is going to be

                  used for.

    src2obj     - function that produces build file lines to build an

                  object file from source files and associated data.

                  It's called like this:

                        src2obj(obj => "PATH/TO/objectfile",

                                srcs => [ "PATH/TO/sourcefile", ... ],

                                deps => [ "dep1", ... ],

                                incs => [ "INCL/PATH", ... ]

                                intent => one of "lib", "dso", "bin" );

                  'obj' has the intended object file *without*

                  extension, src2obj() is expected to add that.

                  'srcs' has the list of source files to build the

                  object file, with the first item being the source

                  file that directly corresponds to the object file.

                  'deps' is a list of explicit dependencies.  'incs'

                  is a list of include file directories.  Finally,

                  'intent' indicates what this object file is going

                  to be used for.

    obj2lib     - function that produces build file lines to build a

                  static library file ("libfoo.a" in Unix terms) from

                  object files.

                  called like this:

                        obj2lib(lib => "PATH/TO/libfile",

                                objs => [ "PATH/TO/objectfile", ... ]);

                  'lib' has the intended library file name *without*

                  extension, obj2lib is expected to add that.  'objs'

                  has the list of object files (also *without*

                  extension) to build this library.

    libobj2shlib - function that produces build file lines to build a

                  shareable object library file ("libfoo.so" in Unix

                  terms) from the corresponding static library file

                  or object files.

                  called like this:

                        libobj2shlib(shlib => "PATH/TO/shlibfile",

                                     lib => "PATH/TO/libfile",

                                     objs => [ "PATH/TO/objectfile", ... ],

                                     deps => [ "PATH/TO/otherlibfile", ... ]);

                  'lib' has the intended library file name *without*

                  extension, libobj2shlib is expected to add that.

                  'shlib' has the corresponding shared library name

                  *without* extension.  'deps' has the list of other

                  libraries (also *without* extension) this library

                  needs to be linked with.  'objs' has the list of

                  object files (also *without* extension) to build

                  this library.

                  This function has a choice; it can use the

                  corresponding static library as input to make the

                  shared library, or the list of object files.

    obj2dynlib  - function that produces build file lines to build a

                  dynamically loadable library file ("libfoo.so" on

                  Unix) from object files.

                  called like this:

                        obj2dynlib(lib => "PATH/TO/libfile",

                                   objs => [ "PATH/TO/objectfile", ... ],

                                   deps => [ "PATH/TO/otherlibfile",

                                   ... ]);

                  This is almost the same as libobj2shlib, but the

                  intent is to build a shareable library that can be

                  loaded in runtime (a "plugin"...).  The differences

                  are subtle, one of the most visible ones is that the

                  resulting shareable library is produced from object

                  files only.

    obj2bin     - function that produces build file lines to build an

                  executable file from object files.

                  called like this:

                        obj2bin(bin => "PATH/TO/binfile",

                                objs => [ "PATH/TO/objectfile", ... ],

                                deps => [ "PATH/TO/libfile", ... ]);

                  'bin' has the intended executable file name

                  *without* extension, obj2bin is expected to add

                  that.  'objs' has the list of object files (also

                  *without* extension) to build this library.  'deps'

                  has the list of library files (also *without*

                  extension) that the programs needs to be linked

                  with.

    in2script   - function that produces build file lines to build a

                  script file from some input.

                  called like this:

                        in2script(script => "PATH/TO/scriptfile",

                                  sources => [ "PATH/TO/infile", ... ]);

                  'script' has the intended script file name.

                  'sources' has the list of source files to build the

                  resulting script from.

Along with the build-file templates is the driving engine

Configurations/common.tmpl, which looks through all the information in

%unified_info and generates all the rulesets to build libraries,

programs and all intermediate files, using the rule generating

functions defined in the build-file template.

As an example with the smaller build.info set we've seen as an

example, producing the rules to build 'libcrypto' would result in the

following calls:

    # Note: libobj2shlib will only be called if shared libraries are

    # to be produced.

    # Note 2: libobj2shlib gets both the name of the static library

    # and the names of all the object files that go into it.  It's up

    # to the implementation to decide which to use as input.

    # Note 3: common.tmpl peals off the ".o" extension from all object

    # files, as the platform at hand may have a different one.

    libobj2shlib(shlib => "libcrypto",

                 lib => "libcrypto",

                 objs => [ "crypto/aes", "crypto/evp", "crypto/cversion" ],

                 deps => [  ]);

    obj2lib(lib => "libcrypto"

            objs => [ "crypto/aes", "crypto/evp", "crypto/cversion" ]);

    src2obj(obj => "crypto/aes"

            srcs => [ "crypto/aes.c" ],

            deps => [ ],

            incs => [ "include" ],

            intent => "lib");

    src2obj(obj => "crypto/evp"

            srcs => [ "crypto/evp.c" ],

            deps => [ ],

            incs => [ "include" ],

            intent => "lib");

    src2obj(obj => "crypto/cversion"

            srcs => [ "crypto/cversion.c" ],

            deps => [ "crypto/buildinf.h" ],

            incs => [ "include" ],

            intent => "lib");

    generatesrc(src => "crypto/buildinf.h",

                generator => [ "util/mkbuildinf.pl", "\"$(CC)",

                               "$(CFLAGS)\"", "\"$(PLATFORM)\"" ],

                generator_incs => [ "util" ],

                generator_deps => [ "util/Foo.pm" ],

                incs => [ ],

                deps => [ ],

                intent => "lib");

The returned strings from all those calls are then concatenated

together and written to the resulting build-file.