System Databases and Name Service Switch
****************************************

Various functions in the C Library need to be configured to work correctly in the local environment.  Traditionally, this was done by using files (e.g., /etc/passwd), but other nameservices (like the Network Information Service (NIS) and the Domain Name Service (DNS)) became popular, and were hacked into the C library, usually with a fixed search order.

Theglibc contains a cleaner solution to this problem.  It is designed after a method used by Sun Microsystems in the C library of Solaris 2.  Theglibc follows their name and calls this scheme Name Service Switch (NSS).

Though the interface might be similar to Sun's version there is no common code.  We never saw any source code of Sun's implementation and so the internal interface is incompatible.  This also manifests in the file names we use as we will see later.

NSS Basics
==========

The basic idea is to put the implementation of the different services offered to access the databases in separate modules.  This has some advantages:

 1. Contributors can add new services without adding them to theglibc.

 2. The modules can be updated separately.

 3. The C library image is smaller.

To fulfill the first goal above, the ABI of the modules will be described below.  For getting the implementation of a new service right it is important to understand how the functions in the modules get called. They are in no way designed to be used by the programmer directly. Instead the programmer should only use the documented and standardized functions to access the databases.

The databases available in the NSS are

aliases
     Mail aliases

ethers
     Ethernet numbers,

group
     Groups of users, see Group Database.

gshadow
     Group passphrase hashes and related information.

hosts
     Host names and numbers, see Host Names.

initgroups
     Supplementary group access list.

netgroup
     Network wide list of host and users, see Netgroup Database.

networks
     Network names and numbers, see Networks Database.

passwd
     User identities, see User Database.

protocols
     Network protocols, see Protocols Database.

publickey
     Public keys for Secure RPC.

rpc
     Remote procedure call names and numbers.

services
     Network services, see Services Database.

shadow
     User passphrase hashes and related information.

More databases may be added later.

The NSS Configuration File
==========================

Somehow the NSS code must be told about the wishes of the user.  For this reason there is the file /etc/nsswitch.conf.  For each database, this file contains a specification of how the lookup process should work.  The file could look like this:

          [file nsswitch.texi not included]

The first column is the database as you can guess from the table above. The rest of the line specifies how the lookup process works.  Please note that you specify the way it works for each database individually. This cannot be done with the old way of a monolithic implementation.

The configuration specification for each database can contain two different items:

  * the service specification like files, db, or nis.

  * the reaction on lookup result like [NOTFOUND=return].

Services in the NSS configuration File
--------------------------------------

The above example file mentions five different services: files, db, dns, nis, and nisplus.  This does not mean these services are available on all sites and neither does it mean these are all the services which will ever be available.

In fact, these names are simply strings which the NSS code uses to find the implicitly addressed functions.  The internal interface will be described later.  Visible to the user are the modules which implement an individual service.

Assume the service NAME shall be used for a lookup.  The code for this service is implemented in a module called libnss_NAME. On a system supporting shared libraries this is in fact a shared library with the name (for example) libnss_NAME.so.2.  The number at the end is the currently used version of the interface which will not change frequently.  Normally the user should not have to be cognizant of these files since they should be placed in a directory where they are found automatically.  Only the names of all available services are important.

Lastly, some system software may make use of the NSS configuration file to store their own configuration for similar purposes.  Examples of this include the automount service which is used by autofs.

Actions in the NSS configuration
--------------------------------

The second item in the specification gives the user much finer control on the lookup process.  Action items are placed between two service names and are written within brackets.  The general form is

     [ ( !? STATUS = ACTION )+ ]

where

     STATUS => success | notfound | unavail | tryagain
     ACTION => return | continue

The case of the keywords is insignificant.  The STATUS values are the results of a call to a lookup function of a specific service.  They mean:

success
     No error occurred and the wanted entry is returned.  The default action for this is return.

notfound
     The lookup process works ok but the needed value was not found.  The default action is continue.

unavail
     The service is permanently unavailable.  This can either mean the needed file is not available, or, for DNS, the server is not available or does not allow queries.  The default action is continue.

tryagain
     The service is temporarily unavailable.  This could mean a file is locked or a server currently cannot accept more connections.  The default action is continue.

The ACTION values mean:

return

     If the status matches, stop the lookup process at this service specification.  If an entry is available, provide it to the application. If an error occurred, report it to the application.  In case of a prior merge action, the data is combined with previous lookup results, as explained below.

continue

     If the status matches, proceed with the lookup process at the next entry, discarding the result of the current lookup (and any merged data).  An exception is the initgroups database and the success status, where continue acts like merge below.

merge

     Proceed with the lookup process, retaining the current lookup result. This action is useful only with the success status.  If a subsequent service lookup succeeds and has a matching return specification, the results are merged, the lookup process ends, and the merged results are returned to the application.  If the following service has a matching merge action, the lookup process continues, retaining the combined data from this and any previous lookups.

     After a merge action, errors from subsequent lookups are ignored, and the data gathered so far will be returned.

     The merge only applies to the success status.  It is currently implemented for the group database and its group members field, gr_mem.  If specified for other databases, it causes the lookup to fail (if the STATUS matches).

     When processing merge for group membership, the group GID and name must be identical for both entries.  If only one or the other is a match, the behavior is undefined.

If we have a line like

     ethers: nisplus [NOTFOUND=return] db files

this is equivalent to

     ethers: nisplus [SUCCESS=return NOTFOUND=return UNAVAIL=continue
                      TRYAGAIN=continue]
             db      [SUCCESS=return NOTFOUND=continue UNAVAIL=continue
                      TRYAGAIN=continue]
             files

(except that it would have to be written on one line).  The default value for the actions are normally what you want, and only need to be changed in exceptional cases.

If the optional ! is placed before the STATUS this means the following action is used for all statuses but STATUS itself. I.e., ! is negation as in the C language (and others).

Before we explain the exception which makes this action item necessary one more remark: obviously it makes no sense to add another action item after the files service.  Since there is no other service following the action always is return.

Now, why is this [NOTFOUND=return] action useful?  To understand this we should know that the nisplus service is often complete; i.e., if an entry is not available in the NIS+ tables it is not available anywhere else.  This is what is expressed by this action item: it is useless to examine further services since they will not give us a result.

The situation would be different if the NIS+ service is not available because the machine is booting.  In this case the return value of the lookup function is not notfound but instead unavail.  And as you can see in the complete form above: in this situation the db and files services are used.  Neat, isn't it?  The system administrator need not pay special care for the time the system is not completely ready to work (while booting or shutdown or network problems).

Notes on the NSS Configuration File
-----------------------------------

Finally a few more hints.  The NSS implementation is not completely helpless if /etc/nsswitch.conf does not exist.  For all supported databases there is a default value so it should normally be possible to get the system running even if the file is corrupted or missing.

For the hosts and networks databases the default value is dns [!UNAVAIL=return] files.  I.e., the system is prepared for the DNS service not to be available but if it is available the answer it returns is definitive.

The passwd, group, and shadow databases are traditionally handled in a special way.  The appropriate files in the /etc directory are read but if an entry with a name starting with a + character is found NIS is used.  This kind of lookup remains possible if theglibc was configured with the --enable-obsolete-nsl option and the special lookup service compat is used.  If theglibc was configured with the --enable-obsolete-nsl option the default value for the three databases above is compat [NOTFOUND=return] files.  If the --enable-obsolete-nsl option was not used the default value for the services is files.

For all other databases the default value is files unless theglibc was configured with --enable-obsolete-rpc option, in which case it the default value is nis [NOTFOUND=return] files.

A second point is that the user should try to optimize the lookup process.  The different service have different response times. A simple file look up on a local file could be fast, but if the file is long and the needed entry is near the end of the file this may take quite some time.  In this case it might be better to use the db service which allows fast local access to large data sets.

Often the situation is that some global information like NIS must be used.  So it is unavoidable to use service entries like nis etc. But one should avoid slow services like this if possible.

NSS Module Internals
====================

Now it is time to describe what the modules look like.  The functions contained in a module are identified by their names.  I.e., there is no jump table or the like.  How this is done is of no interest here; those interested in this topic should read about Dynamic Linking.

The Naming Scheme of the NSS Modules
------------------------------------

The name of each function consists of various parts:

     _nss_SERVICE_FUNCTION

SERVICE of course corresponds to the name of the module this function is found in. (Now you might ask why this information is duplicated.  The answer is that we want to make it possible to link directly with these shared objects.)  The FUNCTION part is derived from the interface function in the C library itself.  If the user calls the function gethostbyname and the service used is files the function

            _nss_files_gethostbyname_r

in the module

            libnss_files.so.2

is used.  You see, what is explained above in not the whole truth.  In fact the NSS modules only contain reentrant versions of the lookup functions.  I.e., if the user would call the gethostbyname_r function this also would end in the above function.  For all user interface functions the C library maps this call to a call to the reentrant function.  For reentrant functions this is trivial since the interface is (nearly) the same.  For the non-reentrant version the library keeps internal buffers which are used to replace the user supplied buffer.

I.e., the reentrant functions can have counterparts.  No service module is forced to have functions for all databases and all kinds to access them.  If a function is not available it is simply treated as if the function would return unavail (see Actions in the NSS configuration).

The file name libnss_files.so.2 would be on a Solaris 2 system nss_files.so.2.  This is the difference mentioned above. Sun's NSS modules are usable as modules which get indirectly loaded only.

The NSS modules in theglibc are prepared to be used as normal libraries themselves.  This is not true at the moment, though. However,  the organization of the name space in the modules does not make it impossible like it is for Solaris.  Now you can see why the modules are still libraries. (There is a second explanation: we were too lazy to change the Makefiles to allow the generation of shared objects not starting with lib but don't tell this to anybody.)

The Interface of the Function in NSS Modules
--------------------------------------------

Now we know about the functions contained in the modules.  It is now time to describe the types.  When we mentioned the reentrant versions of the functions above, this means there are some additional arguments (compared with the standard, non-reentrant versions).  The prototypes for the non-reentrant and reentrant versions of our function above are:

     struct hostent *gethostbyname (const char *name)

     int gethostbyname_r (const char *name, struct hostent *result_buf,
                          char *buf, size_t buflen, struct hostent **result,
                          int *h_errnop)

The actual prototype of the function in the NSS modules in this case is

     enum nss_status _nss_files_gethostbyname_r (const char *name,
                                                 struct hostent *result_buf,
                                                 char *buf, size_t buflen,
                                                 int *errnop, int *h_errnop)

I.e., the interface function is in fact the reentrant function with the change of the return value, the omission of the RESULT parameter, and the addition of the ERRNOP parameter.  While the user-level function returns a pointer to the result the reentrant function return an enum nss_status value:

NSS_STATUS_TRYAGAIN
     numeric value -2

NSS_STATUS_UNAVAIL
     numeric value -1

NSS_STATUS_NOTFOUND
     numeric value 0

NSS_STATUS_SUCCESS
     numeric value 1

Now you see where the action items of the /etc/nsswitch.conf file are used.

If you study the source code you will find there is a fifth value: NSS_STATUS_RETURN.  This is an internal use only value, used by a few functions in places where none of the above value can be used.  If necessary the source code should be examined to learn about the details.

In case the interface function has to return an error it is important that the correct error code is stored in *ERRNOP.  Some return status values have only one associated error code, others have more.

NSS_STATUS_TRYAGAIN
EAGAIN   One of the functions used ran temporarily out of
resources or a service is currently not available.

ERANGE   The provided buffer is not large enough.
The function should be called again with a larger buffer.

NSS_STATUS_UNAVAIL
ENOENT   A necessary input file cannot be found.

NSS_STATUS_NOTFOUND
ENOENT   The requested entry is not available.

NSS_STATUS_NOTFOUND
SUCCESS   There are no entries.
Use this to avoid returning errors for inactive services which may
be enabled at a later time. This is not the same as the service
being temporarily unavailable.

These are proposed values.  There can be other error codes and the described error codes can have different meaning.  With one exception: when returning NSS_STATUS_TRYAGAIN the error code ERANGE must mean that the user provided buffer is too small.  Everything else is non-critical.

In statically linked programs, the main application and NSS modules do not share the same thread-local variable errno, which is the reason why there is an explicit ERRNOP function argument.

The above function has something special which is missing for almost all the other module functions.  There is an argument H_ERRNOP.  This points to a variable which will be filled with the error code in case the execution of the function fails for some reason.  (In statically linked programs, the thread-local variable h_errno is not shared with the main application.)

The getXXXbyYYY functions are the most important functions in the NSS modules.  But there are others which implement the other ways to access system databases (say for the user database, there are setpwent, getpwent, and endpwent).  These will be described in more detail later. Here we give a general way to determine the signature of the module function:

  * the return value is enum nss_status;

  * the name (see NSS Module Names);

  * the first arguments are identical to the arguments of the non-reentrant function;

  * the next four arguments are:

     STRUCT_TYPE *result_buf
          pointer to buffer where the result is stored.  STRUCT_TYPE is normally a struct which corresponds to the database.

     char *buffer
          pointer to a buffer where the function can store additional data for the result etc.

     size_t buflen
          length of the buffer pointed to by BUFFER.

     int *errnop
          the low-level error code to return to the application.  If the return value is not NSS_STATUS_SUCCESS, *ERRNOP needs to be set to a non-zero value.  An NSS module should never set *ERRNOP to zero.  The value ERANGE is special, as described above.

  * possibly a last argument H_ERRNOP, for the host name and network name lookup functions.  If the return value is not NSS_STATUS_SUCCESS, *H_ERRNOP needs to be set to a non-zero value.  A generic error code is NETDB_INTERNAL, which instructs the caller to examine *ERRNOP for further details.  (This includes the ERANGE special case.)

This table is correct for all functions but the set...ent and end...ent functions.

Extending NSS
=============

One of the advantages of NSS mentioned above is that it can be extended quite easily.  There are two ways in which the extension can happen: adding another database or adding another service.  The former is normally done only by the C library developers.  It is here only important to remember that adding another database is independent from adding another service because a service need not support all databases or lookup functions.

A designer/implementer of a new service is therefore free to choose the databases s/he is interested in and leave the rest for later (or completely aside).

Adding another Service to NSS
-----------------------------

The sources for a new service need not (and should not) be part of theglibc itself.  The developer retains complete control over the sources and its development.  The links between the C library and the new service module consists solely of the interface functions.

Each module is designed following a specific interface specification. For now the version is 2 (the interface in version 1 was not adequate) and this manifests in the version number of the shared library object of the NSS modules: they have the extension .2.  If the interface changes again in an incompatible way, this number will be increased. Modules using the old interface will still be usable.

Developers of a new service will have to make sure that their module is created using the correct interface number.  This means the file itself must have the correct name and on ELF systems the soname (Shared Object Name) must also have this number.  Building a module from a bunch of object files on an ELF system using GNU CC could be done like this:

     gcc -shared -o libnss_NAME.so.2 -Wl,-soname,libnss_NAME.so.2 OBJECTS

Options for Linking (GNU CC), to learn more about this command line.

To use the new module the library must be able to find it.  This can be achieved by using options for the dynamic linker so that it will search the directory where the binary is placed.  For an ELF system this could be done by adding the wanted directory to the value of LD_LIBRARY_PATH.

But this is not always possible since some programs (those which run under IDs which do not belong to the user) ignore this variable. Therefore the stable version of the module should be placed into a directory which is searched by the dynamic linker.  Normally this should be the directory $prefix/lib, where $prefix corresponds to the value given to configure using the --prefix option.  But be careful: this should only be done if it is clear the module does not cause any harm.  System administrators should be careful.

Internals of the NSS Module Functions
-------------------------------------

Until now we only provided the syntactic interface for the functions in the NSS module.  In fact there is not much more we can say since the implementation obviously is different for each function.  But a few general rules must be followed by all functions.

In fact there are four kinds of different functions which may appear in the interface.  All derive from the traditional ones for system databases. DB in the following table is normally an abbreviation for the database (e.g., it is pw for the user database).

enum nss_status _nss_DATABASE_setDBent (void)
     This function prepares the service for following operations.  For a simple file based lookup this means files could be opened, for other services this function simply is a noop.

     One special case for this function is that it takes an additional argument for some DATABASEs (i.e., the interface is int setDBent (int)).  Host Names, which describes the sethostent function.

     The return value should be NSS_STATUS_SUCCESS or according to the table above in case of an error (see NSS Modules Interface).

enum nss_status _nss_DATABASE_endDBent (void)
     This function simply closes all files which are still open or removes buffer caches.  If there are no files or buffers to remove this is again a simple noop.

     There normally is no return value other than NSS_STATUS_SUCCESS.

enum nss_status _nss_DATABASE_getDBent_r (STRUCTURE *result, char *buffer, size_t buflen, int *errnop)
     Since this function will be called several times in a row to retrieve one entry after the other it must keep some kind of state.  But this also means the functions are not really reentrant.  They are reentrant only in that simultaneous calls to this function will not try to write the retrieved data in the same place (as it would be the case for the non-reentrant functions); instead, it writes to the structure pointed to by the RESULT parameter.  But the calls share a common state and in the case of a file access this means they return neighboring entries in the file.

     The buffer of length BUFLEN pointed to by BUFFER can be used for storing some additional data for the result.  It is not guaranteed that the same buffer will be passed for the next call of this function.  Therefore one must not misuse this buffer to save some state information from one call to another.

     Before the function returns with a failure code, the implementation should store the value of the local ERRNO variable in the variable pointed to be ERRNOP.  This is important to guarantee the module working in statically linked programs.  The stored value must not be zero.

     As explained above this function could also have an additional last argument.  This depends on the database used; it happens only for host and networks.

     The function shall return NSS_STATUS_SUCCESS as long as there are more entries.  When the last entry was read it should return NSS_STATUS_NOTFOUND.  When the buffer given as an argument is too small for the data to be returned NSS_STATUS_TRYAGAIN should be returned.  When the service was not formerly initialized by a call to _nss_DATABASE_setDBent all return values allowed for this function can also be returned here.

enum nss_status _nss_DATABASE_getDBbyXX_r (PARAMS, STRUCTURE *result, char *buffer, size_t buflen, int *errnop)
     This function shall return the entry from the database which is addressed by the PARAMS.  The type and number of these arguments vary.  It must be individually determined by looking to the user-level interface functions.  All arguments given to the non-reentrant version are here described by PARAMS.

     The result must be stored in the structure pointed to by RESULT. If there are additional data to return (say strings, where the RESULT structure only contains pointers) the function must use the BUFFER of length BUFLEN.  There must not be any references to non-constant global data.

     The implementation of this function should honor the STAYOPEN flag set by the setDBent function whenever this makes sense.

     Before the function returns, the implementation should store the value of the local ERRNO variable in the variable pointed to by ERRNOP.  This is important to guarantee the module works in statically linked programs.

     Again, this function takes an additional last argument for the host and networks database.

     The return value should as always follow the rules given above (see NSS Modules Interface).