# Plugin Developer's Guide #

```eval_rst

.. toctree::

```

## Overview ##

Beginning with OPAE C library version 1.2.0, OPAE implements a plugin-centric

model. This guide serves as a reference to define the makeup of an OPAE C API

plugin and to describe a sequence of steps that one may follow when constructing

an OPAE C API plugin.

## Plugin Required Functions ##

An OPAE C API plugin is a runtime-loadable shared object library, also known as

a module. On Linux systems, the *dl* family of APIs from libdl are used to

interact with shared objects. Refer to "man dlopen" and "man dlsym" for examples

of using the libdl API.

An OPAE C API plugin implements one required function. This function is required

to have C linkage, so that its name is not mangled.

```c

    int opae_plugin_configure(opae_api_adapter_table *table, const char *config);

```

During initialization, the OPAE plugin manager component loads each plugin,

searching for its `opae_plugin_configure` function. If none is found, then

the plugin manager rejects that plugin. When it is found, `opae_plugin_configure`

is called passing a pointer to a freshly-created `opae_api_adapter_table` and

a buffer consisting of configuration data for the plugin.

The job of the `opae_plugin_configure` function is to populate the given adapter

table with each of the plugin's API entry points and to consume and comprehend

the given configuration data in preparation for initialization.

## OPAE API Adapter Table ##

The adapter table is a data structure that contains function pointer entry points

for each of the OPAE APIs implemented by a plugin. In this way, it adapts the

plugin-specific behavior to the more general case of a flat C API. Note that

OPAE applications are only required to link with opae-c. In other words, the

name of the plugin library should not appear on the linker command line. In this

way, plugins are truly decoupled from the OPAE C API, and they are required to

adapt to the strict API specification by populating the adapter table only. No

other linkage is required nor recommended.

`adapter.h` contains the definition of the `opae_api_adapter_table`. An abbreviated

version is depicted below, along with supporting type `opae_plugin`:

```c

    typedef struct _opae_plugin {

        char *path;

        void *dl_handle;

    } opae_plugin;

    typedef struct _opae_api_adapter_table {

        struct _opae_api_adapater_table *next;

        opae_plugin plugin;

        fpga_result (*fpgaOpen)(fpga_token token, fpga_handle *handle,

                                int flags);

        fpga_result (*fpgaClose)(fpga_handle handle);

        ...

        fpga_result (*fpgaEnumerate)(const fpga_properties *filters,

                                     uint32_t num_filters, fpga_token *tokens,

                                     uint32_t max_tokens,

                                     uint32_t *num_matches);

        ...

        // configuration functions

        int (*initialize)(void);

        int (*finalize)(void);

        // first-level query

        bool (*supports_device)(const char *device_type);

        bool (*supports_host)(const char *hostname);

    } opae_api_adapter_table;

```

Some points worth noting are that the adapter tables are organized in memory by

adding them to a linked list data structure. This is the use of the `next`

structure member. (The list management is handled by the plugin manager.)

The `plugin` structure member contains the handle to the shared object instance,

as created by `dlopen`. This handle is used in the plugin's `opae_plugin_configure`

to load plugin entry points. A plugin need only implement the portion of the

OPAE C API that a target application needs. Any API entry points that are not

supported should be left as NULL pointers (the default) in the adapter table.

When an OPAE API that has no associated entry point in the adapter table is

called, the result for objects associated with that plugin will be

`FPGA_NOT_SUPPORTED`.

The following code illustrates a portion of the `opae_plugin_configure` for

a theoretical OPAE C API plugin libfoo.so:

```c

    /* foo_plugin.c */

    int opae_plugin_configure(opae_api_adapter_table *table, const char *config)

    {

        adapter->fpgaOpen = dlsym(adapter->plugin.dl_handle, "foo_fpgaOpen");

        adapter->fpgaClose =

                dlsym(adapter->plugin.dl_handle, "foo_fpgaClose");

        ...

        adapter->fpgaEnumerate =

                dlsym(adapter->plugin.dl_handle, "foo_fpgaEnumerate");

        ...

        return 0;

    }

```

Notice that the implementations of the API entry points for plugin libfoo.so

are prefixed with `foo_`. This is the recommended practice to avoid name

collisions and to enhance the debugability of the application. Upon successful

configuration, `opae_plugin_configure` returns 0 to indicate success. A

non-zero return value indicates failure and causes the plugin manager to

reject the plugin from futher consideration.

## Plugin Optional Functions ##

Once the plugin manager loads and configures each plugin, it uses the adapter

table to call back into the plugin so that it can be made ready for runtime.

This is the job of the `opae_plugin_initialize` entry point, whose signature

is defined as:

```c

    int opae_plugin_initialize(void);

```

The function takes no parameters, as the configuration data was already given

to the plugin by `opae_plugin_configure`. `opae_plugin_initialize` returns 0

if no errors were encountered during initialization. A non-zero return code

indicates that plugin initialization failed. A plugin makes its

`opae_plugin_initialize` available to the plugin manager by populating the

adapter table's `initialize` entry point as shown:

```c

    /* foo_plugin.c */

    int foo_plugin_initialize(void)

    {

        ...

        return 0; /* success */

    }

    int opae_plugin_configure(opae_api_adapter_table *table, const char *config)

    {

        ... 

        adapter->initialize =

                dlsym(adapter->plugin.dl_handle, "foo_plugin_initialize");

        ...

        return 0;

    }

```

If a plugin does not implement an `opae_plugin_initialize` entry point, then

the `initialize` member of the adapter table should be left uninitialized.

During plugin initialization, if a plugin has no `opae_plugin_initialize`

entry in its adapter table, the plugin initialization step will be skipped,

and the plugin will be considered to have initialized successfully.

Once plugin initialization is complete for all loaded plugins, the system

is considered to be running and fully functional.

During teardown, the plugin manager uses the adapter table to call into each

plugin's `opae_plugin_finalize` entry point, whose signature is defined as:

```c

    int opae_plugin_finalize(void);

```

`opae_plugin_finalize` returns 0 if no errors were encountered during teardown.

A non-zero return code indicates that plugin teardown failed. A plugin makes

its `opae_plugin_finalize` available to the plugin manager by populating the

adapter table's `finalize` entry point as shown:

```c

    /* foo_plugin.c */

    int foo_plugin_finalize(void)

    {

        ...

        return 0; /* success */

    }

    int opae_plugin_configure(opae_api_adapter_table *table, const char *config)

    {

        ... 

        adapter->finalize =

                dlsym(adapter->plugin.dl_handle, "foo_plugin_finalize");

        ...

        return 0;

    }

```

If a plugin does not implement an `opae_plugin_finalize` entry point, then

the `finalize` member of the adapter table should be left uninitialized.

During plugin cleanup, if a plugin has no `opae_plugin_finalize` entry

point in its adapter table, the plugin finalize step will be skipped, and

the plugin will be considered to have finalized successfully.

In addition to `initialize` and `finalize`, an OPAE C API plugin has two

further optional entry points that relate to device enumeration. During

enumeration, when a plugin is being considered for a type of device, the

plugin may provide input on that decision by exporting an

`opae_plugin_supports_device` entry point in the adapter table:

```c

    bool opae_plugin_supports_device(const char *device_type);

```

`opae_plugin_supports_device` returns true if the given device type is

supported and false if it is not. A false return value from

`opae_plugin_supports_device` causes device enumeration to skip the

plugin.

Populating the `opae_plugin_supports_device` is done as:

```c

    /* foo_plugin.c */

    bool foo_plugin_supports_device(const char *device_type)

    {

        if (/* device_type is supported */)

            return true;

        ...

        return false;

    }

    int opae_plugin_configure(opae_api_adapter_table *table, const char *config)

    {

        ... 

        adapter->supports_device =

                dlsym(adapter->plugin.dl_handle, "foo_plugin_supports_device");

        ...

        return 0;

    }

```

```eval_rst

.. note::

    The `opae_plugin_supports_device` mechanism serves as a placeholder only.

    It is not implemented in the current version of the OPAE C API.

```

Similarly to determining whether a plugin supports a type of device, a plugin

may also answer questions about network host support by populating an

`opae_plugin_supports_host` entry point in the adapter table:

```c

    bool opae_plugin_supports_host(const char *hostname);

```

`opae_plugin_supports_host` returns true if the given hostname is supported

and false if it is not. A false return value from `opae_plugin_supports_host`

causes device enumeration to skip the plugin.

Populating the `opae_plugin_supports_host` is done as:

```c

    /* foo_plugin.c */

    bool foo_plugin_supports_host(const char *hostname)

    {

        if (/* hostname is supported */)

            return true;

        ...

        return false;

    }

    int opae_plugin_configure(opae_api_adapter_table *table, const char *config)

    {

        ... 

        adapter->supports_host =

                dlsym(adapter->plugin.dl_handle, "foo_plugin_supports_host");

        ...

        return 0;

    }

```

```eval_rst

.. note::

    The `opae_plugin_supports_host` mechanism serves as a placeholder only.

    It is not implemented in the current version of the OPAE C API.

```

## Plugin Construction ##

The steps required to implement an OPAE C API plugin, libfoo.so, are:

* Create foo\_plugin.c: implements `opae_plugin_configure`,

`opae_plugin_initialize`, `opae_plugin_finalize`, `opae_plugin_supports_device`,

and `opae_plugin_supports_host` as described in the previous sections.

* Create foo\_plugin.h: implements function prototypes for each of the

plugin-specific OPAE C APIs.

```c

    /* foo_plugin.h */

    fpga_result foo_fpgaOpen(fpga_token token, fpga_handle *handle,

                             int flags);

    fpga_result foo_fpgaClose(fpga_handle handle);

    ...

    fpga_result foo_fpgaEnumerate(const fpga_properties *filters,

                                  uint32_t num_filters, fpga_token *tokens,

                                  uint32_t max_tokens,

                                  uint32_t *num_matches);

    ...

```

* Create foo\_types.h: implements plugin-specific types for opaque data

structures.

```c

    /* foo_types.h */

    struct _foo_token {

        ...

    };

    struct _foo_handle {

        ...

    };

    struct _foo_event_handle {

        ...

    };

    struct _foo_object {

        ...

    };

```

* Create foo\_enum.c: implements `foo_fpgaEnumerate`,

`foo_fpgaCloneToken`, and `foo_fpgaDestroyToken`.

* Create foo\_open.c: implements `foo_fpgaOpen`.

* Create foo\_close.c: implements `foo_fpgaClose`.

* Create foo\_props.c: implements `foo_fpgaGetProperties`,

`foo_fpgaGetPropertiesFromHandle`, `foo_fpgaUpdateProperties`

* Create foo\_mmio.c: implements `foo_fpgaMapMMIO`, `foo_fpgaUnmapMMIO`

`foo_fpgaWriteMMIO64`, `foo_fpgaReadMMIO64`, `foo_fpgaWriteMMIO32`,

`foo_fpgaReadMMIO32`.

* Create foo\_buff.c: implements `foo_fpgaPrepareBuffer`,

`foo_fpgaReleaseBuffer`, `foo_fpgaGetIOAddress`.

* Create foo\_error.c: implements `foo_fpgaReadError`, `foo_fpgaClearError`,

`foo_fpgaClearAllErrors`, `foo_fpgaGetErrorInfo`.

* Create foo\_event.c: implements `foo_fpgaCreateEventHandle`,

`foo_fpgaDestroyEventHandle`, `foo_fpgaGetOSObjectFromEventHandle`,

`foo_fpgaRegisterEvent`, `foo_fpgaUnregisterEvent`.

* Create foo\_reconf.c: implements `foo_fpgaReconfigureSlot`.

* Create foo\_obj.c: implements `foo_fpgaTokenGetObject`,

`foo_fpgaHandleGetObject`, `foo_fpgaObjectGetObject`,

`foo_fpgaDestroyObject`, `foo_fpgaObjectGetSize`, `foo_fpgaObjectRead`,

`foo_fpgaObjectRead64`, `foo_fpgaObjectWrite64`.

* Create foo\_clk.c: implements `foo_fpgaSetUserClock`,

`foo_fpgaGetUserClock`.