MPI

- spawn/attach

- communicators

- pt2pt requests

- collective

- RMA

- error (handling, FT, reporting)

MPID

- messages

  - tradition MPI message

  - one-sided operations

  - control?

- streams

- process management (via BNR)

Communication Methods

- TCP

- VIA

- Shared Memory

- Loopback

- IMPI

===============================================================================

MPI layer

--------------------

Operations

- point-to-point

  - requests

  - datatypes

  - communicators

  - status

  - errors

- collective

  - datatypes

  - communicators

  - errors

- process management

  - communicators

  - info

  - errors

- RMA

  - datatypes

  - windows

  - communicators

  - groups

  - epoch

  - errors

- I/O

  - depends on

    - files

    - datatypes

    - requests

    - info

    - status

    - errors

  * implemented via ROMIO

    - dependent only on MPI functions

    - future enhancements may use low-level interfaces

- topology

  - communicators

  - errors

  * can be implemented entirely at the MPI layer

- generalized requests

  - errors

  * can be implemented entirely at the MPI layer

--------------------

Structures

- requests

- datatypes

- communicators

  - groups

- groups

  * are groups modified or augmented by low layers?

- windows

- files

  * defined and implemented via ROMIO

    - future enhancements may require access lower layers

- status

- errors

- attributes

  * can be defined and operations implemented entirely at the MPI layer

  - communicator

  - datatypes

  - windows

- info

  * can be defined and operations implemented entirely at the MPI layer

===============================================================================

MPID

--------------------

Operations

- point-to-point

- collective operations

- process management

- RMA

- generalized requests

Structures

--------------------

Concepts

- MPI buffer movement (moving buffers defined by an address, count and

  datatype)

- internal buffer management

- Connection management

  - virtual connection structures

  - low-level connnection management (sockets, etc.) should be handled

    entirely by the device and probably driven by a state machine

===============================================================================

Multi-method design

--------------------

Device-level objects

- group

  - data structures

  - methods

    - set_connection(group, rank, vc_ptr) - associate pointer to virtual

      connection structure with a (group,rank)

    - get_connection(group, rank) - returns pointer to virtual connection

      structure associated with (group,rank)

- communicators

  - data structures

    - group

  - methods

    - set_connection(dcomm, rank, vc_ptr) - associate pointer to virtual

      connection structure with a (dcomm,rank)

    - get_connection(dcomm, rank) - returns pointer to virtual connection

      structure associated with (dcomm,rank)

- virtual connections

  - alloc() - returns a pointer to a virtual connection structure

  - add_ref(vc) - increments the reference count (atomically)

  - release() - decrements the reference count; if the reference count reaches

    zero, the structure is freed

  NOTE: It may be useful to be able to locate a virtual connection based on a

  process group ID and rank, in part so we can detect when multiple virtual

  connections might be formed between a pair of processes.

can connect/accept be called multiple times between a set of processes?

--------------------

Method-level functions

Method descriptors are strings that are used to describe the capabilities of

the methods.  These descriptors can then be used to determine if two processes

miight be capable of "talking" using the method in question.  We say "might be

capable" because in the case of a method like VIA, it may be impossible provide

enough information in the descriptor to determine in two processes can talk.

It may be necessary to simply attempt to form the connection.  This implies

that binding to a particular protocol may need to be deferred until we are

ready to form a real connection.  However, some methods, such as shared memory,

can provide enough information and thus can be bound immediately.

- query/get_descriptor()

- match_descriptors()

===============================================================================

MPI_Init()

- create basic datatypes

  MPIR_Datatype_init()

  {

      foreach dt (all basic datatypes)

      {

          MPID_Datatype_init(dt, ...)

      }

  }

- initialize device

  - BNR initialization

    BNR_Init()

    BNR_Get_group(&my_bnr_group)

    BNR_Get_size(my_bnr_group, &size)

    BNR_Get_rank(my_bnr_group, &rank)

    BNR_Get_parent(&parent_bnr_group)

    BNR_Merge(my_bnr_group, parent_bnr_group, &inter_bnr_group)

  - loop through methods

    - initialize method

    - query for descriptor of method's capabilities

    Q: what about dymanicly loaded methods?  do they have to be initialized now

    or can they be added later?

  - publish capabilities of all known methods

  - initialize AQ, buffer management, etc.

- establish MPI_COMM_WORLD

  - create MPI_GROUP_WORLD (internal) from BNR my_group, etc.

    stores BNR info in group structure

    allocates virtual connection structures

    initializes virtual connections to stubs

  - create MPI_COMM_WORLD from MPI_GROUP_WORLD

- establish inter-communicator with parent (if parent exists)

  - create inter_group from inter_bnr_group

  - create inter-communicator from inter_group

- create inter_group from inter_bnr_gorup

- create inter-communicator from inter_group

MPI_Spawn()

{

    BNR_Open_group(my_bnr_group, &new_bnr_group)

    BNR_Spawn(remote_bnr_group, N, ..., func)

    BNR_Close(remote_bnr_group)

    BNR_Merge(my_bnr_group, remote_bnr_group, &inter_bnr_group);

}

need a BNR_Group_ID which is globally unique in order to implement MPI_Connect/Attach

------------------------------------------------------------------------

Structures that cross layers

- many of the information structures that are passed through the layers contain

  data sections from multiple layers

- ne option is to include device (and method) include files in the MPICH layer

  include file.  Rob and Brian feel this would be bad.

- David suggests that the structure definitions be supplied by the device

  header files and that method specific information be included in those

  definition using unions.  Rob and Brian feel this is ugly (from a software

  engineering standpoint).

- Rob and Brian suggest having each layer define their own portion of the

  structure.  The definitions of the higher layers are known to the lower

  layers, but not vice versa.  To increase cache locality and reduce memory

  allocation, the device (and methods) report the amount of space they need in

  these structures so that the highest layer can allocate sufficient space.

  pointer arithmatic, etc.

Virtual connection

- used by MM implementation to allow late binding to a method

  this implies that the VC contains a pointer to either the function pointer

  table for the method to which it is bound or the function pointer to table to

  a set of functions that perform the binding to such a method

- one-to-one correspondence between a virtual connection and a real connection

- Contains

  - state of binding

  - method specific information

  - function pointer table?

----------

Communicators

- Contains

  - communication group

  - local group (inter-communicator only)

  - send and receive context IDs - same for intra-communicator

  - attributes

  - reference count

  - error handlers

  - device specific information (needed for MPICH-G2)

----------

Groups

- Contains

  - virtual connection table

  - my rank

  - reference count

  - device specific information (???)

----------

Requests

- probably allocated and partially initialized above ADI

- initialization complete by device/method

- Request contains

  - Immutable after initialization

    - type of request

      - persistent request flag

      - send, bsend, rsend, ssend, recv, generalized

    - buffer

    - count

    - datatype

    - rank (src or dest depending on type of request)

    - tag

    - comm

----------

Connection resolution

- needs to talk to with BNR

----------

Communication agent