* Definitions

  - MPI buffer - count, datatype, memory pointer

* Communication subsystem capabilities (requirements)

  - general MPI messaging (MPI_*send() and MPI_*recv())

    - sending messages

      - if send-side MPI buffer is sufficiently contiguous, send data
        directly from MPI buffer

      - if RMA capabilities exist and MPI receive buffer is
        sufficiently contiguous, write message data directly into MPI
        receive buffer

    - receiving messages

      - match incoming messages with posted receives

        - handle posting and matching of wildcard (source) receives

      - special handling for already posted receives

        - if MPI buffer is sufficiently contiguous, receive directly
          into MPI buffer

        - if user buffer is non-contiguous, unpack data as portions of
          message data are received

  - persistent MPI messaging (MPI_*_init())

    - for some network interfaces, we should be able to perform
      one-time initialization to eliminate unnecessary data copies
      (manipulating the MPI buffer directly)

  - collective operations

    - send portions of a MPI buffer

    - receive portions of a MPI buffer

    - forward portions of an incoming message

      Use pipelining instead of store and forward to increase network
      utilization (and thus performance).

      Potentially multicast the same portion to multiple remote
      processes.  Nick's prototype shows this is a big win for TCP and
      vMPI.  I suspect it is a big win in general.

    - share buffers between methods to avoid copying data during forward
      operations

    - perform MPI computations (as defined by MPI_Reduce()) while
      receiving/unpacking data

      Computations may need to be performed at intermediate processes
      (processes not receiving any of the results) which implies that
      computations may need to be performed without the presence of a
      user provided MPI buffer (or datatype).

    - handle multiple simultaneous collective operations on the same
      communicator (multi-threaded MPI)

      We should be able to use the tag field and a rolling counter to
      separate messages from independent collective operations.  This
      would allow us to use the same matching mechanisms that we use
      for general MPI messaging.

  - remote memory operations

    - aggregate operations (from the same exposure epoch?) into a
      single communication

    - perform MPI computations on remote memory

    - match communicated operations with exposure epochs (either
      explicit or implied)

      Is context sufficient for this?  Do we need a tag to separate
      independent access/exposure epochs?

  - unreliable communication and QoS 

    Theoretically, a MPI communicator could be tagged to allow
    unrealiable delivery, QoS, etc.  We haven't thought much what
    impact this has on our design, but we probably don't want to
    prevent these capabilites.

* Communication subsystem components

  - virtual connnections

    allows late binding to a communication device (method)

    provides a function tables for all connection/communication
    related interfaces

  - progress engine

  - matching incoming messages to posted requests

  - message level flow control

  - shared network buffer management

  - network communication

    - network flow control

* Message level flow control

  For simple messaging operations, message envelope meta-data must be
  sent to the remote process immediately.  Failure to do so may cause
  the remote process to block indefinitely awaiting a particular
  message.  However, the method also needs to balance messaging
  performance (sending the entire message immediately) with the memory
  used by the remote process to buffer messages not already posted by
  the remote process.

  Messages are typically converted (by the method?) to one of three
  types to obtain this balance: short, eager, and rendezvous.
  Conversion to a particular message type may depend on the memory
  availability of the remote process.

  NOTE: Some communication interfaces such as vendor MPI will do this
  automatically, which means we shouldn't force message level flow
  control upon the method.

* Method

  - definition of a method

    A method presents an interface which allows upper layers to convey
    actions it wishes to the method perform in the context of a
    virtual connection.  These actions consist of sending and
    receiving messages, performing remote memory operations, and
    providing data and buffers to other methods.

  - flow control at the message level

  - flow control at the network buffer (packet) level

    Some methods need to worry about network buffer availability at
    remote process.

  - reliability

    Under a default environemnt, MPI messages are inherently reliable
    which means that some methods may need concern themselves with
    acknowledgments and retransmission if the underlying network does
    not guarantee reliability.


  - matching incoming messages to requests