Important issues with passive target accumulate

* Simple accumulations should combine lock/unlock with operation when

  latency is high

* Separate accumulations into non-overlapping buffers (within the same

  local window) should be processed with as much parallelism as

  possible.

  For single threaded MPI runtime systems, it is reasonable to

  restrict such optimizations to accumulations issued from separate

  processes since communications about multiple accumulations from a

  single process are likely to be serialized by the networking device

  anyway.

  The trick to parallelizing multiple requests is identifying that

  their target buffers do not overlap.  Detecting this seems extremely

  difficult, however, an assertion at the time the lock is acquired

  could be used to inform the runtime system that no overlapping

  accumulations will be issued by other processes during this epoch.

  * Two-sided

    Unless multiple threads (and processors) are present on the target

    system to process accumulation operations, serialization is

    eminent, and the best optmization is to not have a ton of

    multi-threading optimizations in the way slowing things down.  This

    suggests that two implementations may be necessary: one using

    threads and one that doesn't.

    If multiple threads and processors are available on the target

    system, then one might be able to use separate threads to process

    accumulate requests from different processes.  Multiple thread

    might also be useful to help overlap communication and computation

    for a single request if that request is large enough.

  * Remote memory

    The same issues that apply for the two-sided case, apply here.

    The only real difference is that we might use get/put to fill

    buffers instead of send/recv.

  * Shared memory

  * Multi-method

* Large accumulations into the same buffer should be pipelined to

  achieve maximum parallelism.

  If it is possible to detect that the same target buffer was being

  used by a set of processes, and that no other overlapping target

  buffers were simultaneously being operated upon, then mutexes can

  be associated with areas of progress within that buffer rather

  particular regions of the target's local window.  Defining the

  areas of progress may happen naturally as a result of limited

  buffer space necessitating the use of segments.  So, atomicity

  would need to be guaranteed for the operations associated with a

  particular segment associated with a buffer rather than a

  particular region with the local window.  In practice, detecting

  that the conditions have been met to ensure atomicity may prove

  too difficult to obtain reasonable performance.

  * Two-sided

    If multiple threads are available to process multiple data

    streams, then it should be possible to pipeline the processing.

    It is critical, however, that the data be organized and sent in

    such a way so as to maximize

  * Remote memory

  * Shared memory

    For shared memory, this can be accomplished by dividing local

    windows into regions and providing a separate mutex for each

    region.

  * Multi-method

* Datatype caching

  [traff00:mpi-impl] and [booth00:mpi-impl] discuss the need for

  datatype caching by the target process when the target process

  must be involved in the RMA operations (i.e., when the data cannot

  be directly read and interpreted by the origin process)

  Datatype caching can be either proactive or reactive.

  In the proactive case, the origin process would track whether the

  target process has a copy of the datatype and send the datatype to

  target process when necessary.  This means that each datatype must

  contain tracking information.  Unfortunately, because of dynamic

  processes in MPI-2, something more complex than a simple bit

  vector must be used to track processes already caching a datatype.

  What is the correct, high-performance structure?

  Alternatively, a reactive approach could be used.  In the reactive

  approach, the origin process would assume that the target process

  already knows about the datatype.  If that assumption is false, the

  target process will request the datatype from the origin process.

  This simplifies the tracking on the origin, but does not completely

  eliminate it.  It is necessary for the origin to increase the

  reference count associated with the datatype until the next

  synchronization point to ensure that the datatype is not deleted

  before the target process has had sufficient opportunity to request

  a copy of the datatype.

  * Two-sided

  * Remote memory

  * Shared memory

    For shared memory, data type caching is unnecessary if the origin

    process performs the work.  If the target is involved in the work,

    necessary datatype information can be placed in shared memory.

  * Multi-method