Shared Memory

Memory that is shared among all ranks on a single node. Must be enabled with the compilation flags -DUSE_MPI and -DUSE_SHARED_MEM (see Compilation). reasonably recent linux kernel + headers required (memfd).

There are three backends for shared memory (swapped through Compilation DEFINES):

• The linux kernel via the syscall memfd_create, i.e., anonymous files (default)

• The POSIX interface via shm_open and shm_unlink (SHARED_MEM_POSIX_IMPL)

• An MPI implementation via MPI_Win_create and friends (SHARED_MEM_MPI_IMPL)

void *shared_malloc(size_t sz)

allocate rank shared memory on a single node. signature equivalent to stdlib’s malloc

void *shared_calloc(size_t cnt, size_t sz)

same as shared_malloc() but zero out the memory. signature equivalent to stdlib’s calloc

void shared_free(void *ptr)

free memory allocated with shared_malloc() or shared_calloc(). signature equivalent to stdlib’s free

int shared_exclusive_enter(void *ptr)

enter a region that is exlusively executed by one rank per node. Usage:

if (shared_exclusive_enter(ptr) == 1) {
  // do something exclusively
}
shared_exclusive_wait(ptr);

Note

When using the MPI implementation of shared memory (SHARED_MEM_MPI_IMPL), entering multiple exclusive regions at the same time does not work due to synchronization issues. Therefore, shared_exclusive_wait() includes a call to shared_malloc_barrier().

void shared_exclusive_wait(void *ptr)

wait for an operation entered via shared_exclusive_enter() to finish on all ranks.

int shared_malloc_rank(void)

return the rank on one node. useful for worksharing when operating on shared memory.

int shared_malloc_size(void)

return the number of ranks on one node. useful for worksharing when operating on shared memory.

void shared_malloc_barrier(void)

barrier that must be reached by each rank on one node before execution continues