QUESO::MpiComm Class Reference

The QUESO MPI Communicator Class. More...

#include <MpiComm.h>

Collaboration diagram for QUESO::MpiComm:

Public Member Functions
Constructor/Destructor methods
	MpiComm ()
	Default Constructor. More...
	MpiComm (const BaseEnvironment &env, RawType_MPI_Comm inputRawComm)
	QUESO MpiComm MPI Constructor. More...
	MpiComm (const MpiComm &src)
	Copy Constructor. More...
	~MpiComm ()
	Destructor. More...
Set methods
MpiComm &	operator= (const MpiComm &rhs)
	Assignment operator. More...
Attribute Accessor Methods
RawType_MPI_Comm	Comm () const
	Extract MPI Communicator from a MpiComm object. More...
int	MyPID () const
	Return my process ID. More...
int	NumProc () const
	Returns total number of processes. More...
Methods Overridden from Comm
void	Allreduce (void *sendbuf, void *recvbuf, int count, RawType_MPI_Datatype datatype, RawType_MPI_Op op, const char *whereMsg, const char *whatMsg) const
	Combines values from all processes and distributes the result back to all processes. More...
void	Barrier () const
	Pause every process in *this communicator until all the processes reach this point. More...
void	Bcast (void *buffer, int count, RawType_MPI_Datatype datatype, int root, const char *whereMsg, const char *whatMsg) const
	Broadcast values from the root process to the slave processes. More...
void	Gather (void *sendbuf, int sendcnt, RawType_MPI_Datatype sendtype, void *recvbuf, int recvcount, RawType_MPI_Datatype recvtype, int root, const char *whereMsg, const char *whatMsg) const
	Gather values from each process to collect on all processes. More...
void	Gatherv (void *sendbuf, int sendcnt, RawType_MPI_Datatype sendtype, void *recvbuf, int *recvcnts, int *displs, RawType_MPI_Datatype recvtype, int root, const char *whereMsg, const char *whatMsg) const
	Gathers into specified locations from all processes in a group. More...
void	Recv (void *buf, int count, RawType_MPI_Datatype datatype, int source, int tag, RawType_MPI_Status *status, const char *whereMsg, const char *whatMsg) const
	Blocking receive of data from this process to another process. More...
void	Send (void *buf, int count, RawType_MPI_Datatype datatype, int dest, int tag, const char *whereMsg, const char *whatMsg) const
	Possibly blocking send of data from this process to another process. More...
Miscellaneous Methods
void	syncPrintDebugMsg (const char *msg, unsigned int msgVerbosity, unsigned int numUSecs) const
	Synchronizes all the processes and print debug message. More...

Private Member Functions
void	copy (const MpiComm &src)
	Copies from an existing MpiComm instance. More...

Private Attributes
const BaseEnvironment &	m_env
RawType_MPI_Comm	m_rawComm
	Embedded wrapped opaque MPI_Comm object. More...
int	m_worldRank
	World rank. More...
int	m_myPid
	Process ID of this process. More...
int	m_numProc

Detailed Description

The QUESO MPI Communicator Class.

This class uses MPI (the Message Passing Interface) for distributed-memory communication between one or more parallel processes. It is meant to insulate the user from the specifics of communication that are not required for normal manipulation of linear algebra objects.

Definition at line 75 of file MpiComm.h.

Constructor & Destructor Documentation

QUESO::MpiComm::MpiComm

(

)

Default Constructor.

It should not be used by user.

Definition at line 31 of file MpiComm.C.

References UQ_FATAL_TEST_MACRO, and QUESO::UQ_UNAVAILABLE_RANK.

  :
  m_env( *(new EmptyEnvironment()) )
{
  UQ_FATAL_TEST_MACRO(true,
                      UQ_UNAVAILABLE_RANK,
                      "MpiComm::constructor()",
                      "should not be called");
}

QUESO::MpiComm::MpiComm	(	const BaseEnvironment &	env,
		RawType_MPI_Comm	inputRawComm
	)

QUESO MpiComm MPI Constructor.

This constructs an MpiComm that uses the given "raw" MPI communicator underneath. The MPI_Comm must be valid for the lifetime of this MpiComm.

Definition at line 41 of file MpiComm.C.

References m_myPid, m_numProc, m_worldRank, UQ_FATAL_TEST_MACRO, and QUESO::UQ_UNAVAILABLE_RANK.

  :
  m_env          (env),
#ifdef QUESO_HAS_TRILINOS
  m_epetraMpiComm( new Epetra_MpiComm(inputRawComm) ),
#endif
  m_rawComm      (inputRawComm),
  m_worldRank    (-1),
  m_myPid        (-1),
  m_numProc      (-1)
{
  int mpiRC = MPI_Comm_rank(inputRawComm,&m_worldRank);
  UQ_FATAL_TEST_MACRO(mpiRC != MPI_SUCCESS,
                      UQ_UNAVAILABLE_RANK,
                      "MpiComm::constructor()",
                      "failed MPI_Comm_rank() on full rank");

  mpiRC = MPI_Comm_rank(inputRawComm,&m_myPid);
  UQ_FATAL_TEST_MACRO(mpiRC != MPI_SUCCESS,
                      m_worldRank,
                      "MpiComm::constructor()",
                      "failed MPI_Comm_rank() on inputRawComm");

  mpiRC = MPI_Comm_size(inputRawComm,&m_numProc);
  UQ_FATAL_TEST_MACRO(mpiRC != MPI_SUCCESS,
                      m_worldRank,
                      "MpiComm::constructor()",
                      "failed MPI_Comm_size() on inputRawComm");
}

QUESO::MpiComm::MpiComm

(

const MpiComm &

src

)

Copy Constructor.

Makes an exact copy of an existing MpiComm instance.

Definition at line 72 of file MpiComm.C.

References copy().

  :
  m_env          (src.m_env)
#ifdef QUESO_HAS_TRILINOS
  ,
  m_epetraMpiComm(NULL)
#endif
{
  this->copy(src);
}

QUESO::MpiComm::~MpiComm

(

)

Destructor.

Definition at line 84 of file MpiComm.C.

{
#ifdef QUESO_HAS_TRILINOS
  delete m_epetraMpiComm;
  m_epetraMpiComm = NULL;
#endif
}

Member Function Documentation

void QUESO::MpiComm::Allreduce	(	void *	sendbuf,
		void *	recvbuf,
		int	count,
		RawType_MPI_Datatype	datatype,
		RawType_MPI_Op	op,
		const char *	whereMsg,
		const char *	whatMsg
	)		const

Combines values from all processes and distributes the result back to all processes.

Parameters

sendbuf	starting address of send buffer
count	number of elements in send buffer
datatype	data type of elements of send buffer
op	operation
recvbuf	(output) starting address of receive buffer

Definition at line 131 of file MpiComm.C.

References m_rawComm, m_worldRank, and UQ_FATAL_TEST_MACRO.

Referenced by QUESO::ComputeCovCorrBetweenScalarSequences(), QUESO::ComputeCovCorrMatricesBetweenVectorSequences(), QUESO::MiscCheckForSameValueInAllNodes(), QUESO::GslVector::mpiAllReduce(), QUESO::GslVector::mpiBcast(), QUESO::MHRawChainInfoStruct::mpiSum(), and QUESO::GslMatrix::mpiSum().

{
  int mpiRC = MPI_Allreduce(sendbuf, recvbuf, count, datatype, op, m_rawComm);
  UQ_FATAL_TEST_MACRO(mpiRC != MPI_SUCCESS,
                      m_worldRank,
                      whereMsg,
                      whatMsg);

  return;
}

void QUESO::MpiComm::Barrier

(

)

const

Pause every process in *this communicator until all the processes reach this point.

Blocks the caller until all processes in the communicator have called it; that is, the call returns at any process only after all members of the communicator have entered the call.

Definition at line 143 of file MpiComm.C.

References m_rawComm, m_worldRank, and UQ_FATAL_TEST_MACRO.

Referenced by QUESO::ComputeCovCorrMatricesBetweenVectorSequences(), QUESO::FullEnvironment::FullEnvironment(), QUESO::MiscCheckForSameValueInAllNodes(), QUESO::GslVector::mpiBcast(), and syncPrintDebugMsg().

{
#ifdef QUESO_HAS_TRILINOS
  return m_epetraMpiComm->Barrier();
#endif
  int mpiRC = MPI_Barrier(m_rawComm);
  UQ_FATAL_TEST_MACRO(mpiRC != MPI_SUCCESS,
                      m_worldRank,
                      "MPIComm::Barrier()", // whereMsg,
                      "mpiRC indicates failure");  // whatMsg);
  return;
}

void QUESO::MpiComm::Bcast	(	void *	buffer,
		int	count,
		RawType_MPI_Datatype	datatype,
		int	root,
		const char *	whereMsg,
		const char *	whatMsg
	)		const

Broadcast values from the root process to the slave processes.

Broadcasts a message from the process with rank "root" to all other processes of the communicator.

Parameters

buffer	(input/output) starting address of buffer
count	number of entries in buffer
datatype	data type of buffer
root	rank of broadcast root

Definition at line 157 of file MpiComm.C.

References m_rawComm, m_worldRank, and UQ_FATAL_TEST_MACRO.

Referenced by QUESO::MiscCheckForSameValueInAllNodes(), QUESO::GslVector::mpiAllQuantile(), and QUESO::GslVector::mpiBcast().

{
  int mpiRC = MPI_Bcast(buffer, count, datatype, root, m_rawComm);
  UQ_FATAL_TEST_MACRO(mpiRC != MPI_SUCCESS,
                      m_worldRank,
                      whereMsg,
                      whatMsg);
  return;
}

RawType_MPI_Comm QUESO::MpiComm::Comm

(

)

const

Extract MPI Communicator from a MpiComm object.

Definition at line 103 of file MpiComm.C.

References m_rawComm.

Referenced by QUESO::FullEnvironment::FullEnvironment().

{
#ifdef QUESO_HAS_TRILINOS
  return m_epetraMpiComm->Comm();
#endif
  return m_rawComm;
}

void QUESO::MpiComm::copy ( const MpiComm &  src )

private

Copies from an existing MpiComm instance.

Definition at line 269 of file MpiComm.C.

References m_myPid, m_numProc, m_rawComm, and m_worldRank.

Referenced by MpiComm(), and operator=().

{
#ifdef QUESO_HAS_TRILINOS
  delete m_epetraMpiComm;
  m_epetraMpiComm = new Epetra_MpiComm(*src.m_epetraMpiComm);
#endif
  m_rawComm   = src.m_rawComm;
  m_worldRank = src.m_worldRank;
  m_myPid     = src.m_myPid;
  m_numProc   = src.m_numProc;

  return;
}

void QUESO::MpiComm::Gather	(	void *	sendbuf,
		int	sendcnt,
		RawType_MPI_Datatype	sendtype,
		void *	recvbuf,
		int	recvcount,
		RawType_MPI_Datatype	recvtype,
		int	root,
		const char *	whereMsg,
		const char *	whatMsg
	)		const

Gather values from each process to collect on all processes.

Parameters

sendbuf	starting address of send buffer
sendcnt	number of elements in send buffer
sendtype	data type of send buffer elements
recvcount	number of elements for any single receive
recvtype	data type of recv buffer elements
root	rank of receiving process
recvbuf	(output) address of receive buffer

Definition at line 168 of file MpiComm.C.

References m_rawComm, m_worldRank, and UQ_FATAL_TEST_MACRO.

Referenced by QUESO::GslVector::mpiAllQuantile().

{
  //int MPI_Gather (void *sendbuf, int sendcnt, MPI_Datatype sendtype, 
  //                void *recvbuf, int recvcount, MPI_Datatype recvtype, 
  //                int root, MPI_Comm comm )
  int mpiRC = MPI_Gather(sendbuf, sendcnt, sendtype,
                         recvbuf, recvcount, recvtype,
                         root, m_rawComm);
  UQ_FATAL_TEST_MACRO(mpiRC != MPI_SUCCESS,
                      m_worldRank,
                      whereMsg,
                      whatMsg);
  return;
}

void QUESO::MpiComm::Gatherv	(	void *	sendbuf,
		int	sendcnt,
		RawType_MPI_Datatype	sendtype,
		void *	recvbuf,
		int *	recvcnts,
		int *	displs,
		RawType_MPI_Datatype	recvtype,
		int	root,
		const char *	whereMsg,
		const char *	whatMsg
	)		const

Gathers into specified locations from all processes in a group.

Parameters

sendbuf	starting address of send buffer
sendcount	number of elements in send buffer
sendtype	data type of send buffer elements
recvcounts	integer array (of length group size) containing the number of elements that are received from each process
displs	integer array (of length group size). Entry i specifies the displacement relative to recvbuf at which to place the incoming data from process i
recvtype	data type of recv buffer elements
root	rank of receiving process

Definition at line 188 of file MpiComm.C.

References m_rawComm, m_worldRank, and UQ_FATAL_TEST_MACRO.

{
  //int MPI_Gatherv(void *sendbuf, int sendcnt, MPI_Datatype sendtype, 
  //                void *recvbuf, int *recvcnts, int *displs, MPI_Datatype recvtype, 
  //                int root, MPI_Comm comm )
  int mpiRC = MPI_Gatherv(sendbuf, sendcnt, sendtype,
                          recvbuf, recvcnts, displs, recvtype,
                          root, m_rawComm);
  UQ_FATAL_TEST_MACRO(mpiRC != MPI_SUCCESS,
                      m_worldRank,
                      whereMsg,
                      whatMsg);
  return;
}

int QUESO::MpiComm::MyPID

(

)

const

Return my process ID.

Definition at line 112 of file MpiComm.C.

References m_myPid.

Referenced by QUESO::FullEnvironment::FullEnvironment(), QUESO::MiscCheckForSameValueInAllNodes(), QUESO::GslVector::mpiAllQuantile(), QUESO::GslVector::mpiAllReduce(), QUESO::GslVector::mpiBcast(), and syncPrintDebugMsg().

{
#ifdef QUESO_HAS_TRILINOS
  return m_epetraMpiComm->MyPID();
#endif
  return m_myPid;
}

int QUESO::MpiComm::NumProc

(

)

const

Returns total number of processes.

Definition at line 121 of file MpiComm.C.

References m_numProc.

Referenced by QUESO::FullEnvironment::FullEnvironment(), QUESO::EnvironmentOptions::getMyOptionValues(), QUESO::MiscCheckForSameValueInAllNodes(), QUESO::MiscCheckTheParallelEnvironment(), QUESO::GslVector::mpiAllQuantile(), QUESO::GslVector::mpiBcast(), QUESO::Matrix::numOfProcsForStorage(), QUESO::Vector::numOfProcsForStorage(), and syncPrintDebugMsg().

{
#ifdef QUESO_HAS_TRILINOS
  return m_epetraMpiComm->NumProc();
#endif
  return m_numProc;
}

MpiComm & QUESO::MpiComm::operator=

(

const MpiComm &

rhs

)

Assignment operator.

Definition at line 95 of file MpiComm.C.

References copy().

{
  this->copy(rhs);
  return *this;
}

void QUESO::MpiComm::Recv	(	void *	buf,
		int	count,
		RawType_MPI_Datatype	datatype,
		int	source,
		int	tag,
		RawType_MPI_Status *	status,
		const char *	whereMsg,
		const char *	whatMsg
	)		const

Blocking receive of data from this process to another process.

Parameters

buf	(output) initial address of receive buffer
status	(output) status object
count	maximum number of elements in receive buffer
datatype	datatype of each receive buffer element
source	rank of source
tag	message tag

Definition at line 208 of file MpiComm.C.

References m_rawComm, m_worldRank, and UQ_FATAL_TEST_MACRO.

{
  int mpiRC = MPI_Recv(buf, count, datatype, source, tag, m_rawComm, status);
  UQ_FATAL_TEST_MACRO(mpiRC != MPI_SUCCESS,
                      m_worldRank,
                      whereMsg,
                      whatMsg);
  return;
}

void QUESO::MpiComm::Send	(	void *	buf,
		int	count,
		RawType_MPI_Datatype	datatype,
		int	dest,
		int	tag,
		const char *	whereMsg,
		const char *	whatMsg
	)		const

Possibly blocking send of data from this process to another process.

Parameters

buf	initial address of send buffer
count	number of elements in send buffer
datatype	datatype of each send buffer element
dest	rank of destination
tag	message tag

Definition at line 221 of file MpiComm.C.

References m_rawComm, m_worldRank, and UQ_FATAL_TEST_MACRO.

{
  int mpiRC = MPI_Send(buf, count, datatype, dest, tag, m_rawComm);
  UQ_FATAL_TEST_MACRO(mpiRC != MPI_SUCCESS,
                      m_worldRank,
                      whereMsg,
                      whatMsg);
  return;
}

void QUESO::MpiComm::syncPrintDebugMsg	(	const char *	msg,
		unsigned int	msgVerbosity,
		unsigned int	numUSecs
	)		const

Synchronizes all the processes and print debug message.

Definition at line 234 of file MpiComm.C.

References Barrier(), QUESO::BaseEnvironment::fullRank(), QUESO::BaseEnvironment::inter0Rank(), m_env, MyPID(), NumProc(), QUESO::BaseEnvironment::subId(), QUESO::BaseEnvironment::subRank(), and QUESO::BaseEnvironment::syncVerbosity().

{
  if (m_env.syncVerbosity() >= msgVerbosity) {
    this->Barrier();
    for (int i = 0; i < this->NumProc(); ++i) {
      if (i == this->MyPID()) {
        std::cout << msg
                  << ": fullRank "       << m_env.fullRank()
                  << ", subEnvironment " << m_env.subId()
                  << ", subRank "        << m_env.subRank()
                  << ", inter0Rank "     << m_env.inter0Rank()
                  << std::endl;
      }
      usleep(numUSecs);
      this->Barrier();
    }
    //if (this->fullRank() == 0) std::cout << "Sleeping " << numUSecs << " microseconds..."
    //                                     << std::endl;
    //usleep(numUSecs);
    this->Barrier();
  }

  return;
}

Member Data Documentation

const BaseEnvironment& QUESO::MpiComm::m_env

private

Definition at line 206 of file MpiComm.h.

Referenced by syncPrintDebugMsg().

int QUESO::MpiComm::m_myPid

private

Process ID of this process.

Definition at line 219 of file MpiComm.h.

Referenced by copy(), MpiComm(), and MyPID().

int QUESO::MpiComm::m_numProc

private

Definition at line 222 of file MpiComm.h.

Referenced by copy(), MpiComm(), and NumProc().

RawType_MPI_Comm QUESO::MpiComm::m_rawComm

private

Embedded wrapped opaque MPI_Comm object.

Definition at line 213 of file MpiComm.h.

Referenced by Allreduce(), Barrier(), Bcast(), Comm(), copy(), Gather(), Gatherv(), Recv(), and Send().

int QUESO::MpiComm::m_worldRank

private

World rank.

Definition at line 216 of file MpiComm.h.

Referenced by Allreduce(), Barrier(), Bcast(), copy(), Gather(), Gatherv(), MpiComm(), Recv(), and Send().

---

The documentation for this class was generated from the following files:

• src/core/inc/MpiComm.h
• src/core/src/MpiComm.C