The QUESO MPI Communicator Class. More...

#include <MpiComm.h>

Collaboration diagram for QUESO::MpiComm:

Public Member Functions
Constructor/Destructor methods
	MpiComm (const BaseEnvironment &env, RawType_MPI_Comm inputRawComm)
	QUESO MpiComm MPI Constructor. More...

	MpiComm (const MpiComm &src)
	Copy Constructor. More...

	~MpiComm ()
	Destructor. More...

Set methods
MpiComm &	operator= (const MpiComm &rhs)
	Assignment operator. More...

Attribute Accessor Methods
RawType_MPI_Comm	Comm () const
	Extract MPI Communicator from a MpiComm object. More...

int	MyPID () const
	Return my process ID. More...

int	NumProc () const
	Returns total number of processes. More...

Methods Overridden from Comm
void	Allreduce (void sendbuf, void recvbuf, int count, RawType_MPI_Datatype datatype, RawType_MPI_Op op, const char whereMsg, const char whatMsg) const
	Combines values from all processes and distributes the result back to all processes. More...

void	Barrier () const
	Pause every process in *this communicator until all the processes reach this point. More...

void	Bcast (void buffer, int count, RawType_MPI_Datatype datatype, int root, const char whereMsg, const char *whatMsg) const
	Broadcast values from the root process to the slave processes. More...

void	Gather (void sendbuf, int sendcnt, RawType_MPI_Datatype sendtype, void recvbuf, int recvcount, RawType_MPI_Datatype recvtype, int root, const char whereMsg, const char whatMsg) const
	Gather values from each process to collect on all processes. More...

void	Gatherv (void sendbuf, int sendcnt, RawType_MPI_Datatype sendtype, void recvbuf, int recvcnts, int displs, RawType_MPI_Datatype recvtype, int root, const char whereMsg, const char whatMsg) const
	Gathers into specified locations from all processes in a group. More...

void	Recv (void buf, int count, RawType_MPI_Datatype datatype, int source, int tag, RawType_MPI_Status status, const char whereMsg, const char whatMsg) const
	Blocking receive of data from this process to another process. More...

void	Send (void buf, int count, RawType_MPI_Datatype datatype, int dest, int tag, const char whereMsg, const char *whatMsg) const
	Possibly blocking send of data from this process to another process. More...

Miscellaneous Methods
void	syncPrintDebugMsg (const char *msg, unsigned int msgVerbosity, unsigned int numUSecs) const
	Synchronizes all the processes and print debug message. More...

Private Member Functions
	MpiComm ()
	Default Constructor. More...

void	copy (const MpiComm &src)
	Copies from an existing MpiComm instance. More...

Private Attributes
const BaseEnvironment &	m_env

RawType_MPI_Comm	m_rawComm
	Embedded wrapped opaque MPI_Comm object. More...

int	m_worldRank
	World rank. More...

int	m_myPid
	Process ID of this process. More...

int	m_numProc

Detailed Description

The QUESO MPI Communicator Class.

This class uses MPI (the Message Passing Interface) for distributed-memory communication between one or more parallel processes. It is meant to insulate the user from the specifics of communication that are not required for normal manipulation of linear algebra objects.

Definition at line 75 of file MpiComm.h.

Constructor & Destructor Documentation

QUESO::MpiComm::MpiComm	(	const BaseEnvironment &	env,
		RawType_MPI_Comm	inputRawComm
	)

QUESO MpiComm MPI Constructor.

This constructs an MpiComm that uses the given "raw" MPI communicator underneath. The MPI_Comm must be valid for the lifetime of this MpiComm.

Definition at line 32 of file MpiComm.C.

References m_myPid, m_numProc, m_worldRank, and queso_require_equal_to_msg.

   :
   m_env          (env),
 #ifdef QUESO_HAS_TRILINOS
   m_epetraMpiComm( new Epetra_MpiComm(inputRawComm) ),
 #endif
   m_rawComm      (inputRawComm),
   m_worldRank    (-1),
   m_myPid        (-1),
   m_numProc      (-1)
 {
   int mpiRC = MPI_Comm_rank(inputRawComm,&m_worldRank);
   queso_require_equal_to_msg(mpiRC, MPI_SUCCESS, "failed MPI_Comm_rank() on full rank");
 
   mpiRC = MPI_Comm_rank(inputRawComm,&m_myPid);
   queso_require_equal_to_msg(mpiRC, MPI_SUCCESS, "failed MPI_Comm_rank() on inputRawComm");
 
   mpiRC = MPI_Comm_size(inputRawComm,&m_numProc);
   queso_require_equal_to_msg(mpiRC, MPI_SUCCESS, "failed MPI_Comm_size() on inputRawComm");
 }

QUESO::MpiComm::MpiComm ( const MpiComm & src )

Copy Constructor.

Makes an exact copy of an existing MpiComm instance.

Definition at line 54 of file MpiComm.C.

References copy().

   :
   m_env          (src.m_env)
 #ifdef QUESO_HAS_TRILINOS
   ,
   m_epetraMpiComm(NULL)
 #endif
 {
   this->copy(src);
 }

QUESO::MpiComm::~MpiComm ( )

Destructor.

Definition at line 66 of file MpiComm.C.

 {
 #ifdef QUESO_HAS_TRILINOS
   delete m_epetraMpiComm;
   m_epetraMpiComm = NULL;
 #endif
 }

QUESO::MpiComm::MpiComm ( )

private

Default Constructor.

It should not be used by user.

Member Function Documentation

void QUESO::MpiComm::Allreduce	(	void *	sendbuf,
		void *	recvbuf,
		int	count,
		RawType_MPI_Datatype	datatype,
		RawType_MPI_Op	op,
		const char *	whereMsg,
		const char *	whatMsg
	)		const

Combines values from all processes and distributes the result back to all processes.

Parameters

sendbuf	starting address of send buffer
count	number of elements in send buffer
datatype	data type of elements of send buffer
op	operation
recvbuf	(output) starting address of receive buffer

Definition at line 113 of file MpiComm.C.

References m_rawComm, and queso_require_equal_to_msg.

Referenced by QUESO::ComputeCovCorrBetweenScalarSequences(), QUESO::ComputeCovCorrMatricesBetweenVectorSequences(), QUESO::MiscCheckForSameValueInAllNodes(), QUESO::GslVector::mpiAllReduce(), QUESO::GslVector::mpiBcast(), QUESO::MHRawChainInfoStruct::mpiSum(), and QUESO::GslMatrix::mpiSum().

 {
   int mpiRC = MPI_Allreduce(sendbuf, recvbuf, count, datatype, op, m_rawComm);
   queso_require_equal_to_msg(mpiRC, MPI_SUCCESS, whatMsg);
 
   return;
 }

void QUESO::MpiComm::Barrier ( ) const

Pause every process in *this communicator until all the processes reach this point.

Blocks the caller until all processes in the communicator have called it; that is, the call returns at any process only after all members of the communicator have entered the call.

Definition at line 122 of file MpiComm.C.

References m_rawComm, and queso_require_equal_to_msg.

Referenced by QUESO::ComputeCovCorrMatricesBetweenVectorSequences(), QUESO::FullEnvironment::FullEnvironment(), QUESO::MiscCheckForSameValueInAllNodes(), QUESO::GslVector::mpiBcast(), and syncPrintDebugMsg().

 {
 #ifdef QUESO_HAS_TRILINOS
   return m_epetraMpiComm->Barrier();
 #endif
   int mpiRC = MPI_Barrier(m_rawComm);
   queso_require_equal_to_msg(mpiRC, MPI_SUCCESS, "mpiRC indicates failure");  // whatMsg);
   return;
 }

void QUESO::MpiComm::Bcast	(	void *	buffer,
		int	count,
		RawType_MPI_Datatype	datatype,
		int	root,
		const char *	whereMsg,
		const char *	whatMsg
	)		const

Broadcast values from the root process to the slave processes.

Broadcasts a message from the process with rank "root" to all other processes of the communicator.

Parameters

buffer	(input/output) starting address of buffer
count	number of entries in buffer
datatype	data type of buffer
root	rank of broadcast root

Definition at line 133 of file MpiComm.C.

References m_rawComm, and queso_require_equal_to_msg.

Referenced by QUESO::MiscCheckForSameValueInAllNodes(), QUESO::GslVector::mpiAllQuantile(), QUESO::GslVector::mpiBcast(), QUESO::InterpolationSurrogateIOASCII< V, M >::read(), and QUESO::InterpolationSurrogateData< V, M >::sync_values().

 {
   int mpiRC = MPI_Bcast(buffer, count, datatype, root, m_rawComm);
   queso_require_equal_to_msg(mpiRC, MPI_SUCCESS, whatMsg);
   return;
 }

RawType_MPI_Comm QUESO::MpiComm::Comm ( ) const

Extract MPI Communicator from a MpiComm object.

Definition at line 85 of file MpiComm.C.

References m_rawComm.

Referenced by QUESO::FullEnvironment::FullEnvironment().

 {
 #ifdef QUESO_HAS_TRILINOS
   return m_epetraMpiComm->Comm();
 #endif
   return m_rawComm;
 }

void QUESO::MpiComm::copy ( const MpiComm & src )

private

Copies from an existing MpiComm instance.

Definition at line 230 of file MpiComm.C.

References m_myPid, m_numProc, m_rawComm, and m_worldRank.

Referenced by MpiComm(), and operator=().

 {
 #ifdef QUESO_HAS_TRILINOS
   delete m_epetraMpiComm;
   m_epetraMpiComm = new Epetra_MpiComm(*src.m_epetraMpiComm);
 #endif
   m_rawComm   = src.m_rawComm;
   m_worldRank = src.m_worldRank;
   m_myPid     = src.m_myPid;
   m_numProc   = src.m_numProc;
 
   return;
 }

void QUESO::MpiComm::Gather	(	void *	sendbuf,
		int	sendcnt,
		RawType_MPI_Datatype	sendtype,
		void *	recvbuf,
		int	recvcount,
		RawType_MPI_Datatype	recvtype,
		int	root,
		const char *	whereMsg,
		const char *	whatMsg
	)		const

Gather values from each process to collect on all processes.

Parameters

sendbuf	starting address of send buffer
sendcnt	number of elements in send buffer
sendtype	data type of send buffer elements
recvcount	number of elements for any single receive
recvtype	data type of recv buffer elements
root	rank of receiving process
recvbuf	(output) address of receive buffer

Definition at line 141 of file MpiComm.C.

References m_rawComm, and queso_require_equal_to_msg.

Referenced by QUESO::GslVector::mpiAllQuantile().

 {
   //int MPI_Gather (void *sendbuf, int sendcnt, MPI_Datatype sendtype,
   //                void *recvbuf, int recvcount, MPI_Datatype recvtype,
   //                int root, MPI_Comm comm )
   int mpiRC = MPI_Gather(sendbuf, sendcnt, sendtype,
                          recvbuf, recvcount, recvtype,
                          root, m_rawComm);
   queso_require_equal_to_msg(mpiRC, MPI_SUCCESS, whatMsg);
   return;
 }

void QUESO::MpiComm::Gatherv	(	void *	sendbuf,
		int	sendcnt,
		RawType_MPI_Datatype	sendtype,
		void *	recvbuf,
		int *	recvcnts,
		int *	displs,
		RawType_MPI_Datatype	recvtype,
		int	root,
		const char *	whereMsg,
		const char *	whatMsg
	)		const

Gathers into specified locations from all processes in a group.

Parameters

sendbuf	starting address of send buffer
sendcount	number of elements in send buffer
sendtype	data type of send buffer elements
recvcounts	integer array (of length group size) containing the number of elements that are received from each process
displs	integer array (of length group size). Entry i specifies the displacement relative to recvbuf at which to place the incoming data from process i
recvtype	data type of recv buffer elements
root	rank of receiving process

Definition at line 158 of file MpiComm.C.

References m_rawComm, and queso_require_equal_to_msg.

Referenced by QUESO::InterpolationSurrogateBuilder< V, M >::sync_data().

 {
   //int MPI_Gatherv(void *sendbuf, int sendcnt, MPI_Datatype sendtype,
   //                void *recvbuf, int *recvcnts, int *displs, MPI_Datatype recvtype,
   //                int root, MPI_Comm comm )
   int mpiRC = MPI_Gatherv(sendbuf, sendcnt, sendtype,
                           recvbuf, recvcnts, displs, recvtype,
                           root, m_rawComm);
   queso_require_equal_to_msg(mpiRC, MPI_SUCCESS, whatMsg);
   return;
 }

int QUESO::MpiComm::MyPID ( ) const

Return my process ID.

Definition at line 94 of file MpiComm.C.

References m_myPid.

Referenced by QUESO::FullEnvironment::FullEnvironment(), QUESO::MiscCheckForSameValueInAllNodes(), QUESO::GslVector::mpiAllQuantile(), QUESO::GslVector::mpiAllReduce(), QUESO::GslVector::mpiBcast(), and syncPrintDebugMsg().

 {
 #ifdef QUESO_HAS_TRILINOS
   return m_epetraMpiComm->MyPID();
 #endif
   return m_myPid;
 }

int QUESO::MpiComm::NumProc ( ) const

Returns total number of processes.

Definition at line 103 of file MpiComm.C.

References m_numProc.

Referenced by QUESO::FullEnvironment::FullEnvironment(), QUESO::EnvironmentOptions::getMyOptionValues(), QUESO::MiscCheckForSameValueInAllNodes(), QUESO::MiscCheckTheParallelEnvironment(), QUESO::GslVector::mpiAllQuantile(), QUESO::GslVector::mpiBcast(), QUESO::Matrix::numOfProcsForStorage(), QUESO::Vector::numOfProcsForStorage(), and syncPrintDebugMsg().

 {
 #ifdef QUESO_HAS_TRILINOS
   return m_epetraMpiComm->NumProc();
 #endif
   return m_numProc;
 }

MpiComm & QUESO::MpiComm::operator= ( const MpiComm & rhs )

Assignment operator.

Definition at line 77 of file MpiComm.C.

References copy().

 {
   this->copy(rhs);
   return *this;
 }

void QUESO::MpiComm::Recv	(	void *	buf,
		int	count,
		RawType_MPI_Datatype	datatype,
		int	source,
		int	tag,
		RawType_MPI_Status *	status,
		const char *	whereMsg,
		const char *	whatMsg
	)		const

Blocking receive of data from this process to another process.

Parameters

buf	(output) initial address of receive buffer
status	(output) status object
count	maximum number of elements in receive buffer
datatype	datatype of each receive buffer element
source	rank of source
tag	message tag

Definition at line 175 of file MpiComm.C.

References m_rawComm, and queso_require_equal_to_msg.

 {
   int mpiRC = MPI_Recv(buf, count, datatype, source, tag, m_rawComm, status);
   queso_require_equal_to_msg(mpiRC, MPI_SUCCESS, whatMsg);
   return;
 }

void QUESO::MpiComm::Send	(	void *	buf,
		int	count,
		RawType_MPI_Datatype	datatype,
		int	dest,
		int	tag,
		const char *	whereMsg,
		const char *	whatMsg
	)		const

Possibly blocking send of data from this process to another process.

Parameters

buf	initial address of send buffer
count	number of elements in send buffer
datatype	datatype of each send buffer element
dest	rank of destination
tag	message tag

Definition at line 185 of file MpiComm.C.

References m_rawComm, and queso_require_equal_to_msg.

 {
   int mpiRC = MPI_Send(buf, count, datatype, dest, tag, m_rawComm);
   queso_require_equal_to_msg(mpiRC, MPI_SUCCESS, whatMsg);
   return;
 }

void QUESO::MpiComm::syncPrintDebugMsg	(	const char *	msg,
		unsigned int	msgVerbosity,
		unsigned int	numUSecs
	)		const

Synchronizes all the processes and print debug message.

Definition at line 195 of file MpiComm.C.

References Barrier(), QUESO::BaseEnvironment::fullRank(), QUESO::BaseEnvironment::inter0Rank(), m_env, MyPID(), NumProc(), QUESO::BaseEnvironment::subId(), QUESO::BaseEnvironment::subRank(), and QUESO::BaseEnvironment::syncVerbosity().

 {
   if (m_env.syncVerbosity() >= msgVerbosity) {
     this->Barrier();
     for (int i = 0; i < this->NumProc(); ++i) {
       if (i == this->MyPID()) {
         std::cout << msg
                   << ": fullRank "       << m_env.fullRank()
                   << ", subEnvironment " << m_env.subId()
                   << ", subRank "        << m_env.subRank()
                   << ", inter0Rank "     << m_env.inter0Rank()
                   << std::endl;
       }
       usleep(numUSecs);
       this->Barrier();
     }
     //if (this->fullRank() == 0) std::cout << "Sleeping " << numUSecs << " microseconds..."
     //                                     << std::endl;
     //usleep(numUSecs);
     this->Barrier();
   }
 
   return;
 }

Member Data Documentation

const BaseEnvironment& QUESO::MpiComm::m_env

private

Definition at line 206 of file MpiComm.h.

Referenced by syncPrintDebugMsg().

int QUESO::MpiComm::m_myPid

private

Process ID of this process.

Definition at line 219 of file MpiComm.h.

Referenced by copy(), MpiComm(), and MyPID().

int QUESO::MpiComm::m_numProc

private

Definition at line 222 of file MpiComm.h.

Referenced by copy(), MpiComm(), and NumProc().

RawType_MPI_Comm QUESO::MpiComm::m_rawComm

private

Embedded wrapped opaque MPI_Comm object.

Definition at line 213 of file MpiComm.h.

Referenced by Allreduce(), Barrier(), Bcast(), Comm(), copy(), Gather(), Gatherv(), Recv(), and Send().

int QUESO::MpiComm::m_worldRank

private

World rank.

Definition at line 216 of file MpiComm.h.

Referenced by copy(), and MpiComm().

The documentation for this class was generated from the following files:

src/core/inc/MpiComm.h
src/core/src/MpiComm.C

Public Member Functions

Private Member Functions

Private Attributes

Detailed Description

Constructor & Destructor Documentation

Member Function Documentation

Member Data Documentation