非阻塞 MPI 通信的死锁

Question

以下代码是一个例程，它将鬼点与顶部/底部和左/右邻居进行通信。该例程在迭代方法的循环期间被调用，大约数百次。

问题是，虽然它是用非阻塞通信编写的，但它是死锁的……有趣的是，它似乎可以进行几次迭代，然后突然冻结。我试图评论第二个通信循环（顶部/底部），它也冻结了，但迭代索引更大......所有发生的事情都好像允许最大数量的通信或其他东西一样。

我认为 Isend 和 Irecv 不应该出现死锁。我认为，在调用 MPI_Wait_all() 之前，我还注意不要接触缓冲区。

我是否以不好的方式使用 MPI 例程？

void fields_updateEghosts(Fields *this, Params *parameters)
{


double *ex_in[2], *ex_out[2], *ey_in[2], *ey_out[2];
int neighbors_in[2], neighbors_out[2], ineighb;
MPI_Request requests_lr[4], requests_tb[4];
int ix,iy,nx,ny,i;
YeeCell *yee;

yee = this->yeecell;
nx  = this->nx;
ny  = this->ny;

/* 0,1 = top/bottom for Ey and left/right for Ex*/
for (i=0; i < 2; i++)
{
    MEM_ALLOC(ex_in[i],  ny*sizeof *ex_in[0]);
    MEM_ALLOC(ex_out[i], ny*sizeof *ex_out[0]);
    MEM_ALLOC(ey_in[i],  nx*sizeof *ey_in[0]);
    MEM_ALLOC(ey_out[i], nx*sizeof *ey_out[0]);
}

/* we send the points just inside the boundary */
for (iy=1; iy < ny; iy++)
{
    ex_out[PART_LEFT][iy]  = ex(1   ,iy);
    ex_out[PART_RIGHT][iy] = ex(nx-2,iy);
}

neighbors_in[0]  = PART_LEFT;
neighbors_in[1]  = PART_RIGHT;
neighbors_out[0] = PART_RIGHT;
neighbors_out[1] = PART_LEFT;

for (ineighb=0; ineighb < 2; ineighb++)
{
    MPI_Irecv(ex_in[neighbors_in[ineighb]],
              ny, MPI_DOUBLE,
              parameters->para->neighbors[neighbors_in[ineighb]], /*src rank */
              neighbors_out[ineighb],                           /* tag */
              MPI_COMM_WORLD,
              &requests_lr[ineighb]);

    MPI_Isend(ex_out[neighbors_out[ineighb]],
              ny, MPI_DOUBLE,
              parameters->para->neighbors[neighbors_out[ineighb]],
              neighbors_out[ineighb],
              MPI_COMM_WORLD,
              &requests_lr[ineighb+2]);
}

/* fill the outgoing top and bottom buffers
   while left/right communications are done*/
for (ix=1; ix < nx; ix++)
{
    ey_out[PART_TOP][ix] = ey(ix,ny-2);
    ey_out[PART_BOT][ix] = ey(ix,1);
}


/* now communications for top/bottom */
neighbors_in[0]  = PART_TOP;
neighbors_in[1]  = PART_BOT;
neighbors_out[0] = PART_BOT;
neighbors_out[1] = PART_TOP;

for (ineighb=0; ineighb < 2; ineighb++)
{
    MPI_Irecv(ey_in[neighbors_in[ineighb]],
              nx, MPI_DOUBLE,
              parameters->para->neighbors[neighbors_in[ineighb]],
              neighbors_out[ineighb],
              MPI_COMM_WORLD,
              &requests_tb[ineighb]);

    MPI_Isend(ey_out[neighbors_out[ineighb]],
              nx, MPI_DOUBLE,
              parameters->para->neighbors[neighbors_out[ineighb]],
              neighbors_out[ineighb],
              MPI_COMM_WORLD,
              &requests_tb[ineighb+2]);
}

/* now wait for communications to be done
   before copying the data into the arrays */

MPI_Waitall(4, requests_lr, MPI_STATUS_IGNORE);
MPI_Waitall(4, requests_tb, MPI_STATUS_IGNORE);


for (iy=1; iy < ny; iy++)
{
    ex(0   ,iy) = ex_in[PART_LEFT][iy];
    ex(nx-1,iy) = ex_in[PART_RIGHT][iy];
}

for (ix=1; ix < nx; ix++)
{
    ey(ix,ny-1) = ey_in[PART_TOP][ix];
    ey(ix,0)    = ey_in[PART_BOT][ix];
}



for (i=0; i < 2; i++)
{
    MEM_FREE(ex_in[i]);
    MEM_FREE(ex_out[i]);
    MEM_FREE(ey_in[i]);
    MEM_FREE(ey_out[i]);
}
}

Answer 1

我找到了答案。我现在将解释它以防有人遇到同样的问题。 首先，上面的功能很好，我认为没有问题。 问题在于调用此例程以获取幻影节点值的迭代方法。 迭代方法有一个收敛标准，我忘了为全局域计算它。因此，一些进程将在其他进程之前满足收敛测试，并退出循环......让其他进程永远等待他们的伙伴...... 一个小的 mpi_allreduce() 计算收敛，不再阻塞。