在使用 Linux 的 databricks 集群上运行 Open MPI 时出现“没有足够的插槽”错误

Question

我尝试使用 mpi 在 databricks 集群上运行 C 应用程序。

我已从以下位置下载 Open MPI https://download.open-mpi.org/release/open-mpi/v4.0/openmpi-4.0.3.tar.gz

并安装在 databricks 集群上。

它是基于 Ubuntu 的 databricks 集群构建的。

  Operating system/version: Linux 4.4.0 Ubuntu
  Computer hardware: x86_64
  Network type: databricks

我正在尝试从 databricks 上的 python 笔记本运行：

%sh
mpirun --allow-run-as-root -np 20  MY_c_Application

MY_c_Application 由 C 语言编写并在 databricks Linux 上编译。

我的 databricks 集群有 21 个节点，其中一个作为驱动程序。每个节点有 32 个核心。

当我运行上述命令时，我得到如下错误。

请告诉我这是怎么造成的？ 或者，我错过了什么？

谢谢

---

There are not enough slots available in the system to satisfy the 20
slots that were requested by the application:

   MY_c_application

 Either request fewer slots for your application, or make more slots available for use.

 A "slot" is the Open MPI term for an allocatable unit where we can launch a process. 

 The number of slots available are defined by the environment in which Open MPI processes are run:

 Hostfile, via "slots=N" clauses (N defaults to number of processor cores if not provided)

  The --host command line parameter, via a ":N" suffix on the hostname
  (N defaults to 1 if not provided)

 Resource manager (e.g., SLURM, PBS/Torque, LSF, etc.)

  If none of a hostfile, the --host command line parameter, or an RM 
  is present, Open MPI defaults to the number of processor cores In 
  all the above cases, if you want Open MPI to default to the number 
  of hardware threads instead of the number of processor cores, use 
  the --use-hwthread-cpus option.

 Alternatively, you can use the --oversubscribe option to ignore the 
 number of available slots when deciding the number of processes to launch.

---

更新

添加hostfile后，这个问题就没有了。

 sudo mpirun --allow-run-as-root -np 25 --hostfile my_hostfile ./MY_C_APP 

谢谢

Answer 1

按照原帖者分享答案：

添加主机文件后，问题解决。

 sudo mpirun --allow-run-as-root -np 25 --hostfile my_hostfile ./MY_C_APP