Running MPI task with HyperQueue
# Environment
HyperQueue: v0.18.0
HPC: MareNostrum 5
Host OS: Red Hat Enterprise Linux 9.2
Slurm Version: 23.02.7
Intel MPI version: 2021.10 Build 20230619 (id: c2e19c2f3e)
Intel C Compiler: 2021.10.0 20230609
# Description of the bug (or clarification of the behavior)
Hello, HyperQueue developers:
I have been benchmarking HyperQueue as an in-situ workflow manager with an MPI application. I noticed that with HyperQueue there was a significant performance degradation with respect to the other in-situ workflow managers, around 20% when distributing among four nodes.
I managed to pinpoint this to how the ranks are being distributed among the nodes in the allocation. For some reason, when the Hydra process manager is launched in Hyperqueue, is launched with `-N 2 -n 2` parameters, for the case with two nodes in the allocation .
Below I attach two logs that were produced by my launch script (which I include in the appendix). In the first log we have the execution of the task by hyperqueue. In the second, directly with mpirun.
My question is if this is a configuration issue, design, or something from the environment that is interfering with hyperqueue.
```
(base) [bsc032371@gs02r1b61 ~]$ head job-1/0.stdout
[mpiexec@gs02r1b62] Launch arguments: /usr/bin/srun -N 2 -n 2 --ntasks-per-node 1 --nodelist gs02r1b61,gs02r1b62 --input none /gpfs/apps/MN5/GPP/ONEAPI/2023.2.0/mpi/2021.10.0/bin//hydra_bstrap_proxy --upstream-host gs02r1b62 --upstream-port 37917 --pgid 0 --launcher slurm --launcher-number 1 --base-path /gpfs/apps/MN5/GPP/ONEAPI/2023.2.0/mpi/2021.10.0/bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 0 --debug /gpfs/apps/MN5/GPP/ONEAPI/2023.2.0/mpi/2021.10.0/bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
```
```
[mpiexec@gs02r1b61] Launch arguments: /usr/bin/srun -N 1 -n 1 --ntasks-per-node 1 --nodelist gs02r1b61 --input none /gpfs/apps/MN5/GPP/ONEAPI/2023.2.0/mpi/2021.10.0/bin//hydra_bstrap_proxy --upstream-host gs02r1b61 --upstream-port 36007 --pgid 0 --launcher slurm --launcher-number 1 --base-path /gpfs/apps/MN5/GPP/ONEAPI/2023.2.0/mpi/2021.10.0/bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 0 --debug /gpfs/apps/MN5/GPP/ONEAPI/2023.2.0/mpi/2021.10.0/bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
```
This, therefore, makes the `mpirun` aware of the other nodes in the allocation, even though it should not have been. My application runs in less cores than the CPUs available in MN5. So the ranks are distributed among the nodes. See the below output of my small test application.
```
Hello world from rank 2 out of 10 processors at machine gs02r1b61 with the following cpus allowed to run 22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,
Hello world from rank 6 out of 10 processors at machine gs02r1b61 with the following cpus allowed to run 66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,178,179,180,181,182,183,184,185,186,187,Hello world from rank 8 out of 10 processors at machine gs02r1b61 with the following cpus allowed to run 88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,200,201,202,203,204,205,206,207,208,209,210,211,212,213,214,188,189,190,191,192,193,194,195,196,197,198,199,
215,216,217,218,219,220,221,
Hello world from rank 0 out of 10 processors at machine gs02r1b61 with the following cpus allowed to run 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,112,113,114,115,116,Hello world from rank 4 out of 10 processors at machine gs02r1b61 with the following cpus allowed to run 44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,
117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,
Hello world from rank 1 out of 10 processors at machine gs02r1b62 with the following cpus allowed to run 56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,Hello world from rank 3 out of 10 processors at machine gs02r1b62 with the following cpus allowed to run 78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,205,206,207,208,209,210,211,
Hello world from rank 5 out of 10 processors at machine gs02r1b62 with the following cpus allowed to run 0,1,2,3,4,5,6,7,8,9,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,212,213,214,215,216,217,218,219,220,221,222,223,
Hello world from rank 7 out of 10 processors at machine gs02r1b62 with the following cpus allowed to run 10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,
77,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,
Hello world from rank 9 out of 10 processors at machine gs02r1b62 with the following cpus allowed to run 32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,
```
And this is the output when I just run directly the code.
```
Hello world from rank 3 out of 10 processors at machine gs02r1b61 with the following cpus allowed to run 33,34,35,36,37,38,39,40,41,42,43,145,Hello world from rank 4 out of 10 processors at machine gs02r1b61 with the following cpus allowed to run 44,45,46,47,48,49,50,51,52,53,54,156,157,158,159,160,161,162,163,164,165,166,
Hello world from rank 6 out of 10 processors at machine gs02r1b61 with the following cpus allowed to run 67,68,69,70,71,72,73,74,75,76,77,179,180,181,182,183,184,185,186,187,188,189,
Hello world from rank 7 out of 10 processors at machine gs02r1b61 with the following cpus allowed to run 78,79,80,81,82,83,84,85,86,87,88,190,191,192,193,194,195,196,197,198,199,200,
Hello world from rank 9 out of 10 processors at machine gs02r1b61 with the following cpus allowed to run 100,101,102,103,104,105,106,107,108,109,110,212,213,214,215,216,217,218,219,220,221,222,
Hello world from rank 1 out of 10 processors at machine gs02r1b61 with the following cpus allowed to run 11,12,13,14,15,16,17,18,19,20,21,123,124,125,126,127,128,129,130,131,132,133,
Hello world from rank 2 out of 10 processors at machine gs02r1b61 with the following cpus allowed to run 22,23,24,25,26,27,28,29,30,31,32,134,135,136,137,138,139,140,141,142,143,144,
146,147,148,149,150,151,152,153,154,155,
Hello world from rank 0 out of 10 processors at machine gs02r1b61 with the following cpus allowed to run 0,1,2,3,4,5,6,7,8,9,10,112,113,114,115,116,117,118,119,120,121,122,
Hello world from rank 5 out of 10 processors at machine gs02r1b61 with the following cpus allowed to run 56,57,58,59,60,61,62,63,64,65,66,168,169,170,171,172,173,174,175,176,177,178,
Hello world from rank 8 out of 10 processors at machine gs02r1b61 with the following cpus allowed to run 89,90,91,92,93,94,95,96,97,98,99,201,202,203,204,205,206,207,208,209,210,211,
```
# Reproducible example
I coded this hello world MPI program (I put the whole code in the appendix), that outputs the rank and the machine where it executed (and also the cpu affinity).
Then I launched this hello world with the launch script (also in the appendix) under an interactive allocation that I have used to set up a hyperqueue server and workers (all commands in the appendix).
# Appendix
Task script
```
#!/usr/bin/bash
module purge
module load impi
mpirun -v -np 10 /home/bsc/bsc032371/mpi_cpu_mask_and_rank/hello
```
HyperQueue setup
```
(base) [bsc032371@glogin1 ~]$ salloc -A bsc32 --qos gp_debug -N 2
(base) [bsc032371@gs02r1b61 ~]$ srun -v --overlap --cpus-per-task=1 --ntasks-per-node=1 --nodes=1 hq server start &
(base) [bsc032371@gs02r1b61 ~]$ srun --overlap -v --cpu-bind=none --nodes=${SLURM_NNODES} hq worker start --cpus=2x112 --manager slurm &
```
MPI Program
```
#define _GNU_SOURCE
#include <mpi.h>
#include <stdio.h>
#include <sched.h>
#include <unistd.h>
#include <string.h>
// Source - https://stackoverflow.com/q
// Posted by rishran
// Retrieved 2025-12-11, License - CC BY-SA 3.0
void long_to_binary(unsigned long k, char *c){
c[0] = '\0';
unsigned long val;
for (val = 1UL << (sizeof(unsigned long)*8-1); val > 0; val >>= 1) {
strcat(c, ((k & val) == val) ? "1" : "0");
}
}
int main(int argc, char** argv) {
// Initialize the MPI environment
MPI_Init(NULL, NULL);
// Get the number of processes
int world_size;
MPI_Comm_size(MPI_COMM_WORLD, &world_size);
// Get the rank of the process
int world_rank;
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);
// Get the name of the processor
char processor_name[MPI_MAX_PROCESSOR_NAME];
int name_len;
MPI_Get_processor_name(processor_name, &name_len);
// Get the pid of the current rank
pid_t current_pid;
current_pid = getpid();
// get the mask of the rank
cpu_set_t set;
sched_getaffinity(current_pid, sizeof(set), &set);
//char binary_set[64]; //sixteen bits because of mn5's architecture
//long_to_binary(set.__bits[0], binary_set);
// Print off a hello world message
printf("Hello world from rank %d out of %d processors at machine %s with the following cpus allowed to run ", world_rank, world_size, processor_name);
for (int cpu = 0; cpu < CPU_SETSIZE; cpu++){
if (CPU_ISSET(cpu, &set)){
printf("%d,", cpu);
}
}
printf("\n");
MPI_Finalize();
}
```
关闭于 2025-12-12 9 条评论