Discussion:
even CPU load
Леонид Коньков
2014-07-28 05:59:32 UTC
Permalink
Hi.

I want my CPUs be loaded as even as possible. I have 22 nodes
(motherboards) with 1 CPU each with 8 CPU cores each. ( My English is far
from perfect, and I'm not shure what is what in you terminology.) I want to
load 100 onethread tasks with 4-5 tasks on one CPU. But tasks always fill
all 8 cores of CPU and leave some CPUs idle.

<--distribution=cyclic> and <--hint=memory_bound> don't help.

Leo.
Marcin Stolarek
2014-07-28 14:16:07 UTC
Permalink
Post by Леонид Коньков
Hi.
I want my CPUs be loaded as even as possible. I have 22 nodes
(motherboards) with 1 CPU each with 8 CPU cores each. ( My English is far
from perfect, and I'm not shure what is what in you terminology.) I want to
load 100 onethread tasks with 4-5 tasks on one CPU. But tasks always fill
all 8 cores of CPU and leave some CPUs idle.
you mean doesn't leave any idle cpu? You want node to be responsible and
available for interactive work while tasks are running?
Post by Леонид Коньков
<--distribution=cyclic> and <--hint=memory_bound> don't help.
Leo.
If I understood you correctly, you can limit the slurmd cpuset to a subset
of your cores, so jobs running under this cpuset won't use more than whole
cgroup/cpuset/slurm/.

cheers,
marcin
Rod Schultz
2014-07-28 15:11:44 UTC
Permalink
Slurm’s default strategy is to fill entire nodes. That is why you are getting some nodes with 8 tasks and some idle nodes.

Try srun –n 100 –N22 –ntasks-per-node=5 –l hostname

-n 100 means start 100 tasks,
-N 22 means all 22 nodes
--ntasks-per-node=5 distributes the tasks.
-l appends task number to the output, so you can see the distribution.



From: Marcin Stolarek [mailto:***@gmail.com]
Sent: Monday, July 28, 2014 7:17 AM
To: slurm-dev
Subject: [slurm-dev] Re: even CPU load



2014-07-28 8:00 GMT+02:00 Леонид Коньков <***@gmail.com<mailto:***@gmail.com>>:
Hi.
I want my CPUs be loaded as even as possible. I have 22 nodes (motherboards) with 1 CPU each with 8 CPU cores each. ( My English is far from perfect, and I'm not shure what is what in you terminology.) I want to load 100 onethread tasks with 4-5 tasks on one CPU. But tasks always fill all 8 cores of CPU and leave some CPUs idle.
you mean doesn't leave any idle cpu? You want node to be responsible and available for interactive work while tasks are running?


<--distribution=cyclic> and <--hint=memory_bound> don't help.
Leo.


If I understood you correctly, you can limit the slurmd cpuset to a subset of your cores, so jobs running under this cpuset won't use more than whole cgroup/cpuset/slurm/.
cheers,
marcin
Леонид Коньков
2014-07-28 16:02:44 UTC
Permalink
To Rod Schultz.

I use sbatch. When I try

sbatch -a 0-19 -p mpi -n 20 -N 10 -ntasks-per-node=2 t40z0600s1.sh

or

sbatch -a 0-19 -p mpi -ntasks-per-node=2 t40z0600s1.sh

it says

sbatch: error: Invalid numeric value "tasks-per-node=2" for number of tasks.

Any suggestions?

Thanks anyway.
Post by Rod Schultz
Slurm’s default strategy is to fill entire nodes. That is why you are
getting some nodes with 8 tasks and some idle nodes.
Try srun –n 100 –N22 –ntasks-per-node=5 –l hostname
-n 100 means start 100 tasks,
-N 22 means all 22 nodes
--ntasks-per-node=5 distributes the tasks.
-l appends task number to the output, so you can see the distribution.
*Sent:* Monday, July 28, 2014 7:17 AM
*To:* slurm-dev
*Subject:* [slurm-dev] Re: even CPU load
Hi.
I want my CPUs be loaded as even as possible. I have 22 nodes
(motherboards) with 1 CPU each with 8 CPU cores each. ( My English is far
from perfect, and I'm not shure what is what in you terminology.) I want to
load 100 onethread tasks with 4-5 tasks on one CPU. But tasks always fill
all 8 cores of CPU and leave some CPUs idle.
you mean doesn't leave any idle cpu? You want node to be responsible and
available for interactive work while tasks are running?
<--distribution=cyclic> and <--hint=memory_bound> don't help.
Leo.
[image: Image removed by sender.]
If I understood you correctly, you can limit the slurmd cpuset to a subset
of your cores, so jobs running under this cpuset won't use more than whole
cgroup/cpuset/slurm/.
cheers,
marcin
[image: Image removed by sender.]
j***@public.gmane.org
2014-07-28 16:09:31 UTC
Permalink
Try
--ntasks-per-node=2 reather than
-ntasks-per-node=2
Post by Леонид Коньков
To Rod Schultz.
I use sbatch. When I try
sbatch -a 0-19 -p mpi -n 20 -N 10 -ntasks-per-node=2 t40z0600s1.sh
or
sbatch -a 0-19 -p mpi -ntasks-per-node=2 t40z0600s1.sh
it says
sbatch: error: Invalid numeric value "tasks-per-node=2" for number of tasks.
Any suggestions?
Thanks anyway.
Post by Rod Schultz
Slurm’s default strategy is to fill entire nodes. That is why you are
getting some nodes with 8 tasks and some idle nodes.
Try srun –n 100 –N22 –ntasks-per-node=5 –l hostname
-n 100 means start 100 tasks,
-N 22 means all 22 nodes
--ntasks-per-node=5 distributes the tasks.
-l appends task number to the output, so you can see the distribution.
*Sent:* Monday, July 28, 2014 7:17 AM
*To:* slurm-dev
*Subject:* [slurm-dev] Re: even CPU load
Hi.
I want my CPUs be loaded as even as possible. I have 22 nodes
(motherboards) with 1 CPU each with 8 CPU cores each. ( My English is far
from perfect, and I'm not shure what is what in you terminology.) I want to
load 100 onethread tasks with 4-5 tasks on one CPU. But tasks always fill
all 8 cores of CPU and leave some CPUs idle.
you mean doesn't leave any idle cpu? You want node to be responsible and
available for interactive work while tasks are running?
<--distribution=cyclic> and <--hint=memory_bound> don't help.
Leo.
[image: Image removed by sender.]
If I understood you correctly, you can limit the slurmd cpuset to a subset
of your cores, so jobs running under this cpuset won't use more than whole
cgroup/cpuset/slurm/.
cheers,
marcin
[image: Image removed by sender.]
--
Morris "Moe" Jette
CTO, SchedMD LLC

Slurm User Group Meeting
September 23-24, Lugano, Switzerland
Find out more http://slurm.schedmd.com/slurm_ug_agenda.html
Леонид Коньков
2014-07-28 16:19:34 UTC
Permalink
Yeah, stupid typo. My bad. Thanks for poiting.

Strange. It puts 4 tasks per node with --ntasks-per-node=2.
Post by j***@public.gmane.org
Try
--ntasks-per-node=2 reather than
-ntasks-per-node=2
Леонид Коньков
2014-07-28 16:38:37 UTC
Permalink
Command:

sbatch -a 0-99 -p mpi --ntasks-per-node=5 t40z0600s1.sh

squeue:

JOBID PARTITION NAME USER ST TIME NODES
NODELIST(REASON)
9366_0 mpi t40z0600 leonid R 0:15 1 S
9366_1 mpi t40z0600 leonid R 0:15 1 Sc
9366_2 mpi t40z0600 leonid R 0:15 1 Si
9366_3 mpi t40z0600 leonid R 0:15 1 Ti
9366_4 mpi t40z0600 leonid R 0:15 1 Al
9366_5 mpi t40z0600 leonid R 0:15 1 Ar
9366_6 mpi t40z0600 leonid R 0:15 1 B
9366_7 mpi t40z0600 leonid R 0:14 1 Be
9366_8 mpi t40z0600 leonid R 0:14 1 C
9366_9 mpi t40z0600 leonid R 0:14 1 Ca
9366_10 mpi t40z0600 leonid R 0:14 1 Cl
9366_11 mpi t40z0600 leonid R 0:14 1 F
9366_12 mpi t40z0600 leonid R 0:14 1 H
9366_13 mpi t40z0600 leonid R 0:14 1 He
9366_14 mpi t40z0600 leonid R 0:14 1 K
9366_15 mpi t40z0600 leonid R 0:14 1 Li
9366_16 mpi t40z0600 leonid R 0:14 1 Mg
9366_17 mpi t40z0600 leonid R 0:14 1 N
9366_18 mpi t40z0600 leonid R 0:14 1 Na
9366_19 mpi t40z0600 leonid R 0:14 1 Ne
9366_20 mpi t40z0600 leonid R 0:14 1 O
9366_21 mpi t40z0600 leonid R 0:14 1 P
9366_[22-99] mpi t40z0600 leonid PD 0:00 1
(Resources)

Command:

sbatch -a 0-99 -p mpi -n 99 -N 22 --ntasks-per-node=5 t40z0600s1.sh

squeue:

JOBID PARTITION NAME USER ST TIME NODES
NODELIST(REASON)
9466_0 mpi t40z0600 leonid R 0:33 22
Al,Ar,B,Be,C,Ca,Cl,F,H,He,K,Li,Mg,N,Na,Ne,O,P,S,Sc,Si,Ti
9466_[1-99] mpi t40z0600 leonid PD 0:00 22
(Resources)


Any help?

Леонид Коньков
2014-07-28 16:07:32 UTC
Permalink
Post by Rod Schultz
you can limit the slurmd cpuset to a subset of your cores, so jobs
running under this cpuset won't use more than whole cgroup/cpuset/slurm/.

I guess that it has sense, but for me it looks more like mumbo-jumbo. Sorry.
<https://plus.google.com/u/0/103884178532124129641?prsrc=4>
Post by Rod Schultz
Hi.
Post by Леонид Коньков
I want my CPUs be loaded as even as possible. I have 22 nodes
(motherboards) with 1 CPU each with 8 CPU cores each. ( My English is far
from perfect, and I'm not shure what is what in you terminology.) I want to
load 100 onethread tasks with 4-5 tasks on one CPU. But tasks always fill
all 8 cores of CPU and leave some CPUs idle.
you mean doesn't leave any idle cpu? You want node to be responsible and
available for interactive work while tasks are running?
Post by Леонид Коньков
<--distribution=cyclic> and <--hint=memory_bound> don't help.
Leo.
If I understood you correctly, you can limit the slurmd cpuset to a subset
of your cores, so jobs running under this cpuset won't use more than whole
cgroup/cpuset/slurm/.
cheers,
marcin
Loading...