Requested node configuration is not available

Discussion:

Eva Hocks

2014-09-23 22:16:33 UTC

slurm Version=14.03:

I am trying to run a simple job with

#SBATCH --nodes=1-1
#SBATCH --ntasks=2
#SBATCH --cpus-per-task=1

on a test cluster with 2 nodes both configured:
CPUAlloc=0 CPUErr=0 CPUTot=8

but whenever I try sbatch it refuses: Requested node configuration is
not available. The nodes have 8 CPUs configured so why can't I run a 2
task job on 2 of those CPUs? I can run a 1 task job or if I don't
include the ntasks I get by default 8 tasks, that works. But I cannot
specify anything >1 task. sbatch --verbose isn't very helpful. The slurm
log has:

[2014-09-23T14:58:03.834] cons_res: cr_job_test: evaluating job 152 on 3 nodes
[2014-09-23T14:58:03.834] cons_res: _can_job_run_on_node: 8 cpus on hpc-0-4(1), mem 0/24151
[2014-09-23T14:58:03.834] cons_res: _can_job_run_on_node: 8 cpus on hpc-0-5(0), mem 0/24151
[2014-09-23T14:58:03.834] cons_res: _can_job_run_on_node: 8 cpus on hpc-0-6(1), mem 0/24151
[2014-09-23T14:58:03.834] cons_res: eval_nodes:0 consec c=24 n=3 b=0 e=2 r=-1
[2014-09-23T14:58:03.834] cons_res: cr_job_test: test 0 fail: insufficient resources
[2014-09-23T14:58:03.834] no job_resources info for job 152 rc=-1
[2014-09-23T14:58:03.834] _pick_best_nodes: job 152 never runnable
[2014-09-23T14:58:03.835] _slurm_rpc_submit_batch_job: Requested node configuration is not available

and scontrol shows the job with
JobState=FAILED Reason=BadConstraints
NumNodes=1 NumCPUs=2 CPUs/Task=1

BTW mysql is running and I can with no problem connect to the server
manually.

Any insight how to run a 1 node/2CPU job in slurm?

Thanks
Eva

Eva Hocks

2014-09-23 22:50:35 UTC

Permalink

If I use

#SBATCH -N 1-1
#SBATCH -c 2 the job starts and uses

NumNodes=1 NumCPUs=8 CPUs/Task=2

Eva

Post by Eva Hocks
I am trying to run a simple job with
#SBATCH --nodes=1-1
#SBATCH --ntasks=2
#SBATCH --cpus-per-task=1
CPUAlloc=0 CPUErr=0 CPUTot=8
but whenever I try sbatch it refuses: Requested node configuration is
not available. The nodes have 8 CPUs configured so why can't I run a 2
task job on 2 of those CPUs? I can run a 1 task job or if I don't
include the ntasks I get by default 8 tasks, that works. But I cannot
specify anything >1 task. sbatch --verbose isn't very helpful. The slurm
[2014-09-23T14:58:03.834] cons_res: cr_job_test: evaluating job 152 on 3 nodes
[2014-09-23T14:58:03.834] cons_res: _can_job_run_on_node: 8 cpus on hpc-0-4(1), mem 0/24151
[2014-09-23T14:58:03.834] cons_res: _can_job_run_on_node: 8 cpus on hpc-0-5(0), mem 0/24151
[2014-09-23T14:58:03.834] cons_res: _can_job_run_on_node: 8 cpus on hpc-0-6(1), mem 0/24151
[2014-09-23T14:58:03.834] cons_res: eval_nodes:0 consec c=24 n=3 b=0 e=2 r=-1
[2014-09-23T14:58:03.834] cons_res: cr_job_test: test 0 fail: insufficient resources
[2014-09-23T14:58:03.834] no job_resources info for job 152 rc=-1
[2014-09-23T14:58:03.834] _pick_best_nodes: job 152 never runnable
[2014-09-23T14:58:03.835] _slurm_rpc_submit_batch_job: Requested node configuration is not available
and scontrol shows the job with
JobState=FAILED Reason=BadConstraints
NumNodes=1 NumCPUs=2 CPUs/Task=1
BTW mysql is running and I can with no problem connect to the server
manually.
Any insight how to run a 1 node/2CPU job in slurm?
Thanks
Eva

Eva Hocks

2014-09-23 22:58:39 UTC

Permalink

I found

#SBATCH --ntasks-per-node=2

and now the job runs with NumNodes=1-1 NumCPUs=2 CPUs/Task=2

The docu on www.schedmd.com/slurmdocs/sbatch.html

shows

-n, --ntasks ......

Is this flag still working for job steps to run within the allocation
will launch a maximum of "number" tasks ? Doesn't look like it.

Thanks
Eva

Post by Eva Hocks
If I use
#SBATCH -N 1-1
#SBATCH -c 2 the job starts and uses
NumNodes=1 NumCPUs=8 CPUs/Task=2
Eva

Trey Dockendorf

2014-09-24 00:07:32 UTC

Permalink

Eva,

I think your misunderstanding the purpose and default behavior of "--ntasks".

If you read the last sentence of the description for --ntasks - "The default is one task per node".

So from your first email, using "--nodes=1-1 --ntasks=2 --cpus-per-task=1" will not work. Your asking for 2 tasks and a max of one node, and the default for number of tasks is one per node (see quote above). If you just said "--nodes=1" then that may work since your max was undefined and as the docs mention for --nodes, the job will be allocated enough nodes to satisfy your number of tasks.

Those options work just fine, and as expected, in 14.03.3 up to 14.03.6 (versions I've used in past months).

- Trey

=============================

Trey Dockendorf
Systems Analyst I
Texas A&M University
Academy for Advanced Telecommunications and Learning Technologies
Phone: (979)458-2396
Email: treydock-mRW4Vj+***@public.gmane.org
Jabber: treydock-mRW4Vj+***@public.gmane.org

----- Original Message -----

Sent: Tuesday, September 23, 2014 5:59:28 PM
Subject: [slurm-dev] Re: Requested node configuration is not available
I found
#SBATCH --ntasks-per-node=2
and now the job runs with NumNodes=1-1 NumCPUs=2 CPUs/Task=2
The docu on www.schedmd.com/slurmdocs/sbatch.html
shows
-n, --ntasks ......
Is this flag still working for job steps to run within the allocation
will launch a maximum of "number" tasks ? Doesn't look like it.
Thanks
Eva

Post by Eva Hocks
If I use
#SBATCH -N 1-1
#SBATCH -c 2 the job starts and uses
NumNodes=1 NumCPUs=8 CPUs/Task=2
Eva

Post by Eva Hocks
I am trying to run a simple job with
#SBATCH --nodes=1-1
#SBATCH --ntasks=2
#SBATCH --cpus-per-task=1
CPUAlloc=0 CPUErr=0 CPUTot=8
but whenever I try sbatch it refuses: Requested node
configuration is
not available. The nodes have 8 CPUs configured so why can't I run a 2
task job on 2 of those CPUs? I can run a 1 task job or if I don't
include the ntasks I get by default 8 tasks, that works. But I cannot
specify anything >1 task. sbatch --verbose isn't very helpful. The slurm
[2014-09-23T14:58:03.834] cons_res: cr_job_test: evaluating job 152 on 3 nodes
[2014-09-23T14:58:03.834] cons_res: _can_job_run_on_node: 8 cpus
on hpc-0-4(1), mem 0/24151
[2014-09-23T14:58:03.834] cons_res: _can_job_run_on_node: 8 cpus
on hpc-0-5(0), mem 0/24151
[2014-09-23T14:58:03.834] cons_res: _can_job_run_on_node: 8 cpus
on hpc-0-6(1), mem 0/24151
[2014-09-23T14:58:03.834] cons_res: eval_nodes:0 consec c=24 n=3 b=0 e=2 r=-1
insufficient resources
[2014-09-23T14:58:03.834] no job_resources info for job 152 rc=-1
[2014-09-23T14:58:03.834] _pick_best_nodes: job 152 never
runnable
[2014-09-23T14:58:03.835] _slurm_rpc_submit_batch_job: Requested
node configuration is not available
and scontrol shows the job with
JobState=FAILED Reason=BadConstraints
NumNodes=1 NumCPUs=2 CPUs/Task=1
BTW mysql is running and I can with no problem connect to the server
manually.
Any insight how to run a 1 node/2CPU job in slurm?
Thanks
Eva

Eva Hocks

2014-09-24 00:22:35 UTC

Permalink

Yes, yo are right, I do not understand what --ntasks means and
when to use it or --ntasks-per-node

I assumed it meant " n tasks " for the job, in my case, 2 tasks for
this job, running on 1 node with 1 cpu per task, which would make it 1
nodes/2 cpu allocation.

As for the "default is one task per node" : when using --nodes=1 I get
an allocation of 1 node/ 8 cpus :

NumNodes=1 NumCPUs=8 CPUs/Task=1

so I get a default of 1 task per cpu, not one task per node.

Thanks for your help,
Eva

Post by Trey Dockendorf
Eva,
I think your misunderstanding the purpose and default behavior of "--ntasks".
If you read the last sentence of the description for --ntasks - "The default is one task per node".
So from your first email, using "--nodes=1-1 --ntasks=2 --cpus-per-task=1" will not work. Your asking for 2 tasks and a max of one node, and the default for number of tasks is one per node (see quote above). If you just said "--nodes=1" then that may work since your max was undefined and as the docs mention for --nodes, the job will be allocated enough nodes to satisfy your number of tasks.
Those options work just fine, and as expected, in 14.03.3 up to 14.03.6 (versions I've used in past months).
- Trey
=============================
Trey Dockendorf
Systems Analyst I
Texas A&M University
Academy for Advanced Telecommunications and Learning Technologies
Phone: (979)458-2396
----- Original Message -----

Post by Eva Hocks
If I use
#SBATCH -N 1-1
#SBATCH -c 2 the job starts and uses
NumNodes=1 NumCPUs=8 CPUs/Task=2
Eva

Post by Eva Hocks
I am trying to run a simple job with
#SBATCH --nodes=1-1
#SBATCH --ntasks=2
#SBATCH --cpus-per-task=1
CPUAlloc=0 CPUErr=0 CPUTot=8
but whenever I try sbatch it refuses: Requested node
configuration is
not available. The nodes have 8 CPUs configured so why can't I run a 2
task job on 2 of those CPUs? I can run a 1 task job or if I don't
include the ntasks I get by default 8 tasks, that works. But I cannot
specify anything >1 task. sbatch --verbose isn't very helpful. The slurm
[2014-09-23T14:58:03.834] cons_res: cr_job_test: evaluating job 152 on 3 nodes
[2014-09-23T14:58:03.834] cons_res: _can_job_run_on_node: 8 cpus
on hpc-0-4(1), mem 0/24151
[2014-09-23T14:58:03.834] cons_res: _can_job_run_on_node: 8 cpus
on hpc-0-5(0), mem 0/24151
[2014-09-23T14:58:03.834] cons_res: _can_job_run_on_node: 8 cpus
on hpc-0-6(1), mem 0/24151
[2014-09-23T14:58:03.834] cons_res: eval_nodes:0 consec c=24 n=3 b=0 e=2 r=-1
insufficient resources
[2014-09-23T14:58:03.834] no job_resources info for job 152 rc=-1
[2014-09-23T14:58:03.834] _pick_best_nodes: job 152 never
runnable
[2014-09-23T14:58:03.835] _slurm_rpc_submit_batch_job: Requested
node configuration is not available
and scontrol shows the job with
JobState=FAILED Reason=BadConstraints
NumNodes=1 NumCPUs=2 CPUs/Task=1
BTW mysql is running and I can with no problem connect to the server
manually.
Any insight how to run a 1 node/2CPU job in slurm?
Thanks
Eva

Trey Dockendorf

2014-09-24 00:38:37 UTC

Permalink

My suggestion would be evaluate your job's need and use the parameters that match your job's needed resources. Carefully read docs too. I've found that even skimming the docs when I'm looking at a particular parameter is not enough. Things like the default behaviors and what other parameters change the behavior are also not always directly under a parameter's description. A browser's "Find text" (CTRL+F on my desktop) can be very handy when you want to see all the ways the behavior of a particular parameter can change.

It's hard to describe the difference between the --ntasks and --ntasks-per-node options without quoting the entire description in the docs. When to use one vs another depends on your configuration (ie partitions, qos, SelectTypeParameters, etc) as well as what your job needs. I typically am very explicit in my sbatch allocation requests and will use --nodes, --ntasks-per-node and --cpus-per-task.

- Trey

=============================

Trey Dockendorf
Systems Analyst I
Texas A&M University
Academy for Advanced Telecommunications and Learning Technologies
Phone: (979)458-2396
Email: treydock-mRW4Vj+***@public.gmane.org
Jabber: treydock-mRW4Vj+***@public.gmane.org

----- Original Message -----

Sent: Tuesday, September 23, 2014 7:23:22 PM
Subject: [slurm-dev] Re: Requested node configuration is not available
Yes, yo are right, I do not understand what --ntasks means and
when to use it or --ntasks-per-node
I assumed it meant " n tasks " for the job, in my case, 2 tasks for
this job, running on 1 node with 1 cpu per task, which would make it 1
nodes/2 cpu allocation.
As for the "default is one task per node" : when using --nodes=1 I get
NumNodes=1 NumCPUs=8 CPUs/Task=1
so I get a default of 1 task per cpu, not one task per node.
Thanks for your help,
Eva

Post by Trey Dockendorf
Eva,
I think your misunderstanding the purpose and default behavior of "--ntasks".
If you read the last sentence of the description for --ntasks -
"The default is one task per node".
So from your first email, using "--nodes=1-1 --ntasks=2
--cpus-per-task=1" will not work. Your asking for 2 tasks and a
max of one node, and the default for number of tasks is one per
node (see quote above). If you just said "--nodes=1" then that
may work since your max was undefined and as the docs mention for
--nodes, the job will be allocated enough nodes to satisfy your
number of tasks.
Those options work just fine, and as expected, in 14.03.3 up to
14.03.6 (versions I've used in past months).
- Trey
=============================
Trey Dockendorf
Systems Analyst I
Texas A&M University
Academy for Advanced Telecommunications and Learning Technologies
Phone: (979)458-2396
----- Original Message -----

Sent: Tuesday, September 23, 2014 5:59:28 PM
Subject: [slurm-dev] Re: Requested node configuration is not available
I found
#SBATCH --ntasks-per-node=2
and now the job runs with NumNodes=1-1 NumCPUs=2 CPUs/Task=2
The docu on www.schedmd.com/slurmdocs/sbatch.html
shows
-n, --ntasks ......
Is this flag still working for job steps to run within the
allocation
will launch a maximum of "number" tasks ? Doesn't look like it.
Thanks
Eva

Post by Eva Hocks
If I use
#SBATCH -N 1-1
#SBATCH -c 2 the job starts and uses
NumNodes=1 NumCPUs=8 CPUs/Task=2
Eva

Post by Eva Hocks
I am trying to run a simple job with
#SBATCH --nodes=1-1
#SBATCH --ntasks=2
#SBATCH --cpus-per-task=1
CPUAlloc=0 CPUErr=0 CPUTot=8
but whenever I try sbatch it refuses: Requested node
configuration is
not available. The nodes have 8 CPUs configured so why can't
I
run a 2
task job on 2 of those CPUs? I can run a 1 task job or if I don't
include the ntasks I get by default 8 tasks, that works. But
I
cannot
specify anything >1 task. sbatch --verbose isn't very
helpful.
The slurm
[2014-09-23T14:58:03.834] cons_res: cr_job_test: evaluating
job
152 on 3 nodes
[2014-09-23T14:58:03.834] cons_res: _can_job_run_on_node: 8 cpus
on hpc-0-4(1), mem 0/24151
[2014-09-23T14:58:03.834] cons_res: _can_job_run_on_node: 8 cpus
on hpc-0-5(0), mem 0/24151
[2014-09-23T14:58:03.834] cons_res: _can_job_run_on_node: 8 cpus
on hpc-0-6(1), mem 0/24151
[2014-09-23T14:58:03.834] cons_res: eval_nodes:0 consec c=24
n=3
b=0 e=2 r=-1
insufficient resources
[2014-09-23T14:58:03.834] no job_resources info for job 152 rc=-1
[2014-09-23T14:58:03.834] _pick_best_nodes: job 152 never
runnable
Requested
node configuration is not available
and scontrol shows the job with
JobState=FAILED Reason=BadConstraints
NumNodes=1 NumCPUs=2 CPUs/Task=1
BTW mysql is running and I can with no problem connect to the server
manually.
Any insight how to run a 1 node/2CPU job in slurm?
Thanks
Eva