More odd scheduler reservation behavior

[Bill Barth]

All,

I'm experiencing the following unexpected behavior with SLURM
reservations. If I create a reservation on some nodes and forget to point
it to a specific partition, and I update the reservation later to point at
the correct partition, it doesn't remove any nodes reserved in the wrong
partition and replace them with nodes from the partition specified.

Here's the details, first beginning with some info about the relevant
defined partitions:

PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
SB2.7* up 2-00:00:00 2 down* c3-[401,421]
SB2.7* up 2-00:00:00 26 idle c3-[402-420,422-428]
IB2.2 up 2-00:00:00 12 idle c3-[501-512]


Create the reservation:

-bash-4.2$ sudo scontrol create reservation StartTime=2014-06-06T08:00:00
Duration=1:00:00 NodeCnt=4 Users=bbarth
Reservation created: bbarth_3
-bash-4.2$ scontrol show res
ReservationName=bbarth_3 StartTime=2014-06-06T08:00:00
EndTime=2014-06-06T09:00:00 Duration=01:00:00
Nodes=c3-[402-405] NodeCnt=4 CoreCnt=64 Features=(null)
PartitionName=SB2.7 Flags=
Users=bbarth Accounts=(null) Licenses=(null) State=INACTIVE


Observe that the nodes happen to come from the SB2.7 partition. If update
the partition on the reservation to be IB2.2, we see that the nodes from
SB2.7 are still the ones reserved:

-bash-4.2$ sudo scontrol update ReservationName=bbarth_3 Partition=IB2.2
Reservation updated.
-bash-4.2$ scontrol show res
ReservationName=bbarth_3 StartTime=2014-06-06T08:00:00
EndTime=2014-06-06T09:00:00 Duration=01:00:00
Nodes=c3-[402-405] NodeCnt=4 CoreCnt=64 Features=(null)
PartitionName=IB2.2 Flags=
Users=bbarth Accounts=(null) Licenses=(null) State=INACTIVE

Is this the expected behavior?

I also notice that if I drain a node it doesn't get replaced in the
reservation, and if I stop SLURM on the node (/etc/init.d/slurm stop) it
doesn't get replaced either. I would have sworn up and down that at least
the latter worked.

Can anyone provide some feedback?

Thanks,
Bill.

--
Bill Barth, Ph.D., Director, HPC
bbarth-***@public.gmane.org | Phone: [phone]
Office: ROC 1.435 | Fax: [phone]

[Bill Barth]

No thoughts on this from the list? I wouldn't have thought we were the
only ones encountering this issue.

Best,
Bill.
--
Bill Barth, Ph.D., Director, HPC
bbarth-***@public.gmane.org | Phone: [phone]
Office: ROC 1.435 | Fax: [phone]

Post by Bill Barth
All,
I'm experiencing the following unexpected behavior with SLURM
reservations. If I create a reservation on some nodes and forget to point
it to a specific partition, and I update the reservation later to point at
the correct partition, it doesn't remove any nodes reserved in the wrong
partition and replace them with nodes from the partition specified.
Here's the details, first beginning with some info about the relevant
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
SB2.7* up 2-00:00:00 2 down* c3-[401,421]
SB2.7* up 2-00:00:00 26 idle c3-[402-420,422-428]
IB2.2 up 2-00:00:00 12 idle c3-[501-512]
-bash-4.2$ sudo scontrol create reservation StartTime=2014-06-06T08:00:00
Duration=1:00:00 NodeCnt=4 Users=bbarth
Reservation created: bbarth_3
-bash-4.2$ scontrol show res
ReservationName=bbarth_3 StartTime=2014-06-06T08:00:00
EndTime=2014-06-06T09:00:00 Duration=01:00:00
Nodes=c3-[402-405] NodeCnt=4 CoreCnt=64 Features=(null)
PartitionName=SB2.7 Flags=
Users=bbarth Accounts=(null) Licenses=(null) State=INACTIVE
Observe that the nodes happen to come from the SB2.7 partition. If update
the partition on the reservation to be IB2.2, we see that the nodes from
-bash-4.2$ sudo scontrol update ReservationName=bbarth_3 Partition=IB2.2
Reservation updated.
-bash-4.2$ scontrol show res
ReservationName=bbarth_3 StartTime=2014-06-06T08:00:00
EndTime=2014-06-06T09:00:00 Duration=01:00:00
Nodes=c3-[402-405] NodeCnt=4 CoreCnt=64 Features=(null)
PartitionName=IB2.2 Flags=
Users=bbarth Accounts=(null) Licenses=(null) State=INACTIVE
Is this the expected behavior?
I also notice that if I drain a node it doesn't get replaced in the
reservation, and if I stop SLURM on the node (/etc/init.d/slurm stop) it
doesn't get replaced either. I would have sworn up and down that at least
the latter worked.
Can anyone provide some feedback?
Thanks,
Bill.
--
Bill Barth, Ph.D., Director, HPC
Office: ROC 1.435 | Fax: [phone]