Disabling partitions by time of day

[Bill Barth]

All,

We're trying to see if we can coax SLURM into to helping us institute a
policy to enable certain partitions to only run at night. We'd like to
have two partitions with overlapping nodes, and we'd like have one of the
partitions active 24-7 and the other only active 8am-8pm (say).

Is this possible with SLURM?

I could turn off one partition with a cron job, but that would allow jobs
to schedule at 7:59am and run into the closed window.

Any other thoughts on how this might be accomplished?

Thanks in advance,
Bill.
--
Bill Barth, Ph.D., Director, HPC
bbarth-***@public.gmane.org | Phone: [phone]
Office: ROC 1.435 | Fax: [phone]

[Lennart Karlsson]

Post by Bill Barth
All,
We're trying to see if we can coax SLURM into to helping us institute a
policy to enable certain partitions to only run at night. We'd like to
have two partitions with overlapping nodes, and we'd like have one of the
partitions active 24-7 and the other only active 8am-8pm (say).
Is this possible with SLURM?
I could turn off one partition with a cron job, but that would allow jobs
to schedule at 7:59am and run into the closed window.
Any other thoughts on how this might be accomplished?
Thanks in advance,
Bill.
--
Bill Barth, Ph.D., Director, HPC
Office: ROC 1.435 | Fax: [phone]

Bill,

We run a certain partition (actually two, on the same nodes) only on certain
hours on weekdays and accomplish this with cron and short reservations (the
WEEKLY flag keeps the reservation alive between weeks):

# scontrol show res
ReservationName=devel_prepare_monday StartTime=2014-03-24T08:59:00 EndTime=2014-03-24T09:04:00 Duration=00:05:00
Nodes=m[169-188] NodeCnt=20 CoreCnt=320 Features=(null) PartitionName=core Flags=OVERLAP,IGNORE_JOBS,WEEKLY,SPEC_NODES,STATIC
Users=lka Accounts=(null) Licenses=(null) State=INACTIVE

ReservationName=devel_prepare_tuesday StartTime=2014-03-18T08:59:00 EndTime=2014-03-18T09:04:00 Duration=00:05:00
Nodes=m[169-188] NodeCnt=20 CoreCnt=320 Features=(null) PartitionName=core Flags=OVERLAP,IGNORE_JOBS,WEEKLY,SPEC_NODES,STATIC
Users=lka Accounts=(null) Licenses=(null) State=INACTIVE

ReservationName=devel_prepare_wednesday StartTime=2014-03-19T08:59:00 EndTime=2014-03-19T09:04:00 Duration=00:05:00
Nodes=m[169-188] NodeCnt=20 CoreCnt=320 Features=(null) PartitionName=core Flags=OVERLAP,IGNORE_JOBS,WEEKLY,SPEC_NODES,STATIC
Users=lka Accounts=(null) Licenses=(null) State=INACTIVE

ReservationName=devel_prepare_thursday StartTime=2014-03-20T08:59:00 EndTime=2014-03-20T09:04:00 Duration=00:05:00
Nodes=m[169-188] NodeCnt=20 CoreCnt=320 Features=(null) PartitionName=core Flags=OVERLAP,IGNORE_JOBS,WEEKLY,SPEC_NODES,STATIC
Users=lka Accounts=(null) Licenses=(null) State=INACTIVE

ReservationName=devel_prepare_friday StartTime=2014-03-21T08:59:00 EndTime=2014-03-21T09:04:00 Duration=00:05:00
Nodes=m[169-188] NodeCnt=20 CoreCnt=320 Features=(null) PartitionName=core Flags=OVERLAP,IGNORE_JOBS,WEEKLY,SPEC_NODES,STATIC
Users=lka Accounts=(null) Licenses=(null) State=INACTIVE


# grep devel /etc/crontab
# Tricks with the devel and devcore partitions
0 9-19 * * 1-5 root (/usr/bin/scontrol update part=devel state=up; /usr/bin/scontrol update part=devcore state=up; /usr/bin/scontrol update part=node nodes=m[189-208]; /usr/bin/scontrol update part=core nodes=m[189-208])
0 0-8,20-23 * * 1-5 root (/usr/bin/scontrol update part=devel state=inactive; /usr/bin/scontrol update part=devcore state=inactive; /usr/bin/scontrol update part=node nodes=m[169-208]; /usr/bin/scontrol update part=core nodes=m[169-208])
0 * * * 6-7 root (/usr/bin/scontrol update part=devel state=inactive; /usr/bin/scontrol update part=devcore state=inactive; /usr/bin/scontrol update part=node nodes=m[169-208]; /usr/bin/scontrol update part=core nodes=m[169-208])

It is admittedly very ugly, but it works.

Best wishes,
-- Lennart Karlsson, UPPMAX, Uppsala University, Sweden