How to change how frequently SLURM updates the output file (stdout)?

[Neal Kruis]

I posted this question to Stack Overflow
<http://stackoverflow.com/questions/25170763/how-to-change-how-frequently-slurm-updates-the-output-file-stdout>
:

I am using SLURM to dispatch jobs on a supercomputer. I have set the
--output=log.out option to place the content from a job's stdout into a
file (log.out). I'm finding that the file is updated every 30-60 minutes,
making it difficult for me to check on the status of my jobs.

Any idea why it takes so long to update this file? Is there a way to change
settings so that this file is updated more frequently?

Using SLURM 14.03.4-2

[Christopher Samuel]

Post by Neal Kruis
I am using SLURM to dispatch jobs on a supercomputer. I have set
the |--output=log.out| option to place the content from a job's stdout
into a file (|log.out|). I'm finding that the file is updated every
30-60 minutes, making it difficult for me to check on the status of my jobs.

I have a suspicion that's related to your filesystem, not Slurm.

Certainly we don't see any such issue using Slurm 2.6.x with GPFS and
Panasas filesystems.

All the best,
Chris

[Neal Kruis]

Thanks, Chris,

It turns out it was a buffering issue where the stdout buffer wasn't being
flushed regularly enough.

Neal Kruis
828 482-2088

Big Ladder Software
2549 Irving Street
Denver, CO 80211
http://www.bigladdersoftware.com

Post by Christopher Samuel

Post by Neal Kruis
I am using SLURM to dispatch jobs on a supercomputer. I have set
the |--output=log.out| option to place the content from a job's stdout
into a file (|log.out|). I'm finding that the file is updated every
30-60 minutes, making it difficult for me to check on the status of my

jobs.
I have a suspicion that's related to your filesystem, not Slurm.
Certainly we don't see any such issue using Slurm 2.6.x with GPFS and
Panasas filesystems.
All the best,
Chris
--
Christopher Samuel Senior Systems Administrator
VLSCI - Victorian Life Sciences Computation Initiative
http://www.vlsci.org.au/ http://twitter.com/vlsci