Setting TMPDIR in sbatch

[Michael Gutteridge]

Hi all

I have what I thought was a simple issue, but I can't seem to get it
sorted. We'd like to have job-specific working directories- basically have
TMPDIR=/some/path/${SLURM_JOB_ID} set for the user in batch or interactive
environments. Using use_env from the Spank collection works well, but
only for srun.

I thought I might be able to modify the use_env source slightly to run it
for sbatch as well, but it seems the functions available in the allocator
context (init and init_post_opt) don't have the slurm job ID available.
Total novice to writing things here- still working that angle. Does
context matter for those functions? Would init_post_opt game the job id if
run in the remote or allocator contexts only?

While I realize that we can configure $TMPDIR this way in the job script,
we would prefer to make sure this step doesn't get "forgotten" by our user
community. I have this nagging feeling I'm missing a very simple solution.

Thanks
Michael

[Mark A. Grondona]

Post by Michael Gutteridge
Hi all
I have what I thought was a simple issue, but I can't seem to get it
sorted. We'd like to have job-specific working directories- basically have
TMPDIR=/some/path/${SLURM_JOB_ID} set for the user in batch or interactive
environments. Using use_env from the Spank collection works well, but
only for srun.
I thought I might be able to modify the use_env source slightly to run it
for sbatch as well, but it seems the functions available in the allocator
context (init and init_post_opt) don't have the slurm job ID available.
Total novice to writing things here- still working that angle. Does
context matter for those functions? Would init_post_opt game the job id if
run in the remote or allocator contexts only?

sbatch itself doesn't have access to a jobid until *after* it submits
the job. I don't think there are any plugin callbacks at that point,
but it is probably moot anyway because the job has been submitted and
sbatch isn't going to do much at this point besides print the jobid
and exit.

You could possibly use the use-env plugin for all jobs if you place
the TMPDIR setting in the "in task" blocks. These blocks are run
in per-task context (so they run on the remote node instead of in
srun), but unfortunately that means you are setting TMPDIR N times
for a N task job, a waste, but probably negligible. For example,

in task {
TMPDIR = /some/path/$SLURM_JOB_ID
}

might work. (I didn't try it yet though)

However, I think for your use case, the use-env plugin is probably overkill.
There is a "tmpdir.c" example in the slurm-spank-plugins source, but
it is pretty old and I'm not even sure it works anymore, but you might
check it out just for instruction. You could also use the spank-lua
interface and develop this functionality in a lua plugin, if you are
willing to learn or already know lua. My guess is that you can do
most of what you want in just a few dozen lines of lua. Setting the
TMPDIR directly in a plugin will give you more functionality down
the road (for example you could create the directory, remove old
directories, bind the directory over /tmp in the current namespace,
etc.)

Let me know if you have any further questions,
mark

Post by Michael Gutteridge
While I realize that we can configure $TMPDIR this way in the job script,
we would prefer to make sure this step doesn't get "forgotten" by our user
community. I have this nagging feeling I'm missing a very simple solution.
Thanks
Michael
--
Hey! Somebody punched the foley guy!
- Crow, MST3K ep. 508

[Carles Fenoy]

Hi,

We are doing this in the job prolog. We basically use 2 prologs:
Prolog:
creates de directory ad chowns it to the job user as the tmp directory is
not user writable.
TaskProlog:
sets the TMPDIR environment variable

this is part of our TaskProlog perl script:
my $SLURM_JOBID = $ENV{SLURM_JOBID};

if (!defined($SLURM_JOBID)) {
panic(2, "${Script}: this script should be executed from slurm");
}

print "export TMPDIR=/scratch/tmp/$SLURM_JOBID;";


Regards,
Carles Fenoy

Post by Michael Gutteridge

Post by Michael Gutteridge
Hi all
I have what I thought was a simple issue, but I can't seem to get it
sorted. We'd like to have job-specific working directories- basically

have

Post by Michael Gutteridge
TMPDIR=/some/path/${SLURM_JOB_ID} set for the user in batch or

interactive

Post by Michael Gutteridge
environments. Using use_env from the Spank collection works well, but
only for srun.
I thought I might be able to modify the use_env source slightly to run it
for sbatch as well, but it seems the functions available in the allocator
context (init and init_post_opt) don't have the slurm job ID available.
Total novice to writing things here- still working that angle. Does
context matter for those functions? Would init_post_opt game the job id

if

Post by Michael Gutteridge
run in the remote or allocator contexts only?

sbatch itself doesn't have access to a jobid until *after* it submits
the job. I don't think there are any plugin callbacks at that point,
but it is probably moot anyway because the job has been submitted and
sbatch isn't going to do much at this point besides print the jobid
and exit.
You could possibly use the use-env plugin for all jobs if you place
the TMPDIR setting in the "in task" blocks. These blocks are run
in per-task context (so they run on the remote node instead of in
srun), but unfortunately that means you are setting TMPDIR N times
for a N task job, a waste, but probably negligible. For example,
in task {
TMPDIR = /some/path/$SLURM_JOB_ID
}
might work. (I didn't try it yet though)
However, I think for your use case, the use-env plugin is probably overkill.
There is a "tmpdir.c" example in the slurm-spank-plugins source, but
it is pretty old and I'm not even sure it works anymore, but you might
check it out just for instruction. You could also use the spank-lua
interface and develop this functionality in a lua plugin, if you are
willing to learn or already know lua. My guess is that you can do
most of what you want in just a few dozen lines of lua. Setting the
TMPDIR directly in a plugin will give you more functionality down
the road (for example you could create the directory, remove old
directories, bind the directory over /tmp in the current namespace,
etc.)
Let me know if you have any further questions,
mark

Post by Michael Gutteridge
While I realize that we can configure $TMPDIR this way in the job script,
we would prefer to make sure this step doesn't get "forgotten" by our

user

Post by Michael Gutteridge
community. I have this nagging feeling I'm missing a very simple

solution.

Post by Michael Gutteridge
Thanks
Michael
--
Hey! Somebody punched the foley guy!
- Crow, MST3K ep. 508

[Michael Gutteridge]

Awesome. I'm trying the task prolog approach as it seems a little
more straightforward for our environment- seems to work great.

The LUA plugin looks interesting if we have more "heavy lifting" to
do. Seems a bit overkill for this trivial task.

Thanks for all the advice.

Best

Michael

Hi,
creates de directory ad chowns it to the job user as the tmp directory is not user writable.
sets the TMPDIR environment variable
my $SLURM_JOBID = $ENV{SLURM_JOBID};
if (!defined($SLURM_JOBID)) {
panic(2, "${Script}: this script should be executed from slurm");
}
print "export TMPDIR=/scratch/tmp/$SLURM_JOBID;";
Regards,
Carles Fenoy

Post by Mark A. Grondona

Post by Michael Gutteridge
Hi all
I have what I thought was a simple issue, but I can't seem to get it
sorted. We'd like to have job-specific working directories- basically have
TMPDIR=/some/path/${SLURM_JOB_ID} set for the user in batch or interactive
environments. Using use_env from the Spank collection works well, but
only for srun.
I thought I might be able to modify the use_env source slightly to run it
for sbatch as well, but it seems the functions available in the allocator
context (init and init_post_opt) don't have the slurm job ID available.
Total novice to writing things here- still working that angle. Does
context matter for those functions? Would init_post_opt game the job id if
run in the remote or allocator contexts only?

sbatch itself doesn't have access to a jobid until *after* it submits
the job. I don't think there are any plugin callbacks at that point,
but it is probably moot anyway because the job has been submitted and
sbatch isn't going to do much at this point besides print the jobid
and exit.
You could possibly use the use-env plugin for all jobs if you place
the TMPDIR setting in the "in task" blocks. These blocks are run
in per-task context (so they run on the remote node instead of in
srun), but unfortunately that means you are setting TMPDIR N times
for a N task job, a waste, but probably negligible. For example,
in task {
TMPDIR = /some/path/$SLURM_JOB_ID
}
might work. (I didn't try it yet though)
However, I think for your use case, the use-env plugin is probably overkill.
There is a "tmpdir.c" example in the slurm-spank-plugins source, but
it is pretty old and I'm not even sure it works anymore, but you might
check it out just for instruction. You could also use the spank-lua
interface and develop this functionality in a lua plugin, if you are
willing to learn or already know lua. My guess is that you can do
most of what you want in just a few dozen lines of lua. Setting the
TMPDIR directly in a plugin will give you more functionality down
the road (for example you could create the directory, remove old
directories, bind the directory over /tmp in the current namespace,
etc.)
Let me know if you have any further questions,
mark

Post by Michael Gutteridge
While I realize that we can configure $TMPDIR this way in the job script,
we would prefer to make sure this step doesn't get "forgotten" by our user
community. I have this nagging feeling I'm missing a very simple solution.
Thanks
Michael
--
Hey! Somebody punched the foley guy!
- Crow, MST3K ep. 508

--
--
Carles Fenoy