Discussion:
Pro/Epi-Log scripts not running
Brian Baughman
2014-06-17 20:21:36 UTC
Permalink
Greetings,

I added some prolog and epilog scripts to our cluster environment:

* Copied script files to all nodes
* Updated slurm.conf
* Reconfigured slurm (also tried restarting) on all nodes

Am I missing a step in getting these scripts to run?

Regards,
Brian
Joan Arbona
2014-06-18 12:31:07 UTC
Permalink
Do your scripts have the execution bit
activated?

On 17/06/14 22:21, Brian Baughman wrote:
Greetings,

I added some prolog and epilog scripts to our cluster environment:

* Copied script files to all nodes
* Updated slurm.conf
* Reconfigured slurm (also tried restarting) on all nodes

Am I missing a step in getting these scripts to run?

Regards,
Brian
--

Joan Francesc Arbona
Ext. 2582
Centre de Tecnologies de la Informació
Universitat de les Illes Balears

http://jfdeu.wordpress.com
http://guifisoller.wordpress.com
Brian Baughman
2014-06-18 12:40:31 UTC
Permalink
Indeed they do.

--
Regards,
Brian
Do your scripts have the execution bit activated?
Post by Brian Baughman
Greetings,
* Copied script files to all nodes
* Updated slurm.conf
* Reconfigured slurm (also tried restarting) on all nodes
Am I missing a step in getting these scripts to run?
Regards,
Brian
--
Joan Francesc Arbona
Ext. 2582
Centre de Tecnologies de la Informació
Universitat de les Illes Balears
http://jfdeu.wordpress.com
http://guifisoller.wordpress.com
Joan Arbona
2014-06-18 13:51:32 UTC
Permalink
Ok. I asked that because it was the
first problem I had when I configured epilog scripts.

I think you are not missing anything. However, how do you know
that the scripts are not getting executed? Just remember that
prolog and epilog get executed in the compute node, not in the
slurmctld host.

http://slurm.schedmd.com/prolog_epilog.html

Do logs say something, by the way?

Good luck!
Joan

On 18/06/14 14:40, Brian Baughman wrote:
Indeed they do. 

--
Regards,
Brian
On Jun 18, 2014, at 8:30, Joan Arbona <joan.arbona-e/wFEmVp3/***@public.gmane.org>
wrote:
Do your scripts have the
execution bit activated?

On 17/06/14 22:21, Brian Baughman wrote:
Greetings,

I added some prolog and epilog scripts to our cluster environment:

* Copied script files to all nodes
* Updated slurm.conf
* Reconfigured slurm (also tried restarting) on all nodes

Am I missing a step in getting these scripts to run?

Regards,
Brian
--
Joan Francesc Arbona
Ext. 2582
Centre de Tecnologies de la Informació
Universitat de les Illes Balears

http://jfdeu.wordpress.com
http://guifisoller.wordpress.com
--

Joan Francesc Arbona
Ext. 2582
Centre de Tecnologies de la Informació
Universitat de les Illes Balears

http://jfdeu.wordpress.com
http://guifisoller.wordpress.com
Brian Baughman
2014-06-18 14:13:34 UTC
Permalink
Hello Joan,

The scripts basically just echo stuff and set some environmental variables. If I set TESTVAL in my user prolog file:

export TESTVAL="this is a test variable"

Then run a test job that does something like:

echo ">>>${TESTVAL}<<<"
<<<
So the variable isn’t set.

Regards,
Brian
Joan Arbona
2014-06-19 07:40:33 UTC
Permalink
In my understanding the export command
makes that variable available only to child processes of your
shell. Consequently I doubt this way you can access to variables
exported in a prolog script from any shell that slurm provides.
However, in the documentation I've seen there's a way to do this:

- Create a prolog script and print the command that exports your
variables:
prolog.sh:
/
#!/bin/bash
echo "export VAR1=\"VALUE1\""/
/echo "export VAR2=\"VALUE2\""
...

/- In your slurm.conf file, use the TaskProlog directive
instead of Prolog/:
/Slurm.conf/:
...
TaskProlog=<dir>/prolog.sh
.../

- And execute:
/$srun env/

And you should see VAR1 and VAR2 variables.

Note that this prolog only gets executed in jobsteps (when
executing srun), that means that when you run an sbatch or an
salloc it won't work.

However, the best way to check if your prolog/epilog scripts are
working is to cast something to a file somewhere and then go to
check it manually.

Greetings,
Joan
/
/On 18/06/14 16:13, Brian Baughman wrote:
Hello Joan,

The scripts basically just echo stuff and set some environmental variables. If I set TESTVAL in my user prolog file:

export TESTVAL="this is a test variable"

Then run a test job that does something like:

echo ">>>${TESTVAL}<<<"

I get the output:
<<<
So the variable isn’t set.

Regards,
Brian
--

Joan Francesc Arbona
Ext. 2582
Centre de Tecnologies de la Informació
Universitat de les Illes Balears

http://jfdeu.wordpress.com
http://guifisoller.wordpress.com
Brian Baughman
2014-06-19 08:16:51 UTC
Permalink
Hello Joan,

Indeed that is what I determined but some of the other aspects of the script I was using didn’t work, e.g. making a working temp directory. Those aspects “magically” did started working after I temporarily changed it from a TaskProlog to a SrunProlog and back.

Currently I am still having issues with the TaskEpilog script not printing to the task stdout as per the documentation (echo “print END” should send “END” to the stdout).

Thank you for your help.

Regards,
Brian

Loading...