python multiprocessing job on SLURM

[Sheila the angel]

Hello All,
I am new to SLURM and trying to run Multiprocessing job via python.

import multiprocessing
print multiprocessing.cpu_count()

The number of CPU shown by Python is always same (16) even if I reserve
only 4 CPU.
Does anyone have experience on running Multiprocessing job via python
on SLURM?
How can I run a python multiprocessing job efficiently on SLURM?

Thanks,

--
Best,
Sheila

[Satrajit Ghosh]

This should give you the correct number:

import os
print os.environ['SLURM_JOB_CPUS_PER_NODE']

cheers,

satra

Post by Sheila the angel
Hello All,
I am new to SLURM and trying to run Multiprocessing job via python.
import multiprocessing
print multiprocessing.cpu_count()
The number of CPU shown by Python is always same (16) even if I reserve
only 4 CPU.
Does anyone have experience on running Multiprocessing job via python
on SLURM?
How can I run a python multiprocessing job efficiently on SLURM?
Thanks,
--
Best,
Sheila

[Mark Roberts]

Re: [slurm-dev] python multiprocessing job on SLURM
import os
print os.environ['SLURM_JOB_CPUS_PER_NODE']
cheers,
satra
On Thu, Jul 17, 2014 at 4:56 AM, Sheila the angel
Hello All,
I am new to SLURM and trying to run Multiprocessing job via python.
import multiprocessing
print multiprocessing.cpu_count()
The number of CPU shown by Python is always same (16) even if I
reserve only 4 CPU.
Does anyone have experience on running Multiprocessing job via
python on SLURM?
How can I run a python multiprocessing job efficiently on SLURM?
Thanks,
--
Best,
Sheila

Shelia,
By default the python multi processing module will use all
the cpus it detects so as hinted above take the slurm environment
variable and pass that to the multiprocessing module to set the amount
of threads you wish to use so a very basic example could be :

import math
import multiprocessing as mp
import os

def f2(x):
return reduce(lambda a, b: math.log(a+b), xrange(10**5), x)

if __name__ == '__main__':

SLURM_CPUS = int(os.environ['SLURM_JOB_CPUS_PER_NODE'])
print "Using %s" % SLURM_CPUS
pool = mp.Pool(SLURM_CPUS)
result = pool.map(f2, range(1,10000))

Regards

Mark