Discussion:
Slurm daemon on Xeon Phi cards?
Christopher Samuel
2014-02-18 01:46:07 UTC
Permalink
Hi all,

At the Slurm User Group in Oakland last year it was mentioned that
there was intended to be support for a lightweight Slurm daemon on
Xeon Phi (MIC) cards.

I had a quick look in the git master last night but couldn't spot
anything related, is this still the intention?

Olli-Pekka Lehto from CSC is running a Xeon Phi workshop at VLSCI at
the moment and it's of interest to a number of us.

We're going to run a hack day on Wednesday and we'll see if we can
build an LDAP enabled Xeon Phi stack, if we can then we we'll see if
we can get standard Slurm going too. Nothing like having lofty goals!

All the best!
Chris
- --
Christopher Samuel Senior Systems Administrator
VLSCI - Victorian Life Sciences Computation Initiative
Email: samuel-***@public.gmane.org Phone: +61 (0)3 903 55545
http://www.vlsci.org.au/ http://twitter.com/vlsci
Ralph Castain
2014-02-18 02:05:26 UTC
Permalink
I know others have direct-launched processes onto the Phi before, both with Slurm and just using rsh/ssh. The OpenMPI user mailing list archive talks about the ssh method (search for "phi" and you'll see the chatter)

http://www.open-mpi.org/community/lists/users/

and the folks at Bright talk about how they did it with Slurm here:

https://www.linkedin.com/groups/Yes-we-run-SLURM-inside-4501392.S.5792769036550955008

Ralph
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi all,
At the Slurm User Group in Oakland last year it was mentioned that
there was intended to be support for a lightweight Slurm daemon on
Xeon Phi (MIC) cards.
I had a quick look in the git master last night but couldn't spot
anything related, is this still the intention?
Olli-Pekka Lehto from CSC is running a Xeon Phi workshop at VLSCI at
the moment and it's of interest to a number of us.
We're going to run a hack day on Wednesday and we'll see if we can
build an LDAP enabled Xeon Phi stack, if we can then we we'll see if
we can get standard Slurm going too. Nothing like having lofty goals!
All the best!
Chris
- --
Christopher Samuel Senior Systems Administrator
VLSCI - Victorian Life Sciences Computation Initiative
http://www.vlsci.org.au/ http://twitter.com/vlsci
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.14 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iEYEARECAAYFAlMCuvIACgkQO2KABBYQAh+pwgCcCLPvoUJamArfmpxY5igcJm3I
0p0AnjF51qUgZfoZtIsKTDLCK+pJe+bf
=7HO3
-----END PGP SIGNATURE-----
Christopher Samuel
2014-02-18 02:30:50 UTC
Permalink
Hi Ralph,
Post by Ralph Castain
I know others have direct-launched processes onto the Phi before,
both with Slurm and just using rsh/ssh. The OpenMPI user mailing
list archive talks about the ssh method (search for "phi" and
you'll see the chatter)
Yeah, we know of that one (OPL has a nice micrun command he uses at
OSC), but what I'm after is ideally a way to deal with them as
separate nodes in their own partition using PAM to prevent access
unless you've got a job on the Phi (and not necessarily the host).

Of course that won't help the case where you may want to treat them as
offload devices as well and have Slurm intelligently juggle them so
they don't get overcommitted, which is what would be ideal.

cheers!
Chris
- --
Christopher Samuel Senior Systems Administrator
VLSCI - Victorian Life Sciences Computation Initiative
Email: samuel-***@public.gmane.org Phone: +61 (0)3 903 55545
http://www.vlsci.org.au/ http://twitter.com/vlsci
Olli-Pekka Lehto
2014-02-18 09:39:24 UTC
Permalink
Getting the daemon running on the Phi is certainly possible and we tried this a year or
so ago. The real challenge lies in being able to run offload, host-native, symmetric and
Phi-native mode programs nicely on the same set of nodes. It is something that would
really be needed in order to maximize utilization of a Phi-cluster.

Furthermore, having a simple way to maintain affinity of Phi cards with their associated
hosts when doing symmetric runs would be very useful. It's sort of possible with the topology
plugin but a bit clunky.

Also it would be nice to have a more lightweight daemon in order to conserve the precious
resources of the Phi as was presented in the Slurm User Group presentation. I expect that
this would be a more significant undertaking, however(?)

I'm wondering if people have been working on these kind of things?

Olli-Pekka
Post by Ralph Castain
I know others have direct-launched processes onto the Phi before, both with Slurm and just using rsh/ssh. The OpenMPI user mailing list archive talks about the ssh method (search for "phi" and you'll see the chatter)
http://www.open-mpi.org/community/lists/users/
https://www.linkedin.com/groups/Yes-we-run-SLURM-inside-4501392.S.5792769036550955008
Ralph
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi all,
At the Slurm User Group in Oakland last year it was mentioned that
there was intended to be support for a lightweight Slurm daemon on
Xeon Phi (MIC) cards.
I had a quick look in the git master last night but couldn't spot
anything related, is this still the intention?
Olli-Pekka Lehto from CSC is running a Xeon Phi workshop at VLSCI at
the moment and it's of interest to a number of us.
We're going to run a hack day on Wednesday and we'll see if we can
build an LDAP enabled Xeon Phi stack, if we can then we we'll see if
we can get standard Slurm going too. Nothing like having lofty goals!
All the best!
Chris
- --
Christopher Samuel Senior Systems Administrator
VLSCI - Victorian Life Sciences Computation Initiative
http://www.vlsci.org.au/ http://twitter.com/vlsci
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.14 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iEYEARECAAYFAlMCuvIACgkQO2KABBYQAh+pwgCcCLPvoUJamArfmpxY5igcJm3I
0p0AnjF51qUgZfoZtIsKTDLCK+pJe+bf
=7HO3
-----END PGP SIGNATURE-----
Paddy Doyle
2014-03-05 10:57:26 UTC
Permalink
Hi all,

Just wondering if there has been any developments regarding Phi cards?

We have just installed a small 10-node cluster with two MICs per node, and are
wondering how best to use the cards.

I intend to try the cross-compilation to install slurm on the cards, and then
have a separate queue, similar to that described in the discussion on linkedin
below.

But I think the users would also like to be able to run offload as well, so
that's a bit of an issue.

I have an additional question: the users have asked if a portion of the host
memory can be ``reserved'' for comms with the MICs, with the rest available for
whoever runs on the host. Is that possible??

Thanks,
Paddy
Post by Olli-Pekka Lehto
Getting the daemon running on the Phi is certainly possible and we tried this a year or
so ago. The real challenge lies in being able to run offload, host-native, symmetric and
Phi-native mode programs nicely on the same set of nodes. It is something that would
really be needed in order to maximize utilization of a Phi-cluster.
Furthermore, having a simple way to maintain affinity of Phi cards with their associated
hosts when doing symmetric runs would be very useful. It's sort of possible with the topology
plugin but a bit clunky.
Also it would be nice to have a more lightweight daemon in order to conserve the precious
resources of the Phi as was presented in the Slurm User Group presentation. I expect that
this would be a more significant undertaking, however(?)
I'm wondering if people have been working on these kind of things?
Olli-Pekka
Post by Ralph Castain
I know others have direct-launched processes onto the Phi before, both with Slurm and just using rsh/ssh. The OpenMPI user mailing list archive talks about the ssh method (search for "phi" and you'll see the chatter)
http://www.open-mpi.org/community/lists/users/
https://www.linkedin.com/groups/Yes-we-run-SLURM-inside-4501392.S.5792769036550955008
Ralph
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi all,
At the Slurm User Group in Oakland last year it was mentioned that
there was intended to be support for a lightweight Slurm daemon on
Xeon Phi (MIC) cards.
I had a quick look in the git master last night but couldn't spot
anything related, is this still the intention?
Olli-Pekka Lehto from CSC is running a Xeon Phi workshop at VLSCI at
the moment and it's of interest to a number of us.
We're going to run a hack day on Wednesday and we'll see if we can
build an LDAP enabled Xeon Phi stack, if we can then we we'll see if
we can get standard Slurm going too. Nothing like having lofty goals!
All the best!
Chris
- --
Christopher Samuel Senior Systems Administrator
VLSCI - Victorian Life Sciences Computation Initiative
http://www.vlsci.org.au/ http://twitter.com/vlsci
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.14 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iEYEARECAAYFAlMCuvIACgkQO2KABBYQAh+pwgCcCLPvoUJamArfmpxY5igcJm3I
0p0AnjF51qUgZfoZtIsKTDLCK+pJe+bf
=7HO3
-----END PGP SIGNATURE-----
--
Paddy Doyle
Trinity Centre for High Performance Computing,
Lloyd Building, Trinity College Dublin, Dublin 2, Ireland.
Phone: +353-1-896-3725
http://www.tchpc.tcd.ie/
Pardo Diaz, Alfonso
2014-06-09 12:30:33 UTC
Permalink
Hello!!

Would you mind posting here the steps to slurm cross-compile to xeon phi or any URL? In the linkedin discussion isn't describe the procedure


Thanks in advance
Post by Christopher Samuel
Hi all,
Just wondering if there has been any developments regarding Phi cards?
We have just installed a small 10-node cluster with two MICs per node, and are
wondering how best to use the cards.
I intend to try the cross-compilation to install slurm on the cards, and then
have a separate queue, similar to that described in the discussion on linkedin
below.
But I think the users would also like to be able to run offload as well, so
that's a bit of an issue.
I have an additional question: the users have asked if a portion of the host
memory can be ``reserved'' for comms with the MICs, with the rest available for
whoever runs on the host. Is that possible??
Thanks,
Paddy
Post by Olli-Pekka Lehto
Getting the daemon running on the Phi is certainly possible and we tried this a year or
so ago. The real challenge lies in being able to run offload, host-native, symmetric and
Phi-native mode programs nicely on the same set of nodes. It is something that would
really be needed in order to maximize utilization of a Phi-cluster.
Furthermore, having a simple way to maintain affinity of Phi cards with their associated
hosts when doing symmetric runs would be very useful. It's sort of possible with the topology
plugin but a bit clunky.
Also it would be nice to have a more lightweight daemon in order to conserve the precious
resources of the Phi as was presented in the Slurm User Group presentation. I expect that
this would be a more significant undertaking, however(?)
I'm wondering if people have been working on these kind of things?
Olli-Pekka
Post by Ralph Castain
I know others have direct-launched processes onto the Phi before, both with Slurm and just using rsh/ssh. The OpenMPI user mailing list archive talks about the ssh method (search for "phi" and you'll see the chatter)
http://www.open-mpi.org/community/lists/users/
https://www.linkedin.com/groups/Yes-we-run-SLURM-inside-4501392.S.5792769036550955008
Ralph
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi all,
At the Slurm User Group in Oakland last year it was mentioned that
there was intended to be support for a lightweight Slurm daemon on
Xeon Phi (MIC) cards.
I had a quick look in the git master last night but couldn't spot
anything related, is this still the intention?
Olli-Pekka Lehto from CSC is running a Xeon Phi workshop at VLSCI at
the moment and it's of interest to a number of us.
We're going to run a hack day on Wednesday and we'll see if we can
build an LDAP enabled Xeon Phi stack, if we can then we we'll see if
we can get standard Slurm going too. Nothing like having lofty goals!
All the best!
Chris
- --
Christopher Samuel Senior Systems Administrator
VLSCI - Victorian Life Sciences Computation Initiative
http://www.vlsci.org.au/ http://twitter.com/vlsci
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.14 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iEYEARECAAYFAlMCuvIACgkQO2KABBYQAh+pwgCcCLPvoUJamArfmpxY5igcJm3I
0p0AnjF51qUgZfoZtIsKTDLCK+pJe+bf
=7HO3
-----END PGP SIGNATURE-----
--
Paddy Doyle
Trinity Centre for High Performance Computing,
Lloyd Building, Trinity College Dublin, Dublin 2, Ireland.
Phone: +353-1-896-3725
http://www.tchpc.tcd.ie/
Paddy Doyle
2014-06-27 09:06:38 UTC
Permalink
Hi Pardo,

Sorry for the lack of reply.. I've been away etc.

See below for the notes I took while compiling. I relied on a lot of advice from
Olli-Pekka Lehto, in particular the binfmt_misc hack to allow some of the
./configure checks to be executed directly, on the Phi cards, even while
compiling on the host.

It also needed some hacking of the configure/autoconf scripts for munge and
slurm.

Note that I had to cross-compile munge and openssl as well, for our setup. I
don't know if people would need to do that in general.

Versions (a bit old by now):
slurm: 2.6.5
munge: 0.5.10
openssl: 1.0.0
mpss: 3.1.1



Hope that helps!

I'd still really like to know if anyone has a good (slurm) solution for handling
the queuing of the many modes of MIC/Phi execution: native, offload, symmetric
with Intel MPI ?

Of course our users want to be able to mix all 3 modes, but I'm struggling to
see how that can be done within the confines of the queuing system.

Paddy





#######################################################################
# working finally! use the config/make lines below
#######################################################################

# used the binfmt_misc hack, *** while on the host ***
# https://software.intel.com/en-us/forums/topic/391645#comment-1734339


###############
# Openssl 1.0.0
###############
./Configure --prefix=/home/support/native-mic/install --cross-compile-prefix=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux- enable-camellia enable-seed enable-tlsext enable-rfc3779 enable-cms enable-md2 no-zlib shared linux-generic64
make depend
make
make install


###############
# Munge
###############
# edit the configure to change the LD-ld detection from "-m elf_x86_64"
# to the empty string ""
#env CC=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-gcc LD=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-ld CXX=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-g++ ./configure --prefix=/home/support/native-mic/install --with-openssl-prefix=/home/support/native-mic/install --host=x86_64-k1om-linux
#env CC=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-gcc LD=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-ld CXX=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-g++ NM=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-nm ./configure --prefix=/home/support/native-mic/install --with-openssl-prefix=/home/support/native-mic/install --host=x86_64-k1om-linux --with-gnu-ld
env CC=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-gcc LD=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-ld CXX=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-g++ NM=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-nm ./configure --prefix=/home/support/native-mic/install --localstatedir=/var --with-openssl-prefix=/home/support/native-mic/install --host=x86_64-k1om-linux --with-gnu-ld
make
make install


###############
# Slurm
###############
# disable the printf NULL check in configure.ac and run 'autoconf' to create a
# new 'configure'
# then edit the configure to change the LD-ld detection from "-m elf_x86_64"
# to the empty string ""
# then the ./configure line below will fail because it tells us it can't run a
# test program while cross-compiling -- edit the configure script and force it
# to run the openssl test, then the ./configure should finish
env CC=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-gcc LD=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-ld CXX=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-g++ ./configure --prefix=/home/support/native-mic/install --sysconfdir=/home/support/native-mic/install/etc/slurm --disable-salloc-background --without-readline --enable-pam -with-ssl=/home/support/native-mic/install --with-munge=/home/support/native-mic/install --host=x86_64-k1om-linux
make
make install
Post by Pardo Diaz, Alfonso
Hello!!
Would you mind posting here the steps to slurm cross-compile to xeon phi or any URL? In the linkedin discussion isn't describe the procedure
Thanks in advance
Post by Christopher Samuel
Hi all,
Just wondering if there has been any developments regarding Phi cards?
We have just installed a small 10-node cluster with two MICs per node, and are
wondering how best to use the cards.
I intend to try the cross-compilation to install slurm on the cards, and then
have a separate queue, similar to that described in the discussion on linkedin
below.
But I think the users would also like to be able to run offload as well, so
that's a bit of an issue.
I have an additional question: the users have asked if a portion of the host
memory can be ``reserved'' for comms with the MICs, with the rest available for
whoever runs on the host. Is that possible??
Thanks,
Paddy
Post by Olli-Pekka Lehto
Getting the daemon running on the Phi is certainly possible and we tried this a year or
so ago. The real challenge lies in being able to run offload, host-native, symmetric and
Phi-native mode programs nicely on the same set of nodes. It is something that would
really be needed in order to maximize utilization of a Phi-cluster.
Furthermore, having a simple way to maintain affinity of Phi cards with their associated
hosts when doing symmetric runs would be very useful. It's sort of possible with the topology
plugin but a bit clunky.
Also it would be nice to have a more lightweight daemon in order to conserve the precious
resources of the Phi as was presented in the Slurm User Group presentation. I expect that
this would be a more significant undertaking, however(?)
I'm wondering if people have been working on these kind of things?
Olli-Pekka
Post by Ralph Castain
I know others have direct-launched processes onto the Phi before, both with Slurm and just using rsh/ssh. The OpenMPI user mailing list archive talks about the ssh method (search for "phi" and you'll see the chatter)
http://www.open-mpi.org/community/lists/users/
https://www.linkedin.com/groups/Yes-we-run-SLURM-inside-4501392.S.5792769036550955008
Ralph
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi all,
At the Slurm User Group in Oakland last year it was mentioned that
there was intended to be support for a lightweight Slurm daemon on
Xeon Phi (MIC) cards.
I had a quick look in the git master last night but couldn't spot
anything related, is this still the intention?
Olli-Pekka Lehto from CSC is running a Xeon Phi workshop at VLSCI at
the moment and it's of interest to a number of us.
We're going to run a hack day on Wednesday and we'll see if we can
build an LDAP enabled Xeon Phi stack, if we can then we we'll see if
we can get standard Slurm going too. Nothing like having lofty goals!
All the best!
Chris
- --
Christopher Samuel Senior Systems Administrator
VLSCI - Victorian Life Sciences Computation Initiative
http://www.vlsci.org.au/ http://twitter.com/vlsci
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.14 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iEYEARECAAYFAlMCuvIACgkQO2KABBYQAh+pwgCcCLPvoUJamArfmpxY5igcJm3I
0p0AnjF51qUgZfoZtIsKTDLCK+pJe+bf
=7HO3
-----END PGP SIGNATURE-----
--
Paddy Doyle
Trinity Centre for High Performance Computing,
Lloyd Building, Trinity College Dublin, Dublin 2, Ireland.
Phone: +353-1-896-3725
http://www.tchpc.tcd.ie/
--
Paddy Doyle
Trinity Centre for High Performance Computing,
Lloyd Building, Trinity College Dublin, Dublin 2, Ireland.
Phone: +353-1-896-3725
http://www.tchpc.tcd.ie/
Bill Barth
2014-06-27 11:53:39 UTC
Permalink
Paddy,

Are you trying to allow one user to use a MIC on a node while a different
user works on the host?

We don't allow this at TACC, and as such, we don't do anything special
when it comes to scheduling on the MIC. There's basically no way to
prevent a user from running offload code, so I think you'd be hard pressed
to schedule around the situation where one user wants to run native and
another says they're running host-only but actually runs an offload code.

Maybe you have something else in mind?

Best,
Bill.
--
Bill Barth, Ph.D., Director, HPC
bbarth-***@public.gmane.org | Phone: (512) 232-7069
Office: ROC 1.435 | Fax: (512) 475-9445
Post by Paddy Doyle
Hi Pardo,
Sorry for the lack of reply.. I've been away etc.
See below for the notes I took while compiling. I relied on a lot of advice from
Olli-Pekka Lehto, in particular the binfmt_misc hack to allow some of the
./configure checks to be executed directly, on the Phi cards, even while
compiling on the host.
It also needed some hacking of the configure/autoconf scripts for munge and
slurm.
Note that I had to cross-compile munge and openssl as well, for our setup. I
don't know if people would need to do that in general.
slurm: 2.6.5
munge: 0.5.10
openssl: 1.0.0
mpss: 3.1.1
Hope that helps!
I'd still really like to know if anyone has a good (slurm) solution for handling
the queuing of the many modes of MIC/Phi execution: native, offload, symmetric
with Intel MPI ?
Of course our users want to be able to mix all 3 modes, but I'm
struggling to
see how that can be done within the confines of the queuing system.
Paddy
#######################################################################
# working finally! use the config/make lines below
#######################################################################
# used the binfmt_misc hack, *** while on the host ***
# https://software.intel.com/en-us/forums/topic/391645#comment-1734339
###############
# Openssl 1.0.0
###############
./Configure --prefix=/home/support/native-mic/install
--cross-compile-prefix=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-
enable-camellia enable-seed enable-tlsext enable-rfc3779 enable-cms
enable-md2 no-zlib shared linux-generic64
make depend
make
make install
###############
# Munge
###############
# edit the configure to change the LD-ld detection from "-m elf_x86_64"
# to the empty string ""
#env CC=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-gcc
LD=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-ld
CXX=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-g++ ./configure
--prefix=/home/support/native-mic/install
--with-openssl-prefix=/home/support/native-mic/install
--host=x86_64-k1om-linux
#env CC=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-gcc
LD=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-ld
CXX=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-g++
NM=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-nm ./configure
--prefix=/home/support/native-mic/install
--with-openssl-prefix=/home/support/native-mic/install
--host=x86_64-k1om-linux --with-gnu-ld
env CC=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-gcc
LD=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-ld
CXX=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-g++
NM=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-nm ./configure
--prefix=/home/support/native-mic/install --localstatedir=/var
--with-openssl-prefix=/home/support/native-mic/install
--host=x86_64-k1om-linux --with-gnu-ld
make
make install
###############
# Slurm
###############
# disable the printf NULL check in configure.ac and run 'autoconf' to create a
# new 'configure'
# then edit the configure to change the LD-ld detection from "-m elf_x86_64"
# to the empty string ""
# then the ./configure line below will fail because it tells us it can't run a
# test program while cross-compiling -- edit the configure script and force it
# to run the openssl test, then the ./configure should finish
env CC=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-gcc
LD=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-ld
CXX=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-g++ ./configure
--prefix=/home/support/native-mic/install
--sysconfdir=/home/support/native-mic/install/etc/slurm
--disable-salloc-background --without-readline --enable-pam
-with-ssl=/home/support/native-mic/install
--with-munge=/home/support/native-mic/install --host=x86_64-k1om-linux
make
make install
Post by Pardo Diaz, Alfonso
Hello!!
Would you mind posting here the steps to slurm cross-compile to xeon
phi or any URL? In the linkedin discussion isn't describe the procedure
Thanks in advance
Post by Christopher Samuel
Hi all,
Just wondering if there has been any developments regarding Phi cards?
We have just installed a small 10-node cluster with two MICs per
node, and are
Post by Christopher Samuel
wondering how best to use the cards.
I intend to try the cross-compilation to install slurm on the cards,
and then
Post by Christopher Samuel
have a separate queue, similar to that described in the discussion on
linkedin
Post by Christopher Samuel
below.
But I think the users would also like to be able to run offload as
well, so
Post by Christopher Samuel
that's a bit of an issue.
I have an additional question: the users have asked if a portion of
the host
Post by Christopher Samuel
memory can be ``reserved'' for comms with the MICs, with the rest
available for
Post by Christopher Samuel
whoever runs on the host. Is that possible??
Thanks,
Paddy
Post by Olli-Pekka Lehto
Getting the daemon running on the Phi is certainly possible and we
tried this a year or
Post by Christopher Samuel
Post by Olli-Pekka Lehto
so ago. The real challenge lies in being able to run offload,
host-native, symmetric and
Post by Christopher Samuel
Post by Olli-Pekka Lehto
Phi-native mode programs nicely on the same set of nodes. It is
something that would
Post by Christopher Samuel
Post by Olli-Pekka Lehto
really be needed in order to maximize utilization of a Phi-cluster.
Furthermore, having a simple way to maintain affinity of Phi cards
with their associated
Post by Christopher Samuel
Post by Olli-Pekka Lehto
hosts when doing symmetric runs would be very useful. It's sort of
possible with the topology
Post by Christopher Samuel
Post by Olli-Pekka Lehto
plugin but a bit clunky.
Also it would be nice to have a more lightweight daemon in order to
conserve the precious
Post by Christopher Samuel
Post by Olli-Pekka Lehto
resources of the Phi as was presented in the Slurm User Group
presentation. I expect that
Post by Christopher Samuel
Post by Olli-Pekka Lehto
this would be a more significant undertaking, however(?)
I'm wondering if people have been working on these kind of things?
Olli-Pekka
Post by Ralph Castain
I know others have direct-launched processes onto the Phi before,
both with Slurm and just using rsh/ssh. The OpenMPI user mailing list
archive talks about the ssh method (search for "phi" and you'll see the
chatter)
Post by Christopher Samuel
Post by Olli-Pekka Lehto
Post by Ralph Castain
http://www.open-mpi.org/community/lists/users/
https://www.linkedin.com/groups/Yes-we-run-SLURM-inside-4501392.S.5792769
036550955008
Post by Christopher Samuel
Post by Olli-Pekka Lehto
Post by Ralph Castain
Ralph
On Feb 17, 2014, at 5:46 PM, Christopher Samuel
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi all,
At the Slurm User Group in Oakland last year it was mentioned that
there was intended to be support for a lightweight Slurm daemon on
Xeon Phi (MIC) cards.
I had a quick look in the git master last night but couldn't spot
anything related, is this still the intention?
Olli-Pekka Lehto from CSC is running a Xeon Phi workshop at VLSCI
at
Post by Christopher Samuel
Post by Olli-Pekka Lehto
Post by Ralph Castain
the moment and it's of interest to a number of us.
We're going to run a hack day on Wednesday and we'll see if we can
build an LDAP enabled Xeon Phi stack, if we can then we we'll see
if
Post by Christopher Samuel
Post by Olli-Pekka Lehto
Post by Ralph Castain
we can get standard Slurm going too. Nothing like having lofty
goals!
Post by Christopher Samuel
Post by Olli-Pekka Lehto
Post by Ralph Castain
All the best!
Chris
- --
Christopher Samuel Senior Systems Administrator
VLSCI - Victorian Life Sciences Computation Initiative
http://www.vlsci.org.au/ http://twitter.com/vlsci
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.14 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iEYEARECAAYFAlMCuvIACgkQO2KABBYQAh+pwgCcCLPvoUJamArfmpxY5igcJm3I
0p0AnjF51qUgZfoZtIsKTDLCK+pJe+bf
=7HO3
-----END PGP SIGNATURE-----
--
Paddy Doyle
Trinity Centre for High Performance Computing,
Lloyd Building, Trinity College Dublin, Dublin 2, Ireland.
Phone: +353-1-896-3725
http://www.tchpc.tcd.ie/
--
Paddy Doyle
Trinity Centre for High Performance Computing,
Lloyd Building, Trinity College Dublin, Dublin 2, Ireland.
Phone: +353-1-896-3725
http://www.tchpc.tcd.ie/
Taras Shapovalov
2014-06-27 14:07:02 UTC
Permalink
Hi guys,

Paddy, we are doing the same thing to run Slurm/munge on MICs and I can
confirm this works fine also for the latest Slurm version.

The problem with "mixed modes scheduling" have been known now more then an
year to the Slurm developers and according what I've heard on ISC'14 the
Slurm developers are working now on a proper solution. It would be nice if
somebody confirms this.

Probably the fix is not so simple as could look like, ok, but I agree this
should be fixed sooner or later. For now Slurm on a quite beneficial
position because people are already using it inside MICs (but struggling)
and no one workload manager supports mixed modes (for now).
There's basically no way to prevent a user from running offload code
Bill, there is at least one way: you can restart coi_daemon with
permissions of user that uses the MIC directly or just stop it (in prolog
of a native job), but obviously this solution is not perfect. Maybe there
are some other ways to prevent using MICs in offloading mode, it would be
good to know.

But the problem not only how to prevent user job to use the mic in
offloading mode but also in fact that when mic0 (installed on node001) is
allocated by job1, then Slurm scheduler will not know that job2 should not
be started on node001 if it requests 1 mic gres (if we have 1 mic per node).

Best regards,
Taras
Paddy,
Are you trying to allow one user to use a MIC on a node while a different
user works on the host?
We don't allow this at TACC, and as such, we don't do anything special
when it comes to scheduling on the MIC. There's basically no way to
prevent a user from running offload code, so I think you'd be hard pressed
to schedule around the situation where one user wants to run native and
another says they're running host-only but actually runs an offload code.
Maybe you have something else in mind?
Best,
Bill.
--
Bill Barth, Ph.D., Director, HPC
Office: ROC 1.435 | Fax: (512) 475-9445
Post by Paddy Doyle
Hi Pardo,
Sorry for the lack of reply.. I've been away etc.
See below for the notes I took while compiling. I relied on a lot of advice from
Olli-Pekka Lehto, in particular the binfmt_misc hack to allow some of the
./configure checks to be executed directly, on the Phi cards, even while
compiling on the host.
It also needed some hacking of the configure/autoconf scripts for munge and
slurm.
Note that I had to cross-compile munge and openssl as well, for our setup. I
don't know if people would need to do that in general.
slurm: 2.6.5
munge: 0.5.10
openssl: 1.0.0
mpss: 3.1.1
Hope that helps!
I'd still really like to know if anyone has a good (slurm) solution for handling
the queuing of the many modes of MIC/Phi execution: native, offload, symmetric
with Intel MPI ?
Of course our users want to be able to mix all 3 modes, but I'm struggling to
see how that can be done within the confines of the queuing system.
Paddy
#######################################################################
# working finally! use the config/make lines below
#######################################################################
# used the binfmt_misc hack, *** while on the host ***
# https://software.intel.com/en-us/forums/topic/391645#comment-1734339
###############
# Openssl 1.0.0
###############
./Configure --prefix=/home/support/native-mic/install
--cross-compile-prefix=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-
enable-camellia enable-seed enable-tlsext enable-rfc3779 enable-cms
enable-md2 no-zlib shared linux-generic64
make depend
make
make install
###############
# Munge
###############
# edit the configure to change the LD-ld detection from "-m elf_x86_64"
# to the empty string ""
#env CC=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-gcc
LD=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-ld
CXX=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-g++ ./configure
--prefix=/home/support/native-mic/install
--with-openssl-prefix=/home/support/native-mic/install
--host=x86_64-k1om-linux
#env CC=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-gcc
LD=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-ld
CXX=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-g++
NM=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-nm ./configure
--prefix=/home/support/native-mic/install
--with-openssl-prefix=/home/support/native-mic/install
--host=x86_64-k1om-linux --with-gnu-ld
env CC=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-gcc
LD=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-ld
CXX=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-g++
NM=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-nm ./configure
--prefix=/home/support/native-mic/install --localstatedir=/var
--with-openssl-prefix=/home/support/native-mic/install
--host=x86_64-k1om-linux --with-gnu-ld
make
make install
###############
# Slurm
###############
# disable the printf NULL check in configure.ac and run 'autoconf' to create a
# new 'configure'
# then edit the configure to change the LD-ld detection from "-m elf_x86_64"
# to the empty string ""
# then the ./configure line below will fail because it tells us it can't run a
# test program while cross-compiling -- edit the configure script and force it
# to run the openssl test, then the ./configure should finish
env CC=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-gcc
LD=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-ld
CXX=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-g++ ./configure
--prefix=/home/support/native-mic/install
--sysconfdir=/home/support/native-mic/install/etc/slurm
--disable-salloc-background --without-readline --enable-pam
-with-ssl=/home/support/native-mic/install
--with-munge=/home/support/native-mic/install --host=x86_64-k1om-linux
make
make install
Post by Pardo Diaz, Alfonso
Hello!!
Would you mind posting here the steps to slurm cross-compile to xeon
phi or any URL? In the linkedin discussion isn't describe the procedure
Thanks in advance
Post by Christopher Samuel
Hi all,
Just wondering if there has been any developments regarding Phi cards?
We have just installed a small 10-node cluster with two MICs per
node, and are
Post by Christopher Samuel
wondering how best to use the cards.
I intend to try the cross-compilation to install slurm on the cards,
and then
Post by Christopher Samuel
have a separate queue, similar to that described in the discussion on
linkedin
Post by Christopher Samuel
below.
But I think the users would also like to be able to run offload as
well, so
Post by Christopher Samuel
that's a bit of an issue.
I have an additional question: the users have asked if a portion of
the host
Post by Christopher Samuel
memory can be ``reserved'' for comms with the MICs, with the rest
available for
Post by Christopher Samuel
whoever runs on the host. Is that possible??
Thanks,
Paddy
Post by Olli-Pekka Lehto
Getting the daemon running on the Phi is certainly possible and we
tried this a year or
Post by Christopher Samuel
Post by Olli-Pekka Lehto
so ago. The real challenge lies in being able to run offload,
host-native, symmetric and
Post by Christopher Samuel
Post by Olli-Pekka Lehto
Phi-native mode programs nicely on the same set of nodes. It is
something that would
Post by Christopher Samuel
Post by Olli-Pekka Lehto
really be needed in order to maximize utilization of a Phi-cluster.
Furthermore, having a simple way to maintain affinity of Phi cards
with their associated
Post by Christopher Samuel
Post by Olli-Pekka Lehto
hosts when doing symmetric runs would be very useful. It's sort of
possible with the topology
Post by Christopher Samuel
Post by Olli-Pekka Lehto
plugin but a bit clunky.
Also it would be nice to have a more lightweight daemon in order to
conserve the precious
Post by Christopher Samuel
Post by Olli-Pekka Lehto
resources of the Phi as was presented in the Slurm User Group
presentation. I expect that
Post by Christopher Samuel
Post by Olli-Pekka Lehto
this would be a more significant undertaking, however(?)
I'm wondering if people have been working on these kind of things?
Olli-Pekka
Post by Ralph Castain
I know others have direct-launched processes onto the Phi before,
both with Slurm and just using rsh/ssh. The OpenMPI user mailing list
archive talks about the ssh method (search for "phi" and you'll see the
chatter)
Post by Christopher Samuel
Post by Olli-Pekka Lehto
Post by Ralph Castain
http://www.open-mpi.org/community/lists/users/
https://www.linkedin.com/groups/Yes-we-run-SLURM-inside-4501392.S.5792769
Post by Paddy Doyle
Post by Pardo Diaz, Alfonso
036550955008
Post by Christopher Samuel
Post by Olli-Pekka Lehto
Post by Ralph Castain
Ralph
On Feb 17, 2014, at 5:46 PM, Christopher Samuel
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi all,
At the Slurm User Group in Oakland last year it was mentioned that
there was intended to be support for a lightweight Slurm daemon on
Xeon Phi (MIC) cards.
I had a quick look in the git master last night but couldn't spot
anything related, is this still the intention?
Olli-Pekka Lehto from CSC is running a Xeon Phi workshop at VLSCI
at
Post by Christopher Samuel
Post by Olli-Pekka Lehto
Post by Ralph Castain
the moment and it's of interest to a number of us.
We're going to run a hack day on Wednesday and we'll see if we can
build an LDAP enabled Xeon Phi stack, if we can then we we'll see
if
Post by Christopher Samuel
Post by Olli-Pekka Lehto
Post by Ralph Castain
we can get standard Slurm going too. Nothing like having lofty
goals!
Post by Christopher Samuel
Post by Olli-Pekka Lehto
Post by Ralph Castain
All the best!
Chris
- --
Christopher Samuel Senior Systems Administrator
VLSCI - Victorian Life Sciences Computation Initiative
http://www.vlsci.org.au/ http://twitter.com/vlsci
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.14 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iEYEARECAAYFAlMCuvIACgkQO2KABBYQAh+pwgCcCLPvoUJamArfmpxY5igcJm3I
0p0AnjF51qUgZfoZtIsKTDLCK+pJe+bf
=7HO3
-----END PGP SIGNATURE-----
--
Paddy Doyle
Trinity Centre for High Performance Computing,
Lloyd Building, Trinity College Dublin, Dublin 2, Ireland.
Phone: +353-1-896-3725
http://www.tchpc.tcd.ie/
--
Paddy Doyle
Trinity Centre for High Performance Computing,
Lloyd Building, Trinity College Dublin, Dublin 2, Ireland.
Phone: +353-1-896-3725
http://www.tchpc.tcd.ie/
Hongjia Cao
2014-02-18 04:57:25 UTC
Permalink
I successfully compiled and ran the standard SLURM (2.6.0) on MIC
before. I used cross compiling.

在 2014-02-17一的 17:46 -0800,Christopher Samuel写道:
Post by Christopher Samuel
Hi all,
At the Slurm User Group in Oakland last year it was mentioned that
there was intended to be support for a lightweight Slurm daemon on
Xeon Phi (MIC) cards.
I had a quick look in the git master last night but couldn't spot
anything related, is this still the intention?
Olli-Pekka Lehto from CSC is running a Xeon Phi workshop at VLSCI at
the moment and it's of interest to a number of us.
We're going to run a hack day on Wednesday and we'll see if we can
build an LDAP enabled Xeon Phi stack, if we can then we we'll see if
we can get standard Slurm going too. Nothing like having lofty goals!
All the best!
Chris
Loading...