when it comes to scheduling on the MIC. There's basically no way to
another says they're running host-only but actually runs an offload code.
Bill.
Post by Paddy DoyleHi Pardo,
Sorry for the lack of reply.. I've been away etc.
See below for the notes I took while compiling. I relied on a lot of advice from
Olli-Pekka Lehto, in particular the binfmt_misc hack to allow some of the
./configure checks to be executed directly, on the Phi cards, even while
compiling on the host.
It also needed some hacking of the configure/autoconf scripts for munge and
slurm.
Note that I had to cross-compile munge and openssl as well, for our setup. I
don't know if people would need to do that in general.
slurm: 2.6.5
munge: 0.5.10
openssl: 1.0.0
mpss: 3.1.1
Hope that helps!
I'd still really like to know if anyone has a good (slurm) solution for handling
the queuing of the many modes of MIC/Phi execution: native, offload, symmetric
with Intel MPI ?
Of course our users want to be able to mix all 3 modes, but I'm
struggling to
see how that can be done within the confines of the queuing system.
Paddy
#######################################################################
# working finally! use the config/make lines below
#######################################################################
# used the binfmt_misc hack, *** while on the host ***
# https://software.intel.com/en-us/forums/topic/391645#comment-1734339
###############
# Openssl 1.0.0
###############
./Configure --prefix=/home/support/native-mic/install
--cross-compile-prefix=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-
enable-camellia enable-seed enable-tlsext enable-rfc3779 enable-cms
enable-md2 no-zlib shared linux-generic64
make depend
make
make install
###############
# Munge
###############
# edit the configure to change the LD-ld detection from "-m elf_x86_64"
# to the empty string ""
#env CC=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-gcc
LD=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-ld
CXX=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-g++ ./configure
--prefix=/home/support/native-mic/install
--with-openssl-prefix=/home/support/native-mic/install
--host=x86_64-k1om-linux
#env CC=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-gcc
LD=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-ld
CXX=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-g++
NM=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-nm ./configure
--prefix=/home/support/native-mic/install
--with-openssl-prefix=/home/support/native-mic/install
--host=x86_64-k1om-linux --with-gnu-ld
env CC=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-gcc
LD=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-ld
CXX=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-g++
NM=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-nm ./configure
--prefix=/home/support/native-mic/install --localstatedir=/var
--with-openssl-prefix=/home/support/native-mic/install
--host=x86_64-k1om-linux --with-gnu-ld
make
make install
###############
# Slurm
###############
# disable the printf NULL check in configure.ac and run 'autoconf' to create a
# new 'configure'
# then edit the configure to change the LD-ld detection from "-m elf_x86_64"
# to the empty string ""
# then the ./configure line below will fail because it tells us it can't run a
# test program while cross-compiling -- edit the configure script and force it
# to run the openssl test, then the ./configure should finish
env CC=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-gcc
LD=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-ld
CXX=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-g++ ./configure
--prefix=/home/support/native-mic/install
--sysconfdir=/home/support/native-mic/install/etc/slurm
--disable-salloc-background --without-readline --enable-pam
-with-ssl=/home/support/native-mic/install
--with-munge=/home/support/native-mic/install --host=x86_64-k1om-linux
make
make install
Post by Pardo Diaz, AlfonsoHello!!
Would you mind posting here the steps to slurm cross-compile to xeon
phi or any URL? In the linkedin discussion isn't describe the procedure
Thanks in advance
Post by Christopher SamuelHi all,
Just wondering if there has been any developments regarding Phi cards?
We have just installed a small 10-node cluster with two MICs per
node, and are
Post by Christopher Samuelwondering how best to use the cards.
I intend to try the cross-compilation to install slurm on the cards,
and then
Post by Christopher Samuelhave a separate queue, similar to that described in the discussion on
linkedin
Post by Christopher Samuelbelow.
But I think the users would also like to be able to run offload as
well, so
Post by Christopher Samuelthat's a bit of an issue.
I have an additional question: the users have asked if a portion of
the host
Post by Christopher Samuelmemory can be ``reserved'' for comms with the MICs, with the rest
available for
Post by Christopher Samuelwhoever runs on the host. Is that possible??
Thanks,
Paddy
Post by Olli-Pekka LehtoGetting the daemon running on the Phi is certainly possible and we
tried this a year or
Post by Christopher SamuelPost by Olli-Pekka Lehtoso ago. The real challenge lies in being able to run offload,
host-native, symmetric and
Post by Christopher SamuelPost by Olli-Pekka LehtoPhi-native mode programs nicely on the same set of nodes. It is
something that would
Post by Christopher SamuelPost by Olli-Pekka Lehtoreally be needed in order to maximize utilization of a Phi-cluster.
Furthermore, having a simple way to maintain affinity of Phi cards
with their associated
Post by Christopher SamuelPost by Olli-Pekka Lehtohosts when doing symmetric runs would be very useful. It's sort of
possible with the topology
Post by Christopher SamuelPost by Olli-Pekka Lehtoplugin but a bit clunky.
Also it would be nice to have a more lightweight daemon in order to
conserve the precious
Post by Christopher SamuelPost by Olli-Pekka Lehtoresources of the Phi as was presented in the Slurm User Group
presentation. I expect that
Post by Christopher SamuelPost by Olli-Pekka Lehtothis would be a more significant undertaking, however(?)
I'm wondering if people have been working on these kind of things?
Olli-Pekka
Post by Ralph CastainI know others have direct-launched processes onto the Phi before,
both with Slurm and just using rsh/ssh. The OpenMPI user mailing list
archive talks about the ssh method (search for "phi" and you'll see the
chatter)
https://www.linkedin.com/groups/Yes-we-run-SLURM-inside-4501392.S.5792769
036550955008
Post by Christopher SamuelPost by Olli-Pekka LehtoPost by Ralph CastainRalph
On Feb 17, 2014, at 5:46 PM, Christopher Samuel
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi all,
At the Slurm User Group in Oakland last year it was mentioned that
there was intended to be support for a lightweight Slurm daemon on
Xeon Phi (MIC) cards.
I had a quick look in the git master last night but couldn't spot
anything related, is this still the intention?
Olli-Pekka Lehto from CSC is running a Xeon Phi workshop at VLSCI
at
Post by Christopher SamuelPost by Olli-Pekka LehtoPost by Ralph Castainthe moment and it's of interest to a number of us.
We're going to run a hack day on Wednesday and we'll see if we can
build an LDAP enabled Xeon Phi stack, if we can then we we'll see
if
goals!
Post by Christopher SamuelPost by Olli-Pekka LehtoPost by Ralph CastainAll the best!
Chris
- --
Christopher Samuel Senior Systems Administrator
VLSCI - Victorian Life Sciences Computation Initiative
http://www.vlsci.org.au/ http://twitter.com/vlsci
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.14 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iEYEARECAAYFAlMCuvIACgkQO2KABBYQAh+pwgCcCLPvoUJamArfmpxY5igcJm3I
0p0AnjF51qUgZfoZtIsKTDLCK+pJe+bf
=7HO3
-----END PGP SIGNATURE-----
--
Paddy Doyle
Trinity Centre for High Performance Computing,
Lloyd Building, Trinity College Dublin, Dublin 2, Ireland.
Phone: +353-1-896-3725
http://www.tchpc.tcd.ie/
--
Paddy Doyle
Trinity Centre for High Performance Computing,
Lloyd Building, Trinity College Dublin, Dublin 2, Ireland.
Phone: +353-1-896-3725
http://www.tchpc.tcd.ie/