SLURM installing into $prefix/lib and $prefix/lib64?

[Jeff Squyres (jsquyres)]

We got a report in OMPI the other day that reported by Timothy Brown (see http://www.open-mpi.org/community/lists/users/2014/10/25467.php) that SLURM is installing things into both $prefix/lib and $prefix/lib64.

Can the SLURM developers confirm that this is correct?

I'm a little confused because all prior versions of OMPI have worked fine (i.e., found the Right SLURM libraries), but now there's apparently an issue with SLURM 14.03.7. Did something change in the installation location of perl modules and libraries in this version of SLURM?

If so, can it be changed (back?) such that SLURM only installs into $prefix/lib *or* $prefix/lib64? It's somewhat unexpected behavior to install into *both* lib and lib64.

Thanks!

[Bruce Roberts]

The perl libs install in the perl installarchlib it appears from reading
the slurm.spec and the rest of the libs install in $prefix/lib or lib64
based on your arch. These can all be altered at configure/rpmbuild time.

On a normal 64bit system this is what I get with the latest 14.03...

rpm -qpl slurm-plugins-14.03.8-1.x86_64.rpm
/usr/lib64/slurm
/usr/lib64/slurm/accounting_storage_filetxt.so
/usr/lib64/slurm/accounting_storage_none.so
/usr/lib64/slurm/accounting_storage_slurmdbd.so
/usr/lib64/slurm/acct_gather_energy_ipmi.so
...

rpm -qpl slurm-14.03.8-1.x86_64.rpm
...
/usr/lib64/libpmi.so
/usr/lib64/libpmi.so.0
/usr/lib64/libpmi.so.0.0.0
/usr/lib64/libpmi2.so
/usr/lib64/libpmi2.so.0
/usr/lib64/libpmi2.so.0.0.0
/usr/lib64/libslurm.so
/usr/lib64/libslurm.so.27
/usr/lib64/libslurm.so.27.0.0
/usr/lib64/libslurmdb.so
/usr/lib64/libslurmdb.so.27
/usr/lib64/libslurmdb.so.27.0.0

rpm -qpl slurm-perlapi-14.03.8-1.x86_64.rpm
/usr/lib/perl/5.18.2/Slurm.pm
/usr/lib/perl/5.18.2/Slurm/Bitstr.pm
/usr/lib/perl/5.18.2/Slurm/Constant.pm
/usr/lib/perl/5.18.2/Slurm/Hostlist.pm
/usr/lib/perl/5.18.2/Slurm/Stepctx.pm
/usr/lib/perl/5.18.2/Slurmdb.pm
/usr/lib/perl/5.18.2/auto/Slurm/Slurm.so
/usr/lib/perl/5.18.2/auto/Slurmdb/Slurmdb.so
/usr/lib/perl/5.18.2/auto/Slurmdb/autosplit.ix

Post by Jeff Squyres (jsquyres)
We got a report in OMPI the other day that reported by Timothy Brown (see http://www.open-mpi.org/community/lists/users/2014/10/25467.php) that SLURM is installing things into both $prefix/lib and $prefix/lib64.
Can the SLURM developers confirm that this is correct?
I'm a little confused because all prior versions of OMPI have worked fine (i.e., found the Right SLURM libraries), but now there's apparently an issue with SLURM 14.03.7. Did something change in the installation location of perl modules and libraries in this version of SLURM?
If so, can it be changed (back?) such that SLURM only installs into $prefix/lib *or* $prefix/lib64? It's somewhat unexpected behavior to install into *both* lib and lib64.
Thanks!

[Jeff Squyres (jsquyres)]

What are the defaults when you build from a tarball?

The perl libs install in the perl installarchlib it appears from reading the slurm.spec and the rest of the libs install in $prefix/lib or lib64 based on your arch. These can all be altered at configure/rpmbuild time.
On a normal 64bit system this is what I get with the latest 14.03...
rpm -qpl slurm-plugins-14.03.8-1.x86_64.rpm
/usr/lib64/slurm
/usr/lib64/slurm/accounting_storage_filetxt.so
/usr/lib64/slurm/accounting_storage_none.so
/usr/lib64/slurm/accounting_storage_slurmdbd.so
/usr/lib64/slurm/acct_gather_energy_ipmi.so
...
rpm -qpl slurm-14.03.8-1.x86_64.rpm
...
/usr/lib64/libpmi.so
/usr/lib64/libpmi.so.0
/usr/lib64/libpmi.so.0.0.0
/usr/lib64/libpmi2.so
/usr/lib64/libpmi2.so.0
/usr/lib64/libpmi2.so.0.0.0
/usr/lib64/libslurm.so
/usr/lib64/libslurm.so.27
/usr/lib64/libslurm.so.27.0.0
/usr/lib64/libslurmdb.so
/usr/lib64/libslurmdb.so.27
/usr/lib64/libslurmdb.so.27.0.0
rpm -qpl slurm-perlapi-14.03.8-1.x86_64.rpm
/usr/lib/perl/5.18.2/Slurm.pm
/usr/lib/perl/5.18.2/Slurm/Bitstr.pm
/usr/lib/perl/5.18.2/Slurm/Constant.pm
/usr/lib/perl/5.18.2/Slurm/Hostlist.pm
/usr/lib/perl/5.18.2/Slurm/Stepctx.pm
/usr/lib/perl/5.18.2/Slurmdb.pm
/usr/lib/perl/5.18.2/auto/Slurm/Slurm.so
/usr/lib/perl/5.18.2/auto/Slurmdb/Slurmdb.so
/usr/lib/perl/5.18.2/auto/Slurmdb/autosplit.ix

Post by Jeff Squyres (jsquyres)
We got a report in OMPI the other day that reported by Timothy Brown (see http://www.open-mpi.org/community/lists/users/2014/10/25467.php) that SLURM is installing things into both $prefix/lib and $prefix/lib64.
Can the SLURM developers confirm that this is correct?
I'm a little confused because all prior versions of OMPI have worked fine (i.e., found the Right SLURM libraries), but now there's apparently an issue with SLURM 14.03.7. Did something change in the installation location of perl modules and libraries in this version of SLURM?
If so, can it be changed (back?) such that SLURM only installs into $prefix/lib *or* $prefix/lib64? It's somewhat unexpected behavior to install into *both* lib and lib64.
Thanks!

[Timothy Brown]

I just checked with the admin that installed SLURM here. We built it from tar balls (so it would be on central storage) with the following:

$ ./configure --prefix=/curc/slurm/slurm/14.03.7 --with-munge --sysconfdir=/curc/slurm/slurm/etc/ --with-hdf5=yes

We are using RedHat 6 (x86_64). This configure line gave us what I reported before, perl in $prefix/lib64 and everything else in $prefix/lib.

Regards
Timothy

Post by Jeff Squyres (jsquyres)
What are the defaults when you build from a tarball?

The perl libs install in the perl installarchlib it appears from reading the slurm.spec and the rest of the libs install in $prefix/lib or lib64 based on your arch. These can all be altered at configure/rpmbuild time.
On a normal 64bit system this is what I get with the latest 14.03...
rpm -qpl slurm-plugins-14.03.8-1.x86_64.rpm
/usr/lib64/slurm
/usr/lib64/slurm/accounting_storage_filetxt.so
/usr/lib64/slurm/accounting_storage_none.so
/usr/lib64/slurm/accounting_storage_slurmdbd.so
/usr/lib64/slurm/acct_gather_energy_ipmi.so
...
rpm -qpl slurm-14.03.8-1.x86_64.rpm
...
/usr/lib64/libpmi.so
/usr/lib64/libpmi.so.0
/usr/lib64/libpmi.so.0.0.0
/usr/lib64/libpmi2.so
/usr/lib64/libpmi2.so.0
/usr/lib64/libpmi2.so.0.0.0
/usr/lib64/libslurm.so
/usr/lib64/libslurm.so.27
/usr/lib64/libslurm.so.27.0.0
/usr/lib64/libslurmdb.so
/usr/lib64/libslurmdb.so.27
/usr/lib64/libslurmdb.so.27.0.0
rpm -qpl slurm-perlapi-14.03.8-1.x86_64.rpm
/usr/lib/perl/5.18.2/Slurm.pm
/usr/lib/perl/5.18.2/Slurm/Bitstr.pm
/usr/lib/perl/5.18.2/Slurm/Constant.pm
/usr/lib/perl/5.18.2/Slurm/Hostlist.pm
/usr/lib/perl/5.18.2/Slurm/Stepctx.pm
/usr/lib/perl/5.18.2/Slurmdb.pm
/usr/lib/perl/5.18.2/auto/Slurm/Slurm.so
/usr/lib/perl/5.18.2/auto/Slurmdb/Slurmdb.so
/usr/lib/perl/5.18.2/auto/Slurmdb/autosplit.ix

Post by Jeff Squyres (jsquyres)
We got a report in OMPI the other day that reported by Timothy Brown (see http://www.open-mpi.org/community/lists/users/2014/10/25467.php) that SLURM is installing things into both $prefix/lib and $prefix/lib64.
Can the SLURM developers confirm that this is correct?
I'm a little confused because all prior versions of OMPI have worked fine (i.e., found the Right SLURM libraries), but now there's apparently an issue with SLURM 14.03.7. Did something change in the installation location of perl modules and libraries in this version of SLURM?
If so, can it be changed (back?) such that SLURM only installs into $prefix/lib *or* $prefix/lib64? It's somewhat unexpected behavior to install into *both* lib and lib64.
Thanks!

--
Jeff Squyres
For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

[Jeff Squyres (jsquyres)]

Post by Timothy Brown
$ ./configure --prefix=/curc/slurm/slurm/14.03.7 --with-munge --sysconfdir=/curc/slurm/slurm/etc/ --with-hdf5=yes
We are using RedHat 6 (x86_64). This configure line gave us what I reported before, perl in $prefix/lib64 and everything else in $prefix/lib.

Thanks for the confirmation.

This default behavior in SLURM seems... weird. IMHO: if an explicit --prefix is given, it should install everything into lib or lib64, not a mix between them.

[Michael Jennings]

On Tuesday, 07 October 2014, at 09:53:03 (-0700),

Post by Timothy Brown
$ ./configure --prefix=/curc/slurm/slurm/14.03.7 --with-munge --sysconfdir=/curc/slurm/slurm/etc/ --with-hdf5=yes
We are using RedHat 6 (x86_64). This configure line gave us what I reported before, perl in $prefix/lib64 and everything else in $prefix/lib.

Try passing --libdir=/curc/slurm/slurm/14.03.7/lib64 to ./configure as
well.

Perl generates its own prefix based on how it was installed; this uses
RedHat's RPM configuration which is multilib-aware. Since Perl on
RHEL6 was built to use lib64, SLURM's Perl stuff uses the same
location.

However, autotools by default use <prefix>/lib, not
<prefix>/lib<arch>, so I suspect that's where the "conflict" arises.

Michael

[Jeff Squyres (jsquyres)]

Post by Michael Jennings
However, autotools by default use <prefix>/lib, not
<prefix>/lib<arch>, so I suspect that's where the "conflict" arises.

Still seems weird -- can't SLURM's configure compensate for this, when it detects that the perl installation dir != $libdir? Especially when SLURM is installed into an empty tree (vs., say, / or /usr).

[Michael Jennings]

On Tuesday, 07 October 2014, at 12:57:00 (-0700),

Post by Jeff Squyres (jsquyres)

Post by Michael Jennings
However, autotools by default use <prefix>/lib, not
<prefix>/lib<arch>, so I suspect that's where the "conflict" arises.

Still seems weird -- can't SLURM's configure compensate for this, when it detects that the perl installation dir != $libdir? Especially when SLURM is installed into an empty tree (vs., say, / or /usr).

I would say that step 1 is making sure we've successfully identified
the culprit. :-)

Multiple possible solutions are available, depending on what users'
expectations are and how Moe & Danny want things to be orchestrated by
default. I would say further discussion should ensue if Timothy
confirms that we have indeed pinpointed what's going on and that
--libdir "fixes" the issue.

Is this a problem because OpenMPI uses the existence of the
<prefix>/lib64 directory as a litmus test for something? I'm
wondering why this particular scenario would adversely affect the
OpenMPI code or build process. If you (or Timothy) wouldn't mind
elaborating on the specific problem being encountered, that might be
helpful! :-)

Michael

[Jeff Squyres (jsquyres)]

Post by Michael Jennings
Multiple possible solutions are available, depending on what users'
expectations are and how Moe & Danny want things to be orchestrated by
default. I would say further discussion should ensue if Timothy
confirms that we have indeed pinpointed what's going on and that
--libdir "fixes" the issue.

Fair enough. Timothy: can you confirm that --libdir=.../lib64 does fix the issue?

Post by Michael Jennings
Is this a problem because OpenMPI uses the existence of the
<prefix>/lib64 directory as a litmus test for something? I'm
wondering why this particular scenario would adversely affect the
OpenMPI code or build process. If you (or Timothy) wouldn't mind
elaborating on the specific problem being encountered, that might be
helpful! :-)

If a non-default directory is provided as the location of SLURM (i.e., via --with-slurm=DIR), then we check to make sure that directory exists (to ensure that the user didn't typo the dirname). If it does, we look for DIR/lib64, and if we find that, we add -LDIR/lib64 to LDFLAGS. But if we don't fine that, then we look for DIR/lib, and if we find that we add -LDIR/lib to LDFLAGS. Then we check for some symbols that should be in the PMI library with whatever was put in LDFLAGS.

Does that make sense?

We're reluctant to put both lib and lib64 in LDFLAGS, because that can lead to erroneous builds (e.g., a mix of 32 and 64 bit libraries, especially if SLURM is installed in a tree with other packages).

[Timothy Brown]

Post by Jeff Squyres (jsquyres)

Post by Michael Jennings
Multiple possible solutions are available, depending on what users'
expectations are and how Moe & Danny want things to be orchestrated by
default. I would say further discussion should ensue if Timothy
confirms that we have indeed pinpointed what's going on and that
--libdir "fixes" the issue.

Fair enough. Timothy: can you confirm that --libdir=.../lib64 does fix the issue?

Will do.

Post by Jeff Squyres (jsquyres)

Post by Michael Jennings
Is this a problem because OpenMPI uses the existence of the
<prefix>/lib64 directory as a litmus test for something? I'm
wondering why this particular scenario would adversely affect the
OpenMPI code or build process. If you (or Timothy) wouldn't mind
elaborating on the specific problem being encountered, that might be
helpful! :-)

If a non-default directory is provided as the location of SLURM (i.e., via --with-slurm=DIR), then we check to make sure that directory exists (to ensure that the user didn't typo the dirname). If it does, we look for DIR/lib64, and if we find that, we add -LDIR/lib64 to LDFLAGS. But if we don't fine that, then we look for DIR/lib, and if we find that we add -LDIR/lib to LDFLAGS. Then we check for some symbols that should be in the PMI library with whatever was put in LDFLAGS.
Does that make sense?

Yes and this is what the patch Ralph provided does.

Post by Jeff Squyres (jsquyres)
We're reluctant to put both lib and lib64 in LDFLAGS, because that can lead to erroneous builds (e.g., a mix of 32 and 64 bit libraries, especially if SLURM is installed in a tree with other packages).

Totally agree here.

Timothy=

[Jeff Squyres (jsquyres)]

Post by Timothy Brown

Post by Jeff Squyres (jsquyres)
If a non-default directory is provided as the location of SLURM (i.e., via --with-slurm=DIR), then we check to make sure that directory exists (to ensure that the user didn't typo the dirname). If it does, we look for DIR/lib64, and if we find that, we add -LDIR/lib64 to LDFLAGS. But if we don't fine that, then we look for DIR/lib, and if we find that we add -LDIR/lib to LDFLAGS. Then we check for some symbols that should be in the PMI library with whatever was put in LDFLAGS.
Does that make sense?

Yes and this is what the patch Ralph provided does.

Actually, Ralph's patch doesn't do this -- it specifically looks for libpmi.so in lib64/ and lib/ and adds it to LDFLAGS. It then repeats the whole thing for libpmi2. This is really brittle, doesn't work on all OS's, and I think it doesn't even check for libpmi.a right now.

That's why I'd rather get better default behavior out of SLURM's installer. :-)

[Michael Jennings]

On Wednesday, 08 October 2014, at 12:55:59 (-0700),

Post by Jeff Squyres (jsquyres)
If a non-default directory is provided as the location of SLURM
(i.e., via --with-slurm=DIR), then we check to make sure that
directory exists (to ensure that the user didn't typo the dirname).
If it does, we look for DIR/lib64, and if we find that, we add
-LDIR/lib64 to LDFLAGS. But if we don't fine that, then we look for
DIR/lib, and if we find that we add -LDIR/lib to LDFLAGS. Then we
check for some symbols that should be in the PMI library with
whatever was put in LDFLAGS.
Does that make sense?
We're reluctant to put both lib and lib64 in LDFLAGS, because that
can lead to erroneous builds (e.g., a mix of 32 and 64 bit libraries,
especially if SLURM is installed in a tree with other packages).

Makes total sense, and I agree with your methodology. It would be
possible to check for the presence of actual libraries in each
directory rather than just the existence of the directories
themselves, but I can understand that you might be hesitant to do
that.

Unfortunately it's a complex issue: On RHEL6 x64, all arch-specific
lib paths are lib64-based for multilib reasons, and since SLURM has a
.xs component, it needs to go into one of those. That being said, if
a custom prefix is being used, it should be perfectly reasonable to
use /lib instead of /lib64 *iff* the site doesn't need multilib for
that prefix/module. It's arguable whether or not SLURM can safely
assume that.

The most correct fix, IMHO, is for both SLURM's configure script and
its Perl Makefile(.PL) to be multilib-aware, or both not. Either
force the Perl stuff to go in <prefix>/lib[1] if a custom prefix is
specified (i.e., not /usr or /usr/local), or force the configure
script to be multilib-aware by default on systems which are otherwise
multilib-aware (perhaps by looking for /lib64 or /usr/lib64). But I
don't have the breadth of BlueGene/Cray/non-RHEL expertise required to
judge what the implications of this might be for other systems.

Michael

[1] This might be as simple as passing PERLARCHLIB=${prefix}/lib/perl5

[Jeff Squyres (jsquyres)]

Post by Michael Jennings

Post by Jeff Squyres (jsquyres)
We're reluctant to put both lib and lib64 in LDFLAGS, because that
can lead to erroneous builds (e.g., a mix of 32 and 64 bit libraries,
especially if SLURM is installed in a tree with other packages).

Makes total sense, and I agree with your methodology. It would be
possible to check for the presence of actual libraries in each
directory rather than just the existence of the directories
themselves, but I can understand that you might be hesitant to do
that.

Yes, it can (and has) led to broken scenarios for users (confusion about mixing 32 and 64 bit libraries in the same build).

Post by Michael Jennings
Unfortunately it's a complex issue: On RHEL6 x64, all arch-specific
lib paths are lib64-based for multilib reasons, and since SLURM has a
.xs component, it needs to go into one of those. That being said, if
a custom prefix is being used, it should be perfectly reasonable to
use /lib instead of /lib64 *iff* the site doesn't need multilib for
that prefix/module. It's arguable whether or not SLURM can safely
assume that.

Agreed; it wouldn't be good to assume this. But it should be easy enough to check and see if libtool is going to install into one place and perl is going to install into another. And if that's the case, perhaps configure should error out.

Post by Michael Jennings
The most correct fix, IMHO, is for both SLURM's configure script and
its Perl Makefile(.PL) to be multilib-aware, or both not.

Yes. Or perhaps configure could just error out if it detects a mismatch (and therefore let a human figure out, perhaps by supplying --libdir to override lib tool's default install location, etc., or passing PERLARCHLIB=...).

Post by Michael Jennings
[1] This might be as simple as passing PERLARCHLIB=${prefix}/lib/perl5

I stole this suggestion for the above paragraph. :-)

[Michael Jennings]

On Tuesday, 14 October 2014, at 10:58:53 (-0700),

Post by Jeff Squyres (jsquyres)
Yes, it can (and has) led to broken scenarios for users (confusion
about mixing 32 and 64 bit libraries in the same build).

Understood.

Post by Jeff Squyres (jsquyres)
Agreed; it wouldn't be good to assume this. But it should be easy
enough to check and see if libtool is going to install into one
place and perl is going to install into another. And if that's the
case, perhaps configure should error out.
...
Yes. Or perhaps configure could just error out if it detects a
mismatch (and therefore let a human figure out, perhaps by supplying
--libdir to override lib tool's default install location, etc., or
passing PERLARCHLIB=...).

I agree with you. As for what the right answer is, hopefully Moe or
Danny will chime in at this point. Or someone from SchedMD. It's an
administrative choice as to which is the right way to go.

I think most folks would expect PERLARCHLIB to correlate with $libdir
(and $prefix/lib*) when using a custom prefix; I think it's quite
reasonable to take specific actions when --prefix is given (outside of
/usr and /usr/local) that aren't taken otherwise. But as I said
before, I don't have the BG/Cray experience to say what those systems'
expectations or caveats are. :-)

Michael

[Jeff Squyres (jsquyres)]

Moe / Danny / SchedMD in general --

Comments?

Post by Michael Jennings

Post by Jeff Squyres (jsquyres)
Yes. Or perhaps configure could just error out if it detects a
mismatch (and therefore let a human figure out, perhaps by supplying
--libdir to override lib tool's default install location, etc., or
passing PERLARCHLIB=...).

I agree with you. As for what the right answer is, hopefully Moe or
Danny will chime in at this point. Or someone from SchedMD. It's an
administrative choice as to which is the right way to go.
I think most folks would expect PERLARCHLIB to correlate with $libdir
(and $prefix/lib*) when using a custom prefix; I think it's quite
reasonable to take specific actions when --prefix is given (outside of
/usr and /usr/local) that aren't taken otherwise. But as I said
before, I don't have the BG/Cray experience to say what those systems'
expectations or caveats are. :-)
Michael
--
Senior HPC Systems Engineer
High-Performance Computing Services
Lawrence Berkeley National Laboratory
Bldg 50B-3209E W: 510-495-2687
MS 050B-3209 F: 510-486-8615

[Danny Auble]

What rpmbuild does is sets the --libdir so things should work correctly
using that method. I can verify running configure without the --libdir
results in just $prefix/lib no matter the arch. I could see how that is
confusing. If someone submits a patch with a reasonable solution we
will evaluate it and get it in the next version. Don't worry about the
exotic systems like BG or Cray, we will.

As you are probably aware both rpmbuild and standalone configure both
use perl to the the PERLARCHLIB and only replace the prefix of that, you
can find what we do in the slurm.spec if you are interested. I don't
think changing that should happen. For what it is worth PERLARCHLIB is
lib on a 64bit ubuntu install. Redhat/CentOS appears to have lib64.

Post by Jeff Squyres (jsquyres)
Moe / Danny / SchedMD in general --
Comments?

Post by Michael Jennings

Post by Jeff Squyres (jsquyres)
Yes. Or perhaps configure could just error out if it detects a
mismatch (and therefore let a human figure out, perhaps by supplying
--libdir to override lib tool's default install location, etc., or
passing PERLARCHLIB=...).

I agree with you. As for what the right answer is, hopefully Moe or
Danny will chime in at this point. Or someone from SchedMD. It's an
administrative choice as to which is the right way to go.
I think most folks would expect PERLARCHLIB to correlate with $libdir
(and $prefix/lib*) when using a custom prefix; I think it's quite
reasonable to take specific actions when --prefix is given (outside of
/usr and /usr/local) that aren't taken otherwise. But as I said
before, I don't have the BG/Cray experience to say what those systems'
expectations or caveats are. :-)
Michael
--
Senior HPC Systems Engineer
High-Performance Computing Services
Lawrence Berkeley National Laboratory
Bldg 50B-3209E W: 510-495-2687
MS 050B-3209 F: 510-486-8615

[Timothy Brown]

Post by Jeff Squyres (jsquyres)

Post by Michael Jennings
Multiple possible solutions are available, depending on what users'
expectations are and how Moe & Danny want things to be orchestrated by
default. I would say further discussion should ensue if Timothy
confirms that we have indeed pinpointed what's going on and that
--libdir "fixes" the issue.

Fair enough. Timothy: can you confirm that --libdir=.../lib64 does fix the issue?

Sorry for taking so long in getting back to this.

Yes, “—libdir=…/lib64” meant everything got put into $prefix/lib64.

[node0004 14.03.7]$ ls
bin include lib64 sbin share
[node0004 14.03.7]$ ls lib64/
libpmi.a libpmi.so.0 libslurm.la libslurm.so.27.0.0 libslurmdb.so slurm
libpmi.la libpmi.so.0.0.0 libslurm.so libslurmdb.a libslurmdb.so.27
libpmi.so libslurm.a libslurm.so.27 libslurmdb.la libslurmdb.so.27.0.0

Regards
Timothy=