Date   

Power8/9

Rhian Resnick
 

Afternoon,



Crazy thought, any one on the user list know if they are investigating porting OpenHPC to the Power8/9 Platform? We have a research group investigating Power9 and would hate to have to port all the packages by hand. 


I don't want to bug the developers and steering committee if it has already been discussed. 


Thanks


Rhian Resnick

Assistant Director Middleware and HPC

Office of Information Technology


Florida Atlantic University

777 Glades Road, CM22, Rm 173B

Boca Raton, FL 33431

Phone 561.297.2647

Fax 561.297.0222

 image


Re: Why prun?

Miroslav Hodak <mhodak@...>
 

Karl,

 

thank for the clarification, very helpful to have official word on this.

 

Miro

 

From: OpenHPC-users@groups.io [mailto:OpenHPC-users@groups.io] On Behalf Of Karl W. Schulz
Sent: Monday, April 2, 2018 12:13 PM
To: OpenHPC-users@groups.io
Subject: Re: [openhpc-users] Why prun?

 

 



On Apr 2, 2018, at 10:38 AM, Miroslav Hodak <mhodak@...> wrote:

 

Hi,

 

Thanks for pointing this out. Handling SLURM and PBS is certainly one of the prun functions, but there is more. I have noticed that when I use OpenMPI with pmix, replacing prun with srun does not work, the job fails. Sure enough, looking at the code, prun adds additional parameters to srun when pmix is present. So, it seems to be a single MPI launch command handling multiple schedulers, MPI implementations, and their options.

 

That is fine, but if that is true, the prun explanation in the Installation Guide needs to be changed. It says prun main role  is to work around PMI issues in Slurm, which appears incorrect.

 

Thanks,

Miro

 

 

 

Thanks for this feedback Miro. As others have mentioned, the prun utility is used as a mechanism to abstract job launch across multiple resource managers and MPI toolchains. It also provides a centralizing mechanism where administrators can customize desired environment settings for their users (and you can run prun in debug mode with “prun -v” to see what env variables it currently sets).

 

The discussion which mentions PMI is now outdated and we will update the docs to remove that and clarify the role of prun as you suggest.

 

Thanks,

 

-k

 

 


Re: Why prun?

Karl W. Schulz
 



On Apr 2, 2018, at 10:38 AM, Miroslav Hodak <mhodak@...> wrote:

Hi,
 
Thanks for pointing this out. Handling SLURM and PBS is certainly one of the prun functions, but there is more. I have noticed that when I use OpenMPI with pmix, replacing prun with srun does not work, the job fails. Sure enough, looking at the code, prun adds additional parameters to srun when pmix is present. So, it seems to be a single MPI launch command handling multiple schedulers, MPI implementations, and their options.
 
That is fine, but if that is true, the prun explanation in the Installation Guide needs to be changed. It says prun main role  is to work around PMI issues in Slurm, which appears incorrect.
 
Thanks,
Miro
 
 

Thanks for this feedback Miro. As others have mentioned, the prun utility is used as a mechanism to abstract job launch across multiple resource managers and MPI toolchains. It also provides a centralizing mechanism where administrators can customize desired environment settings for their users (and you can run prun in debug mode with “prun -v” to see what env variables it currently sets).

The discussion which mentions PMI is now outdated and we will update the docs to remove that and clarify the role of prun as you suggest.

Thanks,

-k



Re: Why prun?

Miroslav Hodak <mhodak@...>
 

Hi,

 

Thanks for pointing this out. Handling SLURM and PBS is certainly one of the prun functions, but there is more. I have noticed that when I use OpenMPI with pmix, replacing prun with srun does not work, the job fails. Sure enough, looking at the code, prun adds additional parameters to srun when pmix is present. So, it seems to be a single MPI launch command handling multiple schedulers, MPI implementations, and their options.

 

That is fine, but if that is true, the prun explanation in the Installation Guide needs to be changed. It says prun main role  is to work around PMI issues in Slurm, which appears incorrect.

 

Thanks,

Miro

 

 

 

From: OpenHPC-users@groups.io [mailto:OpenHPC-users@groups.io] On Behalf Of Vinícius Ferrão
Sent: Monday, April 2, 2018 11:11 AM
To: OpenHPC-users@groups.io
Subject: Re: [openhpc-users] Why prun?

 

Hello,

 

prun is just a tool to support PBS Professional and SLURM using the same command line tools.

 

If you take a look on the file: /opt/ohpc/pub/utils/prun/1.2/prun; you’ll see what it does.

 

I’m not aware of libpmi. I’ll be looking forward for this.

 

 

On 31 Mar 2018, at 22:57, Miroslav Hodak <mhodak@...> wrote:

 

Hello,

 

I have been wondering about the role of the prun script/wrapper. It seems to be OHPC-only (slutrm does not mention it). The Guide (v 1.3.3) explains it this way:

 

OpenHPC also provides a companion job-launch

script named prun that is installed in concert with the pre-packaged MPI toolchains. At present, OpenHPC

is unable to include the PMI process management server normally included within Slurm which implies that

srun cannot be use for MPI job launch. Instead, native job launch mechanisms provided by the MPI stacks

are utilized and prun abstracts this process for the various stacks to retain a single launch command.

 

However, libpmi is now part of OpenHPC and has been for some time. So is prun still useful? If so, what for?

 

Thank you,

Miro

 


Re: Why prun?

Vinícius Ferrão
 

Hello,

prun is just a tool to support PBS Professional and SLURM using the same command line tools.

If you take a look on the file: /opt/ohpc/pub/utils/prun/1.2/prun; you’ll see what it does.

I’m not aware of libpmi. I’ll be looking forward for this.


On 31 Mar 2018, at 22:57, Miroslav Hodak <mhodak@...> wrote:

Hello,
 
I have been wondering about the role of the prun script/wrapper. It seems to be OHPC-only (slutrm does not mention it). The Guide (v 1.3.3) explains it this way:
 
OpenHPC also provides a companion job-launch
script named prun that is installed in concert with the pre-packaged MPI toolchains. At present, OpenHPC
is unable to include the PMI process management server normally included within Slurm which implies that
srun cannot be use for MPI job launch. Instead, native job launch mechanisms provided by the MPI stacks
are utilized and prun abstracts this process for the various stacks to retain a single launch command.
 
However, libpmi is now part of OpenHPC and has been for some time. So is prun still useful? If so, what for?
 
Thank you,
Miro


Re: OpenMPI and shared memory

Patrick Goetz
 

Thanks, that was helpful. In this case we're running a program written by another group, so have less control over the auxiliary libraries used.

On 03/29/2018 06:33 PM, kage park wrote:
I can’t say that I’ve seen that particular message before.
If you want running shared memory with multi-thread then you can use OpenMP (Not OpenMPI).
OpenMP case, maximum-thread-number same as max-cores-number in the server.
MPI ( mpich, mvapich, OpenMPI, Intel MPI, ....) case not using shared memory (not using multi-thread, using multi-process)
So, network speed for each process talking data also very important.
(Maximum task number same as all cores in the cluster)
Best Regard,
Kage Park
On Thu, Mar 29, 2018 at 3:33 PM, Karl W. Schulz <karl@... <mailto:karl@...>> wrote:

> On Mar 29, 2018, at 4:21 PM, Patrick Goetz <pgoetz@... <mailto:pgoetz@...>> wrote:
>
> I'm throwing an OpenMPI question at this group, because why not.
>
> My Cryo-EM users use an image resolution program called relion which uses MPI to make use of either multiple threads (single CPU, multiple cores) or multiple CPUs in a cluster.
>
> Anyway, running on a single machine they reported getting errors that looked like this:
>
> [jensen:11426] 5 more processes have sent help message help-btl-vader.txt / cma-permission-denied
>
> and
>
> WARNING: Linux kernel CMA support was requested via the
> btl_vader_single_copy_mechanism MCA variable, but CMA support is
> not available due to restrictive ptrace settings.
>
> I tracked this down to a kernel parameter?  Apparently previously ptrace could snoop on any process, and setting ptrace_scope to 1 (the default) restricts such introspection to child processes only.
>
> # echo 0 > /proc/sys/kernel/yama/ptrace_scope
>
> Makes the error message go away.  The program runs either way, but I think what's happening is the MPI threads are siblings, hence can't access shared memory by default, so if ptrace_scope isn't 0, each thread makes its own copy of the data in memory.  So if you have 12 cores and 2G of data which could be shared, not sharing results in 24G of memory use.
>
> Question: is this the correct fix?  It seems like this would come up more often when MPI is used, given the default setting which restricts such memory sharing (apparently).
I can’t say that I’ve seen that particular message before, but I
checked our CI environment and we have the ptrace_scope setting in
/proc set to zero so this seems like a reasonable fix.  Note that
CMA (and other similar technologies like LiMIC) are about enabling
zero-copy for shared-memory xfers.  If you don’t enable it, you can
still leverage shared memory transfers on-node, but you pay an extra
copy cost that typically manifests as a latency increase in MPI
transfers.
-k


Re: OpenMPI and shared memory

Patrick Goetz
 

That was just what I needed; thanks Karl!

On 03/29/2018 05:33 PM, Karl W. Schulz wrote:

On Mar 29, 2018, at 4:21 PM, Patrick Goetz <pgoetz@...> wrote:

I'm throwing an OpenMPI question at this group, because why not.

My Cryo-EM users use an image resolution program called relion which uses MPI to make use of either multiple threads (single CPU, multiple cores) or multiple CPUs in a cluster.

Anyway, running on a single machine they reported getting errors that looked like this:

[jensen:11426] 5 more processes have sent help message help-btl-vader.txt / cma-permission-denied

and

WARNING: Linux kernel CMA support was requested via the
btl_vader_single_copy_mechanism MCA variable, but CMA support is
not available due to restrictive ptrace settings.

I tracked this down to a kernel parameter? Apparently previously ptrace could snoop on any process, and setting ptrace_scope to 1 (the default) restricts such introspection to child processes only.

# echo 0 > /proc/sys/kernel/yama/ptrace_scope

Makes the error message go away. The program runs either way, but I think what's happening is the MPI threads are siblings, hence can't access shared memory by default, so if ptrace_scope isn't 0, each thread makes its own copy of the data in memory. So if you have 12 cores and 2G of data which could be shared, not sharing results in 24G of memory use.

Question: is this the correct fix? It seems like this would come up more often when MPI is used, given the default setting which restricts such memory sharing (apparently).
I can’t say that I’ve seen that particular message before, but I checked our CI environment and we have the ptrace_scope setting in /proc set to zero so this seems like a reasonable fix. Note that CMA (and other similar technologies like LiMIC) are about enabling zero-copy for shared-memory xfers. If you don’t enable it, you can still leverage shared memory transfers on-node, but you pay an extra copy cost that typically manifests as a latency increase in MPI transfers.
-k


Why prun?

Miroslav Hodak <mhodak@...>
 

Hello,

 

I have been wondering about the role of the prun script/wrapper. It seems to be OHPC-only (slutrm does not mention it). The Guide (v 1.3.3) explains it this way:

 

OpenHPC also provides a companion job-launch

script named prun that is installed in concert with the pre-packaged MPI toolchains. At present, OpenHPC

is unable to include the PMI process management server normally included within Slurm which implies that

srun cannot be use for MPI job launch. Instead, native job launch mechanisms provided by the MPI stacks

are utilized and prun abstracts this process for the various stacks to retain a single launch command.

 

However, libpmi is now part of OpenHPC and has been for some time. So is prun still useful? If so, what for?

 

Thank you,

Miro


Re: OpenHPC Ubuntu

John Hearns <hearnsj@...>
 

MArk, I guess you have examined containerisation for these users?


On 30 March 2018 at 09:20, Mark Irvine <irvine@...> wrote:
Hi 

This might be a (very) silly question but as I have users who use ubuntu I will ask the question: Would it be possible to have an openhpc system based on ubuntu (or debien) ?  Or at least having some nodes under ubuntu with perhaps a separate queue ?

Thanks
Mark



Re: OpenHPC Ubuntu

 

Bash is bash. If they need to use software that is more readily available in one of these OS’s, why not put it into containers?

Alan

On Mar 30, 2018, at 2:20 AM, Mark Irvine <irvine@...> wrote:

Hi 

This might be a (very) silly question but as I have users who use ubuntu I will ask the question: Would it be possible to have an openhpc system based on ubuntu (or debien) ?  Or at least having some nodes under ubuntu with perhaps a separate queue ?

Thanks
Mark


OpenHPC Ubuntu

Mark Irvine
 

Hi 

This might be a (very) silly question but as I have users who use ubuntu I will ask the question: Would it be possible to have an openhpc system based on ubuntu (or debien) ?  Or at least having some nodes under ubuntu with perhaps a separate queue ?

Thanks
Mark


Re: Spack not generating modules

Rhian Resnick
 

We are still new to ohpc but we seem to have the same issue. We resolved it by sourcing the setup-env.sh file for spack.


cat << EOF > /etc/profile.d/spack.sh
#/bin/bash
# Load spack environment source /opt/ohpc/admin/spack/0.10.0/share/spack/setup-env.sh

EOF chmod +x /etc/profile.d/spack.sh


Re: OpenMPI and shared memory

kage park
 

I can’t say that I’ve seen that particular message before.

If you want running shared memory with multi-thread then you can use OpenMP (Not OpenMPI).
OpenMP case, maximum-thread-number same as max-cores-number in the server. 

MPI ( mpich, mvapich, OpenMPI, Intel MPI, ....) case not using shared memory (not using multi-thread, using multi-process)
So, network speed for each process talking data also very important.
(Maximum task number same as all cores in the cluster)

Best Regard,

Kage Park



On Thu, Mar 29, 2018 at 3:33 PM, Karl W. Schulz <karl@...> wrote:


> On Mar 29, 2018, at 4:21 PM, Patrick Goetz <pgoetz@...> wrote:
>
> I'm throwing an OpenMPI question at this group, because why not.
>
> My Cryo-EM users use an image resolution program called relion which uses MPI to make use of either multiple threads (single CPU, multiple cores) or multiple CPUs in a cluster.
>
> Anyway, running on a single machine they reported getting errors that looked like this:
>
> [jensen:11426] 5 more processes have sent help message help-btl-vader.txt / cma-permission-denied
>
> and
>
> WARNING: Linux kernel CMA support was requested via the
> btl_vader_single_copy_mechanism MCA variable, but CMA support is
> not available due to restrictive ptrace settings.
>
> I tracked this down to a kernel parameter?  Apparently previously ptrace could snoop on any process, and setting ptrace_scope to 1 (the default) restricts such introspection to child processes only.
>
> # echo 0 > /proc/sys/kernel/yama/ptrace_scope
>
> Makes the error message go away.  The program runs either way, but I think what's happening is the MPI threads are siblings, hence can't access shared memory by default, so if ptrace_scope isn't 0, each thread makes its own copy of the data in memory.  So if you have 12 cores and 2G of data which could be shared, not sharing results in 24G of memory use.
>
> Question: is this the correct fix?  It seems like this would come up more often when MPI is used, given the default setting which restricts such memory sharing (apparently).


I can’t say that I’ve seen that particular message before, but I checked our CI environment and we have the ptrace_scope setting in /proc set to zero so this seems like a reasonable fix.  Note that CMA (and other similar technologies like LiMIC) are about enabling zero-copy for shared-memory xfers.  If you don’t enable it, you can still leverage shared memory transfers on-node, but you pay an extra copy cost that typically manifests as a latency increase in MPI transfers.

-k







Re: OpenMPI and shared memory

Karl W. Schulz
 

On Mar 29, 2018, at 4:21 PM, Patrick Goetz <pgoetz@...> wrote:

I'm throwing an OpenMPI question at this group, because why not.

My Cryo-EM users use an image resolution program called relion which uses MPI to make use of either multiple threads (single CPU, multiple cores) or multiple CPUs in a cluster.

Anyway, running on a single machine they reported getting errors that looked like this:

[jensen:11426] 5 more processes have sent help message help-btl-vader.txt / cma-permission-denied

and

WARNING: Linux kernel CMA support was requested via the
btl_vader_single_copy_mechanism MCA variable, but CMA support is
not available due to restrictive ptrace settings.

I tracked this down to a kernel parameter? Apparently previously ptrace could snoop on any process, and setting ptrace_scope to 1 (the default) restricts such introspection to child processes only.

# echo 0 > /proc/sys/kernel/yama/ptrace_scope

Makes the error message go away. The program runs either way, but I think what's happening is the MPI threads are siblings, hence can't access shared memory by default, so if ptrace_scope isn't 0, each thread makes its own copy of the data in memory. So if you have 12 cores and 2G of data which could be shared, not sharing results in 24G of memory use.

Question: is this the correct fix? It seems like this would come up more often when MPI is used, given the default setting which restricts such memory sharing (apparently).

I can’t say that I’ve seen that particular message before, but I checked our CI environment and we have the ptrace_scope setting in /proc set to zero so this seems like a reasonable fix. Note that CMA (and other similar technologies like LiMIC) are about enabling zero-copy for shared-memory xfers. If you don’t enable it, you can still leverage shared memory transfers on-node, but you pay an extra copy cost that typically manifests as a latency increase in MPI transfers.

-k


OpenMPI and shared memory

Patrick Goetz
 

I'm throwing an OpenMPI question at this group, because why not.

My Cryo-EM users use an image resolution program called relion which uses MPI to make use of either multiple threads (single CPU, multiple cores) or multiple CPUs in a cluster.

Anyway, running on a single machine they reported getting errors that looked like this:

[jensen:11426] 5 more processes have sent help message help-btl-vader.txt / cma-permission-denied

and

WARNING: Linux kernel CMA support was requested via the
btl_vader_single_copy_mechanism MCA variable, but CMA support is
not available due to restrictive ptrace settings.

I tracked this down to a kernel parameter? Apparently previously ptrace could snoop on any process, and setting ptrace_scope to 1 (the default) restricts such introspection to child processes only.

# echo 0 > /proc/sys/kernel/yama/ptrace_scope

Makes the error message go away. The program runs either way, but I think what's happening is the MPI threads are siblings, hence can't access shared memory by default, so if ptrace_scope isn't 0, each thread makes its own copy of the data in memory. So if you have 12 cores and 2G of data which could be shared, not sharing results in 24G of memory use.

Question: is this the correct fix? It seems like this would come up more often when MPI is used, given the default setting which restricts such memory sharing (apparently).


Re: Adding Lmod modules?

Karl W. Schulz
 

On Mar 29, 2018, at 1:36 PM, Mark Moorcroft <plaktau@...> wrote:

Thanks Derek,

I'm asking because the guy who is taking on maintaining our modules claims that you can't fully embrace the advantages with lmod if you go outside the main tree. We are doing exactly what you suggest now, but he claims a lot of the smarts and inheritance is lost when you do that. Or it could be we just lack experience with lmod.

Mark Moorcroft
NASA/Ames RC

If you just augment the MODULEPATH as Derek suggested, then you are still letting Lmod do it’s magic. You may have some local moduelfiles that don’t take advantage of all the options (for example. the “family” designation that we use in ohpc-generated modulefiles), but you should not lose functionality.

Cheers,

-k


Re: Adding Lmod modules?

Mark Moorcroft
 

Thanks Derek,

I'm asking because the guy who is taking on maintaining our modules claims that you can't fully embrace the advantages with lmod if you go outside the main tree. We are doing exactly what you suggest now, but he claims a lot of the smarts and inheritance is lost when you do that. Or it could be we just lack experience with lmod.

Mark Moorcroft
NASA/Ames RC


Re: Adding Lmod modules?

Derek Simmel
 

At PSC, we typically put our own local modules into another location (say /opt/modulefiles), and then adjust the MODULEPATH environment variable to include all directories of modules, e.g.,

$ echo $MODULEPATH
/opt/ohpc/pub/modulefiles:/opt/modulefiles

$ module avail

-------------------------- /opt/ohpc/pub/modulefiles ---------------------------
EasyBuild/3.3.1 hwloc/1.11.7 pmix/1.2.3
autotools llvm4/4.0.1 prun/1.1
gnu7/7.1.0 papi/5.5.1 valgrind/3.13.0

------------------------------- /opt/modulefiles -------------------------------
cuda/9.1

- Derek

On Mar 28, 2018, at 10:07 PM, Mark Moorcroft <plaktau@...> wrote:


Is there an accepted practice/location for adding additional modules? Is it considered bad form to add them in /opt/ohpc, or is that exactly where you should be putting them?
---
Derek Simmel
Pittsburgh Supercomputing Center
dsimmel@...
+1 (412) 268-1035


Adding Lmod modules?

Mark Moorcroft
 


Is there an accepted practice/location for adding additional modules? Is it considered bad form to add them in /opt/ohpc, or is that exactly where you should be putting them? 


Re: CentOS aarch64 on Raspberry PI 3

Adrian Reber
 

On Wed, Mar 28, 2018 at 06:33:54AM -0700, DARDO ARIEL VIÑAS VISCARDI wrote:
Adrian!! I'm trying to build your guide: https://opensource.com/article/18/1/how-build-hpc-system-raspberry-pi-and-openhpc#comment-152896

How did you do this???
I downloaded the Fedora 27 64bit Raspberry Pi 3 image:

https://fedoraproject.org/wiki/Architectures/ARM/Raspberry_Pi#Raspberry_Pi_3_aarch64_support

Installed it and replaced all user space packages with the CentOS 7
aarch64 files. I do not actually remember, but I guess I was using rsync
with --delete. The whole process is not very pretty, but it worked for
me.

Adrian

2861 - 2880 of 4663