Date   

Re: OpenMPI and shared memory

Patrick Goetz
 

That was just what I needed; thanks Karl!

On 03/29/2018 05:33 PM, Karl W. Schulz wrote:

On Mar 29, 2018, at 4:21 PM, Patrick Goetz <pgoetz@...> wrote:

I'm throwing an OpenMPI question at this group, because why not.

My Cryo-EM users use an image resolution program called relion which uses MPI to make use of either multiple threads (single CPU, multiple cores) or multiple CPUs in a cluster.

Anyway, running on a single machine they reported getting errors that looked like this:

[jensen:11426] 5 more processes have sent help message help-btl-vader.txt / cma-permission-denied

and

WARNING: Linux kernel CMA support was requested via the
btl_vader_single_copy_mechanism MCA variable, but CMA support is
not available due to restrictive ptrace settings.

I tracked this down to a kernel parameter? Apparently previously ptrace could snoop on any process, and setting ptrace_scope to 1 (the default) restricts such introspection to child processes only.

# echo 0 > /proc/sys/kernel/yama/ptrace_scope

Makes the error message go away. The program runs either way, but I think what's happening is the MPI threads are siblings, hence can't access shared memory by default, so if ptrace_scope isn't 0, each thread makes its own copy of the data in memory. So if you have 12 cores and 2G of data which could be shared, not sharing results in 24G of memory use.

Question: is this the correct fix? It seems like this would come up more often when MPI is used, given the default setting which restricts such memory sharing (apparently).
I can’t say that I’ve seen that particular message before, but I checked our CI environment and we have the ptrace_scope setting in /proc set to zero so this seems like a reasonable fix. Note that CMA (and other similar technologies like LiMIC) are about enabling zero-copy for shared-memory xfers. If you don’t enable it, you can still leverage shared memory transfers on-node, but you pay an extra copy cost that typically manifests as a latency increase in MPI transfers.
-k


Why prun?

Miroslav Hodak <mhodak@...>
 

Hello,

 

I have been wondering about the role of the prun script/wrapper. It seems to be OHPC-only (slutrm does not mention it). The Guide (v 1.3.3) explains it this way:

 

OpenHPC also provides a companion job-launch

script named prun that is installed in concert with the pre-packaged MPI toolchains. At present, OpenHPC

is unable to include the PMI process management server normally included within Slurm which implies that

srun cannot be use for MPI job launch. Instead, native job launch mechanisms provided by the MPI stacks

are utilized and prun abstracts this process for the various stacks to retain a single launch command.

 

However, libpmi is now part of OpenHPC and has been for some time. So is prun still useful? If so, what for?

 

Thank you,

Miro


Re: OpenHPC Ubuntu

John Hearns <hearnsj@...>
 

MArk, I guess you have examined containerisation for these users?


On 30 March 2018 at 09:20, Mark Irvine <irvine@...> wrote:
Hi 

This might be a (very) silly question but as I have users who use ubuntu I will ask the question: Would it be possible to have an openhpc system based on ubuntu (or debien) ?  Or at least having some nodes under ubuntu with perhaps a separate queue ?

Thanks
Mark



Re: OpenHPC Ubuntu

 

Bash is bash. If they need to use software that is more readily available in one of these OS’s, why not put it into containers?

Alan

On Mar 30, 2018, at 2:20 AM, Mark Irvine <irvine@...> wrote:

Hi 

This might be a (very) silly question but as I have users who use ubuntu I will ask the question: Would it be possible to have an openhpc system based on ubuntu (or debien) ?  Or at least having some nodes under ubuntu with perhaps a separate queue ?

Thanks
Mark


OpenHPC Ubuntu

Mark Irvine
 

Hi 

This might be a (very) silly question but as I have users who use ubuntu I will ask the question: Would it be possible to have an openhpc system based on ubuntu (or debien) ?  Or at least having some nodes under ubuntu with perhaps a separate queue ?

Thanks
Mark


Re: Spack not generating modules

Rhian Resnick
 

We are still new to ohpc but we seem to have the same issue. We resolved it by sourcing the setup-env.sh file for spack.


cat << EOF > /etc/profile.d/spack.sh
#/bin/bash
# Load spack environment source /opt/ohpc/admin/spack/0.10.0/share/spack/setup-env.sh

EOF chmod +x /etc/profile.d/spack.sh


Re: OpenMPI and shared memory

kage park
 

I can’t say that I’ve seen that particular message before.

If you want running shared memory with multi-thread then you can use OpenMP (Not OpenMPI).
OpenMP case, maximum-thread-number same as max-cores-number in the server. 

MPI ( mpich, mvapich, OpenMPI, Intel MPI, ....) case not using shared memory (not using multi-thread, using multi-process)
So, network speed for each process talking data also very important.
(Maximum task number same as all cores in the cluster)

Best Regard,

Kage Park



On Thu, Mar 29, 2018 at 3:33 PM, Karl W. Schulz <karl@...> wrote:


> On Mar 29, 2018, at 4:21 PM, Patrick Goetz <pgoetz@...> wrote:
>
> I'm throwing an OpenMPI question at this group, because why not.
>
> My Cryo-EM users use an image resolution program called relion which uses MPI to make use of either multiple threads (single CPU, multiple cores) or multiple CPUs in a cluster.
>
> Anyway, running on a single machine they reported getting errors that looked like this:
>
> [jensen:11426] 5 more processes have sent help message help-btl-vader.txt / cma-permission-denied
>
> and
>
> WARNING: Linux kernel CMA support was requested via the
> btl_vader_single_copy_mechanism MCA variable, but CMA support is
> not available due to restrictive ptrace settings.
>
> I tracked this down to a kernel parameter?  Apparently previously ptrace could snoop on any process, and setting ptrace_scope to 1 (the default) restricts such introspection to child processes only.
>
> # echo 0 > /proc/sys/kernel/yama/ptrace_scope
>
> Makes the error message go away.  The program runs either way, but I think what's happening is the MPI threads are siblings, hence can't access shared memory by default, so if ptrace_scope isn't 0, each thread makes its own copy of the data in memory.  So if you have 12 cores and 2G of data which could be shared, not sharing results in 24G of memory use.
>
> Question: is this the correct fix?  It seems like this would come up more often when MPI is used, given the default setting which restricts such memory sharing (apparently).


I can’t say that I’ve seen that particular message before, but I checked our CI environment and we have the ptrace_scope setting in /proc set to zero so this seems like a reasonable fix.  Note that CMA (and other similar technologies like LiMIC) are about enabling zero-copy for shared-memory xfers.  If you don’t enable it, you can still leverage shared memory transfers on-node, but you pay an extra copy cost that typically manifests as a latency increase in MPI transfers.

-k







Re: OpenMPI and shared memory

Karl W. Schulz
 

On Mar 29, 2018, at 4:21 PM, Patrick Goetz <pgoetz@...> wrote:

I'm throwing an OpenMPI question at this group, because why not.

My Cryo-EM users use an image resolution program called relion which uses MPI to make use of either multiple threads (single CPU, multiple cores) or multiple CPUs in a cluster.

Anyway, running on a single machine they reported getting errors that looked like this:

[jensen:11426] 5 more processes have sent help message help-btl-vader.txt / cma-permission-denied

and

WARNING: Linux kernel CMA support was requested via the
btl_vader_single_copy_mechanism MCA variable, but CMA support is
not available due to restrictive ptrace settings.

I tracked this down to a kernel parameter? Apparently previously ptrace could snoop on any process, and setting ptrace_scope to 1 (the default) restricts such introspection to child processes only.

# echo 0 > /proc/sys/kernel/yama/ptrace_scope

Makes the error message go away. The program runs either way, but I think what's happening is the MPI threads are siblings, hence can't access shared memory by default, so if ptrace_scope isn't 0, each thread makes its own copy of the data in memory. So if you have 12 cores and 2G of data which could be shared, not sharing results in 24G of memory use.

Question: is this the correct fix? It seems like this would come up more often when MPI is used, given the default setting which restricts such memory sharing (apparently).

I can’t say that I’ve seen that particular message before, but I checked our CI environment and we have the ptrace_scope setting in /proc set to zero so this seems like a reasonable fix. Note that CMA (and other similar technologies like LiMIC) are about enabling zero-copy for shared-memory xfers. If you don’t enable it, you can still leverage shared memory transfers on-node, but you pay an extra copy cost that typically manifests as a latency increase in MPI transfers.

-k


OpenMPI and shared memory

Patrick Goetz
 

I'm throwing an OpenMPI question at this group, because why not.

My Cryo-EM users use an image resolution program called relion which uses MPI to make use of either multiple threads (single CPU, multiple cores) or multiple CPUs in a cluster.

Anyway, running on a single machine they reported getting errors that looked like this:

[jensen:11426] 5 more processes have sent help message help-btl-vader.txt / cma-permission-denied

and

WARNING: Linux kernel CMA support was requested via the
btl_vader_single_copy_mechanism MCA variable, but CMA support is
not available due to restrictive ptrace settings.

I tracked this down to a kernel parameter? Apparently previously ptrace could snoop on any process, and setting ptrace_scope to 1 (the default) restricts such introspection to child processes only.

# echo 0 > /proc/sys/kernel/yama/ptrace_scope

Makes the error message go away. The program runs either way, but I think what's happening is the MPI threads are siblings, hence can't access shared memory by default, so if ptrace_scope isn't 0, each thread makes its own copy of the data in memory. So if you have 12 cores and 2G of data which could be shared, not sharing results in 24G of memory use.

Question: is this the correct fix? It seems like this would come up more often when MPI is used, given the default setting which restricts such memory sharing (apparently).


Re: Adding Lmod modules?

Karl W. Schulz
 

On Mar 29, 2018, at 1:36 PM, Mark Moorcroft <plaktau@...> wrote:

Thanks Derek,

I'm asking because the guy who is taking on maintaining our modules claims that you can't fully embrace the advantages with lmod if you go outside the main tree. We are doing exactly what you suggest now, but he claims a lot of the smarts and inheritance is lost when you do that. Or it could be we just lack experience with lmod.

Mark Moorcroft
NASA/Ames RC

If you just augment the MODULEPATH as Derek suggested, then you are still letting Lmod do it’s magic. You may have some local moduelfiles that don’t take advantage of all the options (for example. the “family” designation that we use in ohpc-generated modulefiles), but you should not lose functionality.

Cheers,

-k


Re: Adding Lmod modules?

Mark Moorcroft
 

Thanks Derek,

I'm asking because the guy who is taking on maintaining our modules claims that you can't fully embrace the advantages with lmod if you go outside the main tree. We are doing exactly what you suggest now, but he claims a lot of the smarts and inheritance is lost when you do that. Or it could be we just lack experience with lmod.

Mark Moorcroft
NASA/Ames RC


Re: Adding Lmod modules?

Derek Simmel
 

At PSC, we typically put our own local modules into another location (say /opt/modulefiles), and then adjust the MODULEPATH environment variable to include all directories of modules, e.g.,

$ echo $MODULEPATH
/opt/ohpc/pub/modulefiles:/opt/modulefiles

$ module avail

-------------------------- /opt/ohpc/pub/modulefiles ---------------------------
EasyBuild/3.3.1 hwloc/1.11.7 pmix/1.2.3
autotools llvm4/4.0.1 prun/1.1
gnu7/7.1.0 papi/5.5.1 valgrind/3.13.0

------------------------------- /opt/modulefiles -------------------------------
cuda/9.1

- Derek

On Mar 28, 2018, at 10:07 PM, Mark Moorcroft <plaktau@...> wrote:


Is there an accepted practice/location for adding additional modules? Is it considered bad form to add them in /opt/ohpc, or is that exactly where you should be putting them?
---
Derek Simmel
Pittsburgh Supercomputing Center
dsimmel@...
+1 (412) 268-1035


Adding Lmod modules?

Mark Moorcroft
 


Is there an accepted practice/location for adding additional modules? Is it considered bad form to add them in /opt/ohpc, or is that exactly where you should be putting them? 


Re: CentOS aarch64 on Raspberry PI 3

Adrian Reber
 

On Wed, Mar 28, 2018 at 06:33:54AM -0700, DARDO ARIEL VIÑAS VISCARDI wrote:
Adrian!! I'm trying to build your guide: https://opensource.com/article/18/1/how-build-hpc-system-raspberry-pi-and-openhpc#comment-152896

How did you do this???
I downloaded the Fedora 27 64bit Raspberry Pi 3 image:

https://fedoraproject.org/wiki/Architectures/ARM/Raspberry_Pi#Raspberry_Pi_3_aarch64_support

Installed it and replaced all user space packages with the CentOS 7
aarch64 files. I do not actually remember, but I guess I was using rsync
with --delete. The whole process is not very pretty, but it worked for
me.

Adrian


Re: CentOS aarch64 on Raspberry PI 3

DARDO ARIEL VIÑAS VISCARDI
 

Adrian!! I'm trying to build your guide: https://opensource.com/article/18/1/how-build-hpc-system-raspberry-pi-and-openhpc#comment-152896

How did you do this???


Re: CentOS aarch64 on Raspberry PI 3

Adrian Reber
 

On Wed, Mar 28, 2018 at 03:04:36PM +0200, Alexandre Strube wrote:
I’m not sure you can use the aarch64 image for it, but you can you the
minimal image made specifically for the raspberry pi 3:

http://mirror.centos.org/altarch/7/isos/armhfp/
That is 32bit only. What I did for a demo was to use CentOS 64 bit
aarch64 user-space and a Fedora 64 bit kernel which works with the
raspberry pi 3.

Adrian

2018-03-28 14:58 GMT+02:00 DARDO ARIEL VIÑAS VISCARDI <
dardo.vinas@...>:

Has anyone find out a way to run CentOS 7 on RP3?

I want to build a small cluster based on them and can't find a way to load
Centos on aarch64


Re: CentOS aarch64 on Raspberry PI 3

Alexandre Strube
 

I’m not sure you can use the aarch64 image for it, but you can you the minimal image made specifically for the raspberry pi 3:


2018-03-28 14:58 GMT+02:00 DARDO ARIEL VIÑAS VISCARDI <dardo.vinas@...>:

Has anyone find out a way to run CentOS 7 on RP3? 

I want to build a small cluster based on them and can't find a way to load Centos on aarch64




--
[]
Alexandre Strube
surak@...


CentOS aarch64 on Raspberry PI 3

DARDO ARIEL VIÑAS VISCARDI
 

Has anyone find out a way to run CentOS 7 on RP3? 

I want to build a small cluster based on them and can't find a way to load Centos on aarch64


Spack not generating modules

i.porebski@...
 

Hi All,

I have installed SPACK version 0.10.0 from OpenHPC. After that I have installed boost "spack install boost" which install boost@1.63.0 and add to lmod

$ ml spider boost/1.63.0
 
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  boost: boost/1.63.0
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    Description:
      Boost free peer-reviewed portable C++ source libraries
 
 
    You will need to load all module(s) on any one of the lines below before the "boost/1.63.0" module is available to load.
 
      gnu/5.4.0  mpich/3.2
      gnu/5.4.0  mvapich2/2.2
      gnu/5.4.0  openmpi/1.10.6
      gnu7/7.2.0  openmpi/1.10.7

However if I installed different version of the boost via spack the software will be installed but only in admin (/opt/ohpc/admin/spack) folder structure but not in pub (/opt/ohpc/pub). The lmod files are not created as well. What I am missing? I was expecting that spack will create modules and copy the software to pub folder similar like for other version so the users can access it. 

Could you help me troubleshoot this issue? 

Thanks,
Irek


Re: Cant run ICC on my compute nodes

Patrick Goetz
 

I'm pretty sure you're not supposed to have 2 exports with the same fsid. Check to see if /opt/intel is even being mounted.

On 03/22/2018 02:44 PM, DARDO ARIEL VIÑAS VISCARDI wrote:
You know I realized that, so I added the folder on the master in /etc/exports with the others:
/home *(rw,no_subtree_check,fsid=10,no_root_squash)
/opt/ohpc/pub *(ro,no_subtree_check,fsid=11)
/opt/intel *(ro,no_subtree_check,fsid=11)
And on the provosioning image, on the $CHROOT/etc/fstab
[root@n2 ~]# cat /etc/fstab
tmpfs / tmpfs rw,relatime,mode=555 0 0
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
10.0.1.1:/home /home nfs nfsvers=3,nodev,nosuid,noatime 0 0
10.0.1.1:/opt/ohpc/pub /opt/ohpc/pub nfs nfsvers=3,nodev,noatime 0 0
10.0.1.1:/opt/intel /opt/intel nfs nfsvers=3,nodev,noatime 0 0
I ran the command "exportfs -a"
But steel, after rebuilding everthing, rebooting nodes, everything..... the folder /opt/intel is showing the content of /opt/ohpc/pub...
Any ideas why this could be happening?

2941 - 2960 of 4737