PyTROLL/SatPy Starter Kit V3.0 for GNU/Linux


Ernst Lobsiger
 

Dear All,

here comes the Kit for GNU/Linux amd64 as promised. It's as easy to install and use as the one for Windows. It has been
developed under Debian 10 (Buster) but you should have no problems to get it running on any amd64 GNU/Linux distro.



It's based on Miniconda3 as the Windows companion. Christian Peters took "virtualenv" (as described in my GNU/Linux HOWTO 2.0)
to install the PyTROLL/SatPy environment and made a first Meteosat image on a RASPI (8GB RAM needed) with one of my scripts :-).

Have fun
Ernst


nigel
 

Many thanks Ernst, for all your work on this. I'll be giving it a try in the next few days!

Regards

Nigel


On 17/01/2021 18:04, Ernst Lobsiger via groups.io wrote:
Dear All,

here comes the Kit for GNU/Linux amd64 as promised. It's as easy to install and use as the one for Windows. It has been
developed under Debian 10 (Buster) but you should have no problems to get it running on any amd64 GNU/Linux distro.



It's based on Miniconda3 as the Windows companion. Christian Peters took "virtualenv" (as described in my GNU/Linux HOWTO 2.0)
to install the PyTROLL/SatPy environment and made a first Meteosat image on a RASPI (8GB RAM needed) with one of my scripts :-).

Have fun
Ernst


Hugo
 

Thanks Ernst, already tried your window version in Linux ( with a few adaptations ) and it works as advertised . I'am using PyCharm as IDE  with Anaconda, very pleased with it.

Kind regards,

Hugo


Christian Peters
 

Ernst,

as always a great package. Thank you very much. 
Set it up here and did some tests, works like charm. 
If some Win user want to try it, have a look at his Win Version of the starter kit or e.g. have a look at VirtualBox, install your first Linux inside and start a real adventure. :-D 
It's worth the effort! 

Regards,

Christian 

Am 17.01.2021 um 17:04 schrieb Ernst Lobsiger via groups.io <ernst.lobsiger@...>:

Dear All,

here comes the Kit for GNU/Linux amd64 as promised. It's as easy to install and use as the one for Windows. It has been
developed under Debian 10 (Buster) but you should have no problems to get it running on any amd64 GNU/Linux distro.

<299px-Tux.svg.png>

It's based on Miniconda3 as the Windows companion. Christian Peters took "virtualenv" (as described in my GNU/Linux HOWTO 2.0)
to install the PyTROLL/SatPy environment and made a first Meteosat image on a RASPI (8GB RAM needed) with one of my scripts :-).

Have fun
Ernst
<Pytroll_Satpy_EMC_4_GNU_Linux_Starter_Kit_V30.tgz>


nigel
 

Hi Ernst,

I've started on installing your PyTROLL/StPy on a Raspberry Pi4, following your HOWTO_V30.txt and have come across a problem with Miniconda3 not being 32bit. I see that Christian has used "virtualenv" - could you kindly point me towards your GNU/Linux HOWTO 2.0, so that I can try this route, please ?

Many thanks,

Nigel

On 17/01/2021 18:04, Ernst Lobsiger via groups.io wrote:
Dear All,

here comes the Kit for GNU/Linux amd64 as promised. It's as easy to install and use as the one for Windows. It has been
developed under Debian 10 (Buster) but you should have no problems to get it running on any amd64 GNU/Linux distro.



It's based on Miniconda3 as the Windows companion. Christian Peters took "virtualenv" (as described in my GNU/Linux HOWTO 2.0)
to install the PyTROLL/SatPy environment and made a first Meteosat image on a RASPI (8GB RAM needed) with one of my scripts :-).

Have fun
Ernst


Christian Peters
 

Nigel,

did you install the 64 bit ubuntu version!? I think you should use the 64 bit version (as the chip is 64bit). Anyway miniconda and other condo versions did not work on the PI.
I figured it out with virtualenv and pip. I think Ernst will provide the link in short time. You have to compile the packet on the Pi so it took some time, but it works. Just tested basic things, so more to come from you.

So you I just wait for a fresh MiniSDCard to try...but maybe I now can skip this as you are just ahead...! :-D

Regards,

Christian

P.S.: Maybe I will do it anyway...it will be just fun to get it working on the RP4 (but here only 4GB Ram)...

Am 19.01.21 um 14:01 schrieb nigel:

Hi Ernst,

I've started on installing your PyTROLL/StPy on a Raspberry Pi4, following your HOWTO_V30.txt and have come across a problem with Miniconda3 not being 32bit. I see that Christian has used "virtualenv" - could you kindly point me towards your GNU/Linux HOWTO 2.0, so that I can try this route, please ?

Many thanks,

Nigel

On 17/01/2021 18:04, Ernst Lobsiger via groups.io wrote:
Dear All,

here comes the Kit for GNU/Linux amd64 as promised. It's as easy to install and use as the one for Windows. It has been
developed under Debian 10 (Buster) but you should have no problems to get it running on any amd64 GNU/Linux distro.



It's based on Miniconda3 as the Windows companion. Christian Peters took "virtualenv" (as described in my GNU/Linux HOWTO 2.0)
to install the PyTROLL/SatPy environment and made a first Meteosat image on a RASPI (8GB RAM needed) with one of my scripts :-).

Have fun
Ernst


Ernst Lobsiger
 

On Tue, Jan 19, 2021 at 05:01 AM, nigel wrote:

Hi Ernst,

I've started on installing your PyTROLL/StPy on a Raspberry Pi4, following your HOWTO_V30.txt and have come across a problem with Miniconda3 not being 32bit. I see that Christian has used "virtualenv" - could you kindly point me towards your GNU/Linux HOWTO 2.0, so that I can try this route, please ?

Many thanks,

Nigel

Nigel,

this HOWTO is attached to message 28324 on this list

https://groups.io/g/MSG-1/message/28324

Just use the part 2) on how to install the PyTROLL/SatPy environment with "virtualenv" and pip(3).
Instead of conda you will always use pip as your installer (Miniconda3 for RASPI seems not to exist).
You will also have to compile xRITDecompress (I have packed the source with the Starter Kit V3)

Good Luck
Ernst


nigel
 

Hi Christian,

thanks for your reply. I've currently installed the official 2020-12-02-raspios-buster-armhf OS. Perhaps I should reconsider that and go for a 64 bit version because, as you say, the chip is 64 bit. Bit more research required ! I'm doing this initial instal on a 2GB model but have an 8GB on order - who knows when it will arrive here in N Cyprus !!

It'll be interesting to hear how you get on, when you get the new SD card.

Regards

Nigel


On 19/01/2021 15:21, Christian Peters via groups.io wrote:
Nigel,

did you install the 64 bit ubuntu version!? I think you should use the 64 bit version (as the chip is 64bit). Anyway miniconda and other condo versions did not work on the PI.
I figured it out with virtualenv and pip. I think Ernst will provide the link in short time. You have to compile the packet on the Pi so it took some time, but it works. Just tested basic things, so more to come from you.

So you I just wait for a fresh MiniSDCard to try...but maybe I now can skip this as you are just ahead...! :-D

Regards,

Christian

P.S.: Maybe I will do it anyway...it will be just fun to get it working on the RP4 (but here only 4GB Ram)...

Am 19.01.21 um 14:01 schrieb nigel:

Hi Ernst,

I've started on installing your PyTROLL/StPy on a Raspberry Pi4, following your HOWTO_V30.txt and have come across a problem with Miniconda3 not being 32bit. I see that Christian has used "virtualenv" - could you kindly point me towards your GNU/Linux HOWTO 2.0, so that I can try this route, please ?

Many thanks,

Nigel

On 17/01/2021 18:04, Ernst Lobsiger via groups.io wrote:
Dear All,

here comes the Kit for GNU/Linux amd64 as promised. It's as easy to install and use as the one for Windows. It has been
developed under Debian 10 (Buster) but you should have no problems to get it running on any amd64 GNU/Linux distro.



It's based on Miniconda3 as the Windows companion. Christian Peters took "virtualenv" (as described in my GNU/Linux HOWTO 2.0)
to install the PyTROLL/SatPy environment and made a first Meteosat image on a RASPI (8GB RAM needed) with one of my scripts :-).

Have fun
Ernst


Christian Peters
 

Nigel,

I would recommend the 64bit version. You can stay on the raspios version as it's more tested on that platform. I tried the ubuntu server version too as I tried ZFS on the pi with two mirrored USB HDs. They packed zfs inside...! ;-)

I will report.

Good luck with the install!

Regards,

Christian

Am 19.01.21 um 15:22 schrieb nigel:

Hi Christian,

thanks for your reply. I've currently installed the official 2020-12-02-raspios-buster-armhf OS. Perhaps I should reconsider that and go for a 64 bit version because, as you say, the chip is 64 bit. Bit more research required ! I'm doing this initial instal on a 2GB model but have an 8GB on order - who knows when it will arrive here in N Cyprus !!

It'll be interesting to hear how you get on, when you get the new SD card.

Regards

Nigel


On 19/01/2021 15:21, Christian Peters via groups.io wrote:
Nigel,

did you install the 64 bit ubuntu version!? I think you should use the 64 bit version (as the chip is 64bit). Anyway miniconda and other condo versions did not work on the PI.
I figured it out with virtualenv and pip. I think Ernst will provide the link in short time. You have to compile the packet on the Pi so it took some time, but it works. Just tested basic things, so more to come from you.

So you I just wait for a fresh MiniSDCard to try...but maybe I now can skip this as you are just ahead...! :-D

Regards,

Christian

P.S.: Maybe I will do it anyway...it will be just fun to get it working on the RP4 (but here only 4GB Ram)...

Am 19.01.21 um 14:01 schrieb nigel:

Hi Ernst,

I've started on installing your PyTROLL/StPy on a Raspberry Pi4, following your HOWTO_V30.txt and have come across a problem with Miniconda3 not being 32bit. I see that Christian has used "virtualenv" - could you kindly point me towards your GNU/Linux HOWTO 2.0, so that I can try this route, please ?

Many thanks,

Nigel

On 17/01/2021 18:04, Ernst Lobsiger via groups.io wrote:
Dear All,

here comes the Kit for GNU/Linux amd64 as promised. It's as easy to install and use as the one for Windows. It has been
developed under Debian 10 (Buster) but you should have no problems to get it running on any amd64 GNU/Linux distro.



It's based on Miniconda3 as the Windows companion. Christian Peters took "virtualenv" (as described in my GNU/Linux HOWTO 2.0)
to install the PyTROLL/SatPy environment and made a first Meteosat image on a RASPI (8GB RAM needed) with one of my scripts :-).

Have fun
Ernst



Ernst Lobsiger
 

Dear All,

Ferdinand combined a stacked pass script for the VIIRS instrument at night (NOAA-20). As this did not work as expected
I made a couple of tests. As you might know during the night there are two VIIRS file types MX-Bands and DNB-Bands.

We could make the script run for MX only and for DNB only but never for MX + DNB as needed for 'night_overview'.
I finally tracked the problem down to a subtle difference of two list methods list.remove() and list.pop(). I have used
in my stacked passes scripts the method that I thought sounds like what I intended (and also worked in my tests so far):

goodfiles.remove(goodfiles[n])
goodtimes.remove(goodtimes[n])

But the right method -- I found out the very hard way with Suomi-NPP at night -- actually is:

goodfiles.pop(n)
goodtimes.pop(n)

I really recommend you make these changes (twice these 2 lines) in all stacked pass scripts.
I do not update the Kit for this silly mistake that everybody can fix with an Editor in no time.


Regards,
Ernst

P.S. This problem is independant of the OS. It exists under Windows and GNU/Linux. In my
case phdf often exhausted all possible 1024 file descripters and the satpy.reader excepted.







Ferdinand Valk
 

Ernst,

 

Thank you for the update. I’ve made the changes in all the multiple passes scripts (*stacked.py) and they all work well, including the full suite of DNB composites. A simple but very effective solution.

 

Just to make sure that people do not interpret your instruction in a different way than intended:

 

Replace the full expression goodfiles.remove(goodfiles[n]) with goodfiles.pop(n)

And goodtimes.remove(goodtimes[n]) with goodtimes.pop(n)

 

This occurs twice per stack script. So, do not change the other expressions.

 

Cheers,

Ferdinand

 

From: MSG-1@groups.io On Behalf Of Ernst Lobsiger via groups.io
Sent: Wednesday, 20 January, 2021 17:47
To: MSG-1@groups.io
Subject: Re: [MSG-1] PyTROLL/SatPy Starter Kit V3.0 for GNU/Linux

 

Dear All,

Ferdinand combined a stacked pass script for the VIIRS instrument at night (NOAA-20). As this did not work as expected
I made a couple of tests. As you might know during the night there are two VIIRS file types MX-Bands and DNB-Bands.

We could make the script run for MX only and for DNB only but never for MX + DNB as needed for 'night_overview'.
I finally tracked the problem down to a subtle difference of two list methods list.remove() and list.pop(). I have used
in my stacked passes scripts the method that I thought sounds like what I intended (and also worked in my tests so far):

goodfiles.remove(goodfiles[n])
goodtimes.remove(goodtimes[n])

But the right method -- I found out the very hard way with Suomi-NPP at night -- actually is:

goodfiles.pop(n)
goodtimes.pop(n)

I really recommend you make these changes (twice these 2 lines) in all stacked pass scripts.
I do not update the Kit for this silly mistake that everybody can fix with an Editor in no time.


Regards,
Ernst

P.S. This problem is independant of the OS. It exists under Windows and GNU/Linux. In my
case phdf often exhausted all possible 1024 file descripters and the satpy.reader excepted.






Ernst Lobsiger
 

On Thu, Jan 21, 2021 at 01:46 AM, Ferdinand Valk wrote:
Thank you for the update. I’ve made the changes in all the multiple passes scripts (*stacked.py) and they all work well, including the full suite of DNB composites. A simple but very effective solution.
Ferdinand,

thanks for the clarification in better English than mine. The distributed stacked scripts do work as is. Even the stacked FY-3D where we have two different file types does work due to the fact
that the file pairs follow each other in the goodfiles list. I was just lucky and it worked! But as you demonstrated, users might want a stacked NOAA-20 script at night including the breath taking
'night_overview' that uses both DNB and MX channels in the case of VIIRS. Then they are stuck because now the "list.pop()" is the only method that works. It took me one week to find the true
nature of the problem for VIIRS at night: A "list.remove(list[n])" removes THE FIRST element with value "list[n]". A "list.pop(n)" removes as intended "list[n]" which is the element AT POSITION n.
In the distributed scripts the two methods, with extra luck for FY-3D, do have the same effect. That's why it has worked so far. With NOAA-20 or Suomi-NPP VIIRS at night the lucky phase is over.

Well that's how programming is and that's why Windows 10 has to fix numerous bugs every Tuesday ...

Best regards,
Ernst


R. Alblas
 

Ernst,
Since this seems to be special for AMD 64-bits, what about Intel 32-bits?
Trying it with Ubuntu, 32-bits Intel machine ( to see if that would work), with:
conda install satpy

I get:
PackagesNotFoundError: The following packages are not available from current channels:

  - satpy
...
Current channels:

  - https://conda.anaconda.org/conda-forge/linux-32
...

Maybe I missed it, but as far as I can see there is nothing about "Pytroll/satpy only runs on 64-bits machines".

Regards,
Rob.

On 17-01-2021 17:04, Ernst Lobsiger via groups.io wrote:
Dear All,

here comes the Kit for GNU/Linux amd64 as promised. It's as easy to install and use as the one for Windows. It has been
developed under Debian 10 (Buster) but you should have no problems to get it running on any amd64 GNU/Linux distro.



It's based on Miniconda3 as the Windows companion. Christian Peters took "virtualenv" (as described in my GNU/Linux HOWTO 2.0)
to install the PyTROLL/SatPy environment and made a first Meteosat image on a RASPI (8GB RAM needed) with one of my scripts :-).

Have fun
Ernst


Ernst Lobsiger
 

On Fri, Jan 22, 2021 at 01:53 AM, R. Alblas wrote:
Maybe I missed it, but as far as I can see there is nothing about "Pytroll/satpy only runs on 64-bits machines".
Rob,

I never even tried to install PyTROLL/SatPy on a 32Bit GNU/Linux. Let's face it 32Bit Linux is fading away and
the PyTROLL/SatPy developers maybe don't want to maintain both 32Bit and 64Bit packages on conda-forge.
And as you certainly know amd64 is only a name (type of instruction set) but that's also for Intel processors.

Regards,
Ernst

P.S. Maybe you get a better answer on the PyTROLL/SatPy list on google:

https://groups.google.com/g/pytroll



Christian Peters
 

Rob,

form one of the developers
"...I don't see why it wouldn't work on 32 bit architecture, as long as the dependencies are available..."

So it should work, maybe not with conda? You could try to do the install with pip3...?
But as Ernst said....32bit is fading away and many distribution dropped it already....

Regards,

Christian

Am 22.01.21 um 10:53 schrieb R. Alblas:

Ernst,
Since this seems to be special for AMD 64-bits, what about Intel 32-bits?
Trying it with Ubuntu, 32-bits Intel machine ( to see if that would work), with:
conda install satpy

I get:
PackagesNotFoundError: The following packages are not available from current channels:

  - satpy
...
Current channels:

  - https://conda.anaconda.org/conda-forge/linux-32
...

Maybe I missed it, but as far as I can see there is nothing about "Pytroll/satpy only runs on 64-bits machines".

Regards,
Rob.

On 17-01-2021 17:04, Ernst Lobsiger via groups.io wrote:
Dear All,

here comes the Kit for GNU/Linux amd64 as promised. It's as easy to install and use as the one for Windows. It has been
developed under Debian 10 (Buster) but you should have no problems to get it running on any amd64 GNU/Linux distro.



It's based on Miniconda3 as the Windows companion. Christian Peters took "virtualenv" (as described in my GNU/Linux HOWTO 2.0)
to install the PyTROLL/SatPy environment and made a first Meteosat image on a RASPI (8GB RAM needed) with one of my scripts :-).

Have fun
Ernst



Ernst Lobsiger
 

On Fri, Jan 22, 2021 at 02:40 AM, Christian Peters wrote:
So it should work, maybe not with conda? You could try to do the install with pip3...?
Rob and Christian

In any case it must be Python 3.7 / 3.8 now. If you install with pip(3) then use "virtualenv"
as described in my HOWTO 2.0 that I recently referenced for Nigel some posts behind.

https://groups.io/g/MSG-1/message/28324


Good luck
Ernst


Christian Peters
 

Ernst

indeed....3.7/3.8 is recommended. 2.7 support is dropped recently.
They will check if there are 32bit conda builds...

Martin wrote (a developer):
..."ah, it could be that we aren't building 32bit packages for conda..."

Regards,

Christian

Am 22.01.21 um 12:57 schrieb Ernst Lobsiger via groups.io:

On Fri, Jan 22, 2021 at 02:40 AM, Christian Peters wrote:
So it should work, maybe not with conda? You could try to do the install with pip3...?
Rob and Christian

In any case it must be Python 3.7 / 3.8 now. If you install with pip(3) then use "virtualenv"
as described in my HOWTO 2.0 that I recently referenced for Nigel some posts behind.

https://groups.io/g/MSG-1/message/28324


Good luck
Ernst


R. Alblas
 

Ernst,

I thought you always try to use PC's as old as possible ;-)
Anyway, I'll give it a try, otherwise I have to give myself a 64-bitter.
Regards,
Rob.

On 22-01-2021 11:27, Ernst Lobsiger via groups.io wrote:
Rob,

I never even tried to install PyTROLL/SatPy on a 32Bit GNU/Linux. Let's face it 32Bit Linux is fading away and
the PyTROLL/SatPy developers maybe don't want to maintain both 32Bit and 64Bit packages on conda-forge.
And as you certainly know amd64 is only a name (type of instruction set) but that's also for Intel processors.

Regards,
Ernst

P.S. Maybe you get a better answer on the PyTROLL/SatPy list on google:

https://groups.google.com/g/pytroll




Ernst Lobsiger
 

On Fri, Jan 22, 2021 at 08:34 AM, R. Alblas wrote:
I thought you always try to use PC's as old as possible ;-)
Rob,

amd64 GNU/Linux runs on my 12 Years old Fujitsu/Siemens with a Core Duo like a charm ;-).

Regards,
Ernst


Roberto.inghilesi@...
 

Thank you Ernst, with a little effort it works smoothly on an apple iMac Pro.