Topics

Pytroll/SatPy for EUMETCast


Ferdinand Valk
 

Ernst,

After an extended period of health inconvenience I finally took the time to
go through the history of emails related to SatPy on a Windows machine. I'm
impressed with the work you have done and with the straightforward
instructions you wrote. I got it working in a single morning with no
obstacle worth mentioning.
My sincere thanks for this contribution to the community.

Cheers,
Ferdinand Valk


David J Taylor GM8ARV 🏴󠁧󠁢󠁳󠁣󠁴󠁿 🇪🇺
 

Ernst,

After an extended period of health inconvenience I finally took the time to
go through the history of emails related to SatPy on a Windows machine. I'm
impressed with the work you have done and with the straightforward
instructions you wrote. I got it working in a single morning with no
obstacle worth mentioning.
My sincere thanks for this contribution to the community.

Cheers,
Ferdinand Valk
=======================================

It's something I intended to play with - are the instructions in the group's file area, or is there a download URL for them?

Thanks, Ernst!

David
--
SatSignal Software - Quality software for you
Web: https://www.satsignal.eu
Email: david-taylor@blueyonder.co.uk
Twitter: @gm8arv


Ernst Lobsiger
 

On Fri, Oct 30, 2020 at 12:00 PM, David J Taylor GM8ARV 🏴󠁧󠁢󠁳󠁣󠁴󠁿 🇪🇺 wrote:
It's something I intended to play with - are the instructions in the group's file area, or is there a download URL for them?

Thanks, Ernst!

David
David,

there is this long MSG-1 thread from spring 2020. I attached a
Windows10 64Bit starter kit to message #29367. This should still
be an acceptable starting point. Unfortunately only weeks later
there were a couple of improvements in SatPy and also some more
scrips attached then and when (as Graham Woolf kept asking ;-).

Maybe this winter brings the next lockdown and I find the time
to update to a version 2.0 of the HOWTO. As I stated there must
be five Windows users of Hugo's proposed oldtimer club reporting
success with EUMETCast and PyTroll/SatPy first. We have now:

Graham Woolf
Douglas Deans
John ?? john.haslam4@..
Ferdinand Valk  (thanks for your nice report!)

Cheers,
Ernst


Graham Woolf
 

Hi Ernst

I for one would love a version 2.0

Regards

Graham


Ferdinand Valk
 

I’m using SatPy version 23.0.  Is this version backward compatible with 21 or 22? After I had upgraded I get error messages complaining that dll loading failed while importing_arpack. I have no clue what I might have disrupted.

 

I’m a bit confused.

 

Thanks,

Ferdinand

 


Ernst Lobsiger
 

Ferdinand

I have only SatPy V0.21 on Windows and V0.22 on GNU/Linux. Both without problems.
arpack and lapack are scientific libraries (linear algebra, solving Eigenvalue problems).

When i do in environment (pytroll) a "conda list" I only find "liblapack 3.9.0" (Linux) here.

Are you sure you installed your SatPy version 0.23 while you were in environment (pytroll)?

Regards
Ernst


Christian Peters
 

Hi all, 

I would avoid installing Satpy 0.23 at the moment as I have problems/errors with making FY-3D images.

Regards,

Christian 

Am 03.11.2020 um 21:25 schrieb Ernst Lobsiger via groups.io <ernst.lobsiger@...>:

Ferdinand

I have only SatPy V0.21 on Windows and V0.22 on GNU/Linux. Both without problems.
arpack and lapack are scientific libraries (linear algebra, solving Eigenvalue problems).

When i do in environment (pytroll) a "conda list" I only find "liblapack 3.9.0" (Linux) here.

Are you sure you installed your SatPy version 0.23 while you were in environment (pytroll)?

Regards
Ernst


Ferdinand Valk
 

Ernst, I thought I was sure, but start to doubt now. In any case, in order to work from a proven state (and understanding that Christian also encounters errors)  I will deinstall and revert to V0.21.

 

Thanks for the feedback.

 

Cheers,

Ferdinand

 

From: MSG-1@groups.io On Behalf Of Ernst Lobsiger via groups.io
Sent: Tuesday, 03 November, 2020 20:26
To: MSG-1@groups.io
Subject: Re: [MSG-1] Pytroll/SatPy for EUMETCast

 

Ferdinand

I have only SatPy V0.21 on Windows and V0.22 on GNU/Linux. Both without problems.
arpack and lapack are scientific libraries (linear algebra, solving Eigenvalue problems).

When i do in environment (pytroll) a "conda list" I only find "liblapack 3.9.0" (Linux) here.

Are you sure you installed your SatPy version 0.23 while you were in environment (pytroll)?

Regards
Ernst


Ferdinand Valk
 

Geniuses,

 

I started a fresh install but find that when I use:

(pytroll) C:\Users\fvalk\conda install satpy

It returns with the full list of new packages to be installed, amongst which I find conda-forge/noarch::satpy-0.23.0.py_0

Is there a way to force satpy 0.21 to be selected instead? I’ve tried to find that in the documentation but could not come up with an answer (likely because I looked at the wrong places).

A pointer would be very welcome.

 

Thanks,

Ferdinand

 

From: MSG-1@groups.io On Behalf Of Christian Peters via groups.io
Sent: Tuesday, 03 November, 2020 20:53
To: MSG-1@groups.io
Subject: Re: [MSG-1] Pytroll/SatPy for EUMETCast

 

Hi all, 

 

I would avoid installing Satpy 0.23 at the moment as I have problems/errors with making FY-3D images.

 

Regards,

 

Christian 



Am 03.11.2020 um 21:25 schrieb Ernst Lobsiger via groups.io <ernst.lobsiger@...>:

 

Ferdinand

I have only SatPy V0.21 on Windows and V0.22 on GNU/Linux. Both without problems.
arpack and lapack are scientific libraries (linear algebra, solving Eigenvalue problems).

When i do in environment (pytroll) a "conda list" I only find "liblapack 3.9.0" (Linux) here.

Are you sure you installed your SatPy version 0.23 while you were in environment (pytroll)?

Regards
Ernst

 


Ernst Lobsiger
 

On Wed, Nov 4, 2020 at 01:46 AM, Ferdinand Valk wrote:

s there a way to force satpy 0.21 to be selected instead? I’ve tried to find that in the documentation but could not come up with an answer (likely because I looked at the wrong places).

A pointer would be very welcome.

 

Thanks,

Ferdinand

Ferdinand

Try "(pytroll) C:\users\fvalk\conda install satpy=0.21" . If you already installed 0.23 this should even downgrade.
It will probably take a while to sort things out. A "(pytroll) C:\users\fvalk\conda list" will show what's installed ...

Good luck
Ernst


Ferdinand Valk
 

Thanks Ernst,

 

That was indeed it. I had used ‘space’ whereas the ‘=’ is what was needed.

It is now installed and working again.

 

Thanks again,

Ferdinand

 

From: MSG-1@groups.io On Behalf Of Ernst Lobsiger via groups.io
Sent: Wednesday, 04 November, 2020 10:56
To: MSG-1@groups.io
Subject: Re: [MSG-1] Pytroll/SatPy for EUMETCast

 

On Wed, Nov 4, 2020 at 01:46 AM, Ferdinand Valk wrote:

s there a way to force satpy 0.21 to be selected instead? I’ve tried to find that in the documentation but could not come up with an answer (likely because I looked at the wrong places).

A pointer would be very welcome.

 

Thanks,

Ferdinand

Ferdinand

Try "(pytroll) C:\users\fvalk\conda install satpy=0.21" . If you already installed 0.23 this should even downgrade.
It will probably take a while to sort things out. A "(pytroll) C:\users\fvalk\conda list" will show what's installed ...

Good luck
Ernst


Ferdinand Valk
 

Just adding to my previous message. Now, using v0.21 of SatPy all issues I had encountered have gone away and I’ve successfully personalized most of the pyscript files.

I’m now starting to work out scheduling of automated tasks. The platform really offers a great versatility. A pleasant expansion of one’s horizon.

 

From: MSG-1@groups.io On Behalf Of Ferdinand Valk
Sent: Wednesday, 04 November, 2020 11:38
To: MSG-1@groups.io
Subject: Re: [MSG-1] Pytroll/SatPy for EUMETCast

 

Thanks Ernst,

 

That was indeed it. I had used ‘space’ whereas the ‘=’ is what was needed.

It is now installed and working again.

 

Thanks again,

Ferdinand

 

From: MSG-1@groups.io On Behalf Of Ernst Lobsiger via groups.io
Sent: Wednesday, 04 November, 2020 10:56
To: MSG-1@groups.io
Subject: Re: [MSG-1] Pytroll/SatPy for EUMETCast

 

On Wed, Nov 4, 2020 at 01:46 AM, Ferdinand Valk wrote:

s there a way to force satpy 0.21 to be selected instead? I’ve tried to find that in the documentation but could not come up with an answer (likely because I looked at the wrong places).

A pointer would be very welcome.

 

Thanks,

Ferdinand

Ferdinand

Try "(pytroll) C:\users\fvalk\conda install satpy=0.21" . If you already installed 0.23 this should even downgrade.
It will probably take a while to sort things out. A "(pytroll) C:\users\fvalk\conda list" will show what's installed ...

Good luck
Ernst


Ernst Lobsiger
 

Ferdinand

Good to hear it's working now. We should not forget that SatPy is still in full development:

*****************************************************************************************
* "SatPy's interfaces are not guaranteed stable and may change until *
* version 1.0 when backwards compatibility will be a main focus."         *
*****************************************************************************************

So sometimes the latest and greatest version breaks a couple of things. That was the case
when they renamed readers (in a more structured way), when they changed from Python
2.7.x (not supported any more) to 3.6 and later 3.8. We have to live with that. On the other
hand as you said, PyTROLL/SatPy is ultra versatile. My HOWTO only scratches the surface ...

Cheers,
Ernst


Christian Peters
 

FYI: The FY3D issue is fixed and should be available with the next release. 

Regards,

Christian 

Am 04.11.2020 um 18:08 schrieb Ernst Lobsiger via groups.io <ernst.lobsiger@...>:

Ferdinand

Good to hear it's working now. We should not forget that SatPy is still in full development:

*****************************************************************************************
* "SatPy's interfaces are not guaranteed stable and may change until *
* version 1.0 when backwards compatibility will be a main focus."         *
*****************************************************************************************

So sometimes the latest and greatest version breaks a couple of things. That was the case
when they renamed readers (in a more structured way), when they changed from Python
2.7.x (not supported any more) to 3.6 and later 3.8. We have to live with that. On the other
hand as you said, PyTROLL/SatPy is ultra versatile. My HOWTO only scratches the surface ...

Cheers,
Ernst


Ferdinand Valk
 

I encounter the following situation:

When I look at the script Noaa_20_VIS.py and compare it with
Suomi_NPP_VIS.py I cannot find any difference other than the segment source
file location and LTDN/LTAN satellite dependent values. The segment file
names have the same layout with the same offset used to determine start time
of segment.
Nevertheless, whereas Noaa_20 completes as desired, Suomi gets back with:

Traceback (most recent call last):
File "Suomi_NPP_VIS.py", line 125 in <module>
dt = datetime(int(Yea), int(Mon), int(Day), int(Hou), int(Min),
ValueError: second must be in 0..59

After this it closes back to (pytroll) C:\EMCtools\pyscripts>

TLE file is current. Both SatPy v0.21 and v0.22 show the same behaviour.
What am I overlooking?

Cheers,
Ferdinand


Ernst Lobsiger
 

On Fri, Nov 6, 2020 at 05:19 AM, Ferdinand Valk wrote:
names have the same layout with the same offset used to determine start time
of segment.
Ferdinand

There are some more differencies you might have overlooked like:
...
orb = Orbital('Suomi-NPP', tlefil)

# Glob by date, assume that interesting segments are all after midnigt
files = glob(segdir + "/" + "SVMC_npp_d" + Yea + Mon + Day + "*.h5")
...
I just made a Suomi_NPP_VIS image of yesterday without problems.

Regards,
Ernst


Ferdinand Valk
 

Ernst,

 

Those differences you mentioned had already been taken into account as well (Suomi-NPP in orbital and SVMC in filename). I really don’t see any other difference. What bothers me is that the error refers to the second in ‘dt’. See attached screenshot (in which it is also shown that yesterday Suomi was incomplete here but Nov 4 was OK).

Really puzzled...

 

Ferdinand

 

From: MSG-1@groups.io On Behalf Of Ernst Lobsiger via groups.io
Sent: Friday, 06 November, 2020 13:43
To: MSG-1@groups.io
Subject: Re: [MSG-1] Pytroll/SatPy for EUMETCast

 

On Fri, Nov 6, 2020 at 05:19 AM, Ferdinand Valk wrote:

names have the same layout with the same offset used to determine start time
of segment.

Ferdinand

There are some more differencies you might have overlooked like:
...
orb = Orbital('Suomi-NPP', tlefil)

# Glob by date, assume that interesting segments are all after midnigt
files = glob(segdir + "/" + "SVMC_npp_d" + Yea + Mon + Day + "*.h5")
...
I just made a Suomi_NPP_VIS image of yesterday without problems.

Regards,
Ernst


Ernst Lobsiger
 

Ferdinand

The line probably refers to:

        dt = datetime(int(Yea), int(Mon), int(Day), int(Hou), int(Min),
                      int(Sec)) + timedelta(seconds = 43)

Which is actually

        dt = datetime(int(Yea), int(Mon), int(Day), int(Hou), int(Min), int(Sec)) + timedelta(seconds = 43)

if written on one line.

The claim is that Sec is not in range 0..59  which means it's not properly taken from a filename
(or the above line is damaged somehow?). You can try to debug it the old way using print ():

...
    Hou = files[n][offset+20:offset+22]
    Min = files[n][offset+22:offset+24]
    Sec = files[n][offset+24:offset+26]
    print ("-------->", Hou, Min, Sec)
...
You can also try to print the filelist files if you see different behaviour on different days.
...
# Glob by date, assume that interesting segments are all after midnigt
files = glob(segdir + "/" + "SVMC_npp_d" + Yea + Mon + Day + "*.h5")
print ("-------->", files)


Or you can attach the script and the printed files to your reply if you cannot nail it down.

Regards
Ernst


Ferdinand Valk
 

Thanks for the suggestions Ernst. I had already put the full expression on a single line without change in behaviour. I’ll do some debugging first and check if anything wrong or incomplete with segment files (most likely tomorrow).

If it remains a dead end I’ll bother you with attachments :-)

 

Enjoy your evening

Ferdinand

 

From: MSG-1@groups.io On Behalf Of Ernst Lobsiger via groups.io
Sent: Friday, 06 November, 2020 15:53
To: MSG-1@groups.io
Subject: Re: [MSG-1] Pytroll/SatPy for EUMETCast

 

Ferdinand

The line probably refers to:

        dt = datetime(int(Yea), int(Mon), int(Day), int(Hou), int(Min),
                      int(Sec)) + timedelta(seconds = 43)

Which is actually

        dt = datetime(int(Yea), int(Mon), int(Day), int(Hou), int(Min), int(Sec)) + timedelta(seconds = 43)

if written on one line.

The claim is that Sec is not in range 0..59  which means it's not properly taken from a filename
(or the above line is damaged somehow?). You can try to debug it the old way using print ():

...
    Hou = files[n][offset+20:offset+22]
    Min = files[n][offset+22:offset+24]
    Sec = files[n][offset+24:offset+26]
    print ("-------->", Hou, Min, Sec)
...
You can also try to print the filelist files if you see different behaviour on different days.
...
# Glob by date, assume that interesting segments are all after midnigt
files = glob(segdir + "/" + "SVMC_npp_d" + Yea + Mon + Day + "*.h5")
print ("-------->", files)


Or you can attach the script and the printed files to your reply if you cannot nail it down.

Regards
Ernst


Graham Woolf
 

Hi Ernst

I am trying to use a Blue Marble tif file as a background for a composite but Im not sure where I need to but the tif file for it to be read by satpy

Can you tell me in which directory I need the file - its in ppp_config at the moment and its not being picked up

Many thanks

Graham