Dear PyTROLL/Satpy users,

here comes version 4.0 of my Satpy scripts for EUMETCast.
I call these "Satpy Processing System" (SPS) and you name
it: There are two directories C:\SPStools and D:\SPSdata.

It's easy to rename those but I have chosen them that way to
make it possible to test the scripts at the side of a working
version 3.0 with C:\EMCtools and D:\EMCdata. As the new scripts
are radically different you better don't mix everything up.

This is for advanced users that have a working (pytroll)
environment. You must also update to Pycoast 1.6.1 and
Satpy 0.38.0 that came out recently. SPS comes with many
MSLP overlays and produces *.webm movies that do directly
play in any modern browser. This distribution is a snapshot
of two working processing PCs. Take it as an unsupported
Beta. I will not answer questions that are explained in the
HOWTO V 4.0. If a couple of Windows 10 users can get that
running I might think about a better way to distribute it.
All Windows 10 and GNU/Linux Python scripts are identical.
The only difference lies in the shell specific OS helper scripts.

Success reports and reports of reproducible errors are welcome.

Have fun,
Ernst


GNU/Linux amd64 (tgz),  wetransfer link valid 1 week, app. 670MB

https://we.tl/t-JqVEEyknmL
     

Windows10 64Bit (zip),  wetransfer link valid 1 week, app. 670MB

https://we.tl/t-e9uOASj7eq

Ernst,

On 12/11/2022 20:39, Ernst Lobsiger via groups.io wrote:

Dear PyTROLL/Satpy users,
here comes version 4.0 of my Satpy scripts for EUMETCast.
I call these "Satpy Processing System" (SPS) and you name
it: There are two directories C:\SPStools and D:\SPSdata.
It's easy to rename those but I have chosen them that way to
make it possible to test the scripts at the side of a working
version 3.0 with C:\EMCtools and D:\EMCdata. As the new scripts
are radically different you better don't mix everything up.
This is for advanced users that have a working (pytroll)
environment. You must also update to Pycoast 1.6.1 and
Satpy 0.38.0 that came out recently. SPS comes with many
MSLP overlays and produces *.webm movies that do directly
play in any modern browser. This distribution is a snapshot
of two working processing PCs. Take it as an unsupported
Beta. I will not answer questions that are explained in the
HOWTO V 4.0. If a couple of Windows 10 users can get that
running I might think about a better way to distribute it.
All Windows 10 and GNU/Linux Python scripts are identical.
The only difference lies in the shell specific OS helper scripts.
Success reports and reports of reproducible errors are welcome.
Have fun,
Ernst

=========================================================================

Once again Ernst many thanks for all your efforts with this. Very much appreciated. Now downloaded.
I hope to get this all set up this coming week (if my house decorating gets completed) and will report how it goes.

Best regards,
Douglas.

Hi Ernst,

A very great work! Thank you very much.

Probably, I will install and try this new scripts when I will buy a new computer for my weather observatory.

A lot of Thanks!

Cheers,
Samuele

On Sun, Nov 13, 2022 at 01:25 AM, g-woolf@... wrote:

Has the snow-age composite been fixed in the latest Satpy version ?

 

Graham,

yes 'snow_age' works. With a little help from me Dave found the culprit

https://github.com/pytroll/satpy/pull/2190

... an older fix in a PR that just didn't make it into version Satpy 0.37.1.

Ernst

P.S. I just noticed that there are still some 'ETCtools' in a few script comments.
Don't be fooled: This is a big package and I renamed stuff more than once ...

Ernst, just to confirm my first image using the new SPS developed by yourself.
Huge amounts still to be done, but I have now grasped the underlying principles after quite a few false starts.
Plenty of fun ahead.
Do agree though that this will be difficult if you have not played about with version 3.
Now back to my decorating !!

May I be allowed one question. When does the output go into 'frames' folder and when 'image' folder.

Thanks again for all your work.

Best regards,
Douglas.

On Sun, Nov 13, 2022 at 06:58 AM, Douglas Deans wrote:

May I be allowed one question. When does the output go into 'frames' folder and when 'image' folder.

Douglas,

Well done. You are even faster than Graham! That's breath taking ...


Per default the scripts output to ...\products\sat (sat = satellite short name like MSG4 or MetopB).

This is what you find at the end of a basic LEO script:
...
    magick = get_magick(Yea, Mon, Day, Hou, Min, height, logo1, logo2, service, channel, receiver,
                        sat, sensor, composite, area, testrun, NoD, multi,
                        subdir=NoD)  # basedir=basedir, subdir=subdir

This is what you find at the end of a basic GEO script
...
   magick = get_magick(Yea, Mon, Day, Hou, Min, height, logo1, logo2, service, channel,
                        receiver, sat, sensor, composite, area, testrun)  # basedir=basedir, subdir=subdir

As the # comment tries to explain these functions take optional "keyword" arguments "basedir" and "subdir" to fine tune the output.

Now frames are images that are normally low resolution because these are used as movie frames.
I decided there is no point in making 5500 x 5500 pixel images every 10 minutes and finally watch
movies on my FullHD (1920 x 1080) screen. That's why I made special areas for images that will serve as
movie frames. I output these to ..., subdir='frames') and the webm.cmd script will expect them there.
But of course if you have a 4K UHD whatever monitor you might want to increase the movie resolution.
Other GEO images, especially full resolution and maybe hourly images, I send to ..., subdir='image').
There is even a possibility to make movies from subdir images where ffmpeg will reduce these on the fly.

There is a special case of GEO images with MSLP overlays. I output them to ..., basedir='MSLP', subdir=area)
LEO images I send to subdirs 'DAY' or 'NIG' but again it's up to you to have them all in subdir \sat only.

That said when setting up the system to your own needs it's up to you where you
finally want to put your results inside directory Drive:\SPSdata\products.


Best regards,
Ernst

On 13/11/2022 16:39, Ernst Lobsiger via groups.io wrote:

On Sun, Nov 13, 2022 at 06:58 AM, Douglas Deans wrote:
May I be allowed one question. When does the output go into 'frames'
folder and when 'image' folder.
Douglas,
Well done. You are even faster than Graham! That's breath taking ...
Per default the scripts output to ...\products\sat (sat = satellite short name like MSG4 or MetopB).
This is what you find at the end of a basic LEO script:
...
    magick = get_magick(Yea, Mon, Day, Hou, Min, height, logo1, logo2, service, channel, receiver,
                        sat, sensor, composite, area, testrun, NoD, multi,
                        subdir=NoD)  # basedir=basedir, subdir=subdir
This is what you find at the end of a basic GEO script
...
   magick = get_magick(Yea, Mon, Day, Hou, Min, height, logo1, logo2, service, channel,
                        receiver, sat, sensor, composite, area, testrun)  # basedir=basedir, subdir=subdir
As the # comment tries to explain these functions take optional "keyword" arguments "basedir" and "subdir" to fine tune the output.
Now frames are images that are normally low resolution because these are used as movie frames.
I decided there is no point in making 5500 x 5500 pixel images every 10 minutes and finally watch
movies on my FullHD (1920 x 1080) screen. That's why I made special areas for images that will serve as
movie frames. I output these to ..., subdir='frames') and the webm.cmd script will expect them there.
But of course if you have a 4K UHD whatever monitor you might want to increase the movie resolution.
Other GEO images, especially full resolution and maybe hourly images, I send to ..., subdir='image').
There is even a possibility to make movies from subdir images where ffmpeg will reduce these on the fly.
There is a special case of GEO images with MSLP overlays. I output them to ..., basedir='MSLP', subdir=area)
LEO images I send to subdirs 'DAY' or 'NIG' but again it's up to you to have them all in subdir \sat only.
That said when setting up the system to your own needs it's up to you where you
finally want to put your results inside directory Drive:\SPSdata\products.
Best regards,
Ernst

===========================================================================================

Fully understood Ernst. Still much to learn with animations top of my list but I suspect that will take time.

Thanks again.
Best regards,
Douglas.

On Sun, Nov 13, 2022 at 11:12 AM, Douglas Deans wrote:

Fully understood Ernst. Still much to learn with animations top of my list but I suspect that will take time.

Douglas,

for animations David's all integrated software is certainly easier to use ;-). On the
the other hand SPS can define animations in many map projection and composites.
SPS uses Satpy to make frames first and there is no point to do that without using
a scheduler (you cannot start an MSG3 rss frame script every 5 minutes and even
make no typos in datetime). Once you have a couple of frames you can assemble
those with ffmpeg. I already see what's missing now. I must provide a simple case
to schedule just frames for ONE animation and to manually start just an ffmpeg
encoding for that animation. Instead I packed cron.cmd that schedules Satpy for
about everything and webm.cmd that makes all the movies defined every hour.

Maybe it's easier if you begin with my MSLP helper scripts first. You can start
these with a click and see in overlays what has been downloaded and prepared.
Then you can manually start the corresponding MSLP scripts ...


Best regards,
Ernst

On 13-11-2022 21:10, Ernst Lobsiger via groups.io wrote:

... there is no point to do that without using
a scheduler (you cannot start an MSG3 rss frame script every 5 minutes and even
make no typos in datetime).

Instead, you can use a watchdog instead of a scheduler. If all files are available start Satpy-script and then add the new frame to the movie. Works fine here. Watchdog is supported by Python and works on both Linux and Windows. See for an example:

http://www.alblas.demon.nl/wsat/software/satpy.html#live%20movie

(some adaptions needed to fit it with the new script set of Ernst)

Rob.

On Sun, Nov 13, 2022 at 01:12 PM, R. Alblas wrote:

Instead, you can use a watchdog instead of a scheduler.

Rob,

there are a couple of solutions using inotify() already provided by PyTROLL:

https://trollduction.readthedocs.io/en/latest/configuration.html

https://github.com/pytroll/pytroll-collectors

But this is way out of  most peoples comfort zone. Therefore my very simple
approach that entirely relies on timeliness. It has worked quite well so far ...

Regards,
Ernst

Hi Ernst

My first test images with your new scripts

Regards

Graham

Ernst,

This PyTROLL-stuff is indeed complex. But I didn't refer to this; I just use 'watchdog', part of standard Python libs. Much easier. I just wanted to note that there is another way besides using a scheduler.

Regards

Rob.

@Rob
Yes, a watchdog is certainly a possibility. But I personally don't like scripts that loop forever. And I wanted to keep everything as close as possible to my GNU/Linux version where I use cron. It should be easy to setup a schedule to start cron.cmd every minute as I even provide a CRON.xml that you can import into the Windows 10 scheduler.

@Douglas
Yes, cron.cmd (that does just about everything) is too heavy to begin with. I attach a super commented slimmed down script cron-explained.txt below. You can read and rename it to cron-explained.cmd and then click it to learn in smaller steps. It makes MSG3 frames for isomrss and oaserss animations. The respective scripts MSG3-isomrss.py and MSG3-oaserss.py in subdirectory GEOscripts must first be made ready to work of course.

Good luck,
Ernst

On 14/11/2022 10:25, Ernst Lobsiger via groups.io wrote:

@Rob
Yes, a watchdog is certainly a possibility. But I personally don't like scripts that loop forever. And I wanted to keep everything as close as possible to my GNU/Linux version where I use cron. It should be easy to setup a schedule to start cron.cmd every minute as I even provide a CRON.xml that you can import into the Windows 10 scheduler.
@Douglas
Yes, cron.cmd (that does just about everything) is too heavy to begin with. I attach a super commented slimmed down script cron-explained.txt below. You can read and rename it to cron-explained.cmd and then click it to learn in smaller steps. It makes MSG3 frames for isomrss and oaserss animations. The respective scripts MSG3-isomrss.py and MSG3-oaserss.py in subdirectory GEOscripts must first be made ready to work of course.
Good luck,
Ernst

=============================================================================

Much appreciated Ernst. Thanks for your help.

Kind regards,
Douglas.

Douglas,

in cron-explained.cmd above I wrote that we should not start
MSG3-oaserss.py in the same minute as MSG3-isomrss.py.
...
call :GEOmin  1  5 %GEO%\MSG3-isomrss.py  -6
call :GEOmin  2  5 %GEO%\MSG3-oaserss.py  -7
...
After a 1h walk in some fresh air I realized that this is
the GNU/Linux point of view because cron will start these
processes in parallel. Though flock() will prevent disaster
we should omit possible race conditions as far as possible.

Under Windows 10 with my cron.cmd the two lines below are not
...
call :GEOmin  1  5 %GEO%\MSG3-isomrss.py  -6
call :GEOmin  1  5 %GEO%\MSG3-oaserss.py  -6
...
only possible but might even be an advantage performance wise.
This is because the scripts are called in sequence and we will
not "loose" CPU time if the first line only takes 30 seconds. So
the real limit here is that all MSG3 rss scripts that use the data of
one slot must have produced their frames/images within 5 minutes.

There is nothing wrong with the distributed cron.cmd that has been
tested many days here. On the other hand nobody should be astonished
if two or even three cron.cmd will work in parallel under the Windows
scheduler. If the cron.cmd for minute 1 needs more than one minute to
do its different jobs then cron.cmd for minute 2 has already started!

Cheers,
Ernst

On 14/11/2022 18:38, Ernst Lobsiger via groups.io wrote:

Douglas,
in cron-explained.cmd above I wrote that we should not start
MSG3-oaserss.py in the same minute as MSG3-isomrss.py.
...
call :GEOmin  1  5 %GEO%\MSG3-isomrss.py  -6
call :GEOmin  2  5 %GEO%\MSG3-oaserss.py  -7
...
After a 1h walk in some fresh air I realized that this is
the GNU/Linux point of view because cron will start these
processes in parallel. Though flock() will prevent disaster
we should omit possible race conditions as far as possible.
Under Windows 10 with my cron.cmd the two lines below are not
...
call :GEOmin  1  5 %GEO%\MSG3-isomrss.py  -6
call :GEOmin  1  5 %GEO%\MSG3-oaserss.py  -6
...
only possible but might even be an advantage performance wise.
This is because the scripts are called in sequence and we will
not "loose" CPU time if the first line only takes 30 seconds. So
the real limit here is that all MSG3 rss scripts that use the data of
one slot must have produced their frames/images within 5 minutes.
There is nothing wrong with the distributed cron.cmd that has been
tested many days here. On the other hand nobody should be astonished
if two or even three cron.cmd will work in parallel under the Windows
scheduler. If the cron.cmd for minute 1 needs more than one minute to
do its different jobs then cron.cmd for minute 2 has already started!
Cheers,
Ernst

==================================================================================

Thanks Ernst. All noted.
At the moment I am working through my GEO and LEO command files and .py files with surprisingly (for me) good results so far.
Time spent on the last project (3) is now paying dividends. Once that is complete big effort at animations.

Best regards,
Douglas.