Date   

Meteosat-11 SEVIRI decontamination - 17 to 24 January 2022 - a reminder

David J Taylor GM8ARV 🏴󠁧󠁢󠁳󠁣󠁴󠁿 🇪🇺
 

A Meteosat-11 SEVIRI decontamination is planned from 17 to 24 January 2022.

During the Meteosat-11 decontamination period, no Meteosat-11 data will be
available. Meteosat-9 Full Earth Scan (FES) data (rectified to 0°) will be
disseminated instead, and also one week before and one week after Meteosat-11
decontamination, i.e. from 10/01 to 31/01/2022.

The parallel data from the non-prime FES spacecraft will be distributed on
EUMETCast Europe, on EUMETSAT Data Channel 15 (PID: 500, Multicast address:
224.223.222.3). The parallel data will have ‘_PAR’ included in their filenames.

More details:
https://www.eumetsat.int/upcoming-meteosat-11-seviri-decontamination-2022

Users wishing to receive the parallel Meteosat data streams
should register with the User Service Helpdesk now!

Cheers,
David
--
SatSignal Software - Quality software for you
Web: https://www.satsignal.eu
Email: david-taylor@...
Twitter: @gm8arv


Re: Using begiiners guide to PyTROLL/SatPy Starter Kit V3.0 for Windows10 64bit

Andreas Mueller
 

@Ernst,

I might consider to reduce those 50 EUMETCast users to 30.

just a comment from my side - please continue with your great work on providing Sat-Image processing in a rock solid straight forward way (on Linux) !
When reading a little bit through Ernst's comments (and as well in his scripts) its absolutely no problem to e.g. adjust the File/Directory structure, adjust the Layout of the processed Image, adjust the visuals in Images (such as Boarders, Cities, Rivers, Font Families, Colors and many more things ...) . There is no need of any 'File'-Mangers etc. as managing of files can done in a consistent way with OS included tools (on Linux). Also visualizing / organizing / maintaining of processed images can be done via webbased tools so no 'additional' software is needed on local PCs to access them. In addition (when using Linux) powerful visualization tools (such as e.g. Grafana) are avaiable without the need to fiddle around in config files etc. Last but not least system up times counting in months can be expected.

Andreas


Re: Using begiiners guide to PyTROLL/SatPy Starter Kit V3.0 for Windows10 64bit

Ernst Lobsiger
 

On Mon, Jan 3, 2022 at 10:57 AM, David J Taylor GM8ARV 🏴󠁧󠁢󠁳󠁣󠁴󠁿 🇪🇺 wrote:
Ernst,

I thought that I had already provided that with the "Save in EPS". That is the
original data, but perhaps your software needs to accept a wider range of file
names? Perhaps if they can provide more details I can see what can be done?

Cheers,
David
David,

That's in the comment header of my Metop-B Satpy script. Advanced users will understand later how dates and times are extracted.
Most users will just skip this comment that says, that EUMETSAT is sending us *.bz2 compressed files with the nameing explained.

# File Pattern : AVHR_xxx_1B_{platform_short_name}_{start_time:%Y%m%d%H%M%SZ}_{end_time:%Y%m%d%H%M%SZ}_{processing_mode}_{disposition_mode}_{creation_time:%Y%m%d%H%M%SZ}
# Example name : AVHR_xxx_1B_M01_20190326085203Z_20190326085503Z_N_O_20190326103042Z
# Hugo's ruler : 0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012
# copied EMCV  : 0         1         2         3         4         5         6         7         8         9         10
# EUMETCast    : These files arrive as *.bz2 in channel EPS-10 (Basic Service)

That said my script expects the Metop-B files like

AVHR_xxx_1B_M01_20190326085203Z_20190326085503Z_N_O_20190326103042Z.bz2

that we get in BAS channel EPS-10. These files will be decompressed so that the Satpy reader can read them.

Neither my script nor the Satpy reader understands the files your Metop manager saves as raw data with naming

AVHR_xxx_1B_M01_20211217004903Z_20211217005203Z_N_O_20211217012222Z.eps_glb

and maybe even uncompressed (?). It is possible (as Christian and Hartmuth have demonstrated) to tweek the underlying
Satpy infrastructure and possibly adapt my scripts to accept your files. But this is out of reach for 99% of PyTROLL users.
Graham Woolf has explained some posts ago what kind of botch he has done up to now only to get the files as expected!


Hope that helps
Ernst


Re: Using begiiners guide to PyTROLL/SatPy Starter Kit V3.0 for Windows10 64bit

David J Taylor GM8ARV 🏴󠁧󠁢󠁳󠁣󠁴󠁿 🇪🇺
 

On 03/01/2022 16:58, Ernst Lobsiger via groups.io wrote:
I might consider to reduce those 50 EUMETCast users to 30.
But I will certainly *not* write this GEO Quaterly article myself.

Cheers,
Ernst
That's why I was suggesting someone less familiar with the software, and more
familiar with the problems!

David
--
SatSignal Software - Quality software for you
Web: https://www.satsignal.eu
Email: david-taylor@...
Twitter: @gm8arv


Re: Using begiiners guide to PyTROLL/SatPy Starter Kit V3.0 for Windows10 64bit

David J Taylor GM8ARV 🏴󠁧󠁢󠁳󠁣󠁴󠁿 🇪🇺
 

On 03/01/2022 16:23, Ernst Lobsiger via groups.io wrote:
David,

I understand that your "managers" are programs that make images and maybe
animations. But the problem Christian, Graham, Hartmuth and now Philip face is
that they cannot run your Metop manager (s?) and get the raw satellite data
saved with original format and naming. This prevents a straight forward
solution to use PyTROLL/Satpy that works with the original EUMETCast data and
must be tweeked in a geeky way to undertsnd the EPS files your manager finally
lets through.

Regards,
Ernst
Ernst,

I thought that I had already provided that with the "Save in EPS". That is the
original data, but perhaps your software needs to accept a wider range of file
names? Perhaps if they can provide more details I can see what can be done?

Cheers,
David
--
SatSignal Software - Quality software for you
Web: https://www.satsignal.eu
Email: david-taylor@...
Twitter: @gm8arv


Re: Using begiiners guide to PyTROLL/SatPy Starter Kit V3.0 for Windows10 64bit

Philip Robinson
 

David
Thanks for considering me to write in the GEO Quarterly. But I am still very very much a beginner and i need a lot more experience before i would be confident enough.
 I love your software, the ones i have are so easy to work with ,but i felt i needed to expand my knowledge and learn some  basic scripting.
regards
Philip

------ Original Message ------
From: "Ernst Lobsiger via groups.io" <ernst.lobsiger@...>
Sent: 03/01/2022 16:58:37
Subject: Re: [MSG-1] Using begiiners guide to PyTROLL/SatPy Starter Kit V3.0 for Windows10 64bit

David,

these are the last few lines of my SatPy_HOWTO_V30.txt

<cite>
**************************************************************************************
Windows Pytroll/Satpy HOWTO V 3.0. I will not consider a next release until at least
50 Windows or GNU/Linux EUMETCast users report success with PyTroll/SatPy. And like
David Taylor I'd love to read a PyTroll/SatPy article in the GEO Quarterly journal.
But this must be written in clean English by a Windows 10 EUMETCast user in the U.K..

Belp, Oase 2021/01/08                                                   Ernst Lobsiger
**************************************************************************************
</cite>

I might consider to reduce those 50 EUMETCast users to 30.
But I will certainly *not* write this GEO Quaterly article myself.

Cheers,
Ernst

Virus-free. www.avg.com


Re: Using begiiners guide to PyTROLL/SatPy Starter Kit V3.0 for Windows10 64bit

Ernst Lobsiger
 

David,

these are the last few lines of my SatPy_HOWTO_V30.txt

<cite>
**************************************************************************************
Windows Pytroll/Satpy HOWTO V 3.0. I will not consider a next release until at least
50 Windows or GNU/Linux EUMETCast users report success with PyTroll/SatPy. And like
David Taylor I'd love to read a PyTroll/SatPy article in the GEO Quarterly journal.
But this must be written in clean English by a Windows 10 EUMETCast user in the U.K..

Belp, Oase 2021/01/08                                                   Ernst Lobsiger
**************************************************************************************
</cite>

I might consider to reduce those 50 EUMETCast users to 30.
But I will certainly *not* write this GEO Quaterly article myself.

Cheers,
Ernst


Re: Using begiiners guide to PyTROLL/SatPy Starter Kit V3.0 for Windows10 64bit

Ernst Lobsiger
 

David,

I understand that your "managers" are programs that make images and maybe animations. But the problem Christian, Graham, Hartmuth and now Philip face is that they cannot run your Metop manager (s?)  and get the raw satellite data saved with original format and naming. This prevents a straight forward solution to use PyTROLL/Satpy that works with the original EUMETCast data and must be tweeked in a geeky way to undertsnd the EPS files your manager finally lets through.

Regards,
Ernst


Re: Using begiiners guide to PyTROLL/SatPy Starter Kit V3.0 for Windows10 64bit

David J Taylor GM8ARV 🏴󠁧󠁢󠁳󠁣󠁴󠁿 🇪🇺
 

Philip, Christian, Graham, Ernst,

Now you all have this Kit installed and working, it would be helpful to others
if a summary or more detailed article could be made available in the GEO
Quarterly journal. Some of the results produced are very good, but it's clear
that it's not yet quite plug and play, so your experiences could be valuable.

As a software author myself, I know that we are not the best to write guides,
as we don't appreciate the problems which beginners face, perhaps due to lack
of familiarity with either the OS or its less used functions.

I would be happy to proof-read and make suggestions, and Les Hamilton can help
where appropriate.

Do you think this is a good idea - a guide to the kit - written from a user's
viewpoint? It might help answer many of the points raise in the recent rather
long discussions!

Thanks,
David
--
SatSignal Software - Quality software for you
Web: https://www.satsignal.eu
Email: david-taylor@...
Twitter: @gm8arv


Re: Using begiiners guide to PyTROLL/SatPy Starter Kit V3.0 for Windows10 64bit

David J Taylor GM8ARV 🏴󠁧󠁢󠁳󠁣󠁴󠁿 🇪🇺
 

On 03/01/2022 15:13, Ernst Lobsiger via groups.io wrote:
Apparently to use satsignal software you have to run a couple of "managers"
that let or don't let you see the original files.
Ernst,

To clarify, it's the user's choice. Most users do not want the source files
cluttering up their hard disks, so making it optional is the best choice. They
may want some files from a particular channel, and not others, and this gets
worse as more data types are available

My Managers do a lot more than managing the files. They convert the encoded
data into JPG or PNG files which allow users direct access to the images with
standard, widely-available software, and they display the images directly on
the user's screen in both thumbnail and full-size format. Further software can
directly animate the data for real-time display, or reproject the data to
combine multiple channels and add overlays in the native resolution of the
overlay. No extra scripts required.

It's a lot, lot simpler than PyTROLL/SatPy, but your efforts in supporting
these users are most welcome, as they may be want to use those tools for the
third generation satellites when they arrive - launch is planned later this
year - so having well-documented kits available is most helpful.

David
--
SatSignal Software - Quality software for you
Web: https://www.satsignal.eu
Email: david-taylor@...
Twitter: @gm8arv


Re: Using begiiners guide to PyTROLL/SatPy Starter Kit V3.0 for Windows10 64bit

nigel
 

Wonderful news, Philip - well done for persevering and getting a good result.

Onwards and upwards !!

Nigel

On 02/01/2022 21:31, Philip Robinson wrote:

Ernst / Graham / David
Joy oh Joy.
I have an image in my EMCData/Meteosat 11 folder.
I can just hear you now saying "about b-----y time"
So much confusion i was having with the forward and backward slashes
Sometimes you need / with windows and then \ with Python etc etc
Attached is the final outputted message details in Anaconda Prompt after running python MSG4.py 202201021430 and my first image produced
I think i will now spend some time in future days learning a bit of processing these images by editing the MSG4.py script. I will do a copy of MSG4.py  just in case i mess up

But as Ben Richards said "I`ll be back"

Thanks to all. May be some of these messages will be useful to someone else starting from scratch with PyTROLL/SatPy in a windows environment.

thanks again 
Philip Robinson
 
------ Original Message ------
From: "Ernst Lobsiger via groups.io" <ernst.lobsiger@...>
Sent: 02/01/2022 18:09:13
Subject: Re: [MSG-1] Using begiiners guide to PyTROLL/SatPy Starter Kit V3.0 for Windows10 64bit

Phillip,

ONCE AGAIN: Use forward slashes '/' in segdir!

Then if you sit in directory USRscripts with prompt:
(pytroll) c:\EMCtools\USRscripts>

You start your script with typing:

python MSG4.py 202201021430

Ernst

Virus-free. www.avg.com


Re: Using begiiners guide to PyTROLL/SatPy Starter Kit V3.0 for Windows10 64bit

Ernst Lobsiger
 

Dear all,

my Starter Kit just expects to find the satellite raw data the way they come out of the EUMETCast receiver. It does not do any data management and also works over the network from read only archives on a receiver or server. Unlike others, I leave my raw data on the receiver where they usually stay from 3 to 10 days depending on HDD size and received EUMETCast channels. This way I can also reprocess the same data several times with special composites or map projections. For historical reasons other programs do their own data management. Apparently to use satsignal software you have to run a couple of "managers" that let or don't let you see the original files. The most simple method to keep track of processed raw data is to delete them or move them out to another place. When I used Xrit2pic it moved processed files to a directory "done". If a managing program is in full control there are even OS system calls for Windows and GNU/Linux (sorry I forgot the function name) to wait for specific files and trigger processing as soon as those files arrive. E.g. the last file after each Seviri HRIT slot is an EPI-logue file. If this file comes in, you know the whole slot files must be there. This way you can process the files as fast as possible. This system call is even more useful if you wait for LEO Aqua/Terra data where the "timeliness" (the time between the satellite pass overhead and the time you get those data from EUMETCast) can vary from 2 to 6 hours. IIRC Rob's Xrit2pic and Hugo's EUMETCastWatcher use this described OS system call. I have no idea what MDM and what not of David's other managers do. But the point is: You can for sure only have one data manager that does that kind of geek stuff behind the scene. These are the problems EUMETcast users face that want to run and use Satsignal software, Hugo's EUMETCastView, my PyTROLL/Sapty scripts and maybe even Xrit2pic at the same time. The Metop hack presented by Christian and Hartmuth is a geek workaround but nothing to be integrated into a "Starter Kit". That said, I will NEVER try to make my scripts run with what not manager you use and it's also clear that this is mainly a Windows problem as GNU/Linux users will find their own workarounds anyway. There is only one solution I see for this: If you use your EUMETCast clients with one RAM disk, you have to transfer in a seperate process all data to a HDD or SDD. In this step you can make copies for David's and Hugo's programs and still transfer the data as is for PyTROLL/Satpy. You can even choose to get the data on HDD in a bulk or YYYY\MM\DD directory structure. This does not mean that you have to pysically copy the data three times. If everthing goes to the same partition (Windows drive letter like C: or D:) you can use so called hard links (or even soft links) and have to copy the actual data from your RAM disk only once.

I know that Alan Curnow has made such program in C++ with a nice Windows GUI. He would certainly deserve a special corner in the "EUMETCast user's Hall of Fame" if he could make this solution available to others. If put under a license like the GPL3 it's up to the author, whether and to what extent he wants to further support it later and e.g. answer annoying questions.


Cheers,
Ernst


Re: Using begiiners guide to PyTROLL/SatPy Starter Kit V3.0 for Windows10 64bit

David J Taylor GM8ARV 🏴󠁧󠁢󠁳󠁣󠁴󠁿 🇪🇺
 

On 03/01/2022 10:05, Graham Woolf wrote:
Good Morning Philip

There is an issue with David's Metop Manager in that there isnt an option to
save the raw data to a specified location like there is in the MSG Data Manager
and I dont think there ever will be
[]
Regards

Graham
Graham,

If you want files saved in EPS format just tick that box!
Easy!

Cheers,
David
--
SatSignal Software - Quality software for you
Web: https://www.satsignal.eu
Email: david-taylor@...
Twitter: @gm8arv


Re: Using begiiners guide to PyTROLL/SatPy Starter Kit V3.0 for Windows10 64bit

Hartmut Schulla
 

Hi!

Attached the actual image from METOP-C, created with these scripts.

Regards

Hartmut

 

Von: MSG-1@groups.io <MSG-1@groups.io> Im Auftrag von Graham Woolf
Gesendet: Montag, 3. Jänner 2022 12:57
An: MSG-1@groups.io
Betreff: Re: [MSG-1] Using begiiners guide to PyTROLL/SatPy Starter Kit V3.0 for Windows10 64bit

 

Hi Hartmut

Thanks very much for that - I wasnt aware you could do that.

Its a much neater and easier solution

Kind Regards

Graham


Re: Using begiiners guide to PyTROLL/SatPy Starter Kit V3.0 for Windows10 64bit

Graham Woolf
 

Hi Hartmut

Thanks very much for that - I wasnt aware you could do that.

Its a much neater and easier solution

Kind Regards

Graham


Re: Using begiiners guide to PyTROLL/SatPy Starter Kit V3.0 for Windows10 64bit

Hartmut Schulla
 

Hi!

Attached a working sample (for Metop-B) and the required reader, adapted for eps-files.

Read the code carefully and change the paths to your requirements.

Regards

Hartmut

 

Von: MSG-1@groups.io <MSG-1@groups.io> Im Auftrag von Christian Peters via groups.io
Gesendet: Montag, 3. Jänner 2022 11:49
An: MSG-1@groups.io
Betreff: Re: [MSG-1] Using begiiners guide to PyTROLL/SatPy Starter Kit V3.0 for Windows10 64bit

 

Graham

there is an option to save raw files in Metop manager. You have to tick "Save data in EPS format".
This saves the raw data in the EPS-Global folder as "AVHR_xxx_1B_M01...... .epg-glb").

If you tweak the avhrr_l1b_eps.yaml file in Satpy to recognize files like
AVHR_xxx_1B_M01_20211217004903Z_20211217005203Z_N_O_20211217012222Z.eps_glb you can use
this files with Satpy.

Maybe Ernst could add the tweak into his next release, you just have to add a reader dir in your
SATPY_CONFIG_PATH, and the place the tweaked avhrr_l1b_eps.yaml files there.
Then the files stored by Metop manager will be recognized.

Regards,

Christian

Am 03.01.22 um 11:05 schrieb Graham Woolf:

Good Morning Philip

There is an issue with David's Metop Manager in that there isnt an option to save the raw data to a specified location like there is in the MSG Data Manager and I dont think there ever will be

However if you  untick Delete Files from RX in the Metop Manager then it will continually reprocess all the files again and again - it doesnt seem to recognise which files have been processed for that day and which havent so it just does the whole lot over again

The solution I found is to set up a process that watches the EPS-10 directory and as soon as new files come in it copies them to a specified location. It does this before the Metop Manager processes them which means I can leave Delete Files from RX ticked . However for my solution it does involve extra software to monitor the EPS-10 directory

Its just something to be aware of if you decide to use pytroll for Metop images

It doesnt seem to be a problem with Himawari ,GOES16/17s as they do have an option to save the raw data

I dont know why the Metop Manager is different

Regards

Graham

 


Re: Using begiiners guide to PyTROLL/SatPy Starter Kit V3.0 for Windows10 64bit

Christian Peters
 

Graham

there is an option to save raw files in Metop manager. You have to tick "Save data in EPS format".
This saves the raw data in the EPS-Global folder as "AVHR_xxx_1B_M01...... .epg-glb").

If you tweak the avhrr_l1b_eps.yaml file in Satpy to recognize files like
AVHR_xxx_1B_M01_20211217004903Z_20211217005203Z_N_O_20211217012222Z.eps_glb you can use
this files with Satpy.

Maybe Ernst could add the tweak into his next release, you just have to add a reader dir in your
SATPY_CONFIG_PATH, and the place the tweaked avhrr_l1b_eps.yaml files there.
Then the files stored by Metop manager will be recognized.

Regards,

Christian


Am 03.01.22 um 11:05 schrieb Graham Woolf:

Good Morning Philip

There is an issue with David's Metop Manager in that there isnt an option to save the raw data to a specified location like there is in the MSG Data Manager and I dont think there ever will be

However if you  untick Delete Files from RX in the Metop Manager then it will continually reprocess all the files again and again - it doesnt seem to recognise which files have been processed for that day and which havent so it just does the whole lot over again

The solution I found is to set up a process that watches the EPS-10 directory and as soon as new files come in it copies them to a specified location. It does this before the Metop Manager processes them which means I can leave Delete Files from RX ticked . However for my solution it does involve extra software to monitor the EPS-10 directory

Its just something to be aware of if you decide to use pytroll for Metop images

It doesnt seem to be a problem with Himawari ,GOES16/17s as they do have an option to save the raw data

I dont know why the Metop Manager is different

Regards

Graham


Re: Using begiiners guide to PyTROLL/SatPy Starter Kit V3.0 for Windows10 64bit

Philip Robinson
 


Hi Graham
Thanks for informing me about Metop Manager. I will print your email out and file it with  lots of paperwork involving this satellite hobby. When i try using pytroll for Metop images, that info i am sure will be useful.
I can see i will have much to learn, even some of the basics using and processing images, in pytroll. But at least i have a start, and now using what i have learnt so far, i can produce images using PyTroll/SatPy. And its no longer a complete mystery when it crops up in the msg forums. One of the reasons i took up this hobby is that all my life i have been very interested in meteorology, climate etc, and this is another outlet for that. 
regards
Philip

------ Original Message ------
From: "Graham Woolf" <woolfg@...>
Sent: 03/01/2022 10:05:14
Subject: Re: [MSG-1] Using begiiners guide to PyTROLL/SatPy Starter Kit V3.0 for Windows10 64bit

Good Morning Philip

There is an issue with David's Metop Manager in that there isnt an option to save the raw data to a specified location like there is in the MSG Data Manager and I dont think there ever will be

However if you  untick Delete Files from RX in the Metop Manager then it will continually reprocess all the files again and again - it doesnt seem to recognise which files have been processed for that day and which havent so it just does the whole lot over again

The solution I found is to set up a process that watches the EPS-10 directory and as soon as new files come in it copies them to a specified location. It does this before the Metop Manager processes them which means I can leave Delete Files from RX ticked . However for my solution it does involve extra software to monitor the EPS-10 directory

Its just something to be aware of if you decide to use pytroll for Metop images

It doesnt seem to be a problem with Himawari ,GOES16/17s as they do have an option to save the raw data

I dont know why the Metop Manager is different

Regards

Graham

Virus-free. www.avg.com


Re: Using begiiners guide to PyTROLL/SatPy Starter Kit V3.0 for Windows10 64bit

Graham Woolf
 

Good Morning Philip

There is an issue with David's Metop Manager in that there isnt an option to save the raw data to a specified location like there is in the MSG Data Manager and I dont think there ever will be

However if you  untick Delete Files from RX in the Metop Manager then it will continually reprocess all the files again and again - it doesnt seem to recognise which files have been processed for that day and which havent so it just does the whole lot over again

The solution I found is to set up a process that watches the EPS-10 directory and as soon as new files come in it copies them to a specified location. It does this before the Metop Manager processes them which means I can leave Delete Files from RX ticked . However for my solution it does involve extra software to monitor the EPS-10 directory

Its just something to be aware of if you decide to use pytroll for Metop images

It doesnt seem to be a problem with Himawari ,GOES16/17s as they do have an option to save the raw data

I dont know why the Metop Manager is different

Regards

Graham


Re: Using begiiners guide to PyTROLL/SatPy Starter Kit V3.0 for Windows10 64bit

Ernst Lobsiger
 

Phillip,

congratulation! You finally made it! It almost seems to understand and reconfigure MDM was the hardest part for you. There are four more hints:

1) You see in the output that PPP_CONFIG_DIR is deprecated. It still works but the PyTROLL/Satpy developers have renamed that environment
   variable  to SATPY_CONFIG_PATH. This message will disappear as soon as you also rename the Windows environment variable to the new name.
2) There is a typo spelling "Edinbourg" instead of "Edinburgh" (sorry David). This typo is in all scripts of the Starter Kit 3.0. Easy to find and correct.
3) You will probably also want to change the receiver type. I doubt you have a TBS-6909X. That's  the card that "converted" Graham to use GNU/Linux :-).
4) Last but not least: If you post console output you better higlight it with the mouse and do a copy & paste instead of these hard to read screenshots.

Cheers,
Ernst

1381 - 1400 of 33867