Preparing for MTG operations.


geojohnt@...
 

Hello All,

The launch of the first MTG imaging satellite is only a few months away and ground based operations have been gearing up to be ready.

https://www.esa.int/Applications/Observing_the_Earth/Meteorological_missions/meteosat_third_generation/Operation_centres_in_tune_for_upcoming_weather_satellite

Regards,
John Tellick.


Ernst Lobsiger
 

Dear All,

it's also time to get familiar with MTG-i1 FCI data. There is one day of
FCI test data assembled from MODIS and other LEO/GEO data here:

https://sftp.eumetsat.int/public/folder/UsCVknVOOkSyCdgpMimJNQ/User-Materials/Test-Data/MTG/MTG_FCI_L1C_SpectrallyRepresentative_TD-360_May2022/

Each slot is 10 minutes (144 slots) compressed and uncompressed.
There is an easy to follow FCI example on SatPy read the docs.

https://satpy.readthedocs.io/en/latest/examples/fci_l1c_natural_color.html?highlight=FCI

I made a couple of MTG composites with above test data.
This wetransfer link stays active for 1 week (65MB *.zip file):

https://we.tl/t-3MO8OF53DI

While coastlines fit well the data does currently not seem to
fully fill the FCI area definition as distributed with SatPy 0.36.
Not all FCI enhancements are yet defined (some "raw" look pale).

Have fun,
Ernst

P.S.
I had to reduce full disc images that are 11136 x 11136 pixels !!
Users that want to play at that level better save money for RAM.


Douglas Deans
 

On 31/07/2022 21:52, Ernst Lobsiger via groups.io wrote:
Dear All,
it's also time to get familiar with MTG-i1 FCI data. There is one day of
FCI test data assembled from MODIS and other LEO/GEO data here:
https://sftp.eumetsat.int/public/folder/UsCVknVOOkSyCdgpMimJNQ/User-Materials/Test-Data/MTG/MTG_FCI_L1C_SpectrallyRepresentative_TD-360_May2022/ <https://sftp.eumetsat.int/public/folder/UsCVknVOOkSyCdgpMimJNQ/User-Materials/Test-Data/MTG/MTG_FCI_L1C_SpectrallyRepresentative_TD-360_May2022/>
Each slot is 10 minutes (144 slots) compressed and uncompressed.
There is an easy to follow FCI example on SatPy read the docs.
https://satpy.readthedocs.io/en/latest/examples/fci_l1c_natural_color.html?highlight=FCI <https://satpy.readthedocs.io/en/latest/examples/fci_l1c_natural_color.html?highlight=FCI>
I made a couple of MTG composites with above test data.
This wetransfer link stays active for 1 week (65MB *.zip file):
https://we.tl/t-3MO8OF53DI <https://we.tl/t-3MO8OF53DI>
While coastlines fit well the data does currently not seem to
fully fill the FCI area definition as distributed with SatPy 0.36.
Not all FCI enhancements are yet defined (some "raw" look pale).
Have fun,
Ernst
P.S.
I had to reduce full disc images that are 11136 x 11136 pixels !!
Users that want to play at that level better save money for RAM.
===============================================================================================================================================

Many thanks Ernst for putting this together.
I tried many months ago with the Satpy reader for the data but sadly failed miserably to adapt it to my use.
Might give it another try again.

Regards,
Douglas.


Ernst Lobsiger
 

Hi Douglas,

the easiest way to start is to download 1 uncompressed repeat cycle RC0072 (noon) unzipped in a data directory.
There should be 41 'W_XX*.nc' files. Later we probably will have to select the files for one RC by 'repeat_cycle_in_day'
in file names as start and stop times in FCI file names now show actual scanning times. From my notes (RC0001):

reader='fci_l1c_nc'

Repeat cicle (= SLOT) 1 .. 144 (10 minutes 'repeat_cycle_in_day', example below shows 001)
Number of files in repeat cicle 1 .. 41 ('count_in_repeat_cycle', example below shows 0001)

File_pattern:  '{pflag}_{location_indicator},{data_designator},MTI{spacecraft_id:1d}+{data_source}-1C-RRAD-FDHSI-{coverage}-{subsetting}-{component1}-BODY-{component3}-{purpose}-{format}_{oflag}_{originator}_{processing_time:%Y%m%d%H%M%S}_{facility_or_tool}_{environment}_{start_time:%Y%m%d%H%M%S}_{end_time:%Y%m%d%H%M%S}_{processing_mode}_{special_compression}_{disposition_mode}_{repeat_cycle_in_day:>04d}_{count_in_repeat_cycle:>04d}.nc'
Example name: W_XX-EUMETSAT-Darmstadt,IMG+SAT,MTI1+FCI-1C-RRAD-FDHSI-FD--CHK-BODY---NC4E_C_EUMT_20170920000515_GTT_DEV_20170920000008_20170920000015_N__T_0001_0001.nc

Good luck,
Ernst


Douglas Deans
 

On 01/08/2022 09:50, Ernst Lobsiger via groups.io wrote:
Hi Douglas,
the easiest way to start is to download 1 uncompressed repeat cycle RC0072 (noon) unzipped in a data directory.
There should be 41 'W_XX*.nc' files. Later we probably will have to select the files for one RC by 'repeat_cycle_in_day'
in file names as start and stop times in FCI file names now show actual scanning times. From my notes (RC0001):
reader='fci_l1c_nc'
Repeat cicle (= SLOT) 1 .. 144 (10 minutes 'repeat_cycle_in_day', example below shows 001)
Number of files in repeat cicle 1 .. 41 ('count_in_repeat_cycle', example below shows 0001)
File_pattern: '{pflag}_{location_indicator},{data_designator},MTI{spacecraft_id:1d}+{data_source}-1C-RRAD-FDHSI-{coverage}-{subsetting}-{component1}-BODY-{component3}-{purpose}-{format}_{oflag}_{originator}_{processing_time:%Y%m%d%H%M%S}_{facility_or_tool}_{environment}_{start_time:%Y%m%d%H%M%S}_{end_time:%Y%m%d%H%M%S}_{processing_mode}_{special_compression}_{disposition_mode}_{repeat_cycle_in_day:>04d}_{count_in_repeat_cycle:>04d}.nc' Example name: W_XX-EUMETSAT-Darmstadt,IMG+SAT,MTI1+FCI-1C-RRAD-FDHSI-FD--CHK-BODY---NC4E_C_EUMT_20170920000515_GTT_DEV_20170920000008_20170920000015_N__T_0001_0001.nc
Good luck,
Ernst
_._,_._,_
------------------------------------------------------------------------
Many thanks Ernst. Some good pointers there to spur me on.
Will report back.

Regards,
Douglas.


Douglas Deans
 

On 01/08/2022 09:50, Ernst Lobsiger via groups.io wrote:
Hi Douglas,
the easiest way to start is to download 1 uncompressed repeat cycle RC0072 (noon) unzipped in a data directory.
There should be 41 'W_XX*.nc' files. Later we probably will have to select the files for one RC by 'repeat_cycle_in_day'
in file names as start and stop times in FCI file names now show actual scanning times. From my notes (RC0001):
reader='fci_l1c_nc'
Repeat cicle (= SLOT) 1 .. 144 (10 minutes 'repeat_cycle_in_day', example below shows 001)
Number of files in repeat cicle 1 .. 41 ('count_in_repeat_cycle', example below shows 0001)
File_pattern: '{pflag}_{location_indicator},{data_designator},MTI{spacecraft_id:1d}+{data_source}-1C-RRAD-FDHSI-{coverage}-{subsetting}-{component1}-BODY-{component3}-{purpose}-{format}_{oflag}_{originator}_{processing_time:%Y%m%d%H%M%S}_{facility_or_tool}_{environment}_{start_time:%Y%m%d%H%M%S}_{end_time:%Y%m%d%H%M%S}_{processing_mode}_{special_compression}_{disposition_mode}_{repeat_cycle_in_day:>04d}_{count_in_repeat_cycle:>04d}.nc' Example name: W_XX-EUMETSAT-Darmstadt,IMG+SAT,MTI1+FCI-1C-RRAD-FDHSI-FD--CHK-BODY---NC4E_C_EUMT_20170920000515_GTT_DEV_20170920000008_20170920000015_N__T_0001_0001.nc
Good luck,
Ernst
==============================================================================================

I have now found that the main problem (or at least one of them) is that my reader in the yaml files is 'fci_I1c_fdhsi' whereas the software is looking for reader 'fci_I1c_nc'

So I will need to find where to download the correct reader.

Regards,
Douglas.


Ernst Lobsiger
 

Douglas,

this is not the first MTG FCI test data and the reader has envolved too. What you probably need is just an update of your satpy.
When in environment (pytroll) issue at the prompt "conda list satpy". This should result in version 0.36 or later. If not update
by "conda update satpy". With that you should automatically get the latest readers. There is a 'fci_l1c_nc' and a 'fci_l2_nc'.
You need the first one 'fci_l1c_nc'. Do *not* use the '.yaml' suffix if you specify the reader in your script.

Regards,
Ernst


Douglas Deans
 

On 01/08/2022 19:53, Ernst Lobsiger via groups.io wrote:
Douglas,
this is not the first MTG FCI test data and the reader has envolved too. What you probably need is just an update of your satpy.
When in environment (pytroll) issue at the prompt "conda list satpy". This should result in version 0.36 or later. If not update
by "conda update satpy". With that you should automatically get the latest readers. There is a 'fci_l1c_nc' and a 'fci_l2_nc'.
You need the first one 'fci_l1c_nc'. Do *not* use the '.yaml' suffix if you specify the reader in your script.
Regards,
Ernst
======================================================================================

Ah fully understood Ernst. Appreciate your help.
This will be fun !!?

Regards,
Douglas.


Ernst Lobsiger
 

Douglas,

as I said you must be in environment (pytroll). If you open the miniconda prompt first you probably are in the (base) environment.
If so, make a "conda activate pytroll" first. The prompt must show (pytroll). Updatin satpy (resolving dependencies) will take a while ...
... and produce a lot of output.

I hope it's fun :-),
Ernst


Douglas Deans
 

On 01/08/2022 20:30, Ernst Lobsiger via groups.io wrote:
Douglas,
as I said you must be in environment (pytroll). If you open the miniconda prompt first you probably are in the (base) environment.
If so, make a "conda activate pytroll" first. The prompt must show (pytroll). Updatin satpy (resolving dependencies) will take a while ...
... and produce a lot of output.
I hope it's fun :-),
Ernst
====================================================================================

It was Ernst and it worked fine. Now updated from .25 to .36.
I now have the correct reader but there are quite a number of other issues to fix.
If I do this myself I will learn a lot so it may be a little time before I come back reporting success.

Thanks again.
Douglas.


Ernst Lobsiger
 

Douglas,

when I wrote the HOWTO for my Kit 3.0 SatPy was at 0.24. So you probably never have updated anything.
Maybe you have been asked to update conda itself with command "conda update -n base -c defaults conda".
And I do not exactly remember when but the name of the environment variable PPP_CONFIG_PATH has
been changed to SATPY_CONFIG_PATH. The old name should still work but you may be reminded to do
the change with every start of a script ...

Good luck,
Ernst


Ernst Lobsiger
 

On Mon, Aug 1, 2022 at 10:16 PM, Ernst Lobsiger wrote:
environment variable PPP_CONFIG_PATH has
been changed to SATPY_CONFIG_PATH.
Sorry,

the old name is "PPP_CONFIG_DIR" that changed to SATPY_CONFIG_PATH.
I have not had my first cup of coffee yet ..

Ernst


Douglas Deans
 

On 02/08/2022 06:16, Ernst Lobsiger via groups.io wrote:
Douglas,
when I wrote the HOWTO for my Kit 3.0 SatPy was at 0.24. So you probably never have updated anything.
Maybe you have been asked to update conda itself with command "conda update -n base -c defaults conda".
And I do not exactly remember when but the name of the environment variable PPP_CONFIG_PATH has
been changed to SATPY_CONFIG_PATH. The old name should still work but you may be reminded to do
the change with every start of a script ...
Good luck,
Ernst
==========================================================================

Thanks Ernst. Yes I am seeing that but ignoring it for the moment.
Working my way through some issues (when time allows) but I am sure that as before I will get them resolved.

Regards,
Douglas.


R. Alblas
 

FYI:

I have uploaded a satpy MTG script. It works fine on the test data. It's still beta, until we get real data.
Apart from adapting the MSGx.py to MTGx.py I had to update Pytroll, as mentioned by Ernst; that's all.
See:

http://www.alblas.demon.nl/wsat/software/satpy.html#GEO_scripts

Script: MTGx.py.
With the same arguments as with my other scripts; one mandatory: -t <YYYYmmddHHMM>


Cheers,
Rob,

On 02-08-2022 10:54, Douglas Deans via groups.io wrote:
On 02/08/2022 06:16, Ernst Lobsiger via groups.io wrote:
Douglas,

when I wrote the HOWTO for my Kit 3.0 SatPy was at 0.24. So you probably never have updated anything.
Maybe you have been asked to update conda itself with command "conda update -n base -c defaults conda".
And I do not exactly remember when but the name of the environment variable PPP_CONFIG_PATH has
been changed to SATPY_CONFIG_PATH. The old name should still work but you may be reminded to do
the change with every start of a script ...

Good luck,
Ernst
==========================================================================

Thanks Ernst.  Yes I am seeing that but ignoring it for the moment.
Working my way through some issues (when time allows) but I am sure that as before I will get them resolved.

Regards,
Douglas.





Douglas Deans
 

On 02/08/2022 10:30, R. Alblas wrote:
FYI:
I have uploaded a satpy MTG script. It works fine on the test data. It's still beta, until we get real data.
Apart from adapting the MSGx.py to MTGx.py I had to update Pytroll, as mentioned by Ernst; that's all.
See:
http://www.alblas.demon.nl/wsat/software/satpy.html#GEO_scripts
Script: MTGx.py.
With the same arguments as with my other scripts; one mandatory: -t <YYYYmmddHHMM>
Cheers,
Rob,
===================================================================================

Rob, thanks for that.
I have been trying some of your scripts with success but I am having a problem with the MTG one.
I have managed to resolve one or two issues (my own problems) but I am stuck with the following error message that I cannot seem to fix.

Runtime error: NetCDF : Filter error : unimplemented filter encountered.

I am running Windows 11 and have updated to the latest Pytroll/Satpy etc.
Any pointers would be appreciated.

Regards,
Douglas.


R. Alblas
 

On 03-08-2022 18:43, Douglas Deans via groups.io wrote:
===================================================================================

Rob, thanks for that.
I have been trying some of your scripts with success but I am having a problem with the MTG one.
I have managed to resolve one or two issues (my own problems) but I am stuck with the following error message that I cannot seem to fix.

Runtime error: NetCDF : Filter error : unimplemented filter encountered.

I am running Windows 11 and have updated to the latest Pytroll/Satpy etc.
Any pointers would be appreciated.

Regards,
Douglas.

===================================================================================


Douglas,

Are you trying to process compressed data? Because then I get also this error message. As far as I know currently only uncompressed data can be handled by Satpy. Or this has to be worked out. Ernst also mentioned to use uncompressed data.

Last 4 messages I get using compressed data:

  File "src/netCDF4/_netCDF4.pyx", line 4739, in netCDF4._netCDF4.Variable.__getitem__
  File "src/netCDF4/_netCDF4.pyx", line 5688, in netCDF4._netCDF4.Variable._get
  File "src/netCDF4/_netCDF4.pyx", line 1965, in netCDF4._netCDF4._ensure_nc_success
RuntimeError: NetCDF: Filter error: unimplemented filter encountered

Regards,
Rob.


Ernst Lobsiger
 

Rob and Douglas,

This works with compressed and uncompressed data. But as explained in my read the docs link you have to install "hdf5plugin" and import it for compressed data.

https://satpy.readthedocs.io/en/latest/examples/fci_l1c_natural_color.html?highlight=FCI

Regards,

Ernst


Douglas Deans
 

On 03/08/2022 20:17, R. Alblas wrote:
On 03-08-2022 18:43, Douglas Deans via groups.io wrote:
===================================================================================

Rob, thanks for that.
I have been trying some of your scripts with success but I am having a problem with the MTG one.
I have managed to resolve one or two issues (my own problems) but I am stuck with the following error message that I cannot seem to fix.

Runtime error: NetCDF : Filter error : unimplemented filter encountered.

I am running Windows 11 and have updated to the latest Pytroll/Satpy etc.
Any pointers would be appreciated.

Regards,
Douglas.

===================================================================================
Douglas,
Are you trying to process compressed data? Because then I get also this error message. As far as I know currently only uncompressed data can be handled by Satpy. Or this has to be worked out. Ernst also mentioned to use uncompressed data.
Last 4 messages I get using compressed data:
  File "src/netCDF4/_netCDF4.pyx", line 4739, in netCDF4._netCDF4.Variable.__getitem__
  File "src/netCDF4/_netCDF4.pyx", line 5688, in netCDF4._netCDF4.Variable._get
  File "src/netCDF4/_netCDF4.pyx", line 1965, in netCDF4._netCDF4._ensure_nc_success
RuntimeError: NetCDF: Filter error: unimplemented filter encountered
Regards,
Rob.
======================================================================================

Rob thanks for the reply.
I have checked several times that it is uncompressed but it does look like the same message you get with the compressed data so I will double check again.

Thanks again. Will get back shortly.

Regards,
Douglas.


R. Alblas
 


On 03-08-2022 21:46, Douglas Deans via groups.io wrote:


Rob thanks for the reply.
I have checked several times that it is uncompressed but it does look like the same message you get with the compressed data so I will double check again.

Thanks again. Will get back shortly.

Regards,
Douglas.


======================================================================================

Douglas,

I have updated the MTGx.py script, and now it works for compressed data too, thanks to Ernst (I missed the "read the docs link"...)

It is just a matter of installing hdf5plugin, and add one line in the script:

import hdf5plugin

But if you used uncompressed data and it failed something else is wrong. At least try the updated script.

Regards,

Rob.








Douglas Deans
 

On 03/08/2022 20:46, Douglas Deans via groups.io wrote:
On 03/08/2022 20:17, R. Alblas wrote:

On 03-08-2022 18:43, Douglas Deans via groups.io wrote:
===================================================================================

Rob, thanks for that.
I have been trying some of your scripts with success but I am having a problem with the MTG one.
I have managed to resolve one or two issues (my own problems) but I am stuck with the following error message that I cannot seem to fix.

Runtime error: NetCDF : Filter error : unimplemented filter encountered.

I am running Windows 11 and have updated to the latest Pytroll/Satpy etc.
Any pointers would be appreciated.

Regards,
Douglas.

===================================================================================

Douglas,

Are you trying to process compressed data? Because then I get also this error message. As far as I know currently only uncompressed data can be handled by Satpy. Or this has to be worked out. Ernst also mentioned to use uncompressed data.

Last 4 messages I get using compressed data:

   File "src/netCDF4/_netCDF4.pyx", line 4739, in netCDF4._netCDF4.Variable.__getitem__
   File "src/netCDF4/_netCDF4.pyx", line 5688, in netCDF4._netCDF4.Variable._get
   File "src/netCDF4/_netCDF4.pyx", line 1965, in netCDF4._netCDF4._ensure_nc_success
RuntimeError: NetCDF: Filter error: unimplemented filter encountered

Regards,
Rob.
======================================================================================
Rob thanks for the reply.
I have checked several times that it is uncompressed but it does look like the same message you get with the compressed data so I will double check again.
Thanks again. Will get back shortly.
Regards,
Douglas.
==========================================================================================

My apologies to both. My fault. Clearly checking twice is not enough in your late 70's !!
It was the compressed data. All fine now. Not helped by the way, by Eumetsat having the same folder names for both sets of data.

Perfect image now processed.
Also noted re the update for both compressed and uncompressed. Thanks for that. Will update tomorrow.
Thanks Rob and Ernst.

Regards,
Douglas.