Topics

FY3D EARS Data

Ian Deans
 

I see from tomorrow that FY3D image data via the EARS service is to be made available ( on reequest via Portal ) and I understand that it is 25 channels.

As I assume it may have been on test for some time is there any program excluding HDFview which is able to visualise this data.

Regards
Ian.

Ernst Lobsiger
 

Ian,

IIRC MERSI (MERSI-1) was an imager on ENVISAT we received via DDS.

There seems to be a FY3D MERSI-2 l1b data reader for PyTroll/SatPy.

https://github.com/pytroll/satpy/issues/706

Is this the kind of data they want to make available via the EARS service?
I have never used this reader up to now. Christian Peters might comment.

Regards
Ernst

Douglas Deans
 

On 21/04/2020 11:05, Ian Deans via groups.io wrote:
I see from tomorrow that FY3D image data via the EARS service is to be made available ( on reequest via Portal ) and I understand that it is 25 channels.
As I assume it may have been on test for some time is there any program excluding HDFview which is able to visualise this data.
Regards
Ian.
============================================================================

I have had the data since last year but found the same difficulty.
SNAP from ESA will but it is very cumbersome and time consuming and I have never been able to provide colours.

David and Hugo's programs cannot handle the data.

The answer to your question really is no.

Regards,
Douglas.

Ernst Lobsiger
 

Douglas and Ian,

MERSI is also the ancestor of OLCI. I used to process ENVISAT data with "convert".
This command line program is still distibuted with SNAP and works for OLCI data.
I have used it in a simple script for years. Maybe you can give "convert" a try?

Regards
Ernst

Christian Peters
 

Ian and Ernst,

I checked with test data available form the Internet.

Pytroll/Satpy MERSI l1b reader for FY3D seems to work. I think there has to be some tweaking of the file pattern take place if it's available and distributed by EUMETCAST, but it works.
I tested with MERSI_1000M_L1B.HDF.

But a lot of additional data per day...! ;-)

Regards,

Christian

Am 21.04.20 um 12:05 schrieb Ian Deans via groups.io:


I see from tomorrow that FY3D image data via the EARS service is to be made available ( on reequest via Portal ) and I understand that it is 25 channels.

As I assume it may have been on test for some time is there any program excluding HDFview which is able to visualise this data.

Regards
Ian.

Ernst Lobsiger
 

Christian,

This was really quick! FY3D seems to be sun synchonous with LTDN 14:00.

https://directory.eoportal.org/web/eoportal/satellite-missions/f/fy-3

So it's an "afternoon" satellite heading south. It should be easy to adapt
my SatPy script to find best POI overhead passes (EARS coverage only).
Some of the channels are very high resolution. We'll see what we get ...

Cheers,
Ernst

Christian Peters
 

Ernst,

I select the data in the EO Portal today...let's see what we get and whether it's working.
Should be really not too complicated to use your fly-over-script I hope....! :-)
But I read there could be an issue even with PyTroll...let's see.

"...NOTE: For some parameters, the values may exceed the range which is specified by the valid range attribute of these parameters. This could lead to reading problems with HDF-readers that use the valid range attribute to check the validity of the data (e.g. some pytroll routines). The issue is being investigated with CMA. "

Regards,

Christian

Am 21.04.20 um 14:39 schrieb Ernst Lobsiger via groups.io:

Christian,

This was really quick! FY3D seems to be sun synchonous with LTDN 14:00.

https://directory.eoportal.org/web/eoportal/satellite-missions/f/fy-3

So it's an "afternoon" satellite heading south. It should be easy to adapt
my SatPy script to find best POI overhead passes (EARS coverage only).
Some of the channels are very high resolution. We'll see what we get ...

Cheers,
Ernst

Ernst Lobsiger
 

Christian and All,

First of all I didn't recall correctly: The ENVISAT imager was MERIS not MERSI.
No idea what the technical differencies are. The files that we can expect from
EUMETCast are described here (click EUMETCast-Europe --> Details):

https://navigator.eumetsat.int/product/EO:EUM:DAT:0100
   
FY3D_20190517_111100_111200_7786_MERSI_GEO1K_L1B.HDF
FY3D_20190517_111100_111200_7786_MERSI_1000M_L1B.HDF

We may do not need the the GEO-File (that seems to be location data) as my
SatPy script uses orbit calculations starting with the filename time stamps.

The data should arrive under HVS-1, multicast address is as for NOAA-20

E1H-RDS-2     601     224.223.223.2     12000     Regional data: FY3

My receivers are down right now, but I'll have a look at it rather soon.


Good luck
Ernst

Christian Peters
 

Ernst and Douglas,

success! We need the GEO files in Satpy for geolocation information.
Sadly it's only the 1km resolution version which is distributed....! :-(

But it works nice with your modified script Ernst, see .jpgs (raw and with correction)! :-)


Regards,

Christian

Am 22.04.20 um 10:01 schrieb Ernst Lobsiger via groups.io:

Christian and All,

First of all I didn't recall correctly: The ENVISAT imager was MERIS not MERSI.
No idea what the technical differencies are. The files that we can expect from
EUMETCast are described here (click EUMETCast-Europe --> Details):

https://navigator.eumetsat.int/product/EO:EUM:DAT:0100
   
FY3D_20190517_111100_111200_7786_MERSI_GEO1K_L1B.HDF
FY3D_20190517_111100_111200_7786_MERSI_1000M_L1B.HDF

We may do not need the the GEO-File (that seems to be location data) as my
SatPy script uses orbit calculations starting with the filename time stamps.

The data should arrive under HVS-1, multicast address is as for NOAA-20

E1H-RDS-2     601     224.223.223.2     12000     Regional data: FY3

My receivers are down right now, but I'll have a look at it rather soon.


Good luck
Ernst

Ernst Lobsiger
 

Christian,

that's really good looking. Yes my over-head-SatPy script should work for most sun synchronous satellites.
Of course the GEO location file is used for resampling. I didn't think of that because my "convert" script
for OLCI pictures does no resampling. I just restarted my TC receivers. What SatPy version do you use?

Cheers,
Ernst

Douglas Deans
 

On 22/04/2020 14:39, Christian Peters via groups.io wrote:
Ernst and Douglas,
success! We need the GEO files in Satpy for geolocation information.
Sadly it's only the 1km resolution version which is distributed....! :-(
But it works nice with your modified script Ernst, see .jpgs (raw and with correction)! :-)
Regards,
Christian
===================================================================

Very nice Christian.
However for non programming people this method is completely out of their comfort zone.

Regards,
Douglas.

Christian Peters
 

Ernst,

yes...I do resampling. But you are right, works without the files and no resampling too.
Just using different versions of Satpy (14, 18 an 21) because of different environments! :-D But I stay on Python 3.7 version as there are still some issues with 3.8. and some progs.
But the most recent version V0.21.0 should work. Maybe you have to do some updates or start a new env with this version to get thing working....

Regards,

Christian


Am 22.04.20 um 15:57 schrieb Ernst Lobsiger via groups.io:

Christian,

that's really good looking. Yes my over-head-SatPy script should work for most sun synchronous satellites.
Of course the GEO location file is used for resampling. I didn't think of that because my "convert" script
for OLCI pictures does no resampling. I just restarted my TC receivers. What SatPy version do you use?

Cheers,
Ernst

Graham Woolf
 

Hi Ernst

can you possible attach your script or email a link

I would like to be able to decode this data

Kind Regards

Graham

Ernst Lobsiger
 

Graham,

are you familiar with PyTroll/SatPy? If not read my HOWTO attached to message #28324 4/13/19
or just search the list for "SatPy HOWTO". The script Christian used for FY3D is basically what
you find within the HOWTO-tgz for Metop A,B,C, Aqua, Terra, NOAA-20, Suomi_NPP. If you
are on WINDOWS you need a program to decompress my *.tgz first and then add the CRs by
opening and saving the text files with WORDPAD. Then use the EDITOR. SatPy lets you reproject
in whatever map projection you can think of! Feel free to ask if you run into any kind of problems.

Best Regards,
Ernst

Graham Woolf
 

Hi Ernst

I wouldnt say familiar exactly although I have manages to get Sentinel data decoded

I will have a go

Thanks

Graham

Ernst Lobsiger
 

Graham,

as you know I mostly run GNU/Linux. When I wrote my SatPy HOWTO I had one
Windows 7 PC as well. If you are on Windows it's certainly Windows 10 now.
I just checked with a recent Windows 10 install how to give it a go:

A *.tgz is a *.tar archive that is gzipped. In Windows 10 you can:

- use 7zip to gunzip and untar my HOWTO.tgz in two steps
- directly use the EDITOR for extracted *.txt and *.py files

The Windows 10 EDITOR understands UTF8 and missig CRs from GNU/Linux.
You will need an ANACONDA Python (take the latest 3.7 version) environment.

Hope this helps
Ernst

Graham Woolf
 

Hi Ernst

Yes I am running Windows 10 and I have Anaconda installed for the processing of Sentinel 3 data
 
The unzipping etc is not a problem - I have the scripts OK

The biggest problem for me is how to translate the unix in your scripts to Windows

So my first question is where does coast_dir get set or is it defined in this statement 

                      overlay = {'coast_dir': '/home/lobsiger/gshhg-shp/',

Regards

Graham

Ernst Lobsiger
 

Graham,

Yes it's defined there (not good programming :-). Today I installed miniconda3
and PyTROLL/SatPy on my Windows 10 box. I "translated" two scripts. There
was a minor problem with UNIX timestamps. I attach the two scripts and results.

The FY-3D_VIS is a *very* QUICK and DIRTY adaption of Suomi_NPP_VIS!
Just to give you a first idea. If some more Windows users are interested, we
could do some (corona) home schooling. But as I have no clue how average
Windows TC receivers are organized, maybe Thorsten would be needed too.
Hugo could eventually give us a hand with the wavelet decompression stuff.


Cheers,
Ernst

Graham Woolf
 

Hi Ernst

Thanks for that - very useful

I am assuming that your scripts are called with one parameter ie. the date

# Date YYYYMMDD
Dat=sys.argv[1]

as here

I am then assuming your script searches all the relevant files for the current day but Im not clear how it picks the relevant file

Kind Regards

Graham

Ernst Lobsiger
 

Graham,

that's no Voodoo just simple spherical trigonometry. I have organized the data
in a ... /TC-CHANNEL/YEAR/MON/DAY structure as used by EUMETCastView.

You call the script (after installing PyTROLL/Satpy and acivating it's env) with:

(YOUR ENV) python MyScript.py 20200423

The script makes a list of files available for the day. Then it extracts the
UTC stamps of the segments and calculates the sat position in the middle
of the segment. If its close enough to your POI (antenna?) the file name
is added to a goodlist. The closest segment will define the orbit for a
bestlist. This bestlist of files is finally read and treated by SatPy.

Regards,
Ernst