Graham

Your list from yesterday has 11 GEO1k files and 9 1000m files. NOT expected.
Your list from today has 10 GEO1k files and 11 1000m files. NOT expected.

1) You either look too early and the files are not in (as I just explained to Douglas).
2) You simply didn't receive all FY-3D files on HVS-1  (use TCLogSummary.cmd)

Regards
Ernst

P.S.
My two receivers with a TBS-6909X taking all 3 services with 3 demodulators
have not lost one single UDP packet (and so no files) for the last three weeks.

On Fri, Nov 20, 2020 at 06:20 AM, Douglas Deans wrote:

Ernst I probably need to look at this more closely, but when I do have a failure it usually checks out as ok on EUMETCastView in colour.
Having said that today's image has produced fine.

Regards,
Douglas.

Douglas

Be sure to give enough time slack for the data to arrive. If you already had todays picture before your last post then the timing may be tight on some days. My scripts
do not know when the files arrive. They do not predict passes but they look at the available file names with segment times and reverse calculate the orbit positions.
For todays almost overhead pass the last file was 13:17 UTC but yesterday (as the nearest pass was more west of IsleOfMan) it was 13:36 UTC and this can even be later.
You have to add an additional worst case slack of 50 minutes (half an typical sun synchronous sat revolution time) to the time of maximum elevation of a central pass.
Then you have EARS timelyness which IIRC is up to another 30 minutes. If you make a picture while data is comming in you could end up with a list of unpaired
1000m and GEO1k files. Then simply try 30 minutes later again (remember FY-3D has LTAN 14:00 which is also 30 minutes after NOAA-20 and Suomi-NPP).

Regards
Ernst

Hi Ernst

Here is my list and its only 20 files compared to your 22

I:/EUMETCast/HVS/MERSI\FY3D_20201119_132600_132700_15621_MERSI_1000M_L1B.HDF
I:/EUMETCast/HVS/MERSI\FY3D_20201119_132600_132700_15621_MERSI_GEO1K_L1B.HDF
I:/EUMETCast/HVS/MERSI\FY3D_20201119_132700_132800_15621_MERSI_1000M_L1B.HDF
I:/EUMETCast/HVS/MERSI\FY3D_20201119_132700_132800_15621_MERSI_GEO1K_L1B.HDF
I:/EUMETCast/HVS/MERSI\FY3D_20201119_132800_132900_15621_MERSI_1000M_L1B.HDF
I:/EUMETCast/HVS/MERSI\FY3D_20201119_132800_132900_15621_MERSI_GEO1K_L1B.HDF
I:/EUMETCast/HVS/MERSI\FY3D_20201119_132900_133000_15621_MERSI_1000M_L1B.HDF
I:/EUMETCast/HVS/MERSI\FY3D_20201119_132900_133000_15621_MERSI_GEO1K_L1B.HDF
I:/EUMETCast/HVS/MERSI\FY3D_20201119_133000_133100_15621_MERSI_GEO1K_L1B.HDF
I:/EUMETCast/HVS/MERSI\FY3D_20201119_133100_133200_15620_MERSI_1000M_L1B.HDF
I:/EUMETCast/HVS/MERSI\FY3D_20201119_133100_133200_15620_MERSI_GEO1K_L1B.HDF
I:/EUMETCast/HVS/MERSI\FY3D_20201119_133200_133300_15620_MERSI_1000M_L1B.HDF
I:/EUMETCast/HVS/MERSI\FY3D_20201119_133200_133300_15620_MERSI_GEO1K_L1B.HDF
I:/EUMETCast/HVS/MERSI\FY3D_20201119_133300_133400_15620_MERSI_1000M_L1B.HDF
I:/EUMETCast/HVS/MERSI\FY3D_20201119_133300_133400_15620_MERSI_GEO1K_L1B.HDF
I:/EUMETCast/HVS/MERSI\FY3D_20201119_133400_133500_15620_MERSI_GEO1K_L1B.HDF
I:/EUMETCast/HVS/MERSI\FY3D_20201119_133500_133600_15620_MERSI_1000M_L1B.HDF
I:/EUMETCast/HVS/MERSI\FY3D_20201119_133500_133600_15620_MERSI_GEO1K_L1B.HDF
I:/EUMETCast/HVS/MERSI\FY3D_20201119_133600_133700_15620_MERSI_1000M_L1B.HDF
I:/EUMETCast/HVS/MERSI\FY3D_20201119_133600_133700_15620_MERSI_GEO1K_L1B.HDF

And for today I only have 21

I:/EUMETCast/HVS/MERSI\FY3D_20201120_130700_130800_15635_MERSI_1000M_L1B.HDF
I:/EUMETCast/HVS/MERSI\FY3D_20201120_130700_130800_15635_MERSI_GEO1K_L1B.HDF
I:/EUMETCast/HVS/MERSI\FY3D_20201120_130800_130900_15635_MERSI_1000M_L1B.HDF
I:/EUMETCast/HVS/MERSI\FY3D_20201120_130800_130900_15635_MERSI_GEO1K_L1B.HDF
I:/EUMETCast/HVS/MERSI\FY3D_20201120_130900_131000_15635_MERSI_1000M_L1B.HDF
I:/EUMETCast/HVS/MERSI\FY3D_20201120_130900_131000_15635_MERSI_GEO1K_L1B.HDF
I:/EUMETCast/HVS/MERSI\FY3D_20201120_131000_131100_15634_MERSI_1000M_L1B.HDF
I:/EUMETCast/HVS/MERSI\FY3D_20201120_131100_131200_15635_MERSI_1000M_L1B.HDF
I:/EUMETCast/HVS/MERSI\FY3D_20201120_131100_131200_15635_MERSI_GEO1K_L1B.HDF
I:/EUMETCast/HVS/MERSI\FY3D_20201120_131200_131300_15635_MERSI_1000M_L1B.HDF
I:/EUMETCast/HVS/MERSI\FY3D_20201120_131200_131300_15635_MERSI_GEO1K_L1B.HDF
I:/EUMETCast/HVS/MERSI\FY3D_20201120_131300_131400_15634_MERSI_1000M_L1B.HDF
I:/EUMETCast/HVS/MERSI\FY3D_20201120_131300_131400_15634_MERSI_GEO1K_L1B.HDF
I:/EUMETCast/HVS/MERSI\FY3D_20201120_131400_131500_15634_MERSI_1000M_L1B.HDF
I:/EUMETCast/HVS/MERSI\FY3D_20201120_131400_131500_15634_MERSI_GEO1K_L1B.HDF
I:/EUMETCast/HVS/MERSI\FY3D_20201120_131500_131600_15634_MERSI_1000M_L1B.HDF
I:/EUMETCast/HVS/MERSI\FY3D_20201120_131500_131600_15634_MERSI_GEO1K_L1B.HDF
I:/EUMETCast/HVS/MERSI\FY3D_20201120_131600_131700_15634_MERSI_1000M_L1B.HDF
I:/EUMETCast/HVS/MERSI\FY3D_20201120_131600_131700_15634_MERSI_GEO1K_L1B.HDF
I:/EUMETCast/HVS/MERSI\FY3D_20201120_131700_131800_15634_MERSI_1000M_L1B.HDF
I:/EUMETCast/HVS/MERSI\FY3D_20201120_131700_131800_15634_MERSI_GEO1K_L1B.HDF

Regards

Graham

On 20/11/2020 12:48, Ernst Lobsiger via groups.io wrote:

On Fri, Nov 20, 2020 at 04:21 AM, Graham Woolf wrote:
Do you have any lat/long values that appear to work more
successfully or is it pot luck on each pass
Regards
Graham
Graham and Douglas
This looks as if you do not have some data that are needed for "true-color". Maybe some VIS data is not transmitted far north in the dark?
1) Can you give me a date and POI location and range you use that shows this problem (I have data 4 days back here) ?
2) Can you try a FY-3D IR (DAY!) composite with these input data that does not produce a VIS (DAY) "true_color" image ?
3) Can you have a look at the list [bestfiles] or use good old TCLogSummary.cmd to make sure you have no missing files?
Regards
Ernst

===================================================================================

Ernst I probably need to look at this more closely, but when I do have a failure it usually checks out as ok on EUMETCastView in colour.
Having said that today's image has produced fine.

Regards,
Douglas.

On Fri, Nov 20, 2020 at 05:28 AM, Graham Woolf wrote:

Hi Ernst

Those errors were from yesterdays pass and I was using Isle of Man
# Isle of Man
lat = 54.228
lon = -4.532
ran = 20.0

Regards

Graham

Graham,

no problem here under GNU/Linux. The FY-3D files come in pairs 1000m and GEO location.
Not sure what happens if you are missing one file in a pair. Try to print [bestfiles] in the script.

Cheers
Ernst

Hi Ernst

Those errors were from yesterdays pass and I was using Isle of Man
# Isle of Man
lat = 54.228
lon = -4.532
ran = 20.0

Regards

Graham

On Fri, Nov 20, 2020 at 04:06 AM, Douglas Deans wrote:

Yes I get that quite often with FY-3D as well Graham. I have noticed that changing the lat and/or lon can sometimes resolve it so perhaps it has something to do with the tolerance of the pass seeking code.

Regards,
Douglas.

Douglas

I doubt there is a problem in my simple pass seeking code. If you change the POI this may result in another orbit number.
If you only increase search range (ran) the pass will be "longer" and maybe reach far north in the (November) winter dark.

Cheers
Ernst

On Fri, Nov 20, 2020 at 04:21 AM, Graham Woolf wrote:

Do you have any lat/long values that appear to work more successfully or is it pot luck on each pass

Regards

Graham

Graham and Douglas

This looks as if you do not have some data that are needed for "true-color". Maybe some VIS data is not transmitted far north in the dark?

1) Can you give me a date and POI location and range you use that shows this problem (I have data 4 days back here) ?
2) Can you try a FY-3D IR (DAY!) composite with these input data that does not produce a VIS (DAY) "true_color" image ?
3) Can you have a look at the list [bestfiles] or use good old TCLogSummary.cmd to make sure you have no missing files?

Regards
Ernst

Hi Douglas

Thanks - Im glad its not just me

Do you have any lat/long values that appear to work more successfully or is it pot luck on each pass

Regards

Graham

On 20/11/2020 11:47, Graham Woolf wrote:

Hi Ernst
I have reported this to OPS
I am having trouble processing FY3D images though  most days when I run the script I get the following errors which dont mean much to me . I wonder if you could have a look at them
Traceback (most recent call last):
  File "c:\Users\graha\miniconda3\envs\pytroll\lib\site-packages\satpy\readers\__init__.py", line 301, in __getitem__
    return super(DatasetDict, self).__getitem__(item)
KeyError: 'true_color'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "T:\PythonScripts\WindowsScripts\FY3D\FY-3D_VIS.py", line 188, in <module>
    scn.save_dataset(composite,imgdir+'/FY3D-Orbit_'+str(orbmax)+'-'+
  File "c:\Users\graha\miniconda3\envs\pytroll\lib\site-packages\satpy\scene.py", line 1302, in save_dataset
    return writer.save_dataset(self[dataset_id],
  File "c:\Users\graha\miniconda3\envs\pytroll\lib\site-packages\satpy\scene.py", line 679, in __getitem__
    return self.datasets[key]
  File "c:\Users\graha\miniconda3\envs\pytroll\lib\site-packages\satpy\readers\__init__.py", line 303, in __getitem__
    key = self.get_key(item)
  File "c:\Users\graha\miniconda3\envs\pytroll\lib\site-packages\satpy\readers\__init__.py", line 290, in get_key
    return get_key(match_key, self.keys(), num_results=num_results,
  File "c:\Users\graha\miniconda3\envs\pytroll\lib\site-packages\satpy\readers\__init__.py", line 245, in get_key
    raise KeyError("No dataset matching '{}' found".format(str(key)))
KeyError: "No dataset matching 'DatasetID(name='true_color', wavelength=None, resolution=None, polarization=None, calibration=None, level=None, modifiers=None)' found"
Now some days this will run OK but mostly it doesnt
I have tried with satpy 021 and 024 with the same problems
Regards
Graham

==============================================================================================

Yes I get that quite often with FY-3D as well Graham. I have noticed that changing the lat and/or lon can sometimes resolve it so perhaps it has something to do with the tolerance of the pass seeking code.

Regards,
Douglas.

Hi Ernst

I have reported this to OPS

I am having trouble processing FY3D images though  most days when I run the script I get the following errors which dont mean much to me . I wonder if you could have a look at them

Traceback (most recent call last):
  File "c:\Users\graha\miniconda3\envs\pytroll\lib\site-packages\satpy\readers\__init__.py", line 301, in __getitem__
    return super(DatasetDict, self).__getitem__(item)
KeyError: 'true_color'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "T:\PythonScripts\WindowsScripts\FY3D\FY-3D_VIS.py", line 188, in <module>
    scn.save_dataset(composite,imgdir+'/FY3D-Orbit_'+str(orbmax)+'-'+
  File "c:\Users\graha\miniconda3\envs\pytroll\lib\site-packages\satpy\scene.py", line 1302, in save_dataset
    return writer.save_dataset(self[dataset_id],
  File "c:\Users\graha\miniconda3\envs\pytroll\lib\site-packages\satpy\scene.py", line 679, in __getitem__
    return self.datasets[key]
  File "c:\Users\graha\miniconda3\envs\pytroll\lib\site-packages\satpy\readers\__init__.py", line 303, in __getitem__
    key = self.get_key(item)
  File "c:\Users\graha\miniconda3\envs\pytroll\lib\site-packages\satpy\readers\__init__.py", line 290, in get_key
    return get_key(match_key, self.keys(), num_results=num_results,
  File "c:\Users\graha\miniconda3\envs\pytroll\lib\site-packages\satpy\readers\__init__.py", line 245, in get_key
    raise KeyError("No dataset matching '{}' found".format(str(key)))
KeyError: "No dataset matching 'DatasetID(name='true_color', wavelength=None, resolution=None, polarization=None, calibration=None, level=None, modifiers=None)' found"

Now some days this will run OK but mostly it doesnt

I have tried with satpy 021 and 024 with the same problems

Regards

Graham

On Thu, Nov 19, 2020 at 07:51 AM, Graham Woolf wrote:

Hi Ernst

I am seeing missing segments again today on NOAA20 orbit number 15565

Are you seeing the same or is it my system just missing some files

Regards

Graham

Graham and All

I just had a look at U.K. passes of Suomi-NPP, NOAA-20, and FY-3D from the last four days.
11 passes are fine, todays NOAA-20 orbit 15565 shows the old problem again. My bet ist that
this is a software bug in the code that stitchs together the data from LANNION and SVALBARD.

NOAA-20 Orbit 15565 pass over th U.K.

-rw-r--r-- 1 root root 53224098 Nov 19 12:07 /srv/rec_0/NPP-2/SVMC_j01_d20201119_t1155229_e1156474_b15566_c20201119120600000401_eum_ops.h5
-rw-r--r-- 1 root root 51792322 Nov 19 12:09 /srv/rec_0/NPP-2/SVMC_j01_d20201119_t1156486_e1158114_b15566_c20201119120532000145_eum_ops.h5
-rw-r--r-- 1 root root 49438269 Nov 19 12:11 /srv/rec_0/NPP-2/SVMC_j01_d20201119_t1158126_e1159371_b15566_c20201119121036000176_eum_ops.h5
-rw-r--r-- 1 root root 47176429 Nov 19 12:13 /srv/rec_0/NPP-2/SVMC_j01_d20201119_t1159383_e1201029_b15566_c20201119121213000973_eum_ops.h5
-rw-r--r-- 1 root root 52510526 Nov 19 12:13 /srv/rec_0/NPP-2/SVMC_j01_d20201119_t1201041_e1202286_b15566_c20201119121221000333_eum_ops.h5

Here we go again with the same problem: All segments received but one segment pretends to have the normal length of app. 85 seconds
but regarding its size compared with adjecent segments obviously is either too short in scan lines or has many black scan lines. El
-rw-r--r-- 1 root root 34226739 Nov 19 12:22 /srv/rec_0/NPP-2/SVMC_j01_d20201119_t1202299_e1203526_b15566_c20201119121952000550_eum_ops.h5

-rw-r--r-- 1 root root 52193269 Nov 19 12:17 /srv/rec_0/NPP-2/SVMC_j01_d20201119_t1203538_e1205183_b15566_c20201119121634000711_eum_ops.h5
-rw-r--r-- 1 root root 52302947 Nov 19 12:18 /srv/rec_0/NPP-2/SVMC_j01_d20201119_t1205196_e1206441_b15566_c20201119121642000336_eum_ops.h5
-rw-r--r-- 1 root root 44722166 Nov 19 12:20 /srv/rec_0/NPP-2/SVMC_j01_d20201119_t1206453_e1208099_b15566_c20201119121649000895_eum_ops.h5
-rw-r--r-- 1 root root 34437344 Nov 19 12:21 /srv/rec_0/NPP-2/SVMC_j01_d20201119_t1208111_e1209356_b15566_c20201119122012000679_eum_ops.h5
-rw-r--r-- 1 root root 23264076 Nov 19 12:21 /srv/rec_0/NPP-2/SVMC_j01_d20201119_t1209369_e1210596_b15566_c20201119122020000870_eum_ops.h5

I'll again report to OPS but others should do as well.

Regards
Ernst

Cornish Man,

Did you install the libs? decompr, hdf5, for Windows gtk20; for Pi: libs_xrit2pic, all on my website:

http://www.alblas.demon.nl/wsat/software/soft_msg.html#download2

Rob.

FYI:

xrit2pic 2020.4 is now available, for Linux, Windows and Raspberry Pi (last one only command line version).
Supports now also FY3D (just "fly-by" picture), and MSG native format. With a simple python script it is now possible to download near-realtime HRIT MSG/MSG_RSS from the Eumetsat-site, and process it using xrit2pic.
Works fine on a very old RPI (2011, I think one of the first versions).

Regards,
Rob Alblas

To all

FY-3D orbit 15620 afternoon ascending over the UK no missing segments within the omerc_bb projection (roughly N Africa to Greenland)

Rgds

John

John, Graham,

I see the same missing segments for NOAA20. For the moment only basic services, so I can't check for FY3D .

grts,

Hugo

On my system same missing segments NOAA20 orbit 15565. Night orbit 15559 ok
Both night orbit 46955 and day orbit 46962 OK for SUOMI NPP
Will check FY-3D shortly

Rgds  

John

Hi Ernst

I am seeing missing segments again today on NOAA20 orbit number 15565

Are you seeing the same or is it my system just missing some files

Regards

Graham

Hi Christian

Thanks for your help. I put the following in the generic.yaml in the enhancements folder