Hi Ernst

Those errors were from yesterdays pass and I was using Isle of Man
# Isle of Man
lat = 54.228
lon = -4.532
ran = 20.0

Regards

Graham

On Fri, Nov 20, 2020 at 04:06 AM, Douglas Deans wrote:

Yes I get that quite often with FY-3D as well Graham. I have noticed that changing the lat and/or lon can sometimes resolve it so perhaps it has something to do with the tolerance of the pass seeking code.

Regards,
Douglas.

Douglas

I doubt there is a problem in my simple pass seeking code. If you change the POI this may result in another orbit number.
If you only increase search range (ran) the pass will be "longer" and maybe reach far north in the (November) winter dark.

Cheers
Ernst

On Fri, Nov 20, 2020 at 04:21 AM, Graham Woolf wrote:

Do you have any lat/long values that appear to work more successfully or is it pot luck on each pass

Regards

Graham

Graham and Douglas

This looks as if you do not have some data that are needed for "true-color". Maybe some VIS data is not transmitted far north in the dark?

1) Can you give me a date and POI location and range you use that shows this problem (I have data 4 days back here) ?
2) Can you try a FY-3D IR (DAY!) composite with these input data that does not produce a VIS (DAY) "true_color" image ?
3) Can you have a look at the list [bestfiles] or use good old TCLogSummary.cmd to make sure you have no missing files?

Regards
Ernst

Hi Douglas

Thanks - Im glad its not just me

Do you have any lat/long values that appear to work more successfully or is it pot luck on each pass

Regards

Graham

On 20/11/2020 11:47, Graham Woolf wrote:

Hi Ernst
I have reported this to OPS
I am having trouble processing FY3D images though  most days when I run the script I get the following errors which dont mean much to me . I wonder if you could have a look at them
Traceback (most recent call last):
  File "c:\Users\graha\miniconda3\envs\pytroll\lib\site-packages\satpy\readers\__init__.py", line 301, in __getitem__
    return super(DatasetDict, self).__getitem__(item)
KeyError: 'true_color'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "T:\PythonScripts\WindowsScripts\FY3D\FY-3D_VIS.py", line 188, in <module>
    scn.save_dataset(composite,imgdir+'/FY3D-Orbit_'+str(orbmax)+'-'+
  File "c:\Users\graha\miniconda3\envs\pytroll\lib\site-packages\satpy\scene.py", line 1302, in save_dataset
    return writer.save_dataset(self[dataset_id],
  File "c:\Users\graha\miniconda3\envs\pytroll\lib\site-packages\satpy\scene.py", line 679, in __getitem__
    return self.datasets[key]
  File "c:\Users\graha\miniconda3\envs\pytroll\lib\site-packages\satpy\readers\__init__.py", line 303, in __getitem__
    key = self.get_key(item)
  File "c:\Users\graha\miniconda3\envs\pytroll\lib\site-packages\satpy\readers\__init__.py", line 290, in get_key
    return get_key(match_key, self.keys(), num_results=num_results,
  File "c:\Users\graha\miniconda3\envs\pytroll\lib\site-packages\satpy\readers\__init__.py", line 245, in get_key
    raise KeyError("No dataset matching '{}' found".format(str(key)))
KeyError: "No dataset matching 'DatasetID(name='true_color', wavelength=None, resolution=None, polarization=None, calibration=None, level=None, modifiers=None)' found"
Now some days this will run OK but mostly it doesnt
I have tried with satpy 021 and 024 with the same problems
Regards
Graham

==============================================================================================

Yes I get that quite often with FY-3D as well Graham. I have noticed that changing the lat and/or lon can sometimes resolve it so perhaps it has something to do with the tolerance of the pass seeking code.

Regards,
Douglas.

Hi Ernst

I have reported this to OPS

I am having trouble processing FY3D images though  most days when I run the script I get the following errors which dont mean much to me . I wonder if you could have a look at them

Traceback (most recent call last):
  File "c:\Users\graha\miniconda3\envs\pytroll\lib\site-packages\satpy\readers\__init__.py", line 301, in __getitem__
    return super(DatasetDict, self).__getitem__(item)
KeyError: 'true_color'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "T:\PythonScripts\WindowsScripts\FY3D\FY-3D_VIS.py", line 188, in <module>
    scn.save_dataset(composite,imgdir+'/FY3D-Orbit_'+str(orbmax)+'-'+
  File "c:\Users\graha\miniconda3\envs\pytroll\lib\site-packages\satpy\scene.py", line 1302, in save_dataset
    return writer.save_dataset(self[dataset_id],
  File "c:\Users\graha\miniconda3\envs\pytroll\lib\site-packages\satpy\scene.py", line 679, in __getitem__
    return self.datasets[key]
  File "c:\Users\graha\miniconda3\envs\pytroll\lib\site-packages\satpy\readers\__init__.py", line 303, in __getitem__
    key = self.get_key(item)
  File "c:\Users\graha\miniconda3\envs\pytroll\lib\site-packages\satpy\readers\__init__.py", line 290, in get_key
    return get_key(match_key, self.keys(), num_results=num_results,
  File "c:\Users\graha\miniconda3\envs\pytroll\lib\site-packages\satpy\readers\__init__.py", line 245, in get_key
    raise KeyError("No dataset matching '{}' found".format(str(key)))
KeyError: "No dataset matching 'DatasetID(name='true_color', wavelength=None, resolution=None, polarization=None, calibration=None, level=None, modifiers=None)' found"

Now some days this will run OK but mostly it doesnt

I have tried with satpy 021 and 024 with the same problems

Regards

Graham

On Thu, Nov 19, 2020 at 07:51 AM, Graham Woolf wrote:

Hi Ernst

I am seeing missing segments again today on NOAA20 orbit number 15565

Are you seeing the same or is it my system just missing some files

Regards

Graham

Graham and All

I just had a look at U.K. passes of Suomi-NPP, NOAA-20, and FY-3D from the last four days.
11 passes are fine, todays NOAA-20 orbit 15565 shows the old problem again. My bet ist that
this is a software bug in the code that stitchs together the data from LANNION and SVALBARD.

NOAA-20 Orbit 15565 pass over th U.K.

-rw-r--r-- 1 root root 53224098 Nov 19 12:07 /srv/rec_0/NPP-2/SVMC_j01_d20201119_t1155229_e1156474_b15566_c20201119120600000401_eum_ops.h5
-rw-r--r-- 1 root root 51792322 Nov 19 12:09 /srv/rec_0/NPP-2/SVMC_j01_d20201119_t1156486_e1158114_b15566_c20201119120532000145_eum_ops.h5
-rw-r--r-- 1 root root 49438269 Nov 19 12:11 /srv/rec_0/NPP-2/SVMC_j01_d20201119_t1158126_e1159371_b15566_c20201119121036000176_eum_ops.h5
-rw-r--r-- 1 root root 47176429 Nov 19 12:13 /srv/rec_0/NPP-2/SVMC_j01_d20201119_t1159383_e1201029_b15566_c20201119121213000973_eum_ops.h5
-rw-r--r-- 1 root root 52510526 Nov 19 12:13 /srv/rec_0/NPP-2/SVMC_j01_d20201119_t1201041_e1202286_b15566_c20201119121221000333_eum_ops.h5

Here we go again with the same problem: All segments received but one segment pretends to have the normal length of app. 85 seconds
but regarding its size compared with adjecent segments obviously is either too short in scan lines or has many black scan lines. El
-rw-r--r-- 1 root root 34226739 Nov 19 12:22 /srv/rec_0/NPP-2/SVMC_j01_d20201119_t1202299_e1203526_b15566_c20201119121952000550_eum_ops.h5

-rw-r--r-- 1 root root 52193269 Nov 19 12:17 /srv/rec_0/NPP-2/SVMC_j01_d20201119_t1203538_e1205183_b15566_c20201119121634000711_eum_ops.h5
-rw-r--r-- 1 root root 52302947 Nov 19 12:18 /srv/rec_0/NPP-2/SVMC_j01_d20201119_t1205196_e1206441_b15566_c20201119121642000336_eum_ops.h5
-rw-r--r-- 1 root root 44722166 Nov 19 12:20 /srv/rec_0/NPP-2/SVMC_j01_d20201119_t1206453_e1208099_b15566_c20201119121649000895_eum_ops.h5
-rw-r--r-- 1 root root 34437344 Nov 19 12:21 /srv/rec_0/NPP-2/SVMC_j01_d20201119_t1208111_e1209356_b15566_c20201119122012000679_eum_ops.h5
-rw-r--r-- 1 root root 23264076 Nov 19 12:21 /srv/rec_0/NPP-2/SVMC_j01_d20201119_t1209369_e1210596_b15566_c20201119122020000870_eum_ops.h5

I'll again report to OPS but others should do as well.

Regards
Ernst

Cornish Man,

Did you install the libs? decompr, hdf5, for Windows gtk20; for Pi: libs_xrit2pic, all on my website:

http://www.alblas.demon.nl/wsat/software/soft_msg.html#download2

Rob.

FYI:

xrit2pic 2020.4 is now available, for Linux, Windows and Raspberry Pi (last one only command line version).
Supports now also FY3D (just "fly-by" picture), and MSG native format. With a simple python script it is now possible to download near-realtime HRIT MSG/MSG_RSS from the Eumetsat-site, and process it using xrit2pic.
Works fine on a very old RPI (2011, I think one of the first versions).

Regards,
Rob Alblas

To all

FY-3D orbit 15620 afternoon ascending over the UK no missing segments within the omerc_bb projection (roughly N Africa to Greenland)

Rgds

John

John, Graham,

I see the same missing segments for NOAA20. For the moment only basic services, so I can't check for FY3D .

grts,

Hugo

On my system same missing segments NOAA20 orbit 15565. Night orbit 15559 ok
Both night orbit 46955 and day orbit 46962 OK for SUOMI NPP
Will check FY-3D shortly

Rgds  

John

Hi Ernst

I am seeing missing segments again today on NOAA20 orbit number 15565

Are you seeing the same or is it my system just missing some files

Regards

Graham

Hi Christian

Thanks for your help. I put the following in the generic.yaml in the enhancements folder

Hi John,

This inspired me to create a composite night_test  which uses M12 Blue, M14 Green and M15 Red

Ferdinand,

I'am really a scientist but I have worked as a teacher most of the time.
Therefore it's nice to see you arrived at chapter 8.) of my text already:

8) Where do we go from here
-------------------------------------
...
What composites need what instrument channels? Find answers here:
C:\users\username\miniconda3\envs\pytroll\Lib\site-packages\satpy\etc\composites
...
For almost every RGB composite there are interesting short "Guides" like the one I attached.
You find them all in directory "Guides" of Diego Souzas SHOWCast program suite which under
the hood actually uses Pytroll/Satpy for image processing. You find SHOWCast here:

https://geonetcast.wordpress.com/2020/07/08/new-showcast-release-v-2-0-0/

My 2.0 version is still on the TODO list. I have to ready my garden (jungle) for the looming winter.

Best regards
Ernst