Ernst just to confirm, after your hard work, that I now have all available satellites up and running using your package and scripts and have made up a number of different cmd files with personalised requirements, all working (mostly!).

The one area now requiring much more work is understanding scripts and that will continue.

Can I ask one final favour to let me progress.
If I just wanted a simple channel image say from MSG4, example VIS006, ie no composite, what would I replace the line eg :-
composite = 'natural_color' with.

Should that line just be :: out and adjustments made to the subsequent lines in the script.

Obviously needed_file would be VIS006

I have spent a lot of time trying to get this to work without success.
A pointer would be a great help.

Regards,
Douglas.

Douglas,

composite = 'VIS006'
needed_files = ['VIS006']

will work. The list [..] of needed_files refers to the naming of the files.
composite could also be a single wavelength 0.635. The problem
with that is I use composite in the final image naming as a string
format. composite = 'VIS006' is a string (text containing) variable while
composite = 0.635 is a floating point variable (you can do math with).

That's where simple programming starts ... See examples attached.


Best regards
Ernst

On 19/05/2020 18:54, Ernst Lobsiger via groups.io wrote:

Douglas,
composite = 'VIS006'
needed_files = ['VIS006']
will work. The list [..] of needed_files refers to the naming of the files.
composite could also be a single wavelength 0.635. The problem
with that is I use composite in the final image naming as a string
format. composite = 'VIS006' is a string (text containing) variable while
composite = 0.635 is a floating point variable (you can do math with).
That's where simple programming starts ... See examples attached.
Best regards
Ernst

==========================================================================

Many thanks Ernst. Much to my surprise that is what I had already been trying but with the following errors coming up.(Top of cmd screen not copied. All files were decompressed).


Decompressed file: H-000-MSG4__-MSG4________-VIS006___-000002___-202005191000-__
Decompressed file: H-000-MSG4__-MSG4________-VIS006___-000003___-202005191000-__
Decompressed file: H-000-MSG4__-MSG4________-VIS006___-000004___-202005191000-__
Decompressed file: H-000-MSG4__-MSG4________-VIS006___-000005___-202005191000-__
Decompressed file: H-000-MSG4__-MSG4________-VIS006___-000006___-202005191000-__
Decompressed file: H-000-MSG4__-MSG4________-VIS006___-000007___-202005191000-__
Decompressed file: H-000-MSG4__-MSG4________-VIS006___-000008___-202005191000-__
Traceback (most recent call last):
File "C:\Users\Douglas\miniconda3\envs\pytroll\lib\site-packages\satpy\readers\__init__.py", line 301, in __getitem__
return super(DatasetDict, self).__getitem__(item)
KeyError: 'VIS006'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "msg4_experiment.py", line 93, in <module>
global_scene.save_dataset(composite,'./MSG4.png',
File "C:\Users\Douglas\miniconda3\envs\pytroll\lib\site-packages\satpy\scene.py", line 1302, in save_dataset
return writer.save_dataset(self[dataset_id],
File "C:\Users\Douglas\miniconda3\envs\pytroll\lib\site-packages\satpy\scene.py", line 679, in __getitem__
return self.datasets[key]
File "C:\Users\Douglas\miniconda3\envs\pytroll\lib\site-packages\satpy\readers\__init__.py", line 303, in __getitem__
key = self.get_key(item)
File "C:\Users\Douglas\miniconda3\envs\pytroll\lib\site-packages\satpy\readers\__init__.py", line 290, in get_key
return get_key(match_key, self.keys(), num_results=num_results,
File "C:\Users\Douglas\miniconda3\envs\pytroll\lib\site-packages\satpy\readers\__init__.py", line 245, in get_key
raise KeyError("No dataset matching '{}' found".format(str(key)))
KeyError: "No dataset matching 'DatasetID(name='VIS006', wavelength=None, resolution=None, polarization=None, calibration=None, level=None, modifiers=None)' found"
Press any key to continue . . .


Kind regards,
Douglas.

Douglas,

are ylo missing one file 00001 ? I get using msg4.py:

(pytroll) PS C:\EMCtools\pyscripts> python msg4.py 202005191000
Decompressed file: H-000-MSG4__-MSG4________-VIS006___-000006___-202005191000-__
Decompressed file: H-000-MSG4__-MSG4________-VIS006___-000002___-202005191000-__
Decompressed file: H-000-MSG4__-MSG4________-VIS006___-000005___-202005191000-__
Decompressed file: H-000-MSG4__-MSG4________-VIS006___-000001___-202005191000-__
Decompressed file: H-000-MSG4__-MSG4________-VIS006___-000004___-202005191000-__
Decompressed file: H-000-MSG4__-MSG4________-VIS006___-000003___-202005191000-__
Decompressed file: H-000-MSG4__-MSG4________-VIS006___-000007___-202005191000-__
Decompressed file: H-000-MSG4__-MSG4________-VIS006___-000008___-202005191000-__
C:\Users\immpac\miniconda3\envs\pytroll\lib\site-packages\pyproj\crs\crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordinate-reference-systems
  proj_string = self.to_proj4() ..... etc. etc.

with result below.

Regards,
Ernst

On 19/05/2020 20:44, Ernst Lobsiger via groups.io wrote:

Douglas,
are ylo missing one file 00001 ? I get using msg4.py:
(pytroll) PS C:\EMCtools\pyscripts> python msg4.py 202005191000
Decompressed file: H-000-MSG4__-MSG4________-VIS006___-000006___-202005191000-__
Decompressed file: H-000-MSG4__-MSG4________-VIS006___-000002___-202005191000-__
Decompressed file: H-000-MSG4__-MSG4________-VIS006___-000005___-202005191000-__
Decompressed file: H-000-MSG4__-MSG4________-VIS006___-000001___-202005191000-__
Decompressed file: H-000-MSG4__-MSG4________-VIS006___-000004___-202005191000-__
Decompressed file: H-000-MSG4__-MSG4________-VIS006___-000003___-202005191000-__
Decompressed file: H-000-MSG4__-MSG4________-VIS006___-000007___-202005191000-__
Decompressed file: H-000-MSG4__-MSG4________-VIS006___-000008___-202005191000-__
C:\Users\immpac\miniconda3\envs\pytroll\lib\site-packages\pyproj\crs\crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordinate-reference-systems
  proj_string = self.to_proj4() ..... etc. etc.
with result below.
Regards,
Ernst

==========================================================================
No they were all there. That's why I said "top of screen not shown"
Here it is with a bigger window showing all.


Decompressed file: H-000-MSG4__-MSG4________-VIS006___-000001___-202005191000-__
Decompressed file: H-000-MSG4__-MSG4________-VIS006___-000002___-202005191000-__
Decompressed file: H-000-MSG4__-MSG4________-VIS006___-000003___-202005191000-__
Decompressed file: H-000-MSG4__-MSG4________-VIS006___-000004___-202005191000-__
Decompressed file: H-000-MSG4__-MSG4________-VIS006___-000005___-202005191000-__
Decompressed file: H-000-MSG4__-MSG4________-VIS006___-000006___-202005191000-__
Decompressed file: H-000-MSG4__-MSG4________-VIS006___-000007___-202005191000-__
Decompressed file: H-000-MSG4__-MSG4________-VIS006___-000008___-202005191000-__
Traceback (most recent call last):
File "C:\Users\Douglas\miniconda3\envs\pytroll\lib\site-packages\satpy\readers\__init__.py", line 301, in __getitem__
return super(DatasetDict, self).__getitem__(item)
KeyError: 'VIS006'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "msg4_experiment.py", line 93, in <module>
global_scene.save_dataset(composite,'./MSG4.png',
File "C:\Users\Douglas\miniconda3\envs\pytroll\lib\site-packages\satpy\scene.py", line 1302, in save_dataset
return writer.save_dataset(self[dataset_id],
File "C:\Users\Douglas\miniconda3\envs\pytroll\lib\site-packages\satpy\scene.py", line 679, in __getitem__
return self.datasets[key]
File "C:\Users\Douglas\miniconda3\envs\pytroll\lib\site-packages\satpy\readers\__init__.py", line 303, in __getitem__
key = self.get_key(item)
File "C:\Users\Douglas\miniconda3\envs\pytroll\lib\site-packages\satpy\readers\__init__.py", line 290, in get_key
return get_key(match_key, self.keys(), num_results=num_results,
File "C:\Users\Douglas\miniconda3\envs\pytroll\lib\site-packages\satpy\readers\__init__.py", line 245, in get_key
raise KeyError("No dataset matching '{}' found".format(str(key)))
KeyError: "No dataset matching 'DatasetID(name='VIS006', wavelength=None, resolution=None, polarization=None, calibration=None, level=None, modifiers=None)' found"
Press any key to continue . . .
-------------------------------------------------------------------------

I have also copied the relevant portion of my cmd file just in case.
Sorry this is causing some hassle Ernst. I thought (and still do) that I was doing something stupid ! Feel free to leave to a more suitable time for yourself.

Regards,
Douglas.
-------------------------------------------------------------------------
# For composite 'natural_color' we use three channels in needed_files
composite = 'VIS006'
needed_files = ['VIS006']

for b in needed_files:
files = glob(segdir + "/" + "H-000-MSG4*" + str(b) + "*" + Dat + "*-C_")
# Decompress all n files found
for n in range (0, len(files)):
os.system("C:/EMCtools/exefiles/xRITDecompress " + files[n])

# Meteosat-11 sends only 1 PROlogue and 1 EPIlogue file per 15 minute time slot
# But e.g. the hrit_goes reader insisted that there is a separate PRO per band?
files = glob(segdir + "/" + "H-000-MSG4*PRO*" + Dat + "*")
for n in range (0, len(files)):
shutil.copy2(files[n], tmpdir);
files = glob(segdir + "/" + "H-000-MSG4*EPI*" + Dat + "*")
for n in range (0, len(files)):
shutil.copy2(files[n], tmpdir);


# Now all the files needed should be decompressed in tmpdir
files = glob(tmpdir + "/" + "H-000-MSG4*" + Dat + "*")
global_scene = Scene(filenames = files, reader = 'seviri_l1b_hrit')

# This image has South on top, image size is 3712x3712 pixels
global_scene.save_dataset(composite,'./MSG4.png',
overlay = {'coast_dir': 'H:/EMCdata/gshhg-shp',
'color': (255, 255, 0), 'width': 1.0, 'resolution': 'i'})

# Use IM to rotate, annotate (maybe also -equalize) and convert to *.jpg
Slo = Yea + "/" + Mon + "/" + Day + " " + Hou + ":" + Min
cmdstr = "C:/EMCtools/exefiles/convert MSG4.png -rotate 180 -background White -pointsize 82\
label:\"\\n Meteosat-11 " + Slo + " UTC / processed with Miniconda3 + Pytroll/Satpy\
/ (c) EUMETSAT\\n\" -gravity Center -append " +\
imgdir + "/" + "MSG4-" + Dat + "-" + composite + ".jpg"


# *Image Magick*
os.system(cmdstr)
# IM will still be working when this python script returns (subprocess.call?)

On Tue, May 19, 2020 at 01:27 PM, Douglas Deans wrote:

No they were all there. That's why I said "top of screen not shown"
Here it is with a bigger window showing all.


Decompressed file: H-000-MSG4__-MSG4________-VIS006___-000001___-202005191000-__
Decompressed file: H-000-MSG4__-MSG4________-VIS006___-000002___-202005191000-__
Decompressed file: H-000-MSG4__-MSG4________-VIS006___-000003___-202005191000-__
Decompressed file: H-000-MSG4__-MSG4________-VIS006___-000004___-202005191000-__
Decompressed file: H-000-MSG4__-MSG4________-VIS006___-000005___-202005191000-__
Decompressed file: H-000-MSG4__-MSG4________-VIS006___-000006___-202005191000-__
Decompressed file: H-000-MSG4__-MSG4________-VIS006___-000007___-202005191000-__
Decompressed file: H-000-MSG4__-MSG4________-VIS006___-000008___-202005191000-__

Graham,

1) If you try another time slot do you have the same problems?
2) If it fails and you check the relevant tmpdir do you also have the PROlogue and EPIlogue files there?
3) Could you zip and attach the script you use?

Good night
Ernst

Douglas,

here two more hints:

1) Could you first make sure we have the same versions of the packages satpy depends on:
From the Windows menue open the black CMD window and activate your environment pytroll :

(pytroll) C:\users\username>

Then update satpy

(pytroll) C:\users\username> conda update satpy

Then try update pyresample (maybe it's latest already)

(pytroll) C:\users\username> conda update pyresample

Now try again your script for 'VIS006' (it works under Windows 64Bit PRO here!).

2) If this does not help I noticed that Graham had the same kind of error when he tried 'hrv_clouds' without resampling (see error file attached)
Can you try a version of msg4_westminster.py that you edit to only use 'VIS006'. If this works then it sounds like something for the developers.

Good Luck
Ernst

Hi Ernst and Douglas

I have tried this on my PC and it works

Kind Regards

Graham

On 20/05/2020 09:00, Graham Woolf wrote:

Hi Ernst and Douglas
I have tried this on my PC and it works
Kind Regards
Graham

======================================

Excellent Graham. There are a couple of mails from Ernst with lots of things to try (for which thanks Ernst).
Which things did you do that worked.

Thanks.
Douglas.

Hi Douglas

All I did was change this

composite = 'VIS006'
needed_files = ['VIS006']

Regards

Graham

On 20/05/2020 10:09, Graham Woolf wrote:

Hi Douglas
All I did was change this
composite = 'VIS006'
needed_files = ['VIS006']
Regards
Graham

===================================================

Ah thanks Graham. That is what I have been using all along without success.
Just updated pytroll and pyresample (did not need done) but no change.

I'll progress with the other things Ernst ask me to look at.

Regards,
Douglas.

On 20/05/2020 10:15, Douglas Deans via groups.io wrote:

On 20/05/2020 10:09, Graham Woolf wrote:

Hi Douglas

All I did was change this

composite = 'VIS006'
needed_files = ['VIS006']

Regards

Graham

===================================================
Ah thanks Graham. That is what I have been using all along without success.
Just updated pytroll and pyresample (did not need done) but no change.
I'll progress with the other things Ernst ask me to look at.
Regards,
Douglas.

===========================================================

Further progress. It works with the msg4.westminster.py script.

Regards,
Douglas.

On Wed, May 20, 2020 at 02:15 AM, Douglas Deans wrote:

Ah thanks Graham. That is what I have been using all along without success.
Just updated pytroll and pyresample (did not need done) but no change.

Douglas,
really strange, it works for Graham and me but not for you. There is one more workaround you can try:

composite = 0.635
needed_files = ['VIS006']

but then you have to change at the bottom of the script:

 imgdir + "/" + "MSG4-" + Dat + "-" + composite + ".jpg"

to
 
 imgdir + "/" + "MSG4-" + Dat + "-" + str(composite) + ".jpg"


We have to change  the floating point 0.635 to a string '0.635' to concatenate with other strings.
This works here as well.

Regards,
Ernst

Sorted !
Now working for the fulldisc.
Needed to add the line :-

global_scene.load([composite])

although I am not sure why it was not there.

Anyway many thanks Ernst for all your efforts. Good learning curve.
Will leave you in peace now.
Hope to slowly get myself on top of this during the coming weeks/months.

Thanks also to you Graham for your help.

Kind regards,
Douglas.

On Wed, May 20, 2020 at 03:46 AM, Douglas Deans wrote:

global_scene.load([composite])

Douglas,

this line is always necessary. The order is:

a) Ready (e.g. decompress, etc ..) the files needed for a certain composite
b) Define the scene with the filelist for that composite and the appropriate reader
c) Tell the scene to load the composite (your missing line that made all the difference)
d) Resample for a certain area projection or if channels have different resolutions
e) Save to file adding coastlines (grids and decorates possible, *.png or *.jpg possible)
f) I annotate with Image Magick (outside image possible, not possible with decorate)

If you get stuck again, feel free to ask and just attach your script as *.zip.

Regards
Ernst

Douglas,

You actually published the relevant script part some posts back:

...
# Now all the files needed should be decompressed in tmpdir
files = glob(tmpdir + "/" + "H-000-MSG4*" + Dat + "*")
global_scene = Scene(filenames = files, reader = 'seviri_l1b_hrit')

# This image has South on top, image size is 3712x3712 pixels
global_scene.save_dataset(composite,'./MSG4.png',
overlay = {'coast_dir': 'H:/EMCdata/gshhg-shp',
'color': (255, 255, 0), 'width': 1.0, 'resolution': 'i'})
...

Sorry I did not see that the  load() line was missing.

One problem of Python is that it often is very noisy with warnings
and sometimes errors that come from the underlying Satpy and
other modules only very few people really understand. Neither
Christian nor me are real Python programmers. The good side
is that you do not have to know very much to get it running.
Then you experiment and learn by doing ...

Regards,
Ernst

On 20/05/2020 14:51, Ernst Lobsiger via groups.io wrote:

Douglas,
You actually published the relevant script part some posts back:
...
# Now all the files needed should be decompressed in tmpdir
files = glob(tmpdir + "/" + "H-000-MSG4*" + Dat + "*")
global_scene = Scene(filenames = files, reader = 'seviri_l1b_hrit')
# This image has South on top, image size is 3712x3712 pixels
global_scene.save_dataset(composite,'./MSG4.png',
overlay = {'coast_dir': 'H:/EMCdata/gshhg-shp',
'color': (255, 255, 0), 'width': 1.0, 'resolution': 'i'})
...
Sorry I did not see that the  load() line was missing.
One problem of Python is that it often is very noisy with warnings
and sometimes errors that come from the underlying Satpy and
other modules only very few people really understand. Neither
Christian nor me are real Python programmers. The good side
is that you do not have to know very much to get it running.
Then you experiment and learn by doing ...
Regards,
Ernst

==============================================================

No problems Ernst. I had been struggling with that for a few days before seeking help but that is one line I won't forget again.

Tried an IR (Channel 9, 10.8 µm) successfully and even managed to find out how to make a negative (white cold) with IM ( -negate).

Many thanks again.

Best regards,
Douglas.

Douglas,

it's a wide field to play. You will have lot's of fun.

Try "ir108_3d" composite with channel 10.8 and you got a nice b/w picture with white clouds without IM inverse nedde...! ;-)

Regards,

Christian

Am 20.05.20 um 16:18 schrieb Douglas Deans via groups.io:

On 20/05/2020 18:28, Christian Peters via groups.io wrote:

Douglas,
it's a wide field to play. You will have lot's of fun.
Try "ir108_3d" composite with channel 10.8 and you got a nice b/w picture with white clouds without IM inverse nedde...! ;-)
Regards,
Christian

============================================================

Excellent. Will certainly give that a try. Many thanks Christian.

Kind regards,
Douglas.