Using begiiners guide to PyTROLL/SatPy Starter Kit V3.0 for Windows10 64bit


Philip Robinson
 

Hi Ernst or anyone
I am having a go at using PyTroll/SatPy and have printed out and am using the very detailed but easy to follow instructions
All was well until i reached my first stumbling block
After extracting gshhg-shp-2.3.7.zip and then checking the file names and sizes in gshhg-shp. I havent a WDBII_shp or GSHHS_shp directoryb installed
Will this matter. I tried both Html and FTP to download the gshhg-sh2.3.7.zip but in both instances those directory files were not in the extracted downloads
Before i go further with  your instructions i wanted to check if thats still ok that i do not have those 2 folders at this stage.
Happy new Year
Philip


Ernst Lobsiger
 

Philip,

welcome to the club of PyTROLL/Satpy users. GSHHG is the name of a collection that has the two parts WDBII (that originated somewhere in the cold war of the 1970ies) and GSHHS.
You get this collection in different file formats. PyTROLL/Satpy uses shape files (which is different from what Hugo uses IIRC). In my data structure these are in the folder gshhg-shp.



Hope this helps and ATB for 2022,
Ernst


Ernst Lobsiger
 

Sorry Philip,

now I understand. You basically say that the ZIP content has changed? This will certainly matter. I'll check that.

Ernst


Philip Robinson
 

Ernst
thanks for the reply
I will try to explain a little clearer
When i downloaded the gshhg-gmt-2.3.7tar.gz and extracted the contents, the GSHHS_shp and WDBII.shp are not there, and SHAPEFILES.TXT is not there either. I have the files below
COPYING.LESSERv3
COPYINGv3
LICENSE.txt 
READEME.txt
but i also have lots of .inc files
regards
Philip

------ Original Message ------
From: "Ernst Lobsiger via groups.io" <ernst.lobsiger@...>
Sent: 01/01/2022 18:18:46
Subject: Re: [MSG-1] Using begiiners guide to PyTROLL/SatPy Starter Kit V3.0 for Windows10 64bit

Philip,

welcome to the club of PyTROLL/Satpy users. GSHHG is the name of a collection that has the two parts WDBII (that originated somewhere in the cold war of the 1970ies) and GSHHS.
You get this collection in different file formats. PyTROLL/Satpy uses shape files (which is different from what Hugo uses IIRC). In my data structure these are in the folder gshhg-shp.



Hope this helps and ATB for 2022,
Ernst

Virus-free. www.avg.com


Philip Robinson
 

Ernst

Just got your email now. Thats correct, the Zip content has changed . I used  html and ftp to double check but they are not there but as in my previous email  there are .inc files.

Regards

Philip

 

Sent from Mail for Windows

 

From: Ernst Lobsiger via groups.io
Sent: 01 January 2022 18:28
To: MSG-1@groups.io
Subject: Re: [MSG-1] Using begiiners guide to PyTROLL/SatPy Starter Kit V3.0 for Windows10 64bit

 

Sorry Philip,

now I understand. You basically say that the ZIP content has changed? This will certainly matter. I'll check that.

Ernst

 


Philip Robinson
 

Ernst
if They are not available by downloading does it still work if you were kind enough to send me copies of the folders GHSS_shp, WDBII.shp and SHAPEFILES.TXT. As long as i install them in the correct gshhg-shp folder , i presume that will not effect my following of your installation instructions.
regards
Philip

------ Original Message ------
From: "Philip Robinson" <philip57@...>
Sent: 01/01/2022 18:29:13
Subject: Re: [MSG-1] Using begiiners guide to PyTROLL/SatPy Starter Kit V3.0 for Windows10 64bit

Ernst
thanks for the reply
I will try to explain a little clearer
When i downloaded the gshhg-gmt-2.3.7tar.gz and extracted the contents, the GSHHS_shp and WDBII.shp are not there, and SHAPEFILES.TXT is not there either. I have the files below
COPYING.LESSERv3
COPYINGv3
LICENSE.txt 
READEME.txt
but i also have lots of .inc files
regards
Philip
------ Original Message ------
From: "Ernst Lobsiger via groups.io" <ernst.lobsiger@...>
Sent: 01/01/2022 18:18:46
Subject: Re: [MSG-1] Using begiiners guide to PyTROLL/SatPy Starter Kit V3.0 for Windows10 64bit

Philip,

welcome to the club of PyTROLL/Satpy users. GSHHG is the name of a collection that has the two parts WDBII (that originated somewhere in the cold war of the 1970ies) and GSHHS.
You get this collection in different file formats. PyTROLL/Satpy uses shape files (which is different from what Hugo uses IIRC). In my data structure these are in the folder gshhg-shp.



Hope this helps and ATB for 2022,
Ernst

Virus-free. www.avg.com


Christian Peters
 

Phillip,

I think you have to load
gshhg-shp-2.3.7.zip
...not the GMT version.

http://www.soest.hawaii.edu/pwessel/gshhg/gshhg-shp-2.3.7.zip

Regards,

Christian

Am 01.01.22 um 19:29 schrieb Philip Robinson:

Ernst
thanks for the reply
I will try to explain a little clearer
When i downloaded the gshhg-gmt-2.3.7tar.gz and extracted the contents, the GSHHS_shp and WDBII.shp are not there, and SHAPEFILES.TXT is not there either. I have the files below
COPYING.LESSERv3
COPYINGv3
LICENSE.txt 
READEME.txt
but i also have lots of .inc files
regards
Philip
------ Original Message ------
From: "Ernst Lobsiger via groups.io" <ernst.lobsiger@...>
Sent: 01/01/2022 18:18:46
Subject: Re: [MSG-1] Using begiiners guide to PyTROLL/SatPy Starter Kit V3.0 for Windows10 64bit

Philip,

welcome to the club of PyTROLL/Satpy users. GSHHG is the name of a collection that has the two parts WDBII (that originated somewhere in the cold war of the 1970ies) and GSHHS.
You get this collection in different file formats. PyTROLL/Satpy uses shape files (which is different from what Hugo uses IIRC). In my data structure these are in the folder gshhg-shp.



Hope this helps and ATB for 2022,
Ernst

Virus-free. www.avg.com


Ernst Lobsiger
 

Philip,

I just did the download and must tell you that NOTHING has changed. Maybe you opened the download even before you had it? Just try another download.


Ernst


Philip Robinson
 

Hi Christian

It looks as though the link at the bottom of the Hawaii page downloaded the gshhg-gmt-2.3.7 tar.gz

I will use your link and download the gshhg-shp=2.3.7.zip

I just followed Ernst instructions and the lack of my knowledge led me to download the GMT version

Many thanks

Philip

 

Sent from Mail for Windows

 


From: MSG-1@groups.io <MSG-1@groups.io> on behalf of Christian Peters via groups.io <hctpeters@...>
Sent: Saturday, January 1, 2022 6:38:41 PM
To: MSG-1@groups.io <MSG-1@groups.io>
Subject: Re: [MSG-1] Using begiiners guide to PyTROLL/SatPy Starter Kit V3.0 for Windows10 64bit
 
Phillip,

I think you have to load
gshhg-shp-2.3.7.zip
...not the GMT version.

http://www.soest.hawaii.edu/pwessel/gshhg/gshhg-shp-2.3.7.zip

Regards,

Christian

Am 01.01.22 um 19:29 schrieb Philip Robinson:
Ernst
thanks for the reply
I will try to explain a little clearer
When i downloaded the gshhg-gmt-2.3.7tar.gz and extracted the contents, the GSHHS_shp and WDBII.shp are not there, and SHAPEFILES.TXT is not there either. I have the files below
COPYING.LESSERv3
COPYINGv3
LICENSE.txt 
READEME.txt
but i also have lots of .inc files
regards
Philip
------ Original Message ------
From: "Ernst Lobsiger via groups.io" <ernst.lobsiger@...>
Sent: 01/01/2022 18:18:46
Subject: Re: [MSG-1] Using begiiners guide to PyTROLL/SatPy Starter Kit V3.0 for Windows10 64bit

Philip,

welcome to the club of PyTROLL/Satpy users. GSHHG is the name of a collection that has the two parts WDBII (that originated somewhere in the cold war of the 1970ies) and GSHHS.
You get this collection in different file formats. PyTROLL/Satpy uses shape files (which is different from what Hugo uses IIRC). In my data structure these are in the folder gshhg-shp.



Hope this helps and ATB for 2022,
Ernst

Virus-free. www.avg.com


Ernst Lobsiger
 

Philip,

tar.gz is Tape ARchive compressed with GNU zip. That's for UNIX and GNU/Linux.
You simply downloaded the wrong file. Maybe the layout of the soest web page has
changed (but I don't think so).

Ernst


Ernst Lobsiger
 

Philip,

my instructions say:

<CITE>

Go to

http://www.soest.hawaii.edu/wessel/gshhg/

gshhs and wdb files are part of GSHHG. Current version is GSHHG-2.3.7.
We need the native shape files that come in different resolutions and
can all be downloaded on the bottom of the page above. Look for the link:

CAUTION: Do not take the binary files as used by Hugo in EUMETCastView!


"
For ESRI shapefiles of shoreline polygons, rivers, and borders you can choose to get via ftp or http:

    FTP: GSHHG coastlines, political borders and rivers in shapefile format (zip archive)
    HTTP: GSHHG coastlines, political borders and rivers in shapefile format (zip archive)
"

You should get a file named gshhg-shp-2.3.7.zip

</CITE>

The soest page has not changed. The "bottom" does not mean the very last link. Please read my text carefully.
Sigh, I know that reading in the Internet age is a dying art. It's at the same stage as playing the Hammond B3:

https://www.youtube.com/watch?v=ZEST4CJlJkg

Cheers,
Ernst


Philip Robinson
 

Ernst   and Christian
Last problem today
I have now arrived in the instructions for editing the segdir line  in the MSG4.py file 
I cant get my head around the  segdir script line path  and what to enter.
I have opened up MSG4.py for editing and can see the example line thats there 'Z"/rec_0................... I do not know what to change this path to and also the date format.

I have lots of incoming data in several different directories of EUMETCAST/received 
three examples of these directories are below. I am using Windows10.
D:\EUMETCast\received\bas\Data Channel4
D:\EUMETCASTreceived\hvs-1\E1H-TPG-1
D:\EUMETCAST/received\hvs-2\default

and all the individual files contained in the received folder have date modified  eg 24/12/2021 20:42

Do I just change the path in  segdir to 'D:\EUMETCAST\received'    This is where my incoming data is received.
I am also not sure about what to enter  for the YEAR/MONTH/DAY structure

I could be barking up the wrong tree, and the segdir path is nothing like the above.

regards
Philip


------ Original Message ------
From: "Ernst Lobsiger via groups.io" <ernst.lobsiger@...>
Sent: 01/01/2022 19:30:26
Subject: Re: [MSG-1] Using begiiners guide to PyTROLL/SatPy Starter Kit V3.0 for Windows10 64bit

Philip,

my instructions say:

<CITE>

Go to

http://www.soest.hawaii.edu/wessel/gshhg/

gshhs and wdb files are part of GSHHG. Current version is GSHHG-2.3.7.
We need the native shape files that come in different resolutions and
can all be downloaded on the bottom of the page above. Look for the link:

CAUTION: Do not take the binary files as used by Hugo in EUMETCastView!


"
For ESRI shapefiles of shoreline polygons, rivers, and borders you can choose to get via ftp or http:

    FTP: GSHHG coastlines, political borders and rivers in shapefile format (zip archive)
    HTTP: GSHHG coastlines, political borders and rivers in shapefile format (zip archive)
"

You should get a file named gshhg-shp-2.3.7.zip

</CITE>

The soest page has not changed. The "bottom" does not mean the very last link. Please read my text carefully.
Sigh, I know that reading in the Internet age is a dying art. It's at the same stage as playing the Hammond B3:

https://www.youtube.com/watch?v=ZEST4CJlJkg

Cheers,
Ernst

Virus-free. www.avg.com


Philip Robinson
 

Hi
Just to add i have incoming files in my EUMETCast received folder eg AVHR_xxx_1B_MO1_2021124185203Z_20211224185503Z_N_O_20211224194312Z.bz2

I feel i am so close to producing my first image.
regards
Philip

------ Original Message ------
From: "Philip Robinson" <philip57@...>
Sent: 01/01/2022 23:20:16
Subject: Re: [MSG-1] Using begiiners guide to PyTROLL/SatPy Starter Kit V3.0 for Windows10 64bit

Ernst   and Christian
Last problem today
I have now arrived in the instructions for editing the segdir line  in the MSG4.py file 
I cant get my head around the  segdir script line path  and what to enter.
I have opened up MSG4.py for editing and can see the example line thats there 'Z"/rec_0................... I do not know what to change this path to and also the date format.

I have lots of incoming data in several different directories of EUMETCAST/received 
three examples of these directories are below. I am using Windows10.
D:\EUMETCast\received\bas\Data Channel4
D:\EUMETCASTreceived\hvs-1\E1H-TPG-1
D:\EUMETCAST/received\hvs-2\default

and all the individual files contained in the received folder have date modified  eg 24/12/2021 20:42

Do I just change the path in  segdir to 'D:\EUMETCAST\received'    This is where my incoming data is received.
I am also not sure about what to enter  for the YEAR/MONTH/DAY structure

I could be barking up the wrong tree, and the segdir path is nothing like the above.

regards
Philip


------ Original Message ------
From: "Ernst Lobsiger via groups.io" <ernst.lobsiger@...>
Sent: 01/01/2022 19:30:26
Subject: Re: [MSG-1] Using begiiners guide to PyTROLL/SatPy Starter Kit V3.0 for Windows10 64bit

Philip,

my instructions say:

<CITE>

Go to

http://www.soest.hawaii.edu/wessel/gshhg/

gshhs and wdb files are part of GSHHG. Current version is GSHHG-2.3.7.
We need the native shape files that come in different resolutions and
can all be downloaded on the bottom of the page above. Look for the link:

CAUTION: Do not take the binary files as used by Hugo in EUMETCastView!


"
For ESRI shapefiles of shoreline polygons, rivers, and borders you can choose to get via ftp or http:

    FTP: GSHHG coastlines, political borders and rivers in shapefile format (zip archive)
    HTTP: GSHHG coastlines, political borders and rivers in shapefile format (zip archive)
"

You should get a file named gshhg-shp-2.3.7.zip

</CITE>

The soest page has not changed. The "bottom" does not mean the very last link. Please read my text carefully.
Sigh, I know that reading in the Internet age is a dying art. It's at the same stage as playing the Hammond B3:

https://www.youtube.com/watch?v=ZEST4CJlJkg

Cheers,
Ernst

Virus-free. www.avg.com


Ernst Lobsiger
 

Philip,

first of all I assume you copied the MSG4.py into the USRscripts directory and work with this copy as I proposed!

Then you must have a minimal idea on how your EUMETCast reception system works. In the header of MSG4.py you read:

# EUMETSAT is EUMETCast disseminating MSG4 data in EUMETSAT_Data_Channel_2

As you are on Windows chances are that you have spaces " " (a very bad habit) instead of underlines "_".

Where these data go (the directory in your file system) you define in your "cast-client-channels_bas.ini" file.

Most probably this is "D:\received\bas\EUMETSAT Data Channel 2" . In this case you tell the MSG4.py

segdir = 'D:/received/bas/EUMETSAT Data Channel 2' 

 Python like GNU/Linux EXPECTS  FORWARD SLASHES "/" !!

The problem might be that, as a Windows user, you also use David's software with whatever data manager
I have no idea of. This may or may not interfere with what my scripts expect. If e.g. such a manager reads
data from "D:\received\bas\EUMETSAT Data Channel 2" processes these files making images and then deletes
the original files you are in a bad situation. But you can certainly tell this "manager" to retain the original files
at least for a while "somewhere". Then "somewhere" is the place you must tell MSG4.py to check. And I guess
that "somewhere" can even include a YYYY/MM/DD directory structure ...

Regards,
Ernst


BTW this is a Metop file:
AVHR_xxx_1B_MO1_2021124185203Z_20211224185503Z_N_O_20211224194312Z.bz2


Philip Robinson
 


David
I hope you can help me on this one.
I am working through Ernst Beginners guide to installing and using PyTROLL/SatPy for Windows10 and i am nearly there for producing the first images in the EMCData

The problem i am having is in the MSG4.py script, and that in the SegDir script  line i have to locate the path to my satellite data files that Tellicast receives. I understand they are in EUMETCast\received\\bas\DATAChannel 2 . I never really took much notice of that folder but i can see those files keep popping in and out continually . I presume these are the important raw files that MSG Data Manager uses ( and SatPy requires too) . I need to keep these files retained in there for a period of time so that they can be used in SatPy. I have ticked Keep raw data and HRIT retention period 2 days in the MSGData Manager Options  but the files still keep popping in and out of that Data Channel 2 folder.
How can I keep these files  retained in that folder for a period of time so that SatPy can make use of them
regards
Philip
PS  I am probably not seeing the option for retaining these raw files. I thought Keep raw data was the option i required to do.

------ Original Message ------
From: "Ernst Lobsiger via groups.io" <ernst.lobsiger@...>
Sent: 02/01/2022 09:31:34
Subject: Re: [MSG-1] Using begiiners guide to PyTROLL/SatPy Starter Kit V3.0 for Windows10 64bit

Philip,

first of all I assume you copied the MSG4.py into the USRscripts directory and work with this copy as I proposed!

Then you must have a minimal idea on how your EUMETCast reception system works. In the header of MSG4.py you read:

# EUMETSAT is EUMETCast disseminating MSG4 data in EUMETSAT_Data_Channel_2

As you are on Windows chances are that you have spaces " " (a very bad habit) instead of underlines "_".

Where these data go (the directory in your file system) you define in your "cast-client-channels_bas.ini" file.

Most probably this is "D:\received\bas\EUMETSAT Data Channel 2" . In this case you tell the MSG4.py

segdir = 'D:/received/bas/EUMETSAT Data Channel 2' 

 Python like GNU/Linux EXPECTS  FORWARD SLASHES "/" !!

The problem might be that, as a Windows user, you also use David's software with whatever data manager
I have no idea of. This may or may not interfere with what my scripts expect. If e.g. such a manager reads
data from "D:\received\bas\EUMETSAT Data Channel 2" processes these files making images and then deletes
the original files you are in a bad situation. But you can certainly tell this "manager" to retain the original files
at least for a while "somewhere". Then "somewhere" is the place you must tell MSG4.py to check. And I guess
that "somewhere" can even include a YYYY/MM/DD directory structure ...

Regards,
Ernst


BTW this is a Metop file:
AVHR_xxx_1B_MO1_2021124185203Z_20211224185503Z_N_O_20211224194312Z.bz2


Virus-free. www.avg.com


Graham Woolf
 

Hi Philip

What you need to do is set the Saved raw Data location in the MSG Data Manager (see screenshot)

Then use that location as your segdir

Regards

Graham


Graham Woolf
 

Hi Philip

Your SegDIR will then be

SegDir = "Z:/RawData/MSG" + "/" + Yea + "/" + Mon + "/" + Day

That assumes you have Saved Source Files in Date Tree ticked which I have

Regards

Graham


Ernst Lobsiger
 

Philip,

that's from message #30806 where you probably have the Kit 3.0 from:

<cite>
My definition of "Childs Play" remains the same as for V1.0:

 0) You know that you have Windows 10 64Bit
 1) You are not completely new to EUMETCast
 2) You read and follow the HOWTO inside (RTFM)
 3) You know where you find your satellite files
</cite>

You are now at point 3). I cannot help you as I do not know all these different data managers you might use.
I just hope you can retain your input data per EUMETCast data channel. Graham is certainly a good guide.

Regards,
Ernst



Philip Robinson
 

Hi Graham.
thanks for your info. 
However, when i look in my Raw data folder, which is in c:/MSG-1/Raw Data, it is empty. (See screenshots of my Data manager paths and also the retention i have  ticked) . The image satellite data seems to being received in EUMETCast/received/Data Channel 2 and i presume that is the path required in SegDir. The problem i have is that the satellite data is going in and out continually in that Data Channel 2 folder... i cannot keep seem to find a way of retaining the satellite data in that folder for any period of time, for then SegDir to make use of.
regards
Philip
By the way MsgDataManager produces images with no problem so i must have my folder paths entered  correctly.

------ Original Message ------
From: "Graham Woolf" <woolfg@...>
Sent: 02/01/2022 12:18:49
Subject: Re: [MSG-1] Using begiiners guide to PyTROLL/SatPy Starter Kit V3.0 for Windows10 64bit

Hi Philip

Your SegDIR will then be

SegDir = "Z:/RawData/MSG" + "/" + Yea + "/" + Mon + "/" + Day

That assumes you have Saved Source Files in Date Tree ticked which I have

Regards

Graham

Virus-free. www.avg.com


Graham Woolf
 

Hi Philip

Have you also set up the data under the Channel Selection tab as shown in my system screenshot

Regards

Graham