Date   

Re: Satpy: GOES problems

Ernst Lobsiger
 

On Fri, Feb 4, 2022 at 05:04 AM, Ernst Lobsiger wrote:
/usr/bin/env seems to be of later date. It's hard to keep up with all that and with python especially ...
Rob and Jeskynar,

I had a look at program env (which was new to me) and indeed this finds the right version of python
in different environments like $,  $ (base), $ (pytroll). To use it as shebang "#!/usr/bin/env python3"
in my scripts under GNU/Linux you also *have* to make my scripts executable. The HOWTO says:

    "Please note that you do *NOT* have to set the scripts executable (e.g. with chmod +x *.py)."

To stay in sync with the Windows HOWTO I still recommend to start the scripts with "$ python myScript.py".
If I'll ever issue an advanced PyTROLL/Satpy HOWTO for GNU/Linux only I might reconsider the shebang.
But having to start a script as "$. /myScript.py" or "$ python myScript.py" does not make a big difference to me.

Best regards,
Ernst


Re: EUMETCastView - some segments do not load!

Ernst Lobsiger
 

On Fri, Feb 4, 2022 at 10:28 AM, <lankosher@...> wrote:
Ok, it looks like some files are restricted to share across the home network and this is the reason why EumetCastView does not see some of them in the folders
Hi,

this has certainly nothing to do with EUMETCastView. Hugo's program needs just read only access. It works here from a TC receiver under GNU/Linux
that shares with NFS and SAMBA the raw data RO (ReadOnly) with other machines including Windows 10 with EUMETCastView and PyTROLL/Satpy.

What you need is a better understanding of your file system NTFS and of how to share data with your OS Windows (some maybe version dependant).

Ernst


Re: EUMETCastView - some segments do not load!

lankosher@...
 

Ok, it looks like some files are restricted to share across the home network and this is the reason why EumetCastView does not see some of them in the folders, which are on another computer. Some of them have this little padlock, which means they are restricted. Not really sure If this restriction can be taken down, but my idea of having a software on one machine, and load the files from another does not seem to be feasible anymore, unless you have some ideas :)


Re: EUMETCastView - some segments do not load!

Hugo
 

Hi,

Meteosat-11 segments are received in 'EUMETSAT_Data_Channel_2'.  
It is always wise to use some tool to put the files in directories like 'EUMETSAT_Data_Channel_2/YYYY/MM/DD'. That way you avoid a long search through the entire channel directory.
Can you see files received from Meteosat-10 or -8 ?

grts,

Hugo


EUMETCastView - some segments do not load!

lankosher@...
 

Hi Colleagues,
I have tried to configure EumetCastView to process my BAS and HVS1 channels, setting up the correct paths to the RX folders and to my surprise it is not loading Eumetsat-11 but e.g. GOES 16 & 17 are ok. (MSG-4) Eumetsat-11 segments are dropping into the RX folder, but for some reason when I press reload they're not displaying where they suppose to. At the same time, MSG Data Manager sees and processes them with no issues. To avoid interference between those 2 clients, I switched off MSG DM but it did not help in any way. Chunks keep on falling into the right folder but EumetCastView will not see a thing. The same thing is with NOAA20, despite active subscription, the HVS-1 folder seems to pile up with the segments but they are not loading. TLE's have been updated accordingly. 
Can you please advise what and where I should check to make it run?


Re: Satpy: GOES problems

Ernst Lobsiger
 

On Fri, Feb 4, 2022 at 02:05 AM, Jeskynar wrote:
The more standard way is to replace
#!/usr/bin/python3
 
with
#!/usr/bin/env python3
 
Jeskynar,

thanks for that. When I developed the scripts there was an environment variable PYTHONPATH
that could be set for using virtualenv. Then starting by name worked (but IIRC in one env only).
Starting my scripts in miniconda3 with (pytroll) python myScript.py also takes the right python
version as the environment path is changed accordingly (as described in my HOWTOs 2.0/3.0).

/usr/bin/env seems to be of later date. It's hard to keep up with all that and with python especially ...

Regards,
Ernst


Re: Satpy: GOES problems

Ernst Lobsiger
 

On Fri, Feb 4, 2022 at 02:45 AM, R. Alblas wrote:
Learning all the time ;-)
Rob,

when I started with PyTROLL/Satpy this was still Python 2.7.
There are 3 ways to install PyTROLL/Satpy under GNU/Linux.
I  tested them all, described in my  HOWTO 2.2 issue 2019/04/10

1) Install as admin to the system /usr/lib/... with pip
PRO:
Every user can start the scripts like ./myScript.py
Scripts can more easily be croned
CON:
You interfere with your distro installed Python stuff
Even more problems when Python 3 arrived
No way for a clean uninstall of all that

2) Preferred (recommended) install at that time was
Minimal apt-get install of of python-gdal and libhdf4-alt-dev
You use virtualenv and pip as unprivileged user
You start scripts in env (pytroll) python myScript.py
That's the way you can also make it work on a RASPI

3) HOWTO 3.0 is for GNU/Linux as well as Windows 10
You only use miniconda3 as unprivileged user
You start scripts in activated env (pytroll) python myScript.py

The shebang is still there to make it work for method 1)

Regards,
Ernst



Re: Satpy: GOES problems

R. Alblas
 

Learning all the time ;-)


On 04-02-2022 11:05, Jeskynar wrote:
The more standard way is to replace
#!/usr/bin/python3

with
#!/usr/bin/env python3

because that finds python3 in the PATH
and will pick up your conda python correctly


On Fri, 4 Feb 2022 at 10:03, R. Alblas <sat@...> wrote:

Thanks, Ernst.

And now it works!
I did a fresh install, and even MSG didn't work anymore.
Then I found out that it complains about:

ModuleNotFoundError: No module named 'pycoast'

while searching in:

/usr/local/lib/python3.8/dist-packages/satpy/scene.py

etc.
Note: python 3.8, not python3.10 which was installed with miniconda!
This is because I did already have python 3.8 installed. First line of the scripts:

#!/usr/bin/python3

Normally you don't have to start this script using python, so I did start with ./MSG4.py....
Which gives the wrong python, not the one installed by miniconda.

Using 'python MSG4.py ....', it works.

I know, that is also how it is mentioned in the howto, but I think best is to remove the first line from the scripts, because this feature should not be used. (Except if miniconda isn't used for installation.)

Regards,

Rob.





On 03-02-2022 19:49, Ernst Lobsiger via groups.io wrote:
On Thu, Feb 3, 2022 at 08:26 AM, R. Alblas wrote:
No luck, I did remove the extra installed netCDF4 and h5netcdf, only pyhdf is present.
Rob,

I made a user rob and installed everything with latest stuff.
I do not have h5netcdf but do have netcdf4 in env (pytroll).

But I only installed satpy, pycoast and pyhdf, so netcdf4 came
with pyhdf this or another packet. I attach a (pytroll) conda list.

OS is Debian 10, GOES16 image works whithout a problem ...

Cheers,
Ernst


Re: Satpy: GOES problems

Jeskynar
 

The more standard way is to replace
#!/usr/bin/python3

with
#!/usr/bin/env python3

because that finds python3 in the PATH
and will pick up your conda python correctly


On Fri, 4 Feb 2022 at 10:03, R. Alblas <sat@...> wrote:

Thanks, Ernst.

And now it works!
I did a fresh install, and even MSG didn't work anymore.
Then I found out that it complains about:

ModuleNotFoundError: No module named 'pycoast'

while searching in:

/usr/local/lib/python3.8/dist-packages/satpy/scene.py

etc.
Note: python 3.8, not python3.10 which was installed with miniconda!
This is because I did already have python 3.8 installed. First line of the scripts:

#!/usr/bin/python3

Normally you don't have to start this script using python, so I did start with ./MSG4.py....
Which gives the wrong python, not the one installed by miniconda.

Using 'python MSG4.py ....', it works.

I know, that is also how it is mentioned in the howto, but I think best is to remove the first line from the scripts, because this feature should not be used. (Except if miniconda isn't used for installation.)

Regards,

Rob.





On 03-02-2022 19:49, Ernst Lobsiger via groups.io wrote:
On Thu, Feb 3, 2022 at 08:26 AM, R. Alblas wrote:
No luck, I did remove the extra installed netCDF4 and h5netcdf, only pyhdf is present.
Rob,

I made a user rob and installed everything with latest stuff.
I do not have h5netcdf but do have netcdf4 in env (pytroll).

But I only installed satpy, pycoast and pyhdf, so netcdf4 came
with pyhdf this or another packet. I attach a (pytroll) conda list.

OS is Debian 10, GOES16 image works whithout a problem ...

Cheers,
Ernst


Re: Satpy: GOES problems

R. Alblas
 

Thanks, Ernst.

And now it works!
I did a fresh install, and even MSG didn't work anymore.
Then I found out that it complains about:

ModuleNotFoundError: No module named 'pycoast'

while searching in:

/usr/local/lib/python3.8/dist-packages/satpy/scene.py

etc.
Note: python 3.8, not python3.10 which was installed with miniconda!
This is because I did already have python 3.8 installed. First line of the scripts:

#!/usr/bin/python3

Normally you don't have to start this script using python, so I did start with ./MSG4.py....
Which gives the wrong python, not the one installed by miniconda.

Using 'python MSG4.py ....', it works.

I know, that is also how it is mentioned in the howto, but I think best is to remove the first line from the scripts, because this feature should not be used. (Except if miniconda isn't used for installation.)

Regards,

Rob.





On 03-02-2022 19:49, Ernst Lobsiger via groups.io wrote:
On Thu, Feb 3, 2022 at 08:26 AM, R. Alblas wrote:
No luck, I did remove the extra installed netCDF4 and h5netcdf, only pyhdf is present.
Rob,

I made a user rob and installed everything with latest stuff.
I do not have h5netcdf but do have netcdf4 in env (pytroll).

But I only installed satpy, pycoast and pyhdf, so netcdf4 came
with pyhdf this or another packet. I attach a (pytroll) conda list.

OS is Debian 10, GOES16 image works whithout a problem ...

Cheers,
Ernst


Re: Satpy: GOES problems

Ernst Lobsiger
 

On Thu, Feb 3, 2022 at 08:26 AM, R. Alblas wrote:
No luck, I did remove the extra installed netCDF4 and h5netcdf, only pyhdf is present.
Rob,

I made a user rob and installed everything with latest stuff.
I do not have h5netcdf but do have netcdf4 in env (pytroll).

But I only installed satpy, pycoast and pyhdf, so netcdf4 came
with pyhdf this or another packet. I attach a (pytroll) conda list.

OS is Debian 10, GOES16 image works whithout a problem ...

Cheers,
Ernst


Re: Satpy: GOES problems

Christian Peters
 

Rob,

try to install netcdf4-python from pip or netcdf4 from conda, I recommend conda if that's what I'm using.
Maybe try to setup a fresh env.

Regards,

Christian 


Am 03.02.22 um 17:26 schrieb R. Alblas:

No luck, I did remove the extra installed netCDF4 and h5netcdf, only pyhdf is present.

Also with new received files, I get the same result:

ValueError: found the following matches with the input file in xarray's IO backends: ['netcdf4', 'h5netcdf']. But their dependencies may not be installed, see:
http://xarray.pydata.org/en/stable/user-guide/io.html
http://xarray.pydata.org/en/stable/getting-started-guide/installing.html

Hm...

Regards,
Rob.

On 02-02-2022 19:36, Ernst Lobsiger via groups.io wrote:
On Wed, Feb 2, 2022 at 10:14 AM, R. Alblas wrote:
Btw, channel names I have here are M3C02 etc., not M6C03. I need first 'fresh' data, I think...
Rob,

Above channel is C02. M3, M4, M6 is scan mode (see file pattern in *.yaml). M6 is now default.
https://cimss.ssec.wisc.edu/satellite-blog/archives/32657

Ernst


Re: Satpy: GOES problems

R. Alblas
 

No luck, I did remove the extra installed netCDF4 and h5netcdf, only pyhdf is present.

Also with new received files, I get the same result:

ValueError: found the following matches with the input file in xarray's IO backends: ['netcdf4', 'h5netcdf']. But their dependencies may not be installed, see:
http://xarray.pydata.org/en/stable/user-guide/io.html
http://xarray.pydata.org/en/stable/getting-started-guide/installing.html

Hm...

Regards,
Rob.

On 02-02-2022 19:36, Ernst Lobsiger via groups.io wrote:
On Wed, Feb 2, 2022 at 10:14 AM, R. Alblas wrote:
Btw, channel names I have here are M3C02 etc., not M6C03. I need first 'fresh' data, I think...
Rob,

Above channel is C02. M3, M4, M6 is scan mode (see file pattern in *.yaml). M6 is now default.
https://cimss.ssec.wisc.edu/satellite-blog/archives/32657

Ernst


Re: Satpy: GOES problems

Ernst Lobsiger
 

On Wed, Feb 2, 2022 at 10:14 AM, R. Alblas wrote:
Btw, channel names I have here are M3C02 etc., not M6C03. I need first 'fresh' data, I think...
Rob,

Above channel is C02. M3, M4, M6 is scan mode (see file pattern in *.yaml). M6 is now default.
https://cimss.ssec.wisc.edu/satellite-blog/archives/32657

Ernst


Re: Satpy: GOES problems

R. Alblas
 

Same result.

Btw, channel names I have here are M3C02 etc., not M6C03. I need first 'fresh' data, I think...


On 02-02-2022 18:42, Ernst Lobsiger via groups.io wrote:
Rob,

and you could also try to read and display one single channel. "colorized_ir_clouds" just uses C13.

Ernst


Re: Satpy: GOES problems

Ernst Lobsiger
 

Rob,

and you could also try to read and display one single channel. "colorized_ir_clouds" just uses C13.

Ernst


Re: Satpy: GOES problems

Ernst Lobsiger
 

On Wed, Feb 2, 2022 at 08:06 AM, R. Alblas wrote:

OR_ABI-L1b-RadF-M3C02_G16_s20173552145434_e20173552156201_c20173552156235-120000_0.nc

(extra: -120000_0)

I did rename that (removed the extra part) and the debug message is gone now, but still the same result.
Rob,

you should certainly not change these file names! Below is one recent slot of filenames from GOES16:

OR_ABI-L1b-RadF-M6C01_G16_s20220320300206_e20220320309514_c20220320309560-118900_0.nc
OR_ABI-L1b-RadF-M6C02_G16_s20220320300206_e20220320309514_c20220320309549-120000_0.nc
OR_ABI-L1b-RadF-M6C03_G16_s20220320300206_e20220320309514_c20220320309563-119000_0.nc
OR_ABI-L1b-RadF-M6C04_G16_s20220320300206_e20220320309514_c20220320309544.nc
OR_ABI-L1b-RadF-M6C05_G16_s20220320300206_e20220320309514_c20220320309557-119101_0.nc
OR_ABI-L1b-RadF-M6C06_G16_s20220320300206_e20220320309520_c20220320309552.nc
OR_ABI-L1b-RadF-M6C07_G16_s20220320300206_e20220320309526_c20220320309566.nc
OR_ABI-L1b-RadF-M6C08_G16_s20220320300206_e20220320309514_c20220320309569.nc
OR_ABI-L1b-RadF-M6C09_G16_s20220320300206_e20220320309520_c20220320309581.nc
OR_ABI-L1b-RadF-M6C10_G16_s20220320300206_e20220320309525_c20220320309572.nc
OR_ABI-L1b-RadF-M6C11_G16_s20220320300206_e20220320309514_c20220320309578.nc
OR_ABI-L1b-RadF-M6C12_G16_s20220320300206_e20220320309520_c20220320309569.nc
OR_ABI-L1b-RadF-M6C13_G16_s20220320300206_e20220320309525_c20220320309588.nc
OR_ABI-L1b-RadF-M6C14_G16_s20220320300206_e20220320309514_c20220320309584.nc
OR_ABI-L1b-RadF-M6C15_G16_s20220320300206_e20220320309520_c20220320309590.nc
OR_ABI-L1b-RadF-M6C16_G16_s20220320300206_e20220320309525_c20220320309575.nc

Channels C01,C02,C03,C05 have the extra -*_0 in their filenames. They also have different resolutions.
I attach the definition of the file content and corresponding naming that you find in .../satpy/etc/readers/

The reader parses according to the file patterns you find in this *.yaml file (therefore the error message).

Regards,
Ernst


Re: Satpy: GOES problems

R. Alblas
 

Ernst,

DEBUG gives 3 messages, 2 of them about 'Reading ......abi_l1b.yaml' and 'Assigning to abi_l1b..'

One message is strange: Can't parse ..... (one of the .nc files).

The data I have here is rather old, from 2017, and it has a few files with an extra part in the name:

OR_ABI-L1b-RadF-M3C02_G16_s20173552145434_e20173552156201_c20173552156235-120000_0.nc

(extra: -120000_0)

I did rename that (removed the extra part) and the debug message is gone now, but still the same result.
Files are OK, I can read them without problems with xrit2pic.

I have to start receive new data, because this is the only set I have right now. Don't know if that would solve the problem.

Regards,

Rob.

On 02-02-2022 14:38, Ernst Lobsiger via groups.io wrote:
On Wed, Feb 2, 2022 at 05:17 AM, R. Alblas wrote:
Still, GOES16 gives the mentioned error message.
Rob,
...
# Why to hell is it not working?
# from satpy.utils import debug_on
# debug_on()
...

Can you uncomment the second and third line? Maybe this gives some more insight
(seems a reader problem !?). Also maybe you try with another SLOT of GOES16 data.

Regards,
Ernst


Re: Satpy: GOES problems

Ernst Lobsiger
 

On Wed, Feb 2, 2022 at 05:17 AM, R. Alblas wrote:
Still, GOES16 gives the mentioned error message.
Rob,
...
# Why to hell is it not working?
# from satpy.utils import debug_on
# debug_on()
...

Can you uncomment the second and third line? Maybe this gives some more insight
(seems a reader problem !?). Also maybe you try with another SLOT of GOES16 data.

Regards,
Ernst


Re: Satpy: GOES problems

Jeskynar
 

Sometimes those two libraries can clash, as they might both have been built to support hdf4.
Maybe try new conda environments and only install one or the other?


On Tue, 1 Feb 2022 at 21:26, R. Alblas <sat@...> wrote:

After managing without problems MSG and Himawari using satpy I have some problems with GOES.

After running command:

./GOES16.py 201712212200

I get following error message:

ValueError: found the following matches with the input file in xarray's IO backends: ['netcdf4', 'h5netcdf']. But their dependencies may not be installed, see:
http://xarray.pydata.org/en/stable/user-guide/io.html
http://xarray.pydata.org/en/stable/getting-started-guide/installing.html

I did do:

 conda install -c conda-forge xarray dask netCDF4 bottleneck
 conda install -c conda-forge xarray dask h5netcdf bottleneck

and also:

python -m pip install "xarray[complete]"  # Install all the above

as noted in the mentioned URL's.

There is a remark on the second URL:

The above commands should install most of the optional dependencies. However, some packages which are either not listed on PyPI or require extra installation steps are excluded. To know which dependencies would be installed, take a look at the [options.extras_require] section in setup.cfg:

but I am not sure what to do from here; I found several setup.cfg's.

Any idea's?

Regards,
Rob.



721 - 740 of 33391