Satpy: GOES problems


R. Alblas
 

After managing without problems MSG and Himawari using satpy I have some problems with GOES.

After running command:

./GOES16.py 201712212200

I get following error message:

ValueError: found the following matches with the input file in xarray's IO backends: ['netcdf4', 'h5netcdf']. But their dependencies may not be installed, see:
http://xarray.pydata.org/en/stable/user-guide/io.html
http://xarray.pydata.org/en/stable/getting-started-guide/installing.html

I did do:

 conda install -c conda-forge xarray dask netCDF4 bottleneck
 conda install -c conda-forge xarray dask h5netcdf bottleneck

and also:

python -m pip install "xarray[complete]"  # Install all the above

as noted in the mentioned URL's.

There is a remark on the second URL:

The above commands should install most of the optional dependencies. However, some packages which are either not listed on PyPI or require extra installation steps are excluded. To know which dependencies would be installed, take a look at the [options.extras_require] section in setup.cfg:

but I am not sure what to do from here; I found several setup.cfg's.

Any idea's?

Regards,
Rob.




Ernst Lobsiger
 

On Tue, Feb 1, 2022 at 01:26 PM, R. Alblas wrote:
./GOES16.py 201712212200
Rob,

first of all miniconda3 should be installed as an unprivileged user.
Then satpy, pycoast and pyhdf should be installed into env (pytroll).

You start the above script in (pytroll) with: python GOES16.py 201712212200

You should NOT mix pip and conda installs. And conda installs must be made
in (pytroll), otherwise these installs go to (base). Pip and conda packages
can be packed differently. When I used Pip with virtualenv years ago I had to
"apt-get install gdal libhdf4-alt-dev" as root first and then install into the virual env
as unprivileged user with Pip:

satpy
pyorbital
pyspectral
netcdf4
h5netcdf
pycoeast
pyhdf

In miniconda3 I only install the 3 packages mentionned in the HOWTO 3.0
and in the second line of this post (satpy, pycoast and pyhdf)

Please check in conda environment (pytroll) that you have pyhdf:

(base) eumetcast@kallisto:~$ conda activate pytroll
(pytroll) eumetcast@kallisto:~$ conda list pyhdf
# packages in environment at /home/eumetcast/miniconda3/envs/pytroll:
#
# Name                    Version                   Build  Channel
pyhdf                     0.10.2           py38h8c074d7_2    conda-forge


Regards,
Ernst


R. Alblas
 

Hm, I did exactly that, in the first place.

And pyhdf is there:

(pytroll) ralblas@pc4:~$ conda list pyhdf
# packages in environment at /home/ralblas/miniconda3/envs/pytroll:
#
# Name                    Version                   Build  Channel
pyhdf                     0.10.3          py310h87b822d_1    conda-forge

And netcdf4:

(pytroll) ralblas@pc4:~$ conda list netcdf4
# packages in environment at /home/ralblas/miniconda3/envs/pytroll:
#
# Name                    Version                   Build  Channel
netcdf4                   1.5.8           nompi_py310hd7ca5b8_101    conda-forge

Still, GOES16 gives the mentioned error message.

Regards,

Rob.


On 02-02-2022 09:45, Ernst Lobsiger via groups.io wrote:
On Tue, Feb 1, 2022 at 01:26 PM, R. Alblas wrote:
./GOES16.py 201712212200
Rob,

first of all miniconda3 should be installed as an unprivileged user.
Then satpy, pycoast and pyhdf should be installed into env (pytroll).

You start the above script in (pytroll) with: python GOES16.py 201712212200

You should NOT mix pip and conda installs. And conda installs must be made
in (pytroll), otherwise these installs go to (base). Pip and conda packages
can be packed differently. When I used Pip with virtualenv years ago I had to
"apt-get install gdal libhdf4-alt-dev" as root first and then install into the virual env
as unprivileged user with Pip:

satpy
pyorbital
pyspectral
netcdf4
h5netcdf
pycoeast
pyhdf

In miniconda3 I only install the 3 packages mentionned in the HOWTO 3.0
and in the second line of this post (satpy, pycoast and pyhdf)

Please check in conda environment (pytroll) that you have pyhdf:

(base) eumetcast@kallisto:~$ conda activate pytroll
(pytroll) eumetcast@kallisto:~$ conda list pyhdf
# packages in environment at /home/eumetcast/miniconda3/envs/pytroll:
#
# Name                    Version                   Build  Channel
pyhdf                     0.10.2           py38h8c074d7_2    conda-forge


Regards,
Ernst


Jeskynar
 

Sometimes those two libraries can clash, as they might both have been built to support hdf4.
Maybe try new conda environments and only install one or the other?


On Tue, 1 Feb 2022 at 21:26, R. Alblas <sat@...> wrote:

After managing without problems MSG and Himawari using satpy I have some problems with GOES.

After running command:

./GOES16.py 201712212200

I get following error message:

ValueError: found the following matches with the input file in xarray's IO backends: ['netcdf4', 'h5netcdf']. But their dependencies may not be installed, see:
http://xarray.pydata.org/en/stable/user-guide/io.html
http://xarray.pydata.org/en/stable/getting-started-guide/installing.html

I did do:

 conda install -c conda-forge xarray dask netCDF4 bottleneck
 conda install -c conda-forge xarray dask h5netcdf bottleneck

and also:

python -m pip install "xarray[complete]"  # Install all the above

as noted in the mentioned URL's.

There is a remark on the second URL:

The above commands should install most of the optional dependencies. However, some packages which are either not listed on PyPI or require extra installation steps are excluded. To know which dependencies would be installed, take a look at the [options.extras_require] section in setup.cfg:

but I am not sure what to do from here; I found several setup.cfg's.

Any idea's?

Regards,
Rob.




Ernst Lobsiger
 

On Wed, Feb 2, 2022 at 05:17 AM, R. Alblas wrote:
Still, GOES16 gives the mentioned error message.
Rob,
...
# Why to hell is it not working?
# from satpy.utils import debug_on
# debug_on()
...

Can you uncomment the second and third line? Maybe this gives some more insight
(seems a reader problem !?). Also maybe you try with another SLOT of GOES16 data.

Regards,
Ernst


R. Alblas
 

Ernst,

DEBUG gives 3 messages, 2 of them about 'Reading ......abi_l1b.yaml' and 'Assigning to abi_l1b..'

One message is strange: Can't parse ..... (one of the .nc files).

The data I have here is rather old, from 2017, and it has a few files with an extra part in the name:

OR_ABI-L1b-RadF-M3C02_G16_s20173552145434_e20173552156201_c20173552156235-120000_0.nc

(extra: -120000_0)

I did rename that (removed the extra part) and the debug message is gone now, but still the same result.
Files are OK, I can read them without problems with xrit2pic.

I have to start receive new data, because this is the only set I have right now. Don't know if that would solve the problem.

Regards,

Rob.

On 02-02-2022 14:38, Ernst Lobsiger via groups.io wrote:
On Wed, Feb 2, 2022 at 05:17 AM, R. Alblas wrote:
Still, GOES16 gives the mentioned error message.
Rob,
...
# Why to hell is it not working?
# from satpy.utils import debug_on
# debug_on()
...

Can you uncomment the second and third line? Maybe this gives some more insight
(seems a reader problem !?). Also maybe you try with another SLOT of GOES16 data.

Regards,
Ernst


Ernst Lobsiger
 

On Wed, Feb 2, 2022 at 08:06 AM, R. Alblas wrote:

OR_ABI-L1b-RadF-M3C02_G16_s20173552145434_e20173552156201_c20173552156235-120000_0.nc

(extra: -120000_0)

I did rename that (removed the extra part) and the debug message is gone now, but still the same result.
Rob,

you should certainly not change these file names! Below is one recent slot of filenames from GOES16:

OR_ABI-L1b-RadF-M6C01_G16_s20220320300206_e20220320309514_c20220320309560-118900_0.nc
OR_ABI-L1b-RadF-M6C02_G16_s20220320300206_e20220320309514_c20220320309549-120000_0.nc
OR_ABI-L1b-RadF-M6C03_G16_s20220320300206_e20220320309514_c20220320309563-119000_0.nc
OR_ABI-L1b-RadF-M6C04_G16_s20220320300206_e20220320309514_c20220320309544.nc
OR_ABI-L1b-RadF-M6C05_G16_s20220320300206_e20220320309514_c20220320309557-119101_0.nc
OR_ABI-L1b-RadF-M6C06_G16_s20220320300206_e20220320309520_c20220320309552.nc
OR_ABI-L1b-RadF-M6C07_G16_s20220320300206_e20220320309526_c20220320309566.nc
OR_ABI-L1b-RadF-M6C08_G16_s20220320300206_e20220320309514_c20220320309569.nc
OR_ABI-L1b-RadF-M6C09_G16_s20220320300206_e20220320309520_c20220320309581.nc
OR_ABI-L1b-RadF-M6C10_G16_s20220320300206_e20220320309525_c20220320309572.nc
OR_ABI-L1b-RadF-M6C11_G16_s20220320300206_e20220320309514_c20220320309578.nc
OR_ABI-L1b-RadF-M6C12_G16_s20220320300206_e20220320309520_c20220320309569.nc
OR_ABI-L1b-RadF-M6C13_G16_s20220320300206_e20220320309525_c20220320309588.nc
OR_ABI-L1b-RadF-M6C14_G16_s20220320300206_e20220320309514_c20220320309584.nc
OR_ABI-L1b-RadF-M6C15_G16_s20220320300206_e20220320309520_c20220320309590.nc
OR_ABI-L1b-RadF-M6C16_G16_s20220320300206_e20220320309525_c20220320309575.nc

Channels C01,C02,C03,C05 have the extra -*_0 in their filenames. They also have different resolutions.
I attach the definition of the file content and corresponding naming that you find in .../satpy/etc/readers/

The reader parses according to the file patterns you find in this *.yaml file (therefore the error message).

Regards,
Ernst


Ernst Lobsiger
 

Rob,

and you could also try to read and display one single channel. "colorized_ir_clouds" just uses C13.

Ernst


R. Alblas
 

Same result.

Btw, channel names I have here are M3C02 etc., not M6C03. I need first 'fresh' data, I think...


On 02-02-2022 18:42, Ernst Lobsiger via groups.io wrote:
Rob,

and you could also try to read and display one single channel. "colorized_ir_clouds" just uses C13.

Ernst


Ernst Lobsiger
 

On Wed, Feb 2, 2022 at 10:14 AM, R. Alblas wrote:
Btw, channel names I have here are M3C02 etc., not M6C03. I need first 'fresh' data, I think...
Rob,

Above channel is C02. M3, M4, M6 is scan mode (see file pattern in *.yaml). M6 is now default.
https://cimss.ssec.wisc.edu/satellite-blog/archives/32657

Ernst


R. Alblas
 

No luck, I did remove the extra installed netCDF4 and h5netcdf, only pyhdf is present.

Also with new received files, I get the same result:

ValueError: found the following matches with the input file in xarray's IO backends: ['netcdf4', 'h5netcdf']. But their dependencies may not be installed, see:
http://xarray.pydata.org/en/stable/user-guide/io.html
http://xarray.pydata.org/en/stable/getting-started-guide/installing.html

Hm...

Regards,
Rob.

On 02-02-2022 19:36, Ernst Lobsiger via groups.io wrote:
On Wed, Feb 2, 2022 at 10:14 AM, R. Alblas wrote:
Btw, channel names I have here are M3C02 etc., not M6C03. I need first 'fresh' data, I think...
Rob,

Above channel is C02. M3, M4, M6 is scan mode (see file pattern in *.yaml). M6 is now default.
https://cimss.ssec.wisc.edu/satellite-blog/archives/32657

Ernst


Christian Peters
 

Rob,

try to install netcdf4-python from pip or netcdf4 from conda, I recommend conda if that's what I'm using.
Maybe try to setup a fresh env.

Regards,

Christian 


Am 03.02.22 um 17:26 schrieb R. Alblas:

No luck, I did remove the extra installed netCDF4 and h5netcdf, only pyhdf is present.

Also with new received files, I get the same result:

ValueError: found the following matches with the input file in xarray's IO backends: ['netcdf4', 'h5netcdf']. But their dependencies may not be installed, see:
http://xarray.pydata.org/en/stable/user-guide/io.html
http://xarray.pydata.org/en/stable/getting-started-guide/installing.html

Hm...

Regards,
Rob.

On 02-02-2022 19:36, Ernst Lobsiger via groups.io wrote:
On Wed, Feb 2, 2022 at 10:14 AM, R. Alblas wrote:
Btw, channel names I have here are M3C02 etc., not M6C03. I need first 'fresh' data, I think...
Rob,

Above channel is C02. M3, M4, M6 is scan mode (see file pattern in *.yaml). M6 is now default.
https://cimss.ssec.wisc.edu/satellite-blog/archives/32657

Ernst


Ernst Lobsiger
 

On Thu, Feb 3, 2022 at 08:26 AM, R. Alblas wrote:
No luck, I did remove the extra installed netCDF4 and h5netcdf, only pyhdf is present.
Rob,

I made a user rob and installed everything with latest stuff.
I do not have h5netcdf but do have netcdf4 in env (pytroll).

But I only installed satpy, pycoast and pyhdf, so netcdf4 came
with pyhdf this or another packet. I attach a (pytroll) conda list.

OS is Debian 10, GOES16 image works whithout a problem ...

Cheers,
Ernst


R. Alblas
 

Thanks, Ernst.

And now it works!
I did a fresh install, and even MSG didn't work anymore.
Then I found out that it complains about:

ModuleNotFoundError: No module named 'pycoast'

while searching in:

/usr/local/lib/python3.8/dist-packages/satpy/scene.py

etc.
Note: python 3.8, not python3.10 which was installed with miniconda!
This is because I did already have python 3.8 installed. First line of the scripts:

#!/usr/bin/python3

Normally you don't have to start this script using python, so I did start with ./MSG4.py....
Which gives the wrong python, not the one installed by miniconda.

Using 'python MSG4.py ....', it works.

I know, that is also how it is mentioned in the howto, but I think best is to remove the first line from the scripts, because this feature should not be used. (Except if miniconda isn't used for installation.)

Regards,

Rob.





On 03-02-2022 19:49, Ernst Lobsiger via groups.io wrote:
On Thu, Feb 3, 2022 at 08:26 AM, R. Alblas wrote:
No luck, I did remove the extra installed netCDF4 and h5netcdf, only pyhdf is present.
Rob,

I made a user rob and installed everything with latest stuff.
I do not have h5netcdf but do have netcdf4 in env (pytroll).

But I only installed satpy, pycoast and pyhdf, so netcdf4 came
with pyhdf this or another packet. I attach a (pytroll) conda list.

OS is Debian 10, GOES16 image works whithout a problem ...

Cheers,
Ernst


Jeskynar
 

The more standard way is to replace
#!/usr/bin/python3

with
#!/usr/bin/env python3

because that finds python3 in the PATH
and will pick up your conda python correctly


On Fri, 4 Feb 2022 at 10:03, R. Alblas <sat@...> wrote:

Thanks, Ernst.

And now it works!
I did a fresh install, and even MSG didn't work anymore.
Then I found out that it complains about:

ModuleNotFoundError: No module named 'pycoast'

while searching in:

/usr/local/lib/python3.8/dist-packages/satpy/scene.py

etc.
Note: python 3.8, not python3.10 which was installed with miniconda!
This is because I did already have python 3.8 installed. First line of the scripts:

#!/usr/bin/python3

Normally you don't have to start this script using python, so I did start with ./MSG4.py....
Which gives the wrong python, not the one installed by miniconda.

Using 'python MSG4.py ....', it works.

I know, that is also how it is mentioned in the howto, but I think best is to remove the first line from the scripts, because this feature should not be used. (Except if miniconda isn't used for installation.)

Regards,

Rob.





On 03-02-2022 19:49, Ernst Lobsiger via groups.io wrote:
On Thu, Feb 3, 2022 at 08:26 AM, R. Alblas wrote:
No luck, I did remove the extra installed netCDF4 and h5netcdf, only pyhdf is present.
Rob,

I made a user rob and installed everything with latest stuff.
I do not have h5netcdf but do have netcdf4 in env (pytroll).

But I only installed satpy, pycoast and pyhdf, so netcdf4 came
with pyhdf this or another packet. I attach a (pytroll) conda list.

OS is Debian 10, GOES16 image works whithout a problem ...

Cheers,
Ernst


R. Alblas
 

Learning all the time ;-)


On 04-02-2022 11:05, Jeskynar wrote:
The more standard way is to replace
#!/usr/bin/python3

with
#!/usr/bin/env python3

because that finds python3 in the PATH
and will pick up your conda python correctly


On Fri, 4 Feb 2022 at 10:03, R. Alblas <sat@...> wrote:

Thanks, Ernst.

And now it works!
I did a fresh install, and even MSG didn't work anymore.
Then I found out that it complains about:

ModuleNotFoundError: No module named 'pycoast'

while searching in:

/usr/local/lib/python3.8/dist-packages/satpy/scene.py

etc.
Note: python 3.8, not python3.10 which was installed with miniconda!
This is because I did already have python 3.8 installed. First line of the scripts:

#!/usr/bin/python3

Normally you don't have to start this script using python, so I did start with ./MSG4.py....
Which gives the wrong python, not the one installed by miniconda.

Using 'python MSG4.py ....', it works.

I know, that is also how it is mentioned in the howto, but I think best is to remove the first line from the scripts, because this feature should not be used. (Except if miniconda isn't used for installation.)

Regards,

Rob.





On 03-02-2022 19:49, Ernst Lobsiger via groups.io wrote:
On Thu, Feb 3, 2022 at 08:26 AM, R. Alblas wrote:
No luck, I did remove the extra installed netCDF4 and h5netcdf, only pyhdf is present.
Rob,

I made a user rob and installed everything with latest stuff.
I do not have h5netcdf but do have netcdf4 in env (pytroll).

But I only installed satpy, pycoast and pyhdf, so netcdf4 came
with pyhdf this or another packet. I attach a (pytroll) conda list.

OS is Debian 10, GOES16 image works whithout a problem ...

Cheers,
Ernst


Ernst Lobsiger
 

On Fri, Feb 4, 2022 at 02:45 AM, R. Alblas wrote:
Learning all the time ;-)
Rob,

when I started with PyTROLL/Satpy this was still Python 2.7.
There are 3 ways to install PyTROLL/Satpy under GNU/Linux.
I  tested them all, described in my  HOWTO 2.2 issue 2019/04/10

1) Install as admin to the system /usr/lib/... with pip
PRO:
Every user can start the scripts like ./myScript.py
Scripts can more easily be croned
CON:
You interfere with your distro installed Python stuff
Even more problems when Python 3 arrived
No way for a clean uninstall of all that

2) Preferred (recommended) install at that time was
Minimal apt-get install of of python-gdal and libhdf4-alt-dev
You use virtualenv and pip as unprivileged user
You start scripts in env (pytroll) python myScript.py
That's the way you can also make it work on a RASPI

3) HOWTO 3.0 is for GNU/Linux as well as Windows 10
You only use miniconda3 as unprivileged user
You start scripts in activated env (pytroll) python myScript.py

The shebang is still there to make it work for method 1)

Regards,
Ernst



Ernst Lobsiger
 

On Fri, Feb 4, 2022 at 02:05 AM, Jeskynar wrote:
The more standard way is to replace
#!/usr/bin/python3
 
with
#!/usr/bin/env python3
 
Jeskynar,

thanks for that. When I developed the scripts there was an environment variable PYTHONPATH
that could be set for using virtualenv. Then starting by name worked (but IIRC in one env only).
Starting my scripts in miniconda3 with (pytroll) python myScript.py also takes the right python
version as the environment path is changed accordingly (as described in my HOWTOs 2.0/3.0).

/usr/bin/env seems to be of later date. It's hard to keep up with all that and with python especially ...

Regards,
Ernst


Ernst Lobsiger
 

On Fri, Feb 4, 2022 at 05:04 AM, Ernst Lobsiger wrote:
/usr/bin/env seems to be of later date. It's hard to keep up with all that and with python especially ...
Rob and Jeskynar,

I had a look at program env (which was new to me) and indeed this finds the right version of python
in different environments like $,  $ (base), $ (pytroll). To use it as shebang "#!/usr/bin/env python3"
in my scripts under GNU/Linux you also *have* to make my scripts executable. The HOWTO says:

    "Please note that you do *NOT* have to set the scripts executable (e.g. with chmod +x *.py)."

To stay in sync with the Windows HOWTO I still recommend to start the scripts with "$ python myScript.py".
If I'll ever issue an advanced PyTROLL/Satpy HOWTO for GNU/Linux only I might reconsider the shebang.
But having to start a script as "$. /myScript.py" or "$ python myScript.py" does not make a big difference to me.

Best regards,
Ernst


R. Alblas
 

Ernst,

Maybe an idea to at least mention in the howto why to start scripts explicitly as an argument of 'python'.

Yes, the howto says:

"Please note that you do *NOT* have to set the scripts executable (e.g. with chmod +x *.py)."

but that does NOT say: you SHOULD not make them executable, and you MUST start as an argument of python.

"Duped by my foreknowledge" I thought: O, shebang, so I CAN make it executable and run as a script. Which did work partly, because there was some satpy leftover installed without conda (which I have removed now).

Just an idea, but maybe it's typical Dutch to want to know "why" instead of strictly following rules without knowing why... ;-)

Rob.


On 05-02-2022 10:53, Ernst Lobsiger via groups.io wrote:
On Fri, Feb 4, 2022 at 05:04 AM, Ernst Lobsiger wrote:
/usr/bin/env seems to be of later date. It's hard to keep up with all that and with python especially ...
Rob and Jeskynar,

I had a look at program env (which was new to me) and indeed this finds the right version of python
in different environments like $,  $ (base), $ (pytroll). To use it as shebang "#!/usr/bin/env python3"
in my scripts under GNU/Linux you also *have* to make my scripts executable. The HOWTO says:

    "Please note that you do *NOT* have to set the scripts executable (e.g. with chmod +x *.py)."

To stay in sync with the Windows HOWTO I still recommend to start the scripts with "$ python myScript.py".
If I'll ever issue an advanced PyTROLL/Satpy HOWTO for GNU/Linux only I might reconsider the shebang.
But having to start a script as "$. /myScript.py" or "$ python myScript.py" does not make a big difference to me.

Best regards,
Ernst