Satpy: GOES problems
|
R. Alblas
After managing without problems MSG and Himawari using satpy I have some problems with GOES. After running command: ./GOES16.py 201712212200 I get following error message: ValueError: found the following matches with the input file in
xarray's IO backends: ['netcdf4', 'h5netcdf']. But their
dependencies may not be installed, see: I did do: conda install -c conda-forge xarray dask netCDF4 bottleneck and also: python -m pip install "xarray[complete]" # Install all the above as noted in the mentioned URL's. There is a remark on the second URL: The above commands should install most of the optional
dependencies. However,
some packages which are either not listed on PyPI or require
extra
installation steps are excluded. To know which dependencies
would be
installed, take a look at the but I am not sure what to do from here; I found several setup.cfg's. Any idea's? Regards,
|
|
|
|
Ernst Lobsiger
On Tue, Feb 1, 2022 at 01:26 PM, R. Alblas wrote:
./GOES16.py 201712212200Rob, first of all miniconda3 should be installed as an unprivileged user. Then satpy, pycoast and pyhdf should be installed into env (pytroll). You start the above script in (pytroll) with: python GOES16.py 201712212200 You should NOT mix pip and conda installs. And conda installs must be made in (pytroll), otherwise these installs go to (base). Pip and conda packages can be packed differently. When I used Pip with virtualenv years ago I had to "apt-get install gdal libhdf4-alt-dev" as root first and then install into the virual env as unprivileged user with Pip: satpy pyorbital pyspectral netcdf4 h5netcdf pycoeast pyhdf In miniconda3 I only install the 3 packages mentionned in the HOWTO 3.0 and in the second line of this post (satpy, pycoast and pyhdf) Please check in conda environment (pytroll) that you have pyhdf: (base) eumetcast@kallisto:~$ conda activate pytroll (pytroll) eumetcast@kallisto:~$ conda list pyhdf # packages in environment at /home/eumetcast/miniconda3/envs/pytroll: # # Name Version Build Channel pyhdf 0.10.2 py38h8c074d7_2 conda-forge Regards, Ernst |
|
|
|
R. Alblas
Hm, I did exactly that, in the first place. And pyhdf is there: (pytroll) ralblas@pc4:~$ conda list pyhdf And netcdf4: (pytroll) ralblas@pc4:~$ conda list netcdf4 Still, GOES16 gives the mentioned error message. Regards, Rob.
On 02-02-2022 09:45, Ernst Lobsiger via
groups.io wrote:
On Tue, Feb 1, 2022 at 01:26 PM, R. Alblas wrote: |
|
|
|
Jeskynar
Sometimes those two libraries can clash, as they might both have been built to support hdf4. Maybe try new conda environments and only install one or the other? On Tue, 1 Feb 2022 at 21:26, R. Alblas <sat@...> wrote:
|
|
|
|
Ernst Lobsiger
On Wed, Feb 2, 2022 at 05:17 AM, R. Alblas wrote:
Still, GOES16 gives the mentioned error message.Rob, ... # Why to hell is it not working? # from satpy.utils import debug_on # debug_on() ... Can you uncomment the second and third line? Maybe this gives some more insight (seems a reader problem !?). Also maybe you try with another SLOT of GOES16 data. Regards, Ernst |
|
|
|
R. Alblas
Ernst, DEBUG gives 3 messages, 2 of them about 'Reading
......abi_l1b.yaml' and 'Assigning to abi_l1b..' One message is strange: Can't parse ..... (one of the .nc files). The data I have here is rather old, from 2017, and it has a few files with an extra part in the name: OR_ABI-L1b-RadF-M3C02_G16_s20173552145434_e20173552156201_c20173552156235-120000_0.nc (extra: -120000_0) I did rename that (removed the extra part) and the debug message is gone now, but still the same result.Files are OK, I can read them without problems with xrit2pic. I have to start receive new data, because this is the only set I
have right now. Don't know if that would solve the problem. Regards, Rob. On 02-02-2022 14:38, Ernst Lobsiger via
groups.io wrote:
On Wed, Feb 2, 2022 at 05:17 AM, R. Alblas wrote: |
|
|
|
Ernst Lobsiger
On Wed, Feb 2, 2022 at 08:06 AM, R. Alblas wrote:
Rob, you should certainly not change these file names! Below is one recent slot of filenames from GOES16: OR_ABI-L1b-RadF-M6C01_G16_s20220320300206_e20220320309514_c20220320309560-118900_0.nc OR_ABI-L1b-RadF-M6C02_G16_s20220320300206_e20220320309514_c20220320309549-120000_0.nc OR_ABI-L1b-RadF-M6C03_G16_s20220320300206_e20220320309514_c20220320309563-119000_0.nc OR_ABI-L1b-RadF-M6C04_G16_s20220320300206_e20220320309514_c20220320309544.nc OR_ABI-L1b-RadF-M6C05_G16_s20220320300206_e20220320309514_c20220320309557-119101_0.nc OR_ABI-L1b-RadF-M6C06_G16_s20220320300206_e20220320309520_c20220320309552.nc OR_ABI-L1b-RadF-M6C07_G16_s20220320300206_e20220320309526_c20220320309566.nc OR_ABI-L1b-RadF-M6C08_G16_s20220320300206_e20220320309514_c20220320309569.nc OR_ABI-L1b-RadF-M6C09_G16_s20220320300206_e20220320309520_c20220320309581.nc OR_ABI-L1b-RadF-M6C10_G16_s20220320300206_e20220320309525_c20220320309572.nc OR_ABI-L1b-RadF-M6C11_G16_s20220320300206_e20220320309514_c20220320309578.nc OR_ABI-L1b-RadF-M6C12_G16_s20220320300206_e20220320309520_c20220320309569.nc OR_ABI-L1b-RadF-M6C13_G16_s20220320300206_e20220320309525_c20220320309588.nc OR_ABI-L1b-RadF-M6C14_G16_s20220320300206_e20220320309514_c20220320309584.nc OR_ABI-L1b-RadF-M6C15_G16_s20220320300206_e20220320309520_c20220320309590.nc OR_ABI-L1b-RadF-M6C16_G16_s20220320300206_e20220320309525_c20220320309575.nc Channels C01,C02,C03,C05 have the extra -*_0 in their filenames. They also have different resolutions. I attach the definition of the file content and corresponding naming that you find in .../satpy/etc/readers/ The reader parses according to the file patterns you find in this *.yaml file (therefore the error message). Regards, Ernst |
|
|
|
Ernst Lobsiger
Rob,
and you could also try to read and display one single channel. "colorized_ir_clouds" just uses C13. Ernst |
|
|
|
R. Alblas
Same result. Btw, channel names I have here are M3C02 etc., not M6C03. I need first 'fresh' data, I think...
On 02-02-2022 18:42, Ernst Lobsiger via
groups.io wrote:
Rob, |
|
|
|
Ernst Lobsiger
On Wed, Feb 2, 2022 at 10:14 AM, R. Alblas wrote:
Btw, channel names I have here are M3C02 etc., not M6C03. I need first 'fresh' data, I think...Rob, Above channel is C02. M3, M4, M6 is scan mode (see file pattern in *.yaml). M6 is now default. https://cimss.ssec.wisc.edu/satellite-blog/archives/32657 Ernst |
|
|
|
R. Alblas
No luck, I did remove the extra installed netCDF4 and h5netcdf, only pyhdf is present. Also with new received files, I get the same result: ValueError: found the following matches with the input file in
xarray's IO backends: ['netcdf4', 'h5netcdf']. But their
dependencies may not be installed, see: Hm... Regards, On 02-02-2022 19:36, Ernst Lobsiger via
groups.io wrote:
On Wed, Feb 2, 2022 at 10:14 AM, R. Alblas wrote: |
|
|
|
Christian Peters
Rob,
toggle quoted message
Show quoted text
try to install netcdf4-python from pip or netcdf4 from conda, I recommend conda if that's what I'm using. Maybe try to setup a fresh env. Regards, Christian Am 03.02.22 um 17:26 schrieb R. Alblas:
|
|
|
|
Ernst Lobsiger
On Thu, Feb 3, 2022 at 08:26 AM, R. Alblas wrote:
No luck, I did remove the extra installed netCDF4 and h5netcdf, only pyhdf is present.Rob, I made a user rob and installed everything with latest stuff. I do not have h5netcdf but do have netcdf4 in env (pytroll). But I only installed satpy, pycoast and pyhdf, so netcdf4 came with pyhdf this or another packet. I attach a (pytroll) conda list. OS is Debian 10, GOES16 image works whithout a problem ... Cheers, Ernst |
|
|
|
R. Alblas
Thanks, Ernst.
And now it works! I did a fresh install, and even MSG didn't work anymore. Then I found out that it complains about: ModuleNotFoundError: No module named 'pycoast' while searching in: /usr/local/lib/python3.8/dist-packages/satpy/scene.py etc.Note: python 3.8, not python3.10 which was installed with miniconda! This is because I did already have python 3.8 installed. First line of the scripts: #!/usr/bin/python3 Which gives the wrong python, not the one installed by miniconda. Using 'python MSG4.py ....', it works. I know, that is also how it is mentioned in the howto, but I think best is to remove the first line from the scripts, because this feature should not be used. (Except if miniconda isn't used for installation.) Regards, Rob.
On 03-02-2022 19:49, Ernst Lobsiger
via groups.io wrote:
On Thu, Feb 3, 2022 at 08:26 AM, R. Alblas wrote: |
|
|
|
Jeskynar
The more standard way is to replace #!/usr/bin/python3 with #!/usr/bin/env python3 because that finds python3 in the PATH and will pick up your conda python correctly On Fri, 4 Feb 2022 at 10:03, R. Alblas <sat@...> wrote:
|
|
|
|
R. Alblas
Learning all the time ;-)
On 04-02-2022 11:05, Jeskynar wrote:
|
|
|
|
Ernst Lobsiger
On Fri, Feb 4, 2022 at 02:45 AM, R. Alblas wrote:
Learning all the time ;-)Rob, when I started with PyTROLL/Satpy this was still Python 2.7. There are 3 ways to install PyTROLL/Satpy under GNU/Linux. I tested them all, described in my HOWTO 2.2 issue 2019/04/10 1) Install as admin to the system /usr/lib/... with pip PRO: Every user can start the scripts like ./myScript.py Scripts can more easily be croned CON: You interfere with your distro installed Python stuff Even more problems when Python 3 arrived No way for a clean uninstall of all that 2) Preferred (recommended) install at that time was Minimal apt-get install of of python-gdal and libhdf4-alt-dev You use virtualenv and pip as unprivileged user You start scripts in env (pytroll) python myScript.py That's the way you can also make it work on a RASPI 3) HOWTO 3.0 is for GNU/Linux as well as Windows 10 You only use miniconda3 as unprivileged user You start scripts in activated env (pytroll) python myScript.py The shebang is still there to make it work for method 1) Regards, Ernst |
|
|
|
Ernst Lobsiger
On Fri, Feb 4, 2022 at 02:05 AM, Jeskynar wrote:
The more standard way is to replaceJeskynar, thanks for that. When I developed the scripts there was an environment variable PYTHONPATH that could be set for using virtualenv. Then starting by name worked (but IIRC in one env only). Starting my scripts in miniconda3 with (pytroll) python myScript.py also takes the right python version as the environment path is changed accordingly (as described in my HOWTOs 2.0/3.0). /usr/bin/env seems to be of later date. It's hard to keep up with all that and with python especially ... Regards, Ernst |
|
|
|
Ernst Lobsiger
On Fri, Feb 4, 2022 at 05:04 AM, Ernst Lobsiger wrote:
/usr/bin/env seems to be of later date. It's hard to keep up with all that and with python especially ...Rob and Jeskynar, I had a look at program env (which was new to me) and indeed this finds the right version of python in different environments like $, $ (base), $ (pytroll). To use it as shebang "#!/usr/bin/env python3" in my scripts under GNU/Linux you also *have* to make my scripts executable. The HOWTO says: "Please note that you do *NOT* have to set the scripts executable (e.g. with chmod +x *.py)." To stay in sync with the Windows HOWTO I still recommend to start the scripts with "$ python myScript.py". If I'll ever issue an advanced PyTROLL/Satpy HOWTO for GNU/Linux only I might reconsider the shebang. But having to start a script as "$. /myScript.py" or "$ python myScript.py" does not make a big difference to me. Best regards, Ernst |
|
|
|
R. Alblas
Ernst, Maybe an idea to at least mention in the howto why to start scripts explicitly as an argument of 'python'. Yes, the howto says: "Please note that you do *NOT* have to set the scripts executable (e.g. with chmod +x *.py)." but that does NOT say: you SHOULD not make them executable, and you MUST start as an argument of python. "Duped by my foreknowledge" I
thought: O, shebang, so I CAN make it executable and run as a
script. Which did work partly, because there was some satpy
leftover installed without conda (which I have removed now). Just an idea, but maybe it's typical Dutch to want to know "why" instead of strictly following rules without knowing why... ;-) Rob.
On 05-02-2022 10:53, Ernst Lobsiger via
groups.io wrote:
On Fri, Feb 4, 2022 at 05:04 AM, Ernst Lobsiger wrote: |
|
|