Date   

Cloud free Svalbard (Spitzbergen)

Hugo
 

Hi,

A nice image of a cloud free Svalbard. (NOAA-20 image Natural colors)
https://www.flickr.com/photos/137270544@N02/52107010583/in/datetaken-public/ 

grts,
Hugo


Re: NWC 'Cloud Types' on EUMETCast

Ernst Lobsiger
 

Graham,

there is no super simple solution. You need some minimal data manager for that. This program must recognize the file patterns and read out the start dates that is part of the filenames. The problem is somewhat complicated by the fact that some satellite files do not use YYYYMMDD but YYYYDOY (DayOfYear). Then the program moves or copies/deletes the files into your MountPoint/Channel/YYYY/MM/DD structure. As a byproduct not used or wanted files are deleted. Under Windows Hugo has long had his EumetCastWatcher as EUMETCastView "preprocessor" for that purpose:

https://hvanruys.github.io/eumetcastwatcher

AFAIK now also called DirUpdate

https://github.com/hvanruys/EumetcastDirUpdate

Starting with Hugo's ideas I have written a simple GNU/Linux CLI C-program (using mc and gcc) that does the job. I moves the files from my RAM-Disk (where they arrive in an /opt/Channel/segments structure) to HDD (into /srv/Channel/YYYY/MM/DD/segments) with copy and delete as these are two separate file systems. Alan Curnow has written his own graphical C++ solutions for Windows 10 and a CLI C++ program under GNU/Linux. Take it as a homework to write your own solution along the lines of Hugo's published work.

Good luck,
Ernst


Re: NWC 'Cloud Types' on EUMETCast

g-woolf@sky.com
 

Hi Ernst

My Sentinel3 data is just going into two directories E2H-S3A-02 and E2H-S3B-02 on my /srv/data/eumetcast/hvs-2/ E2H-S3A-02 and the same for  E2H-S3B-02 on my Linux receiving PC

What do I need to do to get them sorted into Year/Month/Day like your data

Kind Regards

Graham


Re: NWC 'Cloud Types' on EUMETCast

Ernst Lobsiger
 

Manu,

well done. You managed 10 passes with 16GB. My 24GB receiver also uses memory for a RAM disk + other PyTROLL processing in the background.
I mentionned Aitoff as a similar pseudo conic projection some posts back. It turns out that pycoast has a problem with parallels. So don't waste your
time with trying to setup an Aitoff area in areas.yaml. What works rather well and without gaps is 'scanice' (Scandinavia and Iceland) as distributed.

Cheers,
Ernst


Re: NWC 'Cloud Types' on EUMETCast

Manu
 

Mission successful but only as a ran=60 setting.
 
With this setting my 16 GB RAM was used up to 100%!


Re: NWC 'Cloud Types' on EUMETCast

g-woolf@sky.com
 

Hi Ernst

Thanks I will try those tomorrow hopefully

Kind Regards

Graham


Re: NWC 'Cloud Types' on EUMETCast

Ernst Lobsiger
 

Graham and Manu,

speaking of GSHHG levels

https://www.soest.hawaii.edu/pwessel/gshhg/

here is a little pycoast secret. If you just set level a number like 6, then you might note
that pycoast will draw levels 6, 5 , 4, 3, 2, 1. But pycoast accepts level lists like [1, 6]  !

Cheers,
Ernst


Re: NWC 'Cloud Types' on EUMETCast

Ernst Lobsiger
 

Graham,

well done! Aitoff is nice as well. But you have to make an entry in areas.yaml first.

https://proj.org/operations/projections/aitoff.html

If you miss Antarctica then set coastline level to 5 or 6 (see GSHHG soest page).

Ernst


Re: NWC 'Cloud Types' on EUMETCast

g-woolf@sky.com
 

HI Ernst

Just tried this and got a lovely image similar to yours

It used 35% of my available ram

Kind Regards

Graham


Re: NWC 'Cloud Types' on EUMETCast

Ernst Lobsiger
 

Graham and Manu,

so far so good! If you have monster RAM (Graham 128GB!) then try 'moll' with the same POI as 'ortho'.
As I said before I "only" have 24GB of RAM and my record before heavy disk swapping is 10 passes.
You already have 'moll' in areas.yaml. Below I set lon=0 lat=0 ran=60. Who dares to set ran=90 :-)  ?

Cheers,
Ernst

P.S. Such are the blessings of proj under the hood. But you must have all the ERR files of course ...


Re: NWC 'Cloud Types' on EUMETCast

Manu
 

Ernst,

successful tests also under windows!

Manu


Re: NWC 'Cloud Types' on EUMETCast

g-woolf@sky.com
 

Hi Ernst

Success - thanks for the help

Graham


Re: NWC 'Cloud Types' on EUMETCast

Ernst Lobsiger
 

Graham,

.... and remembering that you have 1.2km resolution at SSP you could even
increase in 'ortho' your image size to height: 10000 width: 10000 .... ;-)

Ernst

P.S. Don't publish an image of this size on MSG-1 though.


Re: NWC 'Cloud Types' on EUMETCast

g-woolf@sky.com
 

Hi Ernst

Thanks I'll give it a go

I have 128Gb of RAM so should be OK :-)

Kind Regards

Graham


Re: NWC 'Cloud Types' on EUMETCast

Ernst Lobsiger
 

Graham,

you will have to add an 'ortho' area to your EMCtools/pppconfig/areas.yaml (if you don't have any yet). I used:

ortho:
  description: Orthographic
  projection:
    proj: ortho
    ellps: WGS84
    lat_0: 20.0
    lon_0: 0.0
  shape:
    height: 1000
    width:  1000
  area_extent:
    lower_left_xy: [-6378137., -6378137.]
    upper_right_xy: [6378137.,  6378137.]

Then I defined a point of interest lon=0 lat=0 and ran=45. I already know that you will want to increase ran
to get more and more passes. My receiver has 24GB of ram and chockes with more than 10 ERR passes ...

Happy hacking,
Ernst


Re: NWC 'Cloud Types' on EUMETCast

g-woolf@sky.com
 

Hi Ernst

Thanks I'll have a look at my files later

Here is my first Sentinel image

How did you get the whole earth image - is it just a case of changing the area to ortho or are more changes needed ?

Its a great image

Kind Regards

Graham


Re: NWC 'Cloud Types' on EUMETCast

Ernst Lobsiger
 

Graham and Manu,

these are the numbers of HVS-2 OLCI ERR files on 2 error free tc receivers:

2022            Sentinel-3A        Sentinel-3B
May 19            14                       17
May 20            13                       16
May 21            10                       18

It seems that Sentinel-3B has more files than expected. Looking at May 21 with 18 files shows:

-rw-r--r-- 1 root root 730224640 May 21 04:10 S3B_OL_1_ERR____20220521T010844_20220521T015309_20220521T034336_2665_066_131______MAR_O_NR_002.SEN3.tar
-rw-r--r-- 1 root root 752793600 May 21 05:54 S3B_OL_1_ERR____20220521T024942_20220521T033408_20220521T053050_2666_066_132______MAR_O_NR_002.SEN3.tar
-rw-r--r-- 1 root root 732190720 May 21 07:35 S3B_OL_1_ERR____20220521T043041_20220521T051507_20220521T071247_2666_066_133______MAR_O_NR_002.SEN3.tar
-rw-r--r-- 1 root root 723302400 May 21 09:13 S3B_OL_1_ERR____20220521T061140_20220521T065606_20220521T084758_2666_066_134______MAR_O_NR_002.SEN3.tar
-rw-r--r-- 1 root root 724756480 May 21 10:48 S3B_OL_1_ERR____20220521T075239_20220521T083705_20220521T102101_2666_066_135______MAR_O_NR_002.SEN3.tar
-rw-r--r-- 1 root root 728811520 May 21 12:41 S3B_OL_1_ERR____20220521T093338_20220521T101804_20220521T120904_2666_066_136______MAR_O_NR_002.SEN3.tar
-rw-r--r-- 1 root root 728023040 May 21 14:25 S3B_OL_1_ERR____20220521T111437_20220521T115902_20220521T135854_2665_066_137______MAR_O_NR_002.SEN3.tar
-rw-r--r-- 1 root root 726958080 May 21 15:53 S3B_OL_1_ERR____20220521T125536_20220521T134001_20220521T152341_2665_066_138______MAR_O_NR_002.SEN3.tar
-rw-r--r-- 1 root root 743127040 May 21 17:37 S3B_OL_1_ERR____20220521T143635_20220521T152100_20220521T170823_2665_066_139______MAR_O_NR_002.SEN3.tar
-rw-r--r-- 1 root root 736358400 May 21 19:19 S3B_OL_1_ERR____20220521T161734_20220521T170159_20220521T185234_2665_066_140______MAR_O_NR_002.SEN3.tar
-rw-r--r-- 1 root root 602214400 May 21 21:02 S3B_OL_1_ERR____20220521T175833_20220521T183431_20220521T203720_2158_066_141______MAR_O_NR_002.SEN3.tar
-rw-r--r-- 1 root root  30914560 May 21 21:20 S3B_OL_1_ERR____20220521T183431_20220521T183631_20220521T211611_0120_066_141______MAR_O_NR_002.SEN3.tar
-rw-r--r-- 1 root root 105297920 May 21 20:45 S3B_OL_1_ERR____20220521T183631_20220521T184258_20220521T203733_0387_066_141______MAR_O_NR_002.SEN3.tar
-rw-r--r-- 1 root root 728832000 May 21 22:42 S3B_OL_1_ERR____20220521T193932_20220521T202357_20220521T221200_2665_066_142______MAR_O_NR_002.SEN3.tar
-rw-r--r-- 1 root root 735918080 May 22 00:22 S3B_OL_1_ERR____20220521T212031_20220521T220456_20220521T235111_2665_066_143______MAR_O_NR_002.SEN3.tar
-rw-r--r-- 1 root root 521789440 May 22 02:10 S3B_OL_1_ERR____20220521T230130_20220521T233328_20220522T014104_1917_066_144______MAR_O_NR_002.SEN3.tar
-rw-r--r-- 1 root root  30330880 May 22 02:22 S3B_OL_1_ERR____20220521T233328_20220521T233528_20220522T021908_0120_066_144______MAR_O_NR_002.SEN3.tar
-rw-r--r-- 1 root root 172339200 May 22 01:53 S3B_OL_1_ERR____20220521T233528_20220521T234555_20220522T014115_0627_066_144______MAR_O_NR_002.SEN3.tar

Apparently there are some ERR files far smaller than the expected 45 minutes (see also file sizes). My system sorts files into a YYYY/MM/DD structures according to start of segment. A full 45 minutes file starting 2022 May 1st 23:59 UTC will still be in directory 2022/05/01. My satpy script assumes 45 minute passes/segments only. I have to think and we'll see whether there could be a problem with these snipplets from Sentinel-3B.

Best Regards,
Ernst


Re: NWC 'Cloud Types' on EUMETCast

g-woolf@sky.com
 

Hi Ernst

Thanks  Ill keep an eye on it

I only set up my system to take this data yesterday morning so havent had a full day yet

Kind Regards

Graham


Re: NWC 'Cloud Types' on EUMETCast

Ernst Lobsiger
 

Graham,

different fom what I just said EUMETSAT expects ~18 files per day in the Product Navigator:

https://navigator.eumetsat.int/product/EO:EUM:DAT:SENTINEL-3:OL_1_ERR___NRT?query=&filter=satellite__Sentinel-3&filter=themes__Ocean&s=advanced

So there is room for further investigations ...

Ernst


Re: NWC 'Cloud Types' on EUMETCast

Ernst Lobsiger
 

Graham,

the second last entry in the TLEs shows the number of revolutions per day. It is 14.26 for Sentinel-3X.
This leads to a (for sun synchronous satellites!) typical nodal period of 1440/14.26 = 101 minutes.

Every ERR file is about 45 minutes showing the sunny side of the orbit (no OLCI data in the dark).

So expect 14 -15 files a day. I even got 16 once but found out that one orbit has been processed and
disseminated twice with slightly different starting times. You may loose files due to reception problems.

Regards,
Ernst

861 - 880 of 33922