new satpy scripts: night passes


R. Alblas
 

I have updated my script-set. For night mode late files from previous day (22:00...23:59 and early files from next day (00:00...01:59) are added to the set of files to process.

I didn't do a 'real' test (I have to receive  a suitable set to really test this), but selecting the extra files seem to work correctly.

See:

http://www.alblas.demon.nl/wsat/software/satpy.html#set2

(note that the common file 'satpy_ecast_rts.py' also changed).

Regards,
Rob.


On 02-04-2022 17:24, Ernst Lobsiger via groups.io wrote:
Rob and users of these scripts

All my scripts use a super simple method to find the files for LEO passes over a point of interest. There is no way to fully image passes when these include a date change. You always use files of just one date. While you will hardly find cases of single pass images that include one of these date changes it's far more probable if you produce images stacking more than one LEO pass. I attach two figures that show where (lon/lat) the problem can show up for NOAA-20. I depict a day pass (yellow) and a night pass (magenta) in othographic projection. Stacked images that span over these ground tracks are in danger. As we get only EARS data of NOAA-20, we only have to consider the night passes. The exact positions of the next few NOAA-20 date changes (about one repeat cycle) are marked as well.

Cheers,
Ernst


Ernst Lobsiger
 

Rob and interested PyTROLL/Satpy users,

I have solved the problem of a possible need for files from two dates in 2021 shortly after the publication of my Starter Kit 3.0. But I have not considered the issue to be all to severe and did not want to stress new PyTROLL/Satpy users that usually have *very* different problems. And further more my Kit doesn't have many stacked imaging scripts where a need for files from two dates is more probable. One thing I have to clarify is that this is not an issue of night passes only (though Ferdinand Valk discovered it while stacking NOOA-20 / Suomi-NPP night passes). If I specify in local time (Mean Solar Time) morning 06:00 - 12:00, afternoon 12:00 - 18:00, evening 18:00 - 24:00 and night 00:00 - 06:00, then we have four possible polar orbits for sun synchronous LEOs (below I use local times 10:00 and 14:00 as example):

1) Morning ascending        LTAN  10:00   --->     Evening descending LTDN = 22:00
2) Morning descending      LTDN  10:00   --->    Evening ascending    LTAN = 22:00
3) Afternoon ascending     LTAN  14:00   --->     Night descending      LTDN = 02:00
4) Afternoon descending   LTDN  14:00   --->     Night ascending        LTAN = 02:00


An observer in Greenwich, where UTC and Mean Solar Time is about the same, taking stacked (multi pass) evening images of a satellite with orbit type 1) or 2) will eventually have to use todays and tomorrows files, while for 3) or 4) orbit type satellites and stacking night passes todays and yesterdays files might be needed. If you go to other longitudes and use e.g. EUMETCast Metop-B/C GDS (Global Data Service) the UTC date change can be in plain local sunshine. The method to solve the problem I used follows:

a) For any given POI and satellite calculate the UTC time for a (theoretical) central ascending or descending pass overhead.
This can be done with Spherical Trigonometry and the satellites LTAN and inclination i (which are allmost constant for years).
b) From this UTC time look some search range (2 to 4 hours) back and forth and check whether this is yesterday or tomorrow.
c) According to the outcome you need segment files from today only, from today and yesterday or even from today and tomorrow.

I attach here four images of Metop-B and Metop-C that show their problematic regions and date changes for one full 29 day repeat cycle.
Rob, if my calculations are as exact as I think they should be ;-), you'll find a couple of dates and places to test your new scripts feature.

Cheers,
Ernst


R. Alblas
 

Ernst,

Just to clarify (maybe this was already clear from my mail): My scripts act as follows:

For night passes (that is, night UTC) I simply add 2 hours of files the day before and 2 hours of files the day after. Whether they are needed or not. I think that would be enough, with a earth pass of about 1.5 hours (but this is easy to extend, if needed).

The right files are then selected using the orbit and the area chosen, using the ' Nautical Triangle for arc distance' from your original script. So there is no intelligence involved in selecting what may be needed outside the day selected.

Calculations of in which cases 'other-day-passes' are needed are of course valuable, but are not used in some way in my scripts.

I'll see if I can catch some METOP files (I am not recording 'everything always').

Cheers,
Rob.

On 05-04-2022 19:57, Ernst Lobsiger via groups.io wrote:
Rob and interested PyTROLL/Satpy users,

I have solved the problem of a possible need for files from two dates in 2021 shortly after the publication of my Starter Kit 3.0. But I have not considered the issue to be all to severe and did not want to stress new PyTROLL/Satpy users that usually have *very* different problems. And further more my Kit doesn't have many stacked imaging scripts where a need for files from two dates is more probable. One thing I have to clarify is that this is not an issue of night passes only (though Ferdinand Valk discovered it while stacking NOOA-20 / Suomi-NPP night passes). If I specify in local time (Mean Solar Time) morning 06:00 - 12:00, afternoon 12:00 - 18:00, evening 18:00 - 24:00 and night 00:00 - 06:00, then we have four possible polar orbits for sun synchronous LEOs (below I use local times 10:00 and 14:00 as example):

1) Morning ascending        LTAN  10:00   --->     Evening descending LTDN = 22:00
2) Morning descending      LTDN  10:00   --->    Evening ascending    LTAN = 22:00
3) Afternoon ascending     LTAN  14:00   --->     Night descending      LTDN = 02:00
4) Afternoon descending   LTDN  14:00   --->     Night ascending        LTAN = 02:00


An observer in Greenwich, where UTC and Mean Solar Time is about the same, taking stacked (multi pass) evening images of a satellite with orbit type 1) or 2) will eventually have to use todays and tomorrows files, while for 3) or 4) orbit type satellites and stacking night passes todays and yesterdays files might be needed. If you go to other longitudes and use e.g. EUMETCast Metop-B/C GDS (Global Data Service) the UTC date change can be in plain local sunshine. The method to solve the problem I used follows:

a) For any given POI and satellite calculate the UTC time for a (theoretical) central ascending or descending pass overhead.
This can be done with Spherical Trigonometry and the satellites LTAN and inclination i (which are allmost constant for years).
b) From this UTC time look some search range (2 to 4 hours) back and forth and check whether this is yesterday or tomorrow.
c) According to the outcome you need segment files from today only, from today and yesterday or even from today and tomorrow.

I attach here four images of Metop-B and Metop-C that show their problematic regions and date changes for one full 29 day repeat cycle.
Rob, if my calculations are as exact as I think they should be ;-), you'll find a couple of dates and places to test your new scripts feature.

Cheers,
Ernst


Ernst Lobsiger
 

Rob,

by adding 2 hours from both yesterday and tomorrow don't you think that for multi pass scripts you risk to stack satellite passes side by side that are 24h off?
A  side effect of knowing UTC times for theoretical central passes overhead POIs is that you can add timeliness + 50 minutes for sheduling the script at hand.


Regards,
Ernst


R. Alblas
 

Eh... you're right. Either yesterday's or tomorrow's files have to be added, not both. So it's a bit more complicated.

Cheers,

Rob.


On 05-04-2022 22:27, Ernst Lobsiger via groups.io wrote:
Rob,

by adding 2 hours from both yesterday and tomorrow don't you think that for multi pass scripts you risk to stack satellite passes side by side that are 24h off?
A  side effect of knowing UTC times for theoretical central passes overhead POIs is that you can add timeliness + 50 minutes for sheduling the script at hand.


Regards,
Ernst


Ernst Lobsiger
 

Rob and interested PyTROLL/Satpy users,

yes indeed, it's more complicated. Here is your MOOSE-TEST. I use Metop-B passes over Lake Taupo.

# Lake Taupo (New Zealand)
lat = -38.78
lon = 175.92
ran =  20.0

laketaupo:
  description: Discussing PyTROLL/Satpy scripts on MSG-1
  projection:
    proj: stere
    ellps: WGS84
    lon_0: 175.92
    lat_0: -38.78
  shape:
    height: 500
    width:  500
  area_extent:
    lower_left_xy:  [ -2000000.0, -2000000.0]
    upper_right_xy: [ +2000000.0, +2000000.0]


Users will mostly image Europe and should not have problems with my scripts. But now Lake Taupo:
With my DAY script as distributed you will use files between 00 and 04 hours (SORRY, NOT GOOD!).
With my NIG script as distributed you will use files between 04 and 24 hours (Includes DAY+NIG).

For my (theoretical) center passes overhead Lake Taupo (output of my python script below) I get:

Satellite Metop-B / Position lat=-38.8 lon=175.9 / sunny=1 / OverHead pass expected at 21:18 UTC
Satellite Metop-B / Position lat=-38.8 lon=175.9 / sunny=0 / OverHead pass expected at 10:15 UTC

Reality is different in that you will have passes east and west of my (theoretical) center pass.
But taking a mean value of all maximum elevation times over a 29 day Metop-B repeat cycle should
bring you pretty much to my theoretical values. I attach DAY and NIG passes of April 1 - 5, 2022.


Happy hacking,
Ernst


R. Alblas
 

Ernst,

I can't reproduce your test here.

In your examples I see that images tagged as 'DAY' are actually UTC night and vice versa, so I think you did use different Metop-B_*.py's? (Because here DAY/NIG in the legend are hard-coded in the original .py scripts.)

To be sure I did try your original scripts, and used Metop-B TLE's of march 28 (day 87: 22087.29017244), as mentioned in your PassOverhead.py.

I have data for 20220401...20220410.

With:

python3 Metop-B_DAY.py 20220401

I get no good files. Same for 20220402 ... 20220406

From your images I would expect results for at least 20220405 and 20220404.

If I increase ran to 40 the I get some results.

With:

python3 Metop-B_NIG.py 20220401

I get 4 good files. Same (but 3 good files) for 20220402 ... 20220406.

Cheers,
Rob.

On 06-04-2022 11:33, Ernst Lobsiger via groups.io wrote:
Rob and interested PyTROLL/Satpy users,

yes indeed, it's more complicated. Here is your MOOSE-TEST. I use Metop-B passes over Lake Taupo.

# Lake Taupo (New Zealand)
lat = -38.78
lon = 175.92
ran =  20.0

laketaupo:
  description: Discussing PyTROLL/Satpy scripts on MSG-1
  projection:
    proj: stere
    ellps: WGS84
    lon_0: 175.92
    lat_0: -38.78
  shape:
    height: 500
    width:  500
  area_extent:
    lower_left_xy:  [ -2000000.0, -2000000.0]
    upper_right_xy: [ +2000000.0, +2000000.0]


Users will mostly image Europe and should not have problems with my scripts. But now Lake Taupo:
With my DAY script as distributed you will use files between 00 and 04 hours (SORRY, NOT GOOD!).
With my NIG script as distributed you will use files between 04 and 24 hours (Includes DAY+NIG).

For my (theoretical) center passes overhead Lake Taupo (output of my python script below) I get:

Satellite Metop-B / Position lat=-38.8 lon=175.9 / sunny=1 / OverHead pass expected at 21:18 UTC
Satellite Metop-B / Position lat=-38.8 lon=175.9 / sunny=0 / OverHead pass expected at 10:15 UTC

Reality is different in that you will have passes east and west of my (theoretical) center pass.
But taking a mean value of all maximum elevation times over a 29 day Metop-B repeat cycle should
bring you pretty much to my theoretical values. I attach DAY and NIG passes of April 1 - 5, 2022.


Happy hacking,
Ernst


Ernst Lobsiger
 

Rob,

as I said my original scripts fail with Lake Taupo. I made all these images with a script that uses the file selection based on the the additional Spherical Trigonometry stuff layed out in my attached PassOverhead.py. The TLEs I have used to calculate LTAN and read out inclination i do not matter at all. These two values will stay pretty much constant for many years. I only deduced these values from TLEs because the LTAN/LTDN values you find on the internet are sometimes hard to find and sometimes just give the orbit design values and have never been updated at all. LTAN as such is not in a TLE but can be calculated with pyorbital + RAAN quite easily.

As I said: I expect the Starter Kit 3.0 distributed Metop-B_DAY.py script not to work at all (no files found) and the *_NIG script will sometimes show NIG and sometimes DAY passes (whatever is closer to Lake Taupo at the choosen date). My post was just to demonstrate that my very simple version of file selection can fail not only in cases where you need files from 2 dates but also with big absolute values of longitudes. And DAY does *NOT* mean UTC day but it means pass in (local) daylight. And NIG does mean pass in (local) darkness.

Some posts back (#32970) I have explained how an improved file selection works. That's how the Metop-B script used here has been programed. And BTW it works for DAY and NIG: You just set sun=1 or sun=0 when calculating the time for a (theoretical) center pass time over the POI. Then the files selected must all be in an interval +/-6 hours from this UTC pass overhead time.


Cheers,
Ernst


R. Alblas
 

Ernst,

Some confusion.


as I said my original scripts fail with Lake Taupo. .....

As I said: I expect the Starter Kit 3.0 distributed Metop-B_DAY.py script not to work at all (no files found) ...

But this was not in your mail of 6-4-2022; instead, you wrote:

But now Lake Taupo:
With my DAY script as distributed you will use files between 00 and 04 hours (SORRY, NOT GOOD!).
With my NIG script as distributed you will use files between 04 and 24 hours (Includes DAY+NIG).

Which suggests that the original scripts do generate at least 'something'.
I now see why it doesn't work for (at least) the day script.

And it was also not clear how you did generate the images you attached.

Cheers,
Rob.


Ernst Lobsiger
 

Rob,

this is all about problems of my very crude version of selecting files. Maybe I was not explicit enough. I took it for granted that interested users play around with the atached script (and also read its comments) that issues for Lake Taupo as posted:

Satellite Metop-B / Position lat=-38.8 lon=175.9 / sunny=1 / OverHead pass expected at 21:18 UTC
Satellite Metop-B / Position lat=-38.8 lon=175.9 / sunny=0 / OverHead pass expected at 10:15 UTC

As my scripts always select the nearest pass over a POI in the first place it's also obvious that this will never be off more than app. +/- 50 minutes (half of a typical revolution of 100 minutes) from the time of a central overhead pass. And as the earth rotates 15° per hour and if the distributed Metop-B_DAY.py script is told to search for segment files 20° around Lake Taupo it's obvious to me that we cannot find such a file while looking at hours 00 to 04 UTC (which is way to far from 21:18 UTC). At the same time if the Metop-B_NIG.py script searches files from hours 04 to 24 UTC this includes ascending and descending passes and cannot work properly either. From the images attached it's at least for some dates easy to see what pass the distributed Metop-B_NIG.py script will select (e.g. I expect the ascending NIG pass at 20220403 but the descending DAY pass at 20220404, though I have not tested that with the original script). Because I wrote these PyTROLL/Satpy scripts all that is or looks maybe too obvious to me.

Regards,

Ernst



Ernst Lobsiger
 

On Thu, Apr 7, 2022 at 10:57 AM, R. Alblas wrote:

Ernst,

Some confusion.


as I said my original scripts fail with Lake Taupo. .....

As I said: I expect the Starter Kit 3.0 distributed Metop-B_DAY.py script not to work at all (no files found) ...
Rob and interested users of my PyTROLL/Satpy Starter Kit 3.0,

as became clear while discussing here my distributed scripts fail with Lake
Taupo. The problem is that my simple test to distinguish DAY from NIG works
with a sunny hours interval [d1,d2] determined from LTAN or LTDN whichever
is on the sunny side. For lon FAR east or lon FAR west this interval can be
shifted out of [0,24] hours and the test can fail. In the case of Lake Taupo we
easily find [d1,d2] = [-7,4] which is "NOT GOOD". Here is a simple way to fix
that. As an example I take the Metop-B_DAY.py and Metop-B_NIG.py scripts:




Find the lines defining d1 and d2 and ADD function sunny() below
...

# Metop-B is a morning satellite heading south
# LTAN = 21:30  (sometimes also called ECT asc)
# LTDN = 09:30  (sometimes also called ECT desc)
d1 = int(9. + 30./60. - lon/15. - 5.5)
d2 = int(9. + 30./60. - lon/15. + 6.5)

# Improved simple function to find sunny Hours
def sunny(iHou):
    if d1 >= 0 and d2 <= 24:
        return iHou > d1 and iHou < d2
    elif d1 < 0:
        return iHou < d2 or iHou > d1 + 24
    elif d2 > 24:
        return iHou > d1 or iHou < d2 - 24
    else:
        return False

...

For DAY scripts find the 2 lines of the Day crossing test below

    # Day crossings between d1 and d2
    if int(Hou) > d1 and int(Hou) < d2:

and REPLACE these 2 lines by the improved test function
 
    # DAY pass is sunny
    if sunny(int(Hou)):


For NIG scripts find the 2 lines of the Night crossing test below

    # Night crossings outside d1 to d2
    if int(Hou) < d1 or int(Hou) > d2:

and REPLACE these 2 lines by the improved test function
 
    # NIG pass is not sunny
    if not sunny(int(Hou)):



This has bee tested to work with Lake Taupo. It does not help
in the rare cases of date changes while passing over a POI.



Best regards,
Ernst


R. Alblas
 

I have updated my scripts.

The problem with 'For lon FAR east or lon FAR west' is solved functionally the same way (but a bit different coded).

Passes around midnight also work fine now. See:

http://www.alblas.demon.nl/wsat/software/satpy.html


Ernst, I did try your new code (MetopB-test.py); it works fine for australia.

You mentioned:


It does not help
in the rare cases of date changes while passing over a POI.
I don't know what you mean exactly, because it does work. Files used are:

AVHR_xxx_1B_M01_20220406235503Z_20220406235803Z_N_O_20220407002047Z.bz2
AVHR_xxx_1B_M01_20220406235803Z_20220407000103Z_N_O_20220407002125Z.bz2
AVHR_xxx_1B_M01_20220407000103Z_20220407000403Z_N_O_20220407002159Z.bz2
AVHR_xxx_1B_M01_20220407000403Z_20220407000703Z_N_O_20220407002235Z.bz2

I think you mean the second file with timestamps 20220406235803 to 20220407000103 would give problems?

Regards,

Rob.


Ernst Lobsiger
 

On Sat, Apr 9, 2022 at 05:56 AM, R. Alblas wrote:
You mentioned:


It does not help
in the rare cases of date changes while passing over a POI.
Rob,

if you take my original scripts and just add the sunny() function then Lake Taupo works but NOT Australia DAY 20220407.

On the other hand my recent script MetopB-test.py using module OverHead shows for 20220407DAY on the image and
in the filename 20220407 23:59. But the maximum elevation is actually 20220406 23:59. It turns out that there are two
daylight passes on 20220406: One with maximum elevation at 00:20 and the second with maximum elevation at 23:59.
So if you use in the legend the maximum elevation date/time and also in the filename, there is no DAY pass on 20220407.

I fixed the MetopB-test.py script to set back the date in such special cases. With 20220406DAY you get the first pass at
20220406 00:20 while entering 20220407DAY you get the second pass in legend and filename at 20220406 23:59 now.

Best regards,

Ernst