GOES ABI Manager


maurizio.calvitti@...
 

Good evening everyone,

for a couple of days GOES ABI manager is not able to decode raw files.

In the Log file I read  a long list like this (both for G17 and G16)

2021-03-20 18:44:18      OR_ABI-L1b-RadF-M6C07_G17_s20210791830321_e20210791839399_c20210791839433.nc failed

2021-03-20 18:44:18      OR_ABI-L1b-RadF-M6C08_G17_s20210791830321_e20210791839388_c20210791839429.nc failed

2021-03-20 18:44:18      OR_ABI-L1b-RadF-M6C09_G17_s20210791830321_e20210791839394_c20210791839436.nc failed

2021-03-20 18:44:18      OR_ABI-L1b-RadF-M6C10_G17_s20210791830321_e20210791839399_c20210791839440.nc failed

 

Someone may help me in understanding what is happening?

 

Thanks a lot

Maurizio

 

 




Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle persone indicate e la casella di posta elettronica da cui è stata inviata è da qualificarsi quale strumento aziendale.

La diffusione, copia o qualsiasi altra azione derivante dalla conoscenza di queste informazioni sono rigorosamente vietate (art. 616 c.p, D.Lgs. n. 196/2003 s.m.i. e GDPR Regolamento - UE 2016/679).

Qualora abbiate ricevuto questo documento per errore siete cortesemente pregati di darne immediata comunicazione al mittente e di provvedere alla sua distruzione. Grazie.

This e-mail and any attachments is confidential and may contain privileged information intended for the addressee(s) only.

Dissemination, copying, printing or use by anybody else is unauthorised (art. 616 c.p, D.Lgs. n. 196/2003 and subsequent amendments and GDPR UE 2016/679).

If you are not the intended recipient, please delete this message and any attachments and advise the sender by return e-mail. Thanks.



Graham Woolf
 

Hi Maurizio

I have seen this a few times

What works for me is to empty the directory containing the raw data . It seems that if you have too  many files to decode the ABI Manager seems to stop working, I dont know why

Regards

Graham


maurizio.calvitti@...
 

Hi Graham,

to tell you the truth I empty the directory at  1 h interval….but without solving the problem.

It is like GOES ABI Manager are not able to decode files.

 

I will try with Eumetcastview to test the files and let you know.

 

Regards,

Maurizio

 

Da: MSG-1@groups.io <MSG-1@groups.io> Per conto di Graham Woolf
Inviato: sabato 20 marzo 2021 20:06
A: MSG-1@groups.io
Oggetto: Re: [MSG-1] GOES ABI Manager

 

Hi Maurizio

I have seen this a few times

What works for me is to empty the directory containing the raw data . It seems that if you have too  many files to decode the ABI Manager seems to stop working, I dont know why

Regards

Graham


David J Taylor GM8ARV 🏴󠁧󠁢󠁳󠁣󠁴󠁿 🇪🇺
 

On 20/03/2021 19:06, Graham Woolf wrote:
Hi Maurizio
I have seen this a few times
What works for me is to empty the directory containing the raw data . It seems that if you have too  many files to decode the ABI Manager seems to stop working, I dont know why
Regards
Graham
Folks,

I run multiple copies of the GOES ABI Manager continually 24x7 with both local disk and network access without seeing any problems such as you describe.

However, there are many unwanted files in the \received\E1H-TPG-1\ directory, and here I run a job every thirty minutes which removes those files. It includes commands like:

~~~~~~~~~~~~~~~~~~~~~~
SET received=L:\Tools\EUMETCast\received
TrimTree 30m %received%\E1H-TPG-1 OR_SEIS-L1b-????_G16*
TrimTree 30m %received%\E1H-TPG-1 OR_GLM-L2-LCFA_G16*
TrimTree 30m %received%\E1H-TPG-1 OR_ABI-L2-*
TrimTree 30m %received%\E1H-TPG-1 ABI-L2-?SRF-M6_v?r?_g16*
~~~~~~~~~~~~~~~~~~~~~~

It's been developed over time as more and more data which isn't required has been added, hence the multiple TrimTree commands. I spotted these issues from seeing the \received\ growing unexpectedly. I have more in that 30-minute job than E1H-TPG-1 trims.

I hope that's the issue you're seeing.

Cheers,
David
--
SatSignal Software - Quality software for you
Web: https://www.satsignal.eu
Email: david-taylor@blueyonder.co.uk
Twitter: @gm8arv


Thorsten Miglus
 

Hi Maurizio,

did your computer restart after a power cut?
I recommend to close ABI Manager.
Then delete the latest images within the Images folder.
Then start ABI manager again.
ABI manager loads the latest images at startup.
If it detects a corrupted image at startup it will not process the incoming files any more.

Cheers,
Thorsten


On Sat, Mar 20, 2021 at 07:53 PM, <maurizio.calvitti@...> wrote:

Good evening everyone,

for a couple of days GOES ABI manager is not able to decode raw files.

In the Log file I read  a long list like this (both for G17 and G16)

2021-03-20 18:44:18      OR_ABI-L1b-RadF-M6C07_G17_s20210791830321_e20210791839399_c20210791839433.nc failed

2021-03-20 18:44:18      OR_ABI-L1b-RadF-M6C08_G17_s20210791830321_e20210791839388_c20210791839429.nc failed

2021-03-20 18:44:18      OR_ABI-L1b-RadF-M6C09_G17_s20210791830321_e20210791839394_c20210791839436.nc failed

2021-03-20 18:44:18      OR_ABI-L1b-RadF-M6C10_G17_s20210791830321_e20210791839399_c20210791839440.nc failed

 

Someone may help me in understanding what is happening?

 

Thanks a lot

Maurizio

 


David J Taylor GM8ARV 🏴󠁧󠁢󠁳󠁣󠁴󠁿 🇪🇺
 

On 21/03/2021 10:28, Thorsten Miglus wrote:
Hi Maurizio,
did your computer restart after a power cut?
I recommend to close ABI Manager.
Then delete the latest images within the Images folder.
Then start ABI manager again.
ABI manager loads the latest images at startup.
If it detects a corrupted image at startup it will not process the incoming files any more.
Cheers,
Thorsten
Thorsten,

Thanks for that. I wasn't aware of that problem. Is there an error message when the bad image is loaded? In any case, I'll try to improve the robustness of that operation.

Cheers,
David
--
SatSignal Software - Quality software for you
Web: https://www.satsignal.eu
Email: david-taylor@blueyonder.co.uk
Twitter: @gm8arv


Thorsten Miglus
 

Hi David,

there is no error message when loading the bad image.

Cheers,
Thorsten


On Sun, Mar 21, 2021 at 11:54 AM, David J Taylor GM8ARV 🏴󠁧󠁢󠁳󠁣󠁴󠁿 🇪🇺 wrote:
Thorsten,

Thanks for that. I wasn't aware of that problem. Is there an error message when the bad image is loaded? In any case, I'll try to improve the robustness of that operation.

Cheers,
David


David J Taylor GM8ARV 🏴󠁧󠁢󠁳󠁣󠁴󠁿 🇪🇺
 

On 21/03/2021 11:12, Thorsten Miglus wrote:
Hi David,
there is no error message when loading the bad image.
Cheers,
Thorsten
OK, and does the program start if you then press the Start button?

David
--
SatSignal Software - Quality software for you
Web: https://www.satsignal.eu
Email: david-taylor@blueyonder.co.uk
Twitter: @gm8arv


geojohnt@...
 

David and All,

I was interested to read about 'unprocessed files clogging up the system' in the above postings.

>However, there are many unwanted files in the \received\E1H-TPG-1\
>directory,

I recall a time - when? - that I suffered all sorts of accumulating unwanted files when setting up a new system(?) and or going over to the 'new' TC software?
Probably when I started to receiver HVS-1 as well as Basic and my HD was filling up by the hour.

My solution (along with your advice) was to cull a lot data channels via TC cast_client_channels.ini and possibly being more specific in data routing not just pointing many of them to a default folder.
And of course the 'behind the scenes' auto deleting of unwanted data built into to your processing programme software.

Also not to request in ones personal EUMETSAT data account all the EUMETCast services that are disseminated.
Originally I just ticked everything. 

Of course all this is well known by experienced users and goes without saying.
But hopefully any new members here might find this of use.

Regarding TrimTree, you have quite often suggested I use this but 'you know me.'
I don't like fiddling around.
And actually I have never needed to use this to my knowledge.
My HD Data E: just sits there year after year, 24/7 operation, at 1.45 TB free space on a 1.81 TB HD.

However, I have just noticed that my received > bas > Info-Channel 2 folder contains 4.40 GB's of accumulated E-UNS files dating from 22/07/2019 to the present.

Now I know what you are going to say .......
But I have just manually deleted them (:-))

Regards,
John.

++++++++++++++++++++++++

Folks,

I run multiple copies of the GOES ABI Manager continually 24x7 with both
local disk and network access without seeing any problems such as you
describe.

However, there are many unwanted files in the \received\E1H-TPG-1\
directory, and here I run a job every thirty minutes which removes those
files.  It includes commands like:

~~~~~~~~~~~~~~~~~~~~~~
SET  received=L:\Tools\EUMETCast\received
TrimTree 30m  %received%\E1H-TPG-1  OR_SEIS-L1b-????_G16*
TrimTree 30m  %received%\E1H-TPG-1  OR_GLM-L2-LCFA_G16*
TrimTree 30m  %received%\E1H-TPG-1  OR_ABI-L2-*
TrimTree 30m  %received%\E1H-TPG-1  ABI-L2-?SRF-M6_v?r?_g16*
~~~~~~~~~~~~~~~~~~~~~~

It's been developed over time as more and more data which isn't required
has been added, hence the multiple TrimTree commands.  I spotted these
issues from seeing the \received\ growing unexpectedly.  I have more in
that 30-minute job than E1H-TPG-1 trims.

I hope that's the issue you're seeing.

Cheers,
David
--

+++++++++++++++++++++

-----Original Message-----
From: David J Taylor GM8ARV 🏴󠁧󠁢󠁳󠁣󠁴󠁿 🇪🇺 via groups.io <david-taylor@...>
To: MSG-1@groups.io
Sent: Sun, 21 Mar 2021 8:43
Subject: Re: [MSG-1] GOES ABI Manager

On 20/03/2021 19:06, Graham Woolf wrote:
> Hi Maurizio
>
> I have seen this a few times
>
> What works for me is to empty the directory containing the raw data . It
> seems that if you have too  many files to decode the ABI Manager seems
> to stop working, I dont know why
>
> Regards
>
> Graham





maurizio.calvitti@...
 

Thank you for your support.

Eumetcastview works perfectly and decode my GOES ABI Files (both GOES 16 and 17)

 

At this moment I have no idea how let GOES ABI manager work!

It continues to give me the message: failed for all files found in the directory.

 

2021-03-21 11:40:13      OR_ABI-L1b-RadF-M6C01_G16_s20210801120192_e20210801129500_c20210801129544-118900_0.nc failed

2021-03-21 11:40:13      OR_ABI-L1b-RadF-M6C02_G16_s20210801120192_e20210801129500_c20210801129534-120000_0.nc failed

2021-03-21 11:40:13      OR_ABI-L1b-RadF-M6C03_G16_s20210801120192_e20210801129500_c20210801129547-119000_0.nc failed

2021-03-21 11:40:13      OR_ABI-L1b-RadF-M6C04_G16_s20210801120192_e20210801129500_c20210801129528.nc failed

2021-03-21 11:40:13      OR_ABI-L1b-RadF-M6C05_G16_s20210801120192_e20210801129500_c20210801129542-119101_0.nc failed

2021-03-21 11:40:13      OR_ABI-L1b-RadF-M6C06_G16_s20210801120192_e20210801129506_c20210801129539.nc failed

2021-03-21 11:40:13      OR_ABI-L1b-RadF-M6C07_G16_s20210801120192_e20210801129511_c20210801129554.nc failed

2021-03-21 11:40:13      OR_ABI-L1b-RadF-M6C08_G16_s20210801120192_e20210801129500_c20210801129550.nc failed

2021-03-21 11:40:13      OR_ABI-L1b-RadF-M6C09_G16_s20210801120192_e20210801129506_c20210801129559.nc failed

2021-03-21 11:40:13      OR_ABI-L1b-RadF-M6C10_G16_s20210801120192_e20210801129511_c20210801129553.nc failed

2021-03-21 11:40:13      OR_ABI-L1b-RadF-M6C11_G16_s20210801120192_e20210801129500_c20210801129562.nc failed

2021-03-21 11:40:13      OR_ABI-L1b-RadF-M6C12_G16_s20210801120192_e20210801129505_c20210801129556.nc failed

2021-03-21 11:40:13      OR_ABI-L1b-RadF-M6C13_G16_s20210801120192_e20210801129511_c20210801129575.nc failed

2021-03-21 11:40:13      OR_ABI-L1b-RadF-M6C14_G16_s20210801120192_e20210801129500_c20210801129573.nc failed

2021-03-21 11:40:13      OR_ABI-L1b-RadF-M6C15_G16_s20210801120192_e20210801129505_c20210801129570.nc failed

2021-03-21 11:40:13      OR_ABI-L1b-RadF-M6C16_G16_s20210801120192_e20210801129511_c20210801129566.nc failed

2021-03-21 11:40:36.068  Closing GoesAbiManager-16.exe.exe V2019.4.20.39

 

 

 

Da: MSG-1@groups.io <MSG-1@groups.io> Per conto di Thorsten Miglus
Inviato: domenica 21 marzo 2021 12:13
A: MSG-1@groups.io
Oggetto: Re: [MSG-1] GOES ABI Manager

 

Hi David,

there is no error message when loading the bad image.

Cheers,
Thorsten


On Sun, Mar 21, 2021 at 11:54 AM, David J Taylor GM8ARV 🏴󠁧󠁢󠁳󠁣󠁴󠁿 🇪🇺 wrote:

Thorsten,

Thanks for that. I wasn't aware of that problem. Is there an error message when the bad image is loaded? In any case, I'll try to improve the robustness of that operation.

Cheers,
David


David J Taylor GM8ARV 🏴󠁧󠁢󠁳󠁣󠁴󠁿 🇪🇺
 

On 21/03/2021 12:53, maurizio.calvitti@enea.it wrote:
Thank you for your support.
Eumetcastview works perfectly and decode my GOES ABI Files (both GOES 16 and 17)
At this moment I have no idea how let GOES ABI manager work!
It continues to give me the message: failed for all files found in the directory.
Maurizio,

Sorry to hear you are still having problems. If the program used to work, and no longer does, you need to ask what changed on your PC.

- are any disks full?

- do the three directories on the setup page exist?

- are those directories all writeable - can the program delete the files it has processed?

My recommended settings are:

Save as NetCDF: unchecked.
Save as JPEG: checked.
Save as PNG: unchecked.

Dates in folders: checked.

Clear images: unchecked.
Short image names: checked.

Select bands: all mine are checked.

GeoSatSignal job number: all mine are 0.

Ciao,
David
--
SatSignal Software - Quality software for you
Web: https://www.satsignal.eu
Email: david-taylor@blueyonder.co.uk
Twitter: @gm8arv


David J Taylor GM8ARV 🏴󠁧󠁢󠁳󠁣󠁴󠁿 🇪🇺
 

On 21/03/2021 12:48, geojohnt via groups.io wrote:
David and All,
I was interested to read about 'unprocessed files clogging up the system' in the above postings.

However, there are many unwanted files in the \received\E1H-TPG-1\
directory,
I recall a time - when? - that I suffered all sorts of accumulating unwanted files when setting up a new system(?) and or going over to the 'new' TC software?
Probably when I started to receiver HVS-1 as well as Basic and my HD was filling up by the hour.
My solution (along with your advice) was to cull a lot data channels via TC cast_client_channels.ini and possibly being more specific in data routing not just pointing many of them to a default folder.
And of course the 'behind the scenes' auto deleting of unwanted data built into to your processing programme software.
Also not to request in ones personal EUMETSAT data account all the EUMETCast services that are disseminated.
Originally I just ticked everything.
Of course all this is well known by experienced users and goes without saying.
But hopefully any new members here might find this of use.
Regarding TrimTree, you have quite often suggested I use this but 'you know me.'
I don't like fiddling around.
And actually I have never needed to use this to my knowledge.
My HD Data E: just sits there year after year, 24/7 operation, at 1.45 TB free space on a 1.81 TB HD.
However, I have just noticed that my received > bas > Info-Channel 2 folder contains 4.40 GB's of accumulated E-UNS files dating from 22/07/2019 to the present.
Now I know what you are going to say .......
But I have just manually deleted them (:-))
Regards,
John.
John,

The E-UNS files are handled (if enabled) by the MSG Data Manager. Here's how I have mine sent:

[channel]
name=Info-Channel-2
target_directory=Z:\EUMETCast\received\bas\Info-Channel-2

In Setup, Channel Selection, MSG-HRIT, you may want to set "Text message retention" and "Daily log retention" to a non-zero value to ensure the processed files are deleted.

The current beta version is: MSG Data Manager - V3.1.15.1317

Likely you could just comment out "Info-Channel-2" if you don't need the data:

# [channel]
# name=Info-Channel-2
# target_directory=Z:\EUMETCast\received\bas\Info-Channel-2

My programs are typically call "Manager" because whilst receiving the data is relatively straight-forward (ahem) controlling the terabyte of data which could be flowing into your system every day is not!

Cheers,
David
--
SatSignal Software - Quality software for you
Web: https://www.satsignal.eu
Email: david-taylor@blueyonder.co.uk
Twitter: @gm8arv


Thorsten Miglus
 

David,

The program is started with the -START option.
The images are loaded until the bad image is found.
Afterwards processing starts but a fail message appers for all incoming files.
You can simulate this by yourself.
Just replace an image with a file of zero length and the same name.
Then start the ABI Manager.
Now watch what happens...

Cheers
Thorsten


On Sun, Mar 21, 2021 at 12:47 PM, David J Taylor GM8ARV 🏴󠁧󠁢󠁳󠁣󠁴󠁿 🇪🇺 wrote:
OK, and does the program start if you then press the Start button?

David


David J Taylor GM8ARV 🏴󠁧󠁢󠁳󠁣󠁴󠁿 🇪🇺
 

On 21/03/2021 15:23, Thorsten Miglus wrote:
David,
The program is started with the -START option.
The images are loaded until the bad image is found.
Afterwards processing starts but a fail message appers for all incoming files.
You can simulate this by yourself.
Just replace an image with a file of zero length and the same name.
Then start the ABI Manager.
Now watch what happens...
Cheers
Thorsten
Thorsten,

Thanks for reporting that. V2019.4.20.39 fixed the zero-length file on restore images issue. I've just updated the program to fix the new issue for zero length downloaded data and uploaded an update:

https://www.satsignal.eu/software/GOES-ABI-Manager.html
V2021.3.21.40 is at the bottom of the page.

Cheers,
David
--
SatSignal Software - Quality software for you
Web: https://www.satsignal.eu
Email: david-taylor@blueyonder.co.uk
Twitter: @gm8arv


geojohnt@...
 

David,

Thanks for your comments.

My MSG Data Manager is set to:
Text message retention 3.
Daily log retention 2.

Ah, I see what seems to be happening.

My Info-Channel-2 data goes to \bas\Info-Channel-2 folder - as you show below.
But, my MDM Tellicast received files location is:
E:\EUMETCast\received\bas\default\

So MDM text/log retention has nothing to regularly delete. 
Meanwhile the contents of the Info-Channel-2 folder continues to fill 'untrimmed.'

Which means(?) if I direct Info-Channel-2 data to the EUMETCast default folder, MDM will pick it up and trim it depending on the retention period.

I see from my bas\default folder that 

E1B-TPG-1.
E1B-TPL-1.
Data Channel-2.
Data Channel-12.

all go to this folder.

And of course as you say, I could just comment out Info-Channel-2.

Regards,
John

++++++++++++++++++++++++++++++++++++++

John,

The E-UNS files are handled (if enabled) by the MSG Data Manager.
Here's how I have mine sent:

[channel]
name=Info-Channel-2
target_directory=Z:\EUMETCast\received\bas\Info-Channel-2

In Setup, Channel Selection, MSG-HRIT, you may want to set "Text message
retention" and "Daily log retention" to a non-zero value to ensure the
processed files are deleted.

The current beta version is: MSG Data Manager - V3.1.15.1317

Likely you could just comment out "Info-Channel-2" if you don't need the
data:

# [channel]
# name=Info-Channel-2
# target_directory=Z:\EUMETCast\received\bas\Info-Channel-2

My programs are typically call "Manager" because whilst receiving the
data is relatively straight-forward (ahem) controlling the terabyte of
data which could be flowing into your system every day is not!

Cheers,
David
--


-----Original Message-----
From: David J Taylor GM8ARV 🏴󠁧󠁢󠁳󠁣󠁴󠁿 🇪🇺 via groups.io <david-taylor@...>
To: MSG-1@groups.io
Sent: Sun, 21 Mar 2021 15:10
Subject: Re: [MSG-1] GOES ABI Manager

On 21/03/2021 12:48, geojohnt via groups.io wrote:
> David and All,
>
> I was interested to read about 'unprocessed files clogging up the
> system' in the above postings.
>
>>However, there are many unwanted files in the \received\E1H-TPG-1\
>>directory,
>
> I recall a time - when? - that I suffered all sorts of accumulating
> unwanted files when setting up a new system(?) and or going over to the
> 'new' TC software?
> Probably when I started to receiver HVS-1 as well as Basic and my HD was
> filling up by the hour.
>
> My solution (along with your advice) was to cull a lot data channels via
> TC cast_client_channels.ini and possibly being more specific in data
> routing not just pointing many of them to a default folder.
> And of course the 'behind the scenes' auto deleting of unwanted data
> built into to your processing programme software.
>
> Also not to request in ones personal EUMETSAT data account all the
> EUMETCast services that are disseminated.
> Originally I just ticked everything.
>
> Of course all this is well known by experienced users and goes without
> saying.
> But hopefully any new members here might find this of use.
>
> Regarding TrimTree, you have quite often suggested I use this but 'you
> know me.'
> I don't like fiddling around.
> And actually I have never needed to use this to my knowledge.
> My HD Data E: just sits there year after year, 24/7 operation, at 1.45
> TB free space on a 1.81 TB HD.
>
> However, I have just noticed that my received > bas > Info-Channel 2
> folder contains 4.40 GB's of accumulated E-UNS files dating from
> 22/07/2019 to the present.
>
> Now I know what you are going to say .......
> But I have just manually deleted them (:-))
>
> Regards,
> John.






maurizio.calvitti@...
 

Many thanks David,
I have checked the configuration and applied your suggested settings and now it works!!

Ciao
Maurizio

-----Messaggio originale-----
Da: MSG-1@groups.io <MSG-1@groups.io> Per conto di David J Taylor GM8ARV ?????????????? ???? via groups.io
Inviato: domenica 21 marzo 2021 15:50
A: MSG-1@groups.io
Oggetto: Re: [MSG-1] GOES ABI Manager

On 21/03/2021 12:53, maurizio.calvitti@enea.it wrote:
Thank you for your support.

Eumetcastview works perfectly and decode my GOES ABI Files (both GOES
16 and 17)

At this moment I have no idea how let GOES ABI manager work!

It continues to give me the message: failed for all files found in the
directory.
Maurizio,

Sorry to hear you are still having problems. If the program used to work, and no longer does, you need to ask what changed on your PC.

- are any disks full?

- do the three directories on the setup page exist?

- are those directories all writeable - can the program delete the files it has processed?

My recommended settings are:

Save as NetCDF: unchecked.
Save as JPEG: checked.
Save as PNG: unchecked.

Dates in folders: checked.

Clear images: unchecked.
Short image names: checked.

Select bands: all mine are checked.

GeoSatSignal job number: all mine are 0.

Ciao,
David
--
SatSignal Software - Quality software for you
Web: https://www.satsignal.eu
Email: david-taylor@blueyonder.co.uk
Twitter: @gm8arv