What happens with old spots not yet uploaded (probably because of Internet failure) but stored in txt files...


ON5KQ
 

Dear,
During last night I probably had an Internet connection failuer, so from 20h utc yesterday until 4h utc in the morning, there was nothing uploaded.
The wsprdaemon script created a lot of txt files instead (containing the decoded spots during that time). In the meantime the wsprdaemon.conf file switched from nighttime to daylight schedule and a little later during the new daylight schedule the Internet came back.

Because of the new daylight schedule running, the wsprdaemon script didn't realize , that from the nighttime schedule (not active anymore) there were still a lot of ,txt files containing the spots to be uploaded. 
If I remember correctly all the .txt files with the nighttime spots were in this directory
/home/wsprdaemon/uploads.d/wsprnet.d/spots.d/ ....   and channel by channel the txt files were written for every 2min of transmission.

I was wondering, why the waiting spots were not uploaded immediately as the Internet came back in the morning...

To find out,  I restarted with "./wsprdaemon.sh -vvva"  to force high verbosity to check the various log files.
I found that the script searched for txt files to be uploaded, but not in the nighttime folder, but the current job folder for the running daytime schedule - so it would not immediately find the hundreds of waiting txt files only after roughly another 12h, when the night schedule becomes active again.

However, I didn't want to wait that long so...
(I am not sure, if it was correct, what I did then....):
- to upload the waiting txt files immediately, I stopped wsprdaemon with
./wsprdaemon.sh -z
then changed wsprdaemon.conf in that way, that the nighttime schedule was resumed instead of starting the daytime schedule.
Started again with:
./wsprdaemon.sh -vvva

Now In the logs I could see, that immediately more than 300 txt files waiting for upload were found from the script
Quickly the logs also showed that after a while many many uploads were done - it must be thousands of spots...

After a while, all txt files were erased and I found in the curl-logs the confirmation of upload acceptance... again thousands. I was wondering however that all that was only a matter of minutes - the missing upload was very quick...

So, I was happy and stopped wsprdaemon, as I did not want to write too much data in the log-files with that high verbosity level.
So, I restarted with
./wsprdaemon.sh -a

Now what::
Hours later now, still there are no spots at all between 22h yesterday and 6h this morning - neither on wsprnet.org, nor on vk7jj's website...

It would be great to understand what really happened with the uploaded spots from partly today and partly from yesterday (based on utc time)

Will the missing spots ever appear ?
Why I got a confirmation of acceptance for thousands of spots in the curl-file, when these spots never appear ?
What did I wrong, that still no missing spot is delivered and how should I proceed next time to prevent data loss.

Or do just need to wait longer for the data to appear (may be one day or so ?!)
I am confused...hi

May be Rob can explain in the next zoom-meeting, how the upload of old spots really works - either because of Internet connection problems or even after portable kiwi operation in the field with later upload from home (Glenn's application ?!)

regards,
Ulli, ON5KQ


Rob Robinett
 

Yes, I think the WD upload service only looks for spot files in receivers which are currently scheduled to be running. That does seem to be a potential source of late uploaded spots, although spots are never flushed until they have been successfully uploaded.   I have added working on that to the v3.0 TODO list.


I don’t know how wsprnet handles such spots, although Glenn has reported success with uploads delayed by hours or a day.  wsprnet assigns a unique 64 bit number to each spot it receives, and in cloning the wsprnet DB I validate that there are no gaps in the spot number sequence.

On Mon, Jun 14, 2021 at 8:17 AM ON5KQ <ON5KQ@...> wrote:
Dear,
During last night I probably had an Internet connection failuer, so from 20h utc yesterday until 4h utc in the morning, there was nothing uploaded.
The wsprdaemon script created a lot of txt files instead (containing the decoded spots during that time). In the meantime the wsprdaemon.conf file switched from nighttime to daylight schedule and a little later during the new daylight schedule the Internet came back.

Because of the new daylight schedule running, the wsprdaemon script didn't realize , that from the nighttime schedule (not active anymore) there were still a lot of ,txt files containing the spots to be uploaded. 
If I remember correctly all the .txt files with the nighttime spots were in this directory
/home/wsprdaemon/uploads.d/wsprnet.d/spots.d/ ....   and channel by channel the txt files were written for every 2min of transmission.

I was wondering, why the waiting spots were not uploaded immediately as the Internet came back in the morning...

To find out,  I restarted with "./wsprdaemon.sh -vvva"  to force high verbosity to check the various log files.
I found that the script searched for txt files to be uploaded, but not in the nighttime folder, but the current job folder for the running daytime schedule - so it would not immediately find the hundreds of waiting txt files only after roughly another 12h, when the night schedule becomes active again.

However, I didn't want to wait that long so...
(I am not sure, if it was correct, what I did then....):
- to upload the waiting txt files immediately, I stopped wsprdaemon with
./wsprdaemon.sh -z
then changed wsprdaemon.conf in that way, that the nighttime schedule was resumed instead of starting the daytime schedule.
Started again with:
./wsprdaemon.sh -vvva

Now In the logs I could see, that immediately more than 300 txt files waiting for upload were found from the script
Quickly the logs also showed that after a while many many uploads were done - it must be thousands of spots...

After a while, all txt files were erased and I found in the curl-logs the confirmation of upload acceptance... again thousands. I was wondering however that all that was only a matter of minutes - the missing upload was very quick...

So, I was happy and stopped wsprdaemon, as I did not want to write too much data in the log-files with that high verbosity level.
So, I restarted with
./wsprdaemon.sh -a

Now what::
Hours later now, still there are no spots at all between 22h yesterday and 6h this morning - neither on wsprnet.org, nor on vk7jj's website...

It would be great to understand what really happened with the uploaded spots from partly today and partly from yesterday (based on utc time)

Will the missing spots ever appear ?
Why I got a confirmation of acceptance for thousands of spots in the curl-file, when these spots never appear ?
What did I wrong, that still no missing spot is delivered and how should I proceed next time to prevent data loss.

Or do just need to wait longer for the data to appear (may be one day or so ?!)
I am confused...hi

May be Rob can explain in the next zoom-meeting, how the upload of old spots really works - either because of Internet connection problems or even after portable kiwi operation in the field with later upload from home (Glenn's application ?!)

regards,
Ulli, ON5KQ


WA2TP
 

I too had seen this in the past where wsprdaemon buffered over 3000 spots while my internet was down for several hours, and when the internet connection restored, after ~15 minutes, WSPRNET accepted only 60 of those 3000 buffered spots for some reason. 

Nothing remained in any of the wsprdaemon folders. No idea where the other 2900+ spots went to.

I think I had sent this info to Rob back when this occurred.

Tom
WA2TP 


On Jun 14, 2021, at 8:00 PM, Rob Robinett <rob@...> wrote:


Yes, I think the WD upload service only looks for spot files in receivers which are currently scheduled to be running. That does seem to be a potential source of late uploaded spots, although spots are never flushed until they have been successfully uploaded.   I have added working on that to the v3.0 TODO list.


I don’t know how wsprnet handles such spots, although Glenn has reported success with uploads delayed by hours or a day.  wsprnet assigns a unique 64 bit number to each spot it receives, and in cloning the wsprnet DB I validate that there are no gaps in the spot number sequence.

On Mon, Jun 14, 2021 at 8:17 AM ON5KQ <ON5KQ@...> wrote:
Dear,
During last night I probably had an Internet connection failuer, so from 20h utc yesterday until 4h utc in the morning, there was nothing uploaded.
The wsprdaemon script created a lot of txt files instead (containing the decoded spots during that time). In the meantime the wsprdaemon.conf file switched from nighttime to daylight schedule and a little later during the new daylight schedule the Internet came back.

Because of the new daylight schedule running, the wsprdaemon script didn't realize , that from the nighttime schedule (not active anymore) there were still a lot of ,txt files containing the spots to be uploaded. 
If I remember correctly all the .txt files with the nighttime spots were in this directory
/home/wsprdaemon/uploads.d/wsprnet.d/spots.d/ ....   and channel by channel the txt files were written for every 2min of transmission.

I was wondering, why the waiting spots were not uploaded immediately as the Internet came back in the morning...

To find out,  I restarted with "./wsprdaemon.sh -vvva"  to force high verbosity to check the various log files.
I found that the script searched for txt files to be uploaded, but not in the nighttime folder, but the current job folder for the running daytime schedule - so it would not immediately find the hundreds of waiting txt files only after roughly another 12h, when the night schedule becomes active again.

However, I didn't want to wait that long so...
(I am not sure, if it was correct, what I did then....):
- to upload the waiting txt files immediately, I stopped wsprdaemon with
./wsprdaemon.sh -z
then changed wsprdaemon.conf in that way, that the nighttime schedule was resumed instead of starting the daytime schedule.
Started again with:
./wsprdaemon.sh -vvva

Now In the logs I could see, that immediately more than 300 txt files waiting for upload were found from the script
Quickly the logs also showed that after a while many many uploads were done - it must be thousands of spots...

After a while, all txt files were erased and I found in the curl-logs the confirmation of upload acceptance... again thousands. I was wondering however that all that was only a matter of minutes - the missing upload was very quick...

So, I was happy and stopped wsprdaemon, as I did not want to write too much data in the log-files with that high verbosity level.
So, I restarted with
./wsprdaemon.sh -a

Now what::
Hours later now, still there are no spots at all between 22h yesterday and 6h this morning - neither on wsprnet.org, nor on vk7jj's website...

It would be great to understand what really happened with the uploaded spots from partly today and partly from yesterday (based on utc time)

Will the missing spots ever appear ?
Why I got a confirmation of acceptance for thousands of spots in the curl-file, when these spots never appear ?
What did I wrong, that still no missing spot is delivered and how should I proceed next time to prevent data loss.

Or do just need to wait longer for the data to appear (may be one day or so ?!)
I am confused...hi

May be Rob can explain in the next zoom-meeting, how the upload of old spots really works - either because of Internet connection problems or even after portable kiwi operation in the field with later upload from home (Glenn's application ?!)

regards,
Ulli, ON5KQ


Rob Robinett
 

The wsprnet upload API is not well documented and so I had to experiment with its usage and failure modes.
I found that some batch uploads are accepted by wsprnet, but in its status response wsprnet would say that it accept only some of the spots in the batch (e.g. 190 of 200 spots).
It does not identify which spots were not accepted and if I attempt to reload the whole batch it will reject them all.
Those problems are among the reasons we implemented the wsprdaemon.org extended spots database where we have complete control of the server.
Perhaps someone with more patience and/or influence with wsprnet can resolve this problem, but I think it more productive to work on V3.0

On Mon, Jun 14, 2021 at 5:41 PM WA2TP <myis300@...> wrote:
I too had seen this in the past where wsprdaemon buffered over 3000 spots while my internet was down for several hours, and when the internet connection restored, after ~15 minutes, WSPRNET accepted only 60 of those 3000 buffered spots for some reason. 

Nothing remained in any of the wsprdaemon folders. No idea where the other 2900+ spots went to.

I think I had sent this info to Rob back when this occurred.

Tom
WA2TP 


On Jun 14, 2021, at 8:00 PM, Rob Robinett <rob@...> wrote:


Yes, I think the WD upload service only looks for spot files in receivers which are currently scheduled to be running. That does seem to be a potential source of late uploaded spots, although spots are never flushed until they have been successfully uploaded.   I have added working on that to the v3.0 TODO list.


I don’t know how wsprnet handles such spots, although Glenn has reported success with uploads delayed by hours or a day.  wsprnet assigns a unique 64 bit number to each spot it receives, and in cloning the wsprnet DB I validate that there are no gaps in the spot number sequence.

On Mon, Jun 14, 2021 at 8:17 AM ON5KQ <ON5KQ@...> wrote:
Dear,
During last night I probably had an Internet connection failuer, so from 20h utc yesterday until 4h utc in the morning, there was nothing uploaded.
The wsprdaemon script created a lot of txt files instead (containing the decoded spots during that time). In the meantime the wsprdaemon.conf file switched from nighttime to daylight schedule and a little later during the new daylight schedule the Internet came back.

Because of the new daylight schedule running, the wsprdaemon script didn't realize , that from the nighttime schedule (not active anymore) there were still a lot of ,txt files containing the spots to be uploaded. 
If I remember correctly all the .txt files with the nighttime spots were in this directory
/home/wsprdaemon/uploads.d/wsprnet.d/spots.d/ ....   and channel by channel the txt files were written for every 2min of transmission.

I was wondering, why the waiting spots were not uploaded immediately as the Internet came back in the morning...

To find out,  I restarted with "./wsprdaemon.sh -vvva"  to force high verbosity to check the various log files.
I found that the script searched for txt files to be uploaded, but not in the nighttime folder, but the current job folder for the running daytime schedule - so it would not immediately find the hundreds of waiting txt files only after roughly another 12h, when the night schedule becomes active again.

However, I didn't want to wait that long so...
(I am not sure, if it was correct, what I did then....):
- to upload the waiting txt files immediately, I stopped wsprdaemon with
./wsprdaemon.sh -z
then changed wsprdaemon.conf in that way, that the nighttime schedule was resumed instead of starting the daytime schedule.
Started again with:
./wsprdaemon.sh -vvva

Now In the logs I could see, that immediately more than 300 txt files waiting for upload were found from the script
Quickly the logs also showed that after a while many many uploads were done - it must be thousands of spots...

After a while, all txt files were erased and I found in the curl-logs the confirmation of upload acceptance... again thousands. I was wondering however that all that was only a matter of minutes - the missing upload was very quick...

So, I was happy and stopped wsprdaemon, as I did not want to write too much data in the log-files with that high verbosity level.
So, I restarted with
./wsprdaemon.sh -a

Now what::
Hours later now, still there are no spots at all between 22h yesterday and 6h this morning - neither on wsprnet.org, nor on vk7jj's website...

It would be great to understand what really happened with the uploaded spots from partly today and partly from yesterday (based on utc time)

Will the missing spots ever appear ?
Why I got a confirmation of acceptance for thousands of spots in the curl-file, when these spots never appear ?
What did I wrong, that still no missing spot is delivered and how should I proceed next time to prevent data loss.

Or do just need to wait longer for the data to appear (may be one day or so ?!)
I am confused...hi

May be Rob can explain in the next zoom-meeting, how the upload of old spots really works - either because of Internet connection problems or even after portable kiwi operation in the field with later upload from home (Glenn's application ?!)

regards,
Ulli, ON5KQ



--
Rob Robinett
AI6VN
mobile: +1 650 218 8896