Dear,
During last night I probably had an Internet connection failuer, so from 20h utc yesterday until 4h utc in the morning, there was nothing uploaded.
The wsprdaemon script created a lot of txt files instead (containing the decoded spots during that time). In the meantime the wsprdaemon.conf file switched from nighttime to daylight schedule and a little later during the new daylight schedule the Internet came back.

Because of the new daylight schedule running, the wsprdaemon script didn't realize , that from the nighttime schedule (not active anymore) there were still a lot of ,txt files containing the spots to be uploaded. 
If I remember correctly all the .txt files with the nighttime spots were in this directory
/home/wsprdaemon/uploads.d/wsprnet.d/spots.d/ ....   and channel by channel the txt files were written for every 2min of transmission.

I was wondering, why the waiting spots were not uploaded immediately as the Internet came back in the morning...

To find out,  I restarted with "./wsprdaemon.sh -vvva"  to force high verbosity to check the various log files.
I found that the script searched for txt files to be uploaded, but not in the nighttime folder, but the current job folder for the running daytime schedule - so it would not immediately find the hundreds of waiting txt files only after roughly another 12h, when the night schedule becomes active again.

However, I didn't want to wait that long so...
(I am not sure, if it was correct, what I did then....):
- to upload the waiting txt files immediately, I stopped wsprdaemon with
./wsprdaemon.sh -z
then changed wsprdaemon.conf in that way, that the nighttime schedule was resumed instead of starting the daytime schedule.
Started again with:
./wsprdaemon.sh -vvva

Now In the logs I could see, that immediately more than 300 txt files waiting for upload were found from the script
Quickly the logs also showed that after a while many many uploads were done - it must be thousands of spots...

After a while, all txt files were erased and I found in the curl-logs the confirmation of upload acceptance... again thousands. I was wondering however that all that was only a matter of minutes - the missing upload was very quick...

So, I was happy and stopped wsprdaemon, as I did not want to write too much data in the log-files with that high verbosity level.
So, I restarted with
./wsprdaemon.sh -a

Now what::
Hours later now, still there are no spots at all between 22h yesterday and 6h this morning - neither on wsprnet.org, nor on vk7jj's website...

It would be great to understand what really happened with the uploaded spots from partly today and partly from yesterday (based on utc time)

Will the missing spots ever appear ?
Why I got a confirmation of acceptance for thousands of spots in the curl-file, when these spots never appear ?
What did I wrong, that still no missing spot is delivered and how should I proceed next time to prevent data loss.

Or do just need to wait longer for the data to appear (may be one day or so ?!)
I am confused...hi

May be Rob can explain in the next zoom-meeting, how the upload of old spots really works - either because of Internet connection problems or even after portable kiwi operation in the field with later upload from home (Glenn's application ?!)

regards,
Ulli, ON5KQ

I too had seen this in the past where wsprdaemon buffered over 3000 spots while my internet was down for several hours, and when the internet connection restored, after ~15 minutes, WSPRNET accepted only 60 of those 3000 buffered spots for some reason. 

Nothing remained in any of the wsprdaemon folders. No idea where the other 2900+ spots went to.

I think I had sent this info to Rob back when this occurred.

Tom

WA2TP 

On Jun 14, 2021, at 8:00 PM, Rob Robinett <rob@...> wrote:



Yes, I think the WD upload service only looks for spot files in receivers which are currently scheduled to be running. That does seem to be a potential source of late uploaded spots, although spots are never flushed until they have been successfully uploaded.   I have added working on that to the v3.0 TODO list.

I don’t know how wsprnet handles such spots, although Glenn has reported success with uploads delayed by hours or a day.  wsprnet assigns a unique 64 bit number to each spot it receives, and in cloning the wsprnet DB I validate that there are no gaps in the spot number sequence.

On Mon, Jun 14, 2021 at 8:17 AM ON5KQ <ON5KQ@...> wrote:

Dear,
During last night I probably had an Internet connection failuer, so from 20h utc yesterday until 4h utc in the morning, there was nothing uploaded.
The wsprdaemon script created a lot of txt files instead (containing the decoded spots during that time). In the meantime the wsprdaemon.conf file switched from nighttime to daylight schedule and a little later during the new daylight schedule the Internet came back.

Because of the new daylight schedule running, the wsprdaemon script didn't realize , that from the nighttime schedule (not active anymore) there were still a lot of ,txt files containing the spots to be uploaded. 
If I remember correctly all the .txt files with the nighttime spots were in this directory
/home/wsprdaemon/uploads.d/wsprnet.d/spots.d/ ....   and channel by channel the txt files were written for every 2min of transmission.

I was wondering, why the waiting spots were not uploaded immediately as the Internet came back in the morning...

To find out,  I restarted with "./wsprdaemon.sh -vvva"  to force high verbosity to check the various log files.
I found that the script searched for txt files to be uploaded, but not in the nighttime folder, but the current job folder for the running daytime schedule - so it would not immediately find the hundreds of waiting txt files only after roughly another 12h, when the night schedule becomes active again.

However, I didn't want to wait that long so...
(I am not sure, if it was correct, what I did then....):
- to upload the waiting txt files immediately, I stopped wsprdaemon with
./wsprdaemon.sh -z
then changed wsprdaemon.conf in that way, that the nighttime schedule was resumed instead of starting the daytime schedule.
Started again with:
./wsprdaemon.sh -vvva

Now In the logs I could see, that immediately more than 300 txt files waiting for upload were found from the script
Quickly the logs also showed that after a while many many uploads were done - it must be thousands of spots...

After a while, all txt files were erased and I found in the curl-logs the confirmation of upload acceptance... again thousands. I was wondering however that all that was only a matter of minutes - the missing upload was very quick...

So, I was happy and stopped wsprdaemon, as I did not want to write too much data in the log-files with that high verbosity level.
So, I restarted with
./wsprdaemon.sh -a

Now what::
Hours later now, still there are no spots at all between 22h yesterday and 6h this morning - neither on wsprnet.org, nor on vk7jj's website...

It would be great to understand what really happened with the uploaded spots from partly today and partly from yesterday (based on utc time)

Will the missing spots ever appear ?
Why I got a confirmation of acceptance for thousands of spots in the curl-file, when these spots never appear ?
What did I wrong, that still no missing spot is delivered and how should I proceed next time to prevent data loss.

Or do just need to wait longer for the data to appear (may be one day or so ?!)
I am confused...hi

May be Rob can explain in the next zoom-meeting, how the upload of old spots really works - either because of Internet connection problems or even after portable kiwi operation in the field with later upload from home (Glenn's application ?!)

regards,
Ulli, ON5KQ