Dropping spots.


Rolf Ekstrand
 

Thank you Rob,

Merry X-mas to you and yours

Rolf


Rob Robinett
 

I have just increased the curl timeouts on my scrapers from 10 seconds to 120 seconds which seems to result in many more successful scrapes from the wsprnet.org server.  
Some successful API calls take 40-70 seconds, while occasionally curl returns error 56.
 
There remains a second problem of gaps in the downloaded spots, and I have some code changes to try which might address that issue.
Only now do I appreciate the size of the downloads. Some of them approach 1 MB.
 
In any case, I expect that vk7jj and others who use wsprdaemon.org will be once again getting most, if not all, of the spots in wsprnet.org 
 


G7ELK
 

Works for me as well - thank you Rob.

-----Original Message-----
From: wsprdaemon@groups.io <wsprdaemon@groups.io> On Behalf Of Stefan HB9TMC
Sent: 24 December 2020 00:41
To: wsprdaemon@groups.io
Subject: Re: [wsprdaemon] Dropping spots.

Thank you Rob, it seems to be uploading again after moving the cache.

73
Stefan


Stefan HB9TMC
 

Thank you Rob, it seems to be uploading again after moving the cache.

73
Stefan


Rob Robinett
 

Since this morning's WD Users Zoom we uncovered one bug in WD spot uploading which is the result of the wsprnet.org overload problems.
I think I can implement and check in a fix before tomorrow morning.
But until then, you should be able to resume uploads of new spots by:
1) stop WD: '~/wsprdaemon/wsprdaemon.sh -z'
2) save away your current cache of spots:  'mv ~/wsprdaemon/uploads.d/wsprnet.d/ ~/wsprdaemon/uploads.d/wsprnet.d.save/'
3) start WD:   '~/wsprdaemon/wsprdaemon.sh -a'

My updated version of WD will allow you to restore and upload those saved spots/


On Wed, Dec 23, 2020 at 12:40 AM <rob@...> wrote:
Ulli,

Mine has failed completely too.

I see 1545 lines in "upload_spots_file_list.txt" and "uploads.log" has grown to 2601 lines with "broken Pipe" and "line 3420: echo: write error: Broken pipe".

No doubt this is caused by the Wsprnet issues but I suspect I will need to delete some spot   files and restart Wsprdaemon to get things started again.  Rob, do you agree?

Regards,

       Rob



--
Rob Robinett
AI6VN
mobile: +1 650 218 8896


Stefan HB9TMC
 

Since yesterday morning, no more spots are uploaded from me as well. (My spots on 2200m/630m/160m are from WSJT-X)

But other WD users seem to be able to upload fine? For example OE9GHV, KD2OM, ...


73
Stefan


Rolf Ekstrand
 

I have no pending uploads here just now.  For what it is worth I observed a time gap on VK7JJ last night   from 18:12 to 18:20  ( that is 00:12 to 00:20 Z) but wspr.net  had no such gap. 

73

Rolf



 


G7ELK
 

Ulli,

Mine has failed completely too.

I see 1545 lines in "upload_spots_file_list.txt" and "uploads.log" has grown to 2601 lines with "broken Pipe" and "line 3420: echo: write error: Broken pipe".

No doubt this is caused by the Wsprnet issues but I suspect I will need to delete some spot   files and restart Wsprdaemon to get things started again.  Rob, do you agree?

Regards,

       Rob


Rob Robinett
 

I have been told these lost spots are due to overloads at wsprnet.org and that is under investigation.

On Tue, Dec 22, 2020 at 10:57 PM ON5KQ <ON5KQ@...> wrote:
No spots at all from my kiwis...for the last day or so...
However all received at wsprdaemon servers, I think...

So I leave everything running here and just wait...

Just strange idea:  Could it be one of the users computer, being hacked and then this infected computer is overwhelming the wsprnet.org site ot make it overload by constantly polling ? The user of the infected computer has (of coarse) no idea about the problem...
In that case the only way is to block the ip-adres of the infected users computer...
However: I just checked my PC.... so don't block me any longer at wsprnet.org..... hihi...

Ulli, ON5KQ



--
Rob Robinett
AI6VN
mobile: +1 650 218 8896


ON5KQ
 

No spots at all from my kiwis...for the last day or so...
However all received at wsprdaemon servers, I think...

So I leave everything running here and just wait...

Just strange idea:  Could it be one of the users computer, being hacked and then this infected computer is overwhelming the wsprnet.org site ot make it overload by constantly polling ? The user of the infected computer has (of coarse) no idea about the problem...
In that case the only way is to block the ip-adres of the infected users computer...
However: I just checked my PC.... so don't block me any longer at wsprnet.org..... hihi...

Ulli, ON5KQ


Rob Robinett
 

Thanks Rolf,

wsprnet.org is overloaded and we are trying to debug the interface between it and the wsprdaemon.org server.

Rob


On Tue, Dec 22, 2020 at 4:25 PM Rolf Ekstrand <rekstrand@...> wrote:
Now it looks OK for the last 2 hours.   At least no time gaps in reporting. 

Rolf



--
Rob Robinett
AI6VN
mobile: +1 650 218 8896


Rolf Ekstrand
 

Now it looks OK for the last 2 hours.   At least no time gaps in reporting. 

Rolf


Rolf Ekstrand
 

Have observed that VK7JJ is dropping spots. Just found 10 minutes  missing whereas they are OK on WSPRnet. 
I suspect that there are more to this, but have not had time to dive into it.   It appears that jimlill.com  is  also affected.

73 Rolf  K9DZT