Date   

Re: What happens with old spots not yet uploaded (probably because of Internet failure) but stored in txt files...

WA2TP - Tom
 

I too had seen this in the past where wsprdaemon buffered over 3000 spots while my internet was down for several hours, and when the internet connection restored, after ~15 minutes, WSPRNET accepted only 60 of those 3000 buffered spots for some reason. 

Nothing remained in any of the wsprdaemon folders. No idea where the other 2900+ spots went to.

I think I had sent this info to Rob back when this occurred.

Tom
WA2TP 


On Jun 14, 2021, at 8:00 PM, Rob Robinett <rob@...> wrote:


Yes, I think the WD upload service only looks for spot files in receivers which are currently scheduled to be running. That does seem to be a potential source of late uploaded spots, although spots are never flushed until they have been successfully uploaded.   I have added working on that to the v3.0 TODO list.


I don’t know how wsprnet handles such spots, although Glenn has reported success with uploads delayed by hours or a day.  wsprnet assigns a unique 64 bit number to each spot it receives, and in cloning the wsprnet DB I validate that there are no gaps in the spot number sequence.

On Mon, Jun 14, 2021 at 8:17 AM ON5KQ <ON5KQ@...> wrote:
Dear,
During last night I probably had an Internet connection failuer, so from 20h utc yesterday until 4h utc in the morning, there was nothing uploaded.
The wsprdaemon script created a lot of txt files instead (containing the decoded spots during that time). In the meantime the wsprdaemon.conf file switched from nighttime to daylight schedule and a little later during the new daylight schedule the Internet came back.

Because of the new daylight schedule running, the wsprdaemon script didn't realize , that from the nighttime schedule (not active anymore) there were still a lot of ,txt files containing the spots to be uploaded. 
If I remember correctly all the .txt files with the nighttime spots were in this directory
/home/wsprdaemon/uploads.d/wsprnet.d/spots.d/ ....   and channel by channel the txt files were written for every 2min of transmission.

I was wondering, why the waiting spots were not uploaded immediately as the Internet came back in the morning...

To find out,  I restarted with "./wsprdaemon.sh -vvva"  to force high verbosity to check the various log files.
I found that the script searched for txt files to be uploaded, but not in the nighttime folder, but the current job folder for the running daytime schedule - so it would not immediately find the hundreds of waiting txt files only after roughly another 12h, when the night schedule becomes active again.

However, I didn't want to wait that long so...
(I am not sure, if it was correct, what I did then....):
- to upload the waiting txt files immediately, I stopped wsprdaemon with
./wsprdaemon.sh -z
then changed wsprdaemon.conf in that way, that the nighttime schedule was resumed instead of starting the daytime schedule.
Started again with:
./wsprdaemon.sh -vvva

Now In the logs I could see, that immediately more than 300 txt files waiting for upload were found from the script
Quickly the logs also showed that after a while many many uploads were done - it must be thousands of spots...

After a while, all txt files were erased and I found in the curl-logs the confirmation of upload acceptance... again thousands. I was wondering however that all that was only a matter of minutes - the missing upload was very quick...

So, I was happy and stopped wsprdaemon, as I did not want to write too much data in the log-files with that high verbosity level.
So, I restarted with
./wsprdaemon.sh -a

Now what::
Hours later now, still there are no spots at all between 22h yesterday and 6h this morning - neither on wsprnet.org, nor on vk7jj's website...

It would be great to understand what really happened with the uploaded spots from partly today and partly from yesterday (based on utc time)

Will the missing spots ever appear ?
Why I got a confirmation of acceptance for thousands of spots in the curl-file, when these spots never appear ?
What did I wrong, that still no missing spot is delivered and how should I proceed next time to prevent data loss.

Or do just need to wait longer for the data to appear (may be one day or so ?!)
I am confused...hi

May be Rob can explain in the next zoom-meeting, how the upload of old spots really works - either because of Internet connection problems or even after portable kiwi operation in the field with later upload from home (Glenn's application ?!)

regards,
Ulli, ON5KQ


Re: What happens with old spots not yet uploaded (probably because of Internet failure) but stored in txt files...

Rob Robinett
 

Yes, I think the WD upload service only looks for spot files in receivers which are currently scheduled to be running. That does seem to be a potential source of late uploaded spots, although spots are never flushed until they have been successfully uploaded.   I have added working on that to the v3.0 TODO list.


I don’t know how wsprnet handles such spots, although Glenn has reported success with uploads delayed by hours or a day.  wsprnet assigns a unique 64 bit number to each spot it receives, and in cloning the wsprnet DB I validate that there are no gaps in the spot number sequence.

On Mon, Jun 14, 2021 at 8:17 AM ON5KQ <ON5KQ@...> wrote:
Dear,
During last night I probably had an Internet connection failuer, so from 20h utc yesterday until 4h utc in the morning, there was nothing uploaded.
The wsprdaemon script created a lot of txt files instead (containing the decoded spots during that time). In the meantime the wsprdaemon.conf file switched from nighttime to daylight schedule and a little later during the new daylight schedule the Internet came back.

Because of the new daylight schedule running, the wsprdaemon script didn't realize , that from the nighttime schedule (not active anymore) there were still a lot of ,txt files containing the spots to be uploaded. 
If I remember correctly all the .txt files with the nighttime spots were in this directory
/home/wsprdaemon/uploads.d/wsprnet.d/spots.d/ ....   and channel by channel the txt files were written for every 2min of transmission.

I was wondering, why the waiting spots were not uploaded immediately as the Internet came back in the morning...

To find out,  I restarted with "./wsprdaemon.sh -vvva"  to force high verbosity to check the various log files.
I found that the script searched for txt files to be uploaded, but not in the nighttime folder, but the current job folder for the running daytime schedule - so it would not immediately find the hundreds of waiting txt files only after roughly another 12h, when the night schedule becomes active again.

However, I didn't want to wait that long so...
(I am not sure, if it was correct, what I did then....):
- to upload the waiting txt files immediately, I stopped wsprdaemon with
./wsprdaemon.sh -z
then changed wsprdaemon.conf in that way, that the nighttime schedule was resumed instead of starting the daytime schedule.
Started again with:
./wsprdaemon.sh -vvva

Now In the logs I could see, that immediately more than 300 txt files waiting for upload were found from the script
Quickly the logs also showed that after a while many many uploads were done - it must be thousands of spots...

After a while, all txt files were erased and I found in the curl-logs the confirmation of upload acceptance... again thousands. I was wondering however that all that was only a matter of minutes - the missing upload was very quick...

So, I was happy and stopped wsprdaemon, as I did not want to write too much data in the log-files with that high verbosity level.
So, I restarted with
./wsprdaemon.sh -a

Now what::
Hours later now, still there are no spots at all between 22h yesterday and 6h this morning - neither on wsprnet.org, nor on vk7jj's website...

It would be great to understand what really happened with the uploaded spots from partly today and partly from yesterday (based on utc time)

Will the missing spots ever appear ?
Why I got a confirmation of acceptance for thousands of spots in the curl-file, when these spots never appear ?
What did I wrong, that still no missing spot is delivered and how should I proceed next time to prevent data loss.

Or do just need to wait longer for the data to appear (may be one day or so ?!)
I am confused...hi

May be Rob can explain in the next zoom-meeting, how the upload of old spots really works - either because of Internet connection problems or even after portable kiwi operation in the field with later upload from home (Glenn's application ?!)

regards,
Ulli, ON5KQ


What happens with old spots not yet uploaded (probably because of Internet failure) but stored in txt files...

ON5KQ
 

Dear,
During last night I probably had an Internet connection failuer, so from 20h utc yesterday until 4h utc in the morning, there was nothing uploaded.
The wsprdaemon script created a lot of txt files instead (containing the decoded spots during that time). In the meantime the wsprdaemon.conf file switched from nighttime to daylight schedule and a little later during the new daylight schedule the Internet came back.

Because of the new daylight schedule running, the wsprdaemon script didn't realize , that from the nighttime schedule (not active anymore) there were still a lot of ,txt files containing the spots to be uploaded. 
If I remember correctly all the .txt files with the nighttime spots were in this directory
/home/wsprdaemon/uploads.d/wsprnet.d/spots.d/ ....   and channel by channel the txt files were written for every 2min of transmission.

I was wondering, why the waiting spots were not uploaded immediately as the Internet came back in the morning...

To find out,  I restarted with "./wsprdaemon.sh -vvva"  to force high verbosity to check the various log files.
I found that the script searched for txt files to be uploaded, but not in the nighttime folder, but the current job folder for the running daytime schedule - so it would not immediately find the hundreds of waiting txt files only after roughly another 12h, when the night schedule becomes active again.

However, I didn't want to wait that long so...
(I am not sure, if it was correct, what I did then....):
- to upload the waiting txt files immediately, I stopped wsprdaemon with
./wsprdaemon.sh -z
then changed wsprdaemon.conf in that way, that the nighttime schedule was resumed instead of starting the daytime schedule.
Started again with:
./wsprdaemon.sh -vvva

Now In the logs I could see, that immediately more than 300 txt files waiting for upload were found from the script
Quickly the logs also showed that after a while many many uploads were done - it must be thousands of spots...

After a while, all txt files were erased and I found in the curl-logs the confirmation of upload acceptance... again thousands. I was wondering however that all that was only a matter of minutes - the missing upload was very quick...

So, I was happy and stopped wsprdaemon, as I did not want to write too much data in the log-files with that high verbosity level.
So, I restarted with
./wsprdaemon.sh -a

Now what::
Hours later now, still there are no spots at all between 22h yesterday and 6h this morning - neither on wsprnet.org, nor on vk7jj's website...

It would be great to understand what really happened with the uploaded spots from partly today and partly from yesterday (based on utc time)

Will the missing spots ever appear ?
Why I got a confirmation of acceptance for thousands of spots in the curl-file, when these spots never appear ?
What did I wrong, that still no missing spot is delivered and how should I proceed next time to prevent data loss.

Or do just need to wait longer for the data to appear (may be one day or so ?!)
I am confused...hi

May be Rob can explain in the next zoom-meeting, how the upload of old spots really works - either because of Internet connection problems or even after portable kiwi operation in the field with later upload from home (Glenn's application ?!)

regards,
Ulli, ON5KQ


Re: first time more than 3000spots/hour on all bands...

ON5KQ
 

I mean the current one, which wsprdaemon is using. However I remember the last update of wsprd decoder engine, it was said by the developers, that it should be more efficient in busy bands. Also the summer QRN is a real stress test for the decoder. The developers said, that also in summer qrn their latest decoder (which we have now for at least 6 month or so) should work better....
So nothing to do...hi

Yesterday was just a good test case to verify the robustness of the decoder - I think we must be happy to have such good software.

Today I received my first UK-station on 4m (70Mhz) over a distance of a little more than 450km - no special propagation just groundwave...
The last couple of days I am testing how to use my terminated rhombic antenna on 6m and above. It is pointing to the UK and it turns out to be very broadband. I use it from 160m to 70Mhz for reception with good results.
VSWR as measured with my DG8SAQ VNA:  (30kHz to 60Mhz trace)


Not that bad...

Ulli, ON5KQ


Re: first time more than 3000spots/hour on all bands...

WA2TP - Tom
 

Ulli,

Which new decoder are you referring to?



On Jun 11, 2021, at 2:40 AM, ON5KQ <ON5KQ@...> wrote:

Jim,
on 20m (and the higher bands) usually I hear the locals (100km distance) best, when both antennas are in the same direction (back-scatter)
So in the evening in 20m CW or SSB if stations put their beams to Northamerica, I would hear them best with the beam to Northamerica too. Although the many Germans heard like this are all in the back of the antenna...

So, I think the usually not very directive Wspr-wire antennas (both TX and RX, compared to a Yagi beam antenna) will work in the same way: Backscatter signals are usually stronger than direct ground wave propagation.

Ulli


Re: first time more than 3000spots/hour on all bands...

ON5KQ
 

Jim,
on 20m (and the higher bands) usually I hear the locals (100km distance) best, when both antennas are in the same direction (back-scatter)
So in the evening in 20m CW or SSB if stations put their beams to Northamerica, I would hear them best with the beam to Northamerica too. Although the many Germans heard like this are all in the back of the antenna...

So, I think the usually not very directive Wspr-wire antennas (both TX and RX, compared to a Yagi beam antenna) will work in the same way: Backscatter signals are usually stronger than direct ground wave propagation.

Ulli


Re: first time more than 3000spots/hour on all bands...

ON5KQ
 

yes, you can filter.... Phil's website (VK7JJ) offers the possibility.
Obviously, due to the short skip "QRM" the reception capability for the much weaker DX stations is limited. When there is so much short skip sporadic E reception often F2 reflections (being the normal propagation on shortwave) is not possible or attenuated due to absorbtion.
Yesterday the 20m band opened to North-America after the short skip signals dropped in signal strength.

I was surprised, how well the latest decoder really can cope with extremely busy bands when such propagation overloads the available wspr-frequency space with transmitting stations. I must say it is impressive, that the decoder can reliably pick more than 40stations in a 2min wspr circle from just 200Hz wide band...
How far this will develop in the coming years, when we probably will see a fully open 10m band even during the night and at the same time have good condx on the low bands ?
That is the moment, where I need a new PC for decoding...hi. The current solution with a dual CPU XEON workstation might be not enough..

The sporadic E max usable frequency yesterday was 47Mhz at about 21h (at the time of my first post above) according to the Ionosonde here in Belgium. Still not enough for opening on 6m although the regular stations were heard via tropo, I guess.

Ulli


Re: KFS reception quality - IMD and other (internal?) noise sources...

Rob Robinett
 

I have just returned from 5 hours at KFS and have time for only a brief report:

1) I removed the Nooelec AMB BF and 6/9 dB pads which were between the Qbit LNA outputs and the Kiwi RF inputs.  
    There are 2 Mhz HPF out at the antennas and I have muted the whole of the AM band, so no user can tune there on a Kiwi.  
    So the Omni Kiwis have additional 6 dB gain and the others an additional 9 dB gain.  I will monitor WD's overload log to verify those changes don't result in significant overload events.
2) I confirmed that the NW feed from the primary multicoupler is suffering from severe AM IMD in the 2-4 Mhz region, so this IMB is generated in that multicoupler or more likely by the LNA out at the antenna
    Omni A and B also show that IMD but to much lesser extent.  Fixing these will require help from ARINC
3) Thanks to Glenn's AA and the tinySA, I tracked down the source of the every 40 Khz switcher RFI:  it is a vertically polarized signal generated at the cell phone equipment shed at the base of the cell phone tower shared with the NW antenna.  Again, ARINC is going to have to get  that fixed if they care about it.  That same RFI can be heard on the Omni which is the other antenna closest to the cell phone shed (about 100M away).  The SE and SW antennas are about 200+M away from that shed and so that RFI is barely noticable on those antennas.
4)  I see other switcher RFIs, so I'm sure there are many other bugs to squash, but the signals we see on the Kiwis are pretty much what the portable Kiwi sees when connected to the primary multicouplers.  So all indications are the problems are outside the building. 
4) Glen''s AA + the portable Kiwi showed a very clean spectrum when I got 300M away from the cell shed and main building.  There is very good wifi coverage out in the fields, so I think deploying a solar powered AA + Kiwi as far as possible (e.g 400M)  from the cell shed would result in a very clean full spectrum.

I now understand the location of all the antennas and can see why the NW and Omni will always be impaired by the cell shed.

Rob


On Thu, Jun 10, 2021 at 12:33 PM ON5KQ <ON5KQ@...> wrote:
In the meantime, I must correct my report from above. There are indeed also during the night severe man-made noise sources, which I believe are produced somewhere in the signal distribution system.,,,,more detailed reports I have sent to Rob, to help to find a solution and support kfs...



--
Rob Robinett
AI6VN
mobile: +1 650 218 8896


Re: WSPRDaemon creating empty spot files and nothing for WSPRNet

Rob Robinett
 

Duncan gave me ssh access to his Pi and I was able to see that his RaspSDR wasn't sending audio through the kiwirecorder session, nor was autowspr able to decode on the RaspSDR.  Duncan later reported that an update to his RaspSDR SW fixed those problems and WD was running fine.  However I don't know any more detail about the RaspSDR SW change.

On Thu, Jun 10, 2021 at 1:33 PM Jim Lill <jim@...> wrote:

does wsprd decoder ever run?

On 6/9/21 7:45 PM, Duncan GM5JET wrote:
Hi Rob,

There is only the entry in the log file for starting the pid and then nothing else.

I have watched the updates and I get the message that states that the files contained no spots and that the files should be flushed.  I'm rather stumped!

73
Duncan



--
Rob Robinett
AI6VN
mobile: +1 650 218 8896


Re: first time more than 3000spots/hour on all bands...

WA2TP - Tom
 

Perhaps there is way to filter out by distance and disregard spots that are under xxx Km?


From: wsprdaemon@groups.io <wsprdaemon@groups.io> on behalf of Jim Lill <jim@...>
Sent: Thursday, June 10, 2021 4:30 PM
To: wsprdaemon@groups.io <wsprdaemon@groups.io>
Subject: Re: [wsprdaemon] first time more than 3000spots/hour on all bands...
 

Looking at the data, I see that the EU advantage over North America is on 17-10m  While I'm sure that much of that is from E and F skip I'd bet there is some GW and Ground Scatter  in there too.  When bands are "supposed to be hot" more people turn there TX on and those shorter paths in EU have those  TX to take advantage of.

When I have time, I'll try to do some analysis on that

-Jim

WA2ZKD

On 6/10/21 3:48 PM, KD2OM wrote:
We US stations might as well pack up our tents and go home:)  As soon as the cycle moves along we will be lucky to make the top ten World Wide.

73
Steve KD2OM

.
 

On Jun 10, 2021, at 15:40, ON5KQ <ON5KQ@...> wrote:

just checked: 20m only spots frequently about 40stations per 2min... currently at 900spots/hour -  incredible and only possible with the current decoder, I guess - I don't think earlier versions were able to handle such overcrowded wspr-channels...
Ulli, ON5KQ


Re: WSPRDaemon creating empty spot files and nothing for WSPRNet

Jim Lill
 

does wsprd decoder ever run?

On 6/9/21 7:45 PM, Duncan GM5JET wrote:
Hi Rob,

There is only the entry in the log file for starting the pid and then nothing else.

I have watched the updates and I get the message that states that the files contained no spots and that the files should be flushed.  I'm rather stumped!

73
Duncan


Re: first time more than 3000spots/hour on all bands...

Jim Lill
 

Looking at the data, I see that the EU advantage over North America is on 17-10m  While I'm sure that much of that is from E and F skip I'd bet there is some GW and Ground Scatter  in there too.  When bands are "supposed to be hot" more people turn there TX on and those shorter paths in EU have those  TX to take advantage of.

When I have time, I'll try to do some analysis on that

-Jim

WA2ZKD

On 6/10/21 3:48 PM, KD2OM wrote:
We US stations might as well pack up our tents and go home:)  As soon as the cycle moves along we will be lucky to make the top ten World Wide.

73
Steve KD2OM

.
 

On Jun 10, 2021, at 15:40, ON5KQ <ON5KQ@...> wrote:

just checked: 20m only spots frequently about 40stations per 2min... currently at 900spots/hour -  incredible and only possible with the current decoder, I guess - I don't think earlier versions were able to handle such overcrowded wspr-channels...
Ulli, ON5KQ


Re: first time more than 3000spots/hour on all bands...

KD2OM
 

We US stations might as well pack up our tents and go home:)  As soon as the cycle moves along we will be lucky to make the top ten World Wide.

73
Steve KD2OM

.
 

On Jun 10, 2021, at 15:40, ON5KQ <ON5KQ@...> wrote:

just checked: 20m only spots frequently about 40stations per 2min... currently at 900spots/hour -  incredible and only possible with the current decoder, I guess - I don't think earlier versions were able to handle such overcrowded wspr-channels...
Ulli, ON5KQ


Re: first time more than 3000spots/hour on all bands...

ON5KQ
 

just checked: 20m only spots frequently about 40stations per 2min... currently at 900spots/hour -  incredible and only possible with the current decoder, I guess - I don't think earlier versions were able to handle such overcrowded wspr-channels...
Ulli, ON5KQ


Re: KFS reception quality - IMD and other (internal?) noise sources...

ON5KQ
 

In the meantime, I must correct my report from above. There are indeed also during the night severe man-made noise sources, which I believe are produced somewhere in the signal distribution system.,,,,more detailed reports I have sent to Rob, to help to find a solution and support kfs...


first time more than 3000spots/hour on all bands...

ON5KQ
 

Short skip conditions here in Europe make my 2x Xeon quadcore Workstation be real busy for the first time...
300km far away stations on 20m are 40db over S9 on the rhombic to northwest. This antenna favours rather high angle radiation on 20m.... so the short skip propagation put the kiwi's on fire...hi
10db att-pads nessesary for the first time as well this evening...
I saw Holgers kiwi also overload for short moments for the strong UK BC-signals from Wooferton (250...500kW on 20dbi ant)
My kiwi shows S9+55db for these stations today...
about 19h utc, 21h local, local sunset expected at 19h58 in roughly one hour.... the computer needs almost 70sec - never experienced before

Frank LX1DQ also spots really al lot (also almost 3000spots all band/hour), Holger similar
All bands full throttle (incl. 10m) at the moment...
Only 6m suffers now, due to another 6db att.pad needed to prevent overloading at the rhombic to northwest ...
I think within 30min I can remove it again, as signal strength will weaker a bit...


Has anyone run auto-wspr and/or WD on a RaspSDR?

Rob Robinett
 

One of our users is tying to use a RaspSDR and neither auto-wspr nor WD are seeing signals which are present on the main waterfall display.


Re: WSPRDaemon creating empty spot files and nothing for WSPRNet

Duncan GM5JET
 

Details sent via PM


Re: WSPRDaemon creating empty spot files and nothing for WSPRNet

Rob Robinett
 

Can you give me ssh access to your Pi?
Or we could do a Zoom session


Re: WSPRDaemon creating empty spot files and nothing for WSPRNet

Duncan GM5JET
 

Hi Rob,

There is only the entry in the log file for starting the pid and then nothing else.

I have watched the updates and I get the message that states that the files contained no spots and that the files should be flushed.  I'm rather stumped!

73
Duncan

701 - 720 of 990