Date   

Dropping spots.

Rolf Ekstrand
 

Have observed that VK7JJ is dropping spots. Just found 10 minutes  missing whereas they are OK on WSPRnet. 
I suspect that there are more to this, but have not had time to dive into it.   It appears that jimlill.com  is  also affected.

73 Rolf  K9DZT


Re: To Proxy or not to Proxy?

G7ELK
 

Thanks for explaining Rob.


Local Monitoring

G7ELK
 

The recent Wsprnet issues have coincided with me moving to using Wsprdaemon.  I hope it wasn't me....

Good morning All,

As a result of the recent Wsprnet  disruption, it has been quite difficult to see if my new Wsprdaemon setup has been working properly - and of course I have repeatedly jumped to the conclusion that there is some kind of issue at my end, leading to lots of fiddling with settings and multiple unnecessary reboots of both the Pi and the Kiwisdr.  

I now know how to look at the Wsprdaemon upload files to see what is going on, but it would be nice to have a continuous output of spots to a console or something to provide some reassurance that all is ok.  I guess I could dig into my 25 year out of date Unix knowledge and try and do something with a script, but I'm wondering if this is something that has already been done?

Is there a Wsprdaemon command line option which I am missing which continually echos spots to the console?  Has anybody else implemented something to allow continuous local monitoring of spots?

Regards,
             Rob


Re: Problem with wspr-server ??

Stu C
 

That makes sense as it felt like wsprnet was there, but just too busy.
I do wonder why people attack that sort of site, perhaps they are doing some atmospheric heating and would rather it not show up in propagation maps.
Stu


Re: To Proxy or not to Proxy?

Rob Robinett
 

Hi Rob,

I have not implemented the proxy upload feature, so currently "proxy" is the same as "noise".  Even when it get implemented, I worry about making WD responsible for delivery of spots to wsprnet.org.  The "proxy" feature is intended for use at sites where internet connections are extremely bandwidth limited like an arctic station which uses a satellite connection.

So for now, if uploads to wsprdaemon.org are enabled you will continue to deliver spots directly to wsprnet.org.

Rob


On Mon, Dec 21, 2020 at 6:01 AM <rob@...> wrote:
All,

I am new to WsprDaemon, having only had things up and running for a couple of days.  Pretty modest station - single KiwISDR  with 8 WSPR channels into an end fed half-wave cut for 20M.  Currently I have a default setup with spots sent directly to Wsprnet, but have been wondering whether using the WsprDaemon proxy might me a better idea.  What are the pros and cons please?

I realize this might not quite be the best time to be asking this question - at various times in the last 24H the number of my spots recorded at wsprnet (when I could get in) was different and always higher than reported by WSPRD querying the Wsprdaemon database, although they seem to be coming close to being in sync again...

BTW it was a very easy automated setup on an Pi4 2GB with the latest standard Raspberry Pi OS "full fat" image.  Many Thanks for your hard work Rob!

Regards,

             Rob (G7ELK, Fareham UK)



--
Rob Robinett
AI6VN
mobile: +1 650 218 8896


Re: Problem with wspr-server ??

Rob Robinett
 

I have just learned from Gary that wsprnet.org was being subjected to DOS attacks.  

On Mon, Dec 21, 2020 at 12:54 AM ON5KQ <ON5KQ@...> wrote:
Many thanks for all the explainations... it is clear now.

So as a conclusion from the client point of view:
In any case it is best, to just keep the system running and not to try some "tweek-action" or even reboot at the client site, as soon as it is verified, that wsprd decodes correctly at the client site.

Ulli, ON5KQ

--
Rob Robinett
AI6VN
mobile: +1 650 218 8896


To Proxy or not to Proxy?

G7ELK
 

All,

I am new to WsprDaemon, having only had things up and running for a couple of days.  Pretty modest station - single KiwISDR  with 8 WSPR channels into an end fed half-wave cut for 20M.  Currently I have a default setup with spots sent directly to Wsprnet, but have been wondering whether using the WsprDaemon proxy might me a better idea.  What are the pros and cons please?

I realize this might not quite be the best time to be asking this question - at various times in the last 24H the number of my spots recorded at wsprnet (when I could get in) was different and always higher than reported by WSPRD querying the Wsprdaemon database, although they seem to be coming close to being in sync again...

BTW it was a very easy automated setup on an Pi4 2GB with the latest standard Raspberry Pi OS "full fat" image.  Many Thanks for your hard work Rob!

Regards,

             Rob (G7ELK, Fareham UK)


Re: Problem with wspr-server ??

ON5KQ
 

Many thanks for all the explainations... it is clear now.

So as a conclusion from the client point of view:
In any case it is best, to just keep the system running and not to try some "tweek-action" or even reboot at the client site, as soon as it is verified, that wsprd decodes correctly at the client site.

Ulli, ON5KQ


Re: Problem with wspr-server ??

Rob Robinett
 

The WD clients independently upload spots to two servers:

1) 'Classic spots' (like those from WSJT-x) are sent directly to wsprnet.org
2) 'Enhanced spots'  (the same spots with additional information) to wsprdaemon.org

The WD client caches both sets of spots in the non-volatile directory tree ~/wsprdaemon/uploads.d/..... until the destination server acknowledges successfully receiving those spots.  WD limits wsprnet.org upload transactions to at most 1000 spots, so after an extended outage it can take minutes or hours for all of the spots in the WD cache to be uploaded to the wsprnet.org server.

There is a diagram of that data flow at: http://wsprdaemon.org/technical.html

Since Nov 20th Jim's jillil.com 'top spotters' site has obtained its spots from the wsprdaemon.org scrape.
Phil's wsprd.vk7jj.com site and Peter's WsprWatch IOS app also use the wsprdaemon.org scrape.

++++++++++

The WD server scrapes the wsprnet.org database 3 times during each 2 minute wspr cycle. Occasional timeouts are common, but in the last 24 hours the scraper has been frequently reporting multiple consecutive timeouts.  The WD scraper reports missing sequence numbers ('gaps') in those spot reports, but in spite of all the timeouts I see relatively few large gaps in the spot sequence numbers once they are finally delivered.

However, neither the WD client nor the WD scraper can verify if the wsprnet.org database is dropping the spots it acknowledges that it has received.  It is on my TODO list to add a 'VALIDATE' option to the WD client's wsprnet.org functions.  Until then, I know of nowhere to learn if wsprnet.org has accepted but then invisibly dropped the spots being uploaded.  I can think of no way for the WD scraper to more fully validate the spot data it receives from its API call to the wsprnet.org server.

Rob

On Sun, Dec 20, 2020 at 9:22 AM ON5KQ <ON5KQ@...> wrote:
Hi Gwyn,
Does WA2ZKD's list reflect the spots in the wsprnet.org site, or does it show wsprdaemon database ?

New spots are showing up now again on the WA2ZKD site as well as on Phil's website wsprd.vk7jj

However where are the older spots from today - only 9000 spots showed up so far - it should be about 20000 instead at this time... Are these spots stored somewhere or lost ?
The reason,  why I want to know is, that I do not know, what to monitor is such a case as today. Perhaps I should have taken action, as there is something wrong here with my PC ?
However:
- I know, decoder worked ok
- I know in the afternoon all decodes where stored in many files waiting to be accepted as uploaded. However it doesn't happen. Instead at a certain point all files with old spots from today disappeared and never showed up again... at least not on WA2ZLD site as well as VK7JJ's site.

There is a gap with no spots from 8h28utc to 16h46 utc. with no spots in between, which explains only 9000 spots so far... Does it mean both websites are not showing wsprdeamon database but wsprnet.org, which is faulty ?
Unfortunately I do not understand the data flow.... hope to learn it sometimes...hi

73s
Ulli, ON5KQ



--
Rob Robinett
AI6VN
mobile: +1 650 218 8896


Re: Problem with wspr-server ??

Gwyn Griffiths
 

Hello Ulli
Some time ago I put a simplified data flow at the bottom of the webpage at:   http://wsprdaemon.org/technical.html

JIm, WA2ZKD has moved to using spots from wsprdaemon, database wsprnet, table spots - that is data from wsprnet.org obtained via Rob's program and the API. Of course, as today, when wsprnet.org has problems then all users of wsprdaemon spots table, such as WA2ZKD, VK7JJ etc will not have the complete data.

As far as I can see from here with my own spots, there is nothing at all wrong with our own wsprdaemon installations. 
I could see this morning that my spots were being accepted into wsprnet.org but not being shown on wsprd.VK7JJ.com for example because the API to bring them in to wsprdaemon, database wsprnet, table spots was timing out - just like our direct queries to wsprnet.org.

Gwyn G3ZIL


Re: Problem with wspr-server ??

ON5KQ
 

Hi Gwyn,
Does WA2ZKD's list reflect the spots in the wsprnet.org site, or does it show wsprdaemon database ?

New spots are showing up now again on the WA2ZKD site as well as on Phil's website wsprd.vk7jj

However where are the older spots from today - only 9000 spots showed up so far - it should be about 20000 instead at this time... Are these spots stored somewhere or lost ?
The reason,  why I want to know is, that I do not know, what to monitor is such a case as today. Perhaps I should have taken action, as there is something wrong here with my PC ?
However:
- I know, decoder worked ok
- I know in the afternoon all decodes where stored in many files waiting to be accepted as uploaded. However it doesn't happen. Instead at a certain point all files with old spots from today disappeared and never showed up again... at least not on WA2ZLD site as well as VK7JJ's site.

There is a gap with no spots from 8h28utc to 16h46 utc. with no spots in between, which explains only 9000 spots so far... Does it mean both websites are not showing wsprdeamon database but wsprnet.org, which is faulty ?
Unfortunately I do not understand the data flow.... hope to learn it sometimes...hi

73s
Ulli, ON5KQ


Re: Problem with wspr-server ??

Gwyn Griffiths
 

I've emailed Rob with a note that many of the API requests to wsprnet.org are timing out and there are large gaps in the sequence numbers of spots being returned by wsprnet.org. There is certainly congestion at wsprnet,org today.

However, I've found that upload into the wsprdaemon_spots table in the tutorial database is working well, as this is entirely independent of wsprnet.org. I know terminology can sometimes be confusing, so in case it helps, here is a reminder:

On any wsprdaemon server (wsprdaemon.org, logs1.wsprdaemon.org, logs2.wsprdaemon.org):

1. Database 'wsprnet' has a table 'spots' gathered via an API from wsprnet.org

2. Database 'tutorial' has two tables 'wsprdaemon_spots' and 'wsprdaemon_noise' from data sent directly to the wsprdaemon server from your client wsprdaemon application, and so is unaffected by any congestion at wsprnet.org

regards
Gwyn G3ZIL


Re: Problem with wspr-server ??

ON5KQ
 

13h50 utc
can't find any spots on the wsprdaemon server database when connecting via Grafana... since early morning all bands.
Originally I saw a lot of files on my local workstation in the wsprdaemon folder, waiting to be uploaded, however they are all erased now.
Curl.log only zeros... no upload confirmations...
There must be something broken... in the meantime, I keep all running here, although it doesn't seem that decodes are written in files, because there is a network error....
The merge.log shows all the spots from the kiwi on each band, but nothing arrived at the wsprdaemon server...

Ulli, ON5KQ


Re: Problem with wspr-server ??

Stu C
 

Might be a loading/DDOS thing as I can see some of my spots (one band) but tried to refresh for all bands and no luck.
The aggregation site pe1itr.com failed to tally today.
On WA2ZKD's site i'm too far up the listing today so might be worse from some locations as a lot of the top users are missing spots?
Stu


Problem with wspr-server ??

ON5KQ
 

No spots sinds several hours...

My local PC works normally with well above 2000decodes/hour...
But nothing appears at the servers

???

Ulli


Re: Noiseblanker in kiwirecorder as a pre-processor to reduce local fence noise before wd writes the wav-files

Cathal Ferris
 

I have a suspicion that the fence pulse might be overloading the Kiwi frontt-end, in which case information is lost anyway before it gets to the noise blanker stage.

Another option, assuming that you have the room at the bottom of your signal spectrum, is to put an attenuator on the Kiwi antenna input and push your received noise floor down closer to the Kiwi's hardware limit. If the numbers work out right it should mean that the peak of the fence pulse doesn't saturate the Kiwi frontend, but still allowing the Kiwi to have the full bit depth at the quiet end to have all of the information decoded from the incoming signal.

If my reading of the flow of wsprdaemon decoding is right, then the presence of the pulse is of no relevance in itself as the protocol will happily work around the presence of the impulse noise; but that makes the assumption that the inbound signal hasn't been clipped or otherwise affected between the pulses due to the impulse noise.

My two KiwiSDRs and the RaspberrySDR that I have pulling signal from an Active-Antenna AAA-1d amplified loop antenna (via one of SV1AFN's "DXer's High-Linearity Pre-Amp (LNA) for HF Receivers" to split and amplify the signal) all appear at times to have overloading, and I am suspecting the ethernet over powerline that the neighbours are claiming not to have. I have yet to try attenuating the signal into the mixer to see the difference, it's a project for over the Christmas.

Regards,
-Cathal


Re: Noiseblanker in kiwirecorder as a pre-processor to reduce local fence noise before wd writes the wav-files

ON5KQ
 

Yes,

It is my intention to compare the Noise blanker enabled WD script against a standard (original) WD script directly and see, if there is any significant difference in reporting statistics. For that purpose I will use two different computers. But antenna will be exactly the same (power splitter to two different kiwi-SDRs)

Most likely I will set it up beginning of next week.

The strength of the noise influencing the North West antenna is however changing - sometimes stronger, sometimes less... we will see.

I will set-up a different call sign for the modified WD-script such as on5kq-nb, so that we can compare easily the results in Grafana.

Ulli, ON5KQ

Yes, you must modify the WD.sh source code.
So do as you say, run a modified version on one Pi reporting as one call sign, and compare it against a standard versionof WD.sh

 



Re: Noiseblanker in kiwirecorder as a pre-processor to reduce local fence noise before wd writes the wav-files

John
 

Ulli,
I remember articles written when WSPR was new and many were thinking of using noise reduction features. The developer recommended to not use any processing on the I/Q files as the WSPR algorithms were designed to find information deep down in the noise. Noise reductions remove information that WSPR  is able to use.

John
TI4JWC


Re: Noiseblanker in kiwirecorder as a pre-processor to reduce local fence noise before wd writes the wav-files

Rob Robinett
 

Yes, you must modify the WD.sh source code.
So do as you say, run a modified version on one Pi reporting as one call sign, and compare it against a standard versionof WD.sh

On Tue, Dec 15, 2020 at 9:42 AM ON5KQ <ON5KQ@...> wrote:

hi Rob,

I updated to latest version of WD with git pull now on my main WSPR Supermicro workstation...


I realize, that it is not possible to use the .conf-file, but edit directly in the wsprdaemon.sh file ?

Because changing the code in wsprdaemon.sh  it will take effect for all kiwis at once and not just a a specific kiwi with the disturbed antenna. It may not harm and I will try tomorrow.


But it is not possible to report any improvement with decent data from such a simple test, because there is no reference. Instead, I would need a different set-up:

- One kiwi run on a raspberry with the same (fence noisy) antenna and original WD version 2.10i

- Another kiwi controlled by a different PC with WD and enabled Noiseblanker with same fence noise antenna perhaps with modified callsign like ON5KQ-NB  ...

Then we can see any difference over time...

I need to set it up for testing... please qrx a while ...hi


Ulli, ON5KQ


Hi Ulli,
 
Make sure you are running v2.10i:
 
pi@Pi-2-KJ6MKI:~/wsprdaemon $ git pull
Already up to date.
pi@Pi-2-KJ6MKI:~/wsprdaemon
 
 
It will be interesting to learn the results of your experiment.  Before making changes, save a copy of the unmodified wsprdaemon.sh, e.g.
 
pi@Pi-2-KJ6MKI:~/wsprdaemon $ cd ~/wsprdaemon/
pi@Pi-2-KJ6MKI:~/wsprdaemon $ cp -p wsprdaemon.sh wsprdaemon.sh.save

Then you can add the "-nb" to line 1446 of V2.10i, i.e::
 
        python3 -u ${KIWI_RECORD_COMMAND} -nb \
 
Rob

On Tue, Dec 15, 2020 at 1:07 AM ON5KQ <ON5KQ@...> wrote:
Rob, this is a question for the code-expert...:

Sometimes I have some fence pulse noise on one of my antennas.
I found out, that when listening on this antenna with the kiwi, I can completely erase the noise, when switching on the kiwi noise blanker in standard mode.
I believe that the wav-files written by wsprdaemon do no process the audio information before it is written in memory for later decoding. It means that kiwirecorder do NOT use the noiseblanker normally, when writing the WAV files.

Now I found there are many options, which can be set for the kiwirecorder.
One can find the options in the wsprdaemon/kiwiclient directory by doing:

$ python3 kiwirecorder.py --help

This calls a list of all available options for the kiwirecorder. In this list of available options we can find for example:

--nb         Enable noise blanker with default parameters

This is exactly I was looking for...
Now my question:
- Can I use this option somehow as a parameter setting in the wsprdaemon.conf file, to activate the noise blanker as an pre-processor for Wsprdaemon ?
- How would I do it ? What is the syntax, I would need to use, to activate the noise blanker by an additional line in Wsprdaemon.conf...

I am thinking about something similar, as you once showed how to change the parameters for 'wsprd'  decoding engine...
Could you post an example, what exactly I would need to add in wsprdaemon.conf ?

If this is not possible in the conf - file, is there a different method how to activate the noise blanker for some tests ?

Many thanks for some help....

Ulli, ON5KQ




 
--
Rob Robinett
AI6VN
mobile: +1 650 218 8896




--
Rob Robinett
AI6VN
mobile: +1 650 218 8896


Re: Noiseblanker in kiwirecorder as a pre-processor to reduce local fence noise before wd writes the wav-files

ON5KQ
 

hi Rob,

I updated to latest version of WD with git pull now on my main WSPR Supermicro workstation...


I realize, that it is not possible to use the .conf-file, but edit directly in the wsprdaemon.sh file ?

Because changing the code in wsprdaemon.sh  it will take effect for all kiwis at once and not just a a specific kiwi with the disturbed antenna. It may not harm and I will try tomorrow.


But it is not possible to report any improvement with decent data from such a simple test, because there is no reference. Instead, I would need a different set-up:

- One kiwi run on a raspberry with the same (fence noisy) antenna and original WD version 2.10i

- Another kiwi controlled by a different PC with WD and enabled Noiseblanker with same fence noise antenna perhaps with modified callsign like ON5KQ-NB  ...

Then we can see any difference over time...

I need to set it up for testing... please qrx a while ...hi


Ulli, ON5KQ


Hi Ulli,
 
Make sure you are running v2.10i:
 
pi@Pi-2-KJ6MKI:~/wsprdaemon $ git pull
Already up to date.
pi@Pi-2-KJ6MKI:~/wsprdaemon
 
 
It will be interesting to learn the results of your experiment.  Before making changes, save a copy of the unmodified wsprdaemon.sh, e.g.
 
pi@Pi-2-KJ6MKI:~/wsprdaemon $ cd ~/wsprdaemon/
pi@Pi-2-KJ6MKI:~/wsprdaemon $ cp -p wsprdaemon.sh wsprdaemon.sh.save

Then you can add the "-nb" to line 1446 of V2.10i, i.e::
 
        python3 -u ${KIWI_RECORD_COMMAND} -nb \
 
Rob

On Tue, Dec 15, 2020 at 1:07 AM ON5KQ <ON5KQ@...> wrote:
Rob, this is a question for the code-expert...:

Sometimes I have some fence pulse noise on one of my antennas.
I found out, that when listening on this antenna with the kiwi, I can completely erase the noise, when switching on the kiwi noise blanker in standard mode.
I believe that the wav-files written by wsprdaemon do no process the audio information before it is written in memory for later decoding. It means that kiwirecorder do NOT use the noiseblanker normally, when writing the WAV files.

Now I found there are many options, which can be set for the kiwirecorder.
One can find the options in the wsprdaemon/kiwiclient directory by doing:

$ python3 kiwirecorder.py --help

This calls a list of all available options for the kiwirecorder. In this list of available options we can find for example:

--nb         Enable noise blanker with default parameters

This is exactly I was looking for...
Now my question:
- Can I use this option somehow as a parameter setting in the wsprdaemon.conf file, to activate the noise blanker as an pre-processor for Wsprdaemon ?
- How would I do it ? What is the syntax, I would need to use, to activate the noise blanker by an additional line in Wsprdaemon.conf...

I am thinking about something similar, as you once showed how to change the parameters for 'wsprd'  decoding engine...
Could you post an example, what exactly I would need to add in wsprdaemon.conf ?

If this is not possible in the conf - file, is there a different method how to activate the noise blanker for some tests ?

Many thanks for some help....

Ulli, ON5KQ




 
--
Rob Robinett
AI6VN
mobile: +1 650 218 8896


501 - 520 of 579