Deep Decoding and CPU Loading
Jim Lill
With the Data Integrity effort done on my "8088" page, I shifted my efforts to understanding the trade off between "deep decoding" and CPU loading.
At one extreme you can run a I9 super box, not worry about CPU loading, and use values like -C 10000 -o4 -d and dig as deep as possible. You'll get more decodes along with false decodes but will need a lot of CPU to do it Those values can be controlled by the top line you add to the wsprdaemon.conf file, eg: WSPRD_CMD_FLAGS="-C 3000 -o 4 -d" which overrides the more conservative -C 500 -o3 Through many tests on my Atomic Pi cluster, with only one critical band per APi I was able to determine that WSPRD_CMD_FLAGS="-C 3000 -o 4 -d" provides a good balance between deep decoding and available MIPS. The limit your computing platform will have is a function of how many channels you decode and how aggressive you set the -C and -o values. I suggest you start with -C 1000 -o 4 and see it that completes recording/decoding cycles. If it does, you have increase the -C value until you run out of MIPS. Use of the top command will show CPU usage. Touching 100% isn't terrible but if you see two kiwirecorder instances running for the same band, at the same time, you've gone too far. Obviously, an older Pi may already be at the limit. -Jim WA2ZKD |
|
Erwin - PE3ES - F4VTQ
Hi Jim,
Can something be said on the increase in number/percentage of false decodes due to deeper seeking. That trade-off line would be interesting. Thanx |
|
Jim Lill
Erwin, Since false decodes can result from various mechanisms, a precise relationship is not possible. However we can come close by looking at a day's false tally for a station. For 22 Sep, every reporter in the top 200 had at least one false. When I was using values of -C 1000 o-4, I would typically have 1-2 false spots. Yesterday, was a very active day and with my current -C 3000 -o 4, I had 6 false spots. That data may be viewed for a current day at: http://www.jimlill.com:8088/data/tally.false and for previous days at http://www.jimlill.com:8088/previous. The impact of false decodes on overall spot count is trivial, < .02% typically. However, false decodes are very impactful on accuracy of the unique transmitter count since every false appears as a unique. My data integrity effort on my "8088" page solved that data corruption with its use of an adaptive whitelist scheme and unique errors are essentially zero, perhaps 1-2 overall per day, < .04% Jim
On 9/23/22 03:18, Erwin - PE3ES - F4VTQ
via groups.io wrote:
Hi Jim, |
|
Andrew Cowan
Jim
toggle quoted message
Show quoted text
I have just added that line to my conf running WSPR on 80m to 10m with daemon. So we will see if that changes anything. Andrew GM0UDL On Thu, Sep 22, 2022 at 01:38 PM, Jim Lill wrote:
With the Data Integrity effort done on my "8088" page, I shifted my |
|
Hans V alphen
I have stopped my WSPR decoders as for this time I feel that the competition is to much disturbed by some reporting stations who let their system running outputing way too many false decodes. It would be acceptable that during such an experiment a station reports more than 10 false uniques for one or at the most two days. We now see stations reporting much more false decodes for many days after each other. In that way competition becomes an illusion no longer worth to put in some effort. I have been on listening and reporting for WSPR at least 4 years and had fun in improving my setup, mainly hardware wise, better antenna and better receiver and also better software. |
|
Jim Lill
Please note that both my own http://jimlill.com:8088 and kb9amg pages excise false decodes thus any ranking is accurate. -Jim
On 9/24/22 05:48, Hans V alphen via
groups.io wrote:
|
|
Hans V alphen
There are other pages showing the ranking which do not correct for false decodes. |
|
Hans,
If you use advanced search on wspr.rocks with the phantom spots query which searches for single spots you will get an idea of the false spots. The single spot method is not completely accurate but is in the ballpark. The wspr challenge page by PE1ITR used to show single spots as errors but doesn't report them any longer. 73 Steve KD2OM |
|
Hans V alphen
Mail to WA2TP I noticed that you are cheating for more than one month in WSPR receiving and sending very many false spots. You are top score nr 1 for the whole month in the falls spots as reported at: http://jimlill.com:8088/previous/2022-10-08/tally.false In practical you have even more falls spots, not all are found at jimlill.com. It's obvious that you would like to have a high score in the WSPR reporters, that's why I decided to give you some help. In my opinion my help is quite succesfull with an absolute top score on 80 mtr's. |
|
Rob Robinett
Hans, Your reporting of false spots with Tom's call sign is corrupting the wsprnet.org and wsprdaemon.org spot databases. Tom has been making crucial contributions to our wsprdaemon development effort and our study of the use of FST4W on the HF bands. Tom's very few false decodes are the result of limitations in the wsprd and jt9 decode software which we hope his work will help improve. Please join our effort to improve WSPR and FST4W and not interfere with it. Rob
--
|
|
WA2TP - Tom
Hello Hans,
Since this is our first introduction, Nice to meet you. My use of the available -C values built into the WSJTX and WSPRDAEMON software are by no means any form of cheating. In fact, those values help decode poor SNR spots in my noisy environment. They are available to use by anyone and are currently utilized by many who understand the software. At my site with 64RX channels, many RX antennas, and lots of processing power, it leads to roughly a .04% output of false decodes as proven by Jim Lill. These false decodes are simply a product of the JT9 decoder which identifies it can be improved: this is goal for much of this work being done by others.
While these decodes are not deliberate on my part whatsoever: rather a product of the technology, your actions are not only deliberate: they are in a way direct cyberattack on WSPRNET and the entire WSPRNET community by way of using my callsign without my consent, intentionally uploading erroneous data to WSPRNET, and holding the wsprnet community hostage to this false data. This does constitute fraud.
Sadly, this is far from the comradery that Amateur Radio is known for.
What to do? Do I contact FCC and CEPT? Or do I let this go provided that you cease and desist from this deliberate activity? In the spirit of the hobby, I am more than willing to let this dissipate and move on without further action.
I will close by saying that I have invested a significant amount of time and money, (well over 10k US) in just the past 2 years alone, on processing and rx equipment solely for WSPR. I contribute significantly to the community: I have provided test platforms form WSPRDAEMON development, as well as other sites to use the data from these systems such as Jim Lill and WSPRrocks. The most recent contribution was the use of my systems by many others, namely Gwyn Griffiths, for the analysis of spectral width and spreading effects on 4W modes, which helped uncover several bugs within the KIWI GPS integration which resulted in significant frequency drift. This bug was affecting ALL KIWI users and has now been fixed thanks to these efforts. This one kiwi bug fix has improved kiwi spot reporting by orders of magnitude. All of my actions are with the focus of improving receiving capabilities at suburban sites challenged by manmade RFI. It's not warming that your actions spawned from no ill will meant of my R&D. What say you? are you willing to cease and desist? I will leave it up to you. |
|
Hans V alphen
Hello Tom and Rob, After my mail in this group nothing happened, the hams causing lot's of false spots remained in this way of working. In my opinion it's causing just as much corruption in the data base as my action. Looking at last month I see that Tom has corrupted the database with at least 3000 false unique decoded spots. This is a lot more than I caused by sending spots in 2 days.
|
|
John K5MO
I've only played with WSPR for a few years, but I only learned today that it's a competition. Who knew? 73 John K5MO. |
|
Hans V alphen
Tom, You gave me some information on your setup, so perhaps it's nice if I do the same. As antenna I am using a single antenna a magnetic loop home build based on a Alford Loop. |
|
WA2TP - Tom
Hi Hans,
I would like to add that as a show of good faith, and in the spirit of our HAM community, that as of a few hours ago I have reduced the -C decode value by 30% of its maximum value. We won’t know what that impact is for several days. |
|
Jim Lill
The effects of reducing the -C value to 3000 will align you with many others who run that or less. I would anticipate that the false count will drop although you may still be high as you run so many receivers. The hard thing to quantify is how many valid "deep" spots you'll
miss, if any, running 3000. I ran a lot of tests here, as did
KD2OM with different C values and I was never able to correlate
that value with any increase in valid spots. There are simply too
many variables at play, band condx, TX hopping variance etc.. The
only method that would have any certainty would be to have
multiple identical RX systems hooked to the sample antenna and
take a large sample size of spots. Even that has accuracy limits
as the one time I tried that, with equal C values, the A/B results
were dicey on very low SNR signals. -Jim WA2ZKD
On 10/10/22 14:20, WA2TP - Tom wrote:
(in part.......)
|
|
WA2TP - Tom
Jim,
There are just so many variables often not considered, some of which were left out are how it can help at a site that has much higher noise floors, and compromise antennas. The example of KD2OM is not necessarily a great one since he has room for large antennas like beverages which outperform most antennas on the low bands. Also, as a note, RFI often changes with weather and simply daily activity. For another example, on the weekends in my densely populated neighborhood when everyone is home, my noise floor is often elevated by 10dbm. The simple answer is, that it is not that simple, to say "one size" fits all. |
|
Jim Lill
Tom, your place is "spacious" compared to mine. My testing was
done both here, at one extreme, and at Steve's, closer to the
other. On 10/10/22 15:30, WA2TP - Tom wrote:
Jim, |
|
Wow! |
|
Rob Robinett
There is a "WSPR Challenge" web site http://wspr.pe1itr.com which treats WSPR reporting as a contest. But I don't think anyone in this WD views WSPR in that way. On Mon, Oct 10, 2022 at 8:36 AM John K5MO <johnk5mo@...> wrote:
--
|
|