Weird WSPR decode anomaly - Probably a FEC problem with wspr decoding software


tony.volpe.1951@...
 

A few days ago, I loaned one of my QCX radios to a mate so he could see how it worked. He is interested in starting WSPR himself and we programmed his call into my QCX40 classic and he ran it for a couple of days on a twenty minute frame setting.

I then got it back and  used it myself WITH MY CALL set up in it.

If I do a database search on his call, I still find a few current decodes which show his call, even though there were no transmissions with his call - only mine is in use. The wrong decodes are on the same frequency as mine with the same Maidenhead square and power. Is it possible that the Forward Error Correction at the receiving station is looking at other parts of the transmission and guessing a callsign the rx did not receive. I am absolutely certain that the call shown was not transmitted for the last two days.


tony.volpe.1951@...
 

I should have added that only 2 stations have 'spotted' these phantom 'transmissions' of the call. It isn't general at all.


Ron Carr
 

The receiving software keeps a database that it uses to match complex calls and 6 character grid squares to the correct call.  And the matching algorithm is not perfect as I believe it uses a hash or similar scheme.  I could see something like this happening if you or he used a call like  K1URC/3   or a grid like FN54fk.


Hans Summers
 

Hi Ron, Tony

That is indeed a mystery. 

Ron, you are correct about 6-char grid squares in the extended WSPR transmission format - but the QCX is not capable of transmitting this extended format. It can only handle the simple normal WSPR transmissions. 

Tony, it cannot be a problem with Forward Error Correction because this relates to the encoding of the transmission itself, on the QCX. Forward Error Correction is, somewhat loosely, the term applied to the encoding of data with additional bits via the used encoding protocol, that allow the decoding station to correct errors in the transmission. The QCX cannot be transmitting or encoding your friend's callsign, since you have replaced it with your own. 

Furthermore it cannot be a problem with the QCX somehow remembering the encoding of your friend's callsign, since the encoded sequence of 162 symbols in a WSPR transmission is not stored in EEPROM, it is always calculated dynamically and exists in volatile RAM only. Therefore it is not persisted through power cycles. It is hard to see therefore, how this mystery could be caused by any problem in the QCX firmware itself. 

Some digital modes augment their decoding logic with a probability-based fuzzy match, looking for probable (but not algorithmically certain) matches between the decoded data, and a database of existing callsigns - which could be callsigns the decoding station has heard before, or in some cases an internet-based list of callsigns. This is a somewhat controversial practice, as far as I know it is not done in the WSJT-X software for WSPR at all; I know JT65 and Opera are two modes that use this controversial technique but I have not heard of it being applied to WSPR, though I could be wrong. 

All of which makes it a bit of a mystery... I do not see how this phenomenon could ever be caused by the QCX firmware itself, though equally well I don't see that it could be caused by WSJT-X either... unless there is some feature of WSJT-X I don't know of, or maybe there are other more aggressive WSPR decoder software packages out there which DO apply the controversial probability match. 

73 Hans G0UPL
http://qrp-labs.com

On Fri, Feb 26, 2021 at 12:53 PM Ron Carr <rcarr@...> wrote:
The receiving software keeps a database that it uses to match complex calls and 6 character grid squares to the correct call.  And the matching algorithm is not perfect as I believe it uses a hash or similar scheme.  I could see something like this happening if you or he used a call like  K1URC/3   or a grid like FN54fk.


Alan G4ZFQ
 

Ron Carr wrote:
The receiving software keeps a database that it uses to match complex calls and 6 character grid squares to the correct call.  And the matching algorithm is not perfect as I believe it uses a hash or similar scheme.  I could see something like this happening if you or he used a call like  K1URC/3   or a grid like FN54fk.
Tony,

Yes, some decodes are difficult or impossible to fathom. Due to the hashtable feature complex calls are suspect and best not used if not really required.
Was there a complex call involved?
You did the right thing, check how many others spotted the "transmission", whether propagation even was possible.
This one seems very strange but so do some other false decodes.

73 Alan G4ZFQ


Alan G4ZFQ
 

as I know it is not done in the WSJT-X software for WSPR at all;
Hans,

It is, the "hashtable" I referred to.
It may be disabled by disabling the "Deep" decode. But that just stops you from making a small number of false decodes.

73 Alan G4ZFQ


Hans Summers
 

Hello Alan
 
>as I know it is not done in the WSJT-X
> software for WSPR at all;

Hans,

It is, the "hashtable" I referred to.

My understanding is that the hashtable is only used for the type 2 and type 3 WSPR transmissions, which are "extended" type transmissions, for callsigns with prefixes or for transmitting 6-character callsigns. 

QCX is only capable of the standard type 1 WSPR transmission. The Ultimate3S kit can do type 2/3 (extended). 
 
It may be disabled by disabling the "Deep" decode. But that just stops
you from making a small number of false decodes.

I have seen that setting but I don't think it is applicable to WSPR. 

I could be wrong - but I am fairly familiar with the WSJT-X documentation and I don't see anywhere in the documentation that would prove me wrong... am I missing something? 

73 Hans G0UPL
http://qrp-labs.com 


tony.volpe.1951@...
 

Thanks for the information and ideas.

I was sure that it was nothing to do with the QCX firmware.

I think somehow it is to do with an aggressive form of decoding which looks in the data for other features in common in the transmitted string.

It can't be anything else and the qcx has been switched off between the transmission of the two calls.


Alan G4ZFQ
 

My understanding is that the hashtable is only used for the type 2 and type 3 WSPR transmissions, which are "extended" type transmissions, for I could be wrong - but I am fairly familiar with the WSJT-X documentation and I don't see anywhere in the documentation that would prove me wrong... am I missing something?
Hans,

I'm not sure if or where any real detailed information is published.
From time to time such things are mentioned in the WSJT-X group.

The hashtable is used for complex calls I do not know if the decoder can "confuse" the transmission types.

https://www.sm2cew.com/Digital%20communications%20using%20minimal%20transfer.pdf

is possibly what you refer to about JT65, in any case it details "Deep Search".
I cannot confirm whether this is relevant to WSPR.

There are quite a few other decoders that have their data uploaded to WSPRnet, I've never looked into the way they work.

But fundamentally we look desperately for answers instead of simply accepting that false decodes are inherent in these modes.
One of the developers has said that the decoder is not working properly if no false decodes are made.

There is also confusion created by the WSPRnet site. The map seems to have bugs, shows stations in the wrong place when another site using the same data shows correct. Maybe a few bugs in the data presentation?

73 Alan G4ZFQ


Ron Carr
 

I located my hashtable ( in appdata local ) and had a look at what it contains.   It stores all calls and not just the extended ones as I had thought.  It appears to use 15 bits and is limited to 32767 entries.  So I would say that any receiving station that needed to use the hash table lookup could locate your friends call if the station hadn't yet received newer information of a transmitter that hashes down to the same 15 bits.

The end of my hashtable:

32618 WY0QGI
32627 VA2ONE
32637 KN4YK
32650 G4BYM
32655 K3MMM
32657 IU0ICC
32668 K2SAQ
32712 W4GHV
32720 W4RYW
32725 M0XDC
32737 VA1VM
32739 KN4ZHP
32745 W9SNE
32746 PY3FJ
32758 G4CAO
32766 KC0TYZ


Allan Nelsson
 

It does sound rather odd yet not FEC related. I am not sure if it is a hash problem but you can try to verify the possibility using this tool:

https://www.rfzero.net/documentation/tools/wspr-hash-collision/

73, Allan OZ5XN


tony.volpe.1951@...
 

Just to finish off this topic I can tell you that the issue has now ceased to show itself. It is as if the records which seemed to be embedded (perhaps in shared hash tables) have now been over written. As I said earlier, the weirdness only existed in a couple of decoding stations anyway. I have seen similar issues of out of date info being displayed when I was experimenting with lower powers by reducing the system voltage on the QCX transceivers (I operate two sometimes and both have had wrong power shown in decodes hours after I changed both the actual power and the power encoded into the wspr stream). These 'old' power values just had to be coming from a receiving station using some old data they had cached somewhere because the same stations repeatedly showed 'old' power values - they were not just single bad decodes. Again - the issue only arose from a very few decoding stations.

Alan's link above took me to this information:

"When sending type 3 messages, i.e. a message with a six character locator e.g. JO65FR, in WSPR the call sign is hashed. This means that instead of sending the call sign a 15 bits hash code is sent. But the hash code is not unique to the specific call sign but “shared” with some other call signs. In such cases hash collision can happen if both transmitting stations are decoded by other stations irrespective of only one of the transmitting stations is sending type 3 messages."  <my bold>

In my case and my pal's, neither of us had complex call signs. Both were 5 character calls encoded into the qcx with a leading space as per the instructions, but every other feature of the transmitted string was the same.

In the end - it doesn't matter a bit in the great scheme of the world's problems. It was just a bit of a puzzle to people who like to understand what is going on behind the wspr scenes.)

Thanks again to all for your thoughts and ideas.

de G0BZB
op Tony


Alan G4ZFQ
 

In the end - it doesn't matter a bit in the great scheme of the world's problems. It was just a bit of a puzzle to people who like to understand what is going on behind the wspr scenes.)
Tony,

Yes, but as Hans reminded me most explanations of false decodes are based on assumptions.
Questions have come up regularly for many years, some causes are identified but the more obscure ones like you have experienced remain unexplained except by guesses.

In WSJT-X Hashtable.txt contains the hashtag of a call and it's 4 figure locator. The hashgtag is calculated for all received spots, not just those using complex transmissions. Does anyone actually know every use of hashtag.txt? Is it used for normal one-part messages?
Received power figure is stored in ALL_WSPR.txt. Is it referred to when decoding?

We often do not even know what decoder is being used by stations that give false spots. There are at least two other decoders in common use. WSPR daemon in the raspberry pi etc. and that in the Kiwi are the ones I know of. There are others and variants.
Looking at worldwide spot summaries makes me wonder if the majority of reports now are not from WSJT?

Some false callsigns like LQ0HGF in RP22 have come up from various stations for years.

It needs research from someone who fully understands all that goes on, I'm not aware of anyone who has done that. Perhaps those people realise there will never be a complete answer..

73 Alan G4ZFQ


tony.volpe.1951@...
 

Thanks for your input Alan. I certainly don't disagree, but I am far from understanding what is happening. Another annoyance is when there are false locations posted on the map, although I can easily see why they happen if a signal is decoded and is on the edge f intelligibility. Once again though, it is often the same receiver stations which keep on posting these bits of nonsense in my experience and sometimes the reported snr is far from being weak.

Cheers to all who put in their thoughts.

Tony G0BZB


Greg Winterflood VK8MD
 

Tony,

I do not wish to stir the possum, but why is this not a failure of memory on your part? You give no exact dates and do not give the call sign of your friend. His call was already in your QCX when it was returned.

Could you have fired it up before you remembered to change his call back to your callsign? Why did you check to see if his call sign had been received after your QCX was returned to you?

What are the odds that FEC errors would have come up coincidentally with the call sign of your mate? A simpler explanation is needed. Life is too short to entertain any other idea. Forensically your QCX is guilty of transmitting his call sign along with your QTH, Power and Frequency. 

At 72 soon to be 73, I am forgetting things all the time ;-)

73 Greg VK8KMD

On Fri, Feb 26, 2021 at 07:10 PM, <tony.volpe.1951@...> wrote:
A few days ago, I loaned one of my QCX radios to a mate so he could see how it worked. He is interested in starting WSPR himself and we programmed his call into my QCX40 classic and he ran it for a couple of days on a twenty minute frame setting.

I then got it back and  used it myself WITH MY CALL set up in it.

If I do a database search on his call, I still find a few current decodes which show his call, even though there were no transmissions with his call - only mine is in use. The wrong decodes are on the same frequency as mine with the same Maidenhead square and power. Is it possible that the Forward Error Correction at the receiving station is looking at other parts of the transmission and guessing a callsign the rx did not receive. I am absolutely certain that the call shown was not transmitted for the last two days.
Greg Winterflood VK8KMD
Alice Springs, Central Australia
PG66wh


Alan G4ZFQ
 

On 27/02/2021 11:19, Alan G4ZFQ via groups.io wrote:

Some false callsigns like LQ0HGF in RP22 have come up from various stations for years.
Sorry to drag this on, you revived my interest in this. I've just checked 3 stations currently reporting this non-existent station. They are not using WSJT-X. Some time ago I did query one station "spotting" this station, he was using WSPRdaemon.
Therefore any guesses about this false particular decode must discount WSJT-X.

What stations gave your false reports Tony? If they appear in the Challenge http://wspr.pe1itr.com/index.php then software used (K1JT's and Kiwi, some Linux) is listed. Other decoders are not. If the decoding software is unknown then there is no way to even guess reasons for false spots.

(Of the top 10 reporters on 80m yesterday none were using WSJT.)

73 Alan G4ZFQ


tony.volpe.1951@...
 

Re WSPR decoding errors.

This matter is not an issue of user idiocy. They happen frequently.

I have long seen errors in power level. Just change the power level entered into your transmitting device and you will see a massive proportion of stations continue to record the old power level. These can not be down to the transmitting station, or to memory lapses on the part of presumed dribbling idiot time wasters on the QRP labs list. This could only happen if the receiving stations were in fact consulting a stored cache of data when reporting decodes. These power level errors are not confined to marginal reception levels. Some of them are seen at high signal levels.
 
This morning, out of 256 wspr reports (252 correct) one station has reported my 40 meter beacon to be in 2 false locations when the received snr is as strong as -21db. Another reported a false location when the signal level was -13db. The vast majority of stations show correct data, often at very much lower snr levels. When errors of various types occur they are often repeated, sometimes several times by the same small number of receivers. 

I have emailed one reporter this morning asking about what software he is using.

73s

G0BZB


Alan G4ZFQ
 

I have emailed one reporter this morning asking about what software he is using.
Tony,

And what way do you see these false reports?
There have been cases when it is said the correct data has been uploaded by a spotter but appears in the wrong place on the map.
There are lots of places where errors or bugs can slip in.

Although I do not specifically look at my spot reports for incorrect reported power or location I rarely notice any in the thousands I scan through each day.

73 Alan G4ZFQ


Greg Winterflood VK8MD
 

Tony,

Please accept my apologies. I am way behind the times and have not been keeping up with changes in the WSPR world.

I began WSPRing 7.5 years ago and tapered my activity around the time Google put the kybosh on the Map.

My current use is virtually nil. I did not appreciate that there are reporting systems other than the one based at Princeton.

73 Greg VK8KMD 
----
Greg Winterflood VK8KMD
Alice Springs, Central Australia
PG66wh 


tony.volpe.1951@...
 

No probs Greg and no need to apologise. No harm done either way.