Date   
locked Spur 4.8kHz upper from Centre Freq with recent Firmware Releases of HF+ 1.6.7 & 1.7.0

Ian DXer
 

Hi Youssef & all,

A couple of weeks ago I updated firmware from 1.6.3 to 1.6.7 with the HF+ ...and all seamed well...but..

However today whilst tuning around parts of the FM BCB I'd noted a spur that consistently sticks to 4.8kHz above the 'centre tuning' freq
on many frequencies & varies in intensity from anywhere from approx 5-7db to 25dB (and sometimes briefly disappears) above the noise floor depending on freq.
Spur presence is enough to disrupt dxing when signal stuck in centre of spectrum display.
I had not noted this until today for some reason (though haven't spent a lot of time tuning past two weeks).

After many hours investigating today I've made these edited observations on the FM BCB (at this stage.)

1. Spur only noted with BW's 192 to 768kHz (not noted with BWs 96kHz & below)
2. Can avoid the problematic spur with sticky tuning & free tuning modes as opposed to the centre tuning option where part of the problem resides.

The spur is still there at reduced level even with 50ohm dummy loads on both VHF & HF antenna inputs & noted on both HF & VHF freqs with firmware 1.6.7

Noted the problem disappeared when going back to firmware 1.6.3. But returned when re-flashing with firmware 1.6.7 & 1.7.0
Given Youssef's extensive AGC work  with Firmware 1.6.7  & given my present HF+ spur symptoms I wanted to introduce some loss into the antenna feedline to see if it would help the situation.
No attentuators available so tried my homebrew 2 stage variable FM BCB preselector which I could use to also introduce attenuation.
This enabled me to attenuate the desired BCB signal a little, but was enough to cause the spur to disappear below the noise floor.

So some questions, comments & decisions now arise.

1. Youssef are you aware of this problem?
2. Can I be presumptive and guess that your future development work with addon preselector for HF+ (and maybe software) might assist with removal of the spur (hopefully not just HF, but also VHF?)
3. Whilst the new AGC, gain & attention controls with recent SDR# releases assist with reception on HF & MW etc, they do nothing for FM BCB reception, or to rephrase; there are no
software gain, attenuation controls for FM BCB or VHF with the HF+ on SDR#. So can VHF (FM BCB) gain/attenuation controls be considered for future with SDR# (if not already on your planning development list)
if it fixes the problem without need for external hardware?

Hope the above is of interest.
Would welcome any advice as I decide whether or not to go back to firmware 1.6.3 - there's pros & cons with that choice re performance.
Advice & plans by yourself Youssef at this stage would be most helpful.

(Tried to keep message short, but wanted it to be helpful to others.)

Cheers

Ian

Re: ADSBSpy Output format

Andrew Jones
 

Okay, so -v and send to a file whilst also asking airspy_adsb to connect to another service does not work. I can gather the raw data if I don't add the -c option when I run airspy_adsb, but then I can't actually look at whats going on to spot any of the oddities. I am just wondering if I could set up a simple logging script to act as another service to connect to and then hoover up the data that way.

Re: ADSBSpy Output format

Andrew Jones
 

Thanks for that.

Yes, I thought some form of log would be good to do. I think I need to capture the raw decodes at the same time as observing whats going on so I can identify a few of the spurious tracks being created to give some indicators of what icao codes they result in to give some markers to look for in the data.

I run under linux but thats not a problem. What exact format raw data would be best? I just need to figure the best way to log the data and send it through the rest of the applications in parallel without compromising the validity of the logging of the raw data (by logging it after it has been filtered or converted to a different format, etc...). Perhaps its as simple as -v and pipe to a file at the same time as letting airspy_adsb connect to other things. I'll have a play...

Andy

Re: ADSBSpy Output format

jdow
 

Is this a confusion of 11 hex and 11 decimal?
{^_^}

On 20180420 15:38, Support@... wrote:
Andy,
No need for apologies - it's not a subject that many people look at too closely. It would be helpful if you could log a short (say 1 minute) output from airspy_adsb showing the issue (with FEC = 1) and EMAIL it to me. I think the command line is "airspy_adsb -v -f 1 > logfile.txt" (assuming you're under windows)
DF11 only contains aircraft identity - the ICAO code. DF17/18 contain ICAO plus one of callsign, lat/lon, speed, heading etc. All of these are covered by CRC which should be able to FEC 100% for single bit errors. It's these frames which populate the whitelist. If you are seeing 'random' ICAO codes pop up in the Dump1090 list, then somehow a DF11/17/18 is escaping the FEC checking and polluting the whitelist.
DF0/4/16/20 contain Altitude, and DF5/21 contain Squawk. These are not CRC checked, so rely on the whitelist being populated correctly.
The DF specification Is "icao annex 10 volume iv". If you're interested in how it all works Google that and download the pdf (It's free).

Re: ADSBSpy Output format

Support@...
 

Andy,
No need for apologies - it's not a subject that many people look at too closely. It would be helpful if you could log a short (say 1 minute) output from airspy_adsb showing the issue (with FEC = 1) and EMAIL it to me. I think the command line is "airspy_adsb -v -f 1 > logfile.txt" (assuming you're under windows)
 
DF11 only contains aircraft identity - the ICAO code. DF17/18 contain ICAO plus one of callsign, lat/lon, speed, heading etc. All of these are covered by CRC which should be able to FEC 100% for single bit errors. It's these frames which populate the whitelist. If you are seeing 'random' ICAO codes pop up in the Dump1090 list, then somehow a DF11/17/18 is escaping the FEC checking and polluting the whitelist.

DF0/4/16/20 contain Altitude, and DF5/21 contain Squawk. These are not CRC checked, so rely on the whitelist being populated correctly.

The DF specification Is "icao annex 10 volume iv". If you're interested in how it all works Google that and download the pdf (It's free).


Re: Truncated RT data for RDS within SDR#

jdow
 

When something you are working on just isn't coming together making the rds stuff a little more forgiving is not all that hard to do. The two fills I have seen so far are spaces and nulls. So testing for either will probably handle almost all cases. However, I can believe an encoder exists that has either random or previous frame's data in it. Since there is no way to reliability check such frames I'd toss them. I suspect the problem is really old equipment that was shoved out the door while the specification was not quite finished. And aside from blown transmitter tubes and the like getting a radio station to spend money on something the owner does not consider vitally important just does not happen. "It ain't broke so why fix it?" (And REAL radios get it right - somehow. {^_-})

{^_^}

On 20180420 04:47, prog wrote:
On Fri, Apr 20, 2018 at 04:38 am, Ian DXer wrote:
Thanks for putting the job on your list (low priority) Youssef - totally
understand your attention to matters that drive your interest.
Hope your to-do list isn't as deep as mine ;-)
However to put the problem in perspective at my QTH; I did a quick survey of
most FM RDS stations with RT tonight.
Test sample total: 13
Stations with RT displaying correctly: 6 (consisting of low budget community
stations & high powered commercial)
* Stations with RT displaying incorrectly: 7 (consisting of high powered
(HP) commercial, HP community & all HP National broadcasters)
Not a good result....
So for the interest of Simon (and maybe some others) it looks like the
situation here is similar to the problems within the US with poorly
implemented RDS standard with many stations.
Did a count of numbers of displayed characters in instances where truncation
occurs & yes is a multiple of 4. Thanks Joanne.
Regards
Ian
I prefer being honest. Changing that part of the code requires a lot of testing and testing requires a lot time. RDS and RBDS seem to be two different things. I'm not using a COTS library to handle them like it is the case with other software. It will have to wait until I'm either really bored of what I'm currently doing or reached the performance objectives.
Hope your to-do list isn't as deep as mine ;-)
Unless you are also managing tens of Corporate and Investment Banking projects, there's little chance you beat me at this.

Re: Re :[airspy] New firmware update for the Airspy HF+ Rev 1.7.0 #airspyhfplus #firmware

camelman123
 

Hi Prog,
Can you tell us more about the upcoming pre-selector and minor enhancements you've mentionned.
Thanks in advance
Rick

Re :[airspy] New firmware update for the Airspy HF+ Rev 1.7.0 #airspyhfplus #firmware

Abdou DAROUF
 

 Hello Prog, 
Can i update the new firmware without use the old versions 

Le ven., avr. 20, 2018 à 11:02, prog
<info@...> a écrit :
This update mainly adds support for the upcoming pre-selector addon on top of other minor enhancements.

The new firmware package can be downloaded from https://airspy.com/airspy-hf-plus/#firmware-updates

Note also that libairspyhf was updated as well and added to the SDR# r1664 package.

Q: Are you ready for the next SAQ transmission? Report your RX here.

Re: ADSBSpy Output format

Andrew Jones
 

All,

First up, my most sincere apologies, I now believe that my assertion regarding a spurious aircraft report that included a location to be wrong. Between the explanations given on this topic and my much more thorough observation of things this evening I think I must have seen/remembered what I thought to be the case, and actually probably untrue (a mixture of confirmation bias, laziness and being in a bit of a rush). Again, apologies if anybody has acted specifically upon this bit of information (which prog may well have done in the release of v1.19... sorry).

So, having run v1.19 for a few hours now and had plenty going on, here is what I can observe:

Certainly by looking in VRS I can see spurious aircraft reports popping up when FEC is on, compared to when FEC is off (where none are obvious).
The aircraft reports I believe to be spurious (you get a small number of messages and ICAO code that can't be looked up anywhere or is for an unassigned country and then nothing) DO NOT ever seem to include lat/long data (hence my apology above)
A number of aircraft reports I believe are likely to be spurious DO occasionally include height reports.
The aircraft appearing in the track list VRS always have a message count of at least 2 messages before they are shown in the list of tracked aircraft (Again, another reason why I doubt my previous statement). This also makes me think that, despite dump1090 reporting a large number of 'single message tracks', these are being filtered out somewhere in the chain up to reporting things to the user in VRS (this could be happening in dump1090 or VRS, I don't know, Its also perhaps not really relevant to the airspy side of things).
Turning FEC on in v1.19 has pretty much the same effect on dump1090 reporting a large proportion of new tracks being created from a single message as happened in v1.18.

This leads me to conclude that there is no real evidence that position messages (DF11? - I am still learning!) are being incorrectly decoded with FEC enabled.
It is perhaps still feasible that there is a larger proportion of badly decoded messages of other formats that are not protected by secure CRC that are being decoded and this would explain what I now think I am observing (and this, in turn, may simply be back to what prog and I discussed about greater sensitivity = more messages in and around the noise = more risk of errors getting in). However, if I have understood correctly regarding the white list, this is supposed to stop this - by only letting messages recovered by FEC processing through where it matches a known ICAO code from the whitelist? If that's true, I'm not sure I should be observing these reports that I believe to be spurious unless they are not being recovered by FEC? But they must be being recovered by FEC as this never happens if I turn FEC off. Or have I misunderstood something?

Once again, sorry for the misinformation previously stated, this was a bit silly of me!

Andy



Re: ADSBSpy Output format

Support@...
 

RE: Yes and it does not work always so well because ICAO allocation is far from random and in particular there is a prefix by country. Around  LFPG , for example, you wil have lots of Air France plane that  have close icao (in term of hamming distance). So with one bit of error the probability to be in the white list is not so low ...

True, that's what results in a corrupted DF4/5/20/21 being allocated to the wrong plane but it shouldn't result in the whitelist becoming contaminated with bad ICAO's.

Andrew's observation of spurious tracks seems to indicate bad ICAO's are being entered into the whitelist - otherwise how can he be seeing Altitude from a DF4/20 ?

Re: Truncated RT data for RDS within SDR#

prog
 

On Fri, Apr 20, 2018 at 04:38 am, Ian DXer wrote:
Thanks for putting the job on your list (low priority) Youssef - totally understand your attention to matters that drive your interest.
Hope your to-do list isn't as deep as mine ;-)

However to put the problem in perspective at my QTH; I did a quick survey of most FM RDS stations with RT tonight.
Test sample total: 13
Stations with RT displaying correctly: 6 (consisting of low budget community stations & high powered commercial)
* Stations with RT displaying incorrectly: 7 (consisting of high powered (HP) commercial, HP community & all HP National broadcasters)
Not a good result....

So for the interest of Simon (and maybe some others) it looks like the situation here is similar to the problems within the US with poorly
implemented RDS standard with many stations.

Did a count of numbers of displayed characters in instances where truncation occurs & yes is a multiple of 4. Thanks Joanne.

Regards

Ian
I prefer being honest. Changing that part of the code requires a lot of testing and testing requires a lot time. RDS and RBDS seem to be two different things. I'm not using a COTS library to handle them like it is the case with other software. It will have to wait until I'm either really bored of what I'm currently doing or reached the performance objectives.

Hope your to-do list isn't as deep as mine ;-)
Unless you are also managing tens of Corporate and Investment Banking projects, there's little chance you beat me at this.

Re: Truncated RT data for RDS within SDR#

Ian DXer
 

Thanks for putting the job on your list (low priority) Youssef - totally understand your attention to matters that drive your interest.
Hope your to-do list isn't as deep as mine ;-)

However to put the problem in perspective at my QTH; I did a quick survey of most FM RDS stations with RT tonight.
Test sample total: 13
Stations with RT displaying correctly: 6 (consisting of low budget community stations & high powered commercial)
* Stations with RT displaying incorrectly: 7 (consisting of high powered (HP) commercial, HP community & all HP National broadcasters)
Not a good result....

So for the interest of Simon (and maybe some others) it looks like the situation here is similar to the problems within the US with poorly
implemented RDS standard with many stations.

Did a count of numbers of displayed characters in instances where truncation occurs & yes is a multiple of 4. Thanks Joanne.

Regards

Ian

Re: ADSBSpy Output format

TLeconte
 

On Fri, Apr 20, 2018 at 02:08 am, <Support@...> wrote:
To solve this problem, most software keeps a whitelist (the opposite of a blacklist) of aircraft ICAO's which have been observed recently with CRC secure DF11/17/18/19. Then, when one of the other DF's arrives, it can check if the ICAO decoded from the insecure frame is in the whitelist. If it is, then the message is probably Ok. If not, then the message is discarded.
Yes and it does not work always so well because ICAO allocation is far from random and in particular there is a prefix by country.
Around  LFPG , for example, you wil have lots of Air France plane that  have close icao (in term of hamming distance). So with one bit of error the probability to be in the white list is not so low ...

New firmware update for the Airspy HF+ Rev 1.7.0 #airspyhfplus #firmware

prog
 

This update mainly adds support for the upcoming pre-selector addon on top of other minor enhancements.

The new firmware package can be downloaded from https://airspy.com/airspy-hf-plus/#firmware-updates

Note also that libairspyhf was updated as well and added to the SDR# r1664 package.

Q: Are you ready for the next SAQ transmission? Report your RX here.

Re: Truncated RT data for RDS within SDR#

prog
 

On Fri, Apr 20, 2018 at 01:40 am, Ian DXer wrote:
Would be more satisfying for Youssef to program a fix rather than be disappointed with often lack of action (after reporting) to broadcast stations.
No. I optimize my own satisfaction quite differently. There are many other hot topics that require more attention and are much, much more rewarding.
Adding code to work around cheap FM transmitters is something I'd put in my todo list with a low priority, though.

Re: ADSBSpy Output format

prog
 

On Fri, Apr 20, 2018 at 02:08 am, <Support@...> wrote:
re: Going back though, I think what prog and I discussed earlier seems to be a plausible explanation for this.

Maybe, Maybe not. The reason it's important to know if it's just a message count of '1' is as follows....

The only messages that are protected by secure CRC are DF11*, DF17, DF18 and DF19. So barring any massive corruption (> 2 bits) these should be decoded properly. The ICAO of aircraft can then be safely extracted from these frames.  With all the other DF's, the decoded CRC is not zero - it's the ICAO of the aircraft that transmitted it. Therefore these other frames cannot be safely decoded in isolation because the software can't know if that aircraft is actually present, or if it's a corrupted frame that produces a different CRC, and hence a different aircraft.

To solve this problem, most software keeps a whitelist (the opposite of a blacklist) of aircraft ICAO's which have been observed recently with CRC secure DF11/17/18/19. Then, when one of the other DF's arrives, it can check if the ICAO decoded from the insecure frame is in the whitelist. If it is, then the message is probably Ok. If not, then the message is discarded. It is vital that the whitelist is kept pure, otherwise spurious decodes of other DF's can and do get through.

What all this means is that you should only ever see messages for ICAO's for which DF11/17/18/19's have been successfully decoded. Altitude or Squawk's come from DF4/DF5/DF20/DF21, so if you see these randomly then there must have also been a previous 'valid' decode of a DF11/17/18/19 as well, so at least 2 frames must have been received using that ICAO. Lat/Lon (usually) requires 3 frames. Therefore it's difficult to understand how a single (corrupt or not) frame can result in an ICAO, 450kts and FL300. That required at least 3 frames I think. 

If two aircraft with similar ICAO's (one bit different) are both being received at the same time then it is possible for corrupted messages from one aircraft to be wrongly allocated to the other. I doubt much can be done about that without much stronger/stricter decoding of the RF signal possibly including RSSI for each bit in the message, but that will incur significant extra processor workload. Sounds like Youssef is on the case though.

*Providing II/SI == 0;
There's a new update of the decoder with tighter filtering of DF11's. Check it out.

Re: ADSBSpy Output format

Support@...
 

re: Going back though, I think what prog and I discussed earlier seems to be a plausible explanation for this.

Maybe, Maybe not. The reason it's important to know if it's just a message count of '1' is as follows....

The only messages that are protected by secure CRC are DF11*, DF17, DF18 and DF19. So barring any massive corruption (> 2 bits) these should be decoded properly. The ICAO of aircraft can then be safely extracted from these frames.  With all the other DF's, the decoded CRC is not zero - it's the ICAO of the aircraft that transmitted it. Therefore these other frames cannot be safely decoded in isolation because the software can't know if that aircraft is actually present, or if it's a corrupted frame that produces a different CRC, and hence a different aircraft.

To solve this problem, most software keeps a whitelist (the opposite of a blacklist) of aircraft ICAO's which have been observed recently with CRC secure DF11/17/18/19. Then, when one of the other DF's arrives, it can check if the ICAO decoded from the insecure frame is in the whitelist. If it is, then the message is probably Ok. If not, then the message is discarded. It is vital that the whitelist is kept pure, otherwise spurious decodes of other DF's can and do get through.

What all this means is that you should only ever see messages for ICAO's for which DF11/17/18/19's have been successfully decoded. Altitude or Squawk's come from DF4/DF5/DF20/DF21, so if you see these randomly then there must have also been a previous 'valid' decode of a DF11/17/18/19 as well, so at least 2 frames must have been received using that ICAO. Lat/Lon (usually) requires 3 frames. Therefore it's difficult to understand how a single (corrupt or not) frame can result in an ICAO, 450kts and FL300. That required at least 3 frames I think. 

If two aircraft with similar ICAO's (one bit different) are both being received at the same time then it is possible for corrupted messages from one aircraft to be wrongly allocated to the other. I doubt much can be done about that without much stronger/stricter decoding of the RF signal possibly including RSSI for each bit in the message, but that will incur significant extra processor workload. Sounds like Youssef is on the case though.

*Providing II/SI == 0;

Re: Truncated RT data for RDS within SDR#

Ian DXer
 

Hi Joanne,

Thanks 4 feedback. This sounds exactly like the cause of the problem.
re: 'Some equipment does not terminate the left over bytes in a four byte text frame'
I haven't proven this, not sure yet how to test this with my available resources atm without further investigation.
I'm sure i could record a 16bit PCM baseband signal for Youssef (if required) of a sample effected station.

Joanne you mention: "..this is a lack of defensive programming I've brought to Youssef's attention several times now"
This reminds me of my Digital Electronics 2 'project' from my college years where students were required to
design a 'programmable security lock' using TTL/CMOS logic chips & other electronics.
I created most of the design whilst quietly lying in bed at 4am.
Unfortunately at project completion time I was marked down from 95% to 90% because my teacher found that my design had not catered for
users performing a mis-operation outside the design specification. I had actually thought about this error scenario occurring (and proved in testing) earlier
and knew it would
only take two logic gates to prevent this error, but foolishly decided during design & construction phase
it was just extra work, time & money to implement during a busy study schedule.
Moral of the story:- design engineers should cater for (all possible) error scenarios occurring during initial design or eg. patch/correct software where
later informed by users & in this example where broadcast engineers aren't feeding data as per RDS design specification, so end (SDR#) users aren't effected.
Just my personal learned experience ;-)

If it is indeed an easy fix for Youssef I'd be very appreciative of his investment of time to correct software for us both & no doubt more users.

Not sure how many of my local radio stations in Australia have this RDS RT problem (might do a count later tonight), not all use RT.
Certainly several stations are slack with correcting 'clock time' with their RDS despite me reporting problem.
Would be more satisfying for Youssef to program a fix rather than be disappointed with often lack of action (after reporting) to broadcast stations.

Thanks for feedback Simon. 

Ian




Re: ADSBSpy Output format

Andrew Jones
 

So, first of all, to get this out of the way, my methods of measuring 'performance' are not perfect, either using my eyeballs to watch whats going on through VRS, or using somebody else's code in the way of the statistics measurements that come from dump1090.

My setup comprises a co-linear antenna feeding a mini-circuits amplifier and then an EPCOS SAW filter into my airspy mini. I am located just north of London, so around this time of year I can expect to receive messages from 200-250 aircraft every minute throughout the day. Flightaware shows something like 800k-900k messages per day.

The 'single message tracks' comes from one of the field outputs from dump1090 stats. I do not know how this is calculated. However, it does correlate well with what I can see with my eyeballs in VRS. With my eyeballs I can see the following things happen: tracks are created that that have ICAO codes that look to be odd (looking up on several online databases they do not match any known active aircraft), they also tend to only ever get one or two messages and then disappear. This happened with the old airspy_adsb and -f 1 but it was not very often. With the new airspy_Adsb and -f 1 I can usually see 4/5 of these types of track almost constantly (and not the same ICAO code - just fairly random, indicating they are just bad decodes) my flightaware stats also show a massive jump in 'other' classified reports when I use the new version with -f 1. The other things that I can see hinting at bad decodes is things like call signs being given to the wrong ICAO code and mis-reported altitudes/speeds (last night there was a Robinson R44 doing ~450kts at 30kft nearby which I thought was a bit funny, turned out it was just in front of a descending airbus out to my Northern max range, looking back to the R44 - it was only a single message again - therefore leading me to believe it was simply an ICAO code decode error and was actually the airbus - I must admit I did not record the icao codes to compare if there was a single bit error possibility). All of this goes away the moment I set -f 0 and restart.

Going back though, I think what prog and I discussed earlier seems to be a plausible explanation for this.

Andy 

Re: Truncated RT data for RDS within SDR#

Simon Brown
 

There are some very poor FM Stereo/RDS transmissions in the US, all the exceptions to the rules can be found there.

Round here the BBC appears to be excellent, as do all national broadcasts I can pick up (France always, all of Europe and N. Africa when there's Sporadic-E).

Simon Brown, G4ELI
www.sdr-radio.com

-----Original Message-----
From: main@airspy.groups.io <main@airspy.groups.io> On Behalf Of jdow
Sent: 20 April 2018 01:41
To: main@airspy.groups.io
Subject: Re: [airspy] Truncated RT data for RDS within SDR#

Does the length of the text modulo four suggest anything to you? If so this is a lack of defensive programming I've brought to Youssef's attention several times now. Some equipment does not terminate the left over bytes in a four byte text frame according to specification. So the frame gets marked as an error and is never displayed. The fix is very easy to make. (I have a very old version of SDRSharp from when the source was available but not "open" in the conventional sense. I made the change and use it for the radio I keep running 24/7 on the local listener supported classical music station here in Los Angeles. (Pumps hand for KUSC.)