New WAV file overload monitoring feature. ADC overload reporting coming soon. More 'wd...' commands and a help menu


Rob Robinett
 

WD status commands enhanced and with new wd-help command
if you 'git pull', restart WD,  and then execute 'source ~/wsprdaemon/.wd_bash_aliases' you will gain access to these new features and commands.

WD WAV file overload monitoring and viewing
One of those is a new WD feature which logs the minimum and maximum wav file sample values found in the 16 bit samples of each 2 minute long wav file fed to the 'wsprd' decoder.  A size-limited 'wav_stats.log file' contains a list of those results and the new 'wdov'' command finds all the log lines on all those log files which record 1 or more max sample value of 1.0000 and/or min sample value of -1.000.
By occasionally running 'wdov' you will over time learn when/if your wav files are overloading.  If users report finding many such wav overloads, then I will work to address that problem by making the AGG setting give to kwiirecorder a configurable value.

WD ADC overload monitoring and viewing TDB
There is a second ADC overload monitoring feature which I will next address.  ADC overloads are due to excessive RF levels on the Kiwi's SMA input jack and those can only be addressed by introducing pads and/or filters in the RF feed to the Kiwi.  Such overloads are becoming much more common with the much improved propagation we are experiencing, so I know that users need access to that information.  

Until then, you can find a running count of your Kiiw's ADC overloads on among the lines on its /status page, e.g at KFS.  'adc_ov=13543'.


Rob Robinett
 

I fixed and refined the wd-query command which is mostly useful because it will show alll FST4W mode spots, including the -120 ones which are mixed with WSPR-2 by wsprnet.


John
 

I am trying to see if I have KIWI overload errors.
I did a wd -le command and received more than a page of lines. Should I be looking for the word "ERROR"?
I did a couple of CR and see a new line with the same one line of info with a new time stamp.
I tryed to exit the command with a ctrl C but no joy, I could not get out of the command.
I had to cancel the SSH session to exit the command.
I need more training.

John
TI4JWC


Rob Robinett
 

Run 'wdov' to get a list of all the overload events.
Since there are many, you will want to pipe its output to 'less' and/or 'grep'.
It looks like your overloads happen mostly around greyline times:

wsprdaemon@daemonServer:~$ wdov  | grep "KIWI_1/80/"
/dev/shm/wsprdaemon/recording.d/KIWI_1/80/kiwi_ovs.log:                221112_2202.wav:        288        286
/dev/shm/wsprdaemon/recording.d/KIWI_1/80/kiwi_ovs.log:                221112_2204.wav:        528        240
/dev/shm/wsprdaemon/recording.d/KIWI_1/80/kiwi_ovs.log:                221112_2248.wav:        530          2
/dev/shm/wsprdaemon/recording.d/KIWI_1/80/kiwi_ovs.log:                221112_2254.wav:       1143        613
/dev/shm/wsprdaemon/recording.d/KIWI_1/80/kiwi_ovs.log:                221112_2256.wav:       1315        172
/dev/shm/wsprdaemon/recording.d/KIWI_1/80/kiwi_ovs.log:                221112_2302.wav:       1909        594
/dev/shm/wsprdaemon/recording.d/KIWI_1/80/kiwi_ovs.log:                221113_0040.wav:       2567        658
/dev/shm/wsprdaemon/recording.d/KIWI_1/80/kiwi_ovs.log:                221113_0044.wav:       2947        380
/dev/shm/wsprdaemon/recording.d/KIWI_1/80/kiwi_ovs.log:                221113_0046.wav:       3995       1048
/dev/shm/wsprdaemon/recording.d/KIWI_1/80/kiwi_ovs.log:                221113_0048.wav:       4486        491
/dev/shm/wsprdaemon/recording.d/KIWI_1/80/kiwi_ovs.log:                221113_0054.wav:       6310       1824
/dev/shm/wsprdaemon/recording.d/KIWI_1/80/kiwi_ovs.log:                221113_0056.wav:       9220       2910
/dev/shm/wsprdaemon/recording.d/KIWI_1/80/kiwi_ovs.log:                221113_0058.wav:      10977       1757
/dev/shm/wsprdaemon/recording.d/KIWI_1/80/kiwi_ovs.log:                221113_0100.wav:      11827        850
/dev/shm/wsprdaemon/recording.d/KIWI_1/80/kiwi_ovs.log:                221113_0102.wav:      13468       1641
/dev/shm/wsprdaemon/recording.d/KIWI_1/80/kiwi_ovs.log:                221113_0104.wav:      15291       1823
/dev/shm/wsprdaemon/recording.d/KIWI_1/80/kiwi_ovs.log:                221113_0106.wav:      16676       1385
/dev/shm/wsprdaemon/recording.d/KIWI_1/80/kiwi_ovs.log:                221113_0108.wav:      17293        617
/dev/shm/wsprdaemon/recording.d/KIWI_1/80/kiwi_ovs.log:                221113_0110.wav:      17357         64
/dev/shm/wsprdaemon/recording.d/KIWI_1/80/kiwi_ovs.log:                221113_1314.wav:      17710        353
/dev/shm/wsprdaemon/recording.d/KIWI_1/80/kiwi_ovs.log:                221113_1316.wav:      18008        298
/dev/shm/wsprdaemon/recording.d/KIWI_1/80/kiwi_ovs.log:                221113_1324.wav:      19252       1244
/dev/shm/wsprdaemon/recording.d/KIWI_1/80/kiwi_ovs.log:                221113_1416.wav:      19261          9
/dev/shm/wsprdaemon/recording.d/KIWI_1/80/kiwi_ovs.log:                221113_1418.wav:      19553        292
/dev/shm/wsprdaemon/recording.d/KIWI_1/80/kiwi_ovs.log:                221113_1444.wav:      20303        750
/dev/shm/wsprdaemon/recording.d/KIWI_1/80/kiwi_ovs.log:                221113_1456.wav:      20614        311
/dev/shm/wsprdaemon/recording.d/KIWI_1/80/kiwi_ovs.log:                221113_1500.wav:      21015        401
/dev/shm/wsprdaemon/recording.d/KIWI_1/80/kiwi_ovs.log:                221113_1552.wav:      21428        413
/dev/shm/wsprdaemon/recording.d/KIWI_1/80/kiwi_ovs.log:                221113_1624.wav:      21994        566
/dev/shm/wsprdaemon/recording.d/KIWI_1/80/kiwi_ovs.log:                221113_1632.wav:      21998          4
/dev/shm/wsprdaemon/recording.d/KIWI_1/80/kiwi_ovs.log:                221113_1638.wav:      22298        300
/dev/shm/wsprdaemon/recording.d/KIWI_1/80/kiwi_ovs.log:                221113_1644.wav:      22450        152
/dev/shm/wsprdaemon/recording.d/KIWI_1/80/kiwi_ovs.log:                221113_1646.wav:      22866        416
/dev/shm/wsprdaemon/recording.d/KIWI_1/80/kiwi_ovs.log:                221113_1656.wav:      23512        646
/dev/shm/wsprdaemon/recording.d/KIWI_1/80/kiwi_ovs.log:                221113_1722.wav:      24157        645
/dev/shm/wsprdaemon/recording.d/KIWI_1/80/kiwi_ovs.log:                221113_1724.wav:      26002       1845
/dev/shm/wsprdaemon/recording.d/KIWI_1/80/kiwi_ovs.log:                221113_1726.wav:      27138       1136
/dev/shm/wsprdaemon/recording.d/KIWI_1/80/kiwi_ovs.log:                221113_1758.wav:      27578        305
wsprdaemon@daemonServer:~$



On Sun, Nov 13, 2022 at 10:33 AM John via groups.io <n0ure=yahoo.com@groups.io> wrote:
I am trying to see if I have KIWI overload errors.
I did a wd -le command and received more than a page of lines. Should I be looking for the word "ERROR"?
I did a couple of CR and see a new line with the same one line of info with a new time stamp.
I tryed to exit the command with a ctrl C but no joy, I could not get out of the command.
I had to cancel the SSH session to exit the command.
I need more training.

John
TI4JWC



--
Rob Robinett
AI6VN
mobile: +1 650 218 8896


John
 

Thank you Rob.
Now if I could move these numbers onto the Noise graph plots.
J


wayne roth
 
Edited

Rob - Is the OV_TOTAL column in the log file is the count of Kiwi A/D samples that equal A/D full scale 2^14?  Since it's coming from a demodulated stream, I wouldn't think so and would be more likely to be the recorded stream has maxed out. Possibly Kiwirecorder just includes the number of overflows the kiwi code shows at the /status page and thus there is no accumulation of actual clipped samples.  If the latter is the case, then I wonder how many A/D samples it takes to set the overflow flag on the kiwi?

What I'm trying to understand is how catastrophic an occasional clipping is to decoding a WSPR signal.  At my station, I keep a log of the /status page of the Kiwi overflows with the following bash script and occasionally look at it to see when overflows happen:

when=$(date -u +%D' '%T);echo -n $when " " >> /home/wayne/adc94_history.dat
curl -s http://192.168.1.94:8073/status | grep adc_ov  >> /home/wayne/adc94_history.dat

Today I accumulated over 2000 overflows from 1342Z to 1354Z (per your log file).  I dropped the padding after the LNA by 3db which should reduce overflows.  I am wondering what the tradeoff will be between decoding weak signals vs those lost by overflow during those times.

Wayne WA2N
 
 


Rob Robinett
 

Yes, ov_total is reported on the Kiwi's /status page and is I believe the total number of 66 Mspsp samples since Kiwi powerup/reboot where the input signal reached the ADC's full scale value.
At the end of each 2 minute WSPR cycle WD compares the the current count with the count reported at the end of the previous 2 minute cycle,
Our measurements of Kiwi's RF and Audio gain structure led us to choose to configure the Kiwi in a fixed -60 dB AGC so that the 16 bit audio range completely includes the 14 bit ADC range.  With that AGC setting all ADC samples should output more than 2-4 lsb in the wav file.

As a result, I infer that at most sites seeing more than 1-9 overloads in a cycle is cause to reduce the RF gain in the path to the Kiwi's input.   So I expect that you can add even more than 3 dB of padding without impairing your system's SNR. 

On Mon, Nov 14, 2022 at 10:29 AM wayne roth <io@...> wrote:

[Edited Message Follows]

Rob - Is the OV_TOTAL column in the log file is the count of Kiwi A/D samples that equal A/D full scale 2^14?  Since it's coming from a demodulated stream, I wouldn't think so and would be more likely to be the recorded stream has maxed out. Possibly Kiwirecorder just includes the number of overflows the kiwi code shows at the /status page and thus there is no accumulation of actual clipped samples.  If the latter is the case, then I wonder how many A/D samples it takes to set the overflow flag on the kiwi?

What I'm trying to understand is how catastrophic an occasional clipping is to decoding a WSPR signal.  At my station, I keep a log of the /status page of the Kiwi overflows with the following bash script and occasionally look at it to see when overflows happen:

when=$(date -u +%D' '%T);echo -n $when " " >> /home/wayne/adc94_history.dat
curl -s http://192.168.1.94:8073/status | grep adc_ov  >> /home/wayne/adc94_history.dat

Today I accumulated over 2000 overflows from 1342Z to 1354Z (per your log file).  I dropped the padding after the LNA by 3db which should reduce overflows.  I am wondering what the tradeoff will be between decoding weak signals vs those lost by overflow during those times.

Wayne WA2N
 
 



--
Rob Robinett
AI6VN
mobile: +1 650 218 8896


John
 

I have so few unique  80m spots in a day 1 -- 15 max. I am guessing that a pad might not make any change in my count.

John