Topics

Dupes...

Joe
 

My setup is a Flex 6600M/SSDR/WSJTX/JTALert/DXCommander/DXKeeper on a relatively fast 6 core i7 3.1 Ghz desktop with lots of RAM, and SSD.

Yesterday, I was using the machine remotely from another room of the house (had to watch some football!) using TeamViewer for the remote link.  When I checked later, there was a block of about 60 QSOs that were missing from the DXKeeper log.  No signs of anything wrong on the screen, and I always saw the prompt for the log entry and the Success alert that follows.  So that is issue number one.

Being the resourceful person that I am, I imported the current WSJTX log file (standard ADIF, Check duplicates on import +/- 0. minutes, Query Callbook for missing items),trusting the dupe filter to work and expecting the import to skip everything that might be in the file except for the missing QSOs.  After all, the entries in DXKeeper came originally from WSJTX for the most part. (I have been using FT8/4 almost exclusively since returning to radio about a year ago.)  It didn't work.  Still no missing QSOs, but LOTS of dupes.  Not sure what was there before.  Since I sync LoTW and the log on each QSO, I checked LoTW for the missing QSOs...not there either.  Getting desperate, I manually uploaded (TQSL) the WSJTX log file, checked that the missing QSOs got there, and then sync'd DXKeeper with LoTW.  Good news...the missing QSOs showed up...bad news....even more dupes!  I think I now have 1500-2000 dupes in the log.  And LotW has about 5000, but most of that may have come from initial attempts to upload an old Logic log from the nineties.

I have read the Wiki info on the dupe filter.  I ran it and it found only 6 dupes!  Looking back thru the log I see lots of duplicate entries now that seem exactly the same, based on the information that I can see.  Judging by the log size now, I must have 1500-2000 dupes.  So...the big question is...does anyone know how to really remove dupes from DXKeeper??  Theoretically, I guess I could export a log from DXKeeper or from LotW, sort and process it with some Python code and then re-import it, but this seems like such a common problem, I am wondering how others have tackled this problem???

Dave AA6YQ
 

+ AA6YQ comments below

My setup is a Flex 6600M/SSDR/WSJTX/JTALert/DXCommander/DXKeeper on a relatively fast 6 core i7 3.1 Ghz desktop with lots of RAM, and SSD.

Yesterday, I was using the machine remotely from another room of the house (had to watch some football!) using TeamViewer for the remote link. When I checked later, there was a block of about 60 QSOs that were missing from the DXKeeper log. No signs of anything wrong on the screen, and I always saw the prompt for the log entry and the Success alert that follows. So that is issue number one.

+ In your setup, JT-Alert logs the QSOs to DXKeeper. Since it reported success, I suggest that you ask Laurie VK3AMA for assistance in determining what might have gone wrong.

Being the resourceful person that I am, I imported the current WSJTX log file (standard ADIF, Check duplicates on import +/- 0. minutes, Query Callbook for missing items),trusting the dupe filter to work and expecting the import to skip everything that might be in the file except for the missing QSOs. After all, the entries in DXKeeper came originally from WSJTX for the most part. (I have been using FT8/4 almost exclusively since returning to radio about a year ago.) It didn't work. Still no missing QSOs, but LOTS of dupes. Not sure what was there before. Since I sync LoTW and the log on each QSO, I checked LoTW for the missing QSOs...not there either. Getting desperate I manually uploaded (TQSL) the WSJTX log file, checked that the missing QSOs got there, and then sync'd DXKeeper with LoTW. Good news...the missing QSOs showed up...bad news....even more dupes! I think I now have 1500-2000 dupes in the log. And LotW has about 5000, but most of that may have come from initial attempts to upload an old Logic log from the nineties.

I have read the Wiki info on the dupe filter. I ran it and it found only 6 dupes! Looking back thru the log I see lots of duplicate entries now that seem exactly the same, based on the information that I can see. Judging by the log size now, I must have 1500-2000 dupes. So...the big question is...does anyone know how to really remove dupes from DXKeeper?? Theoretically, I guess I could export a log from DXKeeper or from LotW, sort and process it with some Python code and then re-import it, but this seems like such a common problem, I am wondering how others have tackled this problem???

+ If you don’t yet understand why something didn't work, you're not ready to take corrective action.

+ If you configured DXKeeper's Import duplicate checker with a range of +/- 0, that means that only QSOs whose "QSO Begin" times match to the second will be considered duplicates. Do the QSOs in the ADIF file you imported specify "QSO Begin" times to the second? Did you have any other boxes checked in the "Duplicate checking" panel on DXKeeper's Main window's "Import QSOs" tab?

73,

Dave, AA6YQ

Dave AA6YQ
 

+ AAA6YQ comments below

Ah, Dave, Dave. I really don't understand why you must be so sarcastic.

+ Not one word of my response was sarcastic.

No other options were checked in the Import QSOs tab.

I downloaded the LotW data and sure enough, some of the entries that are otherwise duplicates, have the time truncated to minutes. QSOs submitted via TQSL have the seconds as in the log. Looking at my latest WSJTX log entries in DXKeeper, they show all zeros for the seconds, whereas the WSJTX logs show the seconds in the time fields. Is DXKeeper truncating the seconds field?

+ DXKeeper does not modify the contents of a QSO's "QSO Begin" or "QSO End" items. You can configure it to suppress the display of seconds if you wish.

73,

Dave, AA6YQ