Topics

Issue Importing Duplicate QSos

fjk5146
 

I have been making a number of FT8 QSOs using WSJT-X.  I use JTAlert to seamlessly bring the QSOs into DXKeeper.  However, I had several QSOs that had not been imported into to DXKeeper because they preceded my installation of JTAlert.  I decided to save an ADIF file from WSJT-X and import it into DXKeeper.  I knew there were duplicate QSOs but I assumed they would be identified and not imported.  Unfortunately that is not what happened.  All of the QSOs were imported, including those that had already been logged in DXKeeper using JTAlert.  I examined the QSOs and found them to have the exact same callsign, time and date information.  However the differences appeared to be the information added by DXKeeper such as SFI and other information looked up and inserted by DXKeeper.

I had to manually remove the duplicates because I was not sure how to write an SQL script to identify each of the duplicates.  

Is there something I could have done to avoid this problem other than manually editing the ADIF file prior to the import?

Thanks,

Fred, KC9QQ

Gilbert Baron W0MN
 

From Wiki

If you are importing into a populated log and wish to prevent the importing of duplicate QSOs, check the Check Duplicates on Import box.

  • Specify the range (in minutes) that defines a duplicate QSO
    • 0 means an imported QSO must exactly match a logged QSO's begin date/time  to be considered a duplicate
    • a number larger than 0 specifies a range before and after the imported QSO's begin date/time; if a matching logged QSO's begin date/time falls in this range, the imported QSO will be considered a duplicate
  • If the  Consider primary and secondary administrative subdivisions box is checked,  an imported QSO is only rejected if its primary and secondary administrative subdivisions also match
  • If the  Consider gridsquares box is checked,  an imported QSO is only rejected if its gridsquare also matches the existing QSO's Grid 1 item

 

Perhaps you did not match this?

 

 

Outlook Laptop Gil W0MN

Hierro Candente Batir de Repente

44.08226N 92.51265 W en34rb

 

 

From: DXLab@groups.io <DXLab@groups.io> On Behalf Of fjk5146
Sent: Wednesday, November 13, 2019 15:08
To: DXLab@groups.io
Subject: [DXLab] Issue Importing Duplicate QSos

 

I have been making a number of FT8 QSOs using WSJT-X.  I use JTAlert to seamlessly bring the QSOs into DXKeeper.  However, I had several QSOs that had not been imported into to DXKeeper because they preceded my installation of JTAlert.  I decided to save an ADIF file from WSJT-X and import it into DXKeeper.  I knew there were duplicate QSOs but I assumed they would be identified and not imported.  Unfortunately that is not what happened.  All of the QSOs were imported, including those that had already been logged in DXKeeper using JTAlert.  I examined the QSOs and found them to have the exact same callsign, time and date information.  However the differences appeared to be the information added by DXKeeper such as SFI and other information looked up and inserted by DXKeeper.

I had to manually remove the duplicates because I was not sure how to write an SQL script to identify each of the duplicates.  

Is there something I could have done to avoid this problem other than manually editing the ADIF file prior to the import?

Thanks,

Fred, KC9QQ


--

W0MN EN34rb 44.08226 N 92.51265 W

Hierro candente, batir de repente

HP Laptop

Gilbert Baron W0MN
 

I may be wrong here but I my opinion adding to the log in 2 ways is not a good idea. I think the best way to do this is to let your program in use create its own file and then after you are done to import that file. This would be a failsafe way to avoid duplicates.  

 

 

Outlook Laptop Gil W0MN

Hierro Candente Batir de Repente

44.08226N 92.51265 W en34rb

 

 

From: DXLab@groups.io <DXLab@groups.io> On Behalf Of Gilbert Baron W0MN via Groups.Io
Sent: Wednesday, November 13, 2019 15:36
To: DXLab@groups.io
Subject: Re: [DXLab] Issue Importing Duplicate QSos

 

From Wiki

If you are importing into a populated log and wish to prevent the importing of duplicate QSOs, check the Check Duplicates on Import box.

  • Specify the range (in minutes) that defines a duplicate QSO
    • 0 means an imported QSO must exactly match a logged QSO's begin date/time  to be considered a duplicate
    • a number larger than 0 specifies a range before and after the imported QSO's begin date/time; if a matching logged QSO's begin date/time falls in this range, the imported QSO will be considered a duplicate
  • If the  Consider primary and secondary administrative subdivisions box is checked,  an imported QSO is only rejected if its primary and secondary administrative subdivisions also match
  • If the  Consider gridsquares box is checked,  an imported QSO is only rejected if its gridsquare also matches the existing QSO's Grid 1 item

 

Perhaps you did not match this?

 

 

Outlook Laptop Gil W0MN

Hierro Candente Batir de Repente

44.08226N 92.51265 W en34rb

 

 

From: DXLab@groups.io <DXLab@groups.io> On Behalf Of fjk5146
Sent: Wednesday, November 13, 2019 15:08
To: DXLab@groups.io
Subject: [DXLab] Issue Importing Duplicate QSos

 

I have been making a number of FT8 QSOs using WSJT-X.  I use JTAlert to seamlessly bring the QSOs into DXKeeper.  However, I had several QSOs that had not been imported into to DXKeeper because they preceded my installation of JTAlert.  I decided to save an ADIF file from WSJT-X and import it into DXKeeper.  I knew there were duplicate QSOs but I assumed they would be identified and not imported.  Unfortunately that is not what happened.  All of the QSOs were imported, including those that had already been logged in DXKeeper using JTAlert.  I examined the QSOs and found them to have the exact same callsign, time and date information.  However the differences appeared to be the information added by DXKeeper such as SFI and other information looked up and inserted by DXKeeper.

I had to manually remove the duplicates because I was not sure how to write an SQL script to identify each of the duplicates.  

Is there something I could have done to avoid this problem other than manually editing the ADIF file prior to the import?

Thanks,

Fred, KC9QQ


--

W0MN EN34rb 44.08226 N 92.51265 W

Hierro candente, batir de repente

HP Laptop


--

W0MN EN34rb 44.08226 N 92.51265 W

Hierro candente, batir de repente

HP Laptop

Dave AA6YQ
 

+ AA6YQ comments below

I have been making a number of FT8 QSOs using WSJT-X. I use JTAlert to seamlessly bring the QSOs into DXKeeper. However, I had several QSOs that had not been imported into to DXKeeper because they preceded my installation of JTAlert. I decided to save an ADIF file from WSJT-X and import it into DXKeeper. I knew there were duplicate QSOs but I assumed they would be identified and not imported.

+ DXLab applications are accompanied by comprehensive reference and task-oriented documentation. Referring to either one would have informed you that by default, the "Import QSOs" operation will import all QSOs unless you check the "Check duplicates on import" box in the "Duplicate checking" panel on the Main window's "Import QSOs" tab.


Unfortunately that is not what happened. All of the QSOs were imported, including those that had already been logged in DXKeeper using JTAlert. I examined the QSOs and found them to have the exact same callsign, time and date information. However the differences appeared to be the information added by DXKeeper such as SFI and other information looked up and inserted by DXKeeper.

+ Those difference would not have impeded the import duplicate checker from ignoring duplicate QSOs, had you enabled it.


I had to manually remove the duplicates because I was not sure how to write an SQL script to identify each of the duplicates.

+ Both the reference documentation and the task-oriented documentation described DXKeeper's mechanism for automatically removing duplicates; for example,

<https://www.dxlabsuite.com/dxlabwiki/RemovingDuplicates>


Is there something I could have done to avoid this problem other than manually editing the ADIF file prior to the import?

+ Yes: familiarize yourself with DXKeeper before attempting to employ it by reviewing its documentation. Had you first reviewed

<https://www.dxlabsuite.com/dxlabwiki/QSOImport>

+ for example, step 3a would have corrected your incorrect assumption about the handling of duplicate QSOs.

73,

Dave, AA6YQ

fjk5146
 

Thanks for the replies.  However, I did have the check box checked to remove duplicates within +/- 2 minutes.  I frequently import logs from N1MM after contests and always check remove duplicates.  That's why I was surprised they were not rejected when I imported the ADIF file created by WSJT-X.  The only reason I imported the ADIF file was because it contained about 30 QSOs I had made before I started using JTAlert.  Lesson learned.

Fred, KC9QQ

Dave AA6YQ
 

+ AA6YQ comments below

On Wed, Nov 13, 2019 at 05:38 PM, fjk5146 wrote:

Thanks for the replies.  However, I did have the check box checked to remove duplicates within +/- 2 minutes.  I frequently import logs from N1MM after contests and always check remove duplicates.  That's why I was surprised they were not rejected when I imported the ADIF file created by WSJT-X.  The only reason I imported the ADIF file was because it contained about 30 QSOs I had made before I started using JTAlert.  Lesson learned.
+ Please place your log file (probably KC9QQ.mdb) in a zip archive. Attach the zip archive to an email message along with the ADIF file you imported. If the email message could identify one or two of the duplicate QSOs, that would be helpful. Then send the email message to me via

aa6yq (at) ambersoft.com

          73,

               Dave, AA6YQ

fjk5146
 

On Wed, Nov 13, 2019 at 05:47 PM, Dave AA6YQ wrote:
aa6yq (at) ambersoft.com
Thanks Dave,

I have sent you an email with the two files.

Fred, KC9QQ

Dave AA6YQ
 

I received your email message, Fred, but only one file was attached: wsjtx_log.adi

To understand what happened, I also need your log file (please place this in a zip archive and attach the zip archive to an email message), and the callsign of one of the duplicate QSOs.

73,

Dave. AA6YQ

-----Original Message-----
From: DXLab@groups.io [mailto:DXLab@groups.io] On Behalf Of fjk5146
Sent: Thursday, November 14, 2019 8:55 AM
To: DXLab@groups.io
Subject: Re: [DXLab] Issue Importing Duplicate QSos

On Wed, Nov 13, 2019 at 05:47 PM, Dave AA6YQ wrote:


aa6yq (at) ambersoft.com

Thanks Dave,

I have sent you an email with the two files.

Fred, KC9QQ


<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> Virus-free. www.avg.com <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>

Dave AA6YQ
 

After receiving and analyzing the log and ADIF file received From Fred KC9QQ, I found that the logged and imported QSOs do not have identical start times: the logged QSOs have their the seconds component of their start times set to 00, whereas the QSOs being imported specify their start times to the second.

Importing the ADIF file with duplicate checking enabled and the "range" set to +/- 1 minute correctly rejects the duplicate QSOs. Importing the ADIF file with duplicate checking enabled and the "range" unspecified (meaning exact time match) did not reject the duplicate QSOs.

DXKeeper's "Import QSOs" panel's "Duplicate Checking" mechanism is working as expected.

73,

Dave, AA6YQ