Topics

QSO validation API suggestion

Gary Hinson
 

I’d like to suggest an LoTW facility (maybe an API?) through which contest adjudicators and awards administrators could submit entrants’ claimed QSOs as ADIF records for validation – either singly or in batches. 

 

That, in turn, would increase pressure for stations to submit their logs to LoTW routinely, benefiting the global community of LoTW users. 

 

To be clear, I envisage the need for controls over access to and use of the third party QSO validation function in order to avoid various risks e.g. people speculatively checking DX logs for matches, or overloading the system by bulk-checking for major contests.  I also appreciate that there would be a cost involved in specifying, developing, testing, using, managing and maintaining the service.  It would not be unreasonable for ARRL to charge third parties for use of the service in order to recover the costs.  If necessary, ARRL might even request contributions up front in order to set the ball rolling with, maybe, a feasibility study. 

 

73
Gary  ZL2iFB

Joe Subich, W4TV
 

*NOT POSSIBLE* LotW derives the callsign and entity from the
uploading station's "certificate". It is not possible for any
other source (including contest sponsors) to obtain access to
that certificate as to do so would eliminate station authentication.

73,

... Joe, W4TV

On 2019-05-23 6:46 PM, Gary Hinson wrote:
I’d like to suggest an LoTW facility (maybe an API?) through which contest adjudicators and awards administrators could submit entrants’ claimed QSOs as ADIF records for validation – either singly or in batches.
That, in turn, would increase pressure for stations to submit their logs to LoTW routinely, benefiting the global community of LoTW users.
To be clear, I envisage the need for controls over access to and use of the third party QSO validation function in order to avoid various risks e.g. people speculatively checking DX logs for matches, or overloading the system by bulk-checking for major contests. I also appreciate that there would be a cost involved in specifying, developing, testing, using, managing and maintaining the service. It would not be unreasonable for ARRL to charge third parties for use of the service in order to recover the costs. If necessary, ARRL might even request contributions up front in order to set the ball rolling with, maybe, a feasibility study.
73
Gary ZL2iFB

Dave AA6YQ
 

Providing Radiosport sponsors with the (retractable) ability to submit to LoTW an ADIF file describing QSOs and receive a confirmed/unconfirmed result for each QSO was discussed by the ARRL's LoTW Committee years ago as a way to further encourage the use of LoTW. My recollection is that it has been implemented; unfortunately, it has never been rolled out.

LoTW stopped moving forward after WAZ support was added in early 2018.

73,

Dave, AA6YQ

-----Original Message-----
From: ARRL-LoTW@groups.io [mailto:ARRL-LoTW@groups.io] On Behalf Of Joe Subich, W4TV
Sent: Thursday, May 23, 2019 8:26 PM
To: ARRL-LoTW@groups.io
Subject: Re: [ARRL-LoTW] QSO validation API suggestion


*NOT POSSIBLE* LotW derives the callsign and entity from the uploading station's "certificate". It is not possible for any other source (including contest sponsors) to obtain access to that certificate as to do so would eliminate station authentication.


73,

... Joe, W4TV


On 2019-05-23 6:46 PM, Gary Hinson wrote:
I d like to suggest an LoTW facility (maybe an API?) through which contest adjudicators and awards administrators could submit entrants claimed QSOs as ADIF records for validation either singly or in batches.



That, in turn, would increase pressure for stations to submit their logs to LoTW routinely, benefiting the global community of LoTW users.



To be clear, I envisage the need for controls over access to and use of the third party QSO validation function in order to avoid various risks e.g. people speculatively checking DX logs for matches, or overloading the system by bulk-checking for major contests. I also appreciate that there would be a cost involved in specifying, developing, testing, using, managing and maintaining the service. It would not be unreasonable for ARRL to charge third parties for use of the service in order to recover the costs. If necessary, ARRL might even request contributions up front in order to set the ball rolling with, maybe, a feasibility study.



73
Gary ZL2iFB







---
This email has been checked for viruses by AVG.
https://www.avg.com

Gary Hinson
 

I don't understand Joe. I don't think what I'm suggesting has anything to do with station authentication or certificates.

If I submit an ADIF QSO record for a 20m QSO between W4TV and P5DX, and if both W4TV and P5DX have previously signed and uploaded their logs containing that QSO to LoTW (so both submissions were authenticated, a QSO match was generated and confirmations are available to both of them), why can't LoTW ALSO confirm to me that the QSO was matched? I presume LoTW maintains a database of the QSOs and matches: I'm simply asking for the ability to do a lookup.

For bonus points, I could sign my request for a lookup and LoTW could sign the response to me but that's a separate issue if fake requests and responses are of concern.

73,
Gary ZL2iFB

-----Original Message-----
From: ARRL-LoTW@groups.io <ARRL-LoTW@groups.io> On Behalf Of Joe Subich, W4TV
Sent: 24 May 2019 12:26
To: ARRL-LoTW@groups.io
Subject: Re: [ARRL-LoTW] QSO validation API suggestion


*NOT POSSIBLE* LotW derives the callsign and entity from the uploading station's "certificate". It is not possible for any other source (including contest sponsors) to obtain access to that certificate as to do so would eliminate station authentication.

73,

... Joe, W4TV


On 2019-05-23 6:46 PM, Gary Hinson wrote:
I’d like to suggest an LoTW facility (maybe an API?) through which contest adjudicators and awards administrators could submit entrants’ claimed QSOs as ADIF records for validation – either singly or in batches.



That, in turn, would increase pressure for stations to submit their logs to LoTW routinely, benefiting the global community of LoTW users.



To be clear, I envisage the need for controls over access to and use of the third party QSO validation function in order to avoid various risks e.g. people speculatively checking DX logs for matches, or overloading the system by bulk-checking for major contests. I also appreciate that there would be a cost involved in specifying, developing, testing, using, managing and maintaining the service. It would not be unreasonable for ARRL to charge third parties for use of the service in order to recover the costs. If necessary, ARRL might even request contributions up front in order to set the ball rolling with, maybe, a feasibility study.



73
Gary ZL2iFB




Gary Hinson
 

Wow! Impressive foresight! Tnx info Dave.

Re the LoTW freeze, is that a resourcing issue, a strategy/policy to let the system stabilize [it's not broken so don't fix it!], or some other reason I wonder?

73,
Gary ZL2iFB

-----Original Message-----
From: ARRL-LoTW@groups.io <ARRL-LoTW@groups.io> On Behalf Of Dave AA6YQ
Sent: 24 May 2019 14:32
To: ARRL-LoTW@groups.io
Subject: Re: [ARRL-LoTW] QSO validation API suggestion

Providing Radiosport sponsors with the (retractable) ability to submit to LoTW an ADIF file describing QSOs and receive a confirmed/unconfirmed result for each QSO was discussed by the ARRL's LoTW Committee years ago as a way to further encourage the use of LoTW. My recollection is that it has been implemented; unfortunately, it has never been rolled out.

LoTW stopped moving forward after WAZ support was added in early 2018.

73,

Dave, AA6YQ



-----Original Message-----
From: ARRL-LoTW@groups.io [mailto:ARRL-LoTW@groups.io] On Behalf Of Joe Subich, W4TV
Sent: Thursday, May 23, 2019 8:26 PM
To: ARRL-LoTW@groups.io
Subject: Re: [ARRL-LoTW] QSO validation API suggestion


*NOT POSSIBLE* LotW derives the callsign and entity from the uploading station's "certificate". It is not possible for any other source (including contest sponsors) to obtain access to that certificate as to do so would eliminate station authentication.


73,

... Joe, W4TV


On 2019-05-23 6:46 PM, Gary Hinson wrote:
I d like to suggest an LoTW facility (maybe an API?) through which contest adjudicators and awards administrators could submit entrants claimed QSOs as ADIF records for validation either singly or in batches.



That, in turn, would increase pressure for stations to submit their logs to LoTW routinely, benefiting the global community of LoTW users.



To be clear, I envisage the need for controls over access to and use of the third party QSO validation function in order to avoid various risks e.g. people speculatively checking DX logs for matches, or overloading the system by bulk-checking for major contests. I also appreciate that there would be a cost involved in specifying, developing, testing, using, managing and maintaining the service. It would not be unreasonable for ARRL to charge third parties for use of the service in order to recover the costs. If necessary, ARRL might even request contributions up front in order to set the ball rolling with, maybe, a feasibility study.



73
Gary ZL2iFB







---
This email has been checked for viruses by AVG.
https://www.avg.com

Dave AA6YQ
 

+ AA6YQ comments below

Wow! Impressive foresight! Tnx info Dave.

Re the LoTW freeze, is that a resourcing issue, a strategy/policy to let the system stabilize [it's not broken so don't fix it!], or some other reason I wonder?

+ Resources were diverted to what the ARRL considers higher-priority tasks - which means that the Board's decision a few years ago to recruit and hire two developers for LoTW was temporary. I don't disagree with the priority of the tasks to which the resources were diverted, but at minimum the development resources taken from LoTW should have been replaced.

+ The timing could not have been worse. The improved reliability and capacity generated by preventing duplicate uploads and re-implementing the LoTW Server followed by the crisp delivery of WAZ support generated significant momentum. The "RadioSport Sponsor query mechanism" and support for several non-ARRL awards were queued up.

+ The proposal to offset LoTW's operational costs by offering an optional annual subscription for a basket of premium LoTW services has also gone nowhere.

+ Complaining about this situation here will have no beneficial impact. Engage with your ARRL Director and remind him or her that software not moving forward is moving backward. If you're outside the US and don't have a Director, engage with the ARRL's CEO.

73,

Dave, AA6YQ

-----Original Message-----
From: ARRL-LoTW@groups.io <ARRL-LoTW@groups.io> On Behalf Of Dave AA6YQ
Sent: 24 May 2019 14:32
To: ARRL-LoTW@groups.io
Subject: Re: [ARRL-LoTW] QSO validation API suggestion

Providing Radiosport sponsors with the (retractable) ability to submit to LoTW an ADIF file describing QSOs and receive a confirmed/unconfirmed result for each QSO was discussed by the ARRL's LoTW Committee years ago as a way to further encourage the use of LoTW. My recollection is that it has been implemented; unfortunately, it has never been rolled out.

LoTW stopped moving forward after WAZ support was added in early 2018.

73,

Dave, AA6YQ



-----Original Message-----
From: ARRL-LoTW@groups.io [mailto:ARRL-LoTW@groups.io] On Behalf Of Joe Subich, W4TV
Sent: Thursday, May 23, 2019 8:26 PM
To: ARRL-LoTW@groups.io
Subject: Re: [ARRL-LoTW] QSO validation API suggestion


*NOT POSSIBLE* LotW derives the callsign and entity from the uploading station's "certificate". It is not possible for any other source (including contest sponsors) to obtain access to that certificate as to do so would eliminate station authentication.



73,

... Joe, W4TV


On 2019-05-23 6:46 PM, Gary Hinson wrote:
I d like to suggest an LoTW facility (maybe an API?) through which contest adjudicators and awards administrators could submit entrants claimed QSOs as ADIF records for validation either singly or in batches.



That, in turn, would increase pressure for stations to submit their logs to LoTW routinely, benefiting the global community of LoTW users.



To be clear, I envisage the need for controls over access to and use of the third party QSO validation function in order to avoid various risks e.g. people speculatively checking DX logs for matches, or overloading the system by bulk-checking for major contests. I also appreciate that there would be a cost involved in specifying, developing, testing, using, managing and maintaining the service. It would not be unreasonable for ARRL to charge third parties for use of the service in order to recover the costs. If necessary, ARRL might even request contributions up front in order to set the ball rolling with, maybe, a feasibility study.



73
Gary ZL2iFB







---
This email has been checked for viruses by AVG.
https://www.avg.com

Gary Hinson
 

Thanks again Dave.

I quite like the sound of premium services, provided the core services remain free for all as now and the premium pricing is realistic. I'm guessing they might be things such as QSL card printing? 3rd party QSO validation? Fast track single-QSO uploads? Hmmm, something to mull over at the weekend!

I also like the sound of a strategic approach to continue driving up the popularity of LoTW. FWIW 58k of my 100k QSOs since emigrating to NZ in 2005 are confirmed on LoTW - I passed the half way mark a couple of years back. Some 61% of my QSOs this year have been on FT8 - clearly a popular computer mode, so I'm a little disappointed that some hams are sufficiently computer-literate to use FT8 but don't seem to want to learn and use LoTW.

I'm just outside the US, a mere 12 hours flight across the Pacific from SFO ... but I'm an ARRL member so I guess there's someone at HQ to deal with us 'forreners'!

Have a good weekend all: WPX CW ahead which means all sorts of weird-n-wonderful prefixes to hunt down while figuring out roughly which way to point the beam.

73
Gary ZL2iFB

-----Original Message-----
From: ARRL-LoTW@groups.io <ARRL-LoTW@groups.io> On Behalf Of Dave AA6YQ
Sent: 24 May 2019 18:07
To: ARRL-LoTW@groups.io
Subject: Re: [ARRL-LoTW] QSO validation API suggestion

+ AA6YQ comments below

Wow! Impressive foresight! Tnx info Dave.

Re the LoTW freeze, is that a resourcing issue, a strategy/policy to let the system stabilize [it's not broken so don't fix it!], or some other reason I wonder?

+ Resources were diverted to what the ARRL considers higher-priority tasks - which means that the Board's decision a few years ago to recruit and hire two developers for LoTW was temporary. I don't disagree with the priority of the tasks to which the resources were diverted, but at minimum the development resources taken from LoTW should have been replaced.

+ The timing could not have been worse. The improved reliability and capacity generated by preventing duplicate uploads and re-implementing the LoTW Server followed by the crisp delivery of WAZ support generated significant momentum. The "RadioSport Sponsor query mechanism" and support for several non-ARRL awards were queued up.

+ The proposal to offset LoTW's operational costs by offering an optional annual subscription for a basket of premium LoTW services has also gone nowhere.

+ Complaining about this situation here will have no beneficial impact. Engage with your ARRL Director and remind him or her that software not moving forward is moving backward. If you're outside the US and don't have a Director, engage with the ARRL's CEO.

73,

Dave, AA6YQ


-----Original Message-----
From: ARRL-LoTW@groups.io <ARRL-LoTW@groups.io> On Behalf Of Dave AA6YQ
Sent: 24 May 2019 14:32
To: ARRL-LoTW@groups.io
Subject: Re: [ARRL-LoTW] QSO validation API suggestion

Providing Radiosport sponsors with the (retractable) ability to submit to LoTW an ADIF file describing QSOs and receive a confirmed/unconfirmed result for each QSO was discussed by the ARRL's LoTW Committee years ago as a way to further encourage the use of LoTW. My recollection is that it has been implemented; unfortunately, it has never been rolled out.

LoTW stopped moving forward after WAZ support was added in early 2018.

73,

Dave, AA6YQ



-----Original Message-----
From: ARRL-LoTW@groups.io [mailto:ARRL-LoTW@groups.io] On Behalf Of Joe Subich, W4TV
Sent: Thursday, May 23, 2019 8:26 PM
To: ARRL-LoTW@groups.io
Subject: Re: [ARRL-LoTW] QSO validation API suggestion


*NOT POSSIBLE* LotW derives the callsign and entity from the uploading station's "certificate". It is not possible for any other source (including contest sponsors) to obtain access to that certificate as to do so would eliminate station authentication.



73,

... Joe, W4TV


On 2019-05-23 6:46 PM, Gary Hinson wrote:
I d like to suggest an LoTW facility (maybe an API?) through which contest adjudicators and awards administrators could submit entrants claimed QSOs as ADIF records for validation either singly or in batches.



That, in turn, would increase pressure for stations to submit their logs to LoTW routinely, benefiting the global community of LoTW users.



To be clear, I envisage the need for controls over access to and use of the third party QSO validation function in order to avoid various risks e.g. people speculatively checking DX logs for matches, or overloading the system by bulk-checking for major contests. I also appreciate that there would be a cost involved in specifying, developing, testing, using, managing and maintaining the service. It would not be unreasonable for ARRL to charge third parties for use of the service in order to recover the costs. If necessary, ARRL might even request contributions up front in order to set the ball rolling with, maybe, a feasibility study.



73
Gary ZL2iFB







---
This email has been checked for viruses by AVG.
https://www.avg.com

Dave AA6YQ
 

+ AA6YQ comments below

I quite like the sound of premium services, provided the core services remain free for all as now and the premium pricing is realistic.

+ Core services, specifically the ability to submit QSOs to LoTW, would remain free.

+ At the time, we were thinking that an optional annual subscription for premium services would be in the $10 to $25 range.


I'm guessing they might be things such as QSL card printing? 3rd party QSO validation? Fast track single-QSO uploads? Hmmm, something to mull over at the weekend!

+ A survey was conducted at the 2017 Dayton Hamvention. These were the candidates (in no particular order):


- An LoTW-based log backup service

- Online logging program—log direct into LoTW

- Upload logs directly to LoTW website (without having to use TQSL)

- Notification via email when submitted QSOs have been processed

- Notification when you receive a new QSL match in LoTW.

- Notification when you reach a new award level for ARRL awards (configurable e.g., DXCC, WAS, VUCC, etc.)

- Notification when you receive a new QSL “counter" (configurable (DXCC ATNO, WAS new state, etc.)

- Notification of results when award applications have been processed.

- Notification for awards sponsored by IARU member societies

- Receive customizable QSL card, delivered via bureau system (or directly for an additional fee)

- Ability to hide submitted but unconfirmed QSOs

- Optimal submission generation (LoTW identifies the minimum set of QSOs to submit that would maximize progress toward an award)

- Access to leaderboards and alerts for "short-lived" operating activities (contest-like activities, like—"who can confirm the most DXCC entities during a weekend?")

- Access to leaderboards and alerts for "long-lived" operating activities (like Centennial and NPOTA)

- Ability to message a QSO partner who’s also on LoTW

- Ability to message ARRL Support from within LoTW

- Access to analytics generated from ARRL data (LoTW, various award credit databases)

- User customizable queries with results in graphical or tabular form

- Yearly Premium membership level that enables you to have two free LoTW applications for ARRL awards in that calendar year

- Yearly “Sponsor” level (tax deductible in U.S.) identifying your station as an LoTW Sponsor on a special LoTW page


+ The purpose of the survey was to identify a subset of features that would produce maximum adoption. An online version of the survey, though planned, was never posted.

73,

Dave, AA6YQ

Gary Hinson
 

What a superb wish-list! Love it!

73
Gary ZL2iFB

-----Original Message-----
From: ARRL-LoTW@groups.io <ARRL-LoTW@groups.io> On Behalf Of Dave AA6YQ
Sent: 24 May 2019 19:11
To: ARRL-LoTW@groups.io
Subject: Re: [ARRL-LoTW] QSO validation API suggestion

+ AA6YQ comments below

I quite like the sound of premium services, provided the core services remain free for all as now and the premium pricing is realistic.

+ Core services, specifically the ability to submit QSOs to LoTW, would remain free.

+ At the time, we were thinking that an optional annual subscription for premium services would be in the $10 to $25 range.


I'm guessing they might be things such as QSL card printing? 3rd party QSO validation? Fast track single-QSO uploads? Hmmm, something to mull over at the weekend!

+ A survey was conducted at the 2017 Dayton Hamvention. These were the candidates (in no particular order):


- An LoTW-based log backup service

- Online logging program—log direct into LoTW

- Upload logs directly to LoTW website (without having to use TQSL)

- Notification via email when submitted QSOs have been processed

- Notification when you receive a new QSL match in LoTW.

- Notification when you reach a new award level for ARRL awards (configurable e.g., DXCC, WAS, VUCC, etc.)

- Notification when you receive a new QSL “counter" (configurable (DXCC ATNO, WAS new state, etc.)

- Notification of results when award applications have been processed.

- Notification for awards sponsored by IARU member societies

- Receive customizable QSL card, delivered via bureau system (or directly for an additional fee)

- Ability to hide submitted but unconfirmed QSOs

- Optimal submission generation (LoTW identifies the minimum set of QSOs to submit that would maximize progress toward an award)

- Access to leaderboards and alerts for "short-lived" operating activities (contest-like activities, like—"who can confirm the most DXCC entities during a weekend?")

- Access to leaderboards and alerts for "long-lived" operating activities (like Centennial and NPOTA)

- Ability to message a QSO partner who’s also on LoTW

- Ability to message ARRL Support from within LoTW

- Access to analytics generated from ARRL data (LoTW, various award credit databases)

- User customizable queries with results in graphical or tabular form

- Yearly Premium membership level that enables you to have two free LoTW applications for ARRL awards in that calendar year

- Yearly “Sponsor” level (tax deductible in U.S.) identifying your station as an LoTW Sponsor on a special LoTW page


+ The purpose of the survey was to identify a subset of features that would produce maximum adoption. An online version of the survey, though planned, was never posted.

73,

Dave, AA6YQ