Re: NewBe Question

John N0TA <reillyjf@...>

Got it to work. Would NOT download with Chrome. Changed to MS Edge. Edge said it couldn't download securely, but I over road it. Anyway, Launcher and DX Keeper are installed.

What got me interested in DXlabs is I have close to 200k QSO's in N3FJP, and it has become very slow. I imported the 200k QSO's into DXkeeper, and it seems OK so far.

I recognized your name and call from the LoTW group. I think we had some communications awhile back concerning a LoTW problem I was having with a location problem (operating away from my Home location).

FWIW, I have never been a fan of the current ARRL website - I agree it needs rework. On the other hand, LoTW has been working remarkably well since the work you did. How much longer that will continue is a big question.

Thanks for your help,
  - John, N0TA

On 1/14/2022 2:54 PM, Dave AA6YQ wrote:
+ AA6YQ comments below

I'm using the link from the website:

+ That URL is working correctly here; no one else has reported a problem.

Aren't you the LoTW guy?

+ I began extending DXLab to interoperate with LoTW in DXKeeper version 2.5.0, which was released in May 2003. That resulted in lots of interaction with Jon KE3Z, who was single-handedly responsible for designing, implementing, testing, and documenting LoTW, and providing support to early logging application developers like me. The project was ridiculously under-scoped and under-resourced, leading to enormous amounts of dirt to be swept under the carpet -- like testing and user documentation. ARRL management compounded the problem by directing Jon to add more functionality -- like support for new awards like WPX and "Triple Play" -- rather than address the "technical debt" they'd accumulated.

+ In November 2012, users began reporting that some of their submitted QSOs were not being processed. That led users to pre-emptively resubmit their QSOs (there was no "duplicate submission" checking back then), which resulted in even more QSOs being reported as not processed. The escalating resubmissions - the software equivalent of thermal runaway - brought LoTW to its knees.

+ I feared that ARRL management would abandon LoTW, so I drove down to ARRL headquarters in Newington and convinced then-CEO Dave K1ZZ to let me help. Mike K1MK, who'd recently been hired as a programmer to replace Jon KE3Z, found the defect responsible for loss of submitted QSO: if the processing queue exceeded a particular depth, subsequently submitted QSOs were erroneously discarded. Even the most basic stress test would have revealed this long-latent defect.

+ The ARRL formed the "ARRL-LoTW" Committed with me as a member. I convinced Dave KM3T, Ken K1EA, and Michael G7VJR -- all database experts -- to conduct an architectural review. As a result of this review, we suggested that the ARRL first focus improving LoTW's reliability and performance problems, and then address its usability problems. We made three immediate recommendations:

1. replace the obsolete, slow, unsupported version of the SAP database engine used in the LoTW Server with the current , faster version of that engine; this required converting from a proprietary C++ interface to an industry standard ODBC interface, but meant the ARRL could further improve the LoTW Server by replacing the SAP engine with a higher performance ODBC-compliant database engine from a commercial supplier or open source.

2. hire 2 developers and dedicate them to LoTW

3. purchase hardware that will enable the LoTW developers to test changes before releasing them to public use

+ The recommendations were accepted. By 2017, LoTW's measured availability had increased from less than 90% to more than 99.9%, and the rate at which submitted QSOs were processed had doubled. Rick K1MU and I rebooted the TQSL open source project, and combined what had been two cryptic applications (TQSL and TQSLCert) into a single application with a modern user interface that could be localized (and now supports 16 languages!). We found that many users were resubmitting their entire logs to LoTW after making a few more QSOs; these "already processed" QSOs, comprised ~50% of the load on the LoTW Server, so we extended TQSL to detect and reject logs that contain already-submitted QSOs.

+ In early 2018 we introduced WAZ support. IOTA was next in line, and discussions were underway with JARL. Mike K1MK had prototyped a new interface that would enable any award sponsor or radio club to submit an ADIF file containing QSOs and receive a report showing which of those QSOs had been confirmed by LoTW: without requiring the user's LoTW credentials. The plan was to license the use of this interface to award sponsors like JARL, DARC, and WIA (for a fee), but to also allow local radio clubs to use it (at no charge) to score sprints and other on-air activities using LoTW confirmations.

+ Just after WAS support was rolled out, the ARRL re-assigned the two LoTW developers to other projects it claimed were more important; 4 years later, neither of those projects has born fruit. In my ~50 years of professionally developing computer hardware and software, I have never seen a more flagrant example of "snatching defeat from the jaws of victory".

+ The ARRL's new CEO proposes to start LoTW over from scratch, saying that the current version of LoTW will be fine until he recruits a new development team that will over the course of several years develop a beautiful new LoTW. This is a mistake of epic proportions. I can count the number of software companies on one hand that have successfully maintained one product or service while another team worked for years to replace it.

+ The right next step, for the record, is to first choose a new modern web framework for as what's there now is obsolete, user-hostile, and extremely inefficient for ARRL staff to utilize. Using it to develop was like building a ship in a bottle: writing raw HTML in a tiny window. With a new framework chosen, a small section of LoTW's current user interface - say "DXCC award status" - should be re-implemented atop the new framework, and publicly released. Feedback from users can then be used to refined the user interface, and tested by modifying LotW to use the new framework for another small section of its user interface. Sequentially, the entire LoTW user interface can be replaced with the new framework, addressing the last issue remaining from the ARRL's naïve under-scoping of LoTW back in 2001.

+ At this point, the ARRL no longer has anyone who knows anything about LoTW internals. Only two or three Board Members are sufficiently knowledgeable to understand the technical tradeoffs; the rest are all-too-easily mislead.

+ So am I "the LoTW guy"? Definitely not. I tried to help, but LoTW's prognosis now looks terrible.


Dave, AA6YQ

Join to automatically receive all group messages.