FW: [ARRL-LoTW] Update on Logbook downloads

Dave AA6YQ

The ARRL's explanation for the recent rise in "invalid request" errors when invoking DXKeeper's "Sync LoTW QSOs" or "Sync LoTW QSLs" functions is appended below. GridTracker is the application that could be configured to download all of the user's LoTW QSOs on startup.


Dave, AA6YQ

-----Original Message-----
From: ARRL-LoTW@groups.arrl.org [mailto:ARRL-LoTW@groups.arrl.org] On Behalf Of Greg K0GW
Sent: Friday, June 18, 2021 9:01 AM
To: ARRL-LoTW@groups.arrl.org
Subject: [ARRL-LoTW] Update on Logbook downloads

Here’s an update on what is being done to understand and deal with the download issues that some users have been reporting here and elsewhere recently. There are actually at least two different but related problems. Work is in progress to resolve them.
Greg, K0GW

Update on recent issues downloading user logs from Logbook of the World.

In the last several weeks, there have been some reports of Logbook rejecting download requests. This is an overview of the situation.

In 2020, ARRL IT staff noticed that certain high-volume users were stacking multiple concurrent requests to retrieve users' logs, and often did so at one minute intervals. Further, many of these requests appeared to start from the beginning of DXCC (1945), rather than being incremental, thus posing a significant and unnecessary load to the database server. This was occurring again, despite discussions with these services several years ago requesting that the practice be discontinued.

Accordingly, rate-limiting penalties were implemented to deny follow-on retrievals within a short period. Code was added to LoTW to detect and deter abuse of its "download all QSOs" capability by returning a "503" error in response to excessive downloading. When this is seen by the logging program, depending on how its developer has coded it, this may appear to the user as an invalid response error. Though the error message returned by the server includes a description of the error, logging programs typically strip this information off when informing their users.

Recently, users had reported receiving an "invalid request" message when they attempted to retrieve copies of their logs from Logbook. At this point, it appears that, as an undocumented default, one software package was configured to download all of the user's LoTW QSOs on startup, an operation that can take some time. When users then attempted to retrieve their QSOs in other logging software, they received the "invalid request" error. The developer has already moved to correct this problem.

It remains to fine tune and document Logbook's rate-limiting approach. The Logbook developer reflector has been informed of the general practice of rate-limiting, and will be informed about the specifics of the algorithm used when it is stabilized.

Users who experience problems with retrieving Logbook records should notify the developer of their logging program, including specifics of the operation they attempted and the results returned. At that point, the developer can investigate whether there is a problem with Logbook or in the manner in which the logging program interacts with Logbook in retrieving the user's information.

Join DXLab@groups.io to automatically receive all group messages.