Date
1 - 9 of 9
update on measured LoTW Performance
Dave AA6YQ
Since LoTW became unstable in early January of this year, I have occasionally measured the rate at which LoTW is processing
submitted QSOs by taking snapshots of the "QSO records entered into the system" statistic displayed on the LoTW web account home
page:
https://lotw.arrl.org/lotwuser/default
A snapshot is obtained by a single web page refresh, so the measurement process is not impacting the LoTW Server's performance.
These measurements showed that LoTW was processing 4-8 QSOs per second, far less than the ~40 QSO per second rate measured after
completion of the LoTW Server upgrade project in 2017.
Some of this decrease is likely attributable to the adverse impact of an early-January GridTracker release that erroneously hammered
LoTW with download requests.
There has been speculation that submitting files containing a single QSO -- which is less efficient for the LoTW Server to process
than files containing multiple QSOs -- are responsible, but a sample taken on February 8 showed that 2463 files processed over the
course of 11 hours averaged 41 QSOs per file:
https://groups.arrl.org/g/ARRL-LoTW/message/37534
There has been speculation that some users may be re-submitting large numbers of-already submitted QSOs, thereby reducing the LoTW
Server's measured processing rate. Note that this requires such users to override TQSL's "already submitted" detection and
prevention mechanism.
GridTracker users have been urged to upgrade to a corrected release, likely reducing the adverse impact caused by its predecessor.
After the completion of this weekend's ARRL International CW DX contest, LoTW's Queue contained more than enough QSOs to keep the
LoTW Server busy. I measured the QSO processing rate from 0207Z to 0222Z. During those 15 minutes, LoTW processed 39,321 QSOs - a
processing rate of 44 QSOs per second! At that time, the average file in the LoTW Queue contained 45 QSOs -- 10% higher than the
February 8 sample cited above.
The good news is that the LoTW Server is still capable of handling a post-contest surge at more than 40 QSOs per second.
The bad news is that the cause of the Server's significantly lower performance when sampled during the past 6 weeks remains unknown.
One could attribute it all to the defective GridTracker release, but that would be speculation. The only accurate way to understand
the LoTW Server's performance under various loads is to run a load testing application on the second LoTW hardware instance whose
acquisition was approved by the Board of Directors in 2013 for this and other testing purposes. If there are serious performance
problems triggered by specific load scenarios, they should be identified and corrected, as the current implementation of LoTW must
support the user community for many years until the new "LoTW for realtime ingestion of contest QSOs and spots" project is proposed,
approved, funded, staffed, designed, implemented, tested, and documented (iteratively, one hopes).
de AA6YQ
submitted QSOs by taking snapshots of the "QSO records entered into the system" statistic displayed on the LoTW web account home
page:
https://lotw.arrl.org/lotwuser/default
A snapshot is obtained by a single web page refresh, so the measurement process is not impacting the LoTW Server's performance.
These measurements showed that LoTW was processing 4-8 QSOs per second, far less than the ~40 QSO per second rate measured after
completion of the LoTW Server upgrade project in 2017.
Some of this decrease is likely attributable to the adverse impact of an early-January GridTracker release that erroneously hammered
LoTW with download requests.
There has been speculation that submitting files containing a single QSO -- which is less efficient for the LoTW Server to process
than files containing multiple QSOs -- are responsible, but a sample taken on February 8 showed that 2463 files processed over the
course of 11 hours averaged 41 QSOs per file:
https://groups.arrl.org/g/ARRL-LoTW/message/37534
There has been speculation that some users may be re-submitting large numbers of-already submitted QSOs, thereby reducing the LoTW
Server's measured processing rate. Note that this requires such users to override TQSL's "already submitted" detection and
prevention mechanism.
GridTracker users have been urged to upgrade to a corrected release, likely reducing the adverse impact caused by its predecessor.
After the completion of this weekend's ARRL International CW DX contest, LoTW's Queue contained more than enough QSOs to keep the
LoTW Server busy. I measured the QSO processing rate from 0207Z to 0222Z. During those 15 minutes, LoTW processed 39,321 QSOs - a
processing rate of 44 QSOs per second! At that time, the average file in the LoTW Queue contained 45 QSOs -- 10% higher than the
February 8 sample cited above.
The good news is that the LoTW Server is still capable of handling a post-contest surge at more than 40 QSOs per second.
The bad news is that the cause of the Server's significantly lower performance when sampled during the past 6 weeks remains unknown.
One could attribute it all to the defective GridTracker release, but that would be speculation. The only accurate way to understand
the LoTW Server's performance under various loads is to run a load testing application on the second LoTW hardware instance whose
acquisition was approved by the Board of Directors in 2013 for this and other testing purposes. If there are serious performance
problems triggered by specific load scenarios, they should be identified and corrected, as the current implementation of LoTW must
support the user community for many years until the new "LoTW for realtime ingestion of contest QSOs and spots" project is proposed,
approved, funded, staffed, designed, implemented, tested, and documented (iteratively, one hopes).
de AA6YQ
Gregg W6IZT
Hi Dave:
Just curious. Is LoTW running on ARRL hardware or is it hosted?
Gregg W6IZT
toggle quoted message
Show quoted text
Just curious. Is LoTW running on ARRL hardware or is it hosted?
Gregg W6IZT
-----Original Message-----
From: DXLab@groups.io <DXLab@groups.io> On Behalf Of Dave AA6YQ
Sent: Monday, February 20, 2023 18:54
To: DXLab@groups.io
Subject: Re: [DXLab] update on measured LoTW Performance
+ Aa6YQ comments below
good info Dave.
I am wondering if the new VOTA contest had something to do with it?
+ Without instrumentation, which ARRL management has so far not implemented, one can only speculate.
73,
Dave, AA6YQ
From: DXLab@groups.io <DXLab@groups.io> On Behalf Of Dave AA6YQ
Sent: Monday, February 20, 2023 18:54
To: DXLab@groups.io
Subject: Re: [DXLab] update on measured LoTW Performance
+ Aa6YQ comments below
good info Dave.
I am wondering if the new VOTA contest had something to do with it?
+ Without instrumentation, which ARRL management has so far not implemented, one can only speculate.
73,
Dave, AA6YQ
Rick Murphy
There were zero changes to Logbook for VOTA. That's been publicly stated.
73,
73,
-Rick
On Mon, Feb 20, 2023 at 2:03 PM Dave AA6YQ <aa6yq@...> wrote:
+ AA6YQ comments below
Just curious. Is LoTW running on ARRL hardware or is it hosted?
+ ARRL hardware. The SAP database engine isn't licensed for hosting.
Dave
--
Rick Murphy, D. Sc., CISSP-ISSAP, K1MU/4, Annandale VA USA
Dave AA6YQ
+ AA6YQ comments below
Something like Dynatrace would show where the problem lies in no time. Wonder what HQ uses for full-stack monitoring, if anything.
+ Since I first began working the ARRL's IT staff in late 2012 until I resigned from the ARRL-LoTW Committee in late 2017, no performance analysis or load/stress testing of LoTW was ever conducted beyond monitoring the number of QSOs LoTW reports as processed to compute its processing rate. Since all LoTW developers were re-assigned to other projects in late 2017, I doubt that any such analysis or testing has been conducted since then.
+ Earlier today, LoTW processed 97,434 QSOs between 0613Z and 0844Z -- a rate of 11 QSOs per second - about a quarter the rate measured two days ago:
https://groups.io/g/DXLab/message/213277
73,
Dave, AA6YQ
Something like Dynatrace would show where the problem lies in no time. Wonder what HQ uses for full-stack monitoring, if anything.
+ Since I first began working the ARRL's IT staff in late 2012 until I resigned from the ARRL-LoTW Committee in late 2017, no performance analysis or load/stress testing of LoTW was ever conducted beyond monitoring the number of QSOs LoTW reports as processed to compute its processing rate. Since all LoTW developers were re-assigned to other projects in late 2017, I doubt that any such analysis or testing has been conducted since then.
+ Earlier today, LoTW processed 97,434 QSOs between 0613Z and 0844Z -- a rate of 11 QSOs per second - about a quarter the rate measured two days ago:
https://groups.io/g/DXLab/message/213277
73,
Dave, AA6YQ