Dave - slight correction/clarification:
Perhaps it needs to be raised with the LOTW administrative groups for a function to be implemented, perhaps via the web interface, to "weed out" old duplicates - only leaving the newest version? Perhaps the whole policy of being unable to delete records from LOTW needs to be reviewed? Perhaps the simplest fix is for TQSL to be updated to permit upload of non-duplicate records yet only fail the duplicates?
The way that eQSL handles duplicates checking is perhaps the "Gold standard" - offering a function to identify duplicates via the web interface and then offering either an "auto-fix" based on the latest record received OR permitting manual selection of the record to be retained. Perhaps such a function is needed for LOTW based on your identification of so many duplicates entering onto the system?
Yet I still feel that this is perhaps unnecessary and undesirable. Duplicates are NOT necessarily a bad thing under LOTW's policy … and restate that many duplicates from responsible Amateurs are actually corrected / updates / manually improved records ! I myself was always taught to only maintain ACCURATE data. Sometimes it becomes obvious that not all Amateurs out there do that - and one must go in and fix that data !
Having duplicates in the LOTW system maintains a record - a trail - of changes that have taken place :-) But as we know, it can also be considered to consume resources.
I still restate that t0he simplest fix overall is to make "Permit uploading of QSO's already uploaded" sticky. Yes I have thought this through long and hard before making the request.
VK3VM / VK3SIR