Scan rate for real network analysers is a variable like a O'scope and should be slow for narrow band measurements.
For my Rigol 815T and the HP8568B or a PNA like the HP 8753C, Agilent-8357A I use 100HZ and not faster
than 500mS ( about 5 seconds to scan the frequency span of about 8khz). This has nothing to do with baud
rate that is the transfer speed after the data is acquired. You didn't say what instrument and PSHNA is not
something I've used.
If you sweep too fast what happens it the filter rigs like a bell and the propagation delay through the filter results
is a very lop sided curve. An effect of generate a discrete frequency and then measure the output of DUT before
things have settled.
Your cables and transformers must be terrible! The usual measurements (S21 transmission or though measurement)
for RG316 1 meter cables and 1:4- 4:1 back to back transformers at 12mhz range is on the order of maybe 1-2 DB total.
The usual transformer I make up is a Trifilar would 2 hole balun (fT43-202) and usually measures out as .6-9db each
(two of them back to back for 50 to 200, 200 to 50) in the HF range. However you did back out the measured response
to get a valid one.
To get losses higher like that I'd be playing at maybe 200-300mhz or used really poor ferrite likely both!