It may be a long shot, but here goes my idea...
Any ideas about the decreasing separation of lines when I increase repetitionSo you have a 7704A mainframe and no other (7K series) to compare to... and using the calibrator at highest speeds, the vertical steps are closer...
Well, it may be that your mainframe's middle frequency response (the higher repetition rate of the Calibrator's GAIN Test mode, is just middle freq to a 200MHz scope) is set too low... and this "lack" of middle freq response is being compensated by an increased middle frequency response in your vertical plugin.
Overall, the frequency response of the set: Vertical Plug-in + Mainframe can be correct because they have been calibrated as one set and, would this be an integrated scope, this would be just fine...
But on a modular scope, one doesn't just want to have the end-to-end response right, but to make sure that the Vertical Output Amplifier (i.e. the Mainframe) have a flat frequency response (and that's where the calibrator comes in), and then, after one knows the MF's freq response is right, the Vertical Plugins are calibrated for even frequency response.
There's hardly anything that can be wrong with the calibrator's staircase generator that could explain a difference in deflection factors when it's set to a higher repetition rate (in comparison to itself, set to a lower repetition rate) but, I can think of two ways to rule this out:
1. Using the calibrator's leveled sine wave test mode (you will need to source a sine wave from an external generator).
If this hypothesis is right, as you sweep from a lower frequency to a higher frequency while ensuring that the sine-wave is leveled using the calibrator's CW Leveled indicator, you should notice a decrease of the sine wave's waveform amplitude on the screen (at some middle frequency, compared to the calibrator's higher repetition rate).
2. If you have another scope, whose frequency response is known correct up to at least 1MHz or so (not much), you can probe the + and - outputs of the calibrator (either using an extender, or probing the signals at the mainframe's back-plane's pins A11 and B11) and while changing from the lower repetition rates to the 1MHz repetition rate, taking note of the steps' voltage difference.
If there's no voltage difference between the steps, when changing from lower rep. rates to the 1MHz repetition rates, then there's nothing wrong with the calibrator, and it's your mainframe who's uncalibrated.
It's not that much difficult to tap into the MF's left bay's back plane pins A11 and B11... The leftier of them is immediately reachable, while the other one (the inner column of pins) can be reached with an insulated piece of rigid wire (I use the wires taken out of an UTP network cable) with just a mm of the insulation stripped off. You can insert the wire between pins 11 and 12 of the outer column, inclined down so that it can touch the inner pin 11.
Ideally, on your test scope, you should connect both your probes and measure them diferentially, and, at this kind of frequency, even using a regular dual channel scope in differential mode will do well enough to confirm if the problem is on what's coming out of the Calibrator, or if it's the MF that is not correctly calibrated.
Well, that's my hunch only.