I have started from the beginning. Replaced 4 aluminum caps on the A5 board (checked the ESR and capacity and they were fine). Then borrowed two 6.5 digit Agilent multimeters (to make sure about the readings) and set the 10V reference voltage to 10.0001V (can't do any better, the resistor is crusty). To measure things I use the ground hole under the calibrator output for the banana plug and pointy probe to measure things on J119.
Then I've continued to set the DAC reference voltage. The whole band is 2.501 volt, however it's not super symmetric: -1,236 for CCW and -1.265 for CW.
Then without touching anything else and re-running the CAL01 I've jumped on CAL02. Unfortunately CAL02 fails in the same fashion -> LIMIT ERROR.
Also noticed, that the gain of the scope is a little bit higher. I.e. instead of getting 0.5V p-p for 500mV calibration signal I get 0.55V. I wonder if it needs some fixing before going to CAL02.
For the vertical calibration I don't have the calibrator module like PG506. I substitute for that the AWG. The signal is 1KHz 50% duty cycle. Low level is 0V, HIGH level is what menu asks for into 50 Ohm, i.e. for 500mV required the AWG is set to provide LO=0V HI=0.5V into 50 Ohm. The output was tested on the "reference" scopes terminated to 50 Ohm.
I'm a bit lost in what could go wrong. I'll try checking the voltage ripple at the output of the power rails.
I just wonder if CAL02 would be affected by bad CAL01?