I managed to understand the working and schematics of the 7854's digital section since I last reported this issue, and fixed the problem. For posterity and for the benefit of anyone who is trying to troubleshoot problems with the 7854 I thought I'd document what I understood. Hope it turns out useful to someone!
- There are several test points of the 7854 accessible from the top side where the digital board get plugged in. These are exceedingly useful. Some waveforms described in the manual are, however, not accessible from these test points. An easy way to create access is to pull out the respective board and solder a header pin to the point in question. Then you can simply plug in a receptacle pin onto the header whenever you need to make an observation. (You need to bend the header pin so it points upwards.)
- The scope display alternates between XY mode (not to be confused with the X vs Y displays for obtaining Lissajous figures) and YT mode. YT mainly handles real-time waveforms and XY mainly handles digitized data and readouts. The two kinds of signals meet at the Vertical Amplifier (and Horizontal Amplifier for horiz signals). One of them is suppressed by the XYINH signal (which is not TTL).
- In this particular case since both signals were getting mixed up, the investigation quickly narrowed down to a few signals including the XYINH signal. This signal is generated in schematic 34 in the Display D/A and Amplifier section, using !DSYOFF, by a few transistors including Q1830 (CA3046). Turned out that one of the transistors in the CA3046 had gone bad and so the XYINH was not able to go positive. Replacing the chip has solved this problem - my scope which was apparently not able to store any waveforms earlier now does so!
It still stops acquisition with an error beep because it is not able to store the readout information (which is missing in the SCOPE mode) and I believe that is due to a bad XY signal. If and when I get round to solving that I'll post my findings here.
Best - Ram