So, what are we looking at there? The output of the Si5351 or the output of your heterodyne mixer? Either way, unless the signal goes through a band pass filter, there will be significant harmonic content which can cause the fuzzy trace.
The jumpiness on the oscilloscope might be harmonics and other signals, perhaps digital signals, leaking into the output.
As the extraneous signals go in and out of phase with the desired signal, instantaneously adding and subtracting from the desired signal, the trace on the scope will demonstrate the jumpiness you're seeing.