"When I put my finger on the crystal I see the frequency jumping for 100Hz or more."
Is this frequency jump from your hot fingers? I'm wondering if it has another source, because your
plot shows a fairly fast change of frequency.
I'm wondering if some local RF fields are re-biasing the oscillator, causing quick drift.
I'd be tempted to try grounding the crystal case. It is currently floating, and can act as an antenna.
And perhaps adding some shielding above/below the Si5351 + crystal. A tin hat & underpants.
It is also possible that local RF fields affect the power supply regulator feeding Si5351. Twist those
Vcc & ground wires together to minimize loop area.