Am a masochistic PIC programmer, calculating the Si5351 parameters in raw assembler. Yeah - painful, but I enjoy doing weird math.
Sticking with Hans' approach where Silicon Labs AN619 app-note "a+b/c" equation uses a fixed value for "c" of 0xFFFFF or decimal 1048575.
And also to reduce jitter, the VCO divider down to the desired frequency is an integer with no fractional part, so the Si5351 runs in "integer-divider mode". The PLL multiplier that transforms the 27 MHz up to a VCO frequency near to (but less than) 900 MHz uses the full fractional range of 15+0/1048575 -to- 90+0/1048575.
I have accommodated in the math that the ovenized 27 MHz crystal oscillator settles at some small, fixed offset, after warm-up.
Example runs using a PIC16F1455 in MPLAB simulator finds a full solution for all the multisynths in about 0.5 milliseconds (with a 16 MHz clock).
Am still concerned with phase noise caused by fractional-synth jitter. Have no way to measure it. Silicon Labs suggests
that lowest jitter would employ integer-only PLL multiplier going from 27 MHz up to UHF VCO, and then an integer-mode
divider going down to the output frequency. But that rule severely limits the resolution of output frequency. Next lowest
jitter employs fractional-synth PLL up to VCO, combined with integer-mode divider (my approach). It also suggests
to me that forcing "c" of a fractional-synth PLL multiplier to 1048575 might cause jitter to be large - at least for some frequencies.
My software is too optimized to recognize if "c" is a multiple of "b". For example, b/c = 13981/1048575 = 1/75.
Silicon Labs is reluctant to reveal Si5351 internal details. Does anyone know if there'd be any jitter difference
between setting "b/c" ratio to 13981/1048575 versus 1/75?