I believe that you have the wrong KD8CEC version loaded on your v5 ubitx. That is why it is not working. In the KD8CEC software, there is a boundary check that keeps you from setting the USB Calibration (BFO) to a value outside of a reasonable range (do not remember the exact value). The version of ubitx prior to v5 had a 12 MHz crystal SSB filter, the v5, and v6 have 11.059 MHz filters. The BFO then needs to be 2,700 Hz lower than the nominal SSB crystal frequency, with adjustments for minor crystal variations.
Go to this GitHub page:
and be sure to get the version uBITXV5 and load that sketch into your Nano. Then set the USB Calibration to the 11055500 value and perform the BFO rough calibration. Verify the calibration using the zero beat method against a known short wave station, then do a final BFO calibration. The steps for the v6 are similar to the v5 with the KD8CEC software. The stock v5 software put the rig into transmit mode at 10.000000 MHz for the calibration step into a dummy load.
The Master Calibration value is to set the displayed frequency to the correct value. Since you have adjusted this, you will need to recalibrate the ubitx once you can receive a signal like WWV.
Hopefully this will fix your issues.