First off, I do not have a v6. I HAVE worked FT8 with both the v4 and v5 uBITX.
The jacks are the same for the v4, v5, and v6. All are TRS (Tip Ring Sleeve) NOT TRRS that is standard for a laptop or cell phone headphones with microphones. A TR only would short out the PTT on the mic input and the audio out of the headphone jacks. DO NOT use tip and ring only (TR) plugs with the uBITX.
The way that I connect for FT8 is with a cheap USB sound card dongle to split the mic and speakers into two separate TRS jacks. This makes it easier to wire as I have had issues with laptops not recognizing the mic input at the laptop jack. I also use an EasyDigi kit to isolate grounds, though some have reported not needing it. Since the headphone jack on the uBITX has both tip and ring connected to audio out of the rig, you do not need a special cable for the rig headphone connection to the PC (laptop) dongle mic input. A standard 3.5 mm TRS on both ends works. Not so for the mic input on the rig, Here the sleeve is the PTT connection, so need to connect only the tip to tip and sleeve to the sleeve of the other cable (you do not need both channels of the stereo signal). You need to have the ring of the microphone cable unconnected on the rig side. One last point, I have added clamp-on ferrites to all lines to and from the rig. This is to reduce any RF issues in the shack causing distortion or erratic operation.
I am using KD8CEC software for CAT control. That is how the PTT is activated. In WSJT-X I have the comport set to the one that shows up when I plug in the USB cable to the uBITX Nano. Radio selected is Yeasu FT-857, Baud for KD8CEC software is 38400, 8 bits, 2 stop bits, none for handshaking. PTT is CAT Mode is USB. Split operation is set to none.
The biggest issue that I have is the sensitivity of the volume control of the uBITX. I have to run it with very little volume or I get the RED bar on the left of the FT8 window signaling input overload. I also run the transmit level low, adjusting the power out measured with a Nissei SWR/Power meter to be the same power as CW (do not know of a better way to verify that I am not overdriving the audio).
Hope the above helps. I would be interested in any experience with the v6 software, as I believe that cat control is included with the stock software.