I believe that there is a grounding issue, though not a "loop". When you have only the Teensy connected, there is not a reference to ground on the computer, so you get the hum. This would be irrespective of the transformers. When you connect the Raduino, you get the ground from the uBITX.
I would be curious if you are using a laptop or a desktop computer?
On the other issue, feedback into the uBITX headphones, I have had that issue, along with distortion in transmitted audio. For me it was traced to RF getting back into the uBITX along the coax shield and the mic input. I solved that with snap on ferrites at the entrance of the coax to the shack. This is a trait of the uBITX as the audio is not completely muted to allow for the side tone in CW mode, at least that is what I surmized, as grounding that input removed the issue as well. Since you are bread boarding, the total audio system could be an antenna feeding back into the uBITX. Ferrites on the audio lines might help if you determine that it is RF causing the issue. Try transmitting into a good dummy load that would have minimal radiated RF. If the tx distortion goes away, then RF is the issue.
Above are just some ideas to try.