You're right, I think, a simple cheap standardized UBitx test set would really be a great tool, not just for a Ubitx but for anyone starting out without any RF test equipment.
I started down that road for a club talk last year and built a single 2N3904 oscillator with a color burst crystal that gave signals in or very close to all the HF bands except 60 meters.
By juggling a few values I was getting reliable output on all those freqs at -20 dBm plus or minus probably 5 dBm, and this was reproduceable with 5 different color burst crystals (tho from the same batch). Not enough to drive a diode ring mixer but otherwise pretty useful and consistent, I think close enough that you could count on the output being what you expected without measuring it.
As an option adding a buffer with another 2N3904 would make it more useful, or maybe just use one of Diz's inexpensive new SMT HF amp kits instead of reinventing the wheel.
Added an IF crystal from Kits and Parts (it was for a V3) with a cap and coil to fudge it a bit and got a useful IF signal at the same level.
Added a couple 50 dB Pi attenuators to give a choice of 50 over 9, S9 and a few tenths of a microvolt for signal level. Standard resistors used that way yielded decent attenuators plus or minus 3 or 4 dB.
Did it on a piece of perf board using 0.1 inch headers and jumpers to switch crystals and attenuators so there were no switches or other hardware needed.
It was dirt cheap and pretty handy, adding the diode detector would be simple, and the attenuators possible useful with that, same for adding an audio generator. That's got to be a 10 dollar or less project with bulk parts purchases.
It would make a great first build project too.
If I had any board skills I would make a board and put in Osh Park... :) But if anyone with some skills wants to collaborate on it, I'd be happy to get together, as I expect people troubleshooting with a known test kit would make the great advice you, Evan, Allison and others give on here even quicker and more powerful.
73 Scott ka9p