The solder joint on the inside end of the BNC was broken. Someone had done a repair before - plastic was melted nearby from their soldering iron. There had been nothing but a bridge of solder between the cup on the BNC pin, and the post that it should attach to.
I don't know what the original connection should look like, but I made a stronger joint by wrapping a piece of wire-wrap wire around the joint, then solder-blobbed the whole thing.
The "looseness" is gone, now that the joint is attached. I couldn't see anything wrong with the BNC, and the set-screw was tight. If I treat it gently, it should be fine now.