Thanks for the responses.
Just and FYI, I do have a mid size mill, just not a lot of experience. The milling attachment was basically free as I got it with all of the stuff I bought from the widow of a friend a few years ago. The reason I thought about putting the milling attachment on the lathe was - I had something mounted in the mill already - mounting can be a major undertaking with the stuff and experience I have and I needed to make a quick part, but I didn't want to loose my reference on the mounted part - so I thought I would try doing it on the lathe instead, which was problematic since the milling attachment did not come with an adapter as far as I know.
Next thing to add to the small project list is a way to drill proper holes in the sides of parts mounted in the lathe since I don't seem to be very good at going from the lathe to the drill press and getting holes drilled correctly. I am talking about holes on the circumference of parts done on the lathe, not in line with the Z axis. I have a rotary table for the mill, but I have not figured out how to use and mount it correctly yet, but that would be a big help I think.