how is your output power on CW? take power readings on CW even if you don't use this mode. this will tell us how the PA etc are working.
some experience weak audio drive that hampers SSB transmit power - a few in our club build. I am fortunate that using a surplus speaker microphone I had works swell, but others are not so happy. some are adding a preamp or even a speech processor IC to their audio chain. I think you need some patient experimenting, after affirming CW power is okay.
if you have a different microphone to try - I would. also consider building an audio oscillator as a test source.