I would verify that the speaker is 8ohms and not 4. If that is an AmateurRadioKits speaker, then I would look into an audio amp. That is what I had to do for my v5 that I put into a 3.5" case. Luckily it came with an audio amp. That made a big difference AFTER I nocked the gain of the LM386 internal to the yBITX down to 20 (Pop out pins 1 and 8). Any 1 wat or better audio amp module on Amazon or eBay would work.
The other option is to use a more efficient speaker. I need to run now so do not have time to look up what I bought. It was on a recommendation on this site.