I would agree with what you are saying based on the v5 that I have. I did not need to add the outboard amp as I had plenty of sound from the speaker. However I still run the volume almost maxed out.
I did need the amp on the v4 boards that I have. This is why Sunil included it in the kits.
Since I do not have a v6, you will need to work out the best place on your own, or hopefully someone else has done it already.
Before you cut into your rig, you might want to run the calibration again. If the master crystal is off there may be an impact on the 45 MHz roofing filter. Just a guess on my part and I have not done the math. As stated above, my v5 with basically the same audio circuits does not have the volume issue as you have stated.
73 and good hunting for the solution.