If you are running milliwatts then a few more milliwatts isn't going to lift you out of the noise if you are struggling. It takes a fourfold increase in output power level to gain 1 S point.
G4BUE did some remarkable long distance QSO's at both milliwatt and microwatt levels.
He didn't try to screw every last milli or micro out of the rigs but he did invest in very good antennas and there lies the magic. Getting 3 watts or 5 watts out of a rig isn't going to make a discernible difference. Changing to a better antenna could make a big difference.
If you are trampling around on the hills near where I live then the height is going to help you. I live surrounded by sea water - that's always good for an extra S point or two even with portable antennas. Have a look at my photos on QRZ.com