The SA602 only requires a 50 mV p-p signal to fully unbalance it. Larger signals create all kinds of unwanted mixer products. Since the VOX requires a 1200 mV p-p signal to trigger it, there needs to be a lot of attenuation between the audio input and the mixer, of about a 25:1 ratio. A 22K/1K divider would be close enough.
The apparent reduction of power is the result of a cleaner, linear output and the fact that a watt meter reads the average power of the DSB signal. If nothing else PLEASE MAKE THIS MOD!
The best way to go would be to add a trimmer resistor to the mixer input. Adjust the sound card output level until the board trips into transmit, then turn up the audio level to the mixer with the trimmer until the power output no longer increases. Now back down the level until the power drops off by a little more then 1/2. You are now operating the mixer in it's linear mode and not "flat topping". If you can monitor the output signal with a Scope, this is obvious.
You can make the VOX more sensitive by adding a resistor from the base of Q5 to ground. A 150K resistor would make the VOX much more sensitive. The idea is to pre-bias the base just below the point at which it starts to turn on, with a little margin for supply voltage and temperature variations. If you do this, then the attenuation to the mixer can be reduced.
Unequal loads on the outputs of the SA602 will unbalance the mixer and allow carrier to leak through. This happens when Q1 is turned off during transmit. All of a sudden all the current needed for Q1's emitter load is being supplied by the mixer output. To fix this, add a diode between the +5 supply and the collector of Q1 (cathode to Q1). It would probably also help to make R2 and R3 the same value to further balance the mixer output loads, but just adding the diode should be enough. This will significantly reduce or eliminate the carrier.
You can test if your transmitting a lot of carrier by manually switching the board into transmit by grounding the relay at Q7with no audio input.
73, Steve KD1JV