Correct you cannot filter IMD with wide filters unless its outside the crystal filter bandwidth.
For bitx that is about 2.5khz give or take and for ubitx that's 15khz (45mhz filer is for FM, wide).
However... Limiting the audio bandwidth to the modulator helps. A good compressor
can help. Isolating the tx audio from the RX audio may help. Why? The RX preamp
Q70 even unpowered the base emitter junction can look like a diode across the mic amp
(Q6) output teh clipping level is about .5 to .6V P-P and it would be noticeable. Inserting
an audio low pass filter between C222 and C63. Then you get a free audio clipper with
lower IMD at that point. Suggested cutoff is around 3.5Khz.
If the N6DT pop stopper is used that would isolate the RX and TX audio paths, worth a
try and it could help more than the pop.
The next step is insuring the drive from the modulator to the succeeding stages does not
push the DBMs into compression. Generally that means staying 10db or more below the
LO for each mixer. That helps both inband (IMD) and out of band (spectrum). You trade
that for less power out and lower average power. You need very little gain from the
modulator to the next mixer and the amp plus filter should have a net gain of maybe
only a few DB otherwise the mixers are being hammered and will compress and
you will hear that.
Now filtering from the last mixer to the power chain does help the out of band especially out
of ham band products. The fewer undesired products you feed the amplifier chain with the
better the result. The key is only out power to the RF you want even weak spurs and splatter
is lost power.
Then you have to deal with the amplifier chain itself. Every stage must have enough current
to insure you do not hit even -1db compression. Note the output matching may not be
optimal and can have the device in voltage or current limiting. Underloading and overloading
can be troublesome. That's tricky as what may work well for 80-20M might be really poor at
10M. If you do not need above 20M then optimize for that, you should be able to get a few DB
improvement. For example the low pass filter could then have a 3db corner of 16mhz. The
output low pass filter can be then for split for 80 through 20M (or 40M).
A 24 volt battery in the field... 2 8AH gells will do that nicely or 6 series LiPo. Both cases give
more watt-hours with less current.
However for the ubitx that means protecting the most everything before the pre-driver from
more than 12V. The TBA2822 seems fragile.
I do know that with the monobanders I have for 80/40/20/15/10/6M that they all perform
better as they are optimum fot the band rather than trying to be everything for ever where.
Multiband with wide tuning is not easy ad requires some compromise to accomplish at that
uBitx does pretty well.