It is always a challenge to fit a rig into a small space.
As Bill Schmidt says, watch out for stray RF. You may have to mount some
shielding (which is true of almost any ham and any rig). Hi. The easiest solution
is usually coax cable with included shielding. That probably should be
used for the three RF signals from the Raduino to the board anyway. Ground
the coax on the Raduino side, but NOT on the motherboard side. The cable
to the display can be wrapped with aluminum foil -- that is usually adequate.
Ground the foil on the Raduino side.
A good cable, if you can find it, is RG-48 with teflon insulation and 100%
shielding. It works well every time, and it is a bit smaller than regular RG-48.
It is a natural for smd connectors and GPS.
I wonder about the heat sinks. Perhaps you should drill 2 small holes
and tighten them against the case with long 4-40 screws. As long as the
IRF510's are isolated, the heat sink should be at ground, and the case is a
good place to anchor them. Every little bit helps. They can be tapped to
accommodate the screws so no nuts are needed...But nuts can be used as well.
Alternately, you could place a larger heat sink on the outside of the case
a la the method of Graham, VE3WGW. That kind of heat sink is easily found
as it is usually used in power inverters and voltage converters, especially
solar power units.