If I am reading the filter specification right, you might want to start with a 50ohm termination on the output of the filter before the input to your scope for your test. Rerun the test to see what you get.
You may also want to verify the impedance of the transmitter to be sure that it is expecting a 50ohm source. If it is different, then either a resistor pad or a transformer may be needed.
Just a couple of suggestions.