N1JM de WB0TEV,
I run RTTY using N1MM+ and MMTTY to a TS-590SG. Here are my settings when I do AFSK RTTY. Note that I send the audio to/from the radio via the USB cable and invoke PTT that way a well. (I assume you have the USB CODEC software installed).
Configurer, Hardware Tab, click the set button on the row that has Radio=TS590 selected. On the Com window that opens check the box for PTT via Radio Command Digital Mode.
Configurer, Digital Modes tab, under the Digital Interface 1 and 2 columns in the TU Type pulldown choose Soundcard.
Configurer, Digital Modes tab, under MMTTY path, browse to the location where you have placed a copy of the MMTTY.EXE engine. Note that this must not be the same as where you might have and use a stand-alone version of MMTTY and it must NOT be under the windows Program Files folder.
Configurer, Mode Control Tab: Under Mode recorded in Log check Always and select RTTY from the pulldown. REMEMBER TO UNDO THIS WHEN YOU GO BACK TO SSB/CW/PSK/FT8.
Configurer, Mode Control Tab: Under Mode sent to Radio, choose Mode RTTY to LSB.
Type RTTY in the Entry window to get N1MM+ into RTTY mode which will launch the DI-1 RTTY window and the MMTTY Engine 1 window.
On the DI-1 window click on Interface and choose MMTTY.
On the DI-1 window click on Setup, then click the 2nd menu item from the top to Turn Auto TRXUpdate on. This will make the compensation between your LSB dial frequency and the mark tone radio frequency based on the audio tone you use for mark.
On the DI-1 window click on Setup MMTTY (or from the RTTY Engine window click on Option.. Setup) which will open the 7 tabbed Setup Ver1.68A window.
On that MMTTY setup window, Demodulator Tab, set Mark to 1300 Hz. (You can choose something else as long as you set up your radio filters accordingly). Make sure the shift is set to 170 Hz.
On the MMTTY setup window, sound card tab, select the USB Audio CODEC under BOTH Reception and Transmission). These sometimes undo themselves, I don't know why.
On the MMTTY setup window, TX tab, under the TX section check the UOS box (Unshift on Space).
Now, on the TS-590SG (note that the menu #s on a TS-590S may be different)
Verify that the radio is in LSB mode; put in it Data mode by pressing the front panel DATA button.
Rotate the HI/Shift knob to select 1400 Hz. Rotate the LO/Width knob to select 400 Hz (or 300 Hz if you prefer).
Go to Menu 69 and select USB, since you'll be bringing audio in over the USB cable.
I have Menu 71 (Audio Level of USB input for Data Comms) set to 2 and Menu 72 (Audio Level of USB input for Data Comms) set to 8. These will be a function of how you set levels on your computer's sound card settings.
FWIW, here are my sound card settings on my Windows 7 machine. Note that my settings are probably a little non-standard and were set to make my QSORDER program I use for recording SSB QSO's happy.
Right click on the speaker icon in the tray in the lower right corner (system tray) and select Recording Devices. That brings up the Sound window with the Recording tab selected. On this recording tab note the level on the bar graph for the Microphone USB Audio CODEC. Click the properties button then the levels tab on the new window that comes up. Adjust the gain slider to get a reasonable level as shown on the Recording tab. (I have mine set all the way down to 2)
Back on the Sound window, click on the sounds tab and under the Sound Scheme pulldown select "No Sounds". Nobody wants to hear your windows "music" on the ham bands!
You may want to play with the level setting on the Playback tab in conjunction with Menu 71 on the TS-590 if need be.
I don't know if any of this is the piece you are missing to get it working, but hopefully it will help someone somewhere.
Victor - WB0TEV / V31VP