1. Topics
  2. Question about XMLRPC - Line Feed/Carriage Return

Question about XMLRPC - Line Feed/Carriage Return #fldigi

makkiato@...
 

Hello,

I am trying to play a bit with FLDIGI's RPC.

I see a strange behaviour. Basically when adding a string of BYTES to the TX-widget, using the method "text.add_tx_bytes"

Any occurrence of the byte `\n` (anywhere in the string, not just at the end) is replaced with the the two bytes '\r\n',
where:
- \n is the ASCII Line Feed character, 0A in hexadecimal. 
- \r is the ASCII Carriage Return, 0D in hexadecimal.

I understand that \r\n is the standard in the Microsoft Windows world.

However this automatic replacement seems odd to me as it makes hard to transmit a byte sequence directly in FLDIGI.
One would have to first 'normalise' the string to be transmitted, like in Base64 or something, to avoid unwanted replacements. But this undesirable as it increases the size of the message.

Do you know if there is any way to avoid this behaviour, allowing correct bytes transmissions (using the RPC)?

Thank you all in advance!
Dave
 

fldigi does not convert the \n to \r\n during execution of the text.add_tx_bytes method.  What software are you using to
  1. create the string message
  2. send it to fldigi using an xmlrpc method (library)

David


On 11/29/22 17:40, makkiato via groups.io wrote:

Hello,

I am trying to play a bit with FLDIGI's RPC.

I see a strange behaviour. Basically when adding a string of BYTES to the TX-widget, using the method "text.add_tx_bytes"

Any occurrence of the byte `\n` (anywhere in the string, not just at the end) is replaced with the the two bytes '\r\n',
where:
- \n is the ASCII Line Feed character, 0A in hexadecimal. 
- \r is the ASCII Carriage Return, 0D in hexadecimal.

I understand that \r\n is the standard in the Microsoft Windows world.

However this automatic replacement seems odd to me as it makes hard to transmit a byte sequence directly in FLDIGI.
One would have to first 'normalise' the string to be transmitted, like in Base64 or something, to avoid unwanted replacements. But this undesirable as it increases the size of the message.

Do you know if there is any way to avoid this behaviour, allowing correct bytes transmissions (using the RPC)?

Thank you all in advance!
makkiato@...
 

Thank you Dave for the kind reply.

I am using FLDIGI (ver. 4.1.23, the latest version) for MacOSX.
I am interacting with the XMLRPC using Python3. 
The following sequence of commands (from standard python3 interpreter) illustrates the problem:

-------------------------------------
# Create RPC

>>> rpc = xmlrpc.client.ServerProxy("http://localhost:7362")


# Clean up transmitted data buffer.

>>> print(rpc.tx.get_data().data)

b''


# Add some bytes to the TX-widget. 
>>> rpc.text.add_tx_bytes(b'Hello\nWorld\n')

''

# Start transmitting

>>> rpc.main.tx()

''


# After a few seconds of transmission...

>>> print(rpc.tx.get_data().data)

b'Hello\r\nWorld\r\n' 
-------------------------------

As you can see, what has been transmitted (at least as reported by the tx.get_data() RCP method) is actually the string with \n replaced by \r\n.
Michael Black
 

Use \r instead of \n

Mike W9MDB




On Wednesday, November 30, 2022 at 06:48:22 AM CST, makkiato via groups.io <makkiato@...> wrote:


Thank you Dave for the kind reply.

I am using FLDIGI (ver. 4.1.23, the latest version) for MacOSX.
I am interacting with the XMLRPC using Python3. 
The following sequence of commands (from standard python3 interpreter) illustrates the problem:

-------------------------------------
# Create RPC

>>> rpc = xmlrpc.client.ServerProxy("http://localhost:7362")


# Clean up transmitted data buffer.

>>> print(rpc.tx.get_data().data)

b''


# Add some bytes to the TX-widget. 
>>> rpc.text.add_tx_bytes(b'Hello\nWorld\n')

''

# Start transmitting

>>> rpc.main.tx()

''


# After a few seconds of transmission...

>>> print(rpc.tx.get_data().data)

b'Hello\r\nWorld\r\n' 
-------------------------------

As you can see, what has been transmitted (at least as reported by the tx.get_data() RCP method) is actually the string with \n replaced by \r\n.
makkiato@...
 

Use \r instead of \n


Thanks Michael but that is not the point of my question.

My goal is to have an arbitrary binary sequence as input, and to send it out *as is* with FLDIGI, without any modifications.

Remark: Of course, I can always take a byte-sequence and 'undo' the replacement \n --> \n\r by applying a replacement \n\r --> \n. 
But I am not sure if this is the cleanest way to proceed. 

If anybody could tell me why (and when, exactly) the  \n --> \n\r is applied, it would be very useful. 

Thanks!
Dave
 

Are you building the fldig app from source or using one of the dmg's ?

David

On 11/30/22 06:59, makkiato via groups.io wrote:

Use \r instead of \n


Thanks Michael but that is not the point of my question.

My goal is to have an arbitrary binary sequence as input, and to send it out *as is* with FLDIGI, without any modifications.

Remark: Of course, I can always take a byte-sequence and 'undo' the replacement \n --> \n\r by applying a replacement \n\r --> \n. 
But I am not sure if this is the cleanest way to proceed. 

If anybody could tell me why (and when, exactly) the  \n --> \n\r is applied, it would be very useful. 

Thanks!
Michael Black
 

Python (and all other software) is will interpret \n as either LF or CR/LF -- it's OS dependent

\n is "new line" -- whatever that takes on the device being used.

Windows is well known to need CR/LF where Unix most always is just LF

So if you want a specific byte to be portable you must add a specific byte.

The \r is just a 0x0d which works everywhere.

Mike W9MDB



On Wednesday, November 30, 2022 at 06:59:51 AM CST, makkiato via groups.io <makkiato@...> wrote:


Use \r instead of \n


Thanks Michael but that is not the point of my question.

My goal is to have an arbitrary binary sequence as input, and to send it out *as is* with FLDIGI, without any modifications.

Remark: Of course, I can always take a byte-sequence and 'undo' the replacement \n --> \n\r by applying a replacement \n\r --> \n. 
But I am not sure if this is the cleanest way to proceed. 

If anybody could tell me why (and when, exactly) the  \n --> \n\r is applied, it would be very useful. 

Thanks!
makkiato@...
 

It is the file:

fldigi-4.1.23_BS.dmg

downloaded from https://sourceforge.net/projects/fldigi/files/fldigi/

>> cksum fldigi-4.1.23_BS.dmg
2046818309 14831010 fldigi-4.1.23_BS.dmg
Dave
 

You might want to learn how to build from source: https://sourceforge.net/p/fldigi/wiki/osx_howto/

that will give you the ability to execute using lldb, the MacOS debugger, to set break points, examine memory, etc.

fldigi source includes a Perl scripts, scripts/fldigi-shell, when provides a developer/user with the ability to test the various xmlrpc procedures



Note that the byte array is literal,  "\n" is not to be interpreted as a new line, but the characters '\' followed by 'n'

I think your problem is a result of the Python interpreter's string and byte array handlers.

David


On 11/30/22 07:20, makkiato via groups.io wrote:

It is the file:

fldigi-4.1.23_BS.dmg

downloaded from https://sourceforge.net/projects/fldigi/files/fldigi/

>> cksum fldigi-4.1.23_BS.dmg
2046818309 14831010 fldigi-4.1.23_BS.dmg
Dave
 

I modified the event logger in fldigi to show the hex characters received from the Python interpreter



This is on a Linux machine.  Behavior is idential on High Sierra, Big Sur, etc.  Note that the \n is not converted to \r\n when inserted into the TX buffer.  The conversion occurs during transmission.  It occurs to maintain compatibility with other digital modem software which expects the \r\n sequence.  Nearly 100% of the early development of digital modem software took place on M$ platforms, thereby forcing it's end-of-line sequence on the rest of the ham world 😭.

This occurs for all modem types in the function get_tx_char(), source file src/dialogs/fl_digi.cxx

    else if (c == '\n') {
        tx_encoder.push("\r\n");
    }
    else {
        char utf8_char[6];
        int utf8_len = fl_utf8encode(c, utf8_char);
        tx_encoder.push(std::string(utf8_char, utf8_len));
    }

If you want to transmit binary data you must first convert it to base64.

David

makkiato@...
 

Thank you A LOT Dave for digging this deeply in the issue.
I was going crazy trying to find where the issue was within the Python libraries, etc. 

Good to know that it is a design choice of FLDIGI, which is entirely reasonable.

If you want to transmit binary data you must first convert it to base64.

Unless I am missing something, this is not needed. 
 
I can just apply the substitution \r\n --> \n on any received data to get back to the original bytes sequence. This approach seems better to me in terms of message length. 
 
Thank you again!
Dave
 

That will work for the instance of \n.  You would need other escape sequences for \t \b etc.

Code review is always good for the gray matter.  😉

The UI for fldigi etal is FLTK.  Check out https://pyfltk.sourceforge.io/examples.php

Dave

On 11/30/22 09:44, makkiato via groups.io wrote:

Thank you A LOT Dave for digging this deeply in the issue.
I was going crazy trying to find where the issue was within the Python libraries, etc. 

Good to know that it is a design choice of FLDIGI, which is entirely reasonable.

If you want to transmit binary data you must first convert it to base64.

Unless I am missing something, this is not needed. 
 
I can just apply the substitution \r\n --> \n on any received data to get back to the original bytes sequence. This approach seems better to me in terms of message length. 
 
Thank you again!
1 - 12 of 12