Got to playing with the python code.
The reason for some of the serial issues:
VID = 5824 #was 1155
PID = 1155 #was 22336
Looks like the VID and PID are mixed and PID was wrong? Even though I was pasting the correct port in and it ran it keeps coming back and looking at the VID and PID to determine button and connection status so was getting confused and crashed.
Since you don't seem to have this problem something is different with your copy of nanoVNA-saver. Maybe I'm using the wrong one from GitHub.
I noticed that nanoVNA-saver is sending a 'finished' and a 'updated' command that is not handled and may be causing the ANVA to do another sweep.
Guess I need to try your hex file and see if it has these problems.