I do not have direct experience with replacing a mechanical encoder with an optical one. You will at least need to run power to the new encoder, and verify that it outputs a low pulse rather than a high (NC vs NO type of switch). The program for reading the encoder value is based on measuring the voltage level with a pull-up resistor (internal to the Nano). This was done so that an analog input could be used for the connections. Normally that would have been digital inputs. There just are not enough digital inputs on the Nano.
So other than running the 5volt power to the encoder, and ensuring it is active low pulse output, that there are no more than 20-30 pulses per revolution, then there should not be any other changes required.
If you do go to active high, then you will need to change the logic in the Nano program.