The Swan Island fort tramway (from Queenscliff) was installed in 1883.
Other Port Phillip sites that definitely had tramways are Point Nepean and the South Channel fort.
If other defence sites that are not strictly 'forts' are included we might include the Williamstown torpedo store (by 1886) and perhaps military magazines.
Colin