Here is what you are missing:
The LED color relies on two things.
1. The position of the points (where are the points) and
2. Are they in the position requested? (are they where I want)
You must use both pieces of information to light the LEDs.
I would use switch machine contacts to answer question number 1.
I would use the toggle switch position to answer question number 2.
If 1 & 2 are yes, then light the LED green. You need to ask and answer
the questions for all four places (A, B, C and D).
I uploaded so typical circuits that satisfy the above conditions.
If you look at the first route on the diagram, B to D (or D to B), both sets of points are in a down position and both LED's are green.
Then look at the second route on the diagram, A to C (or C to A), both sets of points are in the up position and both LED's are green.
But, here is the problem as I see it. If both LED's are green in the first route (B-D) and the two sets of points are in the down position, then in the second route (A-C) with the two sets of points in the up position, the LED's would be red. The same is true for the other two routes as well. The B-C route would be red-green, and the A-D route would be green -red.
So, what am I missing?