If i look on the diagram, R36 has one end grounded. So putting the pot on this side SHOULD produce zero output !
The pot is also a logarithmic version and sound level SHOULD follow a smooth curve.
I suspect that you have a defective pot or something is not correct in your wiring. Check, visualy and with an Ohm-meter, if the ground side of the pot is really connected to ground.