If I'm understanding you correctly yes all of the caps are in parallel going from B+ to ground. Ideally you want the hot side of the cap with the shortest possible lead and let the ground side run as needed. Here's what Bruce had to say from one of his earlier missives:
"In decoupling applications, I have long been an advocate of multiple capacitors - each creating more rejection in the stop band. At the point of power entry,
an aluminum or tantalum with a resonance of about 100 kHz, then near IC's a 0.1uF for 10 MHz, in parallel with a 0.01uF for about 70 to 80 MHz, in parallel
I recommend a small series resistor in line with the + and - supplies, 1 to 10 ohms or so, with the bypassing right at the op amp power pin. This vastly improves power supply rejection ratio of the op amp. Also, at minimum, 33 pF NPO in parallel with the 0.1 uF cap to get rid of cell phone interference. Adding other caps in parallel such as 1000 pF NPO will improve FM rejection in the audio circuit."