scroll/zoom conflict

When I scroll through a notebook using the mouse wheel, I may pause while hovering a graph, in which case the graph captures the next wheel movement and interpret it as zoom in/out.
Is there a way to force the use of shift+wheel for zoom instead of just the wheel?