Cycles: using HDRI for lighting but removing from refraction

I’m using an HDRI to light a soda bottle, and while I want the HDRI in the glossy shader, I don’t want it in the refraction (it muddies it). I’m rendering with the Transparent Film box checked and need the bottle to be transparent and to refract itself, but not the HDRI. Is it possible to remove the HDRI from being caught by the refraction rays?

HDRI in reflections only:


HDRI in everything but refractions: