I have a motherboard with on-board sound that can be configured for playing different sources to speakers and headphones. Your motherboard or sound card may already have this option.
If not, I don't think software would be able to provide it, as it seems like a feature that would need to be enabled at the driver level - so that the different applications would be able to see different sound devices.
The alternative is to install a second sound card - they can be very cheap. The simplest would be a USB dongle type, which you can buy for about $10.