It really depends on what you're trying to do.
If all you trade is front month verticals, i.e. you're using options spreads as a way to trade delta I more or less agree.
However, it ends right there. All other combinations and especially the market neutral ones are really subject to changes in the price of optionality which basically is IV.
On top of that, IV changes your delta profile, meaning that if you for example buy a +gamma call spread and IV goes down, your delta goes up which increases your risk. As soon as the market trades towards your short strike and IV increases, your delta decreases.
The more you go out in time, the more prominent these IV changes will be.
So absolutely do not discard these variables as meaningless, because they have a huge influence on your performance