This is a factor, but not it's not THE deciding factor.Quote from moarla:
when you can show that you have an office there with employies working for you , and the decision making is there, it will be ok. but if you trade from Florida, no no my friend
Many multinationals have letterbox companies (even headquarters) with no physical offices or employees in Europe and the Caribbean and the IRS doesn't seem to mind.
On the other hand, the IRS might argue that your offshore physical office and employees have no economic purpose other than avoid tax and dismiss them as a sham and they might even accuse you of money laundering to shut you up, settle quietly and do whathever they want.