If the IRA custodian is not broker-affiliated then I see no reason why you couldn't have multiple brokerage accounts. For example, North Star Trust Company lets you open brokerage accounts with a ton of different brokers. You can hold your IRA investments in cash, a commodities account, whatever. You can chop and change brokers and move your money around between different brokerage accounts as you please. There is no rollover to worry about because the money has never left the IRA.
In essence, when the IRA custodian is neutral, you can open and shut brokerage accounts, pull money out of one account into cash, stick it in another broker account, as you please. The entire time your money remains under the umbrella of being an IRA at North Star. Because it's never the broker who is the custodian for the IRA.
There are other trust companies too. Millenium comes to mind. Google for others. Perhaps contact them to ask questions. By having most of the money in brokerage accounts within the IRA, it gives added peace of mind that if the custodial trust company ever went belly up or committed fraud, your money is physically held in segregated accounts at a brokerage house where they couldn't get it.
Professional advice would be a good idea, but it might be worth making some inquiries of these independent trust companies. Their main marketing jist is that they provide mucy greater flexibility than most traditional firms when it comes to IRAs.