I wonder how can the business model of prop shops succeed
For three main reasons, I think ...
First, there are
some people with professional trading skills but without adequate capital to trade on their own, for whom it makes sense to use a prop shop temporarily (but "temporarily" can be quite a long time, and meanwhile they're helping the prop shop's business model to succeed, perhaps mostly in the form of volume commissions);
Secondly, there are many more people with what they
mistakenly imagine are professional trading skills but without adequate capital to trade on their own, to whom it seems to make sense to use a prop shop temporarily (and the rest is as above: the contracts are typically drawn up and the procedures executed in such a way that the prop shop's own funds are exposed to only minimal - if any - risk);
Thirdly, some so-called prop shops are making a significant part of their incomes by providing other services (I don't mean to suggest that that they're a "front" for escort services, but I think probably some of them are principally education vendors).
Why would they want to pay higher commission to a prop shop when Interactive Brokers is cheap?
A mixture of reasons, perhaps even including some social ones, but probably "undercapitalization" is the commonest.
I think there's a big turnover of these people, and I strongly suspect that the majority of them probably remain undercapitalized, long-term.
There's some turnover of prop shops, too, of course ...