Yeah. Gold is not a convenient asset. If you invest indirectly, you have to have money to spare while that sits, hedging. If you hold gold bars, that will lack even more convenience. It's not like you'll pay for groceries in gold.
In short, only a small amount of people could afford to do that...