Honestly, I can understand and do share the same feeling in this.
First thing I tend to notice is if the broker is regulated with strong regulation bodies, because if they are, it means that they can be trusted and relied on.
But yes, even I do sometimes think of the fact that even a regulated broker can also scam me. This is not uncommon to think of. But what can be helpful is to see if those broker are member of ICF (Investor's Compensation Funds), which is a good indicator to tell you that the regulated bodies will have to compensate the funds to you, when your broker goes south.