Quote from Jerry030:
I don't have time to actually run this for you but I'll guess that somewhere between 200 and 2000 observations would give you some good confidence levels, depending on how you structure what you want to prove and the level of confidence required.
Similar work in pharmaceutical and consumer testing uses about this number of subjects. A recent consumer testing study on weight loss diets where I did the IT consulting but not the statistics had 460 subjects. This was enough to support a claim of efficacy with the FDA.
For Phase III trials most pharmaceutical companies want several thousand subjects as they have a large liability when a drug goes bad and they get a class action lawsuit.
Some good input by different people in this thread, please continue

I'm in the process of figuring out how to calculate the odds in my case.