Quote from Maverick74:
It's a "little" bit more complicated then that. LOL. This is a controlled experiment you offered. You have the coin, you know it's fair and you know exactly the outcome possibilities. In the real world, you have thousands of coins you are flipping. You don't know if they are fair and you don't even know their distributions. So when this trader has a lucky run, we won't know if it's him or the coins. The coins in this example is analogous to the example I gave earlier regarding trading environment.
Indeed. There are so many participants and so many "coins", that it's incredibly difficult to ascertain whether a given participant is skillful or lucky, even in the presence of consistent outperformance.
A decent approach that has been used to evaluate fund managers (with depressingly poor results) is the comparison of investment/trading decisions with the result of random choices made under similar constraints. For those that are interested:
Discovering the quality of portfolio decisions:
http://www.portfolioprobe.com/2012/11/26/discovering-the-quality-of-portfolio-decisions/
Not Fooled by Randomness: Using Random Portfolios to Analyze Investment Funds
http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2143293