Case 1... pay as you go:
You start with after-tax income. So you earned $1200 taxed at 25% = $900 to invest
900(1.2) >> 1080 - .25(1080 - 900) = 1035
1035(1)>> 1035
1035(.95) >> 983.25
983.25(1.1) >> 1081.57 - .25(1081.57 - 983.25) = 1056.99
1056.99(1.25) >>1321.23 - .25(1321.23 - 1056.99) = 1255.27
Case 2: pay later:
You start with pretax income. So you get the full $1200 to invest:
1200(1.2)(1)(.95)(1.1)(1.25) = 1881
Then you have to pay the tax man:
capital gain = 1881 - 1200 = 681
tax on capital gain = .25(681) = 170.25
penalty on capital gain = .1(681) = 68.1
Net:
1881 - 170.25 - 68.1 = 1642.65
You have the right idea/intuition, but not quite the right math (or good enough returns to make the idea work out financially, although it could). As others pointed out, for a pretax IRA/401k, you pay ordinary income tax plus penalty if young on all withdrawals. There is no concept of capital gains within an IRA - it's just money in might get a tax break, money out might get a tax bill and/or penalty ("Might" referring to traditional vs Roth accounts with their different tax rules).
So in your case 2, it should go:
1881 - 0.25*1881(tax) -0.1*1881(penalty) = $1,222.65
Which is $30 worse than the taxable version in Case 1.
In general, doing this things with Roth money is more clear, since you dont have to be as careful about pretax money in vs aftertax money, different tax rates at different times, etc.
However, your general idea that compounding tax free within a retirement account can overcome the penalty, even if you need the money earlier than retirement age, is correct. Suppose a 33% tax rate as you go and Roth money for simplicity.
Case 1: taxable
$1000 in taxable account (post tax)
5 years of 30% pretax returns each year
So after tax 20% returns compounded
$1000 * 1.2^5 = $2488 (after tax money)
Case 2: Roth
$1000 in Roth retirement account (no tax break for contribution, same as after tax above)
5 years of 30% returns
$1000 * 1.3^5 = $3713
Cash it out, remembering that the original $1000 contribution isn't taxable for a Roth
$3713 - 0.33*2713(tax) - 0.1*2713(penalty) = $2546 (after taxes and penalties)
So you can come out ahead, you just need more time and better returns to overcome the 10% penalty. Obviously if you can, you should try to let this compound and live off other money since paying taxes on all those gains, plus penalties, is very expensive and an entirely avoidable cost if this ends up being your retirement stash.
Also, there's no problem with a 401k trading a lot, assuming you can get a brokerage option offered by the plan.