Pseudo-random runs around Visual Studios

[TSGannGalt]

Quote from propseeker:

imo, you're being a little pretentious and specific in your particular and personal use of the word 'backtest'. a backtest is just that: a test on historical data. it can be anything imaginable, including but not limited to risk assessment, but also expectation. your indicator research was a 'backtest'. there is no singular 'proper' way, as the 'proper' way can only be defined by the hypotheses of the test in question.

i get your point on one level, but you're contradicting yourself by communicating it as specifically as you do.

ps
also, maybe this is a case of differing semantics like you mentioned earlier, but, assessing risk without expectation is a bit meaningless. there is always an expectation for any given risk, or else why asses or take the risk?

Well... you're partly correct. I used the term "backtest" from 3 standpoints. Apologies for the confusion.

I say partly because the significance and the nature of a tool becomes a completely different thing based on "how" it is used, "why" it is used, and "who" (what perspective) it is used.

OK... I'm starting to sound like jack hershey... let me restate this...

Most people on ET look for a clean slope on their equity curve, hence you read a lot of "System looks good". Further more, if the model covers all the costs and slippage, you read a lot of "System works". Filter that more, you have a bunch of people who run Monte Carlo and out-sample / forward testing, you read a lot of "System is robust". Filtering it even more, there's a little crowd who wrote about the importance of having a set of contingeny plans of when to cut the model (on/off switch). Filter that more and you end up with 6-10 posters who has discussed about potential ways of managing them as a portfolio. (Agreeing to it and actually writing about it are 2 different things.)

But then... the 6-10 posters never write about the model development prior to the management. Because to manage a portfolio, you need to test them properly, it's already a given.... So there is a major gap between systematic traders who are struggling with models and those who are struggling with management.

It's kind of like... posters writing about ways to implement a RTM models. We discuss or hint the bits and pieces of the model's components. But we rarely write about how to be sure that the tests are done correctly. It's pretty much a given that you know it already. Though, there's a definite gap between "how they are backtested"

Anyways... I agree. I didn't word it correctly. I'll watch out from next time.

 

[TSGannGalt]

Quote from TSGannGalt:

Back to the topic...

A while back I ran a series of tests about how Technical Analysis work together. I was trying to find a methodical way for combining different indicators. For example I ran the following test:

....

* I'm well aware about the fact that the results can change based on the type of tests I ran. Looking back, the 90+% is a bit "curve-fitted" towards an idea, but my point still stands.

Anyways... still waiting for some thoughts and opinions... Also... more coming later.

Quote from TSGannGalt:

Actually, I was going to write about predictive value...

....

* All the things I write are based on very weak propositions.

Adding...

I've already known that the test results are flawed and very biased...



Also...

I'm not dissing on anyone. I just find a large gap between different posters in ET.

Also, the whole issue is not psychological, from my point of view. Some of it has to do with the backtesting software you use. Your conclusions are limited on the reports you get from the software you use. This business as a whole has limited information in the public, systematic trading is somewhat a combination of treasure hunting for resources on the internet and trial/error.

 

[MarkBrown]

something i have learned from using industry standard backtesting is that you need a much better systems then what you think to get the job done.

so if you happy to make 150 per trade on the emini sp with 150 risk and win about 62% you better shoot for 350 a trade with 150 risk and 68% winners.

also you need to take the opto report and not take the best results in the bunch - you need to take the settings that are in the 75% range. the two items alone combined will eliminate most every systems you will make.

 

[TSGannGalt]

Quote from MarkBrown:

something i have learned from using industry standard backtesting is that you need a much better systems then what you think to get the job done.

so if you happy to make 150 per trade on the emini sp with 150 risk and win about 62% you better shoot for 350 a trade with 150 risk and 68% winners.

also you need to take the opto report and not take the best results in the bunch - you need to take the settings that are in the 75% range. the two items alone combined will eliminate most every systems you will make.

Agreeing mostly...

1. Definitely agree with the industry backtesting that it needs to get better.

2. :eek: 150 trades >>> 350 trades... This largely depends on whether the market allows you to. I think it was you (or Mark Johnson???) who mentioned this a whole lotta years back on Curtis' site, my answer to this "challenge" was to develop another model that utilizes the same tendency with different entries. Of course, it's not the best way because if one fails, you have another one failing with it but that's what I did. I'd love to get some advice on this.

3. This is a toughy... Reports for a single run are generally more detailed and opt. reports obviously becomes very "focused" based on your needs. I tried binding test results on 3D charts but my brain doesn't function well enough to perceive something out of it.

Also, more than 1/2 of what is significant in one report / test is useless for another report / test. I really haven't had time to consider this until you mentioned this but if I have time, I need start thinking about a reporting paradigm that automatically detects the "probable" significant measures based on the type of results the system shows...

Anyways, thank you for your post.

 

[MarkBrown]

of course two things when i talk i am being very general in terms and numbers. also i am exercising talking out loud rather than really at someone. so it may sound like i am talking at someone rather that just talking to think through some of my own thoughts.

i have lots of things that make it possible for me to do well even making as little net as 5.00 a trade off the emini's - fast feeds low clearing cost rebates soft dollar stuff etc. so for me i can break even on most systems that others would go broke on.

i have found generally there is a scale of human wants and deliverance. the human wants 98% winners and will justify the drawdown to get there. the other spectrum is the 20% winner who only wants to hit home runs - enduring many losses to get there.

generally higher percentage winners equates out to lower profits per trade. to achive high percentage winners you typically have to endure larger risk on a per trade basis.

exactly the opposite for low percentage winning systems - they typically make huge profits and have small losses.

so lets say you have a system that wins 50% and makes 100 and risk 100 you would still be a loser cause of cost. however i could trade that model and prob make money.

this is because i understand randomness and embrace it as my own. even randomness can have texture and clusters of sortable events. which with enough data can give one a feeling which could be quantified.

m

 

[TSGannGalt]

Quote from MarkBrown:

of course two things when i talk i am being very general in terms and numbers. also i am exercising talking out loud rather than really at someone. so it may sound like i am talking at someone rather that just talking to think through some of my own thoughts.

i have lots of things that make it possible for me to do well even making as little net as 5.00 a trade off the emini's - fast feeds low clearing cost rebates soft dollar stuff etc. so for me i can break even on most systems that others would go broke on.

i have found generally there is a scale of human wants and deliverance. the human wants 98% winners and will justify the drawdown to get there. the other spectrum is the 20% winner who only wants to hit home runs - enduring many losses to get there.

generally higher percentage winners equates out to lower profits per trade. to achive high percentage winners you typically have to endure larger risk on a per trade basis.

exactly the opposite for low percentage winning systems - they typically make huge profits and have small losses.

so lets say you have a system that wins 50% and makes 100 and risk 100 you would still be a loser cause of cost. however i could trade that model and prob make money.

this is because i understand randomness and embrace it as my own. even randomness can have texture and clusters of sortable events. which with enough data can give one a feeling which could be quantified.

m

I understood your previous post and if you were directing your reply to me, you probably would have quoted my post. No biggie.

Now that you mention it, it's good to hear someone mentioning "Randomness" from a systematic trading perspective. You rarely hear about it.

The topic was hinted by another, who mentioned about making money through random behaviors in the market. I wouldn't conclude that it's the only way the person views it but his example given was based on Mean Reversion. Given that the market tendency that the market moves with in a range, the random market moves will be "random" (or "non-biased") acting as a noise within the tendency. For this example, it will be making moves within the boundaries of the x-Sigma of the tendency "wrapping" the randomness. Hence, the market will flunctuate creating arbitrage opportunities.

Would "wrapping" tendency be the same idea as what you mention as "texture and clusters of sortable events"? Or am I missing more about the topic?

Thank you in advance.

 

[MarkBrown]

well of course what all you said. that was my point there are systems - that everyone builds. then there are events which happen within the boundaries of the system which provide other opportunity.

this is probably the hardest thing i have ever coded because the trick is to not interrupt the primary system yet be back on even keel when the positions change. really some deep shit right there.

i have accomplished this all by using switches, logic and just eyeballing with my developed feel. then when i get close i let the numbers run their business and that tells me further what is reality.

as a whole i would say most everyone is still looking for a method let alone developing methods within methods, that is rare and mostly scoffed at being impossible. it's like a song writer who only writes songs around notes that he himself can hit.

this is what makes great thinkers - is to know you are ill equipped and accept there is more out there than fits within your minds boundaries. flip back the top of your skull and let the knowledge rain in - i say to myself.

m

 

[TSGannGalt]

 

[asiaprop]

just curious, where in this large gap do you actually place yourself? I have a hard time seeing where you try to get to with this thread, you seem to have touched on pretty much all aspects of system design, emphasis on "touched". I am not a coder but place myself more in the discretionary trading corner, maybe thats why I kind of fail to see how you actually try to get to isolate a tradable, persistent edge...

...which system are you working on...why "Visual Studio" in the title...what about correlation between different strategie (correlation of what actually?)...suddenly serial correlation...and. and.and. That in itself looks very uncorrelated...maybe I miss years of earlier posts?

anyways...the posted text files reminded me on my old days at CMU, thanks for that déjà vu... ;-)

M

Quote from TSGannGalt:

Adding...

I've already known that the test results are flawed and very biased...



Also...

I'm not dissing on anyone. I just find a large gap between different posters in ET.

Also, the whole issue is not psychological, from my point of view. Some of it has to do with the backtesting software you use. Your conclusions are limited on the reports you get from the software you use. This business as a whole has limited information in the public, systematic trading is somewhat a combination of treasure hunting for resources on the internet and trial/error.

 

[TSGannGalt]

I'm involved in another web venture so I shouldn't be posting much in ET (last thing I want is to be thought that I'm advertising...)

Last post I'll be making in quite a while...

1. I rant randomly. Hence the pseudo-random.

2. I place myself as one of the top 3 "systematic" trader in the whole universe of trading. Why? Because people in GSAM, LEH, Citadel and other firms told me so. To be fair, I was also noted that:

- Worst quant. ever. Because I just don't know shit about pricing models. Seriously, I still don't understand Greeks and I don't trade options.
- Bad portfolio manager. All my buy/hold stock picks never make money.
- Reasonable IT. There's plenty of people better than me.
- Pretty good discretionary trader. I used to be an outright prop.

3. I'll come back to this thread... some time soon. Until then...

hasta la vista... pinchi bendejo