For many of our retired brethren, trading is more of a hobby than a vocation. The hope is that by applying time to manage their investments that they can do marginally better than the mutual funds that they would otherwise have available. Since the mutual funds come across as sloppy and lazy, this looks easy. Best to test ones trading ideas before risking capital. Will it give you an edge, not. Will it stop you from making bad future trades, not. But it will provide a cheap initial expectation setting. If, after five months of investigation, the trader hasn't found any system which was profitable in the last two years of the back-test, then he knows its not that easy. Better that than getting emotionally attached and blowing half his retirement savings to find out the same thing.
I'm at step 0 regarding systems.
- Thread starter rainmonkey
- Start date
Agreed, there are many limitations to back testing.
Not to mention, it shouldn't be the first phase of system development, but it often is.
It is also true considering that most software packages available main focus is on technical analysis. Very few allow for data-mining, which should be the STARTING point in system development. When you mine data, you are able to discover things that the charts don't tell you. Because your eyes can't see the events and can't calculate their probability of occurrence.
Data-mining gives you ideas of what to expect. Just knowing the Open and the first directional move to a specific point (measured move) will give you the direction of the day in most stocks 60-70% of the time. This is simply because if I mine the data, I find this out because I am testing 2 events that are measured moves and determine that one price has a 50-60% chance of occurring and another that has a 70-80% chance of occurring there is only roughly a 30-40% chance that both will occur. This is done with data-mining, not back-testing. Total price-action, no news, zero technical analysis, and no human interference.
There is a wide range here due to once again, to an often over-looked aspect - the volatility and phase of the instrument. More volatile stocks bring the percentages down due to their intra-day movement.
This is similar to what I mentioned in an earlier post related to Gaps being filled by the SPY, DIA, Qs. There were two break-away gaps this past week in opposite directions and both closed the gap to their High / Low respectively. Both situations had a high probability of playing out. You find this information out by data-mining.
That's the discovery phase of system development and it's often left out. Not to mention, it's the most important because it can give you probabilities from which to develop a strategy.
Hey trhudson,
You have been an extremely informative poster, thank you for that.
It seems like the more I read about system development, the more I realize I know far less than I initially thought.
I'm still working on learning mySQL with XAMPP, found a tutorial about how to set up, and another tutorial about how to build a futures trading system (which I haven't started yet given that I'm learning XAMPP first).
I noticed that you mention data mining a lot, correct me if I'm wrong, but this is looking for patterns/events that occur not randomly and can be 'assigned' probabilities? Is this done through PHP?
Once again, thank you so much for everything.
Regards,
rainmonkey
You have been an extremely informative poster, thank you for that.
It seems like the more I read about system development, the more I realize I know far less than I initially thought.
I'm still working on learning mySQL with XAMPP, found a tutorial about how to set up, and another tutorial about how to build a futures trading system (which I haven't started yet given that I'm learning XAMPP first).
I noticed that you mention data mining a lot, correct me if I'm wrong, but this is looking for patterns/events that occur not randomly and can be 'assigned' probabilities? Is this done through PHP?
Once again, thank you so much for everything.
Regards,
rainmonkey
Quote from gianno:
You will be wasting your time and end up with nothing of value. System Writer Plus was invented in 1987 by a company called Omega Research, nowadays called Tradestation Technologies. I think the price was about 2K then. This is probably close to 15K in today's money after what Bernanke has done to debase the USD.
Using that program and others that came after it like Trading Blox and Metastock, to name a few titles, traders were able to extract every possible edge that can be extracted by backtesting.
Now, you come and you want to go back 30 years in the past, and use excel or some free tools that everyone else is using, I mean hundreds of thousands like you who have no money to spend and think that putting 0, I mean zip, down, can make them money.
Learn this simple mathematical fact, you will need it in your life:
0 x N = 0
where N is any number > 0
This is what traders use nowadays and it may not even be enough:
http://www.elitetrader.com/vb/showthread.php?s=&threadid=220286
Good luck, you will need it
If we applied your logic to trading in general, then theoretically, there'd be no point in trying to trade as individuals. I mean after all, anything you do, can be done by anyone else. You're basically saying that there's no point in trying to do anything since you've got literally hundreds of thousands, perhaps millions, trying to do the same thing with the same tools.
You sound like you're trying to discourage people from trading on their own. If that's the case, why're you even on these forums? Or perhaps you're trying to scare away new comers to lessen your competition. Either way, you're not being helpful.
And you're wishing ME luck? I wish YOU good luck, get out of your pessimistic hovel.
I can't speak for Gianno, but the premise, its-all-been-done-before, is both right and wrong. Economists, in their pleasure to have everything wrapped-up neatly, usually get this wrong.
True. When alot of money and talent are committed to exploiting an inefficiency, the inefficiency lessons. Either to zero, because the cause is fixed or to a point of diminishing returns. For example, a government stops offering to pay a coupon rate which is not justified. Aluminum can recycling becomes popular, but there will always be a few cans in the trash.
False. Despite the fact that thousands or millions of people have worked on the same problem, there are usually things that haven't been tried. The reason is that the landscape of knowledge if fractal.
I have seen this over-and-over again in software development. No matter how similar the problems are that two solutions are meant to solve. Every design decision, explicit or implicit, leads to different understandings, different reasons and different choices down the road.
I see this equally so with hedge funds and mutual funds. Two groups may be both long-short funds, work across the street from each other, and hire people with the same background, but there are a thousand differences in the details. The performance of the funds could well have no correlation.
The same is true for individual trading. You want to pick a method that is profitable, but all of the choices matter.
For this reason, I try to keep my designs very open-minded. Try to explore a few steps ahead on each path. Being pedantic for an hour can save weeks and months of work down the road.
Take calculated risks, but also don't be impatiently certain.
I have pointed a few people to a similar tutorial. However, the tutorial that you read, was a bad example. I say this because the author calculated the moving average in the spreadsheet before inserting it into the database. Not necessary and it's extremely limiting. I asked him, why didn't you calculate the MA in PHP, he said it would be too much work, but it could be done. So, as with everything else I have learned in my desire to create a successful system...If you want something done, you've gotta do it yourself.
This is exactly why I developed my own platform...all the others were basically useless based on what I wanted to do. Combine data-mining with system development. I am able to back-test intra-day 8 years of data and mine that data (trade results) to find higher-probability trades. I am also able to tell what patterns work and what patterns don't by mining the data. I will be adding a feature that will find out what conditions were present before the winners vs. the losers and see if patterns exist there as well.
With all the commercially available platforms, you have to re-write the code and run another back-test. This is the extent of their data-mining. Five years of data on tradestation took anywhere from 45 minutes to an hour and sometimes longer. Seems like a waste of time. Not to mention, at the time none of the commercial platforms could use a stop-loss that I was trying to implement. And they will all tell you that they are capable of working with multiple timeframes at the same time...it's just not as easy as some here will tell you it is. How do I know? I tried, paid certified programmers to write code and they gave up, said it couldn't be done. Now, I am able to test 8 years much faster. And better still I am able to mine the data (trade results). I honestly can't believe that all of the commercial platforms forgot this part. It's as if they were all developed by the same people who were all thinking inside the same box. The funny thing is the newer commercially available system development tools are getting worse, not better. They make them look 'pretty' with flashing lights and smooth sleek user interfaces, and let's not forget the 'pretty graphs' (because they are so useful...lol) so the public will like them and sacrifice what really matters - speed, functionality, and features (mainly data-mining). Remember, they are solely in the business to make money and they aren't really concerned whether or not you make money.
Just this week I was introduced to a new platform that allows you to 'visualize the market', what a bad idea. Totally going the wrong way with this one. Take the human OUT of the EQUATION, don't give them more ways to mess up a system.
Back to what we are here for:
There is really nothing to learn with MySQL. I do all of my
calculations and algorithms in PHP. This is because I ran into a wall trying to write MySQL code that would find certain patterns. Therefore, I re-wrote everything in PHP.
1. Download, Install XAMPP, and Run Apache and MySQL
Now you have a working web and database server on your local computer
2. Go to localhost/ in your browser and click on phpMyadmin
3. Create a database with 7 columns call it historical_quotes if you want
4. Name the columns:
symbol
date
open
high
low
close
volume
5. Download your data using whatever tool or website you want.
6. Import the data into the database. The date MUST BE IN THE FOLLOWING FORMAT:
YYYY-MM-DD
Many software packages will allow you to do this automatically. However, if you use excel, you can get it in this format if you open it in excel and change the entire column using the Format Column function. Select Custom and enter the letters exactly as I did above and it will store it correctly.
I may create a blog to demonstrate this as my posts are becoming very large and they may be useful to some and get lost in here.
This week I will demonstrate how to create a form that will pass a symbol to the database and display the quotes using PHP. I will also show how to do a few simple things with the data.
One good idea to do as well would be to develop this in a framework such as CodeIgniter. This will allow for some structure in your code and keep things organized.
When you start using a platform like this and get comfortable with your coding, you are going to see things you never saw looking at the charts, this I promise. It will change your mind about trading and system development.
TRH
The best edge is frontrunning. I'm up 100% in a few months with it. But with higher leverage I can get it to 200% next year.
Where you got that? You should read my post again. All I said is that zero investment will generate zero output for you. Below is a quote from my post:
Why do you think I am discouraging people from trading? Actually, I think I am saving them money. Look around you and be a realist. Do you think you will be able to compete with hedge funds that employ hundreds of PhDs and pay them 500K + bonus per year with excel and MySQL? Are you kidding us? Are you in any kind of a touch with reality? These people as failed_trad3r said will frontrun you. They will be faster in identifying market opportunities, get in the market before you and then when you come they will be happy to serve you any contracts or shares you ask for.
trhudson is right about data mining. This is the only way to get something going in the right direction. Backtesting is 30 years old practice. How to know what to backtest anyway?
He is also right about charting and other visual types of tools. They will get you nowhere.
At present, my point is that if you want to seriously compete and make money trading you have to invest at least 15K for software and hardware. This is absolute minimum. Let me give you an example. I was talking to a guy via Skype the other day who is running a commercial pattern data mining program on 25 high speed computers and he is using another commercially available program to analyze the results.
Do you think you can beat this guy in this game?
You know, you will need luck because you do not understand a basic market principle. I can't make money if I 'm in the market alone against professionals. I need to compete with losers. You are not competition to me. Actually, if I were a dishonest individual I would tell you that you are moving along in the right path because I would end up taking your money. Try to understand this. Try to appreciate what people are telling you. Nobody is 100% correct. Everyone has a piece of the truth.