I'm at step 0 regarding systems.

[trhudson]

There are many different forms and approaches to trading ...both intraday as well as short-term. While I will not discuss the details of my personal method, I will say that it is not scalping. Most people here believe that scalping is the only form of intraday trading. Once again people thinking inside the same box. If you get it in your head that you will lose, you should definitely not even try.

My method is based on statistics and math. It is versatile enough that while I am implementing a form of a breakout one could easily trade it as a pullback applying the exact same math and be profitable. Saying that I can be front-run is believable up to a point, but in the long run getting in front of me isn't important...getting out is where the profit and loss are calculated. Let me also point out that front-running this method will lead to losses if you hold it too long, I know, I already calculated the probability. You have to remember, my platform was developed to trade my method using unlimited variables. Using this method, a high frequency trader is not my direct competition, he's scalping and I am looking for a larger move.

Over the next few weeks, I will focus on Pivot Points. While I don't use them and have never tested them using data mining, I'm going to help you develop a data-mining platform that will tell you if it is at all possible to create a system using them.

As for system development, I totally agree I spent much more than that. Not too mention, the most valuable input Is knowing what to test. That is priceless.

 

[trhudson]

Quote from gianno:


At present, my point is that if you want to seriously compete and make money trading you have to invest at least 15K for software and hardware. This is absolute minimum. Let me give you an example. I was talking to a guy via Skype the other day who is running a commercial pattern data mining program on 25 high speed computers and he is using another commercially available program to analyze the results.

Do you think you can beat this guy in this game?

He is playing a different game, he's a high frequency trader...look outside the box. That's not the only game, the market is open 6.5 hours. If you want to jump in that game, go ahead, but as you said it's going to be tough. However, it's a totally different game.

What's his average holding time? How do you know that we would even be on opposite sides of the trade?

You have made a few of assumptions and obviously don't understand that if this guy is looking to make .10 to .25 on a trade in a stock that trades on average 2.00+ from high to low a day, it's a non-event. You act like this guy is ruling the world and can move the market for 6.5 hours straight...damn, I wish he could, cause I would bet we'd be on the same side of the trade.

Do you have any idea what the probability of the market opening and closing within 30% of a day's range to each other is? If a stock trades on average 2.00 from high to low, and the open and close are within .60 of each other, in the middle of the range (which by the way is a low probability), there was still .70 on each side of that range from which you could profit. So, if you are trading for .30 in a stock that moves 2.00 / day you have only made 15% of the move. If you are correct about the direction, you can make at the very least, a little more.

Do you have any idea what percentage of the time a stocks open is within 30% of it's high or low? Any idea what the probability of a stock filling a gap to a high / low or close is? These are all statistics that can be used to help create a trading system that isn't scalping nor high-frequency and their holding time may be hours.

 

[zedDoubleNaught]

Quote from trhudson:

...
Do you have any idea what the probability of the market opening and closing within 30% of a day's range to each other is? If a stock trades on average 2.00 from high to low, and the open and close are within .60 of each other, in the middle of the range (which by the way is a low probability), there was still .70 on each side of that range from which you could profit. So, if you are trading for .30 in a stock that moves 2.00 / day you have only made 15% of the move. If you are correct about the direction, you can make at the very least, a little more.
...

I think I understand your points about data mining. I realized some of the custom indicators I've made for my backtesting software are not for signals or trades but for gathering information, it occurred to me this may be data mining.

When you get your probabilities, are you interested in how they vary over time or across time intervals, or samples, or instruments (ie, any way to group data)? Or do you go for one probability figure based on the 8 years of data?
thanks

 

[goodgoing]

Quote from trhudson:

While I don't use them and have never tested them using data mining, I'm going to help you develop a data-mining platform that will tell you if it is at all possible to create a system using them.

IMO this is a wired use of the data-mining term. Usually, data-mining invloves backtesting a number of rules on the same data sample and selecting the best performing ones based on some criterion.

Maybe I missed something. Could you clarify a little your methodology?

 

[trhudson]

Quote from goodgoing:

IMO this is a wired use of the data-mining term. Usually, data-mining invloves backtesting a number of rules on the same data sample and selecting the best performing ones based on some criterion.

Maybe I missed something. Could you clarify a little your methodology?

What you are describing is data-mining, but it's not a phase of initial discovery. You have already made assumptions by creating a system and back-testing it. You may be mining your back-tested data to determine the best method, but that's hardly using data mining as a tool of discovery. You are discovering what method you back-tested presents the best results.

Data mining is a tool of discovery. If you have already assumed something to be true, what are you looking to discover?

In this example, I am using it to determine what has a probability of some event occurring and then attempt to create a system based from those probabilities.

 

[trhudson]

Quote from zedDoubleNaught:


When you get your probabilities, are you interested in how they vary over time or across time intervals, or samples, or instruments (ie, any way to group data)? Or do you go for one probability figure based on the 8 years of data?
thanks

I am interested in how they vary, but the variance often depends on the volatility of the stock. Stocks that trade in wide ranges are very similar and stocks with small ranges are similar.

Market Based ETFs are much different than Sector Based ETFs because on an intra-day basis, the Sector Based ETFs trend much better.

The data could easily be grouped and this could be done in the code.

 

[Mr_RC]

Not sure it has been mentioned but Tradelink is quite nice. You may have to patch a few things here and there but that is the nice part too, Open source.

 

[trhudson]

Quote from Mr_RC:

Not sure it has been mentioned but Tradelink is quite nice. You may have to patch a few things here and there but that is the nice part too, Open source.

I would not call Tradelink a data-mining tool. It is an open source automated trading platform with back-testing capabilities.

It seems like it could be useful to somebody that wants to learn how to use it. And you would definitely have to piece a lot together. Apparently there are numerous programs inside TradeLink that you must open to perform many different functions.

I looked at it before developing my platform as a base platform to start from and my programmer decided that would be a bad idea. He saw a product that wasn't quite organized the way he wanted it. He had written many trading programs for various products, so I took his word for it. My platform actually does the exact same things, back-tests, and records quotes for playback, but my user interface is much more organized and I don't have to open another program to get something done.

Not to mention 8 yrs of data for back-testing held in a database with direct access from the platform. My results are stored in a database so that I can retrieve them and mine the results if I so choose. The algorithm settings that made the trades are held in the database. It can also automate back-testing with a spreadsheet that feeds it symbols, dates, strategies and you can view the results later.

 

[zedDoubleNaught]

Quote from trhudson:

I am interested in how they vary, but the variance often depends on the volatility of the stock. Stocks that trade in wide ranges are very similar and stocks with small ranges are similar.

Market Based ETFs are much different than Sector Based ETFs because on an intra-day basis, the Sector Based ETFs trend much better.

The data could easily be grouped and this could be done in the code.

My problem is I can get a probability estimate from the data, but the estimates vary across samples. Eg the probability from Jan-Apr is different from the probability found in May-Aug. Then, if I get the probability estimate from all of my data instead of smaller samples, they all tend towards 50% or some value that makes a decision more like a guess.

But "wide ranges are similar and small ranges are similar" ... that reminds me of the NOVA I saw on Fractals. I think I should mine my data as looking for the whole data set probability, then split the data in half to get 2 estimates, then split those in half for 4 estimates, ...etc, then use the estimates that show the least variance from larger to smaller sample sizes. Thanks for the idea, this will be interesting to research !!

On another topic, how long did it take you to code your platform? I estimate, for me to code a platform from scratch, would take me anywhere from 2-6 months, then I would probably have a few undiscovered bugs.

 

[trhudson]

Quote from zedDoubleNaught:


On another topic, how long did it take you to code your platform? I estimate, for me to code a platform from scratch, would take me anywhere from 2-6 months, then I would probably have a few undiscovered bugs.

It was a part-time project and I started with one programmer who had numerous issues and he had a 'mac' (that should explain a lot), thought he knew what he was doing. So, six months later I found a guy who really did know what he was doing. The trading part didn't take all that long, but the data-mining, database development and getting the data for the platform took quite some time. I hired my first programmer 3 years ago and we went nowhere for 6 months. It was trading within a month of my second programmer being hired.

The project was just going to be a simple trading platform, but it turned into a much bigger project. In the initial phase, my programmer set it up to record the data and I decided to give it the ability to back-test as well. I also tested the platform to make certain that my back-tested results matched my live trading results. Then came the data-mining another feature that just simply isn't available.

I must say that the back-testing and data-mining features were really eye-opening. I had no idea that I could apply the exact same rules everyday and make money (not everyday, but over 50% of the time) over the previous 8 years.

We are looking to possibly allow others to use it and develop their own strategies and this will be in C# code, but will come with a few explanations. However, this will take a few months.