Trading System Development

[AdrianHagh81]

For what the OP is trying to do, sma crossovers. Then yeah Tradestation will do just fine.

Found some new buzz words on the internet? Adrian or whoever you are, I am not condescending I just do not have much patience with those who try to impress or otherwise talk about things they know little to nothing about.

Your mere mentioning of Multicharts or Tradestation reveals that you are not in this for a career. Nobody in the professional space would ever touch the platforms you mentioned simply because they are not proficient and do not expose enough functionality. So here are my 3 questions for you that I promised to ask:

I'm not trading professionally, but am making money while keeping my day job.
Last few months been earning more from trading then my job

I think for the sort of question asked in this thread, it is directly assumed that the OP
is probably retail like me.

1) How does your platform reliably handle non-base currency generated p&l? Can you easily reconcile trading in assets that are denominated in different currencies?

I don't trade anything right now that generate non base currency pl, and most
retail traders don't.

2) Can your platform(s) easily handle different asset classes? Currency options, futures options, index options, ETFs, bonds, bond futures, commodity futures?

I don't do options much, except to hedge.
Currently execute it manually and use optionvue to model greeks.

3) Does your platform enable you to analyze time series as detailed as tick based quoting and trade reporting? How performant is it? Can you visualize single ticks and look at micro market dynamics in a visual sense and/or statistical sense?

For any retail affordable datafeed there is no accurate tick based quoting.
Well, theres the tick chart option in multicharts.

anything fancier gotta export the whole
thing to R or mathlab and do your own visualization.

I can assure you that non of the platforms you mentioned are capable of the 3 above. But please provide insight if I missed something or am not fully up to date. You might say that you do not engage in hft and hence do not care about tick based data. But you should be concerned with such as well because you are directly exposed to micro market dynamics at the time of entering and exiting positions. If you find this condescending then I do not know what else to say. Again, if your Multicharts approach works for you then great for you, it would not for me.

theres the extended backtesting mode with bar magnifier, that allows me to do tick by tick backtesting with bid ask spread. It lets me model fills more accurately especially on markets where spreads are volatile and see which tick my entry landed on.

Yes, besides MC .net that is sometimes linked to my external R or mathlab module
I have 16 HP blade 460 servers running 24/7 crunching out alphas.

Thats pretty decent setup for a retail guy like me don't you think ?


Anyways most of the post in this forum is asking technical stuff that is achievable and
doable by retail anyway.


So mister HFT Punter, just how big is your AUM anyway ?

 

[volpunter]

* Why would you need a day job if you were remotely profitable on the trading side? Maybe it is worth listening to those who make a career and living out of what they do best. That applies to all sort of skills and industries.

* Lol, even trading 2-3 currency pairs would expose you to non-base currency (or at least divergent non-base currency) p&l. Just because you do not trade such and hence do not care about it does not mean that the platform you mentioned are sufficient in handling non-base currencies. Most everyone who trades for a career most likely has had p&l exposure in different currencies.

* So your platforms fail on the options side as well. Hence the point I made that your mentioned platform suit the absolutely most basic needs of retail punters, those that you describe you are yourself. Again, I wished you well for what you do but it remains basic and non-professional. More like a hobby than a serious endeavor.

* No tick based data feeds that are relatively affordable? Sure if you are hackling and having sleepless nights over whether to spend 30 usd over a close-to-free charting platform then you most likely will have issues to invest a few hundred dollars per month in a quality data feed.

* You wanna export tick based data to R and crunch numbers? Why dont you do that with a simple set of 30 or so securities over a year worth of data and around 100 million data points and report back when you are done. R is about the worst tool to do such. Please talk about things you truly know about rather than making slippery assumptions.

* You mean random ticks between bars? Lol. Have fun with that and your assumption a random number generator replicates reality. If not random then why would you not just run your backtest over tick based data but called it some fancy term "bar magnifier"???

You have "16hp blade 460 servers crunching out alphas"??? Lol not sure what it is and what you are trying to say but using MultiCharts and talking about server hardware makes me chuckle. It just does not sound right.

Listen, I am not trying to be confrontational but your talk sounds very BS-y. Sure, what OP looks to achieve can be done on very simple hardware and software but he also asked about architectural questions and I gave the simple advice not to start with SQL which is precisely when you went onto your retail-punter-rant and attacked me how I would dare to say SQL is not sufficient. SQL is NOT GREAT for time series analysis, whether you deal with 10 rows or 10 billion rows, period. You and also I do not have any idea what OP wants to do next. Can your platforms for example dynamically run screeners and you submit a script dynamically (without stopping your platform) which the screener runs over and applies to existing or streaming data? No, Multicharts cannot nor can Tradestation, so for heaven's sake, give me a little credit for knowing what I am talking about. You can hate me and the way I deliver the message but the message nonetheless is a message based on a lot of expertise in this area.

For what the OP is trying to do, sma crossovers. Then yeah Tradestation will do just fine.




I'm not trading professionally, but am making money while keeping my day job.
Last few months been earning more from trading then my job

I think for the sort of question asked in this thread, it is directly assumed that the OP
is probably retail like me.


I don't trade anything right now that generate non base currency pl, and most
retail traders don't.



I don't do options much, except to hedge.
Currently execute it manually and use optionvue to model greeks.



For any retail affordable datafeed there is no accurate tick based quoting.
Well, theres the tick chart option in multicharts.

anything fancier gotta export the whole
thing to R or mathlab and do your own visualization.



theres the extended backtesting mode with bar magnifier, that allows me to do tick by tick backtesting with bid ask spread. It lets me model fills more accurately especially on markets where spreads are volatile and see which tick my entry landed on.

Yes, besides MC .net that is sometimes linked to my external R or mathlab module
I have 16 HP blade 460 servers running 24/7 crunching out alphas.

Thats pretty decent setup for a retail guy like me don't you think ?


Anyways most of the post in this forum is asking technical stuff that is achievable and
doable by retail anyway.


So mister HFT Punter, just how big is your AUM anyway ?

 

[cjbuckley4]

@volpunter will the SQL column store index not decrease the number of joins in the same way that a full on column oriented DB like monetDB or kdb+ would? My understanding of the main issue with using SQL (the kx website has a few articles on this subject) was that the relational model suffers mainly because of the multiple joins, so when I read this https://msdn.microsoft.com/en-us/library/gg492088.aspx, it seemed to imply that issue has been largely mitigated by this feature. I totally agree that a column oriented approach is the way to go here just from an algorithmic standpoint, so could you briefly explain the difference between this column index and the result of using a column oriented db?

And yeah, I see what you're saying about teafiles. The only real advantage a teafile confers over binary is that the structure of the file will be provided by the file, similar to HDF5. So, for our intents and purposes, the format of each asset's file will likely be identical, so having the file tell us it's struct before we read it isn't going to be necessary in this case. I still think its interesting though, might play around with it a bit when I have time. I think it's a simple and clean idea that could be really useful. Some of the people in industry on other forums seem to have taken an interest in it as well.

 

[volpunter]

I do not know enough about how SQL (particularly MSSQL) implements its columnstore indexes to comment with insight. But I do know that even a 10x speedup over a well indexed rdbms will not even come close to the performance of pure columnar based databases. So I do not bother learning more about the SQL approach. If you know which tool is best in this case why don't you then just use such tool? Nobody asks you to learn Q and handle kdb. You can just pick up HDM5 or other approaches and get a columnar db setup in less than 2 hours.

The schema/header of a teafile is actually very similar to how I have implemented my own binary data store. I keep information in the header such as symbology, what type of data is contained, how large each datapackage is (which is essential knowledge for a binary reader), how many data points are included (not necessary as you can deduce from the size of the binary file and the size of each data point after you subtract the size of the header), and other information. Everything else is basically elementary operations on binary data. The basic idea and underlying schema of a teafile is good imho but how they then implemented the storage and retrieval of such data is not optimized at all.

@volpunter will the SQL column store index not decrease the number of joins in the same way that a full on column oriented DB like monetDB or kdb+ would? My understanding of the main issue with using SQL (the kx website has a few articles on this subject) was that the relational model suffers mainly because of the multiple joins, so when I read this https://msdn.microsoft.com/en-us/library/gg492088.aspx, it seemed to imply that issue has been largely mitigated by this feature. I totally agree that a column oriented approach is the way to go here just from an algorithmic standpoint, so could you briefly explain the difference between this column index and the result of using a column oriented db?

And yeah, I see what you're saying about teafiles. The only real advantage a teafile confers over binary is that the structure of the file will be provided by the file, similar to HDF5. So, for our intents and purposes, the format of each asset's file will likely be identical, so having the file tell us it's struct before we read it isn't going to be necessary in this case. I still think its interesting though, might play around with it a bit when I have time. I think it's a simple and clean idea that could be really useful. Some of the people in industry on other forums seem to have taken an interest in it as well.

 

[Handle123]

Um, ever heard of easylanguage ? use multicharts the easylanguage edition with
portofolio backtester.


But anyway, it's just EOD data, with 2K stocks 10 years its just 5020000 data points.

AND your using very simple indicators , RSI and SMA

I'll just write something up in Java and GA for a possible optimum solution.
Since it's mostly linear so there won't be any local maxima trap i think.

WOW, I started with Super Charts which was first attempt to do backtesting before Tradestation came out, then many versions of Tradestation where you first bought the software before they came out with month subscriptions. I did all my testing back then until certain limitations were stopping me from testing more than one contract of tick data. MultiCharts certainly a very good software to have and get started on and go on to more advanced tools or making their own platforms. One stays with very long trading off weeklies, I think advanced software just a waste most of the time depending on timeframe. But as I get older I see K.I.S.S. to actually better with good skills on price patterns than complex systems.

 

[AdrianHagh81]

@volpunter will the SQL column store index not decrease the number of joins in the same way that a full on column oriented DB like monetDB or kdb+ would? My understanding of the main issue with using SQL (the kx website has a few articles on this subject) was that the relational model suffers mainly because of the multiple joins, so when I read this https://msdn.microsoft.com/en-us/library/gg492088.aspx, it seemed to imply that issue has been largely mitigated by this feature. I totally agree that a column oriented approach is the way to go here just from an algorithmic standpoint, so could you briefly explain the difference between this column index and the result of using a column oriented db?

See the attached File for more in depth explanation on mssql column index implementatoin

* Why would you need a day job if you were remotely profitable on the trading side? Maybe it is worth listening to those who make a career and living out of what they do best. That applies to all sort of skills and industries.

I make 200 grand a year on my day job and can grow silicon right next door
with access to all the toys I could ever want. Why would I leave when they allow me
such flexible work time ?

* You wanna export tick based data to R and crunch numbers? Why dont you do that with a simple set of 30 or so securities over a year worth of data and around 100 million data points and report back when you are done. R is about the worst tool to do such. Please talk about things you truly know about rather than making slippery assumptions.

Most people use R and Mathlab for the handy Matrix functions and other calculus
aproximation, just take a look at the library, and call it directly from your own C program.
problem solved.

* You mean random ticks between bars? Lol. Have fun with that and your assumption a random number generator replicates reality. If not random then why would you not just run your backtest over tick based data but called it some fancy term "bar magnifier"???

It is the actual tick data, most signal are based on m1> OHLC data, but for precise
entry and slippage info the bid/ask tick by tick info is provided in the bar magnifier
function.

You have "16hp blade 460 servers crunching out alphas"??? Lol not sure what it is and what you are trying to say but using MultiCharts and talking about server hardware makes me chuckle. It just does not sound right.

I've developed my own pattern finding tool in java and some C called with JNI.
its a GA and uses SVM + NN as the final filter condition.
I can set the treedepth, tournament size, kernel type and node number via parameter.
I unleash them on m1> OHLC data, then if they find any intresting patterns
implement that in MC .net to verify the results and get a more visual feel and
more detailed performance report.

Can your platforms for example dynamically run screeners and you submit a script dynamically (without stopping your platform) which the screener runs over and applies to existing or streaming data?

MC 9 does, its called portofolio trader.

Listen, I am not trying to be confrontational but your talk sounds very BS-y. Sure, what OP looks to achieve can be done on very simple hardware and software but he also asked about architectural questions and I gave the simple advice not to start with SQL which is precisely when you went onto your retail-punter-rant and attacked me how I would dare to say SQL is not sufficient. SQL is NOT GREAT for time series analysis, whether you deal with 10 rows or 10 billion rows, period. You and also I do not have any idea what OP wants to do next. Can your platforms for example dynamically run screeners and you submit a script dynamically (without stopping your platform) which the screener runs over and applies to existing or streaming data? No, Multicharts cannot nor can Tradestation, so for heaven's sake, give me a little credit for knowing what I am talking about. You can hate me and the way I deliver the message but the message nonetheless is a message based on a lot of expertise in this area.

Hm I suspect you do work in the industry but in the IT support side, as in
the guy that the traders calls asking why can't they open redtube during lunchbreaks or
why the office printer is jamming again.

 

[AdrianHagh81]

Replying to couple of replies in single post:



AdrianHagh81: Spending money on a back testing software I agree, but usually the coding style used is harder to learn. Specifically when i am trying to deviate from the standard indicators. Once I asked question to Ninja Trader support about how to rank stocks by a given technical indicator and their answer was - we offer training and custom coding/indicators.

What they mean to say is that if you can't infer how to do it from the provided documentation
then its probably futile to try and accomplish that in an acceptable timeframe better pay us
to do it instead.

 

[volpunter]

* I looked at the file and that is where I took the stated speedup from (10x). It is by far not enough to even spend another second thinking to use SQL for columnar data. Let alone the fact that you have to pay hundreds if not thousands of dollars for a full MSSQL license if you want to store the amount of data a full-fledged time series data base requires. MySql and other SQL versions do not really offer such columnar indexing features (I think Oracle does but not 100% sure) and are a real pain to setup in a .Net environment (which I target, and no I won't go into any debate of pros and cons of this choice). Above all, writing a simplistic binary data store would take a proficient programmer not more than a few days.

* Great if you make 200 grand a year then you should honestly stick to what you do best and not talk about things you understand unarguably very little about. (fyi, 200/yr is the salary a senior associate-junior VP makes in ibanking, not to talk down your salary but by any measures it is nothing special at all in the financial industry and certainly does not impress me at all).

* Sure they use R for matrix computations, but R and Matlab stink big time when it comes to running code that is path dependent which most backtests are. It does not matter how you access R it simply is way too slow for code iteration (which is required for backtests).

* Sorry but regarding your "BarMagnifier" example it is nonsense: Here is why: If you ran a system over a full tick-based data feed then you do just that, period. Why do you need a "bar magnifier" or other toys? You can build whatever algorithms please you but it does not change the fact that a platform that is capable of handling tick based data runs over every singe tick, allows submission of orders at every tick and fills your order at whatever tick your simulated orderbook fills your order.

* And for your exercise you need some hardware "blades"? Why? Sounds more like you try to show off. If you have efficient code then you can run most screeners on commodity hardware. I can stream several million ticks a second on a commodity quad core machine, which includes query and loading of binary database data, deserialization to ticks/quotes, sorting by timestamp in case you request multiple time series, streaming of those to the strategy core engine, running strategy algos, submitting orders when each strategy generates orders, filling them in a simulated order book, generating statistics and reports as well as visualizations. What do you need a server or blade for? ;-)

* You confuse things here. "Portfolio trader" simply means that it runs backtests over multiple strategies and generates performance statistics across the whole book of strategies rather than running each strategy one after another. It has nothing to do with dynamic script execution. What I mean is that you have a live data analytics engine or live trading engine and inject a new analytics or strategy script and the engine will execute and run the same without interruption.

* You are wrong about your last assumption as well. Someone who takes care of jamming printers would not school you about even basic issues such as we discussed. We have not even gone in depth yet. We have argued before and equally you have refused to learn from anyone. You got all agitated just because their/my delivery of message did not sit right with you. If you had ever worked on the trading side in banks or funds you would have grown a thicker skin and had a keener attitude to learning rather than trying to shoot the messenger just because he did not kiss your ego.

I do not intend to spend more time debating issues with you after you displayed a very limited insight into the topic under discussion (remember, you stated "SQL is GRRREAATT for time series data storage and retrieval"). Not saying I cannot learn from you but I do not see much I can learn from you in this particular area. Good luck with your endeavor but please be careful what you recommend to newbies because you mislead others with your erroneous statements.

See the attached File for more in depth explanation on mssql column index implementatoin


I make 200 grand a year on my day job and can grow silicon right next door
with access to all the toys I could ever want. Why would I leave when they allow me
such flexible work time ?


Most people use R and Mathlab for the handy Matrix functions and other calculus
aproximation, just take a look at the library, and call it directly from your own C program.
problem solved.



It is the actual tick data, most signal are based on m1> OHLC data, but for precise
entry and slippage info the bid/ask tick by tick info is provided in the bar magnifier
function.



I've developed my own pattern finding tool in java and some C called with JNI.
its a GA and uses SVM + NN as the final filter condition.
I can set the treedepth, tournament size, kernel type and node number via parameter.
I unleash them on m1> OHLC data, then if they find any intresting patterns
implement that in MC .net to verify the results and get a more visual feel and
more detailed performance report.


MC 9 does, its called portofolio trader.





Hm I suspect you do work in the industry but in the IT support side, as in
the guy that the traders calls asking why can't they open redtube during lunchbreaks or
why the office printer is jamming again.

 

[AdrianHagh81]

And for your exercise you need some hardware "blades"? Why? Sounds more like you try to show off. If you have efficient code then you can run most screeners on commodity hardware. I can stream several million ticks a second on a commodity quad core machine, which includes query and loading of binary database data, deserialization to ticks/quotes, sorting by timestamp in case you request multiple time series, streaming of those to the strategy core engine, running strategy algos, submitting orders when each strategy generates orders, filling them in a simulated order book, generating statistics and reports as well as visualizations. What do you need a server or blade for? ;-)

Hmm, unless you've developed a new OS and filesystem I've never heard about
then this I find hard to believe.

Anyone using the standard linux kernel fread() will not accomplish this.

Guys Listening to volpunter makes you think that retail automated trading is hopeless,

well it's not, I'm proof of it

 

[AdrianHagh81]

You confuse things here. "Portfolio trader" simply means that it runs backtests over multiple strategies and generates performance statistics across the whole book of strategies rather than running each strategy one after another. It has nothing to do with dynamic script execution. What I mean is that you have a live data analytics engine or live trading engine and inject a new analytics or strategy script and the engine will execute and run the same without interruption.

WRONG

http://www.multicharts.com/trading-software/index.php/Portfolio_Trader

See, it even have currency conversion.

Heheheheheh

Ready made tools available for retail are starting to make serious gains.

It'll be a shame not to take full advantage of them.