Another aspect came back to me that I think will clear up some questions you have. When we split the CICS/IBM DB2 mainframe transaction servers off to UNIX they had their own DB2 database at that point. The transaction servers used these distributed DB2 databases to hold the instructions from the main application and as a holding place for the transaction content. So yes that distributed database was âcollocatedâ with the transaction server. But it was wiped clean after transmissions back to the z/OS just like the files where wiped out when it was a mainframe only app.
You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an alternative browser.
You should upgrade or use an alternative browser.
how to protect trading strategies for a colocated server
- Thread starter trend2009
- Start date
Another aspect came back to me that I think will clear up some questions you have. When we split the CICS/IBM DB2 mainframe transaction servers off to UNIX they had their own DB2 database at that point. The transaction servers used these distributed DB2 databases to hold the instructions from the main application and as a holding place for the transaction content. So yes that distributed database was âcollocatedâ with the transaction server. But it was wiped clean after transmissions back to the z/OS just like the files where wiped out when it was a mainframe only app.
Jerry,Quote from Jerry030:
Iâd like to express my sincere thanks for your post, which is outstanding. It contributes well written, concrete, detailed information on real world trading in terms of the topic on this thread.
Hats off to you!
It also shows the exception to the just prior post, which notes that nothing of value is ever shared, on-line anymore.
A few questions if you care to respond:
1) What kind of profitability did this system generate, expressed in any metric you know or care to share? One assumes that with this kind of effort the trading was profitable enough to make this worthwhile long term.
2) You mentioned compensation based on performance, if the system kept working. I donât want to know the numbers but could you give me an idea of the ratio between base pay and incentive bonus that was used to keep everyone working happily?
My personal theory is that this has to have a relationship to the net profitability of the operation in order to keep folks from trying to steal it and or jump ship and try to duplicate it independently.
3) You mention extensive compartmentalization in hardware, software and people. What would have prevented a gathering of yourself and a half dozen peers in a bar on Friday after work from turning into a plot to put some of the pieces toghter either for fun or to get the system? You as DBA working with others who had access to the parts you didnât.
A number of the basic concepts you mention can be adapted to the kind of operations in the range of folks on ET. I suspect none can afford the mainframes you mention, however the concepts of breaking the system and its operation into parts running on multiple separate independent servers at different locations has been discussed along with
other some of the other elements you mentioned.
You also lend some support to my perspective that the best way to steal the system is to get access to the trades being executed and use advanced CI to clone a functional equivalent even if precise algos or parameters are unknown. I have done a portion of this process for a client. It is called a Failure Model. The client had a very complex proprietary system used to trade private money. He wanted to improve performance but of course wouldnât reveal anything about the structure of the system. So I used neural networks and other advanced CI to create a predictive model of the systems performance. When the model predicted that the system would generate a loosing trade then the system would not trade. When there was no prediction of a loss the system traded normally. This eliminated over 20% of the negative trades and boosted net profitability considerably. The next step in this logical process would be a Success Model: predicting winning trades. The two together would essentially duplicate most of the original system (functionally), without ever knowing anything about it except the market, time granularity, market data in, and trades out.
Thanks, again for a high quality post.
Jerry
You asked some questions that are at the heart of any security meeting. I sat through meetings where many of these same questions were asked. In these meetings we had to use existing bank security plus some we invented.
One of the basic security needs was filled by the banks two signature audit policy. If the system needed programmers to make changes then to it then a minimum of two signatures from bank officers responsible for their parts of the system were required before any code could be pull back for change. It was during this open period that the system was vulnerable to theft.
To counteract this theft risk the bank used test boxes for secured systems. The problem with this box is you could work on the objects inside all you wanted but nothing came out of it. You could not even screen scrape the images on the tubes. The code that came out of this box was the difference changes between the current production and modification (mod) that would be applied to production systems (version control was CA-Panvalet if I remember right). This means you did not see the entire code set again after it made production the first time. And it took a minimum of two signatures to get these mods to remake to the production code.
The bottom line would have taken an act of god to get the full production code out in the open where multiple people had an opportunity to get their hands on it to steal it. The second problem is with the system broken into parts you would need almost all of the management to sign off two at a time to get the system in the open. Donât think the auditors would not catch something like this. Doing this at a bar on Friday night would have drawn a lot of attention with a crowd that large.
For example my area secured all of the Database platform design, DB2 database definitions, SQL and Database processing code used by the system. It was rare we changed platform design, added new tables, designed new SQL and database code in the same mod. That would have required a full meeting of all the security team and many eyes looking over my shoulder.
As you mention the logs of the trades, trade transactions, execution output, trade transmissions any thing that pointed to interaction with exchanges had a big role in the design of this trading system. These objects were all cleaned up as they happened and stored as database objects rather than files to increase security (which is rare on regular systems). The idea behind this, I was told, was the system was ramped in stages. Once all functionality was operational the audit trail of the trades was like the source code.
I heard many times that if a competing bank got a âsnifferâ on any of our communication lines this system was toast. These types of comments towards the transactions were ongoing. These were rehashed more that the original systems design. We did things like change encryption because too many hands had touched codes and they were worried one end might be broken into. With every new release of DB2 we installed this system had to be tested (along with the others) for holes that IBM may have opened up. So there was no set it and forget with this system. We were at war and this was a main battle resource.
I got to hear about compensation based on performance in meetings with my boss and at other events. Discussing pay or bonuses on work time except with your boss even in jest could get you fired. Banks are way beyond touchy with this subject especially since I was a bank officer. In that meeting with my boss I got the largest bonus of my life. But my boss was perplexed. He was a straight guy. He told me to keep it quiet but the bonuses we had gotten were âa pittanceâ of what the âtrade managementâ guys had gotten. What I heard later on the sly at a Christmas party where lips were loosened by elixir was some bonuses were in the hundredâs of thousands for that team. Whew. But I still bought a car on my bonus. It was very nice compared to other leaner years.
About a month after that party I was at a change control meeting where I found out why the bonuses happened. They had ramped the system too full out for the last quarter of the year (which I already knew from seeing the database activity). But what I did not know is it blew out profit expectations because of volatility. I never did get close to learning net profitability and compensation numbers. I handled technical functionality and was not invited to system management meetings. All I heard was the rumors and offline conversations. But seeing the toys that team bought from that experience was an excellent motivational tool to get me into trading on my own full time.
Thanks,
RabbitOne
excellent! thank you a lot for sharing with us, few of us in the forum have experience of yours.
Quote from Rabbitone:
Jerry,
You asked some questions that are at the heart of any security meeting. I sat through meetings where many of these same questions were asked. In these meetings we had to use existing bank security plus some we invented.
One of the basic security needs was filled by the banks two signature audit policy. If the system needed programmers to make changes then to it then a minimum of two signatures from bank officers responsible for their parts of the system were required before any code could be pull back for change. It was during this open period that the system was vulnerable to theft.
To counteract this theft risk the bank used test boxes for secured systems. The problem with this box is you could work on the objects inside all you wanted but nothing came out of it. You could not even screen scrape the images on the tubes. The code that came out of this box was the difference changes between the current production and modification (mod) that would be applied to production systems (version control was CA-Panvalet if I remember right). This means you did not see the entire code set again after it made production the first time. And it took a minimum of two signatures to get these mods to remake to the production code.
The bottom line would have taken an act of god to get the full production code out in the open where multiple people had an opportunity to get their hands on it to steal it. The second problem is with the system broken into parts you would need almost all of the management to sign off two at a time to get the system in the open. Donât think the auditors would not catch something like this. Doing this at a bar on Friday night would have drawn a lot of attention with a crowd that large.
For example my area secured all of the Database platform design, DB2 database definitions, SQL and Database processing code used by the system. It was rare we changed platform design, added new tables, designed new SQL and database code in the same mod. That would have required a full meeting of all the security team and many eyes looking over my shoulder.
As you mention the logs of the trades, trade transactions, execution output, trade transmissions any thing that pointed to interaction with exchanges had a big role in the design of this trading system. These objects were all cleaned up as they happened and stored as database objects rather than files to increase security (which is rare on regular systems). The idea behind this, I was told, was the system was ramped in stages. Once all functionality was operational the audit trail of the trades was like the source code.
I heard many times that if a competing bank got a âsnifferâ on any of our communication lines this system was toast. These types of comments towards the transactions were ongoing. These were rehashed more that the original systems design. We did things like change encryption because too many hands had touched codes and they were worried one end might be broken into. With every new release of DB2 we installed this system had to be tested (along with the others) for holes that IBM may have opened up. So there was no set it and forget with this system. We were at war and this was a main battle resource.
I got to hear about compensation based on performance in meetings with my boss and at other events. Discussing pay or bonuses on work time except with your boss even in jest could get you fired. Banks are way beyond touchy with this subject especially since I was a bank officer. In that meeting with my boss I got the largest bonus of my life. But my boss was perplexed. He was a straight guy. He told me to keep it quiet but the bonuses we had gotten were âa pittanceâ of what the âtrade managementâ guys had gotten. What I heard later on the sly at a Christmas party where lips were loosened by elixir was some bonuses were in the hundredâs of thousands for that team. Whew. But I still bought a car on my bonus. It was very nice compared to other leaner years.
About a month after that party I was at a change control meeting where I found out why the bonuses happened. They had ramped the system too full out for the last quarter of the year (which I already knew from seeing the database activity). But what I did not know is it blew out profit expectations because of volatility. I never did get close to learning net profitability and compensation numbers. I handled technical functionality and was not invited to system management meetings. All I heard was the rumors and offline conversations. But seeing the toys that team bought from that experience was an excellent motivational tool to get me into trading on my own full time.
Thanks,
RabbitOne
RabbitOne, Trend2009 and others,
On the topic of system security I'd like to suggest a thought experiment. This process is used in quantum physics to explore a domain that hasn't yet been detected or measured in the physical universe, such a parallel realities, communication beyond the speed of light, or hidden dimensions in our universe.
Goal
Create a security process for a automated trading system whose performance would be viewed as impossible by the general public community, as science fiction by expert system developers and as something to be cracked or stolen by an elite few, assuming they became aware of its existence
To make this fun lets assume that each of us has either developed or acquired such a system. What world we do to protect it and how would we use it?
To get us thinking Iâll list some possible categories and theoretical possibilities.
Feel free to add others that spark you imagination. Lets call this imaginary system the X Project.
X Project
Imaginary Creation Scenarios:
1) The discovery of obscure new mathematical methods and their implementation in an academic research software at a major university falls into your hands. While the system is devoted in DNA analysis, you plug in market time series data and find it has unbelievable ability to identify or predict market turns.
2) You find a laptop in the trunk of a rental car you just picked up at the airport. Just for fun you decide to see what all is on it before you call the rental company. You discover it contains the prototype to a voice pattern matching system for NSA use in tracing terrorist cell phone intercepts. You test on market patterns and find the results astounding.
2) Others â¦â¦. Add you own below
X Project System Operating Characteristics
1) Totally automatic, with no human intervention or
monitoring necessary
2) Othersâ¦..add your own below
X Project System Performance Characteristics
1) Average win/loss ratio in net dollars per
100 trades: > 93%
2) Time to doubling of total trading account
capital: < 14 trading days
3) Othersâ¦. Add your own below
Operational Challenges
1) Physical Security: A major problem as anyone
other then yourself who:
a) Knew of itâs existence
and
b) Believed it to be real
would be tempted to steal it.
In additional to individuals this might include corporate entities, criminal elements and other state and non-state players.
How would you insure physical security of the system hardware/software and insure it was non-operational if physically stolen?
2) Operational Security
Assume you have a plan for physical security but that leaves operational challenges
A) Unless you are an exchange member or clearing firm
or market maker, you have to execute trades
through another entity.
The performance of such a system would rapidly
get attention from a broker, auditor or others
where you execute trades.
How your you hide the system and its performance from those holding your account and executing the trades
3) Market impact
The X Project would rapidly make so much capital that it would alter markets depending on their size and volume.
How would you mitigate this to keep the system operational at the largest scale that is safe in a given market?
You might ask why play around with such a fantasy. Well, its fun to imagine things, as it gets our creativity flowing. Also, perhaps Iâm working on a plot for a book or screenplay about high tech investing, orâ¦..well, who knows.
Jerry
On the topic of system security I'd like to suggest a thought experiment. This process is used in quantum physics to explore a domain that hasn't yet been detected or measured in the physical universe, such a parallel realities, communication beyond the speed of light, or hidden dimensions in our universe.
Goal
Create a security process for a automated trading system whose performance would be viewed as impossible by the general public community, as science fiction by expert system developers and as something to be cracked or stolen by an elite few, assuming they became aware of its existence
To make this fun lets assume that each of us has either developed or acquired such a system. What world we do to protect it and how would we use it?
To get us thinking Iâll list some possible categories and theoretical possibilities.
Feel free to add others that spark you imagination. Lets call this imaginary system the X Project.
X Project
Imaginary Creation Scenarios:
1) The discovery of obscure new mathematical methods and their implementation in an academic research software at a major university falls into your hands. While the system is devoted in DNA analysis, you plug in market time series data and find it has unbelievable ability to identify or predict market turns.
2) You find a laptop in the trunk of a rental car you just picked up at the airport. Just for fun you decide to see what all is on it before you call the rental company. You discover it contains the prototype to a voice pattern matching system for NSA use in tracing terrorist cell phone intercepts. You test on market patterns and find the results astounding.
2) Others â¦â¦. Add you own below
X Project System Operating Characteristics
1) Totally automatic, with no human intervention or
monitoring necessary
2) Othersâ¦..add your own below
X Project System Performance Characteristics
1) Average win/loss ratio in net dollars per
100 trades: > 93%
2) Time to doubling of total trading account
capital: < 14 trading days
3) Othersâ¦. Add your own below
Operational Challenges
1) Physical Security: A major problem as anyone
other then yourself who:
a) Knew of itâs existence
and
b) Believed it to be real
would be tempted to steal it.
In additional to individuals this might include corporate entities, criminal elements and other state and non-state players.
How would you insure physical security of the system hardware/software and insure it was non-operational if physically stolen?
2) Operational Security
Assume you have a plan for physical security but that leaves operational challenges
A) Unless you are an exchange member or clearing firm
or market maker, you have to execute trades
through another entity.
The performance of such a system would rapidly
get attention from a broker, auditor or others
where you execute trades.
How your you hide the system and its performance from those holding your account and executing the trades
3) Market impact
The X Project would rapidly make so much capital that it would alter markets depending on their size and volume.
How would you mitigate this to keep the system operational at the largest scale that is safe in a given market?
You might ask why play around with such a fantasy. Well, its fun to imagine things, as it gets our creativity flowing. Also, perhaps Iâm working on a plot for a book or screenplay about high tech investing, orâ¦..well, who knows.
Jerry
Quote from WinstonTJ:
holy long quotes and posts in this thread...
I can't believe you guys are still going around in circles over the same thing.
If you really know how to do it why not actually DO IT instead of bitch about it over the internet?
Winstin,
Thanks, (I guess) for your participation. However you're on the loosing side of this trade as a number of us have found the discussion to be very useful. I especially liked confirmation that the easiest way to steal a system is to reverse engineer it from the actual trades.
If the amount of reading material presented taxes your abilities perhaps you should switch to picture books or comics. I know some folk with ADD have problems with long sentences.
Jerry
The answer is, YOU locate yourself and servers to an office building with appropriate connections, appropriate security measures. YOU control permissions and access to the server. The closer the machine is to you, the safer it is.
Colocation is stupid. It bleeds of you having no financial backbone to backup the trading system itself. If you fully trust this system and its proven to make you money, then put the damn servers in your office and hire a security professional to administrate it - someone you trust.
Why in the world would you put your crown jewels on a colocated server in the first place? If something so valuable is put on equipment which is untrusted and for which can be tampered with, don't you think you should keep the servers in-house and have policies in place only for those who work for you? You should never trust your sensitive applications to a hosting company. Even though they say they have security measures in place, that doesn't mean a rogue hosting personnel can get to your files let alone an outside hacker.
For whatever reason, the safest route is to build the application in C/C++ and compile it into binary native format to the server (i.e, Intel). Using any interpreted scripting language is a risk as hackers can pull the source files and read easily. At least with binary they are going to have to spend a hell of a time reverse engineering the code.
This question is really ridiculous when you think about it. If you were real with something that makes you tons of money, how can you be so naive to ask this question in the first place?
Clearly this op is a newb and has nothing worth of a system even protecting!
Colocation is stupid. It bleeds of you having no financial backbone to backup the trading system itself. If you fully trust this system and its proven to make you money, then put the damn servers in your office and hire a security professional to administrate it - someone you trust.
Why in the world would you put your crown jewels on a colocated server in the first place? If something so valuable is put on equipment which is untrusted and for which can be tampered with, don't you think you should keep the servers in-house and have policies in place only for those who work for you? You should never trust your sensitive applications to a hosting company. Even though they say they have security measures in place, that doesn't mean a rogue hosting personnel can get to your files let alone an outside hacker.
For whatever reason, the safest route is to build the application in C/C++ and compile it into binary native format to the server (i.e, Intel). Using any interpreted scripting language is a risk as hackers can pull the source files and read easily. At least with binary they are going to have to spend a hell of a time reverse engineering the code.
This question is really ridiculous when you think about it. If you were real with something that makes you tons of money, how can you be so naive to ask this question in the first place?
Clearly this op is a newb and has nothing worth of a system even protecting!
What are the "appropriate" connections? How do you assure your physical connections aren't being snooped?Quote from FutsTrader111:
The answer is, YOU locate yourself and servers to an office building with appropriate connections. YOU control permissions and access to the server. The closer the machine is to you, the safer it is.
What's the latency to the various exchanges? How much redundancy do you have? In case of hard drive or power supply failure, how long until you're back up? How large is your IT department supporting your system?
I personally value being in 350 E Cermak.
Quote from heech:
What are the "appropriate" connections? How do you assure your physical connections aren't being snooped?
What's the latency to the various exchanges? How much redundancy do you have? In case of hard drive or power supply failure, how long until you're back up? How large is your IT department supporting your system?
I personally value being in 350 E Cermak.
The question is how to protect trading strategies for a colocated server. The answer is DONT COLOCATE.
If the application can be entirely run in-house then do it. Colocating puts up another level of worry having to go through a secure connection. Why risk that at all? Eliminate all the potential security threats. That starts by not being a cheapskate!
Secure the application to be Intranet based. Hire the appropriate personnel in house to administrate and hold them accountable.
The security policy is to always be paranoid and aggressively monitoring. If you don't approach your application this way, you are asking for the unexpected.
After exchanging a few PMs I thought we were getting along... I'll stick to picture books, adderall and Redtube from now on.Quote from Jerry030:
Winstin,
Thanks, (I guess) for your participation. However you're on the loosing side of this trade as a number of us have found the discussion to be very useful. I especially liked confirmation that the easiest way to steal a system is to reverse engineer it from the actual trades.
If the amount of reading material presented taxes your abilities perhaps you should switch to picture books or comics. I know some folk with ADD have problems with long sentences.
Jerry
Quote from heech:
What are the "appropriate" connections? How do you assure your physical connections aren't being snooped?
What's the latency to the various exchanges? How much redundancy do you have? In case of hard drive or power supply failure, how long until you're back up? How large is your IT department supporting your system?
I personally value being in 350 E Cermak.
This is what is so funny about ET and this post. If you are paranoid about "the man" spying on you then you have bigger issues than worrying about trading.
There is a difference between bandwidth and latency - this thread does not seem to understand that difference. You can colo the crappiest box on fiber in tim-buk-tu and you will still be SOL. It has more to do with a balance of processing power, tier-1 business ISP and the trade than does co-lo and espionage. Anyone who trades via mouse/manually does not need to co-lo - ever. Anyone who holds for more than 10min does not need to co-lo - ever (unless you trade futures and are setting up overnight).
heech - if I break/blow up... I get flat and take the day off. I'm not the NYSE and I don't try to be. Five 9's is not my goal or even a remote requirement - HDD/PSU failure - get flat, figure it out, get back on line ASAP... no big deal, such is life. My next meal does not hinge on my hardware making it through 4pm tomorrow and the same is probably true for anyone paying for rack space (which I don't).
PS - what are you going to do when CME builds their own prop datacenter? 350 E Cermak is nice but do you co-lo in all 4 CME leased datacenters and ping every morning to decide where to run from? If not you are wasting your money paying for rack space. Get a 10mb (byte not bit) line out of your house, anywhere in IL and be equally competative.