A proposal for testing black-box systems

botpro · Jan 6, 2016

A proposal for testing black-box systems

The best method to test a black-box system seems to be
if the system is online for a registered user (the interessent),
and offers a simple API for feeding the system with tick or bar data.
The system has to send its every decision to the user,
so he/she can verify that the system did not "look forward" into the data.

The API should have these main functions:

- login
- logout
- set_system_params
- start_system
- stop_system
- abort_system
- feed_system_with_market_data
- ...

and optionally:
- get_current_summary (ie. positions list and account data)
- ...

It means that the user has to send continously _single_ tick or bar data, maybe making a small pause of some milliseconds after each package sent.
And user-side must check if any data from the system-side has arrived.

Of course the system has to send status messages if it encounters an error
(for example if it gets no data from the user anymore and a timeout happens).

For security reasons an SSL link should be used (like https).

The market data (either real or simulated) has to come from the user.
The system must collect the data and inform the user immediately after making a trading decision (ie. buy/sell).

All data must have a timestamp or a tick/bar number, and maybe even a CRC for each data package.

The user has to log all the data he/she sends to the system, as well all the data he/she receives.
Same for the system side.

This gives a very good method for testing a remote black-box system and its advertised performance.
Of course it could still be cheated if both sides conspire and have that in mind.
But otherwise it is IMO the best testing method between system developer and a potential customer.

Handle123 · Jan 7, 2016

As name implies, "Black Box", not knowing how signals are generated, back testing is close to impossible to back test. There are so many ways to alter results, I have seen where if you back test six months or six years, that six month that both have, can have different results cause the guts of the system might rely on equity curve of winners or losers, so many losers in a row, will give signals, so many winning trades in a row, stops giving signals. Signals are skipped based on reports.. I understand why many have them cause fears of others stealing and reselling, thinking Collective2 would be better option for both one who designed system and audited for customer to make decision.

botpro · Jan 7, 2016

Handle123 said:
As name implies, "Black Box", not knowing how signals are generated, back testing is close to impossible to back test. There are so many ways to alter results, I have seen where if you back test six months or six years, that six month that both have, can have different results cause the guts of the system might rely on equity curve of winners or losers, so many losers in a row, will give signals, so many winning trades in a row, stops giving signals. Signals are skipped based on reports.. I understand why many have them cause fears of others stealing and reselling, thinking Collective2 would be better option for both one who designed system and audited for customer to make decision.

Can you give a concrete example why the above procedure would fail?
It is the user who sends the _single_ market data, and checks the response from the system after each sent data,
so by this he/she can be sure that the system is not looking ahead into the data.

And: backtesting is very well possible with this method: just create a data set (or even use real historic data).
What it does is forward-testing, but if you supply it historical data then the net effect is backtesting.

And there is just one result: that from the system. The user can verify each action (buy/sell) the system does;
that's the whole idea behind this.

Sorry, I don't have any experience with the mentioned firm Collective2, or any other.
The above method is intended for direct end-to-end testing a black-box system, without any intermediary 3rd party.
It's just an idea how such a test could be done.

Handle123 · Jan 7, 2016

I have not bought any Black Box systems since the 1980s, so I don't know who or what they are producing now, but most of the vendor software produced then and perhaps now would change the data from real data to continuous. Even though you might plug in by contract like March, June, Sept and Dec singles then rerun all, would get different signals. That is what I encountered back then.

Anything can add to your education to finding truth in vendor's software, but still comes down to this, if the software worked well, why sell it? So many who are good at decoding(break) software will get the guts. Future's Truth is very good service to see if systems making or losing. The really good software have so many they sell then that's it.

Alex27 · Feb 19, 2016

I'm guessing you know collective 2 quite well - what does the guy get from them ?

who has a winning strategy?

Its not very clear on site why a winning trader would bother?

Handle123 · Feb 19, 2016

botpro said:
Can you give a concrete example why the above procedure would fail?
It is the user who sends the _single_ market data, and checks the response from the system after each sent data,
so by this he/she can be sure that the system is not looking ahead into the data.

And: backtesting is very well possible with this method: just create a data set (or even use real historic data).
What it does is forward-testing, but if you supply it historical data then the net effect is backtesting.

And there is just one result: that from the system. The user can verify each action (buy/sell) the system does;
that's the whole idea behind this.

Sorry, I don't have any experience with the mentioned firm Collective2, or any other.
The above method is intended for direct end-to-end testing a black-box system, without any intermediary 3rd party.
It's just an idea how such a test could be done.

LOL I bought systems in late 80s and early 90s, the guy you bought the system from has tainted data, but as a buyer you don't know that as the system and data is black box. I have bought system where you enter open, high, low and close from the exchange and out pops tomorrows buy/sell signals, when it loses the black box system faxes out no signal and when you complain they said I have to buy their data as mind is wrong, so they get another $100 bucks out of you and their data is not any better cause as you have losses, you go back to run software day of loss, it shows no signal, WTF? And back then there was no clever way to take images as they had that blocked.

it all comes down to who you bought it from and if he is honest. But here is the bottom line, I don't care how much you spent on it, even $5,000, a good day trader can make that in a week or a day, why mess with the public as most as going to cry to someone about it as people who buy systems, generally can't trade and don't know when to adapt or not take signals. And black boxes are even worse cause you don't know what is going on.

And just cause you can run your data in the software, I found some can alter the results bypassing losing trades. And you can't check that cause t black box.

I have spent like $200k in my lifetime of buying everything and anything out there, take it apart and see if any of it will improve what I do, I have bought maybe five books in ten years now and have donated away 95% of the crap I bought.

Depending on what you want to trade any far as timeframe, long term systems do far better than day trading, they do have big drawdowns and people never start at right times, they should start after healthy drawdown but most start after making new equity highs. There can be good day trading systems sold, but I never look any more as nothing can be better than what I trade, I developed it and backtested and traded it fourteen years, but I am certainly not going to sell software of it.

Am sure there are thousands of Forex systems sold, this site is pretty good for reviews.
http://www.forexpeacearmy.com/

Free systems within ET are often a great staring point than buying anything. And often times many of those who start journals show what they are doing or journals from years ago. Really, unless you already know how to trade, I'd never recommend to anyone to buy systems.

Alex27 · Feb 19, 2016

Im not looking to buy a black box system !

Im looking to protect my algo, you see on some sort of custom platform instead of multicharts

botpro · Feb 19, 2016

Alex27 said:
Im not looking to buy a black box system !

Im looking to protect my algo, you see on some sort of custom platform instead of multicharts

You can protect your algo if you compile your code and create a shared library (ie. a DLL) which then would be loaded by the host as a "plugin".
Ie. the other side would get just the binary file, not the human readable source code.
If that minimum is a given, then you can even go further and additionally encode portions inside your code, so that it requires a keyfile or password
to decode in memory some essential tables/data needed by the system first, before starting its trading job...

And, of course you can also make your own system a black-box system. The above procedure is already half a black-box system.

Said differently: after all the sole purpose of a black-box system is nothing but just protecting the algo!... ;-)

Alex27 · Feb 19, 2016

Like It ! so whats best way to achieve this black box set up at the moment got multicharts would like c++ custom solution like this. With what you described above would this work with IB gateway, your suggestions?

botpro · Feb 19, 2016

Alex27 said:
Like It ! so whats best way to achieve this black box set up at the moment got multicharts would like c++ custom solution like this. With what you described above would this work with IB gateway, your suggestions?View attachment 162109

I have experience with the IB suite (TWS, IB gateway, and of course the IB API),
but I have no experience yet with Multicharts. But what I read on their web site indicates
that it is possible to write your own DLL:
http://www.multicharts.com/forex-trading-platform/tech-specs/
"Scripting Language
MCFX Pro supports EasyLanguage to a maximum possible extent. Studies are first translated into C++ and then efficiently compiled, resulting in extremely fast execution speed that is so critical for handling very large volumes of data. Nearly all of the advanced capabilities of EasyLanguage, including custom DLLs, are supported by MCFX Pro.

Development Technologies
MCFX’ architecture is written in C++, and it uses hardware resources to the fullest possible extent. It is a multi-process application that efficiently passes information between the processes to achieve maximum efficiency. MCFX runs parallel tasks on each processor, which dramatically improves the system performance. MultiCharts technology is also multiple-monitor friendly, allowing multiple chart windows to be detached from the main application and displayed across all of the available monitors simultaneously.
"

So, then what you want to achieve should be IMHO possible, but as said I don't know MC or your specific environment.
One would need to know the full environment (products/programs used, languages used etc.), locations (where has each component to run, local or remote at a 3rd party server etc.), ie. the full requirements of your specific project, without disclosing your intellectual property (IP)...

Update: I quickly read your posting. So, you mean you currently use MultiCharts but would like to realize your project in C++?
IMO no problem, you just would need to code it yourself or find someone who can do that for you.
I could do that too if you want, but it would of course cost some bucks, depending on the complexity and timeframe of the project...