Dilemma in modeling idiosyncratic high-frequency noise

it would really help to quantify your data structures and base set.

what markets are you using and applying your data gathering and sampling to?

futures?
options?
options on futures?
equities?
bonds?
otc bonds? (derivatives, credit swaps, etc.)
emini futures?
pit traded contracts only?
equity - preferreds?
equity - (high) dividend yields
equity - (no dividend yields)
equity - etf's
equity - etf sub-category (underlying heavily basket traded)


if you knew enough about trading these markets, each being and having their own distinct volume signatures and market participants
 
Quote from bluelou:

Wayne,
I think there's been a misunderstanding. I was referring to a simulated data set for a given contract such as the ES contract, not running a simulation/backtest.

By a simulated data set I meant a data set that replicates the statistical properties of the ES contract.

With simulated data we could run backtests on a variety of random realizations of a facsimile of the ES contract without falling into the path dependencies that can occur when optimizing a trading model to a single data set.

I'm assuming that you do not have this kind of data? Is this correct?

-Lou
More...


Sorry, if I misunderstood.

A simulated data set to build strategies?

I actually created data for doing that. I have no idea where that code is now.

Mainly I was just impressed at how much it looked like real data.

There's only one problem.

Much of market activity is "expected" as to timing. But nothing in that simulated data can be expected.

What do I mean by expected?

The market always reacts, for example, when the Fed makes an announcement or speaks before congress.

How do you "simulate" that?

Same is true with many news announcements.

Some are surprise news announcements.

Some get rescheduled. You can account for all of that in a real market situation.

How do you do it with simulated "statistical" data?

Can you make it go through phases due to news that may last weeks or months?

How statistical is it to simulate a 8 or 10 year bull market?

I don't think it's impossible. No. It's just there's SO MANY variables it seems kind of like it would be close to cracking RSA 1024 strength keys.

I'm dubious at best. But willing to learn.

Wayne
 
Quote from limitdown:

it would really help to quantify your data structures and base set.

what markets are you using and applying your data gathering and sampling to?
More...

Were you addressing me? I'll try to answer anyway.

2 answers, I'm only using 12 pairs of Forex (didn't I mention that?) and also, I play with MSFT for equities, personally.

However, I've discretionary traded most on your list below. And, of course, you're right they all have different "signatures".

But the system I build can only handle what I stated right now which is quotes, trades (aka T&S), and 5 levels of DOM.

It would be cool to use on ES for example. Then it would be useful to tweak the code (open source) to make it recognize the contract rollovers, etc of futures.

futures?
options?
options on futures?
equities?
bonds?
otc bonds? (derivatives, credit swaps, etc.)
emini futures?
pit traded contracts only?
equity - preferreds?
equity - (high) dividend yields
equity - (no dividend yields)
equity - etf's
equity - etf sub-category (underlying heavily basket traded)

if you knew enough about trading these markets, each being and having their own distinct volume signatures and market participants
More...

Sorry, this seems like a fragment. I can't quite get your point.

Certainly, for equities you have other, different players, agreed.

My personal preferences are Forex and Futures. I love options and will play with those mostly when my capital level warrants it.

Sincerely,
Wayne
 
no, I was responding to the thread starter,

surprising how it caught others in the sites, as it were.


there is a whole difference between those who are academics and those who have managed and returned real results....

these academic ideas never break down market substructures into their component parts and always blanket the discussion with broadband terms.

every wonder why all those brokerages with their preferential hiring practices lost billions?

hint:
implementing academic discussions like these.

no wonder we find guys like this trying to see why (those approachs) failed
 
Quote from bluelou:

Why does idiosyncratic noise exist in the first place? What causes the idiosyncracies? Is there any academic research on idiosyncratic noise signatures in high frequency data (of any type) that you would recommend? I'm open to your suggestions.

Background:
I use a model based on ideas derived from statistical physics to trade FX and other futures on an intraday basis. The only inputs to the model are price and bar count.

I'm refering to idiosyncratic noise, not signal. That is, I've detrended the data series so that all that is remaining is a noise residual. I believe that I'm finding that FX pairs (and other futures instruments) can have unique and persistent high-frequency noise signatures.
More...


I played with "detrended" data also. But let's assume you model it and get your system well trading the "noise" signal.

How do you translate that to the real market? (I'm asking to learn.)

My issue was that my detrended signal said to buy but the real market actually when down because it was in a strong down trend.

So I'm not sure how useful detrended data can be.

I presonal feel, that the what some successful traders is true, our "smarts" for those of us into statistcal modeling and such, work against us.

I always get a kick thinking of when Larry Williams pointed out that when he meets with mathematicians to discuss the market, they always conclude it's statistically random. But he's the one paying the drinks for the group from money earned by playing the markets in a non-random manner.

Wayne
 
Quote from limitdown:

no, I was responding to the thread starter,

surprising how it caught others in the sites, as it were.


there is a whole difference between those who are academics and those who have managed and returned real results....

these academic ideas never break down market substructures into their component parts and always blanket the discussion with broadband terms.

every wonder why all those brokerages with their preferential hiring practices lost billions?

hint:
implementing academic discussions like these.

no wonder we find guys like this trying to see why (those approachs) failed
More...


Well, there certainly is value in all the statistically modeling somewhere.

I was amazed when using it to model sales and marketing response to see how predictable (from the law of averages) people can be.

The statistical modeling done in some industries is nothing short of mind boggling.

But if anyone has really found a solid use for it in trading the markets, they're keeping it more secret than the CIA, for sure.

Wayne
 
Quote from greaterreturn:

Well, there certainly is value in all the statistically modeling somewhere.

I was amazed when using it to model sales and marketing response to see how predictable (from the law of averages) people can be.

The statistical modeling done in some industries is nothing short of mind boggling.

But if anyone has really found a solid use for it in trading the markets, they're keeping it more secret than the CIA, for sure.

Wayne
More...

no, actually,

they were quite public in their advertising of how they used these methods in their hedge funds....

the Madoff's of the world were low tech enough to take advantage of the "poof" factor that was sold to the public....

hedge funds have been economic terrorists in that they have been strangling the net worth of most trading markets and economies of a whole lot of country's corporate businesses,

in that their taking the so called statistacal annomoloies and market inefficiencies were really not there to be taken, and once gone helped to destabalize those markets, economies and financial systems....

no, these methods were used, and not as secret as any of the 3 letter bunch... (CIA being the most popular and commonly known, there are so many others too)....
 
Quote from limitdown:

no, actually,

they were quite public in their advertising of how they used these methods in their hedge funds....

the Madoff's of the world were low tech enough to take advantage of the "poof" factor that was sold to the public....

hedge funds have been economic terrorists in that they have been strangling the net worth of most trading markets and economies of a whole lot of country's corporate businesses,

in that their taking the so called statistacal annomoloies and market inefficiencies were really not there to be taken, and once gone helped to destabalize those markets, economies and financial systems....

no, these methods were used, and not as secret as any of the 3 letter bunch... (CIA being the most popular and commonly known, there are so many others too)....
More...

Okay. Understood.

Still, it seems we agree on the ineffectiveness.

Wayne
 
Quote from bluelou:

The last few e-mails remind me of Plato's "Allegory of the Cave". This is quality.
More...
Meaning what? You understand the reality outside the cave and we are interpreting the shadows or visa-versa? Just curious.
 
You must log in or register to reply here.
Back
Top