Not using any external framework , all is custom code and thats why it would be easyer to implement data filtering or preprocessing.
I've recently gone the all custom code-route myself. One codebase, one place, multiple scripts though to separate data harvesting (daily data for now) and algo + simulation. Not at real execution-stage yet and plan to use another broker than my current production system, since my current broker went bananas with poor usage of javascript (no API's offered here
).
Machine learning implementation and strategy tester is made and have been thinking of implementing filtered data for algos and using unfiltered for simulationg excutions.
Ok this brings some useful context. So, I guess you want to "filter", aka. smooth data, in order to help your algo. This choice will directly affect the underlying ML/algo being fed different data of course, which will be "per design".
However, then it begs the question: If the ML doesn't extract everything it needs in order to make robust and stable decision, do you actually need ML at all, what do you need it for and how can you ensure it converges to optimality (however you define it)? How can you trust it?
Problematic aspect is too few trades made by system.
That's a strange problem indeed, but is shared by my current production system. Either: Are we too impatient, or is some foundational understanding of the markets missing, would ML bridge that gap better than a human and how to prove that?
Most important aspect i dont know, everything is interconnected and to know weakest link would be good but not easy to conclusively find.
Maybe make it more simple?
Would like filter to try reduce chaos in data, maybe by removing mostly unrepetative noise(if it possible) to have less differences troughout data to make it easyer for algos to classify data movements. Would that make any sense?
On the surface, it makes perfect sense. However, it also tells me that the ML algos must then have limitations if reducing noise is necessary. Maybe wrong algo is used if it cannot pick movements from noise or is too unstable? How to solve that problem and if you do solve it, will simple if-conditions be enough afterwards?
Currently using only unfiltered price data, only part that may be filter is timeframe converter to make small timeframes to bigger.
Filtering irrevocably converts your data, for both good and evil
The good is that the data may be more understandable, but it'll also contain more "lag" and contain less original information. Your timeframe converter may also create artifacts or bugs, and be very wary of stealing information from the future in backtests (you probably already know this).
Idea is to implement multiple filters , and at runtime enable/disable, configure.
For example has someone used moving avg to slightly smooth data vs using unfiltered data?
If you configure at runtime in production, you're creating problems for yourself, as this invalidates your system performance and backtests. It's better to try to minimize configuration and not mess with the runtime once it's operational.
MA, EMAs, all sorts of smoothing filters are being used by everyone. However, it's faulty thinking that you just need "filter + ML + runtime config", and bang that solves your problems somehow sometime in the future. Where's the proof and backtest, or even a believable hypothesis? Additionally, if you optimize parameters based on past performance, you risk finding meaningless statistical spikes, especially the more dimensions you explore (more parameters * more datasources).
I can only find out hypothesis about filtering by testing filtering but it would be nice to know others experiences about feeding algos with unfiltered price data vs filtered (Same implementation might not be meaningful for different algo but still good to know).
I'd say test it out. Make it cheap & easy to test everything out. I don't structure my code too heavily, but make big procedures just for that purpose, flexibility to test stuff out instead of locking myself in. It's gotten better after my first attempt which is running live and doing it's break-even thing
, but still, need to explore possibilities and then all structure and external dependencies may rein you in unnecessarily.
In the end, it's not a magic system, a magic process or a magic teacher that'll make you land on your feet, send you running and later flying, it'll be you and your built-up experience, which can be broad and then 99% shaven off.
.