Vectorized backtesting with pandas

[dholliday]

Seems to be a big concern of yours given you want to backtest 10 years of data in 15 seconds - your first post.

15 seconds per security is too long. 15 seconds for several thousand securities is difficult without some serious thought to parallel processing.

Hard to figure out what you are really wanting.

jharmon is correct.
I just checked my app.
1049 symbols, 1000 EOD data records (four years) per symbol, approx. 10 sec. total time.
This is on a very old i7 desktop. Using 8 threads.
By far the majority of this 10 sec. is spent loading the CSV files from disk. When optimizing I only load a file once. Quite quick.
Computers are very fast. You have to have incredibly poor code to not be fast. Unfortunately, most trading platforms are made to display pretty charts and are slow. Also, most libraries are general purpose, not made to be fast and require you to get the data into a format they understand rather than how you want to write your code.
I don't use Python but:
Use a threadpool
Only load data once
don't do any graphics until your done with your calculations
If it's not fast enough:
Write your own libraries in a more efficient manner.
Compile your Python code.
Use a compiled language like C++, Java JVM language, C# CLI language.

 

[nooby_mcnoob]

jharmon is correct.
I just checked my app.
1049 symbols, 1000 EOD data records (four years) per symbol, approx. 10 sec. total time.
This is on a very old i7 desktop. Using 8 threads.
By far the majority of this 10 sec. is spent loading the CSV files from disk. When optimizing I only load a file once. Quite quick.
Computers are very fast. You have to have incredibly poor code to not be fast. Unfortunately, most trading platforms are made to display pretty charts and are slow. Also, most libraries are general purpose, not made to be fast and require you to get the data into a format they understand rather than how you want to write your code.
I don't use Python but:
Use a threadpool
Only load data once
don't do any graphics until your done with your calculations
If it's not fast enough:
Write your own libraries in a more efficient manner.
Compile your Python code.
Use a compiled language like C++, Java JVM language, C# CLI language.

Definitive answer that applies to everything (nevermind that I load years of ticks at a time, this guy definitely knows what he's doing)