So you don't know R right now, but are thinking about learning? My two cents worth is I wouldn't use R for this kind of application, and I'd spend my time learning something else.
Although in theory there is no issue, I've found there are problems with the way R allocates and then doesn't free up memory, particularly when running over loops. You need to write your code very carefully to avoid this kind of problem. As a relative novice you'll potentially waste a lot of time learning how to do this.
Where I used to work both Matlab and python were used successfully in the implementation of exactly the kind of system you have. I have a well known preference for Python so I won't trot out the pros and cons once again. Other languages are of course available.
I'd personally use a simple database for storage rather than say flat files, although as others have said this is more for robustness than to get more memory back.
One more thing; backtesting this beast could be done very easily as a parallel process (since todays CS ranking has no bearing on tommorrows). I guess in a world where most people have multiple cores on one machine some paralleisation would be done by the interpreter or complier (I'm not an expert on this) but you still have to write your code in such a way to make it possible (eg list comprehensions in python, or the equivalent). Alternatively we used to run stuff on a big cluster where we had to make the parallel stuff explicit (using something like
http://www.parallelpython.com/).
Another option could be using something like
https://www.quantconnect.com/ (no connection, never used it, but looks interesting) to get parallel computing power.
GAT
