IB TWS taking a lot of memory

Tried 933.5.
I forgot to tell java version:1.6.0_45.I think I read somewhere that IB advises to use 1.6 instead of 1.7.

Gateway memory consumption: 123 MB
My application memory consumption 30 MB

I see that when it downloads additional data (every 30 mins each security) memory increases: after two downloads we are at 129 MB, then after 2 minutes goes to 127 MB. This didn't happen with 917.9.
There seems to be a memory leak actually.
I would suggest a memory optimizer anyway: mine sent gateway used memory to 17 MB; after 10 minutes it has grown to 70 MB: at this level seems quite stable now. But I'm just looking at quotes,not sending orders/requesting portfolio,downloading historical data etc.
 
Quote from pfranz:

Tried 933.5.
I forgot to tell java version:1.6.0_45.I think I read somewhere that IB advises to use 1.6 instead of 1.7.

Gateway memory consumption: 123 MB
My application memory consumption 30 MB

I see that when it downloads additional data (every 30 mins each security) memory increases: after two downloads we are at 129 MB, then after 2 minutes goes to 127 MB. This didn't happen with 917.9.
There seems to be a memory leak actually.
I would suggest a memory optimizer anyway: mine sent gateway used memory to 17 MB; after 10 minutes it has grown to 70 MB: at this level seems quite stable now. But I'm just looking at quotes,not sending orders/requesting portfolio,downloading historical data etc.
More...

How are you monitoring the memory consumption? I presume you're aware that the number shown in windows is the JVM foootprint, not the memory in use by the application. You need to profile or turn on GC logging if you want to see what's actually going on. A little premature to confirm a memory leak I think. Increases in allocation during download seems entirely reasonable.

I haven't read the whole thread, but why are you obsessing over a few MB? I run the gateway with a fixed maximum heap size of 512MB and have no issues running for a week at a time, although I run on Linux.

FYI, the Java 6 recommendation was in one of the TWS release notes late last year given the security issues that had arisen in Java 7 at the time.
 
I'm not obsessed by memory consumption,I was just replying to regulus who had a more serious problem. These are tests for him (see his last posts).
I don't know why his figures are so much bigger than mine,I was just noting that 933.5 has a different behaviour than previous versions.
I don't care what java garbage collector does or does not,what I believe is important is how much memory javaw.exe takes (and I think task manager value is correct). I don't know if it is a java gc problem or a gateway problem, I know that this didn't happen with 917.9: after downloading data, memory used remained more or less stable. Now it increases and doesn't come back.
A few MB are not a problem for me,I won't even use this gateway version.
If you say that the problem with java 7 was only a security problem in an old version,it can surely be.I thought there was something about execution correctness too,but I don't remember very well.
 
Quote from pfranz:

This didn't happen with 917.9.
There seems to be a memory leak actually.
More...
Thanks for confirming that there actually IS a memory leak with TWS 933.5. If you try the more recent TWS versions as well (up to the latest TWS 938.1f) you should notice the same effect.
So it's not just on my end.

Maybe you want to try to download some historical tick data (say ~2 months) for some futures contracts (say 5 - 10). Then the memory leak should become very obvious.

Quote from pfranz:

I would suggest a memory optimizer anyway
More...
Which memory optimizer would you actually recommend?

P.S. My java version is 1.7.0_21, OS is Windows 7 64 Bit.
 
Wait, ddecker is right when he says that it's not enough to seriously confirm a memory leak.
But I think the difference in behaviour is clear (even if I didn't write it,I watched carefully what happened to memory usage while software was working),and should be enough to start an investigation.
Don't forget that the leaks I saw were small,much smaller than yours.
I'm using the memory optimizer that comes with "wise care 365"
My Windows 7 is 32 bit
 
Quote from pfranz:

I'm not obsessed by memory consumption,I was just replying to regulus who had a more serious problem. These are tests for him (see his last posts).
I don't know why his figures are so much bigger than mine,I was just noting that 933.5 has a different behaviour than previous versions.
I don't care what java garbage collector does or does not,what I believe is important is how much memory javaw.exe takes (and I think task manager value is correct). I don't know if it is a java gc problem or a gateway problem, I know that this didn't happen with 917.9: after downloading data, memory used remained more or less stable. Now it increases and doesn't come back.
A few MB are not a problem for me,I won't even use this gateway version.
If you say that the problem with java 7 was only a security problem in an old version,it can surely be.I thought there was something about execution correctness too,but I don't remember very well.
More...

Didn't mean to imply that you were, perhaps obsessed was a poor word choice ;)

You should care what the garbage collector does, because it seems you are concerned about the performance and memory footprint of the application, and the GC is directly impacting this. I can say with a good degree of confidence, that there is no memory leak in the JVM version you are using. As such, looking that the JVM footprint isn't telling you anything. Seeing an increase in memory usage during activity that is not subsequently released is entirely normal.

The JVM doesn't know at runtime how much memory the gateway is going to use, so it sets a reasonable initial heap size, which is what you see in the task manager. This is not how much memory the gateway is using. During execution, the gateway allocates memory and if it deems necessary, the JVM will increase the heap size and you'll see the footprint in task manager increase. This is unlikely to be given back, it simply reflects the fact that too small a heap was initially allocated. If your system has adequate memory, the JVM will use it.

My point here is that if you're actually worried about a memory leak (which I wouldn't be unless you're running into an out of memory condition), then you need to turn on GC logging and use something like HPjmeter to investigate. A slowly increasing JVM footprint over a limited time period means little.

I would also suggest that if you care about performance, somehow forcing (no idea if your optimizer does this) the JVM to decrease it's memory consumption is not a good idea.

regulus, I'd recommend using the gateway, not TWS, and setting the heap size to 512MB. I run version 930.5 and have no memory issues.
 
Quote from ddecker:


regulus, I'd recommend using the gateway, not TWS, and setting the heap size to 512MB. I run version 930.5 and have no memory issues.
More...

ddecker, I am not quite sure if we are on the same page. That you do not have any memory issues with TWS 930.5 is perfectly consistent with my experience. As I have outlined in my previous posts:

TWS / IB Gateway 932.4 and older: no memory leak issue
TWS / IB Gateway 933.5 and newer: memory leak issue

Further, I have experienced the memory leak issue under the following, specific conditions:

Multicharts 64 as API client and at least one chart of tick data (Futures or ETFs; with Forex or Spot Metals no issue occurs)

512 MB is the default max heap size for TWS versions up to 932. Beginning with TWS 933 (or some versions later, don't remember exactly), IB set the default max heap size to 768 MB, obviously in an attempt to circumnavigate this issue. But even setting Xmx to 5 GB does not help; RAM usage increases to several GB and TWS will freeze just a few hours later. Of course I tried several Xmx settings from 512 MB to 5 GB; all to no avail
Even upgrading my workstation to 64 GB of RAM (which I will do anyway; as of now, I have 16 GB) will not resolve the problem. Maybe this will be good for one or two days, but not for a full trading week.

IB Gateway versions 933 and newer show exactly the same issue, as I had outlined before.

At least from my perspective, when a small, slim API application like IB Gateway allocates several GB of memory, something must be flawed.
 
(EDIT: I wrote this post before reading regulus reply; regulus, did the optimizer help?)

Thanks for your information,ddecker,but it seems you didn't read what regulus wrote.
1) The problem is just from 933.5 onward, so 930.5 is known to work right. If you want to help, why don't you try on linux, version 933.5 or later and carefully watch memory consumption?
2) For him the problem is a big problem because after some hours memory consumption is many gigabytes and it freezes the machine,so any heap setting won't do
3) He only wanted to find if someone saw the same thing or if it was only him,in order to decide if he'd tell Interactive Brokers
4) I'm a programmer and though I know very little java, I think I know allocation memory issues and garbage collectors; I already had in mind what you wrote about gc
5) Supposing (as you seem to do) that memory optimizer reduces heap size (but I'm not sure), it's not easy to say whether performance will be hurted: probably at first java has to re-allocate all the memory it needs which was freed by the optimizer (and in this phase performance will probably decrease), but at a certain point memory consumption stabilizes and then there is the chance that performance not only remains stable,but even increase.In regulus case,an optimizer should definitely help.
6) The heap size I set seems to be not immediately allocated: I see that after launch - at logon - gateway is consuming far less memory (for me, around 30 MB).Probably it's a trick with virtual memory,transparent to java.So if you want to avoid system memory allocation to increase performance or to avoid the problem,this doen't seem the way
 
pfranz, thanks for your reply. I really appreciate this.

I've had a test run with Wise Memory Optimizer. The tool has a remarkable effect: after optimization, javaw.exe will be "crunched" to 10 - 20 MB, but then rapidly grows to the threshold defined in Wise.
The only drawback with Wise is that I can't precisely define a threshold for javaw.exe but just for the overall memory allocation. Are there any memory optimizers available which provide this functionality?

However, in my understanding, using a memory optimizer is more a workaround and not a solution for this specific issue. Let's hope IB will tackle this issue in due time.
 
Of course it is only a workaround. I hoped it could stop memory growing some time after crunching,but if you have to set a threshold then it means that memory consumption doesn't stop.
There are a lot of memory optimizers which generally have time and memory thresholds to set,but I don't know them.you can do a search and see which fits you best.
Supposing that the problem is historical data download,I don't understand why multicharts has to do it so much.
I plot charts only from data stored on disk,and use data downloading only to complete the missing data in the disk.this way downloads are not so many.
Another possibility is that problem comes also from the 5 seconds realtime bars,which I don't use.
If problem is when plotting charts,you are supposed to be in front of the screen and can do memory optimizing manually: this is very convenient as you can choose the best moment for your trading(=when you don't have to trade) as shrinking & enlarging memory use can impact performance as ddecker said.
Before asking IB maybe you could first of all try java 1.6 instead of 1.7,then windows 7 32bit or any other windows version you can have access to.
Then finally try another API client different from multicharts and see how things go.
 
You must log in or register to reply here.
Back
Top