IB TWS taking a lot of memory

[regulus]

Quote from pfranz:

regulus,
I use only 1 sec. data,so I'm in the same case as you are.I have 1 GB of memory and set java heap to 96 MB.
So I don't understand why my problem is much smaller than yours (actually I have no problem)

pfranz,
maybe the issue occurs only for the specific combination of Multicharts and TWS/Gateway 933+?
But that's pure speculation, I didn't try any other API client than Multicharts yet.

 

[pfranz]

Just to be precise, here are the switches I use:
-Dsun.java2d.noddraw=true
-Dsun.java2d.opengl=false
-Dsun.java2d.pmoffscreen=false
-Xmx96M

 

[ddecker]

Quote from regulus:

I tried to integrate the Java log commands you suggested into the TWS shortcut target field, but there is a limitation of characters. Then I tried to start TWS via batch file, but I am not sure how to set the Java class path correctly. I will have another try later on.

The verbose:gc switch is redundant so leave that out if it helps it fit. You can probably omit -XX:+PrintGCTimeStamps too, as I think it's turned on by default with Xloggc, but not 100%.

For the classpath on windows, just use -cp followed by a semicolon separated list of jars

 

[regulus]

Below is the heap usage log of TWS 938.1f on Win 7 64 bit with Java 1.7.0_21, API client is Multicharts 64. TWS froze after 3 hrs.

The log shows that heap usage is constantly growing when TWS runs.

I used the following parameters for TWS:

C:\Windows\System32\javaw.exe -cp jts.jar;total.2012.jar -Dsun.java2d.noddraw=true -Dswing.boldMetal=false -Dsun.locale.formatasdefault=true -Xloggc:mygc.log -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -Xmx1G -XX:MaxPermSize=128M jclient/LoginFrame C:\Jts

Any suggestions what I should try next to identify the issue are welcome.

 

[ddecker]

Quote from regulus:

Below is the heap usage log of TWS 938.1f on Win 7 64 bit with Java 1.7.0_21, API client is Multicharts 64. TWS froze after 3 hrs.

This is indeed troubling. It is freezing because the JVM is eventually spending all of it's time doing GC to try to stay below the -Xmx1G limit. You can see this in the purple "Other full GC" entries. Presumably a larger maximum heap size just delays the time taken to reach this point.

I know you prefer TWS, but for completeness, you should run the same test with the Gateway.

I don't know anything about Multicharts, but I presume one of the main differences between our setups (assuming this happens on the Gateway as well) is that yours uses the historical data API, while I only stream data/submit orders. One might therefore hypothesize that a memory leak has been introduced somewhere in this area.

There are further things that can be done to narrow down where this is occurring, but it's not really your job to be doing IB's testing for them. I'd repeat the test with the Gateway, submit all of the results to IB and demand they investigate.

 

[regulus]

Quote from ddecker:

I know you prefer TWS, but for completeness, you should run the same test with the Gateway.

Below is the heap usage for IB Gateway 938.1f (Java version is 1.7.0_21 64 bit and OS is Windows 7 64 bit).

The log shows a very similar behavior compared to TWS. After around 8.000 seconds of operation, when the -Xmx1G limit is reached, the Gateway freezes.

Start parameters are:
C:\Windows\System32\javaw.exe -cp jts.jar;hsqldb.jar;jcommon-1.0.12.jar;jhall.jar;other.jar;rss.jar -Dsun.java2d.noddraw=true -Xloggc:mygc.log -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -Xmx1G -XX:MaxPermSize=128M ibgateway.GWClient C:\Jts

For reasons of completeness, I will post a log with 932 in order to demonstrate that this version is not affected by the memory leak.

 

[ddecker]

Quote from regulus:

Below is the heap usage for IB Gateway 938.1f (Java version is 1.7.0_21 64 bit and OS is Windows 7 64 bit).

The log shows a very similar behavior compared to TWS. After around 8.000 seconds of operation, when the -Xmx1G limit is reached, the Gateway freezes.

Start parameters are:
C:\Windows\System32\javaw.exe -cp jts.jar;hsqldb.jar;jcommon-1.0.12.jar;jhall.jar;other.jar;rss.jar -Dsun.java2d.noddraw=true -Xloggc:mygc.log -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -Xmx1G -XX:MaxPermSize=128M ibgateway.GWClient C:\Jts

For reasons of completeness, I will post a log with 932 in order to demonstrate that this version is not affected by the memory leak.

It's interesting that it's happening well before hitting 1G. Could you post the log itself?

Perhaps the TWS code is getting a little overzealous in it's memory micromanagement. For kicks, try the following:

add:
-XX:+DisableExplicitGC
-XX:+UseConcMarkSweepGC
-XX:+CMSClassUnloadingEnabled
-XXermSize=256m

change:
-XX:MaxPermSize=256m

 

[regulus]

Quote from ddecker:

It's interesting that it's happening well before hitting 1G. Could you post the log itself?

Perhaps the TWS code is getting a little overzealous in it's memory micromanagement. For kicks, try the following:

add:
-XX:+DisableExplicitGC
-XX:+UseConcMarkSweepGC
-XX:+CMSClassUnloadingEnabled
-XXermSize=256m

change:
-XX:MaxPermSize=256m

ddecker,
below is the log for IB Gateway 938.1f.

I'll have another test with the parameters you suggested.

 

[regulus]

Quote from regulus:

ddecker,
below is the log for IB Gateway 938.1f.

Files size of the log file was too big. Another try with zip:

 

[regulus]

And here is the heap usage for Gateway 932.4, for completeness.

With this version, Java garbage collection works as it should and no freeze occurs.