Trading bot construction (IB)

doli · Jan 14, 2007

Thanks, edil. That looks useful.

doli · Jan 14, 2007

It looks like "Sample" might be a good framework in which to test code snippets. The first thing I've wanted is a button to get ES data and buttons to buy and sell ES. Bots don't want to fill out forms.

Here is what I did:

In IBJts/java/TestJavaClient, I modified SampleFrame.java:

Code:

        JButton butReqESData = new JButton( "Req ES Data");
        butClose.addActionListener( new ActionListener() {
            public void actionPerformed( ActionEvent e) {
                onReqESData();
            }
        });
        JButton butBuyES = new JButton( "Buy ES");
        butClose.addActionListener( new ActionListener() {
            public void actionPerformed( ActionEvent e) {
                onBuyES();
            }
        });
        JButton butSellES = new JButton( "Sell ES");
        butClose.addActionListener( new ActionListener() {
            public void actionPerformed( ActionEvent e) {
                onSellES();
            }
        });

and:

Code:

        buttonPanel.add( new JPanel() );
        buttonPanel.add( butReqESData );
        buttonPanel.add( butBuyES );
        buttonPanel.add( butSellES );

and:

Code:

    void onReqESData() {
    }

    void onBuyES() {
    }

    void onSellES() {
    }

The first code snippet was added after line 215; the second snippet was added after line 251; the third snippet was added after line 481.

To build, in the IBJts/java/TestJavaClient directory:

javac TestJavaClient/Main.java
javac TestJavaClient/ComboLegDlg.java

To run, in the IBJts/java directory:

java TestJavaClient.Main

The method calls in the last snippet are just stubs. I'll implement them after finding a good place to put them and after finding out which other methods of "Sample" they'll need to interact with. Anyway, when running it, you'll see a new panel, along the right side, at the bottom, with three new do-nothing buttons.

doli · Jan 15, 2007

A mistake has been made. Lesson: test code before posting. The first code snippet, above, should be

Code:

        JButton butReqESData = new JButton( "Req ES Data");
        butReqESData.addActionListener( new ActionListener() {
            public void actionPerformed( ActionEvent e) {
                onReqESData();
            }
        });
        JButton butBuyES = new JButton( "Buy ES");
        butBuyES.addActionListener( new ActionListener() {
            public void actionPerformed( ActionEvent e) {
                onBuyES();
            }
        });
        JButton butSellES = new JButton( "Sell ES");
        butSellES.addActionListener( new ActionListener() {
            public void actionPerformed( ActionEvent e) {
                onSellES();
            }
        });

I found the bug when I attempted to get the new market data button to work. Cut-and-paste is too easy!

Anyway, to get the "Req ES Data" button working, I added:

Code:

    void onReqESData() {

        Contract c = new Contract();
        int id = 4;

        c.m_symbol = "ES";
        c.m_secType = "FUT";
        c.m_expiry = "20070315";
        c.m_right = "";
        c.m_multiplier = "";
        c.m_strike = 0.0;
        c.m_exchange = "GLOBEX";
        c.m_primaryExch = "GLOBEX";
        c.m_currency = "USD";
        c.m_localSymbol = "ESH7";
        c.m_includeExpired = false;
        m_client.reqMktData(id, c, "100,101,104,106,162,165,221,225");
    }

right after the onReqNewsBulletins method, in SampleFrame.java. This addition needs some improvement: the 'id', 'c.m_expiry', and 'c.m_localSymbol' may not always be valid. I am learning Java.

By the way, if you're in the IBJts directory, to compile:

javac TestJavaClient/Main.java
javac TestJavaClient/ComboLegDlg.java

then to run "Sample":

java TestJavaClient.Main

There is a script in that directory, run.ksh, which builds and runs "Sample". On Windows, the commands in that script could be put in a '.bat' file, assuming that you're running the Java TWS and have the paths to the java tools set.

doli · Jan 19, 2007

Does anybody know about copyright with respect to IB and ET?

It seems to be that IB wouldn't mind the posting of snippets of their code or diffs against their code, because they produce the "Sample" API for people to use/extend. But I'd like to get the definitive word.

What about ET? I notice a copyright on the main page. Would I be better off doing my own blog?

doli · Jan 20, 2007

Getting the "Buy ES" and "Sell ES" buttons to work should be similar to what was done to get ES data. I'll get to that, later -- errands to do this weekend: download tax forms/instructions; get the dreaded taxes started and get to the point where I at least know that I've got all the irs pubs that I'll need.

Rediscovered 'xfig' for drawing -- got to review as it's been several years since I used it. It can export jpegs, so ... .

I'm thinking that instead of adding the three new buttons to "Sample," it might have been better to have a new JFrame (?) with just two buttons: buy and sell, along with bid/ask price/size, along with last price/size. Might make a decent scalping tool. Also, I'm thinking that an ability to ask for contract details would get around hard coding some of the info, as was done with getting the ES data to work.

Trading tools are interesting and useful, too. With volume/price ticks, all sorts of studies/indicators can be coded. But how to test them? Visualization? But that requires some charting capability, so that you can see, for example, that your moving average computation -- whatever -- is working. The Java 2D graphics API looks approachable. Before buying a Java book, I recommend that people google for "java tutorial" That's produced by Sun and is an excellent starting point. The only Java book I have is van der Linden's "Just Java 2," 6th edition.

As to starting a blog, I am looking at blojsom, which is written in Java and its use would align with my goal of learning Java, assuming I ever had to get into its guts. Ha ha! Would all these wonderful vendors I mention place ads on the blog?

doli · Jan 23, 2007

What I've got this time is a bare bones interface to the API. No GUI -- nothing but connecting to TWS.
It's done by replacing TestJavaClient/Main.java with this:

Code:

/*
 * Main.java
 *
 */

package TestJavaClient;

import java.io.*;

import com.ib.client.Contract;
import com.ib.client.ContractDetails;
import com.ib.client.EClientSocket;
import com.ib.client.EReader;
import com.ib.client.EWrapper;
import com.ib.client.Execution;
import com.ib.client.Order;
import com.ib.client.TickType;


class Wrapper implements EWrapper {

    int nextOrderId;

    EClientSocket client = new EClientSocket(this);

    Wrapper() {

        client.eConnect("127.0.0.1", 7496, 0);
        if (!client.isConnected()) {
            System.out.printf("Failed to connect\n");
            System.exit(1);
        }
        else
            ;
    }

    public void error(String s) {
        System.out.printf("%s\n", s);
    }

    public void error(int id, int code, String msg) {
        System.out.printf("%s\n", msg);
    }

    public void error(Exception e) {
        System.out.printf("%s\n", e);
    }

    public void nextValidId(int id) {
        System.out.printf("Next order id is %d\n", id);
        nextOrderId = id;
    }

    public void connectionClosed() {
        System.out.printf("Connection closed\n");
    }

    public EReader createReader(EClientSocket socket, DataInputStream dis) {
        return new EReader(socket, dis);
    }


    // to be implemented:

    public void tickPrice(int id, int field, double price, int canAutoExecute) {
    }

    public void tickSize(int id, int field, int size) {
    }

    public void tickOptionComputation(int id, int field, double impliedVol, double delta, double modelPrice, double pvDividend) {
    }

    public void orderStatus(int id, String status, int filled, int remaining, double avgFillPrice, int permId, int parentId, double lastFillPrice, int clientId) {
    }

    public void openOrder(int id, Contract contract, Order order) {
    }

    public void updateAccountValue(String key, String value, String currency, String accountName) {
    }

    public void updatePortfolio(Contract contract, int position, double marketPrice, double marketValue, double averageCost, double unrealizedPNL, double realizedPNL, String accountName) {
    }

    public void updateAccountTime(String timeStamp) {
    }


    public void contractDetails(ContractDetails contractDetails) {
    }

    public void bondContractDetails(ContractDetails contractDetails) {
    }

    public void execDetails(int id, Contract contract, Execution execution) {
    }

    public void updateMktDepth(int id, int position, int operation, int side, double price, int size) {
    }

    public void updateMktDepthL2(int id, int position, String marketMaker, int operation, int side, double price, int size) {
    }

    public void updateNewsBulletin(int id, int msgType, String message, String origExchange) {
    }

    public void managedAccounts(String accountsList) {
    }

    public void receiveFA(int faDataType, String xml) {
    }

    public void historicalData(int reqId, String date, double open, double high, double low, double close, int volume, int count, double WAP, boolean hasGaps) {
    }

    public void scannerParameters(String xml) {
    }

    public void scannerData(int reqId, int rank, ContractDetails contractDetails, String distance, String benchmark, String projection) {
    }

    public void tickGeneric(int tickerId, int tickType, double value) {
    }

    public void tickString(int tickerId, int tickType, String value) {
    }
}

public class Main {

    public static void main (String args[]) {

        Wrapper wrapper = new Wrapper();

        try {
            for (;;)
                Thread.sleep(1000);
        } catch (Exception e) { wrapper.error(e); }
    }
}

That can be built by running:
javac TestJavaClient/Main.java
Then run with:
java TestJavaClient/Main

If all goes well, you see something like:

Code:

Server Version:31
TWS Time at connection:20070123 16:29:19 MST
Next order id is 4
Market data farm connection is OK:hkfarm
Market data farm connection is OK:usfuture
Market data farm connection is OK:usfarm
HMDS data farm connection is inactive but should be available upon demand.:ushmds2a
HMDS data farm connection is inactive but should be available upon demand.:hkhmds2

It can be exited by typing ^C (control-C).

What has happened is that the SampleFrame class has been replaced with the Wrapper class, which implements EWrapper, like SampleFrame used to do. Some of the stubs, in the Wrapper class, could be eliminated if their abstractions in the EWrapper class were to be commented out, but I've left everyting there as something to be implemented; however, I probably won't implement things like financial advisor, scanner and market depth.

I'll proceed with this, later. I think that there will be a some GUI code.

doli · Jan 24, 2007

I have several questions about the last code:

1. Should 'Wrapper' be run as a separate thread? Note the 'sleep' in 'main'. When 'main' sleeps, does 'Wrapper' sleep, too?

2. With the line "package TestJavaClient", does all the code in the TestJavaClient directory get compiled, even though it isn't used?

3. In 'Wrapper', what is the difference between instantiating 'client' outside the constructor vs. inside the constructor?

4. Could a server socket be implemented, too? If so, things like market data could be distributed to other apps. Several bots could run at the same time. There is a limit to the number of connections that tws will accept; there may also be a limit to the number of messages that tws and tws' server will process, so it may not be possible to run a 1000 active trader bots at once. Another possible client is a "chart," and as a client to this rather than tws, its subscription to and processing of data wouldn't depend on any particular API.

dcraig · Jan 24, 2007

Quote from doli:

I have several questions about the last code:

1. Should 'Wrapper' be run as a separate thread? Note the 'sleep' in 'main'. When 'main' sleeps, does 'Wrapper' sleep, too?

The various functions in EWrapper are called from the receive thread created within the API. The receive thread blocks on a socket waiting for messages from TWS. When it receives a message, it decodes it and calls the appropriate function in EWrapper implemented in your Wrapper. As per the JavaDoc, Thread.sleep() causes the currently executing thread to sleep (ie the calling thread which is the thread that main() is executing in in your case).

From the API source:

Code:

class EReader extends Thread

Use the source Luke !

3. In 'Wrapper', what is the difference between instantiating 'client' outside the constructor vs. inside the constructor?

Probably not much.

4. Could a server socket be implemented, too? If so, things like market data could be distributed to other apps. Several bots could run at the same time. There is a limit to the number of connections that tws will accept; there may also be a limit to the number of messages that tws and tws' server will process, so it may not be possible to run a 1000 active trader bots at once. Another possible client is a "chart," and as a client to this rather than tws, its subscription to and processing of data wouldn't depend on any particular API.

Yes. But you wouldn't be using TWS to run 1000 bots.

dcraig · Jan 24, 2007

doli,

One possible way (and the way I do it) of handling "tick events" in the EWrapper methods is to create a tick object and put it in a queue. The receive thread in the API can then get on with doing it's business which is simply to read from the TCP connection to TWS.

I recommend you time stamp each 'tick event' at this stage and use a java.util.concurrent.ConcurrentLinkedQueue which is thread safe.

You can create one or more queue reader threads to do what you like with the queued "ticks".

doli · Jan 24, 2007

Thanks for the input, dcraig.

I've only read the first 6 chapters of van der Linden's book, so far. But that didn't stop me from trying to setup a server socket and get a minimal subscription service going. There is a new Main.java and a test1.java. Main.java now listens for requests, which can be sent from test1.java. Main.java may change after I read some more about Java threads and networking, unless somebody hits me with a clue-by-four before then. I took the "package " statement out of Main.java, too.

This has been built and run on Windows and Solaris 10. One thing about subscribers: if a subscriber dies, then the thread that talks to that subscriber should die and the subscriber should be removed from the subscriber list. That happens, but when a subscriber is killed by typing ^C on 'test1' the behavior is different on Windows than it is on Solaris. With Windows, a connection reset exception occurs, but with Solaris I had to throw the 'OutOfGas' exception to kill the thread. On both systems the subscriber is removed from the subscriber list when the next socket write occurs, because the write attempt causes an exception; however, the write socket exception on Windows is a socket write error, but on Solaris it is a broken pipe exception. I'll have to look into that some more -- it may be working by accident more than design -- on Windows in.readLine() is generating the exception, but on Solaris the OutOfGas exception is generated if in.readLine() returns null. That brings up another point: it may be best to get away from line-oriented message passing.

There may be more overhead this way than with, say, a "chart" thread or a "trader" thread as part of this app. With those threads in the same app (the client of tws), they could wait for work to arrive on a queue in local memory rather than wait for a message to arrive over the network. Something to consider.

Here is the latest Main.java:

Code:

/*
 * Main.java
 *
 */


import java.io.*;
import java.net.*;
import java.util.*;

import com.ib.client.Contract;
import com.ib.client.ContractDetails;
import com.ib.client.EClientSocket;
import com.ib.client.EReader;
import com.ib.client.EWrapper;
import com.ib.client.Execution;
import com.ib.client.Order;
import com.ib.client.TickType;


class Subscriber {

    Connection client;

    Subscriber(Connection c) {
        client = c;
    }
}


class Connection extends Thread {

    Socket sock;
    BufferedReader in;
    DataOutputStream out;

    Connection(Socket s) throws Exception {

        try {

            sock = s;
            in = new BufferedReader(
                      new InputStreamReader(sock.getInputStream())
                     );
            out = new DataOutputStream(sock.getOutputStream());
        }
        catch (Exception e) {
            System.out.printf("Connection: caught %s\n", e);
            System.exit(1);
        }
    }


    class OutOfGas extends Exception {
        OutOfGas(String s) {super(s);}
    }

    public void run() {

        String s;

        for (;;) {
            try {
                s = getRequest();
                // handle the request:
                reply("Not much ...\n");
            } catch (Exception e) {
                System.out.printf("Connection.run: caught %s\n", e);
                // break out of the loop,
                // which will then cause a fall off the end of run(),
                // which should kill this thread.
                break;
            }
        }
    }

    String getRequest() throws Exception {

        String s = null;

        s = in.readLine();

        if (s == null)
            throw new OutOfGas("whoops!");
        else
            return s;
    }

    void reply(String s) throws Exception {
        out.writeBytes(s);
    }
}


    
class Distributor extends Thread {

    LinkedList <Subscriber> subscriber = new LinkedList<Subscriber>();

    public void run() {

        try {

            ServerSocket ssock = new ServerSocket(6500);
            for (;;) {
                System.out.printf("Distributor: listening\n");
                Socket sock = ssock.accept();
                System.out.printf("Distributor: client connected\n");
                Connection client = new Connection(sock);

                // save each 'client', on a 'Subscriber' list
                synchronized(subscriber) {
                    subscriber.add(new Subscriber(client));
                }

                client.start();
                System.out.printf("no. of threads is %d\n", Thread.activeCount());
            }
        }
        catch (Exception e) {
            System.out.printf("Distributor.run: caught %s\n", e);
            System.exit(1);
        }  
    }

    public void echo(String s) throws Exception {    // BUG? throws?

        synchronized(subscriber) {

            Iterator<Subscriber> sub = subscriber.listIterator();

            while (sub.hasNext()) {
                try {
                       sub.next().client.reply(s);
                }
                catch (Exception e) {
                    System.out.printf("Distributor.echo: caught %s\n", e);
                    System.out.printf("no. of threads is %d\n", Thread.activeCount());
                    sub.remove();    // zombie clients are removed from the list
                }
            }
        }
    }
}


class Wrapper implements EWrapper {

    int nextOrderId;
    Distributor distribute;
    EClientSocket client = new EClientSocket(this);

    Wrapper(Distributor distributor) {

        this.distribute = distributor;
        client.eConnect("127.0.0.1", 7496, 0);
        if (!client.isConnected()) {
            System.out.printf("Failed to connect\n");
            System.exit(1);
        }
        else
            ;
    }


    public void error(String s) {
        System.out.printf("%s\n", s);
    }

    public void error(int id, int code, String msg) {
        System.out.printf("%s\n", msg);
    }

    public void error(Exception e) {
        System.out.printf("%s\n", e);
    }

    public EReader createReader(EClientSocket socket, DataInputStream dis) {
        return new EReader(socket, dis);
    }

    public void nextValidId(int id) {
        System.out.printf("Next order id is %d\n", id);
        nextOrderId = id;
    }

    public void tickPrice(int id, int field, double price, int canAutoExecute) {
    }

    public void tickSize(int id, int field, int size) {
    }

    public void tickOptionComputation(int id, int field, double impliedVol, double delta, double modelPrice, double pvDividend) {
    }

    public void orderStatus(int id, String status, int filled, int remaining, double avgFillPrice, int permId, int parentId, double lastFillPrice, int clientId) {
    }

    public void openOrder(int id, Contract contract, Order order) {
    }

    public void updateAccountValue(String key, String value, String currency, String accountName) {
    }

    public void updatePortfolio(Contract contract, int position, double marketPrice, double marketValue, double averageCost, double unrealizedPNL, double realizedPNL, String accountName) {
    }

    public void updateAccountTime(String timeStamp) {
    }


    public void contractDetails(ContractDetails contractDetails) {
    }

    public void bondContractDetails(ContractDetails contractDetails) {
    }

    public void execDetails(int id, Contract contract, Execution execution) {
    }

    public void updateMktDepth(int id, int position, int operation, int side, double price, int size) {
    }

    public void updateMktDepthL2(int id, int position, String marketMaker, int operation, int side, double price, int size) {
    }

    public void updateNewsBulletin(int id, int msgType, String message, String origExchange) {
    }

    public void managedAccounts(String accountsList) {
    }

    public void receiveFA(int faDataType, String xml) {
    }

    public void historicalData(int id, String date, double open, double high, double low, double close, int volume, int count, double WAP, boolean hasGaps) {
    }

    public void scannerParameters(String xml) {
    }

    public void scannerData(int id, int rank, ContractDetails contractDetails, String distance, String benchmark, String projection) {
    }

    public void tickGeneric(int id, int tickType, double value) {
    }

    public void tickString(int id, int tickType, String value) {
    }

    public void connectionClosed() {
        System.out.printf("Connection closed\n");
    }
}


public class Main {

    public static void main (String args[]) {

        Distributor distribute = new Distributor();
        new Thread(distribute).start();

        Wrapper wrapper = new Wrapper(distribute);

        try {
            for (;;) {
                Thread.sleep(10000);
                distribute.echo("main is awake\n");
            }
        } catch (Exception e) { wrapper.error(e); }
    }

}