Topics

Experience with the new ib python api?

Spinor 8
 

Hi All,


  In an earlier post, dam5h mentioned that IB released their own python api this month. This was news to me. I immediately downloaded it (part of IB API v 973 beta) and intend to tinker around with it in the next few days. I was wondering if there are other Pythonistas on the forum who have spent time exploring the new python api and are willing to share their thoughts on it.


Regards,

Stanley.



mdelvaux2
 

This is great news.  I have been working with IbPy for a while, but always felt a little worried about support, an IB supported python API removes that fear.

The API reference has indeed now a python tab, and a quick scan of the source shows that the port is extensive, and the example is extensive.  It will take some time to digest what IB has done and how easy it will be port to translate existing application code to use the new API binding.

ciherraiz@...
 

I coded some lines to try the new API. I'm a begginer on TWS API. I connected to IB Gateway and I received some details about the account and portfolio. But I have a problem, when I execute the run() method of EClient class, I run the loop and I can't use others commands to place orders o something like that. I don't know if I have to use a thread or not.

I think, I don't understand the API well.

Did anybody do an example requesting any data of the account and placing some orders?

Thanks!!!

Ewald de Wit
 

The simplest way to start off is to inherit from EWrapper and override these methods:
  • connectAck, this is called right after connecting and you can place for example data requests here;
  • updateAccountTime, is called every second (or couple of seconds?). You can place your evaluation logic here and fire orders.
To use the updateAccountTime() callback for periodic evaluation is a bit of a hack. To get better control you could replace EClient.run() with something that periodically calls into your evaluation logic.

Another option is to replace the run method with a proper event loop. The loop is then external and calls into the API instead of the other way around. This is however a lot harder to start with for a beginner.

My experience with the beta Python API so far is that it works well enough. It uses a bit more CPU then what I had before (talking directly into a socket, also with Python) but this is negligible in terms of added latency. The biggest complaint would be the excessive logging that it does, and on the root logger too were it can not be filtered away easily.

cheers,

Ewald

On Sun, Feb 12, 2017 at 11:35 AM, ciherraiz via Groups.Io <ciherraiz@...> wrote:

I coded some lines to try the new API. I'm a begginer on TWS API. I connected to IB Gateway and I received some details about the account and portfolio. But I have a problem, when I execute the run() method of EClient class, I run the loop and I can't use others commands to place orders o something like that. I don't know if I have to use a thread or not.

I think, I don't understand the API well.

Did anybody do an example requesting any data of the account and placing some orders?

Thanks!!!

comicpilsen
 

could someone please post a simple sample python script for the 973 api?  I just want to get a feel for the most expedient way to start, just getting a stock quote would be excellent.  I was going to use IBPY but thought this "official" python api would have a longer life.

ciherraiz@...
 

Hi,

I followed the first idea explained by Ewald the Wit (thanks!!!!), inheriting from EWrapper. I post some code:


I inherited from EWrapper (ibwrapper.py):

from ibapi.wrapper import EWrapper
from ibapi.contract import (Contract, ContractDetails)

class IBWrapper(EWrapper):

    def connectAck(self):
            print("\n[Connected]")
            self.reqManagedAccts()
            self.reqAccountUpdates(True, self.account)
    def updateAccountTime(self, timeStamp:str):
        super().updateAccountTime(timeStamp)
        print("UpdateAccountTime. Time:", timeStamp)


I implemented my own Broker class (ibbroker.py):


from technology.broker.ib.ibwrapper import IBWrapper
from technology.broker.ib.ibclient import IBClient

class IB(IBWrapper, IBClient):
    def __init__(self, account):
        IBWrapper.__init__(self)
        IBClient.__init__(self, wrapper=self)
        self.account = account

class IBBroker:

    def __init__(self, account):
        self.ib = IB(account)

    def start(self, cID):
        print("Starting client ID: {}".format(cID))
        self.ib.connect("127.0.0.1", 4001, clientId=cID)
        print("\nServer version: {}   Connection time: {}\n" \
        .format(self.ib.serverVersion(), self.ib.twsConnectionTime()))

    def run(self):
        self.ib.run()

    def stop(self):
        self.ib.disconnect()


And I called an IBBroker object here (app.py):

from technology.broker.ib.ibbroker import IBBroker

if __name__=="__main__":
    print ("\nRunning ...\n")
    broker = IBBroker('DUXXXXXX')
    broker.start(0)
    broker.run()
    print("\n...stopping\n")
    broker.stop()


The TWS API explains about updateAccountTime event:

Just as with the TWS' Account Window, this information is updated every three minutes.

I will try the last way that Ewald the Wit said: "replace the run method with a proper event loop". I will start overriding the run() method of Eclient copying the original code.


Carlos.






Ewald de Wit
 

Carlos, this came up in another thread too; I posted this example of doing it with an event loop. It's a fully working demo and it should be straightforward to try it out. The connection info needs to be adjusted to your 4001 port.

At the end of the script you will see loop.run_forever() instead of the broker.run() in your code. What's the difference? It looks nearly the same but it is actually a complete loop inversion. With the loop from broker.run() there is no way to hook in your own code (other then waiting for a callback), while with the loop from asyncio anything becomes possible. Check the asyncio documentation, for example the methods call_at or call_later.
At the end of the script you will see loop.run_forever() instead of the broker.run() in your code. What's the difference? It looks nearly the same but it is actually a complete loop inversion. With the loop from broker.run() there is no way to hook in your own code (other then waiting for a callback), while with the loop from asyncio anything becomes possible. Check the asyncio documentation, for example the methods call_at or call_later.

good luck,

Ewald

On Fri, Feb 17, 2017 at 7:38 PM, ciherraiz via Groups.Io <ciherraiz@...> wrote:

Hi,

I followed the first idea explained by Ewald the Wit (thanks!!!!), inheriting from EWrapper. I post some code:


I inherited from EWrapper (ibwrapper.py):

from ibapi.wrapper import EWrapper
from ibapi.contract import (Contract, ContractDetails)

class IBWrapper(EWrapper):

    def connectAck(self):
            print("\n[Connected]")
            self.reqManagedAccts()
            self.reqAccountUpdates(True, self.account)
    def updateAccountTime(self, timeStamp:str):
        super().updateAccountTime(timeStamp)
        print("UpdateAccountTime. Time:", timeStamp)


I implemented my own Broker class (ibbroker.py):


from technology.broker.ib.ibwrapper import IBWrapper
from technology.broker.ib.ibclient import IBClient

class IB(IBWrapper, IBClient):
    def __init__(self, account):
        IBWrapper.__init__(self)
        IBClient.__init__(self, wrapper=self)
        self.account = account

class IBBroker:

    def __init__(self, account):
        self.ib = IB(account)

    def start(self, cID):
        print("Starting client ID: {}".format(cID))
        self.ib.connect("127.0.0.1", 4001, clientId=cID)
        print("\nServer version: {}   Connection time: {}\n" \
        .format(self.ib.serverVersion(), self.ib.twsConnectionTime()))

    def run(self):
        self.ib.run()

    def stop(self):
        self.ib.disconnect()


And I called an IBBroker object here (app.py):

from technology.broker.ib.ibbroker import IBBroker

if __name__=="__main__":
    print ("\nRunning ...\n")
    broker = IBBroker('DUXXXXXX')
    broker.start(0)
    broker.run()
    print("\n...stopping\n")
    broker.stop()


The TWS API explains about updateAccountTime event:

Just as with the TWS' Account Window, this information is updated every three minutes.

I will try the last way that Ewald the Wit said: "replace the run method with a proper event loop". I will start overriding the run() method of Eclient copying the original code.


Carlos.






comicpilsen
 

Ewald

thanks so much for posting the code. I got the IB sample code program.py working but it logs EVERYTHING and I'm a little worried about hacking anything out of it just in case I break something. Which thread do you discuss why you are using asynchio instead of IB's ibapi.connection.Connection ?  AND did you see my suggestion as to the new approach to asynch using curio ?

https://curio.readthedocs.io/en/latest/tutorial.html



Ewald de Wit
 

Pilsen, it should be pretty safe to rip out the logging. The amount of logging is just too much, even every int or float conversion is logged. And on the root logger too. The common practice is to log on the module level, like logging.getLogger(__name__), or on the object level, like logging.getLogger(__class__.__name__).

Thank you for you suggestion to look at Curio, I had seen it briefly before and it is a very interesting project. It also has a run() method, but the user has to plug in his code as coroutines. The run method hops from coroutine to coroutine, without using an event loop. This is very efficient when there are many thousands of things happening (like reading and writing sockets). But for the ibapi, with its one socket connection, this is a bit overkill. For the ibapi I prefer an old-fashioned event loop with callback style event handling. Like there is an event called "somethingHappened" and it gets handled by the onSomethingHappened method. This design also makes it straightforward to play back the events (market data) and voila you have a backtester.

cheers,

Ewald

On Fri, Feb 17, 2017 at 11:21 PM, comicpilsen <comicpilsen@...> wrote:

Ewald

thanks so much for posting the code. I got the IB sample code program.py working but it logs EVERYTHING and I'm a little worried about hacking anything out of it just in case I break something. Which thread do you discuss why you are using asynchio instead of IB's ibapi.connection.Connection ?  AND did you see my suggestion as to the new approach to asynch using curio ?

https://curio.readthedocs.io/en/latest/tutorial.html



comicpilsen
 
Edited

Hey Ewald

I'm going to use your code sample and play with that. Nice work by the way.  I would like to take a minimalist approach by just starting with a request for the last prices of stocks ( say 20) whose symbols are held in a list. So I would pump out the 20 requests and watch what order they get completed in. Would you anticipate an UNordered completion or an ORDERED completion?

So I send out requests for AAPL, GE, FCX,BAC  ( and 16 others) and the requests would come back in the same order AAPL, GE, FCX,BAC( ORDERED)

or out of order  GE, BAC,AAPL,FCX (UNordered)

I haven't really started with the new API yet as I just got the test program program.py working yesterday on ubuntu 16.04 mate.  I WAS going to use the approach below ( pluging in the IB calls)  until I read what you said about COROUTINES, I would value your opinion on the approach. BUT could you watch this from DaveB just in case it's something interesting. https://www.youtube.com/watch?v=E-1Y4kSsAFc

import asyncio

@asyncio.coroutine
def hello_world():
    yield from asyncio.sleep(1)
    print('Hello World')
    asyncio.ff(hello_world())

@asyncio.coroutine
def good_evening():
    yield from asyncio.sleep(1)
    print('Good Evening')
    asyncio.async(good_evening())

print('step: asyncio.get_event_loop()')
loop = asyncio.get_event_loop()
try:
    print('step: loop.run_until_complete()')
    asyncio.async(hello_world())
    asyncio.async(good_evening())
    loop.run_forever()
except KeyboardInterrupt:
    pass
finally:
    print('step: loop.close()')
    loop.close()


J G
 

On Sat, Feb 18, 2017 at 02:42 pm, comicpilsen wrote:

I would like to take a minimalist approach by just starting with a request for the last prices of stocks ( say 20) whose symbols are held in a list. So I would pump out the 20 requests and watch what order they get completed in. Would you anticipate an UNordered completion or an ORDERED completion?

So I send out requests for AAPL, GE, FCX,BAC  ( and 16 others) and the requests would come back in the same order AAPL, GE, FCX,BAC( ORDERED)

or out of order  GE, BAC,AAPL,FCX (UNordered)

 I guess you want to do this while the markets are open and trading is ongoing? In that case you should assume that the responses from IB/TWS will be unordered. Why? Because IB sends you a price update once the latest trade price of an instrument changes (and no previous update was sent within x milliseconds, for stocks x = 250). This behavior is not specific to Python, but applies to IB in general.

comicpilsen
 

Hey JG

thanks for taking the time to help a noob out. Yes when the markets are open. One approach I am looking at is to get the Implied vol for a symbol find the 1sd and find out where the symbol's current price is related to that range.

I am planning out my approach and right now would like to look into asynchronous completion ( I understand IB lets you specify asynch vs synch) and finding the docs to be a little confusing.  Right now my approach is to set up a very simple program that establishes a valid connection to a READ ONLY live account, set up an awaitio loop, post requests for last_price on 100 symbols and process the responses from IB using io tag information to track them.  I'm trying to start on a good foundation and the fine people in this group are REALLY helping me out.

Thanks again JG for taking the time.



 

Ewald de Wit
 

Pilsen, looks like you're well on your way. Tomorrow it will get better with live data coming in and the whole thing coming alive. You have probably figured out how to use the reqId to tie request and response together. I wouldn't worry too much about the async stuff. If you're doing one thing at a time then you don't need it. When the need arises for some form of scheduling or concurrency then it well be there.

About the David B talk you linked, you should realize that it is pretty advanced stuff that is abused to the logical limits by a total madman. Thx for the link I enjoyed it very much.

Ewald



On Sun, Feb 19, 2017 at 11:28 AM, comicpilsen <comicpilsen@...> wrote:

Hey JG

thanks for taking the time to help a noob out. Yes when the markets are open. One approach I am looking at is to get the Implied vol for a symbol find the 1sd and find out where the symbol's current price is related to that range.

I am planning out my approach and right now would like to look into asynchronous completion ( I understand IB lets you specify asynch vs synch) and finding the docs to be a little confusing.  Right now my approach is to set up a very simple program that establishes a valid connection to a READ ONLY live account, set up an awaitio loop, post requests for last_price on 100 symbols and process the responses from IB using io tag information to track them.  I'm trying to start on a good foundation and the fine people in this group are REALLY helping me out.

Thanks again JG for taking the time.



 

comicpilsen
 

Hey Ewald

I agree, the first step is the biggest and that's why I want to make sure I set a firm foundation to build on. I want to look at the asynch stuff as that is my linage.  It's just getting over this first step. I want to make sure that my first hand coded program is rational in my new world of Python and Interactive brokers. I don't fear the complexity of asynchronous events, in fact I love it. It's just that I am finding it difficult to get the first step out of the way. Your help is INVALUABLE!!! Just to clarify the code ( bottom of message) is a foundation to the IB api approach?  I'd like to persue the asynch as my intent here is trade DISCOVERY not execution, so I will use some of the tried and tested ideas from my time in industry to find trade candidates ( using variance from 1sd using IV is just one of them) and then use that information to consider the strategy. The world is chaotic and events happen in any order and I want to look at that world in code and approach.  I am a HUGGGGEEE ( sorry couldn't resist it) fan of madmen and I find comfort in their company. DaveB is a wonderful person who I had an excellent breakfast with where he course corrected me on many of my Python misconceptions knowing I am a noob in this world.  So if you are ok with the loop below I was going to amalgamate it into your wonderful sample code and continue from there. Please let me know as I value your opinion. Maybe using my ignorance and your experience we can come up with the missing SIMPLE approach to asynchronous IB api event handling which will form the basis for all that come after us. Not a bad goal for a new world ( Python 3.6 and the python wrapper for IB API).

cp

here is the loop I was going to incorporate into your code



import asyncio

@asyncio.coroutine
def hello_world():
    yield from asyncio.sleep(1)
    print('Hello World')
    asyncio.ff(hello_world())

@asyncio.coroutine
def good_evening():
    yield from asyncio.sleep(1)
    print('Good Evening')
    asyncio.async(good_evening())

print('step: asyncio.get_event_loop()')
loop = asyncio.get_event_loop()
try:
    print('step: loop.run_until_complete()')
    asyncio.async(hello_world())
    asyncio.async(good_evening())
    loop.run_forever()
except KeyboardInterrupt:
    pass
finally:
    print('step: loop.close()')
    loop.close()

Dave Webb
 

Apologizes if this is a dumb noob question.

I seem to have installed the API using a degree of guess work. i.e. I downloaded the API from IB site as per their instructions then guessed I needed to on Mac OSX run sudo -H python setup.py install

But some of the example programs I end up with refer to the package "IBApi" rather than "ibapi". It can't find "IBApi" - Does anyone know if that is that just a hang over from some old naming convention and I should ignore the programs with "IBApi" or is indicative my guess work missed something? 

Thanks

Dave


Ewald de Wit
 

Pilsen, the code you posted is the general idea when to integrate with something non-ibapi related. For example an embedded webserver or telnet server er whatever running inside the trading app.

For the use that you describe check out the call_at method. With that you can periodically calculate your stddevs.

Ewald



On Sun, Feb 19, 2017 at 5:23 PM, comicpilsen <comicpilsen@...> wrote:

Hey Ewald

I agree, the first step is the biggest and that's why I want to make sure I set a firm foundation to build on. I want to look at the asynch stuff as that is my linage.  It's just getting over this first step. I want to make sure that my first hand coded program is rational in my new world of Python and Interactive brokers. I don't fear the complexity of asynchronous events, in fact I love it. It's just that I am finding it difficult to get the first step out of the way. Your help is INVALUABLE!!! Just to clarify the code ( bottom of message) is a foundation to the IB api approach?  I'd like to persue the asynch as my intent here is trade DISCOVERY not execution, so I will use some of the tried and tested ideas from my time in industry to find trade candidates ( using variance from 1sd using IV is just one of them) and then use that information to consider the strategy. The world is chaotic and events happen in any order and I want to look at that world in code and approach.  I am a HUGGGGEEE ( sorry couldn't resist it) fan of madmen and I find comfort in their company. DaveB is a wonderful person who I had an excellent breakfast with where he course corrected me on many of my Python misconceptions knowing I am a noob in this world.  So if you are ok with the loop below I was going to amalgamate it into your wonderful sample code and continue from there. Please let me know as I value your opinion. Maybe using my ignorance and your experience we can come up with the missing SIMPLE approach to asynchronous IB api event handling which will form the basis for all that come after us. Not a bad goal for a new world ( Python 3.6 and the python wrapper for IB API).

cp

here is the loop I was going to incorporate into your code



import asyncio

@asyncio.coroutine
def hello_world():
    yield from asyncio.sleep(1)
    print('Hello World')
    asyncio.ff(hello_world())

@asyncio.coroutine
def good_evening():
    yield from asyncio.sleep(1)
    print('Good Evening')
    asyncio.async(good_evening())

print('step: asyncio.get_event_loop()')
loop = asyncio.get_event_loop()
try:
    print('step: loop.run_until_complete()')
    asyncio.async(hello_world())
    asyncio.async(good_evening())
    loop.run_forever()
except KeyboardInterrupt:
    pass
finally:
    print('step: loop.close()')
    loop.close()

Ewald de Wit
 

Yes the programs in the test directory import from IBApi which doesn't work. It's probably not updated yet. In the zip file these is also a samples directory with working programs.

Ewald

On Sun, Feb 19, 2017 at 6:08 PM, Dave Webb <dave@...> wrote:

Apologizes if this is a dumb noob question.

I seem to have installed the API using a degree of guess work. i.e. I downloaded the API from IB site as per their instructions then guessed I needed to on Mac OSX run sudo -H python setup.py install

But some of the example programs I end up with refer to the package "IBApi" rather than "ibapi". It can't find "IBApi" - Does anyone know if that is that just a hang over from some old naming convention and I should ignore the programs with "IBApi" or is indicative my guess work missed something? 

Thanks

Dave


comicpilsen
 

Hey Ewald

thanks for correcting me, I REALLY am not getting the documentation with the IB API for python. That is my fault not IB's.  here's the pseudo code of what I want to do, could you just look it over and tell me if it makes sense. I am looking to use coroutines ALA https://www.youtube.com/watch?v=l4Nn-y9ktd4    THANK for ANY help you can offer.

here's the VERY high level for the asynch code

establish a valid connection with TWS desktop listening on port 7496


send off  requests for last price and implied volatility for 100 different symbols noting each separate reqid

 start an awaitio loop which loops until process is killed

               when an reqid completes ( successfully) perform some action on the data 

               send off another request for the symbol that just completed

              







J G
 

On Mon, Feb 20, 2017 at 01:57 pm, comicpilsen wrote:
               send off another request for the symbol that just completed

 I think that this is not necessary. Once you subscribe to receiving market data (using reqMktData) you will remain subscribed and will continue to receive new data until you send a message to unsubscribe from it.

Ewald de Wit
 

Pilsen,

The good news is that you don't need coroutines at all. Really just forget about them it will only lead to confusion

Understand that EVERYTHING happens inside the loop. Once you let the loop run you can then only react to events. It's known as the reactor pattern.

The quickest way to learn how the ibapi really works is to start off with running a sample script, see what it does and what results it gives and then add the things you want to it. About reqMktData, it can give either streaming data (snapshot=False) or just the last state (snapshot=True). The first option has a maximum of 100 tickers and second one is subject to pace limitations.

Also take a look at the market scanner as it may do all that is needed.

good luck,

Ewald


On Mon, Feb 20, 2017 at 10:57 PM, comicpilsen <comicpilsen@...> wrote:

Hey Ewald

thanks for correcting me, I REALLY am not getting the documentation with the IB API for python. That is my fault not IB's.  here's the pseudo code of what I want to do, could you just look it over and tell me if it makes sense. I am looking to use coroutines ALA https://www.youtube.com/watch?v=l4Nn-y9ktd4    THANK for ANY help you can offer.

here's the VERY high level for the asynch code

establish a valid connection with TWS desktop listening on port 7496


send off  requests for last price and implied volatility for 100 different symbols noting each separate reqid

 start an awaitio loop which loops until process is killed

               when an reqid completes ( successfully) perform some action on the data 

               send off another request for the symbol that just completed

              







Previous Topic Next Topic