Topics

Blocked after 60 requests

leo tanner
 

Just request historical ticks, there is no 60 request per 10 minutes limit. But the only problem there is that, the ticks within the same second will be labeled as having the same timestamp (egg 8:00:05 price 100 and 8:00:05 100.5) However since they are ranked with the order of arrival, take the last tick to get the 1 sec data they provide.

Hope that helps 

Wiliam Ornelas
 

Do you have recommendations on where to get historical data? If I want historical second or ticker data, is there a source you would recommend?

I've also thought about the following:
 - run my processes very close to a historical data provider
 - such that: I can rely on their own database for storage
 - with the expectation that: I can request large amounts of that data at random times,
 - with a small enough latency.

If that is not possible, I am also willing to host the database, but I still need to find a proper provider of this data.

I find that requesting Interactive Brokers (IB) I get data within 1 to 2 seconds for 1800 second tickers. I think that it would be nice to use IB if I could get data within a few milliseconds. I don't think I've seen any methods to reduce the latency for these requests.

mdelvaux2
 

This limit specifically applies to historical data requests, limitations for streaming data are quite different
- you don't need repeated requests, you do one request then the quotes stream to you without further intervention
- you can get 100 (normal) level 1 streams at the same time, and you can pay to get more of these.  These provide aggregated quotes, 4 times per second for stock
- you can get a smaller number of non aggregated (tick) level 1 streams or level 2 streams

As far as I know, you cannot pay to get better historical data access. 

More details
https://interactivebrokers.github.io/tws-api/top_data.html 
https://interactivebrokers.github.io/tws-api/market_depth.html 

Wiliam Ornelas
 

I am wondering if they would considering lifting these limits to customers who pay more? We are willing to pay more. Also, what if we buy Level-II book data? I think that was one of the subscriptions they provide. Surely, they would let us query that data at the same rate, since level-II book data would mean even more data. More data would take a long time to gather if they set the same limits.

mdelvaux2
 

You are probably hitting the pacing limit for historical data.  If you print the error messages you got, you may see something about that.  In short, IB has a limitation, 1 historical data request per 10 seconds.  Currently IB waives that limit for bar sizes  larger than 30 seconds (it used to limit any historical data requets), and you are below that limit.

The exact text can be found at https://interactivebrokers.github.io/tws-api/historical_limitations.html. The last one (quoted below) is what applies to you

  • Making more than 60 requests within any ten minute period.

Wiliam Ornelas
 

Hi,

I am hitting an issue when querying against IB many times. 
I am using ib_insync.
Note that I am using something similar to https://github.com/ibizaman/docker-ibcontroller to run the client that connects to the IB API.

This is what I do:
For a stock, like TSLA for example,
For a time range, 2018-04-03 00:00:00 until 2018-04-05 00:00:00 for example,
For datatypes ASK, BID, TRADES,
Do:
Request 30 mins of 1 second ticker data (1800 tickers) of the datatype D, for the stock S, for the endTime T.  This amount to about more than 150 requests.
I run these in a loop, and I take a 1 second break AND I store the data to a database between each request. 

I see that after 60 requests are properly serviced, on the 61th request against IB, I get a 0rows (0 tickers) from IB in the response.

I have gone from a very concurrent solution to a simple for loop, so I am pretty sure the issue is not with my implementation.

I have tried the following:
 - Create a client with a different client_id
 - Wait a certain amount of time after every chunk of 20 requests
 - Wait a few more seconds after each request
 - Shuffle the order of the requests
 - Do requests for one stock A first, then a stock B for second request, and again stock A for third request, and stock B for fourth request, and on and on.. 
    - (Essentially, alternate requests against stock A and stock B, in the same time range)
 - Query smaller chunks of data: 1000 tickers instead of 1800.

I realized though that if it stops after 60 requests for that job I was running, and I try to re-run the job right after, I would get 0 rows in the response for the very first request.
This is when I realized that IB was probably not returning anything because of the high amount of requests I was sending.

How do you request large amount of historical 1 second ticker data from Interactive Brokers?

I am thinking maybe IB is not the right tool to get this data in big amounts?
Maybe I am just doing something wrong and someone else is able to not get the limit I am getting.
Maybe I should subscribe to some increased quota?

Thank you,

Wilornel