[ts-gen] History queries -- Part 1: upstream limitations
pippin at owlriver.net
Fri Jun 11 16:50:02 EDT 2010
This is the 1st in a sequence of messages related to history queries
from the shim, via the IB tws api, to its upstream history farms.
IB places a number of restrictions on the frequency of, and values
that may occur in, history queries. This messages focuses on
the key time limits that IB uses to control load on its upstream
history farms, since these pacing limitations are not otherwise
intuitive, and have turned out to add significant complexity to
To place this message in context, history queries sent in violation
of pacing limits receive error messages in return rather than query
answers, and in the past downstream scripts bore responsibility for
obeying those pacing limits. New features in the shim take on this
responsibility, and this message is the first of several to explain
how now to use the shim to collect history via the IB tws api.
The specification material below is drawn in part from p. 20,
"Historical Data Limitations", of the api guide, and other material
in this series will draw from pp 214--216, "reqHistoricalData()",
about arguments for the IB C++ sample client history query request
Please feel free to refer back to the original source text, as
opposed to depending solely on the material below. The api guide
may be downloaded from IB's website, typically as a file named
"APIprintable.pdf", and you should obtain a copy of this file if you
have not already done so.
Other api specification issues for history, and in particular
the detailed format of history requests, and so the attribute
values that may be used for such data elements as "what-to-show",
are covered in the IB api guide manual, and will not be repeated
herein, except as they are related to the shim command language,
in which case they are referred to tangentially in a later message
focused on history query command syntax.
I've rephrased the rules below from the api manual; there the text
simply warns that pacing violations may occur, and the limits are
stated in part as violations, rather than in terms of the safe upper
IB places the following limits on the use of history queries made via
the tws api:
0. The query duration and bar size values must define an answer
with at most 2000 bars.
1. No more than 60 requests may be made in any ten-minute period.
2. At most five requests for the same contract, exchange, and tick
type may be made over the course of two seconds.
3. Identical requests must be spaced more than 15 seconds apart.
Limit (0) above is not the focus of this message, and will be considered
instead in the context of a later message in this sequence on the
detailed syntax of the history query command.
The shim does nothing to stop you from breaking restriction (3),
which makes it easy to trigger pacing violations for testing purposes.
The shim enforces pacing limits for the 1st and 2nd limits, the focus
of the next post, and I'll refer to those as the ten-minute and
Long in the past there was anecdotal evidence of a fourth limit, on
the number of queries that could be in flight at any one time, and
even claims that this limit was originally as low as one. Reading
between the lines of the api guide, the in-flight limits have now
increased either up to, or nearly to, the 60 request limit for the
ten-minute rule. It appears that the same-symbol rule is a more
fine-grained control over upstream history farm resource use, and
that any in-flight query limits that may have existed in the past
are replaced by rules (1) and (2) above.
As noted previously, when pacing violations occur, an error message
is returned in place of the query answer. Downstream user code
should therefore know of and watch for such errors. The pacing
violation error message is numbered as 162, and this and other error
messages of interest are listed below:
162 Historical Market Data Service error message: ...
200 No security definition has been found for the request
300 Can't find EId with tickerId: nnnnn
320 Error reading request:- ...
321 Error validating request:- ...
322 Error processing request:- ...
In the text above, the ellipsis refer to variable parts of the message,
and the string "nnnnn" refers to the tick id, a numeric identifier
originated by the shim and included with the original request. This
number appears as a distinct field in most api error messages, and in
particular, nearly always for each of the six above. When one of the
above messages includes a valid, outstanding tick id for a history
request, such an error message represents the only answer that request
will get; don't wait for the history message, for it's not coming.
As for 162, the typical case for text to replace the ellipsis above
is "Historical data request pacing violation" . Examples of
variations in the trailing text for 162 that I have observed in
the wild include:
162 ... :Historical data request pacing violation
162 ... :API historical data query cancelled: nnnnn|
162 ... :HMDS query returned no data: YM JUN 06 at ECBOT Trades|
162 ... :Socket broken while historical data query is pending.|
162 ... :TWS exited during processing of HMDS query.|
In any case, the query has been cancelled, and no answer will be
forthcoming unless and until the query is resubmitted.
Since the shim enforces the pacing restrictions for the ten-minute and
same-symbol rules in any particular session, 162 pacing violations for
downstream scripts using the shim are probably due to violation of the
third rule, on identical requests; jitter in network communication,
about which more in a later message; or back-to-back sessions against
the same account, in which case, depending on how greedy the first
session was, upstream pace accounting may need as much as ten minutes
before queries are accepted.
More information about the ts-general