[ts-gen] Re: The Shim as a data recorder?

[R P Herrold]

On Mon, 25 Sep 2006, (an anonymous user) wrote:

> I'm currently involved in a (very much amateur) automated 
> trading project, and am looking for an ideal way to build an 
> ultra-fine-grained database of historical data--basically, I 
> want every detail of every order (on the exchanges I 
> monitor) stored in a database the moment TWS reports them.

Some issues off the top: TWS, and so near as we can find, NO 
source short of an extremely expensive data feed with SIAC, 
can provide in real time the 'unified' 'broad tape' of "very 
detail of every order" [we assume here you mean 'trade' rather 
than 'order' as orders are held confidential until they result 
in a trade, in some cases]

TWS updates its 'Market Data' information at a rate not more 
frequently than 300 mSec intervals, when there is detail to 
report. That is, there is a peak data rate load of about 3 
updates a second;  also updates may contain both a trade price 
and volume information, or just a trade price, or just volume 
of a trade, or just Bid/Ask spread data, or when no trades are 
occurring the bid, and the ask prices, so one needs to keep 
track of the last observed values for each, and push out stale 
information of various types.

A trap for the unwary -- one cannot simply start a 'market 
data' collector before the day starts, and stop it after the 
day ends, and 'catch' all trades;  The CBOT make their 
complete trade detail data available for free download as a 
matter of historical record, and we are told that people 
trying to 'reconcile' IB market data to it cannot get it to 
balance.

Also, it is not at all clear in light of 'off floor' trades 
reporting (often delayed up to 15 minutes), error correction, 
and such, we doubt that anyone can see _all_ trades, ever, 
anywhere, with sufficient detail to fully catch 'everything'. 
Perhaps the Exchanges can, perhaps the SEC can, but I do not 
think there is a realtime feed available outside of those 
places to get 'everything'.

> I don't have a ton of programming experience (mostly 
> scripting languages) so I was looking at the third-party 
> Python and Ruby APIs, but then I noticed that the 
> description of the shim seems to hint that it might be 
> capable of doing precisely what I'm interested in. 
> However, after reading most of the documentation (geared 
> mainly towards experts, it seems) I'm still not entirely 
> sure of its focus and capabilities.

The trading-shim is designed so that a non-programmer person 
might:
 	1. do a database insert with the desired Underlyings, 
specified with sufficient particularity as to remove all doubt 
as to what is being sought, send a 'signal' to the shim, and 
obtain back:
 		a.  an ongoing 'market data' (viz., 
tickstream) dataset, visible through the syslog,
 		b.  recurring 'history farm' data (viz., 
OHLC, Vol, and VWAP), which is 'close' to the present, and with
some effort;
 		c.  full historical data to the limits of the 
data farms maintained by IB.

This works for Stocks, Futures, and Indices just fine; support 
for Currencies, and Options is not yet fully complete.  It is 
possible to ask for 'too much' history data 'too fast', and to 
trigger IB's protective mechanisms to prevent denial of 
service attacks, or undue load on the history farms.

We have disabled order management in the released versions 
to date as not in our initial release path.  We did a proof of 
concept some time back on this and there is some remnant code 
lingering, but it is probably not in a working state at the 
moment.

> So, just to clarify, is it possible to instruct the shim to 
> monitor and record (in real time--this is important) all 
> orders of select securities in a MySQL database?

With the prior qualification as to 'all' and redefinition as 
'trades', yes.

The latency introduced by the shim once it is presented the 
data from the TWS, is under a second, and probably much less. 
We have no way to know what latencies exist between IB and the 
Exchanges, and assume it varies but is reasonable short -- on 
the order of under a second or two of a report of trade being 
filed, based on study of the way that market data is 
distributed in the US securities markets; lags or dropouts of 
up to 5 seconds seem to occur getting data from the Sydney 
exchange to the US.

> Would this require much mucking around in C++, or could I 
> simply use the command prompt?

No C++ mucking about other than compiling a binary suitable 
for your system, exists;  with a proper environment (ie, all 
headers present, and so forth), it is as simple as 
untarring it, and typing:
 	make shim

There are also scripts to set up a basic database of 
Underlyings, and Exchanges, and so forth in the ./bin/ 
directory of the tarball, again assuming you have a MySQL 
database up and running, and ACL issues solved.

Personally, I have a set of scripts which take the tarball, 
and patch it as to the database dataset and credential details 
to use (I run an environment separate from another here at the 
office, and we wanted freedom to each use our own database 
instance without 'stepping on' the other.)

> Also, am I to take this statement: "The command interface 
> also serves as an alternative to the existing tws gui 
> interfaces, to permit you to use downstream programs, 
> whether gui or not, to drive the tws through its api."  to 
> mean that I can use pipes to pass text commands to the shim?

yes -- indeed, I do just that

> I apologize for the simplistic questions, but given my 
> overall skill level it would probably take me several hours 
> (minimum) to install and thoroughly examine/test the shim.

smile -- We anticipate a release with much more robust history 
parseing momentarily; the market data portion is fine and I 
have used it for perhaps the last year.

As noted in the release announcement archive, I am running the 
shim against tws version 857; there is some datastream change 
later which we have not needed to solve yet.

The mailing lists are available at:
  	http://www.trading-shim.org/mailman/
with the customary archiving features at:
  	http://www.trading-shim.org/pipermail/ts-general/
for the general list.

-- Russ Herrold