[ts-gen] Re: where are the ticks?

[pippin at owlriver.net]

An anonymous poster asks about the lack of tick data: what starts it, where
it goes, and what might be wrong when it doesn't appear; and also about
the program modes in general.  I'll talk about program modes in the next
message, else this post grow too long.

My short answer to the first question, up front, is that I would indeed expect
that you would be seeing tick stream data in the system log, and that the load
option might be the solution -- "./shim --data load" -- or else there is a
problem, and you should check the SubRequest database table before looking
carefully at the system log.

This is also a good time for me to provide context by explaining the data flow
behind subscriptions.

There are five types of shim-tws mediated data streams:

    IB tws name         shim sub type and command name
    --------------      ------------------------------
    market data              tick
    market depth             book
    history query            past
    market scanner           scan
    contract data            data

Only the the first three are currenty implemented.  The market scanner may
well be a very long time in coming, depending on the quality of documentation
we can obtain for that part of the api, while contract data, in contrast, will
probably be added reasonably soon.  Bottom line, however, is that only the
first three work for now.

Probably the most important, and certainly the most familiar, is market data.
As is true for each of the data stream types above, whether subscription or
query, there are two ways to ask for the data, one relying primarily on the
the database system, and the other on the command channel.  I'll consider
them each in turn.

_______________________________________________________________________________

The load command
_______________________________________________________________________________

Under the proper circumstances, all the entries in the SubRequest table will
be read in by the shim, used to define tws requests, and lead thereby to
messages from the tws.  In particular, for market data requests, if the rows
in SubRequest are of SubType "tick", and given the foreign key constraints for
the TickConfig and Contract references, then the first time that table is read
in by the shim, each tick row read generates a market data request, such
requests are metered out to the upstream tws at the rate of about 1 every
20 milliseconds, and if the tws recognizes the symbol and has upstream data
farm connectivity, market data messages should begin reaching the shim, and
from there go on to appear in the system log.

Three points, one each about the shim load option/command, the SubRequest
table, and diagostics for the subscription process.

1.  Reading the SubRequest table can be triggered in either of two ways,
by the load option, or by the load command.  Given the command line
"./shim --data load", then the table will be read in automatically at
startup, and so you will see messages appear in the log within a second or
so of starting the shim.  Also, whenever the load command is entered to the
command channel by the downstream, (see the subscription regression test
function "reg_subs" in bin/includes for an example of command usage), then
the table will be read, or reread, as the case may be, and subscription
requests and cancels generated, as rows have been added or dropped from the
last time the table was read.

2.  SubRequest defines the set of active subscriptions in --data mode.
The SubRequest table must have subscriptions to cause subrequests; and
it is by deleting records from that table that you begin the process of
cancelling subscriptions, which you then follow up with a load command
to alert the shim.  Each time the shim reads SubRequest, it sends a cancel
for every subscription no longer there, and an add request for each one that
appears new.  The load command, then, is idempotent; execute it twice, without
changing SubRequest, and the second time no new requests will be sent to
the tws.

3.  The log shows your commands, including "load;", the requests sent to
the tws, and the messages returned, including any error messages to the effect
that there is no symbol known to the tws for the contract description it
received.  So, if given data in SubRequest, a successful startup of the
shim, and you want to diagnose problems, you check the log to see what
happened.

Does your load command appear?  What about the subscription requests?  And
does an error message, or price/quantity message, then appear?

_______________________________________________________________________________

The tick command
_______________________________________________________________________________

Subscriptions can also be triggered via explicit command channel
events, using the tick command.  Again, see the regression test program,
this time the function "wasteful", for an example.  The command syntax is
more involved; in addition to the command verb "tick", there is also the
modifier "add" or "del", a contract id, and a tick configuration table
index, e.g. to add a market data subscription for AIG, use: "tick add 5 1;"

Again, the command should be echoed in the log, followed by the request, and
then soon after by messages from the tws.

As for that mysterious tick config index at the trailing end of the tick
command, and for that matter as the num attribute of the SubRequest table,
the bar and bars attributes will be used in the future to control tick
data aggregation, though this is not yet implemented.

Please let me know if any of the above is unclear.  I'll tackle the second
question, about modes and options, in my next post.

Thanks,

Bill