[ts-gen] Getting historical data outside regular hours

[Bill Pippin]

I've just pushed a new release, and it includes a db version
increase.

I'd like to address a number of issues with respect to history
queries in general and that release in particular that Sam's
questions have brought to the fore:

    0. the best currently available link I know of for IB tws api docs;
    1. bar sizes, and the number thereof;
    2. parameter errors for non-trade and bid-ask what-to-show values;
    3. cumulative changes to the format of the history command;
    4. the current structure of the HistoryTag table; and
    5. using IB contract ids in place of contract unique ids
________________________________________________________________________

0.  Due to use of flash, IB's website can be a bit tricky to navigate,
    and I've been hobbled by obsolete links to the api docs.  Sam was
    kind enough to point out a newer link, and for the benefit of others
    I repeat it below:

http://www.interactivebrokers.com/php/apiUsersGuide/apiguide.htm

The api docs have been way behind the sample client sources, and this
seems to bring the online docs much closer to the present.  This is
a great improvement in the available documentation for the IB tws api. 
________________________________________________________________________

1.  Those docs give an expanded list of bar-size intervals, and I reproduce
    the list below, with a line after which the newer values are not yet
    supported by the shim:

        Interval   Code

         1 sec     1
         5 secs    2
        15 secs    3
        30 secs    4
         1 min     5
         2 mins    6
         5 mins    7
        15 mins    8
        30 mins    9
         1 hour   10
         1 day    11
        ____________

         1 week   12
         1 month  13
         3 months 14
         1 year   15
         3 mins   16

Note that the bar-size codes are monotonically increasing up through 
but not past 15.  The IB tws api does not accept the newer bar sizes
until relatively new client versions that the shim does not yet support.

Although we intend to support a wider range of client versions, we are
not yet working on this issue, as other features will come first.  As
always, comments to the mailing list about feature value are read by
me with interest.
________________________________________________________________________

2.  While testing history queries with a range of parameter values,
    I discovered that, (a), the shim displayed transient error messages
    for history bar volume when the N/A value of -1 occurred, as it
    does for all but the TRADE what-to-show case, that is for MIDPOINT,
    BID, ASK, and BID_ASK queries.  This is now fixed.

Also, as part of the same tests, I found that the BID_ASK label was
mispelled as BID/ASK; this is also now fixed.
________________________________________________________________________

3.  History query commands now include the bar size as an explicit
    parameter.  This change is distinct from, though similar in concept
    to, a recent change about which I posted:

> History query commands ... include the query duration, a time period
> with catenated unit.  This parameter used to be provided as part of
> the PastFilter record, and in fact the database version change is due
> to the deletion of the related attributes from that table.

To help users keep track of the changes, I list some sample history
queries below, for the version of early this month and before, of 080918,
and as of today, 080930.

select past 142     8    now;                       wait 11;
select past 142     6    Ymd_T(20080722  14:00:00); wait 10;

select past 142     8 2d now;                       wait 11; # 0918
select past 142     6 2d Ymd_T(20080722  14:00:00); wait 10; # 0918

select past 142 m15 1 2d now;                       wait 11; # 0930
select past 142 m2  1 2d Ymd_T(20080722  14:00:00); wait 10; # 0930

In the oldest form, both the bar size and duration are tucked away in
database records; in the intermediate, about which I've already posted,
the duration is lifted to the top level; and now the bar size has been
lifted out, as well.

Please note that the bar size is a label, and so begins with a character,
e.g., m15 for 15 minute bars, while durations are dimensioned numbers,
with the unit following the numeric value; the first is a finite domain,
while the latter is variable over some range, in general 30 seconds to
whatever duration would give 2000 records.  Hopefully this distinction
will help you keep track of which is which.

These changes are in part to reduce the impact of future support for
additional bar sizes; the history tag numbering will then remain
unchanged.  This is at the cost of changing that table now; since
we appear to be picking up new users, my inclination is to get such
changes over with.

This change will affect the interpretation of accumulated history data;
old history tag indices will no longer be valid.  If you've used
relatively few distinct tag values, you can probably do a bulk update,
e.g., 

    update HistoryBar
       set tid = 1
     where tid = 8;
________________________________________________________________________

4.  This brings me to the current values in the HistoryTag table, the
records of which are referred to in the earlier form of the commands above
as unique id 8 or 6, and most recently as id 1.

With the removal of the bar size from HistoryTag, the number of cases
drops from 220 to 20.  The remaining tuples are:

    +-----+----------+--------+----------+
    | uid | rth_only | format | what     |
    +-----+----------+--------+----------+
    |   1 |        0 | ymdt   | TRADES   |
    |   2 |        0 | ymdt   | MIDPOINT |
    |   3 |        0 | ymdt   | BID      |
    |   4 |        0 | ymdt   | ASK      |
    |   5 |        0 | ymdt   | BID/ASK  |
    |   6 |        0 | epoch  | TRADES   |
    |   7 |        0 | epoch  | MIDPOINT |
    |   8 |        0 | epoch  | BID      |
    |   9 |        0 | epoch  | ASK      |
    |  10 |        0 | epoch  | BID/ASK  |
    |  11 |        1 | ymdt   | TRADES   |
    |  12 |        1 | ymdt   | MIDPOINT |
    |  13 |        1 | ymdt   | BID      |
    |  14 |        1 | ymdt   | ASK      |
    |  15 |        1 | ymdt   | BID/ASK  |
    |  16 |        1 | epoch  | TRADES   |
    |  17 |        1 | epoch  | MIDPOINT |
    |  18 |        1 | epoch  | BID      |
    |  19 |        1 | epoch  | ASK      |
    |  20 |        1 | epoch  | BID/ASK  |
    +-----+----------+--------+----------+
    20 rows in set (0.00 sec)

For most use, users should be able to choose from among cases
1, 6, 11, and 16 for queries.
________________________________________________________________________

5.  Using IB contract ids to refer to symbols --- this makes sense, is
    planned as part of enhancements to the bind operator to support
    contract expressions, and will in fact be one of the earlier
    changes as part of this work.  When I talk about work on the bind
    operator, I'm referring in part to this feature.
________________________________________________________________________

In addition to the above work, some ongoing changes to the scanner
suggested by an earlier bug report from Nils have been completed.

I'd like to thank both Sam and Nils for useful comments that helped
suggest the changes I describe in this post.

Thanks,

Bill