[ts-gen] New stock and futures contract value expressions in cmd language

[Bill Pippin]

I'd like to announce the new contract value form syntax for stocks
and futures.  This syntax is supported by the newest release
of the shim, the shim-090313.tgz tarball.

This release extends to other data mode commands, e.g., market data,
history, and market depth, the value form that was already available
for contract info queries.  

Schematically, contract value expressions in the command language
are now (for stocks and futures) or will be (for the other security
types below) strings of the following form:

     IND:Route:Symbol:Currency
     STK:Route:Symbol:Currency
    CASH:Route:Symbol:Currency
     FUT:Route:Symbol:Currency:Expiry
     OPT:Route:Symbol:Currency:Expiry:Right:Strike
     FOP:Route:Symbol:Currency:Expiry:Right:Strike

So, for market data requests, the code fragments below of value
form contract data expressions would subscribe to market data
for STK:SMART:AMAT:USD and FUT:ECBOT:YM:USD:20090320 over a 
three second interval:

    select tick STK:SMART:AMAT:USD 1;        wait  3;
    cancel tick STK:SMART:AMAT:USD;

    select tick FUT:ECBOT:YM:USD:20090320 1; wait  3;
    cancel tick FUT:ECBOT:YM:USD:20090320;

The examples above, and other parts of this post, are derived from
the exs/tick script.  You may also wish to look at the exs/info
script, which gives rules for how whitespace may be used to format
value form expressions.

In brief, order is significant, symbol values should be in all caps,
and whitespace may be freely used between colons, but not after the
last one.  E.g., the currency and expiry above must immediately
follow the last colon. 

The older index forms continue to be supported:
 
    pid:ProductUid at Route
    ibc:ContractId at Route

Example index form subscription requests include the following:

select tick ibc:  266093 at SMART 1;
select tick ibc:49652404 at ECBOT 1;

select tick pid:    2846 at SMART 1;
select tick pid:   48194 at ECBOT 1;

The ibc index form is preferred over the pid index form, since
the latter exposes internal record keys from the database, and
those may change when you reload new symbol data or upgrade the
database.

The pid index form has been retained since it provides an
alternative unique single key to the ib conid for products,
and the latter is outside our control.  That being said,
the ib conid --- prefix ibc --- will almost certainly remain
available, so you should feel free to use it.  The pid index
form is just engineering redundancy for safety, and hopefully
it will never become necessary.

Recall that the older cid:nnnnn form and its supporting
ContractRow table were both already eliminated, prior to this.
This has had the nice effect of reducing database load time
to less than a second, for a total startup time for the
shim of less than two seconds.  The slow.sql script also
went away, so that repopulating the database is now much
faster as well.

I intend to continue updating the exs/tick script to reflect the
various forms of contract expression that occur in the command
language, so see that script for details whenever you have questions.

Support for additional security types depends not on extensions to
the command parser --- those are already in place --- but rather
on collection of additional symbols data, extensions to the fill.rb
script to manage, preen, and add to that data, and additional
features in the shim to give control over input of the data.

Orders:
-------

You probably noted that value form contract expressions were
described only in the context of data mode commands, and that
no mention was made of using such expressions in order requests.

Value form expressions are not guaranteed to refer to a unique IB
conid, so that they should not be used for orders at this time.

If multiple IB conids match a particular value form, the IB tws
might either reject the contract spec as it appeared in the
request, which would be fine; or else use some form of undocumented
default logic to choose the preferred match --- not so good.

Our planned solution to this issue is three-fold:

    * Russ is collecting updated symbol data from the net, and this
      is meant to give us a better idea of IB's symbol universe;

    * I'm continuing to run large-scale contract data queries, to
      explore the correspondence between the net data and what is
      provided via the api, and I'll continue to extend the fill.rb
      script, to automate these checks.  To the extent that the
      database matches the results of contract info queries, it's
      complete for that particular IB tws account, and ambiguous
      forms, if any, can be flagged up front.

    * In the future you should be able to check the data for
      yourself, either by running the fill.rb script, for batch
      operation, or dynamically, by running contract info queries
      for a symbol during a session before using that symbol
      in an order. 

All these approaches share in common the idea of validating the shim's
database against the upstream IB symbols database on their servers.

The problem would go away if we could redistribute symbols data
that had been collected via contract info queries to the IB tws api,
but their license prohibits that, and some effort on users' part to
check consistency seems unavoidable at this point.

So, for now you should continue to use the index form, with a
specific IB conid, for orders.

Expect more releases over the next month, either of new symbols
data tarballs, new versions of the fill.rb script in shim tarballs,
or both.

Thanks,

Bill