[ts-gen] create & modify orders

[Bill Pippin]

Paul C comments:

> Having experimented with the shim for a few days I'm getting
> interested in the Order commands and associated sql tables.

> The shim looks to be a powerful way to interact with the TWS, and I got
> it up and running without much problem (debian).  It would be
> interesting to know what's currently under development in the code
> (i.e. I'm curious about your roadmap). 

Thanks for the kudos.  About feature enhancements in general and the
evolution of orders in particular, I'd like to first give four general
principles for the shim command interface; it should, ideally:

    1.  provide access to the full power of the IB tws api;
    2.  be as concise as practical, but no more than that; and
    3.  be simple and intuitive, and easy to use.

Although the 1st and 3rd goals would seem to conflict, for the most
part, we can use the database to map brief, simple text commands to
the full binary api format.  In some cases, where we are not [yet, in
any case] interested, e.g., the financial advisor functionality, we've
chosen not to support that part of the IB tws api.  There are also
places where the shim is simply not yet finished, e.g., complete
flexibility for contract and execution reports filters; where we're
not sure if the feature is useful or not, e.g. the market scanner;
and where we have not yet caught up with newer versions of the api,
e.g., real time bars.  FA will probably wait for interested and
supportive users.  Better filters, new event types, and perhaps
even experimentation with the scanner are part of our planned
build-out, though I wish to emphasize that this is part of our
internal planning, subject to change, and will occur after current
work on orders.

The 2nd and 3rd goals are more subtle, particularly in the way
they interact.  As the constrained mimimization of brevity suggests,
encoding parameters via database records can be taken too far.
We've recently worked out two related, clear-cut guidelines that elide
apparent conflicts between concision and obscurity, and goals 2
and 3, as follows:

    A.  fundamental data should be obtained via user-intuitive keys
    B.  voluminous cross-products must be dropped from the database

The first suggests that, where practical, downstream users be shielded
from the use of raw record ids as command parameters, and we are adding
various forms of the bind command to make the definition of contract
and orders-related parameters more intuitive.

The second suggests that the current use of concrete contract records
be replaced with abstract products, and those be specialized with
derivative and other attributes as needed.  E.g., for products with
many possible routes and currencies, rather than requiring that the
user create explicit database contracts specifying the route and
currency, or detail information such as expiry, strike, and right,
the bind command should instead be extended with an expression
language to allow contracts to be specified on-the-fly, by value.

The key insight here is that we can't hope to define all the
possible concrete contracts, but we can include a tremendous number
of products, that is abstract contracts; the downstream can specialize
these symbolically; and journal information can record the concrete
contracts as key-tuples, the cross-product quantity of which it
would be infeasible to define up front via records in the current
Contract and LineItem tables.

The 4th requirement, easy to use, and the 'no large cross-products'
rule lead to the same conclusion here:  asking the downstream to
create zillions of contract and lineitem records is infeasible.

As suggested above, we'll instead have explicit records for abstract
contracts, in effect IB security products; contract detail attributes,
e.g., expiry and strike values; and order exemplars, or templates,
to specify the enormous number of relatively obscure order parameters;
all as startup input to the shim.  Lineitems would be created by
the shim as needed as part of order journalling, and contracts recorded
as low-degree value tuples, e.g., the record id tuples corresponding
to STK:IBM at SMART:USD , or FUT:YM at ECBOT:USD:200806 .

________ orders, future ________________________________________________

This brings me to your question about the orders interface:

> I get the impression that the commands for creating orders are
> evolving ...

Indeed.  As a colleague was wont to say, with understated emphasis,
"That is a ... *true* statement."

> ... In the most recent 'bin/includes' are commands of the form:

>           create item  12 3 LMT 2 50.0  0.0;

>   ... previously the "12"    ^^  was not present.  My understanding is
> that this is a limit order to buy 2 lots of AIG at price 50.0; and
> that it's AIG because it references row "3" in the "Template" table,
> which in turn (pid=2) references row 2 in the "Position" table: this
> ultimately deriving its entries from risk.sql.  But I'm not sure about
> the "12".

You noted that the contract id of 12, for AIG, was previously not present,
and that it is currently redundant, both true.  It is needed currently,
however, to keep the parser happy; and in the future for semantics as
well, since the current line item index will be replaced with an order
Template record specifier in its place.

So, the input format for the create statement now has an extra
parameter; we're giving up some brevity, with the goal of avoiding
the cross-product tarpit.

[Keep in mind that the changes in database semantics are not yet
drilled down; the command above works, but currently still requires
the explicit underlying lineitems of the past.  Also keep in mind
that the following is, in accord with the text of the GPL, *not*
a promise to implement the features described; I'm explaining our
current internal plans, which are, as always, subject to change.]

The create command currently has a syntax that is textually consistent
with a limited form of the following planned semantics, where bracketed
terms must be replaced with actual parameters:

    create {sel} {con-expr} {ord-expr} {ord-type} {q} {lmt-price} {aux-price}

where selector is one of item, buy, or sell; con-expr is a contract
expression; ord-expr is an orders exemplar, or template specifier;
ord-type is one of the order types, {MKT, LMT, ...}; q is the quantity;
and limit price and auxiliary price are interpreted according to the
particular order type, as determined by the IB tws.

The selector "item" is currently required, and would become the neutral
case of three possible choices, with the other two providing a consistency
check with the direction of the order specifier; the buy and sell selectors
would not actually specify the direction, but would have to match it, else
the order would be invalid.

The contract expression would be a contract value expression, or else a
name defined earlier via the bind command; currently it is a raw, explicit
contract record id.  As you point out, the index for the contract
record is currently redundant, since it is also specified via the
lineitem record id.  This will change, as noted above, to avoid the
cross-product tarpit; once the record parameter for the lineitem id
is interpreted instead as an index into the Template table, the contract
specifier will be essential.  At this point it will be convenient to reuse
Template records for multiple distinct orders, as needed; templates will
give the structure, but not the value, of an order.

The order expression would typically be a symbolic variable specified via
the bind command, the name of which would be stored in the journal to
allow the downstream to distinguish individual orders from one another.
There would be one or two levels of indirection possible, as the
downstream or the shim took responsibility for making order template
record variable names unique.

________ orders, present _______________________________________________

The above is really just a sketch of the plans for orders.  More precisely,
I need to provide, or finish providing:

    * improved load files for symbology;
    * the full bind command, for a number of record specifier expression types;
    * command language rules that allow use of those expressions;
    * scheduled tasks to track order state; and
    * order processing to post order state to the journal

Where is the shim now?  For now, create and submit work as shown in various
test scripts, while modify is broken.  Order journalling *only* tracks the
creation of orders; note that completion here depends on working code for
all of the other stages, and that in particular, effective cross-session
order cancellation depends on accurate order tracking.  Please feel free
to keep your eye on the web site for news and updated releases, and the
mailing list for additional explanations.

I'd also like to take this opportunity to thank the list readers who ask
questions; without those dialogue would become monologue.  Our web site,
mailing list, and release process are all means to foster dialogue, and
I'm delighted to see users respond.

Thanks,

Bill