[ts-gen] creating new orders
Bill Pippin
pippin at owlriver.net
Tue Sep 11 14:09:35 EDT 2007
Nils asks:
> I want to create some test orders in shim with some different contracts
> than included in the test suite. As I understand, I have to add some
> entries in LineItem, Position and Protocol, probably just adapt
> something in the file risk.sql. I can see some orders in bin/includes,
> but what is the meaning of the parameters (apart from the obvious
> ones)?
Since I've not yet documented the wire [synonym order] command in the manual,
others may have this question as well, and I'm glad you've asked.
The only way for users to track this kind of issue down at this time is to
either ask, or go to the sources. Fortunately, most of the commands are so
brief that just looking at examples is often sufficient, but not here; the
order command is perhaps the most complicated of all the commands.
So, first, the example of interest is from the regression test scripts, in the
function bracket, and I've extracted the following code fragments:
BuyOrder='order(3,LMT,Create,2,60.0,0.0,0);' # AIG
SellHigh='order(4,LMT,Create,2,80.0,0.0,0);'
Transmit='order(3,LMT,Submit,2,60.0,0.0,0);'
FireRoot='order(3,MKT,Submit,2,00.0,0.0,0);'
Reversal='order(4,MKT,Submit,2,00.0,0.0,0);'
cmd_shim "$BuyOrder" ; sleep 2 # protected entry:
cmd_shim "$SellHigh" ; sleep 2 # limit exit order
cmd_shim "$Transmit" ; sleep 1 # send transaction
cmd_shim "$FireRoot" ; sleep 1 # fire order entry
cmd_shim "$Reversal" ; sleep 1 # reverse position
There is a limit entry, profit taking exit, and then the order is transmitted,
by duplicating the parameters. A moment later the order is converted to a
market order, which forces the entry; and finally the position is exited,
again with a market order. This is, of course, impractical for real life
use, the example here is just a regression test against the paper account.
So, first key point, and Nils hits the nail right on the head:
> ... I can see some orders in bin/includes,
> but what is the meaning of the parameters ...
Nils is asking about the meaining of the values in parentheses. E.g., for
the limit entry defined by the ksh variable BuyOrder, what are the numbers?
BuyOrder='order(3,LMT,Create,2,60.0,0.0,0);' # AIG
I'm going to dig into the sources here, and what I'll show you is admittedly
very cryptic. Later on you'll be shielded from all this by documentation,
and again, the order command is a special case; the underlying API order
request is composed of *80* individual send operations as far back as the 9.00
version. In our defense, I'd like to say that we're doing pretty good by
squeezing this all down to 7 parameters. If you want to skip ahead to where
I summarize the order command as a septuple, feel free.
________ source code explanation begin: ________________________________________
Looking in the sources, in rule.c, within the function tag::CmdK::schema(),
which is the grammar schema for the downstream command language, near the
bottom of the definition we find the following call to a constructor:
494: new (p) cmd::Wire(w, STV(26), c, T16, "wire", xact),// accumulate orders
Moving back up, we find that the table entry T16 ultimately depends on the
row entry R16:
439: R16[] = { Lpar,ItmI,sep ,OrdK,sep ,WrtK,sep , // wire
440: Q___,sep ,P___,sep ,P1__,sep ,
441: Nat , Rpar, Semi};
(I've inserted the line numbers in the left margin; they aren't in the original
sources.)
The grammar symbols of interest are ItmI, the line item key; OrdK, the
order type; WrtK, the request tag; Q___, the quantity; P___, the price;
a mysterious price-like quantity, P1__, about which more later, and a
totally opaque natural number value, with no other clue to its purpose.
The other grammar symbol codes are for punctuation, not abstract syntax.
You can look up grammar symbol keys such as ItmI in generation.h, e.g.
$ grep ItmI generation.h
typedef const Tabs<Index , 27, obj::fun::LineItem> ItmI;
Objects in the 'fun' namespace are for table rows from the database. All
the grammer keys that end with K are string-valued keys, and those ending
with I are natural numbers, here a database table index.
Another way to get some idea what the parameters to the Wire command are
is to look in src/command.h, at the data members that are defined once a
Wire command event has been constructed (again, line numbers are added):
141: Item_c item;
142: OrdT_c type;
143: ReqT_c request;
144: Nat_c quantity;
145: Rat_c lmt_price, // for limit, stop-limit, and relative orders
146: aux_price; // the stop price for limit and stop-limit
147: // orders, the offset amount for relative
148: // orders, and otherwise zero.
149: Nat_c time_out; // time in seconds until a cancel is to be
150: // attempted in the event of a non-fill
151: // Warning: dummy for now; not yet implemented
Note first the explanation for the first and second price parameters, which
correspond directly to the IB tws limit price and aux price properties in
their documentation. Here we see also what that unexplained trailing number
is, a time out value that is not yet being used. You still have to fill it
in, or else there will be a parse error, but you can feel safe for now in
using zero, since that will be interpreted as no timeout once it is
implemented, and so there will be no visible change until you make it so.
________ source code explanation end ___________________________________________
So, to recap, the order command is a septuple:
wire(Lineitem, OrderType, RequestTag, Quantity, Price, AuxPrice, T),
where T should always be zero for now. Let's focus on the other parameters,
which naturally partition into "stuff from the database", and "stuff that
I only just figured out at the last minute, or might want to change later".
The LineItem key is the only entry in that first category. Everything after
that is dynamic, something you can try to modify by resubmitting the order.
________ dynamic values: _______________________________________________________
The possible order types are defined by the table OrdType, itself loaded
from sql/req/OrdType.sql, and identically by the values for the OrdI grammar
symbol in syms.c. Looking there you'll see values such as MKT, LMT, STP,
etc., all strings predefined by IB.
The request tags are one of: Create, Submit, Modify, Attach, and Cancel.
You could find these values also in syms.c, although you'd have to dig.
Looking for WrtK, the request tag keys, you'd see that those were defined
by the request tag codes, symbol WrtI, and the table definition for that
symbol includes the strings listed above. In brief, for now, just limit
yourself to Create, Modify, Submit, and Cancel, and ignore the Attach tag.
Quantity is almost but not quite self-explanatory; what are the units? The
line item includes a foreign key reference to a contract, and the values for
the contract determine the units that quantity refers to. Price also depends
on the contract, via the currency.
The Price is limit or market price, and the AuxPrice is the stop price for
limit orders.
________ database parameters: __________________________________________________
The LineItem key is just the tip of an iceberg. Going to the database table
definitions, and in particular the file xact.sql, the create table statement
for the LineItem table has 10 foreign keys:
122: side int unsigned not null references OrderAct(uid),
123: pid int unsigned not null references `Position`(uid),
124: gid int unsigned references OcaGroup(uid),
125: pro int unsigned not null references Protocol(uid),
126: fid int unsigned not null
127: default 2 references OrderFlags(uid),
128: parent int unsigned references LineItem(uid),
130: gat datetime, -- ib YYYYMMDD HH:MM:SS
131: gud datetime, -- ("%Y%m%d %t" format)
133: size int unsigned, -- iceberg display size
135: fad int unsigned references FinAdvisor(uid),
136: iid int unsigned references Institution(uid),
137: box int unsigned references BoxOptions(uid),
138: vid int unsigned references Volatility(uid),
So, what values should go in a LineItem record in the database? Mostly,
it boils down to choosing the right foreign key values.
Four can be ignored for now. The references to the FinAdvisor, Institution,
BoxOptions, and Volatility tables may be null, and are in effect placeholders;
the shim just sends blanks, zeroes, or default values to the IB tws in its
place order request, so you'll just be wasting your time for now if you try
to define these values. (Please speak up on the mailing list if you would
like some work done here. We need people to *use* this stuff, because we
aren't at this point, so it doesn't get tested.)
The important attributes are the "not null" foreign keys. They do all the
work:
122: side int unsigned not null references OrderAct(uid),
123: pid int unsigned not null references `Position`(uid),
125: pro int unsigned not null references Protocol(uid),
126: fid int unsigned not null
127: default 2 references OrderFlags(uid),
The Position key is arguably the most important, since the Position record
determines the contract. Those records may be thought of as quadruples:
Position(Contract, Open, Closed, Comment)
where the open and closed values are timestamps for accounting purposes, and
the comment is yours to use as you wish. The heavy lifting is done via the
contract key, which pulls in all the symbol and other information that
determines *what* you are buying/selling. You can have as many line items
share a particular position as you wish, although for accounting reasons
you probably want to create new position records once you've gone flat for
a particular contract.
Now, for the rest of the required keys:
________ order flags and codes: ________________________________________________
In req/OrderAct.sql we see that the order action is simple, one of BUY or SELL.
Protocol and OrderFlags, though, group lots and lots of codes together.
Looking at load.sql, we see that Protocol is a cross-product on Trigger,
OcaType, and TifType, that is stop trigger, one-cancels-all action, and
time-in-force, with the actual codes defined via table loads from the
related files in the directory sql/req. Note that since Protocol is
an exhaustive cross-product, that is it enumerates all the possibilities,
you will never need to add new entries, instead you will select one that's
already there.
OrderFlags is again an exhaustive cross-product, this time on boolean-valued
codes, and to see that note that there are eight boolean attributes, and
that the table already has 256 rows. The boolean codes are all from the IB tws
order properties, the codes for block orders, sweep to fill, hidden orders,
firm quote only, ECN trade only, regular trading hours for options and
futures, ignore rth for stocks, and all-or-none.
In practice you'll find at most a few rows in Protocol and OrderFlags that
you use for most of your order line items, and it's up to you to study the
IB tws properties for these attributes to decide which values you want. If
this seems elaborate, keep in mind that it is very compact; just choose a
Prococol and OrderFlags row one each, and you've defined 11 different IB tws
api codes and flags.
The only other significant attributes in LineItem are the the good-until-date
and good-after-time values, gud and gat, about which you may see the IB tws
api properties documentation; and the parent and OcaGroup foreign keys, which
I've purposely left for last.
________ parent and OcaGroup: __________________________________________________
Note that this stuff is important; bracket, oca group, and combo orders
are the *only* way to get any kind of transactional logic from the
IB tws api. The shim does not yet support combo orders, which are
conjunctive, and oca groups are non-deterministically transactional --- the
only way you can control sequencing of order actions is via bracket orders.
So: consider not using the OcaGroup, and relying mostly on parent. If you
create bracket orders by grouping lineitems via a parent-child relationship,
and for most cases you should, then the IB tws will want to assign its own
ocagroup values, and you'll just be buying yourself trouble if you try to
control the oca group.
There is a possible foreign key chain of Position<-OcaGroup<-LineItem,
implying the tree structure of Position->OcaGroup->LineItem. This is
the *general* case. To make it possible to avoid the OcaGroup table for
the common case of bracket orders, however, the database design has been
partly denormalized; the LineItem oca group attribute may and most often
should be null. In this case you should think of the hierarchy as going
Position->LineItem; as mentioned earlier, a Position may have any number
of line items, all of which share the same contract.
If you do choose to have user-defined oca groups, presumably you know what
you are doing, and can figure them out from the IB tws documentation. Just
keep in mind that you will have to keep the Position key consistent between
the LineItem and OcaGroup, with both referring to the same Position. This
is a consequence of the denormalization I mentioned earlier, and the only
way to avoid it would be to have required [dummy] oca groups with bracket
orders, a cure worse than the disease.
So, bottom line, you'll probably want to create bracket orders, and if so,
you'll have two kinds of line items, parents, with the parent attribute
null, and children, otherwise, and for both, the oca group attribute null.
There is an example in risk.sql that has been laid out to make the
distinction between the parent and its children obvious.
Please feel free to ask more if parts of this are unclear, and if it seems
elaborate, please keep in mind that it is an interface to the IB tws orders
api, which has to support orders for all kinds of security types. After all,
as IB advertises, your account is a *universal* account.
And sorry, but we haven't tried out all those security types yet. For those
who are interested in some of them, please feel free to inquire on this list.
Thanks,
Bill
More information about the ts-general
mailing list