[ts-gen] A look behind the curtain; was: create & modify orders

[R P Herrold]

On Thu, 27 Mar 2008, Paul C wrote:

> Having experimented with the shim for a few days I'm getting
> interested in the Order commands and associated sql tables.

Welcome.  Pleased to hear from you. Also welcome to the new 
lurkers on the list.  Every time I update the Freshmeat entry, 
we pick up a couple of new observers.  PLEASE. speak up if you 
have questions.

The shim has been in a usable state for almost two years now, 
and sometimes I feel like Bill and I dominate the list.  It 
_is_ tricky to get started, but almost trivial to use once it 
is set up.

> I get the impression that the commands for creating orders 
> are evolving (?).

Evolution. Intelligent Design. They seem so similar (a bit of 
a joke on the US schizophrenia on that British Darwin chap's 
work, vs. Creationism :)

Yes -- we are aware of the clumsy parts of the shim, and this 
is one of them under active rework presently.  The appearance 
of the 'bind' operator has been planned for at least six 
months by us, to permit a more natural (or at least more 
easily remembered) form of expression of orders.  We are 
drilling in changed in the back end in this regard.

The other, more important missing function, which has required 
MUCH re-work 'under the hood', has been clearing up some 
structures to support journalled accounting of order status 
(and with that completed executions on a given position 
series), able to persist and reach proper state through an 
initial connection, disconnections and re-connections with the 
TWS.

List user Nils Gebhardt did some initial work in areas we had 
not yet done usage testing on, and exposed some lack of 
clarity and missing function on 'modifying' orders with the 
TWS [technically, it appears a 'modification' is **not** how 
IB is doing things under the hood, but it appears that way to 
the user of the Java sample client, or the TWS GUI].  This was 
in a post Sept 15, 2007:
   http://www.trading-shim.org/pipermail/ts-general/2007-September/000088.html

This led us to the VERY long development cycle, really 
identified on that date midway through the series begun Aug 2 
2007 (see the NEWS file: 
http://www.trading-shim.org/capitals/?NEWS )

Bill remarked on the stanza series started Nov 14 2007: "This 
release begins a new series for no better reason than that a 
month has gone by since the last.  That delay is due to 
juggling ... ."

And some of that juggling was planning to get 
the loose ends of the shim API identified, properly fixed in a 
durable fashion, and so to have a product we were proud enough 
to move from a developmental 'beta' status, to a formally 
mature 'Production/Stable' Status in the Freshmeat scheme of 
options:
   http://freshmeat.net/browse/6/

Bill and I formally 'plan' in a meeting each Monday afternoon, 
and review 'progress' against goals each Friday afternoon; it 
took us a few months to settle into this approach, but it 
seems to work well.  We are physically in a common office 
suite, but have found that the intense level of concentration 
each of us needs to bring to bear on coding, and testing are 
such that we usually communicate asynchronously through email 
although we sit 15 feet from one another, rather than speaking 
and potentially interrupting the other.

The shim has grown in approach over time, based on our 
understanding of markets, and then of IB, and through a 
reading and contemplation of the Java sample client, and 
reading and re-reading the documentation of the TWS API.  Oh 
-- and much experimentation.

Bill's regression tests ('test-driven development' per the 
website) are in the tarball; when I think I see a 
misbehavior, I will first try to show it within the four 
corner's of his tools; failing that, then I write a 'minimal 
reproducer' test case, and confirm anticipated behavior with 
the Java sample client.  I have written literally hundreds of 
testing scripts over the last couple years;  In looking just 
now, my main testing script is about presently 1300 lines 
long, and this is after cutting out over a thousand lines 
earlier this year.  I have 66 RCS check-ins so far this year -- 
about one a day.

The recent addition of the 'shebang' capabilities in the shim 
was no accident; the release of the sample Ruby script for 
history harvesting was intentional as well; there is a 
possible additional command and control adjunct data path 
which Bill and I blocked out as well, which we will discuss 
more in tomorrow's meeting.

My tests take long enough that I get to about 20 tests a day. 
I imagine he runs his more focused tests at least 100 times a 
day.  We use 'distcc' to speed the compilation process, and it 
is a Godsend.

We have intra-day 'checkpoints', formal internals (usually at 
end of day), proposed externals, and formal externals in our 
'in-office' routine, all produced by scripts; similarly, 
scripts mechanize and make perfect testing to a design.

When an external release cut is proposed, I fire up a series 
of external release 'smoke test' scripts on 32 and 64 bit 
CentOS 5; Intel and AMD processors; Debian Testing and 
Quantian; and a ppc Apple OS/X 10.4 unit.  If any fail, we see 
it and fix it in minutes.


The shim code has met parts of our research design goals (see 
the first, and last paragraphs on the homepage);  We want to 
meet all of them.  We re-wrote the third paragraph earlier 
this week.

The shim has users in academic quarters, in proprietary 
trading groups, and in hedge funds.  Its users are in North 
America, Europe, Asia and Australia (no South America, Africa, 
or Antarctica, so far as we know).

Earlier this week, I was looking at the FTP and web logs, and 
a couple hundred discrete IP's are pulling the tarballs, and 
reading parts of the website; I make sure that the 'top ten' 
pages are as accurate and as current as I can make them.

We know that the manual.pdf, and my commands.pdf are woefully 
out of date, and cleaning them to match the 'Production 
Stable' release is on our docket.  The mailing list archives 
are the natural place that Bill and I can first present 
examples of usage of the shim, and we hope the archives are 
useful to members of the list.

Google fully indexes the site every couple of weeks, and it is 
pretty clear that they are doing so because we are keeping the 
site fresh, and because people are using Google to search the 
site (the search page relies on this, for a site specific 
search.)

> In the most recent 'bin/includes' are commands of the
> form:
>
> 		create item  12 3 LMT 2 50.0  0.0;
>
> where previously the "12"  ^^ was not present.  My 
> understanding is that this is a limit order to buy 2 lots of 
> AIG at price 50.0; and that it's AIG because it references 
> row "3" in the "Template" table, which in turn (pid=2) 
> references row 2 in the "Position" table: this ultimately 
> deriving its entries from risk.sql.  But I'm not sure about 
> the "12". I see that AIG has a cid of 12 in the table; but 
> is this the same 12 as in the above command? (i.e a 
> reference to AIG in the shims' database).

This is close, but not quite right; Bill will see this email 
and your question as well tomorrow, and I will not give a 
partial response here and now;  The order code has been 
through heavy revision, and it is worth getting the answering 
analysis as accurately stated as possible.

> The shim looks to be a powerful way to interact with the 
> TWS, and I got it up and running without much problem 
> (debian).

yup -- Debian Testing is tested continuously by me, and has 
been fully functional from very early on.  The test only takes 
a few minutes with each candidate.

herrold at debian:~$ time ./shim-debian.sh > /dev/null 2>&1
[sudo] password for herrold:

real    7m51.020s
user    6m52.466s
sys     0m36.174s
herrold at debian:~$

> It would be interesting to know what's currently under 
> development in the code (i.e. I'm curious about your 
> roadmap).

I've tried to give a flavor of process, and the nearest work 
area (Orders).  Further out, confirming completion of all 
Requests (to the TWS) and Messages (from the TWS), save a few 
related to FA's, and such.  IB also 'versions' server and 
client Messages and Requests, respectively.  Bill has a 
sensible and hopefully 'future-proof' approach on this.  ... 
but entropy is such a relentless opponent.  ;)

We know that maintenance and additions to the Symbology we 
carry and doing the CID (ContractID) look-ups is cumbersome 
[your question touches on this], and we have a design in 
process and partially drilled in to address this. The top item 
in NEWS: 'Add full set of exchanges ... ' gives a portent of 
this.

Scanning ahead in my email, it looks as though there may be a 
typo in the cold load scripts, and I'll test this tomorrow 
morning; they were re-written a bit, and I know where to look.

And of course use examples, and documentation.  The full TeX 
is in the tarball for earlier variations of how the shim has 
worked, but it is sad to lose the time spent documenting well 
a failed approach [I know that the re-work of the 'select' and 
'cancel' verbs, whole proper as a matter of design, caused 
about 80 pages of my command reference to go stale in about a 
day ;) ] We had on-line help substantially complete, but then 
that same change obsoleted much, and broke some of my 
documentation building scripts.  Ah well ...

Please be assured that the completion of the doco is already 
part of the 'Production Stable' status checklist to leave 
'Beta'.

with my best regards,

-- Russ herrold