[ts-gen] Examples, and mysql, and ssl

[Bill Pippin]

Justin,

With respect to the manual:

> So far I seem to be able to connect to the api, and access the demo
> account, I'm now at the "Using the Shim" section in the manual.  

You should be aware that the manual has become increasingly out-of-date
over the last year.  The shim is under active development, and we are
adding features on a day to day basis at this point, which features
include new sentences in the command grammar.  The example scripts in
the exs directory plus the mailing list are your best guides at this
point.

I'm glad to see you posting to the mailing list; in my experience,
once most people get started, they find the shim pretty easy to use,
and the people reading the list should be able to help you over the
rough spots.  The big obstacle seems to be the initial installation,
setup, and configuration, which you've already achieved.

> I'm still contemplating on whether I should invest more time
> familiarizing myself with mysql ...

You don't need to have any special skills with mysql.  The configuration
of the sql isolation level is an issue with nearly all sql vendors; they
set it too low, to get artificially high benchmark performance, and sane
dbas reset it to serialize reads. The mysql conf file change is a one
time problem, documented --- correctly --- in the manual, and you've
already taken care of it.

The key skill you need with mysql is starting up the interpreter,
which they call mysql.  Once started, it's equivalent to the isql
(Interpreted-SQL) translator that most vendors ship.  Type or pipe in
a query, and you get a result, either to the screen, or the stdout.
All you need to use the mysql interpreter is an understanding of the
SQL language, in particular the select statement, and there are plenty
of examples in the sql directory.

As for other database understanding, ideally, you might want to be able
to read sql create table statements, and understand the implications of
primary and unique key declarations, and be able to view and understand
the foreign key dependency graph diagrams we ship with the shim.  This
level of understanding is not essential, but it's helpful, especially
if you want to subscribe to unusual securities, or track the performance
of your trading strategy using the order journal.

With respect to ssl:

> ... ssl disabled ... I get:

>           The trading shim has connected to the database server
>           and loaded 51649 products; connected to the IB tws as
>           client id  1; and queried for the account code, which
>           seems to be DU15082.  Program initialization has been
>           completed.

Good.  Congratulations, you've got the shim started with successful
connections to the database and tws.  You should be able to use
the shim, say with example scripts from the directory exs, which is
parallel to, that is in the same unpack directory, e.g., shim-090112,
as src, lib, and sql.

Now, about the timing problems you are seeing:

> I tried connecting to the tws with ssl enabled ...

> 	Problem: 421 the client id  1 is ignored by the IB tws
> 	Problem: 421 the client id  2 is ignored by the IB tws
> 	Problem: 421 the client id  3 is ignored by the IB tws
>       ... 
> 	Problem: 524 could not obtain IB account info at startup

The 421 messages might in other circumstances indicate that multiple
shims were connecting to the IB tws.  Here, however, for the first
shim started, the exception error messages indicate timeouts, at
least four of probably two seconds each.  These timeouts can only
occur at startup, interspersed with or instead of the banner; once
it has displayed, you're home free.

The shim --- and, for that matter, the tws api --- is specified in
terms of a tcp socket connection.  Once it has finished the upstream
connection handshake, the shim uses nonblocking reads, happily
accepting input from either the downstream client scripts or the IB
tws whenever such input arrives.

That being said, we don't want our downstream scripts to hang waiting
on the shim if something goes wrong at startup, so we haven't written
the shim to poll forever during the handshake; instead, it waits at
most some limited time, and then times out.

This time limit is controlled by the shim configuration variable
PollTime, which may be used in a .shimrc configuration file.  The
sample .shimrc file that is shipped with the sources has up 'til
now had an initial value of two seconds, which I've now increased
to three seconds, in part because of your report.  You can set it to
whatever suits your purposes; just change the right-hand side value
for the entry in the .shimrc configuration file, found in the
directory where you unpacked the sources.

So, feel free to increase the value of the PollTime variable in
the .shimrc file, although you may want to read furthur, and consider
carefully whether there is some other issue you want to deal with as
well.

First off, have you marked the IP address the shim is connecting
from as a trusted address in the IB tws?  If you are manually clicking
on a dialog box to accept the connection, you should expect the shim
to timeout.  I realize this is an unlikely problem, but it's worth
checking before we go furthur, if only because it is so easy to fix.

Second, you shouldn't put too much weight on what you see with
the demo account; it's very flaky.  Paper accounts work pretty well,
but the demo accounts can show any number of odd behaviors.  You do
have an IB account, right?  Please be aware that without one, you
can't collect history, or get meaningful price data, or trust the
executions you see.

> Should I be able to connect with ssl enabled?

Third, please realize that the delays you are seeing are due not to
the shim, but rather the IB tws; the choice of ssl determines how it
connects to its upstream.  The tws api, and the shim's connection via
that api, uses a regular, unencrypted socket.

So, the IB tws is connecting to its upstream over a network layer
that adds multi-second time delays.  If those are only at startup,
e.g., for certificate lookups, then the delays may well be harmless,
and you could just wait to start the shim until the IB tws has
finished talking to its servers.  Beyond this, your choices are
simple: either turn off ssl; or increase the time until the shim
would give up waiting.  For the latter, as explained above, use the
shim's timeout variable PollTime to change the maximum wait for the
connection handshake and account query; you can increase it as desired.
Otherwise your problem is outside the control of the shim; if the IB
tws wants to take its time talking to the upstream, there's not much
else the shim can do but wait.

Thanks,

Bill