[ts-gen] Re: ts-md] Re: where are the ticks?

[pippin at owlriver.net]

The following is in response to questions posed by an anonymous Debian
user (ADU) trying to get the trading-shim regression tests running in
order to see a working tick stream.

Problems discussed include:
    1. expired front month
    2. broken regression script shell path
    3. invalid database server host name in regression scripts
    4. intermittent connect failures, typically on first try

The problems he runs into are likely to be of interest to everyone, since
they are all either timely, reflecting recent fixes to the tarball, or else
basic startup problems that, in the absence of documentation beyond this
mailing list, could possibly afflict almost any initial user.  For that
reason, I've copied my response to this list.

_______________________________________________________________________________

    1. expired front month
_______________________________________________________________________________

First problem is that there used to be a default subscription to December YM,
and since the front month has changed, the contract needs to change also.  An
upgrade to our newest release takes care of this, although manual changes to
the load file WatchSets.sql in sql/mod would have worked just as well.

Automatic front-month rollover is a feature that is not yet specified, and
pretty far out in our planning.  For now, you'll need to manually add new
futures contracts in your trading database for any other futures you work
with, leaving the old ones alone, of course, if you've made any trades with
them; and we'll continue to manually modify the database load script for the
WatchSets table to select a newer expiry for the YM exemplar.

_______________________________________________________________________________

    2. broken regression script shell path
_______________________________________________________________________________

Second, ADU fixed the shell path in the regression scripts to find pdksh for
Debian.

> Took a little fiddling (had to install pdksh, and then had to edit the
> makefile to manually invoke it for regress; it didn't automatically figure
> out the proper command/shell to use despite the '#!/usr/bin/pdksh' line)

This will be necessary for any Debian user as, (1) we use an explicit path
to pdksh in place of ksh since ksh takes a brain dead approach to parsing the
empty function; (2) we develop on a Centos system, which uses Redhat-derived
packages; and, (3) Fedora and Debian systems disagree on where pdksh is
located.  The long term approach to this problem will probably be replacing
pdksh with bash, which is located in a predictable place on all Linux systems.

_______________________________________________________________________________

    3. invalid database server host name in regression scripts
_______________________________________________________________________________

Third, there were problems with the database connection function in the
regression scripts.  The bin/regress script autoloads bin/includes, and
then attempts to start up an instance of the mysql sql interpreter.  That
attempt fails in this case, with ADU reporting:

> ... I got this:
>
> ERROR 2005 (HY000): Unknown MySQL server host 'ftp' (1)
> set tws to Detail
> make: *** [test] Broken pipe
> $

The diagnostic text above might seem a bit confusing, since it combines an
error message from mysql, a trace message from the regression script, and
an error message from make.  As is often the case when things go wrong,
the first message is all we need; the regression script is unable to
connect to the MySQL server host, probably because there isn't one with
that name.  ADU goes right to the heart of the problem:

> Not sure if the server host error is significant; "ftp" isn't mentioned in
> the makefile or in regress.

He is correct; the actual reference is in bin/includes.  Starting at the top
level, the bin/regress script calls the run_dbms function indirectly through
the chain start_up -> listener -> run_dbms.  Looking at the text of run_dbms,
we see that the UserName, HostName, and DbmsName values probably need to be
changed:

    function run_dbms
    {
    UserName=shim
    HostName=ftp
    DbmsName=testing
        Dbms="mysql -u $UserName -h $HostName $DbmsName"

        $Dbms |&
        exec 4<&p
        exec 5>&p
    }

In the newest release, I've modified the lines:

    UserName=shim
    HostName=ftp
    DbmsName=testing

to now read:

    UserName=your_dbms_here
    HostName=your_dbms_host
    DbmsName=your_user_name

so that the error message from mysql would now read:

> ERROR 2005 (HY000): Unknown MySQL server host 'your_dbms_host' (1)

Although this should make the error message easier to interpret, you will still
need to edit the bin/includes script, in addition to the edits you are already
making to src/data.c for the same reason, in order to provide the literal
values for the database name, mysql server host name, and database user name.

_______________________________________________________________________________

    4. intermittent connect failures, typically on first try
_______________________________________________________________________________

In a following attempt, with the run_dbms function still unchanged, he
naturally saw the same error message, but also got a little bit furthur
along in the startup process:

> [...]
> ERROR 2005 (HY000): Unknown MySQL server host 'ftp' (1)
> set tws to Detail
 
> The trading shim has finished program initialization, including the
> construction of successful connections to the database and IB tws.
 
> YM history query
> make: *** [test] Broken pipe
> $

That message in the middle is good news; even though the regression script
is confused about how to connect to the database, the trading-shim has
succeeded!  ADU goes on to speculate:

> The disparity might be related to the erratic connection behavior of the
> shim I've been seeing--about 50% of the time I get a handshake failure for
> no particular reason, but if I immediately run the shim again it invariably
> connects without complaint.

I've seen this too.  Almost every morning, right after starting the tws,
the first trading-shim connect attempt to the tws fails, and on the next
attempt succeeds.  Then, for the rest of the day, I almost always succeed.

What is referred to as erratic connection behavior is most often similar in
appearance to the following (here, given a "make test"):

> Fri Dec 15 15:36:13 EST 2006
> make[1]: Leaving directory `/home/pippin/src/tws/src/tsd'
> bin/regress
> set tws to Detail
> terminate called after throwing an instance of
>     'TradingShimDaemon1_0::TradingException::version_handshake_failure'
> make: *** [test] Broken pipe

Clearly the handshake has failed, and the shim has quit, as it should; and,
since the next attempt almost always succeeds, the problem is clearly timing
related, perhaps an insufficiently generous timeout on the part of the
shim, or a tardy tws, depending on your point of view.

I've changed the timeout from 1 to 2 seconds, which I feel is going from
exhorbitant to ridiculous; but we will see.  In any case, since it is an
intermittent problem, I'm not sure whether you will see it from time to
time, and I'm not too worried about it, since restarts always seem to work,
but feel free to let us know if you see it, especially if it occurs often.

Thanks to our ADU for some good questions,

Bill