[ts-gen] Version control

[Bill Pippin]

In the process of trying out the newest version of the IB tws,
build 886.4 from Aug 8, the regression tests indicated that the
contract info message type had changed in some way, presumably
due to the addition of new fields.  (I won't know for sure until
I see a new version of the api sample client sources, since the
release notes don't seem to mention the change.)

Previously, I've updated the message tables in the shim to reflect
the new version, pushed a new release, and left it at that.  This
time I've done something more, changing the way version mismatch
errors are handled in the parsing logic.

I'm taking this opportunity to explain:

    1.  implications of the api message syntax for message matching;
    2.  changes to the shim's parser to relax message version matching;
    3.  the version alphabet soup, and how that affects downstream users;
    4.  how version negotiation between api clients and the tws works; and
    5.  future plans for improved support for multiple client versions.

1. The api message syntax, and message matching

Api messages reaching the shim consist of null terminated tokens, with
those made up of character data.  Messages begin with a message type
index / message version number pair, followed by a variable length
payload the length of which is determined, first, by the message type
index, and for history data, the number of history bars in the repeating
group.  Note that the message version number is determined by the api
server version; it's fixed by the instance of the tws that you have
installed.

There is no end-of-message delimiter, and so clients must begin at an
actual message boundary and count tokens in order to pick off a message.
This means that it's difficult for clients to recover from just about
any kind of syntax error; discarding tokens is simple enough, but how
many, and what [message] next?


2.  The shim's parser, and the relaxation in message version matching

Recall that, given server version and the message type, the message
version is predictable.  This suggests a self-aligning algorithm
with which to parse api messages; discard a token on each parse
error until a valid message (index / version) pair is reached.
Although false positives can and do occur, most of these can be
prevented by strict typechecking for each of the following 
attributes in the message, and this has worked well in practice.
The only problems we've seen are version mismatches as new server
versions come out, and we can fix these via a quick change to the
message tables in tabs.c, which has been fine until now.

Recently IB has been making frequent changes to the message
version numbers for the api, and although this is not a problem
internally, it is a headache for outside users who are not used
to the table structures, or comfortable with editing them.

So, starting with today's release, messages are accepted where
the msg version, given an expected value V, is in {V +/- 1}.
The shim will go ahead and parse, process, and output the
message as for a valid message --- including a log format
display that uses the expected version code --- but will go
on to print a transient error message:

        Problem: 427 api msg version mismatch -- upgrade soon

Which begs the question, upgrade what, exactly?


3.  The version alphabet soup

The shim has it's own version, and uses a (mostly, outside of
order journalling) private database, which also has a version.
The IB tws has, in addition to its build version, the api version,
client version, server version, request versions, and message
versions.

Herewith a version list:

version type         example                    where found
-------------        --------------             ---------------------
shim program               0.83                 log banner
shim datestamp       080812			tarball name, directory
shim database              1.57                 sql/req/Version.sql

IB tws               Build 886.4                gui: Help->About
IB tws api                 9.51                 file: API_VersionNum.txt
api client ver               23			.shimrc cVersion def; log
api server ver               40                 shim log, near banner
request versions     single digits*		net packets; source code; log
message versions     single digits*		net packets; source code; log

* Except for orders,
  where changes have
  been frequent.
---------------------------------------------------------------------

And, now, why you might care:

Don't worry about the shim program version, since I don't change it
often enough to care.  Instead, realize that the shim insists on
using an up-to-date database, and will refuse to continue without such.

There is a script, sql/create.sql, that can be used to (re)create and
populate the database, and you'll typically need to use it if you
upgrade the shim.  Since the tarball includes the database load files
and scripts in addition to the program sources, and the shim checks
the database version at startup, you can try out new tarballs freely;
just compile, start the binary, and if it fails as follows:

        Problem: 513 the shim and database are out of sync

you need to recreate the database as well.

Making sense of the various IB tws-related version numbers depends
in part on understanding the version handshake, in the next section,
but I'll note the following here:

The tws build number, such as 886.4, changes very often, and 
you are *not* free to continue using old versions past some
variable cutoff date, typically less than half a year and
sometimes only a few months.

In practice, you should download each of the unixmacos.jar files as
they become available, whether you intend to use them or not,
because sooner or later you'll have no choice, and the newest might
have problems.

You can ignore the api version number, e.g., 9.51, since you have no
control over it.  Changes here seem to stay reasonably in sync with
the build number, so that the build number is near but less than
the api version number times 100.

The other IB tws version numbers bring me to the connection handshake.


4.  How version negotiation between api clients and the tws works

The server version, currently 40, is determined entirely by the
IB tws; build 886.4 supports server version 40, and that's what
you get.

The client version would typically be some number at most as high
as the server version provided by the latest tws build, but the
tws doesn't seem to care if it's too high.  In any case, it can only
talk up to the highest protocol level it knows of, say 40.

In our case, we've got the shim clamped to client version 23,
although we intend to provide multi-version support in the future.

At this point, api version negotiation probably sounds like it works
the same as other protocols you are familiar with.  Two sides exchange
values, and the lowest common denominator results.  Here, it's close
to, but not quite that, simple.

Every request the client sends has a version, typically in the low
single digits except for orders, where changes have been frequent.
The request versions must be consistent with the client version.

Every message the server sends has a version, also typically in the
low single digits.  NOTE: message version *numbers* reflect what the
server has to offer, not what the client is ready to accept.  NOTE
ALSO: the message *format*, that is the number of tokens in the
message, does reflect the client version, and possibly, for the
specific message, the related request version that triggered that
message.

So, messages would appear to mix protocol versions; they have a
version that reflects the server version, and a payload that
follows the client version.  You can think of this as the server
advertising its newer features, whether the client wants to hear
about them or not.

At this point I can return to the question posed above; given
a 427 transient suggesting you upgrade soon, what exactly 
should you upgrade?

Given the change to the shim's parser, the shim will work with
either the shim or the IB tws one message version number behind 
for any given message type.  So, you should upgrade whichever
is lagging behind, almost certainly the last one upgraded.
If the message appears after you've downloaded a new shim, and
you're trying it out, then probably you should consider downloading
a newer IB tws as well, and vice versa.


5.  Future plans for improved support for multiple client versions.

The shim currently uses a table driven parser, with distinct rules
for each message type, and similar rules to map requests to their
wire format.  Although additional work would be needed to select
these rules from tables according to the client version, we've
already used table lookups to configure the request and message
rules for specific api features, so the design is proven.

Still, given the number of cases involved, there is a fair amount
of work involved; it's not clear to what degree the shim would
support older versions prior to client version 23; and there might
be some version "holes", since the sample client sources from IB
skip version numbers in some cases.  For all these reasons and more,
multi-version support is one of the later features in our planning
process; but we do see a real need for it, and do hope to implement
it.

Given that multi-version support is provided, you'll be able to
control the client version via the cVersion configuration variable,
in the .shimrc file.  (Currently it would be ignored.)  This would
give control over the format of messages sent on to downstream
applications, to simplify the modification of those apps to take
advantage of new IB tws api features.

At the same time, for a given client version, the shim would be sending
message payloads according to that version, so downstream apps could
work to a stable api as desired.

Thanks,

Bill