[ts-gen] database version stability

[Bill Pippin]

An offlist poster has asked, in the context of extracting history
data from the database for analysis by unrelated programs:

> [Is] the database structure ... supposed to be changed again?

Note that by "unrelated" I mean other than downstream scripts,
where those control the shim via commands.

There are actually four points I'd like to address here: (1), the
general question of gaining access to history data; (2), the specific
one of stability for the HistoryBar table; (3), the implied one of
stability for the database as a whole; and, (4), the opportunity
to describe some planned changes.

For the first, I would like to suggest that users try the simplest
approach first, which I claim is feeding sql queries to mysql via
the stdin; bin/get_id.sql is a good example of that approach.  You
shouldn't need anything more complicated.

For the second, I like the way Russ tackled this issue:

> We are closing in on that part of the database (HistoryBar)
> and its foriegn key dependencies not changing; I think you
> will find that it has had only two other forms in the last two
> and a half years.  The addition of the IB ConID value when we
> did an effort to populate out the Products within the last six
> months occasioned the last change, and before that the
> Product, Exchange pairing.

To recap, we haven't changed HistoryBar very often, we hope not
to change it that much in the future, and nevertheless as we
approach a 1.0 release, there may be a need for some changes,
hopefully not major.  By way of example, consider the need for
attribute renames to be explained later in (4) of this post.

For the third, I'd like to say, (a), that the database is
used primarily by the shim itself, and (b), that we are and
have been working to insulate external programs from database
changes, by eliminating the need for such programs to write
to the database, and, in the future, by planned changes to the
command input language to accept contract expressions, so that
external programs can ignore the various shim-specific symbol,
product, and contract record ids.

Finally, with respect to planned changes, I'd like to take
this opportunity to warn users of superficial but widespread
changes to be made in the near future with respect to
attribute names.  Currently the shim has literal sql select
statements as constants in data.c, whereas insert prefixes
are generated automatically from type constants, since those
correspond to attribute names.  Although this approach has
not yet been applied to the select statements for no better
reason that of inertia, it will be used in the near future, and
many table attribute names are likely to change at the same
time.

This will not, in and of itself, change the database structure;
I'm talking about just the names of attributes, not their types.
The revisions will have the beneficial effect that, once made,
maintenance to the shim will be simplified, and table attribute
names will be even more consistent from one table to the next.
Such rationalization of attribute names is in any case a
necessary prelude to the production of database documentation.

Thanks,

Bill