[ts-gen] Ugo's post; was a blank subject line

[R P Herrold]

On Wed, 26 Sep 2007, Ugo Kanain wrote:

> Though my requirements are rather simple, I found the shim 
> to be a rather comprehensive solution to interact with IB's 
> api, and probably the only viable solution to use a sql 
> server. Would like to extend my appreciation for Bill's and 
> Herrold's effort in putting out the shim for all to use. 
> Also on their patience when dealing with newbies, while we 
> try to learn to use the trading-shim.

Thank you, Ugo; we appreciate your post.  As Bill has noted 
previously here, we are pleased to have questions, as it 
permits us to become aware of our blind spots, and the places 
we need to externally document better.

It does not try our patience at all, and part of your design 
for building the shim as a community project anticipated doing 
careful email correspondence, in a timely fashion, to help 
attract users and co-developers.  We appreciate all comment, 
and on-list is best as it permits others to follow along from 
the archive, and by reading their email.

Last week, Bill re-enabled the full form of his TeX 
documentation build in the most recent tarballs; I have placed 
a working copy of my desk notes here:
 	http://www.herrold.com/commands.pdf
which is oriented more to the end user (rather than the 
developer).  The addition of the 'DRAFT' marker just now by me 
has interfered with the in-document pagination, which I need 
to work out.  I will refer to pull outs by section number 
which will work as live links in the table of contents.

> While trying to use the shim, I found that the db is a 
> complex and interrelated structure, which is not easily 
> maintained. The manual recommends that you modify the sql 
> files, and then reload the database. Since it requires 
> changes to be carried onto several tables at the same time. 
> So changes aren't a task easily carried out by hand.

We concur!  It is just too darn hard to do manually.  I am 
reminded of a quote which Tim Berners Lee made, published in a 
December 25, 2005 issue of e-week, summarizing this use case, 
but as to the HTML markup process:

> When I wrote the browser, people were using documents with
> wizzywig editors, so I really assumed that what people were 
> going to use for preparing content was wizzywig, or what you 
> see is what you get.

> So that's why I made it an editor and I was really surprised 
> when on platforms which didn't have wizzywig editors, that 
> people were prepared to go to the trouble of learning all 
> the angle brackets and doing the html.

> And a lot of people still do, so that was something that the 
> user interface engineer in me was horrified to find that 
> people would put up with such a terrible interface.

Bill and I are comfortable in a *nix environment, where one 
learns to builds small tools (sometimes for very short term 
uses) to extend what the Operating System provides, 
fabricating them either out of system commands, or by building 
intermediate tools.  This 'tool-builder' culture may seem 
foriegn to users of Apple's OS/X, or the Windows graphical 
products, but we are used to it.  ;)

> Another layer of complexity and in order to save space, is 
> the use of the numerical foreign keys in the records, so the 
> resulting structure is not easily read by people. 
> Notwithstanding that most of the database has already been 
> setup by Bill, there are times one would like to make 
> changes.

Please ask about a feature, as it may already be there, and 
just not obvious.  I am in process on an example about such an 
extension, to look up WatchSet abbreviations, but had not 
gotten too far.  See:
 	5.1.4 Adding additioanl tables
to decode the index values (WatchSet.tag), which are similar 
to a 'foreign key'.  I'll try to flesh this out shortly.

See also: my FIXME notes at:
 	5.1.1 Why so many tables

> I was thinking of an easier solution that allows the modification of
> the db, through a view of the record. In such a view, all the related
> fields would be shown with more descriptive information (maybe Bill
> can put a link that shows one of the table views). Then all the other
> tables would be affected.

Visualization is hard as there is the top down, and the bottom 
up approach -- see:
 	6.1.1 Looking up an underlying Symbol from the CID -
 	step by step
and compare it to:
 	6.1.3 Looking up a Contract.uid with join

I would love to find an Open Source schemvisualization tool, 
which can handle multiple foreign keys, but it is not clear 
that one exists.  Pointers appreciated.

> Have started playing with a web interface. However, can't 
> decide what would be the best way to do things; top to 
> bottom or bottom to top. Don't know if all tables allow both 
> approaches, but imagine most should.

They permit working from either end.  The tables are extremely 
consistent, and relentlessly normalized.  See for an example 
of emitting HTML ready code:
 	6.1.6 Tabular database listings in other contexts

> To illustrate the idea, suppose we are adding a stock to the 
> database. Though the stock table has to be changed, it also 
> needs to modify the underlying and symbol tables:

> 1) Top to bottom: Modifying the Symbol table would cascade 
> the changes down to the Underlying and Stock tables

> 2) Bottom to Top: Modifying the Stock table would cascade 
> the changes upwards unto the Underlying and Symbol tables.

Yes - either approach is doable. In our planning Bill and I 
have a design for using the shim to query wildcard matches 
from IB upstream, as has been discussed by IB employees 
verbally in some of the older IB 'webinars' when moderated by 
Jay Bailey; and to have the shim emit well formed 'INSERT' 
statements, which could then be post processed, and finally 
applied to the database after testing. See:
 	6.1.4 Adding a new underlying Symbol to the Contract
 	table (after the select of the FutDetail table)
for some work in process cautions [the paragraph presently 
beginning: NOTE: This next process, ...]

  We are some time away from this functionality, however, and I 
would say that you are on the right track.

> Haven't played with the shim to know which approach would be 
> a better solution. So would appreciate to know, first if 
> there is any interest on this project (would require an 
> apache server with php), and secondly your comments as to 
> which approach to follow.

We regularly run each, so the webserver and PHP is no 
impediment here.  We expressly anticipated that there would be 
interest in a web client interface, but one still has to solve 
the underlying command line actions before writing such, and 
so we have been slow to work in that area.  One of my examples 
is in command line PHP, and I use the tool all the time. 
While adding a webbish front end seems slower to us, it is 
perfectly doable, and indeed I have done RO query mockups from 
time to time as we have developed.

Let me ask for a raincheck, as to a more detailed response and 
think about this a bit to help you build a good ROADMAP.  As 
Bill mentioned, so long as the code is GPLv3, or other OSI 
approved FOSS license, and carries the proper COPYING and 
license notices, we are delighted to read what you have and 
will comment.

http://www.trading-shim.org/pipermail/ts-general/2007-September/000092.html

-- Russ Herrold