[ts-gen] (repost) Questions about shim command language

[Bill Pippin]

An off-list message posed a number of questions about the shim, the answers
to which may be of general interest, and so I'm copying my reply to this list.

The message first noted, about the database record ids appearing as values
in the command language:

> A big stumbling block ... were the ... raw record id[s].

Yes, numeric record id literals *are* ugly.  They were a quick hack early
on, but I'll be glad to reduce or eliminate their use.  Variable names
via the bind command are the first step, and Contract values will improve
the command language furthur.

Minor uses, such as for history filter records, will probably remain for
a while to come.  The only good alternative to such uses that I see is to
accept sql select statements as part of a shim command, and pass those
on to mysql.  E.g., 

bind hfcr = pf : 		# the common-case history filter config record
select PastFilter.uid
  from PastFilter, HistoryTag, BarSize
 where PastFilter.duration = 1800
   and PastFilter.tid      = HistoryTag.uid
   and HistoryTag.rth_only = true
   and HistoryTag.format   = 'ymdt'
   and HistoryTag.what     = 'TRADES'
   and HistoryTag.bar      = BarSize.uid
   and    BarSize.secs     = 1;

Given the current default load of the database, the query returns the
record uid of 1, and so the above command would be equivalent to:

bind hfcr = pf:1 		# the common-case history filter config record

Note that neither form of the bind command is yet provided, that the
second, concise form with the manifest constant record id should be in
place reasonably soon, and that the first form, providing access to
sql select statements, may or may not be implemented.

As for determining what *is* the current syntax of the command language
while it undergoes rapid change:

> Is it fair to say that the scripts in exs/ and bin/includes are the best
> guide to the latest command syntax?

Yes, absolutely: exs/* and bin/includes are far and away your best current
guide to the latest form of the command language.  Once it stabilizes, the
help command will also be useful, and once that is updated, I hope to fill
in the command syntax section of the manual.  For now, however, please
see the examples you've noted.

> ... [I]s there a simple way of extracting [the syntax] from the source
> code ... ?

Not exactly simple, but the type vectors for the nonterminal symbols in
rule.c specify the exact semantics of each nonterminal.  E.g., in
CmdK::schema(), you'll see that there's a type vector that gives the
arguments to the create command:

...
    R18[] = { ws  ,Item,ws  ,ConI,ws  ,Mods,     B0E ,       Eoc}, // create
...

Once the initial verb "create" of a command selects this rhs rule in the
parser, and taking whitespace as a separator between arguments as given,
then the verb must be followed by the literal text "item"; a contract record
id; the nonterminal symbol Mods for the modifiable parameters of an order,
about which more later; and some end-of-command indication, currently either
newline or semicolon, although likely to be restricted to the semicolon
in the future to allow multi-line commands.

If you are wondering about the symbol B0E, the epsilon symbols provide
defaults and context-based table lookups without corresponding to any
actual argument text; and as for the nonterminal Mods, as indicated in
the functors schema, FunK::schema(), it expands to require an order
template key, order type, quantity, limit price, and auxiliary price:

template<>
tag::FunK::Map
tag::FunK::schema(Wrap_e w, Tags_c t, Funs_c c) const
{
 ...

 Arg_0 ...
       R02[] = { TmpK, ws  , OrtK, ws  , Q___, ws  , LmtP, ws  , AuxP };

 Arg_tab ...
         T02(vec::Build<Argument, sizeof(R02)>(p, R02).block());

   G_0   tab[] = {
         ...
         new (p) tag::Mods(TagName(w, "mod"), T02)
    };

    return vec::Build<Functor, sizeof(tab)>(up(p), tab).index();
}

So, yes, there is a precise specification for the command language present
in the code, as you would expect, and if you are willing to trace the
meaning of the nonterminals via rule.c, type.c, and generation.i, you
can follow it, given some comfort with the use of templates in the
type hierarchy, and some time looking at the parser driver rules in read.c.
   
>                        ... the same Template record would be used for
> buying AIG and YM (for example)? whereas currently each would have
> separate entries in the LineItem (now Template) table.

Yes, currently you need distinct line item records for each contract you
wish to work with, and in fact you should have distinct line items for
each order over time if you wanted the current approach to order accounting
to give [semi-] useful data.  Once orders are specified by a contract
value - order template variable pair, you will be able to reuse templates
from one order to the next; an order will consist of a template with the
contract "filled in", via association by the text of an order command.

Either the downstream or the shim will disambiguate individual orders
via a template variable (label) name.  E.g. if you bind the same
template record to the variables e1.080401.1544 and e2.080401.1545 during
a session, where the variable names are literal strings in shim commands,
then since order accounting would store the variable names in the
journal, you could distinguish two uses of the same position entry
template via the variable names.  Or, if you used shim-controlled
labels, via dollar-sign variables, e.g., $e1 and $e2, the shim would
expand those in some way, perhaps ultimately using the tws-assigned
perm id once it became available.

> ...  Is prolog in some sense informing your choice of semantics for
> the shim, or the underlying C++ code?

My previous experience with logic programming certainly contributes to
my current programming style, e.g.: my preference for const objects,
monadic state where state is appropriate, conditional logic expressed
as rules, and overall use of applicative style.  See my dissertation
for more information if you want to trace influences:
 
http://www.trading-shim.com/about/pippin/
http://www.trading-shim.com/pdfs/dissert.pdf

As part of my work on optimization of the clpr implementation, I ported
the code from C to C++, and this led to my use of the singletree pattern,
and informs the derivation structure of the object type hierarchy defined
by the type.h, code.h, rule.h, and meta.h header files.

> ... is [there] more info in the tex that doesn't make it to the pdf/html?

At times I've commented out large sections of tex, via percent signs in
the toplevel file manual.tex.  Right now you are seeing essentially all
of the manual that is of interest, but if you want to check to see if that
is still true in the future, look to see if \chapter commands in
doc/manual.tex are commented-out.

> ... are you moving towards ruby instead of korn-shell for scripts, or
> does ksh have some significant advantage (e.g. coprocesses?) for
> interacting with the shim?

You're correct that I used ksh instead of bash due to its support for
coprocessing, in particular to be able to drive both an instance of
the mysql interpreter and the shim as well.  Later on I decided that
shell scripts were too clumsy, that some more powerful scripting
language should be used, and that I preferred ruby.  Other scripting
languages can of course be used, and you should use whatever you like.
If I ever provide a sample-client downstream script, it will probably
be written in ruby.

The ksh text in bin/includes is a vestigial remnant of an early attempt
to restrict language dependencies to C++ and the shell.  I may rewrite
the regression tests to use ruby in the future, with popen used for
access to mysql and the shim.  Although downstream users don't need to
understand or write ruby code, they do need to have it installed; it's
currently used during the installation process, as you've noted.

Thanks,

Bill