[ts-gen] list posting

[Bill Pippin]

An off-list message asks, with respect to messages and the log file ShimText:

> ... doing something wrong [?] or something isn't working.
> ... get the data the shim gets into the database. However  
> the only thing [happening], is ... the [ShimText] file growing  
> huge. Tried the list and load commands, but none gets data stored in  
> the db (historybar?). Is there a setting or a command that stores  
> into the database?

In brief, no.  Although the shim stores history data to the database,
 [and also logs pending orders to the journal] it does not store tick
data there.  Instead, this can be performed by a downstream control
program, e.g. some script in ksh, ruby, python, perl, php, etc.  The
key idea, say for tick events, is to translate a group of events into
an insert statement, and feed that to the mysql interpreter.  The shim
is simply an interface program to the IB tws api, and is not in itself
a complete trading infrastucture.

>               ...  Also any way to prevent the shimtext from growing  
> too large? Can it be disabled or assigned a size limit (maybe by  
> rotating the files)?

All tick data, other messages, and indeed all other events are logged to
the designated output stream, currently either a local log file or the
system logger.

If you want the shim to send to the system logger in place of the output
log file, one approach would be to use the logd [synonym syslog] option.
>From the help topic "help opts":

> help opts;
> ____________________________________________________________________________

>   # Options may be listed following the mode.

>                # real modes; the output options write all events to the:
>       file     # file with name also the ShimText name-value pair value
>       cout     # standard output -- warning, sprays all over the screen
>       logd     # system logger, to provide routing via /etc/syslog.conf
>                # output options may be combined and file is the default

>                ... more stuff deleted

So the shim *will* send output to at least one sink, the log file,
system logger, or local log file, with the latter being the default;
and if you want log rotation and the like, the logd option is the way
to go.

Another alternative is to use the "cout" option to send to the standard out,
and pipe that into a script that filters and discards as needed; or you
could, depending on the file system you're using, create a named pipe as the
local logfile, and the shim would happily write to it, and reading from 
the pipe would delete the data from disk.

We routinely see log files grow into the 10s of megs per day, but rarely if
ever over a hundred meg, and use scripts to discard/route/filter/store on a
per day basis.  Given the size of modern hard disks, double digit megs are
not much that bad for most of us.  If it is for you, use a pipe, so that
material once processed by the downstream reader no longer takes space on
disk.

The real problem is deciding what to *do* with it.  If you store all tick
data into the database, you'll have much bigger problems with disk space
than the ShimText file, which is, after all, meant to be at most one day's
data.  You'll probably want to summarize the tick data, say over 1 minute
intervals, perhaps using the HistoryBar(Open, High, Low, Close, Vol, Wap, Gaps)
relation.  HistoryBar format would compress the 10 basic tick types in a
tick set to one row, and one minute bars, as many as 180 (3 per second) tick
message sets, again to one row, so summarizing to open-high-low-close-vol-etc.
is a big win for compression.  For seven hours of data over 100 tick
subscriptions, one minute bars give 42k records, nontrivial but manageable.

Even then the stuff adds up given enough time.  There's no magic answer.
Either it goes somewhere, or it gets deleted.  The shim log message format
is not designed to be the final storage format; it's formatted for convenient
filtering, and easy reading, e.g., for scrolling to the screen, and not for
minimum space, so you'll want to translate the log output somehow, and
discard each day's log output file.

Thanks for the question,

Bill