[ts-gen] Failed history query [Was: historical data format]

[Bill Pippin]

Mike,

Your 1st questions are about log format; 2nd, about history query
command syntax; and 3rd, about the lack of useful answers to a
history query, what would appear to be a bug.

Since your message raises a variety of points, I'll break my
reply into multiple posts.  I'll take them in reverse order,
focusing here on diagnosing problems with history data queries.

So, log format is outside the scope of this message --- I'll talk
about that in a follow-on --- but if you're eager, it's also
discussed in various previous posts to this list, including the
following messages I wrote that are still in the archives:

http://www.trading-shim.org/pipermail/ts-general/2009-July/000474.html
http://www.trading-shim.org/pipermail/ts-general/2009-September/000576.html

Returning to the subject of history query problems, your first point
of attack for problems with history queries, or for that matter any
process problem with cmd->req->msg events, should be to look at the
log, starting from the top for the current session, that is, for a
permanent log, where you've used the join option to append each
session, searching from the bottom up towards the first occurrence of
the string "star", e.g.:

> 14901|69724|  1.514403|4| 0| 0|# |star|****************|
> 14901|69724|  0.000008|4| 1| 0|# |head|0.95|080820|data|
> 14901|69724|  0.000000|4| 0| 0|# |star|****************|
> 14901|69724|  0.000005|4| 2| 0|# |hand|23|1|45|20100323 \
>  18:22:03 EST|Connect with: cv 23, id 1, sv 45|
> 14901|69724|  0.000000|3| 9| 1|1|

After the initial banner, you'll see the results of an account
data query that the shim has used to discover the account code.
After this, your downstream commands, the resulting requests,
and the asynchronous messages that occur in reply to those
requests, appear, one after another.

For problems with history queries, you'd want to look at the
startup messages about the state of the history farms, and see
what messages followed the history query request; formatting
problems, and rejection due to invalid options, are a common
problem here.

So, dealing specifically with your failed forex query:

> However, what I saw print out in the logfile was

> 23873|59291| 469189941|2|20| 3|23873|59291| 469189941|2|20| 3||
> 23873|59352| 530357165|1| 0| 0||
> 23873|59352| 530357228|1| 0| 0||
> 23873|59352| 530357254|1| 0| 0||
> 23873|59352| 530357278|1| 0| 0||
> 23873|59352| 530357326|1| 0| 0||
> 23873|59352| 530357350|1| 0| 0|| <------------ these lines
> repeat for most of the log file
...

> The last few lines repeated indefinitely and the logfile grew
> to 223MB -- I did not see any valid Forex data.  I had to kill
> the shim to prevent the logfile from growing larger. What did I
> do wrong?

Not knowing how you sent that command to the shim, and in
particular what commands preceded and followed, and without the
first part of the log, the interesting part, before it starts
repeating, it's impossible to say.

No matter how much crud ends up in the log file, it's what appears
before that that is most interesting.

It's possible that the ruby script is sending text to the shim,
and the shim is happily sending log text back.  It's not hard to
make such a loop continue indefinitely, since both ruby and the
shim are designed to keep on keeping on.  So, as is true for
most debugging situations, you need to reduce the problem scope.

One approach would be to insert debug statements in the ruby script,
and modify that script to send ping messages to the shim, in order
to mark the log.

Instead, in what follows below, I've reduce problem scope by using
a shim script, rather than a Turing-complete scripting language
like ruby:

> #!./shim -f

> select past CASH:IDEALPRO:EUR:USD m1 1 1d Ymd_T(20100312 16:00:00);
> wait 10;
> exit;

When I ran the above script --- less the "> " prefix --- as a shell
command, and with arguments " --data file save diff join" , the
following event payload resulted:

1| 0| 0|#!./shim -f|
1| 0| 0| |
1| 0| 0|select past CASH:IDEALPRO:EUR:USD m1 1 1d Ymd_T ...
1| 0| 0|wait 10;|
1| 0| 0|exit;|
1|11| 0|select past CASH:IDEALPRO:EUR:USD m1 1 1d Ymd_T ...
2|20| 3|13539|40395|  0.000020|2|20| 3||
1| 0| 0|exit;|
1| 2| 0|wait 10;|
3| 4| 2|   -1|2106|HMDS data farm connection is OK:ushmds2a|
3| 4| 2|    1| 165|Historical Market Data Service query ...
3| 4| 2|    1| 162|Historical Market Data Service error ...
1| 7| 0|exit;|

Both the enqueue and dequeue events for the history query list the
same text, which I originally cut and pasted from your message: 

|select past CASH:IDEALPRO:EUR:USD m1 1 1d Ymd_T(20100312 16:00:00);|

The history farm status messages, when split over two lines, appear
as follows:

> 165|Historical Market Data Service query message:
> HMDS server connection was successful.|

> 162|Historical Market Data Service error message:
> No historical market data for EUR/CASH at IDEALPRO Last 60|

So, for me, when I run your query, there's no history data at the
farm for your query.

Thanks,

Bill