[ts-gen] redirecting shim output to python script

[Bill Pippin]

I'd like to thank Paul for the cogent report and his excellent
diagnostic testing.  I'll be continuing to look into this, and
might have more to say later, but for now, first, what I believe
to be a work around for Mike's benefit; second, some comments on
what Paul found, and third, some plans for changes to the shim's
initialization setup to help with this issue.

Thanks as well, Mike, for your thorough diagnostic posting; I would
have been able to diagnose this without Paul's message, although his
text handed me the solution on a plate.

So, about what Mike is seeing:

> I am still having a bit of difficultly reading data from a pipe ...
>     ... my python script ... does not fully read all of the
> initial shim account header information.  After reading a total of
> about 8kB of data, the script enters a state where it continually
> polls an empty pipe --- the rest of the ... info is not read.

True; it almost certainly has not been sent yet.

> ... the script is not locking up since it ends up continually
> polling an empty pipe ...

True; it's fair to say the script is livelocked; the shim is waiting
for more messages to fill its buffers, and waiting as well for the
commands that might trigger those messages.

One safety measure that I believe is worth using for this problem is
to always submit a market data subscription for something, anything,
to ensure that messages are arriving on a regular basis.  Then, when
they don't show up, you can begin looking for problems, e.g., with
the IB tws network connection to the upstream servers.

This hides but does not completely solve the problem here, however;
as Paul's report makes clear, log text is being buffered in
unacceptably large blocks.

Mike, please consider using the log file in place of the stdout for
now.  That is, start the shim with the file option, open the log
file, default name log/ShimText, or whatever you've renamed it via
your config file options, and read the log lines from that file.

Yes, this is clumsy, and yes, it's marginally less efficient, and
yes, it sacrifices --- for now --- the elegance of Unix pipes.  It
should, however, work, which is the main virtue of this approach.

Now, about Paul's report:

> ...  The log/ShimText file contains all the expected output,
> irrespective of what the pipe does.

True.  The shim sets line buffering for the log file, in part so
that the tail script, bin/tail.window, will work properly.  This
is how I observe the shim in real time, in particular for testing;
I run scripts, and glance at the tail'd output in a konsole window
to see what's happening.

> ...  Uncommenting the shim.puts line:

>       # shim.puts "wait 10;\nquit;\n" # ...

> causes the shim to quit after 10 seconds, at which point the rest
> of the output then appears.

True.  The stdout is flushed when a process, in this case the shim,
terminates, and similarly, for all of the test scripts in the exs
directory, delays in output are hidden by process termination.

> ...  I can resolve the problem ... by adding a fflush(stdout) [in]
> src/dump.c:

>    Sig(print_cout) {    f.fputs(stdout,      b); fflush(stdout); }

Clearly line buffering is not set for stdout when users include the
cout option on the command line.  Looking at the logic, I see that
I set line buffering for those files that are explicitly opened, 
but not for the stdout.  I vaguely remember deciding not to for
reasons probably having to do with bulk data collection when using
the cout option ...
 
> ... [whether] the shim source code is correct as it is [?] ...

... nevertheless, block buffering makes the cout option useless for
downstream apps using risk mode.  So, the shim needs to provide some
form of user control over the choice of bufffering policy, about
which more in the next section.

>                                                  ... just that it
> seems to resolve it for these python/ruby scripts, and so may give
> us a clue. 

Indeed.  Paul, your diagnostics provide more than a clue, they
pinpoint the issue exactly, and not so incidently suggest a work
around.  Thank you.

Now, about changes to the shim to deal with the problem Mike is
seeing.  My plan for now, once given the cout option, is:

    * for both risk and data modes, to make line buffering for
      stdout the default; and

    * for data mode only, to add an option, "bulk", to let the
      operating system choose the buffer policy, and so maintain
      the current behavior.

Comments about this change are welcome, although I would need to
be given *strong* reasons to justify any plan that fails to make
line buffering the default in risk mode; Mike has uncovered an
important issue on this point.  Also, other semantics might be
added to the bulk option in the future, although this is uncertain.

As for when the change will go out, I'm working in other parts of
the code right now, and so don't plan to make a release today.  I
do, however, expect to get one out soon, perhaps tomorrow, or else
likely early next week.

Mike, if you can't solve the problem you're seeing by using the
log file, please feel free to repost, and again, thanks Paul.

Bill