[ts-gen] MacOS Shim problems

[Bill Pippin]

Ric,

[The last para of this post may also be of interest to Mike and Nils.]

Thanks for your additional efforts at long distance debugging.  I have
been working on a thorough refactoring of the code in wait.c, in the
hopes that this might give us at least some new diagnostic hints, but
you've found the problem elsewhere:

> Had a little bit of time this evening and I made some progress with
> poking at the hang with gdb.  ...  Hope that provides a hint ...

Actually, much more than a hint --- I believe you've supplied what
is most of the way there to being a complete diagnosis.  There is a
bug in a binary search loop, and here's the patch:

*** old/leaf.c	Thu May  7 19:34:16 2009
--- src/leaf.c	Thu May 21 13:57:26 2009
***************
*** 177,185 ****
  //                  d.unit  .text(),
  //                  d.region.text());
  
!     for(nat i(mid(a, b)); a<b && !p; i = mid(a, b)) {
          switch((x = d.compare(t[i]))) {
!             case -1: b = i    ; break;
              case  1: a = i + 1; break;
              case  0: p = &t[i];
          }
--- 177,185 ----
  //                  d.unit  .text(),
  //                  d.region.text());
  
!     for(nat i(mid(a, b)); a<=b && !p; i = mid(a, b)) {
          switch((x = d.compare(t[i]))) {
!             case -1: b = i - 1; break;
              case  1: a = i + 1; break;
              case  0: p = &t[i];
          }

The new code continues on to make one more comparison, or else it
might fail to find an occurring key; and trims the intervals more
aggressively, else it to will hang.  By the way, the midpoint test,
in tables.h, was already correct even for large array lengths,
since it uses an unsigned shift on unsigned indices.

If it's not obvious why the shim was looking up data via binary search
when you've not yet even entered any commands, you should know that:

The shim sends an account data request at startup in order to determine
the account code, for use in order accounting; the IB tws returns
portfolio records as part of this account data; and the portfolio
messages cause the shim to make product lookups in order to determine
what contract product those messages refer to.

In addition, as part of the default install of the IB tws there are a
number of checkboxes selected under Configure->All API settings, and
these options may increase the variety of messages the shim sees at
startup:

    * Download open orders on connection
    * Include FX positions when sending portfolio
    * Send status updates for EFP and Volatility orders
          with 'Continuous Update' flag

The binary search where you found the shim to be hanging was due to a
product search most likely caused by one of the portfolio messages
returned along with the other account data.
 
> I uncommented ... debug ... but it did not work so I [added] a print
statement I could understand and it pretty much just loops.

The printfs within the operator[] function are outside the loop, and so
were of no help.  There is also a commented out call to a helper trace
function print_line() inside the loop, and calling that should have done
the trick.  If by "uncommented" you are referring to the print_line()
procedure, I'd be interested in knowing what the problem you saw was.

There are other places in the code I want to look at, and I'll be
putting out a release either today, tomorrow, or both, to include
this patch, your earlier fix in lib/inet.c, the refactoring for
wait.c, changes for the problem Mike was having, and a change from
object to character level request queuing that I'm making in the
hopes it will help uncover an intermittent problem Nils has observed.

Thanks,

Bill