[ts-gen] Problems with create/submit/cancel [Was: Cancel Order and OCA Orders]

[Bill Pippin]

Ken,

It's not clear exactly what commands you are using, since you only give
me an excerpt.  So, I'll list a full script below, one adapted from your
post:

#!./shim -f

bind key:foobar1 to oid:1;
create item humanreadableID ibc:15016133 at IDEALPRO foobar1 LMT 50000 79.500 0.0;
wait 0.3;
submit item foobar1;	wait 0.3;
cancel item foobar1;	wait 4.0;

exit;

The script above works for me on the first try.  Given the journal,
it only works once.  I've included a trailing four second wait before
the exit to make sure the shim stays around to give the IB tws time
to act on the cancel.  If the shim quits first, all bets are off.

Note that I used 300 millisecond delays otherwise, since anecdotal
evidence indicates that there are problems on the IB tws side if
related order events arrive too close together in time.  An
operational script should instead provide the cout option to the shim,
be reading the log output from the stdin, and match on the related
requests and messages.

Note also that I put the first wait command on a separate line, for
readability.  This should not be necessary, but feel free to put commands
on separate lines if you're trying to track down syntax errors, and if
you find a case where you can't put a wait after a send command, please
let us know on the list, including the exact text of the offending
script.

In any case, when I run the script above against the paper account
for the first time, I see the following in my log:

1| 0| 0|#!./shim -f|
1| 8| 0|bind key:foobar1 to oid:1;|
4|11| 0|# create item humanreadableID ibc:15016133 at IDEALPRO foobar1 LMT 50000 79.500 0.0;|
4|11| 0|# wait 0.3;|
4|11| 0|# submit item foobar1;|
4|11| 0|# wait 0.3;|
4|11| 0|# cancel item foobar1;|
4|11| 0|# wait 4.0;|
4|11| 0|# exit;|
1|18| 0|create item humanreadableID ibc:15016133 at IDEALPRO foobar1 LMT 50000 79.500 0.0;|
2| 3|15|3|15|6|AUD|CASH||0.0000||1|IDEALPRO||JPY||BUY|50000|LMT|79.5000|0.0000|GTC||||0|humanreadableID|0||0|0|0|2|1|0||||||||||||
4|12| 0|# create: CASH:IDEALPRO:AUD:JPY: (LMT, 50000,  79.5000,   0.0000);|
4|12| 0|# submit: CASH:IDEALPRO:AUD:JPY: (LMT, 50000,  79.5000,   0.0000);|
4|12| 0|# cancel: CASH:IDEALPRO:AUD:JPY: (LMT, 50000,  79.5000,   0.0000);|
1| 2| 0|wait 0.3;|
3| 4| 2|    6|2109|Order Event Warning:Attribute 'Outside Regular Trading Hours' is ignored based on the order type and destination. PlaceOrder is now being processed.|
1|21| 0|submit item foobar1;|
2| 3|15|3|15|6|AUD|CASH||0.0000||1|IDEALPRO||JPY||BUY|50000|LMT|79.5000|0.0000|GTC||||0|humanreadableID|1||0|0|0|2|1|0||||||||||||
1| 2| 0|wait 0.3;|
3| 5|10|6|AUD         |CASH||0.000000||IDEALPRO|JPY|AUD.JPY|BUY|50000|LMT|79.500000|0.000000|GTC||DU4181|C|0|humanreadableID|8|1609582724|0|0|0.000000||1609582724.0/DU4181/100|||||||0.000000||0||0|0.000000|0.000000|0.000000|0.000000|0.000000|0|0|0|0|0|0|3|0|0|0.000000|0|2|
3| 3| 6|6|Submitted|0|50000|0.000000|1609582724|0|0.000000|8|
4|12| 0|# journal OrderStatus(1);|
4|12| 0|# journal ActiveOrder(1);|
1|31| 0|cancel item foobar1;|
2| 4| 1|4|1|6||||
1| 2| 0|wait 4.0;|
3| 4| 2|    6| 202|Order Canceled - reason:|
3| 3| 6|6|Cancelled|0|50000|0.000000|1609582724|0|0.000000|8|
4|12| 0|# journal OrderStatus(1);|
1| 7| 0|exit;|

Every thing seems to be working as one would expect; to summarize:

    1. the script shows up as comments, indicating the send commands
       in it have been queued;
    2. the create command is dequeued, and the related place order request
       sent;
    3. journal post operations to the CreateEvent and ChangeOrder tables
       occur, as shown by more comments;
    4. there is an IB tws warning error in response to the original place
       order request, most likely about either the time of day, or the rth
       flag in my order template, neither of which necessarily apply in
       your case;
    5. the submit command is dequeued, and the related place order request
       sent;
    6. open order and order status messages are received, and posted to the
       journal;
    7. the cancel command is dequeued, the related cancel order request
       sent, replied to with another order status message, and that posted;
    8. and the shim exits. 

Note there is always, at least in principle, a race here between the
submission and the cancel; the order could in theory be filled, in which
case the cancel would fail.  Presumably here the limit price is set to
decide the race in favor of the cancel.

Finally, I'm sure you realize that the script would not work a second
time, since the order key, here foobar1, is a constant rather than a
variable.  You're probably aware that the risk.rb command uses
string interpolation to vary the key:

    sequence = "-aa-"
    identity =  sequence  + Time.new.strftime("%y%m%d-%H-%M-%S")
    take_pos = 'take-pos' + identity
    exit_pos = 'exit-pos' + identity
    brand    = 'test-pos'

    create item #{brand} ibc:265598 at SMART #{take_pos} LMT 2 100.0  0.0;

I don't go to the trouble to vary the brand, which for operational
orders and real-life tax accounting is a mistake, but at the very
least the shim *has* to see different order keys, or it thinks you
are trying to modify the order, which is not allowed for the create
command.

> When I try the following 4-liner, the order is created and sent,
> but for some reason, cancel doesn't work (1-3 ok).  Any idea why?

> I'm getting a syntax error on a compound item, but I'm having a
> very hard time figuring out how to fix this ...

Probably something that you think you're doing is not quite what is
happening.  The order process is inherently stateful, and you're
fighting the log by trying to kill it.  Better is to create truly
unique --- original --- order keys as in the risk.rb script.

In the below trace, where the shim has refused to accept a cancel
command, the first and decisive error is during a semantic action,
also an input action, or InAction, named TicA (tick, or order, id
action).
 
> cancel item foobar1;
> Process rule: InAction (TicA)
  ...
> Cursor state: match; text is:
> cancel item foobar1;
> -------------------^
> Drop: ;
> Del;

One possible explanation for the failure of the order id semantic
action is that the key is unknown, and so can't be looked up, say
because the earlier create failed.  I realize that you believe that
the create and submit did succeed, I'm just throwing out possibilities.

By the way, I hope you're redirecting a script here, and not just
typing in from the command line.  The latter is fine for experimenting
with the shim, but once you see problems, you need reproducible tests,
and that requires a file.

The script I give works fine for me on the *first* try as I indicate
above, but a possible scenario for what you've done is suggested by
the battle you're having with the log: 

> While playing with orders on a paper account, I like to wipe some
> tables clean ...  These SQL statements *seem* [emphasis added] to
> clean things up:

> DELETE FROM ActiveOrder;
> DELETE FROM ChangeOrder;
> DELETE FROM CreateEvent;
> DELETE FROM OrderStatus;

> ALTER TABLE ActiveOrder AUTO_INCREMENT = 1;
> ALTER TABLE ChangeOrder AUTO_INCREMENT = 1;
> ALTER TABLE CreateEvent AUTO_INCREMENT = 1;
> ALTER TABLE OrderStatus AUTO_INCREMENT = 1;

You seem to be ignoring the command side of the order journal, which
might be your problem.  You could also use truncate statements.  Here's
probably what you want:

    -- Destroy the order journal; use on the paper account only,
    -- and don't run this unless you know what you're doing.
    -- Note that the order of the first two commands is critical.

    truncate ChangeOrder;
    truncate CreateEvent;
    truncate OrderStatus;
    truncate ActiveOrder;
    truncate OrderResult;
    truncate OrderReport;

I've used the six-statement script above in preparation for rerunning
your order script over and over, and it always worked.  I also tried
it with the wait commands left out, and it sometimes worked fine that
way as well, though once I saw the following, that the submit failed
because the IB tws had not yet finished processing the create:

1|21| 0|submit item foobar1;|
2| 3|15|3|15|17|AUD|CASH||0.0000||1|IDEALPRO||JPY||BUY|50000|LMT|79.5000|0.0000|GTC||||0|humanreadableID|1||0|0|0|2|1|0||||||||||||
3| 4| 2|   17|2102|Unable to modify this order as its still being processed.|
1|31| 0|cancel item foobar1;|
2| 4| 1|4|1|17||||
3| 3| 6|17|ApiCancelled|0|50000|0.000000|0|0|0.000000|8|
4|12| 0|# journal OrderStatus(1);|
1| 7| 0|exit;|

By the way, for money-losing regression tests with the paper account
only, you may want to set the "Bypass order precautions for api orders"
attribute in the IB tws Configure->API->All API settings dialogue;
otherwise a popup might occur.

I'm not really sure what is behind the problem you're seeing, but
you're probably being bitten by hidden state, either on the IB tws
side, or with the shim's per-session accumulated state or its journal,
such as a create that fails because the order key is not unique, or
a cancel that fails because the order key is unknown.

Note that the commands you give only reset the message side of the
log, and leave the command side untouched.  I'm not sure what you
are doing here, but there is accumulated state from the create to
the submit to the cancel, it is echoed to the log and also stored
in the shim for the duration of the session, and you can't rerun those
commands without completely resetting the command side of the order
journal and starting a new session, using e.g., the truncate
ChangeOrder, truncate CreateEvent statements above, in that order. 

Probably you should leave the journal alone, bite the bullet and use
a dynamic order key, and enjoy an accumulated history of all the tests
you make.  The shim has been designed to require unique keys for all
order commands, and you seem to be fighting that requirement by
resetting the journal.  Feel free as long as you're testing with the
paper account, but don't be surprised if it seems tricky.

Thanks,

Bill