From paulq2o0 at yahoo.co.uk Mon Mar 15 10:05:06 2010 From: paulq2o0 at yahoo.co.uk (Paul C) Date: Mon, 15 Mar 2010 14:05:06 +0000 Subject: [ts-gen] error compiling shim-100312 Message-ID: <20100315140506.GA18897@godel> Bill, The last shim didn't compile (on g++ (Debian 4.4.2-9) 4.4.3), failing with: src/mode.c:52: error: lvalue required as unary '&' operand Removing the comma at the end of the previous line seems to fix it. ( By the way, do you prefer we use the Bug Tracker for reporting issues, or is that no longer used? ) Regards Paul C From pippin at owlriver.net Mon Mar 15 13:35:12 2010 From: pippin at owlriver.net (Bill Pippin) Date: Mon, 15 Mar 2010 13:35:12 -0400 Subject: [ts-gen] error compiling shim-100312 Message-ID: Paul, Thanks for the quick report. I'm not sure how that file got into the release, but clearly I made some kind of mistake in pulling together files, running the regression test suite, and rolling the tarball. Thanks for catching it, and I've pushed another tarball with the obvious fix you pointed out. I'm continuing to look at the code for other issues. Thanks, Bill From endoscope at gmail.com Sat Mar 20 18:04:06 2010 From: endoscope at gmail.com (Mike Thornton) Date: Sat, 20 Mar 2010 15:04:06 -0700 Subject: [ts-gen] shim error Message-ID: Hi Bill, I am having difficulty with the latest shim release "shim-100318" (have used a release from last year up to this point). I am using the default values for the ".shimrc" file, and I can connect to the mysql database with a user name of "shim" without a password. Do you have any insights into what is wrong? ----------------------- shim dump ---------------------------- ./shim --data The trading shim has connected to the database server and loaded 51620 products. The IB tws offered api level 47, and connection negotiation resulted in an api level of 23 and a client id of 1. Problem: 414 mysql query evaluation failed Error 1044 (42000): Access denied for user 'shim'@'localhost' to database 'testing' insert ignore into AccountCode(code) values ('UXXXXXXXX'); Problem: 417 AccountCode tuple not found following insert Problem: 525 could not obtain IB account info at startup Exiting ----------------------------------------------------------------- Thanks. Mike -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.trading-shim.org/pipermail/ts-general/attachments/20100320/826e4766/attachment.html From pippin at owlriver.net Mon Mar 22 10:29:59 2010 From: pippin at owlriver.net (Bill Pippin) Date: Mon, 22 Mar 2010 10:29:59 -0400 Subject: [ts-gen] [Re:] shim error Message-ID: Mike, About the startup problem you are seeing, I'll start with the error dump you've provided; add a note about shim internals; highlight the underlying point of failure; point out a work-around I hope you'll skip, and, last, give a hypothesis you can check as to the fundamental problem and solution. With respect to that last point, please let us know what you find. > Problem: 414 mysql query evaluation failed > Error 1044 (42000): Access denied for user 'shim'@'localhost' ... > Problem: 417 AccountCode tuple not found following insert > Problem: 525 could not obtain IB account info at startup > Exiting 1. It's probably obvious from the above that the shim does not have write permission to the AccountCode table. The mysql server returned an error on an attempted insert, the shim translated that to the 414 error, the following checking attempt to read the insert failed for the simple reason that it wasn't there, and the shim quit, unable to confirm the account code. 2. Note that once that account code record reaches the database, the shim doesn't attempt to write it on later sessions; the problem you are seeing is a one-time affair that might show up when you start with a new, empty database, or move to a new account. 3. Why wouldn't the shim have write permissions? This is purely a matter of database setup; nothing in the design or implementation of the shim itself modifies database permissions in any way. So, the fundamental issue lies with the database setup process. 4. Although you've probably figured out already that you can finesse this problem by manually entering the AccountCode tuple into the appropriate database record, using the mysql interpreter and a hand-typed sql insert statement, I hope you'll consider the more fundamental solution. 5. You may have been recreating the database tables with the create.sql script, and indeed that is what it's for, yet in this case, given the elapsed time from your last install, the shim's database permissions are probably incomplete, and you likely need to start from scratch, using the setup.sql script rather than the create.sql script. The setup script drops the old databases, recreates new ones, adds permissions, and rebuilds tables; the create.sql script takes care of the last step only, as it drops, recreates, and repopulates tables. So, once you've unloaded any journal data you want to save, and in order to obtain admin privileges on the mysql dbms and run the setup.sql script: begin from a command line session having as current directory the sql subdirectory of the unpack directory, e.g., shim-100319/sql ; start up the mysql interpreter with the -p option by itself; answer the resulting password prompt with the appropriate value; and source the setup.sql script from with the sql session. This should fix your problem, and in any case, please let us know what happens. By the way, near-future versions of the shim are likely to include some changes to startup account code processing, involving support for the Account configuration variable. That, however, is an issue for another post on another day. Thanks, Bill From endoscope at gmail.com Tue Mar 23 03:21:56 2010 From: endoscope at gmail.com (Mike Thornton) Date: Tue, 23 Mar 2010 00:21:56 -0700 Subject: [ts-gen] historical data format Message-ID: Hi Bill, Quick questions regarding the usage of the shim past command. I ran the example ruby script (past.30.rb) to download data, and I want to clarify the usage of the command and the printing format. I see data printed out like this 23873|58924| 102437959|0| 1| 0|20100312 11:07:00| 29.28| 29.29| 29.25| 29.25| 719| 29.27|false|STK:SMART:MSFT:| 23873|58924| 102437994|0| 1| 0|20100312 11:08:00| 29.25| 29.26| 29.24| 29.25| 492| 29.25|false|STK:SMART:MSFT:| 23873|58924| 102438029|0| 1| 0|20100312 11:09:00| 29.25| 29.26| 29.25| 29.25| 103| 29.25|false|STK:SMART:MSFT:| 23873|58924| 102438064|0| 1| 0|20100312 11:10:00| 29.25| 29.25| 29.24| 29.24| 636| 29.25|false|STK:SMART:MSFT:| What does the initial 23873|58924| 102437959|0| 1| 0 mean? I figure that "102437959" is a timestamp, but what do the other values mean? Also is the line substring 29.28| 29.29| 29.25| 29.25| 719| 29.27|false printed out in the API format to mean the following? open | high | low | close | volume | weighted-average-price | hasGaps For the following quest string, I am guessing that "m1" refers to 1 minute bars, but what does the following "1 1d" indicate b/c it appears the date and the upper daily data cutoff is given inside of the Ymd_T(...) substring? select past STK:SMART:MSFT:USD m1 1 1d Ymd_T(20100312 16:00:00); I tried to mimic this to download Forex data, but was unable to do so. I used the command select past CASH:IDEALPRO:EUR:USD m1 1 1d Ymd_T(20100312 16:00:00); However, what I saw print out in the logfile was 23873|59291| 469189941|2|20| 3|23873|59291| 469189941|2|20| 3|| 23873|59352| 530357165|1| 0| 0|| 23873|59352| 530357228|1| 0| 0|| 23873|59352| 530357254|1| 0| 0|| 23873|59352| 530357278|1| 0| 0|| 23873|59352| 530357302|1| 0| 0|| 23873|59352| 530357326|1| 0| 0|| 23873|59352| 530357350|1| 0| 0|| <---------------------------- these lines repeat for most of the log file ... The last few lines repeated indefinitely and the logfile grew to 223MB -- I did not see any valid Forex data. I had to kill the shim to prevent the logfile from growing larger. What did I do wrong? Thanks. Mike -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.trading-shim.org/pipermail/ts-general/attachments/20100323/92568971/attachment.html From pippin at owlriver.net Tue Mar 23 13:21:16 2010 From: pippin at owlriver.net (Bill Pippin) Date: Tue, 23 Mar 2010 13:21:16 -0400 Subject: [ts-gen] historical data format Message-ID: Mike, Before I respond to your new question about history data, would you please be so kind as to let us know what the result was with respect to your previous query? In particular, I'm interested in whether you corrected the database permissions by using the setup.sql script, or whether you avoided the problem by manually entering the account code, offline, and in either case, what happended when you next started the shim. Your answer may also be of later interest to readers browsing our list archives. Thanks, Bill From endoscope at gmail.com Tue Mar 23 18:36:46 2010 From: endoscope at gmail.com (Mike Thornton) Date: Tue, 23 Mar 2010 15:36:46 -0700 Subject: [ts-gen] historical data format In-Reply-To: References: Message-ID: Hi Bill, OK, no problem. Regarding the database query. Apparently my testing database had become corrupt -- I am not sure whether this was a result of my tinkering with the database or from the scripts. I suspect that the corruption was not a result of the scripts. After your email, I started mysql manually, and mysql detected that the "testing" database was corrupt. I am not sure why mysql did not detect this earlier since I had started mysql manually a few times, albeit with a different user name and not "shim". Mysql said something about needing to repair the database, which it did behind the scenes. I did not need to run any of the scripts again. Then, when I started the shim, it just started working. Before I had run the latest sql scripts for the first time (I had an older version of the shim database setup from over 1 year ago), I manually deleted the trading database. The testing database would not "drop" for some reason -- I cannot remember the error message. However, the setup.sql script had run OK, and the testing database looked fine upon running queries so I had initially assumed it was OK. So to answer your question, I am not sure what I did to fix it. Somehow mysql detected the database corruption. Perhaps I should rerun the "setup.sql" script, but I have not done this since the shim says it connects OK. Thanks. Mike On Tue, Mar 23, 2010 at 10:21 AM, Bill Pippin wrote: > Mike, > > Before I respond to your new question about history data, would > you please be so kind as to let us know what the result was with > respect to your previous query? > > In particular, I'm interested in whether you corrected the database > permissions by using the setup.sql script, or whether you avoided > the problem by manually entering the account code, offline, and in > either case, what happended when you next started the shim. > > Your answer may also be of later interest to readers browsing our > list archives. > > Thanks, > > Bill > > _______________________________________________ > ts-general mailing list > ts-general at trading-shim.org > http://www.trading-shim.org/mailman/listinfo/ts-general > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.trading-shim.org/pipermail/ts-general/attachments/20100323/cb54c67f/attachment.html From pippin at owlriver.net Tue Mar 23 19:44:42 2010 From: pippin at owlriver.net (Bill Pippin) Date: Tue, 23 Mar 2010 19:44:42 -0400 Subject: [ts-gen] Failed dbms account code update [Was: shim error] Message-ID: Mike, With respect to version dependent database problems: > I started mysql manually, and mysql detected that the "testing" > database was corrupt. I am not sure why mysql did not detect this > earlier since I had started mysql manually a few times, albeit with > a different user name and not "shim". Mysql said something about > needing to repair the database, which it did behind the scenes. I > did not need to run any of the scripts again. Then, when I started > the shim, it just started working. The setup.sql script takes dbms admin (root) permissions, drops both databases, testing and trading, recreates the databases, adds permissions, creates tables, and populates them. The create.sql script --- for one database only, determined by context --- drops tables, recreates them, and repopulates them. So, one way to get your databases out of sync is to use create.sql to update one and not the other. > Before I had run the latest sql scripts for the first time (I had > an older version of the shim database setup from over 1 year ago), > I manually deleted the trading database. The testing database > would not "drop" for some reason -- I cannot remember the error > message. However, the setup.sql script had run OK, and the testing > database looked fine upon running queries so I had initially assumed > it was OK. Dropping tables must be performed bottom up, that is postorder for the foreign key dependency graph. When tables are added or renamed, this means that the create.sql script from a newer version of the shim may have trouble with dropping the tables from an older version. The setup.sql script, in contrast, since it drops both databases, just blows everything away all at once. It sounds like you may have had some problems with the create.sql script. > So to answer your question, I am not sure what I did to fix it. > Somehow mysql detected the database corruption. Perhaps I should > rerun the "setup.sql" script, but I have not done this since the > shim says it connects OK. Hard to say from here, but glad it works. Thanks for the explanation, Bill From pippin at owlriver.net Tue Mar 23 20:15:05 2010 From: pippin at owlriver.net (Bill Pippin) Date: Tue, 23 Mar 2010 20:15:05 -0400 Subject: [ts-gen] Failed history query [Was: historical data format] Message-ID: Mike, Your 1st questions are about log format; 2nd, about history query command syntax; and 3rd, about the lack of useful answers to a history query, what would appear to be a bug. Since your message raises a variety of points, I'll break my reply into multiple posts. I'll take them in reverse order, focusing here on diagnosing problems with history data queries. So, log format is outside the scope of this message --- I'll talk about that in a follow-on --- but if you're eager, it's also discussed in various previous posts to this list, including the following messages I wrote that are still in the archives: http://www.trading-shim.org/pipermail/ts-general/2009-July/000474.html http://www.trading-shim.org/pipermail/ts-general/2009-September/000576.html Returning to the subject of history query problems, your first point of attack for problems with history queries, or for that matter any process problem with cmd->req->msg events, should be to look at the log, starting from the top for the current session, that is, for a permanent log, where you've used the join option to append each session, searching from the bottom up towards the first occurrence of the string "star", e.g.: > 14901|69724| 1.514403|4| 0| 0|# |star|****************| > 14901|69724| 0.000008|4| 1| 0|# |head|0.95|080820|data| > 14901|69724| 0.000000|4| 0| 0|# |star|****************| > 14901|69724| 0.000005|4| 2| 0|# |hand|23|1|45|20100323 \ > 18:22:03 EST|Connect with: cv 23, id 1, sv 45| > 14901|69724| 0.000000|3| 9| 1|1| After the initial banner, you'll see the results of an account data query that the shim has used to discover the account code. After this, your downstream commands, the resulting requests, and the asynchronous messages that occur in reply to those requests, appear, one after another. For problems with history queries, you'd want to look at the startup messages about the state of the history farms, and see what messages followed the history query request; formatting problems, and rejection due to invalid options, are a common problem here. So, dealing specifically with your failed forex query: > However, what I saw print out in the logfile was > 23873|59291| 469189941|2|20| 3|23873|59291| 469189941|2|20| 3|| > 23873|59352| 530357165|1| 0| 0|| > 23873|59352| 530357228|1| 0| 0|| > 23873|59352| 530357254|1| 0| 0|| > 23873|59352| 530357278|1| 0| 0|| > 23873|59352| 530357326|1| 0| 0|| > 23873|59352| 530357350|1| 0| 0|| <------------ these lines > repeat for most of the log file ... > The last few lines repeated indefinitely and the logfile grew > to 223MB -- I did not see any valid Forex data. I had to kill > the shim to prevent the logfile from growing larger. What did I > do wrong? Not knowing how you sent that command to the shim, and in particular what commands preceded and followed, and without the first part of the log, the interesting part, before it starts repeating, it's impossible to say. No matter how much crud ends up in the log file, it's what appears before that that is most interesting. It's possible that the ruby script is sending text to the shim, and the shim is happily sending log text back. It's not hard to make such a loop continue indefinitely, since both ruby and the shim are designed to keep on keeping on. So, as is true for most debugging situations, you need to reduce the problem scope. One approach would be to insert debug statements in the ruby script, and modify that script to send ping messages to the shim, in order to mark the log. Instead, in what follows below, I've reduce problem scope by using a shim script, rather than a Turing-complete scripting language like ruby: > #!./shim -f > select past CASH:IDEALPRO:EUR:USD m1 1 1d Ymd_T(20100312 16:00:00); > wait 10; > exit; When I ran the above script --- less the "> " prefix --- as a shell command, and with arguments " --data file save diff join" , the following event payload resulted: 1| 0| 0|#!./shim -f| 1| 0| 0| | 1| 0| 0|select past CASH:IDEALPRO:EUR:USD m1 1 1d Ymd_T ... 1| 0| 0|wait 10;| 1| 0| 0|exit;| 1|11| 0|select past CASH:IDEALPRO:EUR:USD m1 1 1d Ymd_T ... 2|20| 3|13539|40395| 0.000020|2|20| 3|| 1| 0| 0|exit;| 1| 2| 0|wait 10;| 3| 4| 2| -1|2106|HMDS data farm connection is OK:ushmds2a| 3| 4| 2| 1| 165|Historical Market Data Service query ... 3| 4| 2| 1| 162|Historical Market Data Service error ... 1| 7| 0|exit;| Both the enqueue and dequeue events for the history query list the same text, which I originally cut and pasted from your message: |select past CASH:IDEALPRO:EUR:USD m1 1 1d Ymd_T(20100312 16:00:00);| The history farm status messages, when split over two lines, appear as follows: > 165|Historical Market Data Service query message: > HMDS server connection was successful.| > 162|Historical Market Data Service error message: > No historical market data for EUR/CASH at IDEALPRO Last 60| So, for me, when I run your query, there's no history data at the farm for your query. Thanks, Bill From pippin at owlriver.net Wed Mar 24 18:00:04 2010 From: pippin at owlriver.net (Bill Pippin) Date: Wed, 24 Mar 2010 18:00:04 -0400 Subject: [ts-gen] Ib tws api --- shim log record format Message-ID: This message is in response to a query from Mike Thornton about the interpretation of the log record format, and in particular the meaning of history data detail lines. This message is, however, much more general than the particular question that triggered its posting. The following is a general description of the shim's log format, including pointers to reference documents and source files where you can find more information. The log format can only be understand in the context of the IB tws api, so, to start, a brief introduction to that interface: IB Tws Api Intro ---------------- The IB tws api is a tcp-socket-based specification for client requests, and the resulting asynchronous messages produced in response by the IB tws process. It provides access to market data and account information, and accepts user-initiated orders. Of particular interest to the trader, orders may be grouped into one-cancels-all (OCA) groups, either by the user or by default under control of the IB tws, and the latter occurs for those orders that share a common parent order. So, the application can, to some degree, chain together entries and stops. There are a variety of resource limitations that apply to the api, including max counts for subscriptions to market data (100) and market depth (3), and pacing limits for requests in general (20 milliseconds) and history queries in particular (cumulative, 10 seconds each). There is also folklore to the effect that parent-child orders and order modification events should obey a moderate pacing limit, with 300 milliseconds being the collective consensus as to the largest pacing delay needed. The api requests and messages (collectively events) consist of null terminated tokens, of either a fixed number, or else, in a few cases, to form a counted repeating group; and there is *no* message delimiter. Events begin with a pair of natural numbers, the event index and event version, and it is by matching on those that an api program can determine how the tokens that follow should be interpreted, and in particular, given the known length for most events, or the counted length for repeating groups --- history messages are the classic example here --- where the *next* event starts. The tws api process seems to respond to non-trivial errors in request format, including in particular invalid request indices, by terminating the connection. Errors in individual data values, in contrast, seem to produce limited but useful error messages. The features of the api, including both the type and number of elements in events, and the range of possible events themselves, varies from one version to the next, and significantly over time. Api versions are best identified by the (currently) two digit numbers exchanged by the client [shim] and server [tws] processes. The tws server api level is currently into the mid-forties, and continues to provide support for earlier versions, all the way back to the mid or low single digits. Lower version clients lose access, to some degree, to newer api features. Newer versions of the api have included an increasing number of new message types that provide end markers for message lists; e.g., for the sequence of account data records periodically produced in response to an account data query, or the sequence of contract detail records, in response to a contract wildcard query. There has been no change, in contrast, to the use of variable-length events with no message terminators. For more information on the api, please see the official IB tws api guide, and also two source files from their java sample client, EClientSocket.java and EReader.java, which serve as the only trustworthy documentation of the wire formats for requests (EClientSocket) and messages (EReader). The sample client source files will give you name, type, and number of attribute values per api event, while the api guide explains the values that those attributes can assume. So, you'll need to look at both. The api guide is substantially redundant, consisting of similar sections distinguished by language interface, and you can pick just one and use only that; I recommend that you read either the Java or C++ sections. The api guide is available from the IB web site, and the following link should work: http://www.interactivebrokers.com/php/apiUsersGuide/apiguide.htm If you download and unpack the java sample client sources, the EClientSocket.java and EReader.java files should be found via the path: IBJts/java/com/ib/client/ That's all herein about the api in isolation, and now, before I focus on the shim's log record format, some context first about the shim itself: The trading shim ---------------- The trading shim has been described as a "dbms-augmented command interpreter" for the IB tws api. By that is meant that client commands are expanded with information from the symbols database to generate api requests, and that the shim then reads and relays the resulting messages. As an interpreter, the shim is written with the expectation that it be driven by downstream programs; it is of course possible to type commands in by hand, or pipe in short test scripts, but the full power, especially conditional processing for orders, requires a controlling client program, typically written in some scripting language such as Perl, Python, or Ruby. So, what exactly does the shim bring to the table? What does it provide that makes it easier to write such downstream client programs? After all, given the socket api, client scripts could just open a connection directly, poke api tokens down the socket, and read the resulting message tokens that are returned. On the request side, the shim provides translation of simple text command statements into api requests, using the dbms augmentation referred to above, as well as request queueing to honor pacing limits and other resource limits. On the message side, the shim provides both error handling, with recovery from message syntax errors, in contrast to the abrupt termination approach used by the IB tws; and, finally the log formatting of events, about which more to follow. This raises the notion of shim log events, and the resulting log stream. Shim log records ---------------- A trading-shim api event is a command, request, message, or comment, where commands are accepted from the downstream and enqueued and dequeued according to pacing and resource limits; requests are translated from dequeued commands, and sent to the IB tws; messages are received, asynchronously, in response to api requests; and comments are created by the shim to annotate other events. A log event is either a command, request, message, or comment, as above; or a log detail line, an item from a message repeating group, e.g., a history bar detail line. The log stream is the serialized sequence of log events output by the shim. This brings me, finally, to the shim log record format, which in part echoes the wire format of api events, so that you *will* need the above mentioned EClientSocket and EReader files in order to predict, as well as interpret, the api events listed in the log. In fact, the log format makes little sense until considered in the context of the api itself, and the shim's role as a intermediary to simplify the development of downstream --- that is, client --- programs. Log event format ---------------- The format text of log events consists of vertical-bar-terminated tokens with a trailing newline. This format is chosen to allow easy processing by downstream scripts, which can use the newlines to recognize record boundaries, and can split on vertical bars to break records up into tokens. All events have a common prefix, about which more to follow; the payload, ditto; and an optional suffix. Log event prefix ---------------- For the log file and stdout channels for the log, the log event prefix consists of four fields: the process pid; the seconds field of the timestamp; the fraction part of the timestamp; and the event code. More precisely, of the four fields of the prefix: the 1st field ends with the pid, has only the pid for the file and stdout channels, while, for the syslog output channel only, has also textual date, time, and user name prior to the pid, though with colons between parts, so that there is still just one vertical bar terminated field for the pid; the 2nd field is seconds past midnight; the 3rd, the fractional part of the timestamp, is in microseconds, either absolute, or, with the diff option, as the differential from the previous event; and the 4th, the event type, is a numerical code in {1, 2, 3, 4}, as the event is a command, request, message, or comment, respectively. Api payload text ---------------- In addition, the following is true of the event payload, that part of the log record text after the prefix and not including suffix annotation: commands: for syntactically correct commands, the log echoes their text, with some reordering to reflect queueing Note: syntax errors are horribly garbled, and so you must use the stderr text to understand what went wrong. requests: faithfully echo the wire format; the same code that uses the request object to lay out the request with null separators for the upstream, uses vertical bars for the log; the only difference is the output stream object (Logger or Sender) and the buffer type (NulBuf or BarBuf). messages: reflect the wire format, though with both whitespace formatting and often suffix annotation, e.g., the symbol translation for the tick id; the format for market data is explained below. Leaving aside the message prefix, and any suffix annotation, and focusing specifically on requests and messages, it's worth emphasizing the following about the payload for those event types: All log-formatted requests and messages have, as their payload immediately following the prefix, a one-for-one list of the tokens that make up the wire format request or message. So, since request and message logging reflects the wire format, and that format is documented by IB in the EClientSocket.java and EReader.java files of their sample client, you can match the attributes of the api event payload in the log against IB's source file text one-for-one. Some examples may be of interest. E.g., for market data, and with price data obscured to honor IB's license restrictions on redistribution, market data for Apple [last year] appeared in the log as follows: 15481|65972| 0.000017|1| 9| 0|select tick STK:SMART:AAPL:USD 1;| 15481|65972| 0.000016|2| 1| 5|1|5|1|AAPL|STK||0.0000||1|SMART||USD|||||| 15481|65972| 0.022712|1| 2| 0|wait 2;| 15481|65972| 0.229638|3| 1| 5| 1| 1| 1?3.4| 3|1|... 15481|65972| 0.000008|3| 1| 5| 1| 2| 1?3.6| 2|1|... 15481|65972| 0.000004|3| 1| 5| 1| 4| 1?3.6| 1|0|... 15481|65972| 0.000005|3| 2| 5| 1| 0| 3|0|... 15481|65972| 0.000003|3| 2| 5| 1| 3| 2|0|... 15481|65972| 0.000003|3| 2| 5| 1| 5| 1|0|... 15481|65972| 0.000003|3| 2| 5| 1| 8| 179???|0|... 15481|65972| 0.000005|3| 1| 5| 1| 6| 1?5.0| 0|0|... 15481|65972| 0.000004|3| 1| 5| 1| 7| 1?2.9| 0|0|... 15481|65972| 0.000004|3| 1| 5| 1| 9| 1?2.8| 0|0|... 15481|65974| 1.798008|1|10| 0|cancel tick STK:SMART:AAPL:USD;| 15481|65974| 0.000063|2| 2| 1|2|1|1|||| In the above text, the first four columns are prefix information: the process id, seconds since midnight, microseconds, and event type, here: a cmd, req, cmd, 10 market data msgs, cmd, and req. Note from the low-microsecond values of the fractional part of the timestamp for most of the events that the shim was running with the diff option. Payloads are repeated below; note that market data price and size messages have seven and five attributes, respectively: 1| 5| 1| 1| 1?3.4| 3|1| 1| 5| 1| 2| 1?3.6| 2|1| 1| 5| 1| 4| 1?3.6| 1|0| 2| 5| 1| 0| 3| 2| 5| 1| 3| 2| 2| 5| 1| 5| 1| 2| 5| 1| 8| 179???| 1| 5| 1| 6| 1?5.0| 0|0| 1| 5| 1| 7| 1?2.9| 0|0| 1| 5| 1| 9| 1?2.8| 0|0| Of course the question marks above are not part of the logging, and are used herein to avoid running afoul of IB's license restrictions. For market data price events, the attributes are message index, version, tick id, market data subtype, price, quantity, and the "can auto execute" flag. Note also that the whitespace formatting is not found in the original wire format data as received over the socket. The suffix annotation here consists of the 0-value flag padding for market data size events, the tick subtype explanation, and the tick id translation of 1 as STK:SMART:AAPL: 1| 1| 1?3.4| 3|1|price.outcry.bid. |STK:SMART:AAPL: 1| 2| 1?3.6| 2|1|price.outcry.ask. |STK:SMART:AAPL: 1| 4| 1?3.6| 1|0|price.summary.last. |STK:SMART:AAPL: 1| 0| 3|0|size.bid. |STK:SMART:AAPL: 1| 3| 2|0|size.ask. |STK:SMART:AAPL: 1| 5| 1|0|size.last. |STK:SMART:AAPL: 1| 8| 1?9494|0|size.volume. |STK:SMART:AAPL: 1| 6| 1?5.0| 0|0|price.summary.high. |STK:SMART:AAPL: 1| 7| 1?2.9| 0|0|price.summary.low. |STK:SMART:AAPL: 1| 9| 1?2.8| 0|0|price.summary.close.|STK:SMART:AAPL: For another example, recall that Mike asked about the interpretation of history bar detail lines. The following is an excerpt of payload text from the log after running the exs/test script, which includes a history query. The prices have, again, been obscured to honor IB's license restrictions on redistribution. 1|11| 0|select past FUT:ECBOT:YM:USD:20100618 h1 11 1d now;| 2|20| 3|14901|69734| 0.000049|2|20| 3|| 1| 2| 0|wait 6;| 3| 4| 2| -1|2106|HMDS data farm connection is OK:ushmds2a| 3| 4| 2| 6| 165|Historical Market Data Service query message: ... 1| 9| 0|select tick ibc: 266093 at SMART 1;| 2| 1| 5|14901|69740| 0.000052|2| 1| 5|| 1| 2| 0|wait 2;| 3|17| 3| 6|8| 0| 1| 0|20100323 08:30:00|107nn.00|107nn.00|107nn.00|107nn.00| 98nn ... 0| 1| 0|20100323 09:00:00|107nn.00|107nn.00|107nn.00|107nn.00| 203nn ... 0| 1| 0|20100323 10:00:00|107nn.00|107nn.00|107nn.00|107nn.00| 149nn ... 0| 1| 0|20100323 11:00:00|107nn.00|107nn.00|107nn.00|107nn.00| 103nn ... 0| 1| 0|20100323 12:00:00|107nn.00|107nn.00|107nn.00|107nn.00| 106nn ... 0| 1| 0|20100323 13:00:00|107nn.00|107nn.00|107nn.00|107nn.00| 98nn ... 0| 1| 0|20100323 14:00:00|107nn.00|108nn.00|107nn.00|108nn.00| 259nn ... 0| 1| 0|20100323 15:00:00|108nn.00|108nn.00|108nn.00|108nn.00| 28nn ... 3| 2| 5| 7| 8| nnnnnn|0|size.volume. ... 3| 1| 5| 7| 6| nn.nn00| 0|0|price.summary.high. ... 3| 1| 5| 7| 7| nn.nn00| 0|0|price.summary.low. ... 3| 1| 5| 7| 1| nn.nn00| nn|1|price.outcry.bid. ... 3| 2| 5| 7| 0| nn|0|size.bid. ... 3| 1| 5| 7| 2| nn.nn00| n|1|price.outcry.ask. ... 3| 2| 5| 7| 3| n|0|size.ask. ... 4|12| 0|# |post|event: history insert| In the transcript above, there is the dequeue event for the history query command; the resulting request; a wait command, used here to help keep the history query answer reasonably close to the its initiating request; two status messages from the IB tws about the status of the history data farm; a market data command, and request; another wait command; and, finally, the history query answer. History data messages begin with a header, including the history bar detail line count, here 8, as well as, left to right, the history message index, 17; the version, 3; and the temporary contract id, 6; assigned on the fly in sequence to stand for the contract expressions occurring in the command text as FUT:ECBOT:YM:USD:20100618 . By the way, for most messages, the suffix is just the contract expression from the related command, so that the downstream need not understand these temporary contract id indices, what I call tick ids. Now, finally, to the answer for Mike's question, about the interpretation of the prices in the detail lines. The following excerpt is from the file EReader.java: case HISTORICAL_DATA: ... ... date = read ... ... open = read ... ... high = read ... ... low = read ... ... close = read ... ... volume = read ... ... WAP = read ... ... hasGaps = read ... Repeating the history query answer, and focusing on the detail lines, it's clear that the prices are open, high, low, close, and with those followed by volume: 3|17| 3| 6|8| 0| 1| 0|20100323 08:30:00|107nn.00|107nn.00|107nn.00|107nn.00| 98nn ... 0| 1| 0|20100323 09:00:00|107nn.00|107nn.00|107nn.00|107nn.00| 203nn ... 0| 1| 0|20100323 10:00:00|107nn.00|107nn.00|107nn.00|107nn.00| 149nn ... 0| 1| 0|20100323 11:00:00|107nn.00|107nn.00|107nn.00|107nn.00| 103nn ... 0| 1| 0|20100323 12:00:00|107nn.00|107nn.00|107nn.00|107nn.00| 106nn ... 0| 1| 0|20100323 13:00:00|107nn.00|107nn.00|107nn.00|107nn.00| 98nn ... 0| 1| 0|20100323 14:00:00|107nn.00|108nn.00|107nn.00|108nn.00| 259nn ... 0| 1| 0|20100323 15:00:00|108nn.00|108nn.00|108nn.00|108nn.00| 28nn ... I realize that users might prefer some form of documentation besides the IB sample client java sources, the files EClientSocket.java and EReader.java. Nevertheless, the IB sources are the only official "docs"; it's their api, and this is their approach to documentation. I'm not going to try to duplicate it, especially given the version issues involved. Thanks, Bill From herrold at owlriver.com Fri Mar 26 11:17:02 2010 From: herrold at owlriver.com (R P Herrold) Date: Fri, 26 Mar 2010 11:17:02 -0400 (EDT) Subject: [ts-gen] General followup questions, after: Availability of support and development status of Trading Shim In-Reply-To: <302656.50661.qm@web33705.mail.mud.yahoo.com> References: <729422.30943.qm@web33703.mail.mud.yahoo.com> <302656.50661.qm@web33705.mail.mud.yahoo.com> Message-ID: On Fri, 26 Mar 2010, a private email asked: > Thanks for taking the time to prepare your response. A > couple more questions: Initially that poster had asked about some private support contract matters, not generally applicable for discussion here, and which were answered privately. That person's follow on, however, raised matters of general interest, and so I 'surface' it here, with elidement of personally identifying information > * In skimming your documentation (which may be dated) it > didn't seem that Trading Shim parsed the IB tick (well their > equivalent of tick data) data but that was left up to the > individual programmer using it. Is that correct or did I > just get that wrong? We are very focused with the shim to provide as transparent an interpreter as possible, from the binary and ornate command language and state machine that the TWS represents, and into something textually clean and able to be manipulated as a well formed Unix tool feeding a stream. We also solve the two uses of NUL as both a end of line ("EOL") separator, and intra-field separator, which causes others a lot of pain, making the EOL a NEWLINE The documentation is accurate, in this regard, but finding the answer needs a Rosetta stone, and we handle that in the mailing list for the project presently. A manual write / re-write is on the docket, but we have some features to attain first, that make it a bit premature to spend time documenting items that were not yet 'nailed down' as to their final implementation details. The description of the command language in the online material, and the discussion of the configuration file options stand out as sore thumbs that do reflect that we turned out to not anticipate how they would finally end up; that said, most of it remains accurate Turning to consider ticks, you have noted that IB provides 'summarized' ticks: Each tick element type is distinctly marked by IB, and thence the shim, and has been from the first external deliverable release -- The 'marking' is in the 'pipe' ["|" delimited] field values, which pass through the IB message values While these values are potentially changeable by IB, and thus our outreference to their documentation, http://www.trading-shim.org/doc/node55.html it turns out that IB has been extremely good about NOT changing the mappings Bill had a post about reading history this week, but the same principles apply to ticks http://www.trading-shim.org/pipermail/ts-general/2010-March/000715.html and his example here was apple ("AAPL") in the 190s: 1| 1| 1?3.4| 3|1|price.outcry.bid. |STK:SMART:AAPL: 1| 2| 1?3.6| 2|1|price.outcry.ask. |STK:SMART:AAPL: 1| 4| 1?3.6| 1|0|price.summary.last. |STK:SMART:AAPL: 1| 0| 3|0|size.bid. |STK:SMART:AAPL: 1| 3| 2|0|size.ask. |STK:SMART:AAPL: 1| 5| 1|0|size.last. |STK:SMART:AAPL: 1| 8| 1?9494|0|size.volume. |STK:SMART:AAPL: 1| 6| 1?5.0| 0|0|price.summary.high. |STK:SMART:AAPL: 1| 7| 1?2.9| 0|0|price.summary.low. |STK:SMART:AAPL: 1| 9| 1?2.8| 0|0|price.summary.close.|STK:SMART:AAPL: We see that the following tick detail are present both in a numeric and a matching (added by us, expanded) text form: 1| 0 size.bid. 1| 1 price.outcry.bid. 1| 2 price.outcry.ask. 1| 3 size.ask. 1| 4 price.summary.last. 1| 5 size.last. 1| 6 price.summary.low. 1| 7 price.summary.low. 1| 8 size.volume. (here, cumulative, for the session) 1| 9 price.summary.close. There are additional attributes added with later server and client versions, but those tick attributes shown are present for ALL versions of the TWS supported by IB presently > In other words, I want each element of > the tick stream mapped to a distinct data object (or > variable) that can then be manipulated directly. * nod * .. using the left column as a pointer to index to a data object name, you have just such a mapping trivially -- My personal downstream parser maintains just such a table (I use an associative array in the particular implementation I looked at, I see ... slower, but it has never gotten 'overrun' by the data stream, so there is not reason for me to refactor it yet) > It seemed like Trading Shim provided a data stream, but not > a parsed data stream ready for consumption by an analysis > module. The parsing is subtle to a person who has not had to 'muck' through binary traces -- NUL to 'pipe' or NEWLINE, as the context is implied > Is that part of the current functionality of Trading Shim or > is that "an exercise left to the reader"? the former, as noted > * In looking at the doctoral paper, it seems that one of the > project objectives was to create a multi-threaded, > concurrent downloading tool. I'll let Bill speak to his paper and the doctoral studies objectives, as its author. :) > Is that supported in the Trading Shim? The pronoun here referring to 'multi-threaded' concurrency, I think. The shim has never needed to add multiple threads to fully keep up with the content fed to it by the TWS, and has a 'select' based input model which makes it pretty unlikely that it would need to do so. See: man 2 select_tut for a far better discussion of the fine points of 'select(2)' than I could ever write, with worked example code > How are the multiple sessions spawned and > managed? (If this is documented, just point me to the > appropriate materials). How are the different sessions > managed in such a way to keep the sequence of the data > stream in the proper event sequence? Is there a means of > buffering (writing out to temp files possibly) the results > until the backend database can write out the data stream? I will give a high level walk-through, and hope that I do not oversimplify the complexity of what is happening. Particularly, there is an amazing amount of detail checking and precise actions building data structures, and validating inputs happening 'under the hood' in the shim which I almost wholly gloss over. The details are out of scope here, but stated in the common form: the shim is a racehorse, built for speed and extremely fast throughout, consistent with accuracy I sample and track the competition, and have no doubt we 'scream' through data that others wallow and waddle about in http://www.trading-shim.org/faq/?other-voices The first question uses the term 'multiple sessions' in a context that I think means 'multiple connections to a single upstream TWS'. The TWS provides for up to 8 concurrent socket connections from downstream clients (and seemingly one or more additional but non-socket based connections to support the Java AWT GUI displays. The IB .jar providing that functionality is obfuscated and I have not tried to look) One usage model we think sensible for the shim is to have a single 'trading mode' connection, able to enter and supervise orders, and as many 'data mode' connections up to the limit of 7 remaining (== 8 minus 1), as one wishes. It turns out that one can do all 'data mode' operations in 'trading ['risk'] mode' so as few as one socket connection to the TWS by the shim can suffice for both 'risk' and 'data' operations, at the risk of adding latencies, discussed later herein. These are simple 'nix processes, started from the command line; triggered by clicking a GUI wrapper to fire off a command; spawned as a sub-task from an already running parent; or living in the inittab. Whatever. At one point we had the shim firing off a charting data refresh and display thread, but I think that turned out to be a dead end and that code removed. What I am saying is that there is no magic to starting a process to connect to the TWS socket interface, whether it is the shim, or otherwise. The SHIM in its start up process reads an optionally present configuration file, and then reads the standard input for optional additional configuration details in a 'let' command, recently described on the mailing list in a post by Bill. There are also 'hard-coded' configuration values in the source code, which may or may not have been over-ridden by values from the config file or the 'let' command. Thereafter in the usual case, the shim connects to the database, and to the TWS as the net effective configuration settings indicate, and resumes processing input as it appears on the standard in, and talking back and forth to the TWS across the socket interface. It is well possible to graft additional data and trade and control stream connectors onto the shim's current complement, but we have not encountered a compelling use case able to released in the GPLv3 code, to expose such an interface here. Talk to me privately for a quote if you think you need such The next question is: "How are the different sessions managed in such a way to keep the sequence of the data stream in the proper event sequence" and I think a quick review of assumptions is in order. There is a whole collection of 'unknowable unknown' latencies in play as IB marshalls data to deliver to the TWS on the TWS' upstream side. A trivial example: Are ticks for a given instrument being received through multicast by IB, or across a low speed serial TTY link? We can speculate as to some common instruments, but if a stock is essentially traded OTC or on a bid and ask 'quotation' basis, the tick data may be quite stale -- I watched SCOX one morning a few years ago, and there was not a single tick or trade until after 10:45 -- it looked like the sole market maker just overslept ;) What are the data transmission latencies TO IB's plants; what are the data cleanup and transfer latencies inside and between IB's plants? What latency does the particular IB load balancer one is connected to add? Does it differ between .us, ,.ch and .hk? Almost certainly. Look at the 'data farm' up and down messages through a trading day. They come, they go, and one may not be ABLE to even detect an outage for a quarter-minute or two Once delivered to the local TWS, across a data link of variable latency, if there is only a single connection, pulling a big chunk of history, I know that any transaction detail that came in, and all tickstream data, will be delayed until that history transfer is completed, because of the way the TWS message that is carrying history detail is structured [This is part of the reason I mentioned the shim's ability to running a 'data' connection, and in another separate shim, one as the 'risk' connection -- one can then structure and classify one's apparent latencies are expressed. But think this through -- there is also only a single encrypt-able FIX connection between the TWS and its IB counter-party servers. What roadblock did it add? We cannot see, and so cannot know but rather only speculate] A particular symbol's tick datastream ** seems ** to be sequentially delivered, when I have compare recorded tick streams against one second historical data, but I know of no guarantees by IB in this regard. Once down at the TWS, each session between it and a shim instance are TCP connections, and as such, natively serialized. Then the shim processes its 'select' data basically in a round-robin fashion, and enqueues it for (writing to a file, sending to the log, sending to standard out, or a combination thereof) and there are minimal delays (usually a maximum of 20 mSec, usually sub micro-Sec, if one is on the right part of the processing cycle) before it it is done being handed off to the host operating system's routines. ... again latencies of appearance which I cannot control and of which do not speak, as they will vary A trading strategy that requires sub 20 mSec response CANNOT be run across data links that take over 2 times that to transfer data to and from the counter-party [IB's next hop is at least that far away from anyone not co-located at a data-center with them], and it is a fools game to pretend otherwise. Probably a trading strategy that HAS to see EVERY TICK should consider a platform other than IB, as IB does not purport to deliver such, not to deliver without time skew between symbol samples The last question about 'holding back' sending messages along until the database is coherent at a given state is not something we do, nor that I see that we would ever want to do. Do you have a use case please? > * Has your team done any work with NVidia's CUDA platform? > If not, is that on your roadmap (to leverage GPU's for > massaging datastreams) with some type of target date? The 'CUDA' numeric co-processor parallelization is more applicable downstream of the shim's functions {very high speed simulation and solving of projected trading surface topology}. The shim is so fast that the book-keeping overhead of farming out, and synchronizing returns would be wasted. It would be bolting a battery of jet engines onto a barracuda ... not really a good fit ;) I do not discount 'CUDA' at all -- it is just not needed at the layer the shim is working at > * I should have stated that I'm using Ubuntu 64 > Desktop/Server editions as to access the larger memory that > may be necessary as more data is retained in memory to avoid > backend database IO issues. Does your code leverage a 64bit > platform? We develop on 64 bit, and encountered an anomaly with an older Debian Stable compiler on a 32 bit platform recently. I cannot presently recreate the anomaly after a recent re-factoring. Our peak memory uses far below the limits a 32 bit 'nix architecture imposes, but yes, we can use a 64 bit arch quite well ;) > * The Freshmeat site has some broken links - you may want to > check it or have their webmaster fix them We do not have access to alter what appears at freshmeat beyond using their web interface. Please send me a private note of the outages, and I will pass it along to their admins. I know they have been 'refreshing' their site recently, and regret that they have removed some detail I enjoyed reading > * You mentioned exploratory project estimates. At this > point, I'm just looking for IB to provide data. Since the > historical data lacks certain data elements, I'm looking at > using the "tick" feed as the source for all data. This > would require streaming out the tick data to a database > while also using to feed real-time analytics. Does the > "off-the-shelf" Trading Shim provide this or would this be > considered custom or add-on work to achieve this? This database write function is doable natively, and has been 'forever' -- it would be an example in the 'open loop' (no closed feedback, back into the shim's standard in) client model. To implement it, one hooks a simple state machine filter written in the scripting language of your choice to accept its standard in through a pipe on the standard out of the shim, and have that downstream client emit database transactions to taste; additionally you can have the filter (in its state machine aspect) fire off (say through a 'curl' POST) a control event to signal a co-process to wake up and refresh analytics only when new data is present (compared to say, refreshing 'regardless' every 5 seconds) Alternatively one can hook a 'reader' to 'tail' the syslog file which the shim has been set up to write to, and post process there in the same fashion Alternatively one can hook a 'reader' to 'tail' the output file which the shim has been set up to write to, and post process there in the same fashion The shim is designed to be a well-behaved 'nix tool ;) -- Russ herrold From pippin at owlriver.net Tue Mar 30 20:07:55 2010 From: pippin at owlriver.net (Bill Pippin) Date: Tue, 30 Mar 2010 20:07:55 -0400 Subject: [ts-gen] bug fix for debian lenny and other systems with gcc 4.3.2 Message-ID: Today's release fixes a bug in the shim that impacts users with debian lenny, and other systems where g++ is still back at version 4.3.2. The bug does not show up with newer gcc, which is in part how the flawed code was released originally. The *next* release will include a dbms version increment, and non-backward compatable changes to journalling, so you probably want to download this tarball to give yourself an interim checkpoint. The fix occurred as part of a thorough refactoring of the hash table component template class, and the remainder of this post after the next paragraph is about that refactoring. For those using debian lenny and the like, please keep in mind that those details may not concern you; the actual fix was only a minor part of the refactoring. The bug was in two parts; the hash table component did not adequately filter client hints about the desired starting size for the table, and a client constructive call used just such an improperly small starting size. The code used to work because older client code did have a valid size request, and the failure to check in the component was not observed. At some point I changed the client code, though only after a newer version of gcc included some change that allowed the component code to work. As part of the hash table refactoring, that table now uses the same handle component other block-doubling containers use, and that handle class includes a filter step for requested start sizes, so this bug can not re-occur without rapid and dramatic symptoms, i.e., a crash locally on the first try, so, it won't be able to escape again. Debian lenny users may stop reading if they so desire, and, now, from the NEWS file: * Refactor hash map to use the same block-doubled handle objects as other containers use ... There are other significant internal changes to the hash map class, though none other fix known bugs as with the thinko mentioned above. ... Unusually large collision sets spill to a splay tree --- tests show that the max extent size of eight cells for 64 bit hardware rules out all but one such case --- and extents as well as the hash array are allocated by a container-specific allocator from the block handle. Note that for 32 bit hardware, where MaxExtent decreases to four cells, there is a non-trivial number of spills, e.g., in the high single digit percentages, to the splay tree. In other words, with respect to the last two paragraphs, if you are deeply concerned about performance, consider using a 64 bit as opposed to 32 bit system. I've made performance tests of the new code, and given the number of symbols in the database, and the relatively small number of large-collision-set elements spilled to a tree, there is absolutely no measureable performance impact. That being said, if the number of symbols in the database was to increase greatly, as with options chains, there probably would be some small performance impact for a 32 bit architecture. I say this since when I completely replace the hash table with a splay tree, with the same effect as if all elements spilled to the tree, there is a small but measureable latency cost in event processing, on the order of a microsecond for each event. For any given event this is unimportant, given much larger IO delays outside the control of the shim, yet over several thousand events, as with market depth in a fast market, it would add up. Note also that there may be other, much stronger performance-related reasons why you should prefer a 64 bit system. In testing locally I see a big difference, though there are many factors besides word size, so that the differences are apples-and-oranges comparisons. Perhaps a better way to state this point is that relatively new hardware will give you better performance than some old clunker, as you would expect, and that memory architecture is probably the key issue, so that you want a modern processor with large caches. Not a surprise. Thanks, Bill From gene_livingston at yahoo.com Wed Mar 31 13:43:55 2010 From: gene_livingston at yahoo.com (gene livingston) Date: Wed, 31 Mar 2010 10:43:55 -0700 (PDT) Subject: [ts-gen] segfault Message-ID: <68626.98917.qm@web50905.mail.re2.yahoo.com> Introduction: Hi, I've just joined the list, and am looking into using your Trading Shim. Your tastes seem to match closely with mine (Linux & C++) so I am very happy to have found this project. I am hoping to use the shim for 1) gathering historical data, mostly futures 2) gathering end of day data 3) trading history,open trades, portfolio mgmt. Not doing much intra-day. Will be using this data to write risk management software in C++ for Linux for my use (Van K Tharp's "Guide to Position Sizing", and Grant's "Trading Risk" are my main references for this project). Issue: I am receiving a segfault when running the shim with the --data option. I have run the install script in /sql and TWS is running(using the paper account, API socket clients enabled for localhost), and have verified that the username works using MySQL query browser. I have tested the TWS API connection using the Java sample app. Thank you very much in advance for any assistance. I did receive several warnings during compilation, no errors, and a binary was built. My specs are: Ubuntu 9.10(Karmic)/x86 g++ 4.4.1 ruby 1.8.7 Sun Java 6 MySQL Server 5.1.37 **I will admit to having one known wrong configuration, but that's related to mysql, so I doubt that would generate a segfault. In 'my.cnf' I did not set "transaction-isolation = SERIALIZABLE" yet as I am having trouble getting mysql to recognize that option. Will have a look at the .diff to see what I'm doing wrong there. output of "make shim": genel at lopburi:~/Downloads/shim/shim-100330$ make shim g++ -Wall -g -I/usr/include/mysql -c -o obj/main.o src/main.c g++ -Wall -g -I/usr/include/mysql -c -o obj/root.o src/root.c g++ -Wall -g -I/usr/include/mysql -c -o obj/once.o src/once.c g++ -Wall -g -I/usr/include/mysql -c -o obj/self.o src/self.c g++ -Wall -g -I/usr/include/mysql -c -o obj/mode.o src/mode.c src/mode.c: In member function ?virtual void TShimInterpreter::DomainSingletons::Risk::check_api(const TShimInterpreter::LanguageConstant::Exceptions&, MultipleParadigm::nat, bool) const?: src/mode.c:241: warning: format not a string literal and no format arguments g++ -Wall -g -I/usr/include/mysql -c -o obj/help.o src/help.c g++ -Wall -g -I/usr/include/mysql -c -o obj/data.o src/data.c g++ -Wall -g -I/usr/include/mysql -c -o obj/tags.o src/tags.c g++ -Wall -g -I/usr/include/mysql -c -o obj/lets.o src/lets.c g++ -Wall -g -I/usr/include/mysql -c -o obj/fsms.o src/fsms.c g++ -Wall -g -I/usr/include/mysql -c -o obj/type.o src/type.c g++ -Wall -g -I/usr/include/mysql -c -o obj/sets.o src/sets.c g++ -Wall -g -I/usr/include/mysql -c -o obj/rule.o src/rule.c g++ -Wall -g -I/usr/include/mysql -c -o obj/apis.o src/apis.c g++ -Wall -g -I/usr/include/mysql -c -o obj/cusp.o src/cusp.c src/cusp.c: In constructor ?TShimInterpreter::LanguageConstant::NatValue::NatValue(const TShimInterpreter::AppFiniteDomains::ParmName&, const char*, MultipleParadigm::nat)?: src/cusp.c:214: warning: format ?%lu? expects type ?long unsigned int*?, but argument 3 has type ?MultipleParadigm::nat*? g++ -Wall -g -I/usr/include/mysql -c -o obj/vars.o src/vars.c g++ -Wall -g -I/usr/include/mysql -c -o obj/freq.o src/freq.c src/freq.c: In member function ?MultipleParadigm::Long TShimInterpreter::DomainSingletons::TscClock::cpu_info() const?: src/freq.c:183: warning: format not a string literal and no format arguments src/freq.c: In member function ?MultipleParadigm::Long TShimInterpreter::DomainSingletons::TscClock::cpuinfo_hertz() const?: src/freq.c:202: warning: format not a string literal and no format arguments g++ -Wall -g -I/usr/include/mysql -c -o obj/bind.o src/bind.c g++ -Wall -g -I/usr/include/mysql -c -o obj/init.o src/init.c g++ -Wall -g -I/usr/include/mysql -c -o obj/dbms.o src/dbms.c g++ -Wall -g -I/usr/include/mysql -c -o obj/open.o src/open.c g++ -Wall -g -I/usr/include/mysql -c -o obj/late.o src/late.c g++ -Wall -g -I/usr/include/mysql -c -o obj/file.o src/file.c g++ -Wall -g -I/usr/include/mysql -c -o obj/fail.o src/fail.c g++ -Wall -g -I/usr/include/mysql -c -o obj/talk.o src/talk.c src/talk.c: In member function ?const TShimInterpreter::TuplesAttributes::Natural* TShimInterpreter::DomainSingletons::IoBase::talk(const TShimInterpreter::EnvironmentValue::CuspEnv&, const TShimInterpreter::LanguageConstant::VersionTuple&, const TShimInterpreter::TuplesAttributes::Compound&, const TShimInterpreter::TuplesAttributes::Natural&, const TShimInterpreter::TuplesAttributes::MessageAtoms::NextId&) const?: src/talk.c:111: warning: format not a string literal and no format arguments g++ -Wall -g -I/usr/include/mysql -c -o obj/null.o src/null.c g++ -Wall -g -I/usr/include/mysql -c -o obj/shim.o src/shim.c g++ -Wall -g -I/usr/include/mysql -c -o obj/fill.o src/fill.c g++ -Wall -g -I/usr/include/mysql -c -o obj/wait.o src/wait.c g++ -Wall -g -I/usr/include/mysql -c -o obj/read.o src/read.c g++ -Wall -g -I/usr/include/mysql -c -o obj/ctor.o src/ctor.c g++ -Wall -g -I/usr/include/mysql -c -o obj/proc.o src/proc.c g++ -Wall -g -I/usr/include/mysql -c -o obj/nque.o src/nque.c g++ -Wall -g -I/usr/include/mysql -c -o obj/send.o src/send.c g++ -Wall -g -I/usr/include/mysql -c -o obj/time.o src/time.c g++ -Wall -g -I/usr/include/mysql -c -o obj/tick.o src/tick.c g++ -Wall -g -I/usr/include/mysql -c -o obj/wire.o src/wire.c g++ -Wall -g -I/usr/include/mysql -c -o obj/post.o src/post.c g++ -Wall -g -I/usr/include/mysql -c -o obj/past.o src/past.c g++ -Wall -g -I/usr/include/mysql -c -o obj/echo.o src/echo.c src/echo.c: In member function ?void TShimInterpreter::DomainSingletons::Logger::log(const MultipleParadigm::ArraysVectorsAndTables::Buffer&, MultipleParadigm::nat) const?: src/echo.c:68: warning: format not a string literal and no format arguments g++ -Wall -g -I/usr/include/mysql -c -o obj/form.o src/form.c g++ -Wall -g -I/usr/include/mysql -c -o obj/term.o src/term.c g++ -Wall -g -I/usr/include/mysql -c -o obj/acts.o src/acts.c g++ -Wall -g -I/usr/include/mysql -c -o obj/name.o src/name.c g++ -Wall -g -I/usr/include/mysql -c -o obj/leaf.o src/leaf.c g++ -Wall -g -I/usr/include/mysql -c -o obj/base.o src/base.c g++ -Wall -g -I/usr/include/mysql -c -o obj/envs.o src/envs.c g++ -Wall -g -I/usr/include/mysql -c -o obj/else.o src/else.c src/else.c: In member function ?const TShimInterpreter::AppFiniteDomains::Permanent& TShimInterpreter::DomainSingletons::AutoEnum::mysql_connection_failure() const?: src/else.c:256: warning: format not a string literal and no format arguments src/else.c: In member function ?const TShimInterpreter::AppFiniteDomains::Permanent& TShimInterpreter::DatabaseRelation::SdbVersion::shim_and_db_ver_mismatch(const TShimInterpreter::EnvironmentValue::BaseEnv&) const?: src/else.c:390: warning: format not a string literal and no format arguments src/else.c: In member function ?double TShimInterpreter::TuplesAttributes::CommandAtoms::Wait::arg_term_typecheck_error(const TShimInterpreter::LanguageConstant::Exceptions&) const?: src/else.c:484: warning: format not a string literal and no format arguments src/else.c: In member function ?void TShimInterpreter::EnvironmentValue::Cursor::mismatch(const TShimInterpreter::LanguageConstant::Compound&, bool, bool) const?: src/else.c:675: warning: format not a string literal and no format arguments src/else.c:676: warning: format not a string literal and no format arguments g++ -Wall -g -I/usr/include/mysql -c -o obj/scan.o lib/scan.c g++ -Wall -g -I/usr/include/mysql -c -o obj/hash.o lib/hash.c g++ -Wall -g -I/usr/include/mysql -c -o obj/text.o lib/text.c lib/text.c: In member function ?const MultipleParadigm::ArraysVectorsAndTables::Buffer& MultipleParadigm::ArraysVectorsAndTables::Buffer::format(const char*, MultipleParadigm::nat, char*, MultipleParadigm::nat) const?: lib/text.c:21: warning: format ?%lu? expects type ?long unsigned int?, but argument 4 has type ?MultipleParadigm::nat? lib/text.c:21: warning: format ?%lu? expects type ?long unsigned int?, but argument 4 has type ?MultipleParadigm::nat? g++ -Wall -g -I/usr/include/mysql -c -o obj/wrap.o lib/wrap.c g++ -Wall -g -I/usr/include/mysql -c -o obj/pool.o lib/pool.c lib/pool.c: In member function ?void* MultipleParadigm::HashAndTreeMaps::BinsPool::alloc(MultipleParadigm::nat) const?: lib/pool.c:409: warning: format ?%6lu? expects type ?long unsigned int?, but argument 3 has type ?MultipleParadigm::nat? lib/pool.c:409: warning: format ?%2lu? expects type ?long unsigned int?, but argument 4 has type ?unsigned int? lib/pool.c:409: warning: format ?%6lu? expects type ?long unsigned int?, but argument 5 has type ?MultipleParadigm::nat? g++ -Wall -g -I/usr/include/mysql -c -o obj/buds.o lib/buds.c lib/buds.c: In member function ?const char* MultipleParadigm::SimpleEfficientAllocator::Map::memory_map_failure() const?: lib/buds.c:446: warning: format not a string literal and no format arguments g++ -Wall -g -I/usr/include/mysql -c -o obj/inet.o lib/inet.c g++ -Wall -g -I/usr/include/mysql -c -o obj/boot.o lib/boot.c g++ -Wall -g -I/usr/include/mysql -c -o obj/math.o lib/math.c g++ -Wall -g -I/usr/include/mysql -c -o obj/dfsa.o lib/dfsa.c (cd obj; g++ -g -o ../shim main.o root.o once.o self.o mode.o help.o data.o tags.o lets.o fsms.o type.o sets.o rule.o apis.o cusp.o vars.o freq.o bind.o init.o dbms.o open.o late.o file.o fail.o talk.o null.o shim.o fill.o wait.o read.o ctor.o proc.o nque.o send.o time.o tick.o wire.o post.o past.o echo.o form.o term.o acts.o name.o leaf.o base.o envs.o else.o scan.o hash.o text.o wrap.o pool.o buds.o inet.o boot.o math.o dfsa.o -L/usr/lib/mysql -lmysqlclient -lm) genel at lopburi:~/Downloads/shim/shim-100330$ genel at lopburi:~/Downloads/shim/shim-100330$ ./shim Usage: shim [optional feature list] Modes: # real modes, requiring access to an IB tws: --data # process subscriptions and log resulting tick stream events --risk # accept full command set, send requests, and log all events genel at lopburi:~/Downloads/shim/shim-100330$ ./shim --data Segmentation fault genel at lopburi:~/Downloads/shim/shim-100330$ From bbands at gmail.com Wed Mar 31 14:06:04 2010 From: bbands at gmail.com (BBands) Date: Wed, 31 Mar 2010 11:06:04 -0700 Subject: [ts-gen] segfault In-Reply-To: <68626.98917.qm@web50905.mail.re2.yahoo.com> References: <68626.98917.qm@web50905.mail.re2.yahoo.com> Message-ID: On Wed, Mar 31, 2010 at 10:43 AM, gene livingston wrote: > Introduction: > > Hi, I've just joined the list, and am looking into using your Trading Shim. Your tastes seem to match closely with mine (Linux & C++) so I am very happy to have found this project. I am hoping to use the shim for 1) gathering historical data, mostly futures 2) gathering end of day data 3) trading history,open trades, portfolio mgmt. Not doing much intra-day. Will be using this data to write risk management software in C++ for Linux for my use (Van K Tharp's "Guide to Position Sizing", and Grant's "Trading Risk" are my main references for this project). Hold the phone! Here are a few more references for you. The first is a very nicely written survey of the landscape, the next two are the meat. http://www.amazon.com/Fortunes-Formula-Scientific-Betting-Casinos/dp/0809045990/ http://www.amazon.com/Handbook-Portfolio-Mathematics-Formulas-Allocation/dp/0471757683 http://www.amazon.com/Leverage-Space-Trading-Model-Reconciling/dp/0470455950 So, while Bill and Russ work out your problem, you have something to read. Best, John -- John Bollinger, CFA, CMT www.BollingerBands.com If you advance far enough, you arrive at the beginning. From pippin at owlriver.net Wed Mar 31 21:47:20 2010 From: pippin at owlriver.net (Bill Pippin) Date: Wed, 31 Mar 2010 21:47:20 -0400 Subject: [ts-gen] segfault Message-ID: Gene, Welcome; we're delighted to have you. > Hi, I've just joined the list, and am looking into using your Trading > Shim. Your tastes seem to match closely with mine (Linux & C++) so I > am very happy to have found this project. So you have 32 bit hardware, and a reasonably up-to-date g++ : > Ubuntu 9.10(Karmic)/x86 > g++ 4.4.1 > ruby 1.8.7 > Sun Java 6 > MySQL Server 5.1.37 About setup requirements: > ... I did not set "transaction-isolation = SERIALIZABLE" yet ... You're correct that this would not cause a segfault. Instead, you should see a straight-forward error message, followed by termination. The error text would be as follows: Problem 512: mySQL isolation level must be serial Exiting. So, focusing on the crash you are seeing: > I am receiving a segfault when running the shim ... ... > genel at lopburi:~/Downloads/shim/shim-100330$ ./shim --data > Segmentation fault Would you please be so kind as to start up gdb, run the shim, capture a stack trace of where the shim is when it crashes, and post that to the list? To obtain the stack trace, start gdb with the file name of the shim executable as an argument, type "r --data" to run, and "w" (where) to print the stack trace once the shim has crashed. Thanks, Bill From pippin at owlriver.net Wed Mar 31 22:13:37 2010 From: pippin at owlriver.net (Bill Pippin) Date: Wed, 31 Mar 2010 22:13:37 -0400 Subject: [ts-gen] segfault Message-ID: Gene, The stack trace is not necessary; I now have a local reproducer, and I'll be looking into the problem more tomorrow. FYI, the symptoms of the problem seem to be the same as seen earlier this month for an offlist user. The code I pushed last night was explicitly meant to fix that problem, and appeared to do so at the time. I'll keep you posted. Thanks, Bill