[ts-gen] Diagnostic session transcript
R P Herrold
herrold at owlriver.com
Thu May 17 12:33:34 EDT 2007
I had occasion to do ome diagnostic work for a list member one
on one rcently. His environment was: Debian (Ubuntu),
I have interspersed *** N *** markers, so I can refer to
points where interesting things to talk about have occurred
item 1: Note the ERROR, and 'your_dbms_host' -- As Bill
mentioned in his ;reading the code recently, lets use 'egrep'
and read the code:
[herrold at centos-4 shim]$ tar zxf shim-070228.tgz
[herrold at centos-4 shim]$ cd shim-070228
[herrold at centos-4 shim-070228]$ egrep -R 'your_dbms_host' *
bin/includes:HostName=your_dbms_host
[herrold at centos-4 shim-070228]$
This tells us where the value came from.
item 2, 3, 4: During the session, as I was working from
memory as to where substitutions existed, I had to go look a
bit. day to day, doing development and testing, I use a
patching script written many months ago and incrementally
updated when things change, to put in local values
automatically; so I do not remember sometimes ;)
item 5: ... and now it works; very quick, very easy.
item 6: Just observing that the shim is reporting in the
local TZ ...
item 7: We use grep again to read the sources
item 8: Here, I was concerned, as we have repeatedly noted in
the documentation, that one MUST test on a paper accout to
avoid burning real money ;) The script: bin/unsafe does an
entry and immediate exit, and is intentionally NOT in 'make
test'
item 9: pastebins are great tools for remote debugging
item 10: This is a puzzler, and we had not previously had a
test to be dure that database writes are working; we have
been reworking the setup instructions to address assigning
'least privilege' to the mysql database userid the shim uses,
and we are investigating why it works on CentOS, but not
Debian presently.
item 11. Reflect that I have not wrapped the shim up into
.deb (nor .rpm) packaging yet, so that 'dpkg' could protect me
from dependencies disappearing silently -- if I had, I would
have been warned with a mesage when I ran:
'apt-get upgrade'
... of course, I see in my back history on this box, with the
recent Debian release:
apt-get -y --force-yes --ignore-hold dist-upgrade
Not much could protect me there, when I am telling the package
management front end that I know what I am doing, and know how
to repair the damage. Fortunately, I recognized it readily,
and a fresh conpile repaired the damage ;)
item 12. shim-070228 was the last external release in a while
before the recent (silent) resumption of releases [known not
to yet recover Debian function]. We are aware that Debian is
not working correctly post 070228, but we wanted to get the
rather nice documnetation in process 'out there.' for some
other people asking private questions.
PLEASE -- use the mailing list, so that all may benefit from
your questions, and our answers -- we are quite aware of the
need to keep the mailing list turnaround prompt and carefully
written. Google is indexing it, and we have a search box at
the site.
As I say, Bill has written rather nice preliminary
documentation at a high level, which appears in HTML at:
http://www.trading-shim.org/doc/
and in PDF form at:
http://www.trading-shim.org/pdfs/
item 13. covered at item 11
item 14. covered at item 12
And that pretty well wraps up my comments on this diagnostic
session. Remember to write.
-- Russ Herrold
====================================================================
Transcript of an IRC support session on 9 May 2007, commencing at 16:16
US Eastern time (GMT 20:16)
15:54 =kiwifx> ping
16:00 =orc_orc> pong
16:00 =orc_orc> hi there
16:04 =kiwifx> hi
16:04 =kiwifx> good afternoon
16:05 =orc_orc> or morning as the case may admit
16:05 =kiwifx> indeed
... after some 'getting acquainted' preliminaries, the support
session commenced:
16:16 =orc_orc> to your problem getting the shim going, which distro
are you using?
16:17 =kiwifx> linux?
16:17 =orc_orc> yes
16:17 =kiwifx> ubuntu
16:17 =orc_orc> kk
16:17 =kiwifx> dapper drake
16:17 =orc_orc> so Debian
16:17 =kiwifx> yes
16:17 =kiwifx> works fine
16:17 =orc_orc> and you have a mysql server running and can compile
the shim, right?
16:17 =kiwifx> and connect to IB and get connection strings etc.
16:17 =orc_orc> kk
16:18 =orc_orc> so then run: make test
and tell me what it shows or paste at www.pastebin.ca, and
let me know the URL to look at
16:19 =orc_orc> your connection is actually to the local TWS instance
of course, and the TWS then talks, through a FIX variant upstream
and offsite from you
16:20 =kiwifx> make[1]: `shim' is up to date.
16:20 =kiwifx> make[1]: Leaving directory `/usr/local/shim-070228'
16:20 =kiwifx> bin/regress
16:20 =kiwifx> ERROR 2005 (HY000): Unknown MySQL server host
'your_dbms_host' (1)
*** 1 ***
16:20 =kiwifx> test: set tws to Detail
16:20 =kiwifx> test: get account data
16:20 =kiwifx> The trading shim has finished program initialization,
including the
16:20 =kiwifx> construction of successful connections to the database
and IB tws.
16:20 =kiwifx> test: YM history query
16:20 =kiwifx> test: AIG arket depth
16:20 =kiwifx> test: YM market depth
16:20 =kiwifx> test: no market depth
16:20 =kiwifx> make: *** [test] Broken pipe
16:20 =orc_orc> Unknown MySQL server host 'your_dbms_host'
16:20 =kiwifx> i know, but that's wrong, as it does connect
*** 2 ***
16:20 =orc_orc> that means you did not patch data.c or the
includes in bin/includes
16:20 =orc_orc> two [dbms] connections in play here
16:20 =kiwifx> oh
16:21 =kiwifx> i patched data.c
16:21 =orc_orc> the shim which works, and the scripts which need the edit
16:21 =orc_orc> yes - that is it all right
16:21 =kiwifx> i see
16:21 =orc_orc> the shell scripts talk to the database as well
16:21 =kiwifx> do i edit regress?
16:21 =orc_orc> bin/includes as I recall -- lemme look
*** 3 ***
16:22 =orc_orc> I patch all on an automated basis with each build so
I forget
16:22 =kiwifx> got it
16:22 =orc_orc> UserName=rph_shim
16:22 =orc_orc> HostName=xps400.first.lan
16:22 =orc_orc> DbmsName=rph_testing
16:23 =orc_orc> that section
*** 4 ***
16:23 =orc_orc> my initials are rph, and that lets me have a namespace
that Bill does not touch, and vice versa
16:24 =orc_orc> xps400 is a Dell xps400, not surprisingly
16:24 =kiwifx> nice
16:24 =kiwifx> ok i'll run again
16:25 =kiwifx> bin/regress
16:25 =kiwifx> test: set tws to Detail
16:25 =kiwifx> The trading shim has finished program initialization,
including the
16:25 =kiwifx> construction of successful connections to the database
and IB tws.
16:25 =kiwifx> test: get account data
16:25 =kiwifx> test: YM history query
16:25 =kiwifx> test: AIG arket depth
16:25 =kiwifx> test: YM market depth
16:25 =kiwifx> test: no market depth
16:25 =kiwifx> test: get subscriptions
16:25 =kiwifx> #eg_echo
*** 5 ***
16:25 =orc_orc> ok -- so, as root: tail -400 /var/log/messages
16:25 =kiwifx> root at shim:/var/log# tail -f /var/log/syslog
16:25 =kiwifx> May 10 09:00:59 shim : shim|data|0.28|13892|32457|
9063042|3| 1| 5| 157|TXN|STK||0.00||1|SMART||USD||
16:25 =kiwifx> May 10 09:01:00 shim : shim|data|0.28|13892|32457|
9063054|3| 1| 5| 158|TYC|STK||0.00||1|SMART||USD||
16:25 =kiwifx> May 10 09:01:00 shim : shim|data|0.28|13892|32457|
9063113|3| 1| 5| 159|UTX|STK||0.00||1|SMART||USD||
16:25 =kiwifx> May 10 09:01:00 shim : shim|data|0.28|13892|32457|
9063128|3| 1| 5| 161|VRSN|STK||0.00||1|SMART||USD||
16:25 =kiwifx> May 10 09:01:00 shim : shim|data|0.28|13892|32457|
9063140|3| 1| 5| 163|WFMI|STK||0.00||1|SMART||USD||
16:25 =orc_orc> yayyyyy
16:25 =kiwifx> May 10 09:01:00 shim : shim|data|0.28|13892|32457|
9063151|3| 1| 5| 165|WMT|STK||0.00||1|SMART||USD||
16:25 =kiwifx> May 10 09:01:00 shim : shim|data|0.28|13892|32457|
9063162|3| 1| 5| 166|WYNN|STK||0.00||1|SMART||USD||
16:25 =kiwifx> May 10 09:01:00 shim : shim|data|0.28|13892|32457|
9063174|3| 1| 5| 171|TRIN-NYSE|IND||0.00||1|SMART||USD||
16:26 =kiwifx> May 10 09:01:00 shim : shim|data|0.28|13892|32457|
9063185|3| 1| 5| 177|YM|FUT|200703|0.00||1|ECBOT||USD||
16:26 =kiwifx> May 10 09:01:00 shim : shim|data|0.28|13892|32459|
11011472|2| 7| 0|quit;|
16:26 =kiwifx> so, that was the issue.
16:26 =orc_orc> yes indeed
[Note: the local Time Zone for kiwifx is 'across' the International date
line from New York, and into 'tommorrow. As such he has the last 'tick'
data of the day, from 16:01 NY time on the 'prior' day, which ended 24
minutes before his paste; the 'YM' would still have been open for trading,
and I am uncertain as to why it has a value at his '09:01:00']
*** 6 ***
16:26 =kiwifx> now, how to get that data into the database?
16:26 =orc_orc> it already is
16:26 =orc_orc> we [that is, the shim] inserted it for you
16:26 =kiwifx> which table?
16:27 * orc_orc looks
16:27 =orc_orc> what I am doing is: cd src, and grep for insert
16:28 =orc_orc> post.c:char_0 head("insert into HistoryBar\n"
16:28 =orc_orc> post.c:char_0 text("event: history insert");
16:28 =orc_orc> looks like HistoryBar
*** 7 ***
16:28 =kiwifx> empty
16:29 =orc_orc> yeah -- I'll look further -- other scripts do this for me
16:29 =kiwifx> k
16:31 =orc_orc> ok -- are you in a PAPER account???
16:31 =kiwifx> yes
16:31 =orc_orc> run: bin/unsafe
*** 8 ***
16:31 =orc_orc> then on the DB, run:
select * from HistoryBar limit 5 ;
16:32 =orc_orc> we move stuff in and out of the test scripts,
depending on where we are developing
16:32 =kiwifx> running now
16:33 =orc_orc> should look like this: http://www.pastebin.ca/479607
*** 9 ***
16:33 =kiwifx> The trading shim has finished program initialization,
including the
16:33 =kiwifx> construction of successful connections to the database
and IB tws.
16:33 =kiwifx> test: protected entry:
16:33 =kiwifx> test: limit exit order
16:33 =kiwifx> test: send transaction
16:33 =kiwifx> test: open orders info
16:33 =kiwifx> test: fire order entry
16:33 =kiwifx> test: reverse position
16:33 =kiwifx> test: YM history query
16:33 =kiwifx> test: get account data
16:33 =kiwifx> test: AIG arket depth
16:33 =kiwifx> test: YM market depth
16:33 =kiwifx> test: no market depth
16:33 =kiwifx> test: get subscriptions
16:33 =kiwifx> # run_risk ; sleep 2
16:33 =kiwifx> # wasteful ; sleep 1
16:33 =kiwifx> # end_shim
16:33 =kiwifx> # end_dbms
16:33 =kiwifx> # exit
16:33 =kiwifx> #eg_echo
16:34 =kiwifx> no
16:34 =orc_orc> 16:33 =kiwifx> test: YM history query
16:34 =kiwifx> still empty
16:34 =orc_orc> that should have popualted the DB
16:34 =orc_orc> there may be a perms issue, but I doubt it ... hmmm
16:34 =kiwifx> maybe
16:34 =kiwifx> no insert priv.
16:34 =kiwifx> hang on, i'll do an insert manually
16:34 =orc_orc> lemme get a test case for you -- one moment
16:35 =orc_orc> > select count(*) from HistoryBar ;
16:35 =orc_orc> does that return zero?
16:36 =kiwifx> well, it returns 1 now as I just did a test insert
using the same connection parameters
16:36 =kiwifx> permissions should be ok
16:36 =orc_orc> very curious -- I was going to do that same test insert
16:36 =kiwifx> great programmers ....
*** 10 ***
16:36 =orc_orc> let me pop over the Debian unit and repeat this test
16:37 =kiwifx> k
16:38 =orc_orc> hmmm -- debian has upgraded a SO away from me since
I built my test shim case there:
./shim: /usr/lib/libmysqlclient.so.14: version
`libmysqlclient_14' not found (required by ./shim)
16:38 =orc_orc> ... rebuilding
16:39 =kiwifx> k
*** 11 ***
16:39 =orc_orc> what is date of tarball you are using?
16:39 =orc_orc> kk
16:39 =kiwifx> 070228
16:40 =orc_orc> yeah -- that is a good one before we ripped a lot up
*** 12 ***
16:54 =orc_orc> ping
16:59 =kiwifx> ping
17:00 =orc_orc> hey
17:00 =orc_orc> have to run -- roof maintenance
17:00 =kiwifx> any joy?
17:00 =orc_orc> stuff just finished build -- one moment
17:01 =orc_orc> one moment -- I am patching debian unit manually
as our code has been restructured and I missed a patch
17:01 =kiwifx> ok
*** 13 ***
17:04 =kiwifx> ok Russ. Have a good evening. chat later.
17:04 =orc_orc> will advise on [Debian history] insert issue -- it has
worked here in the past -- will sort it
17:04 =orc_orc> good luck
17:04 =kiwifx> cheers
*** 14 ***
More information about the ts-general
mailing list