[ts-gen] Symbol data

[R P Herrold]

On Fri, 16 Apr 2010, gene livingston wrote:

> (1)
> Noticed that 'A0Stk08.sql' and 'A0Stk09.sql' are not included as source
> files in '/sql/load.sql' (shim-100409). Wondered if there was a
> reason for that. I added them into 'load.sql' and got no errs.

* nod * sort of -- The files are there, but in the 
'data-xxxx.tgz' file; I noticed on Wednesday evening that we 
had a missing (or as it turns out, incorrect) 'sym' symlink 
file to that in the scripts that build at the top of the 
'shim-xxx.tgz' unrolled tarball.

The solution temporarily is to simply make correctly the 
symlink, once the two tarballs are unrolled.

As a reminder -- it is in the archives, bue here is a quick 
refresher:

1.  Retrieve the shim tarball. Presently this would be:
 	cd a working directory
 	wget ftp://ftp.trading-shim.org/pub/local/trading-shim/shim-100416.tgz
2.  Unroll the shim tarball.  I use:
 	tar zxf shim-100416.tgz
3.  Retrieve the data tarball
 	wget ftp://ftp.trading-shim.org/pub/local/trading-shim/data-100331.tgz
4.  Position the data tarball down in the unrollsed shim 
directory.  Presently this would be:
 	cp data-100331.tgz shim-100416
5.  Move into that directory:
     	cd shim-100416
6.  Unroll the data tarball
 	tar zxf data-100331.tgz

Examining the unrolled shim and data tarballs, I see:
 	lrwxrwxrwx 1 herrold herrold 14 Apr 18 21:49 sym -> \
 		../data-100331

which points up one level too high at the moment.  This is due 
to some internal changes in release process in the last couple 
of weeks.  As you might expect, there are a number of internal 
releases that never see the external light of day for one 
reason or another as we test a push or address a private 
customer need.  It is easy to fix however

AND here the providing trarball is broken (and thus 
creating the need for a manual) symlink fixup.

Remove the old 'sym' first
 	rm -f sym
And make one at the same directory level
 	ln -s data-100331 sym

That properly creates the needed link so that the data load 
script can find the proper current dataset.  Alternatively one 
can unroll the data tarball up a link, and no change is 
needed

Recappping a quick test from my 'history', and chopping out a 
couple of lines where I was viewing interim results:

  1001  mkdir temp
  1002  cd temp
  1003  wget -q \
ftp://ftp.trading-shim.org/pub/local/trading-shim/shim-100416.tgz
  1004  tar zxf shim-100416.tgz
  1005  wget -q \
ftp://ftp.trading-shim.org/pub/local/trading-shim/data-100331.tgz
  1006  cp data-100331.tgz shim-100416
  1008  cd shim-100416
  1009  tar zxf data-100331.tgz
  1012  rm -f sym
  1013  ln -s data-100331 sym
  1014  ls -al
  1015  ls sym

And the instructions at:
 	http://www.trading-shim.org/capitals/?INSTALL
should simply work at that point.  I am aware that they also 
mention the 'manual' as a potentially better source, about 
which, more in a bit

> (2)
> After reading the section in the manual on adding symbols, I added another
> source file 'A0Stk10.sql' which loads fine. I notice 'LocalSet.sql' is not
> in use anymore...are there any other files I should be modifying when
> adding symbols? My new symbols seem to be working ok.

LocalSet (as you note, still referred to in the manual) was 
removed a while back, and the manual lags, which I thinks had 
been noted here before.  We are very aware of the issue as to 
the documentation, but have had to focus on some not so 
visible internal issues, pretty well described in the NEWS 
file
 	http://www.trading-shim.org/capitals/?NEWS


When the shim project started, manually adding new symbols 
stood out as a very 'touchy' and error prone process.  The 
process was also prone to clerical and trnascription errors 
and at that time, there was simply not a good way to retrieve 
contract information from IB.  IB have, in the intervening 
time, added an interface for electronic, and our understanding 
of methods consistent with data integrety and ability to be 
documented, have matured as well.

I personally continue to wish for a simple and not 'blocked by 
a intellectual property rights assertion' way to retrieve a 
complete statement of IB's symbology universe in a single 
snapshop.  But I am not aware of a present method for doing 
so.  Suggestions welcome


One can, and I have at times in the past written 'web screen 
scraping' routines, but they are slow, touchy to maintain, and 
as you note, subject to changes of symbol names and such, so 
getting an accurate snapshot is quite tricky

So, the constraint of the shim [for data validation and 
parsing purposes], early on, to know the full description of a 
symbol before it would accept portfolio and order information, 
had to be relaxed.  It would be safer perhaps to have 
continued this validation, but it was a both user maintenance 
burden, and its continuation implied that unknown holdings and 
orders appearing in the account and trades log messages would 
cause the shim to fail on a data error.

True from the perspective that the shim's database did not yet 
know of the contract, but probably not the 'right' behaviour 
compared to accepting a partially specified contract into the 
accounting journalling, and noting it in a form so that a post 
process could 'harvest' the needed details for doing the 
required contract info queries. With atht informatoin, new 
load files may be created, and then one can backup, and re-add 
an extensions during the load file process out of market 
hours.

This relaxation also (tangentially) made possible the recent 
addition of the 'blog' and 'dump' files, to relax the writable 
database constraint while in data mode

> (3)
> Do you want new symbol contributions? I found 10 US stock symbols from
> my own db not contained in Shim's db. In addition I found one symbol
> change for Shim's db (changing 'PCU' to 'SCCO'; no change to the ibc).

Thank you for the offer, but I think the answer is 'not at the 
moment'.  I know of several as well, and really the correct 
solution is me publishing some side scripting approaches I am 
presently using, and checking once again if thre is a IB 
provided way to get their full symbol set (I suspect the 
answer remains: no, because to some degree there is a lot of 
simpley irrelevant cruft of non-tradeables, and the answer 
depends to some degree on the country one is located in, and 
investor type one is)

Thank you for the report.  If the 'sym' symlink fix does not 
work (or if it does) for you,  please let us know

-- Russ herrold