[ts-gen] Retrieving a day's second by second price action

[R P Herrold]

Things were pretty wild and wooly on the stock market index 
futures today following yesterday's 'Fed' meeting 
announcement;  the following little script will 'harvest' 
second by second data into the HistoryBar table;

[herrold at centos-4 shim_071212]$ ( for i in `seq 10 16 `; do  \
 	echo "past add 181 21 Ymd_T(20071212 $i:00:00);" ; sleep 20 ;  \
 	echo "past add 181 21 Ymd_T(20071212 $i:30:00);" ; sleep 20 ;  \
 	done ; echo "quit;" ) | ./shim --data logd


'reading' the "shell one liner" script, we can see that 
Contract ID 181 is being gathered; at a pastFilter value of 
21, in half hour increments, preceeding the top and the bottom 
of each interval the market was open for trading.

[herrold at centos-4 dcop]$ ./decodeCID.sh 181
181 FUT.SMART.YM.200803

mysql> select * from  PastFilter where uid = '21';
+-----+-----+--------+------+----------+------+--------+
| uid | tid | period | reps | duration | unit | script |
+-----+-----+--------+------+----------+------+--------+
|  21 | 111 |   NULL |    0 |     1800 | s    |        |
+-----+-----+--------+------+----------+------+--------+

mysql> select * from HistoryTag where uid = '111';
+-----+----------+--------+--------+-----+
| uid | rth_only | format | what   | bar |
+-----+----------+--------+--------+-----+
| 111 |        1 | ymdt   | TRADES |   1 |
+-----+----------+--------+--------+-----+

So we are seeking 1800 one-second interval OHLC slices back, 
during regular trading hours.


I removed prior history so I could get a clean count, before 
running the script:

mysql> delete from HistoryBar ;
Query OK, 45020 rows affected (1.17 sec)


... and then again once it completed:

mysql> select count(*) from HistoryBar ;
+----------+
| count(*) |
+----------+
|    25200 |
+----------+
1 row in set (0.02 sec)

I also examined the head and the tail of the data:

mysql> select * from HistoryBar order by time limit 2 ;
+-------+-----+-----+---------------------+------------+------------+------------+------------+-----+------------+----------+
| uid   | cid | bid | time                | open       | high 
| low | close      | vol | wap        | has_gaps |
+-------+-----+-----+---------------------+------------+------------+------------+------------+-----+------------+----------+
| 80575 | 181 |   1 | 2007-12-12 09:30:00 | 13774.0000 |
 	13774.0000 | 13774.0000 | 13774.0000 |   0 | 13774.0000 |        0 |
| 80576 | 181 |   1 | 2007-12-12 09:30:01 | 13774.0000 |
 	13774.0000 | 13774.0000 | 13774.0000 |   0 | 13774.0000 |        0 |
+-------+-----+-----+---------------------+------------+------------+------------+------------+-----+------------+----------+
2 rows in set (0.05 sec)

mysql> select * from HistoryBar order by time desc limit 2 ;
+--------+-----+-----+---------------------+------------+------------+------------+------------+-----+------------+----------+
| uid    | cid | bid | time                | open       | high 
| low  | close      | vol | wap        | has_gaps |
+--------+-----+-----+---------------------+------------+------------+------------+------------+-----+------------+----------+
| 105774 | 181 |   1 | 2007-12-12 16:29:59 | 13597.0000 |
 	13597.0000 | 13597.0000 | 13597.0000 |   0 | 13597.0000 |        0 |
| 105773 | 181 |   1 | 2007-12-12 16:29:58 | 13597.0000 |
 	13597.0000 | 13597.0000 | 13597.0000 |   0 | 13597.0000 |        0 |
+--------+-----+-----+---------------------+------------+------------+------------+------------+-----+------------+----------+
2 rows in set (0.05 sec)

-- Russ Herrold