[ts-gen] redirecting shim output to python script

[Bill Pippin]

Nils,

About memory usage:

>        ...  What confuses me is the fact that the shim process eats
> quite a bit of memory.  To be more precise, about 1-2 GB til the end
> of the day. The shim process is opened by popen("...", "w"), appears
> as "shim --risk file save", receives a small 2-digit number of
> commands during its lifetime and the log/Shim file keeps a factor
> about 1000 below memory usage. 

> Has anybody observed similar usage in memory?

The current best measure of expected memory use is the number of
events, which as you suggest is much larger than the number of
commands.

Memory reallocation is a known issue to me, and I listed it on a road
map a while back, so you can find the message in the archives if you
are interested.  I didn't say much at the time; no one else has ever
mentioned it before.  Thanks for bringing it up; that increases its
priority.

The numbers you are seeing are in line with what I expect.  I use
memory pools everywhere to allocate memory, and have not yet put in
place the code needed to recover short-term usage.

Note that memory usage typically doubles going from 32bit to 64bit
machines.  Also please keep in mind that the shim's working set is
much, much smaller than the total use you report, which is why the
shim has such excellent time performance.  Indeed much of that
memory could be pushed out to swap with no harm whatsever.
 
The shim uses obj stacks to build events, and the usage you see should
drop considerably once I add Marker objects to the code to indicate
where events can be deallocated.  These will be in the main loop,
mostly, so that many object lifetimes will be limited to a 20 ms time
slot.

Other improvements should be achieved as I refactor the block
allocator, which I was working on week before last.  I put this work
aside when the bug reports of last week surfaced, but I'll go back to
it once I deal with the current issues, including a release that
should help you track down your intermittent journalling issue.
Part of that work --- byte-level request queueing --- is a
precondition for the memory reallocation work anyway.

Thanks,

Bill