Parsing a growing logfile


Parsing a growing logfile



I want to do a logfile parser using ruby, this parser should parse the log file while it grows. It should parse line by line until the end and then wait (somehow?) for more lines to come, so my question is how to best handle it growing?

edit: Would prefer a portable way of doing this, even though my logfile is on Windows (for the moment).




Unable to catch an exception in Ruby

1:



How to highlight all instances of a variable in TextMate?
For Windows, you can use Directory Change Notifications.


test if a PDF file is finished in Ruby (on Solaris/Unix)?
You tell Windows (with FindFirstChangeNotification) to monitor the directory c:/foo/logs, then Windows updates your handle when something happens in that directory.


Ruby gem listed, but wont load (gem in user dir, not ruby dir)
At that point, you check to see if the change involves a file you care about..
How do I figure out what this character is?
Ruby has bindings for the Win32 API, and there is an example of getting these notifications..
Complex DataMapper query association


How to put elements of a file into a hash? -Ruby


Ruby: Boolean attribute naming convention and use

2:


There is a good script posted at http://www.biterscripting.com/SS_WebLogParser.html .

It is a sample script written for web server logs, but can be used as a starting point for writing your own log parser for logs of any kind.

To use it in a continuous way, while the log file keeps growing, here is a script..
# Script LogParser.txt # Go in a continuous loop, sleeping 1 hr each time. 

while (true) do # The number of lines in the log file the last time we checked is in following # variable.

Initially, it will be 0.

var int lines_old # Read the log file into a str variable.

var str log ; cat "file.log" > $log # Get the number of lines found this time.

var str lines_new ; set $lines_new = { len -e $log } # Strip off the first $lines lines.

lex -e (makestr(int($lines))+"]") $log > null # The new lines are now available in $log.

Process them with something similar to # SS_WebLogParser script.

# Update $lines_old, then, sleep.

set $lines_old = $lines_new sleep 3600 # 3600 seconds = 1 hour done
To try,.
  1. Save this script into, say, C:\LogParser.txt (since you are on windows).
  2. Download biterscripting.

    Google it up.
  3. Call our script by entering the following command.. script "\LogParser.txt".
If you need to use any of their sample scripts, install them with the following command..
script "http://www.biterscripting.com/Download/SS_AllSamples.txt" 
Patrick.


3:


For this task you can use IO.popen to get a file stream on the pipe of the growing result of a command line.

Then use the readline function in while loop.

Here is an example with the "adb logcat" command which get the live growing log of an Android device :.
#! /usr/bin/env ruby  IO.popen("adb logcat") do |io|     while line = io.readline         line.strip!          # Process here         print "#{line}\n"      end end 
EDIT. For a file, it is a little different.

I would "readline" in polling on the file stream..
#! /usr/bin/env ruby  File.open("test.log") do |io|     loop do          begin             line = io.readline             line.strip!         rescue             sleep 0.2             retry         end          # Process here         print "#{line}\n"     end  end 



72 out of 100 based on 42 user ratings 642 reviews