I'm on a mission! I have to search log files that are between 150 MB and larger. These are syslogd files generated. Here is sample output.
2007-01-15 00:00:10 Mail.Debug 111.111.111.111 join[00000]: server.steve.net[1.1.1.1] 1168837209-0ff301650000-3Bdfku 1168837209 1168837210 SCAN - info@mydomain.com info@mydomain.com - 2 39 *abcdef.xyz.com SUBJ:Blah this is replaced
I don't confess to be an expert but this has got to be easier than I'm making it. I want to share my experiences, so far and I love adventures like this, because I learn a lot. What am I after? I want the 'server.steve.net[1.1.1.1]' or just [1.1.1.1].
Example #1 - For small files this works good
$sb = new-object System.Text.StringBuilder
$re = new-object regex('\[(\d{1,3})\.(\d{1,3})\.(\d{1,3})\.(\d{1,3})\]')
$m = $re.match((get-content mySysLogfile.txt))
while ($m.Success)
{
$sb.Append($m.value)
$sb.AppendLine()
$m = $m.NextMatch()
}
$sb.ToString() > st1.txt
Example #2 - Works for large files extracting the data, performance takes a couple hours.
$sb = new-object System.Text.StringBuilder
$re = new-object regex('\[(\d{1,3})\.(\d{1,3})\.(\d{1,3})\.(\d{1,3})\]')
$m = [System.IO.File]::OpenText("d:\temp\syslogcatchall16.txt")
while($line = $m.ReadLine())
{
$line = $re.Match($line)
$sb.Append($line)
$sb.AppendLine()
}
$m.Close()
$sb.ToString() > st1.txt
Log Parser example
'Example 1
logparser -i:tsv "select top 50 Count(extract_token(field6,1, '[')) as CountOfIt,extract_token(Field6,1,'[') as IPAddress into Steve.csv from '\\ServerName\ShareName\syslogd11.txt' Group By IPA
ddress order by CountOfIt DESC" -headerRow:off -iSeparator:'spaces'
'Example 2
logparser -i:tsv "select Top 10 Count(field6),Field6 from\\ServerName\ShareName\syslogd11.txt Group By Field6" -headerRow:off -iSeparator:'spaces'
'Findstr
http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/findstr.mspx?mfr=true
'QGrep
http://www.ss64.com/nt/qgrep.html
In conclusion, the clear winner was Log parser, speed and accuracy were great. Powershell was 'cool' but took too long. Maybe as I get better at Powershell, that will change. Findstr & QGrep appear to be more for parsing out entire lines of text. That was my experience, it could be my lack of advanced knowledge with these tools. I use FINDSTR a lot for doing quick searches, it is faster than FIND. I was hoping to use regular expressions, but found Powershell was easier to use for regex. I didn't try a grep utility found on sourceforge, because Log Parser did the trick. If you have other experiences using FINDSTR, QGrep or some other tool, please pass them along. Hope this helps!