Apache log4net & LogParser (Lizard) together

Earlier this week, I quickly read through Peter Vogel’s most recent piece in the Visual Studio Magazine. The article’s title is “Debugging Tools for the .NET Developer”.  The article is very well written and had a lot of pointers and tools references.

One of the tools mentioned in Lizard Lab’s LogParser Lizard.

The article is available @ http://visualstudiomagazine.com/Articles/2011/11/01/Debugging-Tools-for-the-NET-Developer.aspx?Page=1

and the paragraph that appeared readily actionable is:

When it comes to reading those log files, I use Log Parser Lizard from Lizard Labs. The free version has some locked features that are only available if you buy the product, which is about $25, but I haven’t felt the need yet. Log Parser Lizard uses a SQL-like syntax to query log files (including CSV and XML files) and understands the IIS, Windows event logs and the log4net file formats straight out of the box. The results are displayed in a grid….

Sometime about database query via SQL and grid as opposed to loading log files into your favorite editor (Notepad++  in my case)  and attempting to find data via regular text search is technically progressive.

So once in front of a computer, downloaded Lizard Lab’s LogParser and played with reading data from my local box event’s viewer. Everything went effortless easy.

As I use Log4Net quite a bit played with the built-in sample, as well.

But, as life is always GOOD and BAD, had problems when I tried accessing one of the log files that I generated with log4Net.

There is a log4Net’s demo bundled with the Application and the App read it quite gently.  But, the ones I generated was a different story.  Wish I could have conformed the format of my log file to match the author’s, but the App did not document the format of the sample log file.

I thus searched the .Net for common log file patterns.  BTW, these patterns are known as  “conversionPatterns” in the log4Net parlance.

These days it seems everyone has a pattern:

Apache.org have sample ones @ http://logging.apache.org/log4net/release/manual/configuration.html

 


<layout type="log4net.Layout.PatternLayout">

<conversionPattern
value="%date [%thread] %-5level %logger [%ndc] - %message%newline"
/>

</layout>

SAS has sample ones – Saw one @ http://support.sas.com/documentation/cdl/en/logug/61514/HTML/default/viewer.htm#a003261250.htm


   <layout>
      <param
       name="ConversionPattern"
       value="%d %-5p %t %c (%F:%L) %m"
      />
   </layout>

The more I read about patterns the more it seems I will have flexibility in terms of what metadata I will want to place in my configuration specification and resultant log files.

And, only those will better appreciation for obscurity than I will love Regular Expression.  Only a 4th night ago I found out to my total disamay that Microsoft.Net Libraries does not have C-Style sscanf as a built in function.

And, thus started my irresistible push and pull to XML log files. At least, they will  have more\better rigidity in terms of what goes in a log file.   And, be a bit self described.

 


    <appender name="FileAppender" type="log4net.Appender.FileAppender">

       <file value="C:\Temp\app-log4net.xml" />

       <appendToFile value="false" />

       <layout type="log4net.Layout.XmlLayoutSchemaLog4j">

         <locationInfo value="true" />

      </layout>

   </appender>

 

 

 

Tried the syntax pasted below to load\access my XML file via Lizard Lab’ Log Parser:


SELECT *
FROM 'C:\Temp\app-log4net.xml'

But, no luck

Error parsing query : Error loading document <filename> : Reference to undeclared namespace prefix: ‘log4j’, [Unknown Error]

<a href=”https://danieladeniji.files.wordpress.com/2011/11/referencetoundeclarednamespaceprefix-log4j.jpg”><img class=”alignright size-full wp-image-1970″ title=”Reference To Undeclared Namespace Prefix – log4j” src=”https://danieladeniji.files.wordpress.com/2011/11/referencetoundeclarednamespaceprefix-log4j.jpg&#8221; alt=”” width=”640″ height=”202″ /></a>

Went back and reviewed the XML file and found the offending “log4j” tags:

<log4j:event logger=”dbEntail” timestamp=”1322168530203″ level=”INFO” thread=”1″><log4j:message>In objEntail.get:- SQL is dbo.usp_Entail</log4j:message><log4j:properties><log4j:data name=”log4net:UserName” value=”labuser” /><log4j:data name=”log4jmachinename” value=”labmachine” /><log4j:data name=”log4japp” value=”labQuery.exe” /><log4j:data name=”log4net:HostName” value=”labmachine” /></log4j:properties><log4j:locationInfo class=”dbSupport” method=”get” file=”c:\Support.cs” line=”200″ /></log4j:event>

<log4j:event logger=”dbEntail” timestamp=”1322168530203″ level=”INFO” thread=”1″><log4j:message>In objEntail.get:- SQL is dbo.usp_purchase</log4j:message><log4j:properties><log4j:data name=”log4net:UserName” value=”labuser” /><log4j:data name=”log4jmachinename” value=”labmachine” /><log4j:data name=”log4japp” value=”labQuery.exe” /><log4j:data name=”log4net:HostName” value=”labmachine” /></log4j:properties><log4j:locationInfo method=”get” file=”c:\Support.cs” line=”278″ /></log4j:event>

And, so next in line was to determine whether I can add the corresponding namespace references for log4j:

Found it difficulty to offhandedly add them to generated log files (thoroughly read through log4net docs)  and also to Lizard’s Lab logparser (from per-using the environment configuration tools and no progress while trying to augment the SQL Query).

Probably there are many avenues to “forcefully” remove log4j from our logfile, I settled on trying on use search\replace type commands.  Or, in this case search and discard.

So the Text replacement code we finally settled on is:

How Can I Find and Replace Text in a Text File?
http://blogs.technet.com/b/heyscriptingguy/archive/2005/02/08/how-can-i-find-and-replace-text-in-a-text-file.aspx


   OPTION EXPLICIT
   Const ForReading = 1
   Const ForWriting = 2

   Set objFSO = CreateObject("Scripting.FileSystemObject")

   Set objFile = objFSO.OpenTextFile("C:\Scripts\Text.txt", ForReading)

   strText = objFile.ReadAll
   objFile.Close
   strNewText = Replace(strText, "Jim ", "James ")

  Set objFile = objFSO.OpenTextFile("C:\Scripts\Text.txt", ForWriting)
  objFile.WriteLine strNewText
  objFile.Close

Customized and added a bit of flexibility:

  1. Replaced input and output file name  with command line arguments

 

<pre>	Const ForReading = 1
	Const ForWriting = 2

	Const strText = "log4j:"
	Const strTextReplacement = ""

	Dim strFileInput
	Dim strFileOuput

	Dim objFSO
	Dim objFile

	Dim strBuffer
	Dim strBufferNew
	Dim strLog

	If (WScript.Arguments.Count <> 2) Then
	Wscript.Echo "Error: Invalid number of arguments entered. "
	strLog = "Usage: " + WScript.ScriptName + " InputFileName OutputFileName"
	Wscript.Echo strLog
	Wscript.Quit
	Else
	strFileInput = WScript.Arguments.Item(0)
	strFileOuput = WScript.Arguments.Item(1)
	End if

	Set objFSO = CreateObject("Scripting.FileSystemObject")
	Set objFile = objFSO.OpenTextFile(strFileInput, ForReading)
	strBuffer = objFile.ReadAll
	objFile.Close

	Set objFile = Nothing
	strBufferNew = Replace(strBuffer, strText, strTextReplacement)

	If objFSO.FileExists(strFileOuput) Then
	   Set objFile = objFSO.OpenTextFile(strFileOuput, ForWriting)
	Else
	   Set objFile = objFSO.CreateTextFile(strFileOuput)
	End If 

	objFile.WriteLine strBufferNew
	objFile.Close

	Set objFile = Nothing
	Set objFSO = Nothing

Aptly saved the file as searchandreplace.vbs

Later found out that we need a singly rooted XML file:

Error message via LogParser:

 

A couple of other important pieces of business are:

  1. Needed ^ in front of special characters such as when using the echo commands (http://www.robvanderwoude.com/useless.php#EscapeChar)
  2.  When we tried accessing the file via the log Parser encountered an error stating “Error Parsing Query – Invalid at the top level of the document. [Unknown Error]”.Tried to load via Chrome and encountered error stating “Error on line NNN at column 1: Extra content at the end of the document”.Using Notepad++, found a misplaced SUB at the bottom of the file. The error was traced back to standard file merging via copy command – Changed to binary merging by adding a /B argument to copy command.





Errors Encountered (in logParser):

  • Error parsing query: Error loading : Invalid at the top level of the document.  [Unknown Error]
  • Errors Encountered (Chrome Browser):
  • Error : Extra content at the end of the document
  • Errors Encountered (Log file Contents):

 

Final Code:
  1. In echo command, prefixed special characters (< and >) with ^
  2. In copy command, augmented copy command with /B

   REM --------------------------------------------------------------------------------------------------------------------------
   @echo off
   cscript searchandreplace.vbs app-log4net.xml app-log4net-proper.xml
   echo ^<log4net^> > header.txt
   echo ^</log4net^> > footer.txt
   copy /B header.txt + app-log4net-proper.xml + footer.txt app-log4net-proper_v2.xml
   REM --------------------------------------------------------------------------------------------------------------------------

Suggested Reading:
  1. Practical .Net – Debugging Tools for the .Net Developer (Peter Vogel)
    http://visualstudiomagazine.com/Articles/2011/11/01/Debugging-Tools-for-the-NET-Developer.aspx?Page=1

 

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s