Microsoft – OS – Memory – Tools – Poolmon – Installation & Usage

Introduction

For myriad reasons outside of un-kept engineering curiosity one might find value and interest in reviewing the list of device drivers installed and engaged on a system.

With a list one might also want to get hard numbers on how much resource (memory) each driver is taken.

Background

This analysis allows you one to do certain things:

  • Respond to Microsoft and other Vendors explicit\implicit requests for supporting data as a trouble ticket is being worked
  • Account for memory usage.  Each machine has a finite amount of memory and a DBA needs to work with OS Administrators and plan for how much memory to allocate to the OS, each Application, and how much to leave available to 3rd party applications (Backup, Anti-Virus). The rest of the memory is thus left available to device drivers.
  • Address intermittent system crashes.  Compared to Software Applications, Device drivers run a bit closer to the kernel.  Errant Device Driver calls or them not properly managing memory are thus more problematic to overall system stability.There is quite a bit of information on the Internet that covers how to ensure that memory dump files will be created and how to read the generated dump files; i.e. “How to read the small memory dump file that is created by Windows if a crash occurs”  (http://support.microsoft.com/kb/315263) and thus will not cover it here.

Installation Binaries

Depending on your OS, you can acquire poolmon through a few install media and location.

OS Description Version#
Windows NT 4.0 Windows NT 4.0 Resource Kit
Windows 2000, Windows XP, Windows Server 2003 \Support\Tools folder of Windows Install CD-ROMs.
Windows 2003 Windows 2003 Support Tools
http://www.microsoft.com/en-us/download/details.aspx?id=7911
Product Version
5.2.3790.0Date: 3/24/5000
Windows 2008, Windows 2008 R2, Windows 2012  Windows Driver Kit Product Version 6.3.9600.16384Date:
8/22/2013 4:32 AM

Installation Choices

Windows 2003 Support Tools Install on Windows 2008

Install Media location
http://www.microsoft.com/en-us/download/details.aspx?id=15326

Install Steps

Upon kicking off installation of the downloaded “Windows 2003 Support Tools” on my target computer, a Windows 2008-R2 Server, received a warning right away.

Warning Message:

This program has known compatibility issues.  Check online to see if solutions are available from the Microsoft website.  If solutions are found, Windows will automatically display a website that lists steps you can take.

Program: Windows Support Tools
Publisher: Microsoft
Location: Not Available

 

Windows2003SupportoolsInstallOnWindows2008

If you click on the “Check for Solutions online” button, you get a message stating that “No solutions found for Windows Support Tools

ProgramCompatibilityAssistant

Usage

 

Start poolmon.exe by launching a command shell, changing your current directory to the folder you chose as your install folder, and entering poolmon.exe


poolmon.exe

Error Message:


Poolmon: Query perf Failed (returned: c0000004)

Conclusion

So no go!

Windows Driver Kit (WDK) 8.1

What are Windows Driver Kit?
http://msdn.microsoft.com/en-us/windows/hardware/gg454513.aspx
They are the tools to build, test, debug, and deploy drivers.

Install Media Location
http://go.microsoft.com/fwlink/p/?LinkId=317353

Install Steps

There are a couple of items to note as you install Windows Driver Kit (for Windows 8.1):

Those things:

  • The Install path can no longer be changed.  It has to be C:\Program Files (x86)\Windows Kits\8.1\
  • The driver kit does not contain actual Application development software, but rather assisting tools

Specify Location

SpecifyLocation

Important Information

ImportantInformation

 

The install succeeded.

Installation Folder:

Remember that the targeted folder is C:\Program Files (x86)\Windows Kits\8.1.

As my targeted system is a 64-bit OS, we need to sojourn to C:\Program Files (x86)\Windows Kits\8.1\Tools\x64.

To try out poolmon launch a Command Shell and change your current folder to that folder.

Try poolmon

-b order by bytes accumulated

poolmon.exe -b

Here is the output (Development box):

PoolMon-VM

Tabulated Output (Development box):

Tag Allocs Frees Bytes
CM31 48295  14835 189505536
MmSt 24685  3144 47786176
Ntff 16500  2068 20320256
Ntff 13092  2517 13019776
MmRe  2669  1398 11891776
CM25  2621  2621 11694080

-a order by # of Allocations

poolmon.exe -a

Here is the output (Development box):

PoolMon-OrderBy-Allocs

Tabulated Output (Development box):

Tag # of Allocs Frees Bytes
Self 2400  2400 0
NDre 2688  2688 0
Even 2849  9720 1255072
NSpg 2843  2843 0

Explanation (Development box):

  • Our busiest Tags are sourced by Self, Ndre, Even, and NSpg
  • Self is such a generic word and I could not affirmatively link it to a specific Vendor
  • NDre is C:\Windows\System32\drivers\ndis.sys (Microsoft Network Sub-system)
  • Even is Microsoft’s Events Object (http://blogs.technet.com/b/yongrhee/archive/2009/06/24/pool-tag-list.aspx)
  • NSpg is C:\Windows\System32\drivers\nsiproxy.sys (Microsoft Network\Proxy Sub-system)
  • Though busy, memory is being allocated and freed at the same rate for all the listed sources, outside of Even

Review Poolmon output

Once you have Poolmon running, you want to look for a few things:

  • The Tags using the most Bytes.  Anything over 40,00,000 bytes ( 40 MB) deserves your acute look
  • Tags that are steadily Allocating bytes (Allocs column growing), but not de-allocating memory (Frees staying as is) might point at memory management problems

Review Specific Tags

Map Tags to Specific Device Driver – Using SysInternals Strings & Findstr

In some cases, the corresponding device drivers for each Tag entries are easy enough.

But, in some cases the Tags can not easily be mapped to specific Driver and thus vendor.

Here again Mark Russinovich is a true and beloved friend.  His tool (Strings.exe) available @ http://technet.microsoft.com/en-us/sysinternals/bb897439.aspx searches files resource area for strings passed in.

So in this case you pass in a Tag and for well written applications that support Internalization, the messages and tags are bundled a bit differently to allow for various languages to be more easily supported.

Use Scenario

Launch Command Shell and issue command based on the syntax below:

Syntax:

  cd c:\windows\system32\drivers 
  strings * | findstr /i TagID 

Sample:

   cd c:\windows\system32\drivers
   strings * | findstr /i "CM31"

Output:

Strings--SaEe

Map Tags to Specific Device Driver – Using Findstr

Microsoft’s Findstr is useful to find strings embedded in files underneath a folder.

Syntax:

ss64.com has a good and very usable documentation on findstr.  The URL to per-use is http://ss64.com/nt/findstr.html


  cd c:\windows\system32
  findstr /m /i /s TextToSearchFor file-name-pattern

Sample:


 cd c:\windows\system32\drivers
 findstr /m /i /S "SaEe" *.sys 

Output:

findstr

Explanation:

  • From the screen shot above, we moved up a folder and initiated our search from c:\window\system32 rather than c:\windows\system32\drivers and we were still able to match our Tag (SaEe) to a file name (drivers\srtsp64.sys)
  • Also, do not assume that all device drivers will be placed in c:\windows\system32\drivers folder

Map Tags to Specific Device Driver (Interpretation)

Tags Driver Vendor
CM31 Microsoft
SaEe srtsp64.sys Symantec/Norton AntiVirus
CIcr PEAuth.sys Protected Environment Authentication and Authorization Export Driver by Microsoft Corporation
Mdmi
MeRe
MmSt

On a lot of machines that I tested poolmon.exe, I found CM31 to be the driver using the most memory.  Please read this excellent write-up by Microsoft’s HarshDeep Singh.

In the article cited above, HarshDeep does the following:

System Overall Kernel Memory Usage

To back-track a bit, you might want to review your overall Kernel Memory Usage.  To do so, please launch Task Manager and glance over to the “kernel memory” section:

MemoryKernelUsage

You want to track your listed “Paged” and “Nonpaged” memory.

Microsoft’s Internal god, Mark Russinovich, does a good job describing the difference between the two.

Non-Paged Pool
http://blogs.technet.com/b/markrussinovich/archive/2009/03/26/3211216.aspx
The kernel and device drivers use nonpaged pool to store data that might be accessed when the system can’t handle page faults. The kernel enters such a state when it executes interrupt service routines (ISRs) and deferred procedure calls (DPCs), which are functions related to hardware interrupts. Page faults are also illegal when the kernel or a device driver acquires a spin lock, which, because they are the only type of lock that can be used within ISRs and DPCs, must be used to protect data structures that are accessed from within ISRs or DPCs and either other ISRs or DPCs or code executing on kernel threads. Failure by a driver to honor these rules results in the most common crash code, IRQL_NOT_LESS_OR_EQUAL.
Paged Pool

http://blogs.technet.com/b/markrussinovich/archive/2009/03/26/3211216.aspx
Paged pool, on the other hand, gets its name from the fact that Windows can write the data it stores to the paging file, allowing the physical memory it occupies to be repurposed. Just as for user-mode virtual memory, when a driver or the system references paged pool memory that’s in the paging file, an operation called a page fault occurs, and the memory manager reads the data back into physical memory. The largest consumer of paged pool, at least on Windows Vista and later, is typically the Registry, since references to registry keys and other registry data structures are stored in paged pool.
Couple of takeaways:

  • Non-Paged as the name now suggests means memory that will never be persisted to the paging file
  • Paged file can be moved to the paging file

Conclusively, our numbers are good!

Additional Reading & Tools

Vendor Driver Use
Microsoft PoolTag Pool Tag List
Microsoft verifier.exe Using Driver Verifier to identify issues with Windows drivers for advanced users
OSR Online PoolTag You’ve probably used poolmon, the DOS-style console mode app that comes with the DDK to monitor your driver’s pool usage.? PoolTag is a Win32 GUI version that improves on poolmon in several ways.? It easily lets you save output to a text file.? Also helps you find those nasty pool leaks by allowing you to take a snapshot of pool usage, and only show changes from that snapshot.? Check it out!
     

Conclusion

In our case, we did not notice any hypersensitive device drivers.  But, it was still a good exercise as re-discovered Microsoft’s YongRhee and his tireless sharing.

YongRhee does a very comprehensive job aggregating patches and he publicly avails them @ http://blogs.technet.com/b/yongrhee/.
Without over-stating it, Mark Russinovich answered my need to know why CM31 happens to take so much storage.  CM means configuration manager and dummy on me, it is the Registry.

It seems that Registry data is read and kept in memory and thus processes do not have to go to disk to read them.

And, so you really want to be careful as to what you install on your machine and also consider Registry cleaning & pruning tools.

References

References – Windows – Kernel Memory – Paged and NonPaged Pool

References – Pool Corruption

References – Memory Exhaustion

References – Pool Usage Metrics

References – Map Tag ID to File Name ( using findstr )

References – Cache

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s