Storage – Planning – Determining Storage Requirements based on IOPS Requirements

Introduction


http://en.wikipedia.org/wiki/IOPS
IOPS (Input/Output Operations Per Second, pronounced eye-ops) is a common performance measurement used to benchmarkcomputer storage devices like hard disk drives (HDD), solid state drives (SSD), and storage area networks (SAN).

Background

I had a conversation with someone this last week.  And, he asked me a question.  The question is that how many disks does one need if you have requirements for 15000 IOPs.

And, I am thinking it depends.

Never had a need to know.  And, so this Saturday morning, I decided to take to the web; specifically Google and see if I can find anything on-line.

I am all set, as I have B.o.b playing in the background:

B.o.B – Don’t Let Me Fall
http://www.youtube.com/watch?v=ZXeQiS4wMEU

Google brought up very good sites, right away:

There are major faucets to review as you determine your IOPS needs.

  • Factors
  • Current Usage Model
  • Maximum Throughput Storage Apparatus

Factors

The number of IOPS that one actually gets depends on a few things:

  • Raid Level
  • Ratio of read and write operations
  • Balance of Sequential to Random access
  • Number of worker threads
  • The Queue Depth
  • Data Block Sizes

in real world settings other factors can indirectly affect IOPS, including:

  • System Setup
  • Storage drivers
  • OS Background operations
  • Network Choice (Fiber Channel [FC] & SCSI)
  • Work Load (NTFS, Database)

Raid Level

http://en.wikipedia.org/wiki/RAID

RAID (redundant array of independent disks, originally redundant array of inexpensive disks is a storage technology that combines multiple disk drive components into a logical unit. Data is distributed across the drives in one of several ways called “RAID levels”, depending on the level of redundancy and performance required.

The term “RAID” was first defined by David PattersonGarth A. Gibson, and Randy Katz at the University of California, Berkeley in 1987.

Marketers representing industry RAID manufacturers later attempted to reinvent the term to describe a redundant array of independent disks as a means of disassociating a low-cost expectation from RAID technology.

Your disk Raid level impacts your IOP counts.

Raid levels should not impact Reads.  But, because writes end up being multiplexed (written out unto a few disks to increase recoverability), we pay an expected price during writes.

I am very partial to well written and easy to understand pictorials, and I exceedingly like Scott Rowe’s illustration as to the price one pays for the various Raid Levels:

Raid – I/O Impact
Raid Level Read Write
Raid 0 1 1
Raid 1 (and 10) 1 2
Raid 5 1 4
Raid 6 1 6

So from Scott Rowe’s pictorial nakedly copied above, one can see that depending on how much protection you want (in terms of Raid Levels), you end up writing to N Disks or figurative N times.

Reads might actually benefits from RAIDS as multiple disks can be summoned and read from.

We all know that you do not not have straight writes nor reads.  So per you calculation, you will probably be asked the ratio of Reads to Writes.

So if you are told ahead of time that you will probably have 80% reads and thus 20% writes, you can see that only 20% of you workload will suffer the consequences of your RAID choice.

So if you have an IOP figure in mind, you can look back and see how the percentile of reads & writes and RAID Level choice will impact the IOPS you will actually get:

IOPS - Impact of RAIDS -v2

In the example listed below:

  • % of Reads is 80%
  • % of Writes is 20%
  • Raid Level of 1 –> Reduces Write by factor of 2
  • 10000 IOPS
   

    --percentile of reads: 80%
    --percentile of writes: 20%
    --RAID Level - Impact
      -->  (0.8 * IOPS) + ( (0.2 * IOPS) / 2)

     --> (08.8 * 10000) + (0.2 * 10000) / 2
           8000 + 1000 = 9000

–percentile of reads: 80% –percentile of writes: 20% –RAID Level – Impact –> (0.8 * IOPS) + ( (0.2 * IOPS) / 2)

–> (08.8 * 10000) + (0.2 * 10000) / 2
8000 + 1000 = 9000

so though we shot for 10000 IOPs, we will only 9000 IOPS based on expected workload and RAID Configuration.

In the example listed below:

  • % of Reads is 60%
  • % of Writes is 40%
  • Raid Level of 5 –> Reduces Write by factor of 4
  • 15000 IOPS
    --percentile of reads: 60%
    --percentile of writes: 40%
    --RAID Level - Impact
      -->  (0.6 * IOPS) + ( (0.4 * IOPS) / 4)

     --> (0.6 * 15000) + (0.4 * 15000) / 4
           9000 + (6000) / 4  
           9000 + 1500
           10500

so though we shot for 15,000 IOPs, we will only get 10,500 IOPS based on expected workload and RAID Configuration.

Ratio of Read and Write Operations

We have already covered how reads and writes can impact IOPs.  In summary, RAID Levels impacts writes; but reads are not lessened.

Balance of Sequential to Random Access

Sequential reads is much, more faster than Random Access.

The reasons are obvious, outside of SSDs, disks are more likely than not mechanical, rotational apparatus.

The more random the requests are, the more likely the disk heads have to move

  • Rotational speed
  • Average latency
  • Average Speed time

Numerous computational techniques and algorithms have been designed to better coalesce and fetch data.

Will address more on that (in a later write-up).

Number of Worker Threads

Threads are a main-stay of high performant systems.  Multi-threaded applications are written to gracefully tackle the various areas of concerns a distributed applications need to address.

Generally speaking, the more worker threads that be can gracefully managed by the system the more responsive the system will appear to be.

Having said that, threads use up resources and inter-process communications between threads have to be accounted for.

Async I/O

Most modern operating systems support Asynchronous I/O.  And, how they do so depends on each Application’s architecture and desire\ability to push the Technology boundaries.

Here is a good write up by Bob Dorr:

SQL Server IO Presentation http://blogs.msdn.com/b/psssql/archive/2010/03/24/how-it-works-bob-dorr-s-sql-server-i-o-presentation.aspx

SQL Server also exposes the pending (async) I/O requests in the sys.dm_io_pending_io_requests DMV. I specifically point out that the column ‘io_pending’ is a key to understanding the location of the I/O request and who is responsible for it.

If the value is TRUE it indicates that HasOverlappedIOCompleted returned FALSE and the operating system or driver stack has yet to complete the I/O.

Looking at the io_pending_ms_ticks indicates how long the I/O has been pending. If the column reports FALSE for io_pending it indicates that the I/O has completed at the operating system level and SQL Server is now responsible for handling the remainder of the request.

Using Async I/O means that SQL Server will often exceed the recommended disk queue length depth of (2).

This is by design as I just described with the read ahead logic as one example. SQL Server is more concerned with the number of I/O requests and average disk seconds per transfer than the actual queue depth.

Data Block Sizes

Your data block sizes will vary depending on your Application.

  1. NTFS – Up to NT 4.0 maximum was 4 KB
  2. NTFS – Win2K and afterwards different sizes
  3. MS SQL Server – 8192 bytes or 8K

Combinational Storage

Aggregated storage like NetApp Filers can have disks carrying different specs – That is some of the disks will have certain rotational speeds, whereas others will have different specs and speeds).

Due to such mixings, it is best to look a bit deep when your are really concerned about disk speeds.

  

Review Current Usage model

If this is an existing application, review your current usage metrics.

For us it is a Database; specifically MS SQL Server.

To gather usage stats, do the following:

  • Use MS Windows Performance Monitor
  • Use MS SQL Server \ Management Studio – Activity Monitor – Database I/O Chart

We will use these numbers later, but as a place holder we will call them your needed\desired throughput

What is the maximum you will get from your HBA Choice?

Once HBA is mentioned we are referring to external storage sitting outside of the box.

Using Joshua Townsend’s blog:

Storage Basics – Part IX: Alternate IOPS Formula
http://vmtoday.com/2013/01/storage-basics-part-ix-alternate-iops-formula/

Fiber Channel - Convert -- FC to MBps

For 16 GB FC, we use the same equation:



Megabits/sec  

MB/s =   baud_rate * bytes-actually-used-for-data
         ---------------------
         bytes-allocated

MegaBytes/sec

MB/s = MB/s
       ------
       8 bits equal 1 byte

We need values for the various “informants”:



Megabits/sec  

MB/s =   baud_rate (14025) * bytes-actually-used-for-data (62)
         ---------------------
         bytes-allocated (64)

       = 14025 * 62
         ------------
         64

       = 13586.71875

MegaBytes/sec

MB/s = MB/s
       ------
       8 bits equal 1 byte

     = 13586.71875
       -----------
       8

     = 1698.34

That is it!

Our number of 1698.34MB/sec is very close to Joshua Townsend’s number (1700 MB/sec)

What will you get from single disks

In cases where one is using internal or directly attached disks, one needs to look at individual disks capability.

Manufacturer supplies that rating through various performance metrics:

  • Sustained Transfer Rate
  • Rotational Speed
  • Seek time
  • Latency time

A combination of these numbers can be used to determine your drive’s capabilities.

Sustained Transfer Rate

Sustained Transfer Rates is published MB/sec.  This number probably maps to your Vendor’s requirement and so hopefully you will not have to much work.

Rotational Speed

Here is data from a couple of collaborating sources:

Interface Device IOPS (Average)
SATA 7200 rpm 75 to 100 IOPS
SATA 10000 rpm 125 to 150 IOPS
SATA 15000 rpm 150 to 190 IOPS
SAS 10000 rpm 140 IOPS
SAS 15000 rpm 175 to 210 IOPS

Please keep in mind these numbers are only for mechanical storage and not SSDs.  The numbers for SSDs are so over the top; and they should be covered on their own.

For those curious about SSDs, please reach out to Henry Newmans’s coverage per “SSD IOPS Arms race does it matter
http://www.enterprisestorageforum.com/storage-management/ssd-iops-arms-race-does-it-matter.html

What is your IOPs requirements

How do you find out what your IOPs needs are?

I will rest on Joshua Townsend’s once again.  The specific blog that provided clarity was:

Storage Basics – Part IX: Alternate IOPS Formula
http://vmtoday.com/2013/01/storage-basics-part-ix-alternate-iops-formula/

AlternativeIOPSFormula

So there we have it… To determine your IOPs

  • Get your average requirement for MB/sec
  • Determine your block size (set at OS Level or Application Level)

How much IOPs should you ask for

Knowing that your RAID Level will impact your IOPS, the question is now how much IOPS you should ask you SAN Admin for:

Unfortunately, I really struggled with what should be a simple formulae:

After hours, took to the Internet, and Googled for “total iops performance”.

Thanks goodness I did, as I found:

IOPS: Performance Capacity Planning Explained
http://blog.synology.com/blog/?p=146

Array IOPS = (I * r) + (I * w * f)

  • r=read percentage (as a decimal point, ie 0.75)
  • w=write percentage (as a decimal point, ie 0.25)
  • I= Desired IOPS
  • f=raid penalty
    • RAID-0 = 1
    • RAID-1/10 = 2
    • RAID-5/50 = 4
    • RAID-6/60 = 6

Tools

There are some tools for measuring IOPs

References

References – What is IOPS

References – Calculate IOPS

References – Storage Capacity Planning

References – Raid Levels

References – Fibre Channel

References – SSDs Performance

References – NetApp – Mixing Disk Types

References – I/O Queue Depth Strategy

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s