White Paper: Plugging Into Utility Storage for Enterprise-Class Application Servers
Overview
Performance
Performance Conclusion
Guaranteed Data Availability
Guaranteed Data Availability Conclusion
Cost-Effective Scalability
Microsoft Sequel and Exchange using IP-SAN
Conclusion
Overview
Today, whether we’re at work or home, we don’t think twice about plugging our appliances into electrical outlets to instantly receive the power we need. Everything from making coffee to powering factory equipment to building automobiles uses the same basic electricity. Electricity, water and phones are utilities, they are consistent wherever we go, are delivered with varying degrees of reliability and we can have access to a lot or a little depending on our needs. The concept of “utility” services has not escaped IT professionals within large organizations. After-all, every house in America does not have its own power plant so why should every department within a large organization have its own IT infrastructure.
Removing data from workflow and departmental servers and delivering storage to those servers like a utility has been an IT goal for the last 10 years. Server captive data is difficult to protect, reallocate, share and reproduce when it’s kept on disk drives within individual servers. IT professional struggle with:
By delivering data storage capacity as a centralized utility you not only remove the data from the server and centralize it onto a network, but you can eliminate or greatly reduce the number of disk drives from 10’s to 1000’s of workflow servers. Utility storage delivers:
Until recently, the two choices for delivering storage from the network were either network attached storage – “NAS” for sharing storage resources using IP (internet protocol), or fibre-channel storage area networks – “FC-SAN” for sharing block-based storage resources using fibre-channel protocol. Fibre-channel is fast and highly reliable, perfect for revenue critical applications, but proved too expensive, too rigid and too complex for all application servers SANRAD Utility Storage Solution within the data center making it a poor choice for utility storage. NAS, on the other hand, was much simpler and not as expensive as FC-SANs, but proved to be difficult to scale and not reliable enough for all applications within the data center. The limitations of NAS and FC-SAN left the majority of servers still using internal storage and IT professionals still looking for the technology needed to deliver a truly open and flexible “utility” storage model. There was a need for another form of networked storage that combined the simplicity and flexibility of IP with the scalability, reliability and block-based properties of a FC-SAN. The IETF committee and industry giants worked together to create the new iSCSI standard storage protocol. iSCSI is reliable and block-based like FC-SANs so it can run any application, but it’s based on IP so it’s simple and cost effective to deploy like NAS. iSCSI is the technology needed to deliver “storage” like a utility. iSCSI storage volumes are consistent where ever you go, are delivered with adjustable degrees of reliability and performance and you can have access to a lot or a little depending on your needs. Like a utility, iSCSI storage can be accessed by plugging into a socket in the wall and can even be billed to departments depending on their monthly usage.
There are many IT professionals considering an iSCSI IP-SANs as a means to finally deliver their utility storage services to large numbers of application servers. The latest quarterly storage numbers from International Data Corp. show tremendous momentum in iSCSI. The market showed a 32% revenue growth over last quarter. In 2003, worldwide iSCSI revenue amounted to $18 million. In 2004 this soared to $113 million.
Surprisingly, many small businesses are not interested in utility storage. They have a low number of servers and small IT department. They don’t need to build a storage utility model to service multiple departments and large numbers of servers. Small businesses are interested in simplicity and low costs, and will generally select an entry-level iSCSI array. iSCSI arrays, like NAS systems, are single systems, they do centralize storage for a small number of attached servers. But a single iSCSI array represents a single point of failure, can be difficult to scale and limits performance to that of the individual system. In contrast, large businesses deploying utility storage are more concerned with performance, efficient storage utilization, guaranteed storage availability and future scalability costs. Storage area networks or SANs, based on IP, represent the more popular way of interconnecting a large number of application servers with different classes and grades of storage systems.
Because SAN’s are “networks” they are open and support all popular server operating systems, applications and storage systems. Before any user decides on the appropriate utility storage solution, they should consider the architectural attributes of an IP-SAN. There are three basic attributes that distinguish IP-SANs for iSCSI storage arrays. These IP-SAN attributes are continuous real-time data access even during component or system failure, low scalability costs to maintain a consistent CAPEX over time and the ability to deliver high peek load performance to a large number of utility storage customers. Because IP-SANs deliver these three key attributes in addition to the utility properties of iSCSI, they are becoming a popular network-storage solution for IT professionals seeking utility storage.
Diagram of typical iSCSI IP-SAN delivering storage like a utility to servers across an IP network
Performance
Utility storage, like electricity, needs to deliver the required data performance to the application server. After all, slow electricity would halt your appliances and slow data performance would halt your application servers. IP-SANs are designed to sustain high levels of random read and write operations. Intelligent IP-SAN switches have high-speed architectures utilizing network processors, real-time operating systems and powerful backplanes. A single IP-SAN switch can sustain 300MBs or 600MBs (when clustered) of random read and write requests and well over 60,000 IOPs. This delivers raw random read/write performance that is from 4 to 15 times greater than small arrays and can easily support from 10 to 200 standard application servers. IP-SANs can utilize any type of storage system. This allows the IT professional to select the storage systems that best fits the performance and reliability needs of varying application servers receiving utility storage. For example, you can use a storage system with FC drives rated for 200 MBs and 10,000 IOPs for application servers requiring high performance, and use a SCSI storage system with ATA drives rated for 40MBs and 3000 IOPs for application servers requiring lower performance.
In addition to selecting different classes of storage (different cache and drive types), IP-SANs can simultaneously read and write data to multiple independent storage systems. By spreading server volumes across independent storage systems and being able to directly access those storage systems without having to pass thru another control layer, IP-SAN’s can maintain line speed performance to the storage systems (up to 2GBs, 200MBs, 20,000 IOPs per storage system) regardless of the location of the data. Moreover, since the storage systems are independent of the intelligent IP-SAN storage switches, capacity and performance can be increased by simply adding more storage systems.
Performance Conclusion
IP-SANs deliver consistent CAPEX over time because they are not limited to any specific type or brand of storage array enabling IT professionals to select the type of storage systems that best fits different business applications – any interface, cache, drive type or component redundancy. IP-SAN performance is not affected when more arrays are added to increase capacity. In fact, IP-SAN performance can increase with more arrays because random read/write requests can be simultaneously spread and processed by multiple storage systems.
Guaranteed Data Availability
IT professionals realize that moving data from 10’s to 1000’s of individual application servers to a utility storage model would create significant business problems if the utility storage service were to become unavailable because of a component failure. Lost access to data, like lost access to electricity would bring most businesses to a complete halt.
Server level Multi-path Failover with the iSCSI Initiator
The iSCSI initiator, included free in most operating systems, consists of layers that are key to providing multiple data paths between servers and iSCSI storage volumes. The two main layers within the iSCSI initiator are the session and connection layer. The first layer, the “session” layer is an upper layer and is responsible for maintaining the communication to the SCSI layer within the server. It also ensures proper order of SCSI commands and data to and from the server file system to the iSCSI storage volumes. SCSI commands are numbered in sequence as they are sent from the server. The iSCSI storage volume arranges the SCSI commands according to their order, ensuring that commands are not lost, taken out of order or duplicated.
Within every server there is usually only one iSCSI initiator but there can be more than one session established and running within a single initiator. For example, if there are two iSCSI storage volumes being used by the server, then there would be one initiator with two sessions running.
The second layer is the “connection” layer. The connection layer is the TCP/IP connection between the server and the iSCSI storage volumes. The session layer can maintain several connections. In common applications there is only a primary iSCSI TCP/IP connection. But for 99.999 high availability applications there is a primary iSCSI TCP/IP connection and an alternant connection. In most cases all iSCSI traffic between the server and utility storage system travels over the primary connection. In most IP-SAN deployments, this is a Gb Ethernet link and is more than fast enough to handle average server traffic. With some application servers, a 10/100 Ethernet link (100Mb) is adequate since most servers don’t generate more than 50Mbs of sustained iSCSI / storage traffic. A Gb Ethernet link (1000Mb) is recommended to handle I/O spikes if spikes result in I/O bursts greater than 50Mbs assuming a 50% overhead.
Diagram of servers connecting to an IP-SAN in both single and high availability mode
Multi-pathing and path-failover are automatic with iSCSI. Because the iSCSI session is aware of alternate TCP/IP paths to the iSCSI storage volumes, it will automatically transfer traffic through an alternate TCP/IP connection. So, if one TCP/IP connection between a server and iSCSI storage volume fails, the traffic is automatically routed through the alternate TCP/IP connection. New iSCSI initiators in addition to failover, will also support load-balancing across multiple iSCSI connections. Because the SCSI commands are numbered, the iSCSI storage volumes are able to arrange and manage commands received across multiple connections.
SANRAD Demonstration of Server Multi-path Failover
To demonstrate how iSCSI failover functions, SANRAD participated in a third party demonstration where 5 videos where streamed from separate storage volumes delivered by a utility storage IP-SAN using the SANRAD iSCSI V-Switch. The 5 Microsoft Windows 2003 hosts reading the video streams from the IP-SAN used the native Microsoft-supplied iSCSI initiator driver software. Each host had two TCP/IP Ethernet connections used by the iSCSI session. The primary connection was a 10/100 Ethernet CAT5 copper connection between the host and utility storage model via a switched LAN. The second connection was a wireless WI-FI connection between the host and utility storage solution. The following statement reviews the failover test and results:
“We were able to disconnect the CAT5 cable from the hosts and the iSCSI session automatically routed the traffic over to the WI-FI connection. We were able to do video streaming to 5 hosts running iSCSI (wireless) going to an access point, then to a hub, then to the SANRAD V-Switch IP-SAN and then to a core-edge FC fabric, then to a virtual port, and finally mapped to an FC open-9 LUN on an enterprise class storage system. This forced failover test was extremely positive and fast enough to keep all 5 videos streaming.”
Diagram of iSCSI failover to WI-FI and fail-back to primary connection
Microsoft MPIO (multipath IO) eliminates the single connection concern for Windows. MPIO resides on the application server and uses multiple Ethernet connections to create multiple data paths to the storage devices within the IP-SAN. Similar to high-end FC-SAN solutions, MPIO provides load balancing of I/O traffic between the server and the storage utility represented by the IP-SAN. In the event a connection from the server were to fail for any reason, MPIO would shift all the read and writes requirements to the surviving connection (s). This would provide Microsoft servers on an IP-SAN with assured data access even during connection failure.
To provide storage system redundancy and data redundancy, IP-SANs use clustering technology to create system fail-over and fail-back scenarios among multiple IP-SAN storage switches. In addition to this system redundancy, real-time (synchronous) data mirroring option to independent storage systems provides a highly persistent and fault tolerant data environment.
Real-time Data Mirroring within IP-SANs
IP-SAN storage switches can create mirrored volumes where the two members of the mirror are located on independently attached and physically separated storage systems. Mirroring is performed by the IP-SAN storage switch in a similar fashion as a RAID controller. The major difference is that RAID controllers mirror storage devices within a single enclosure or server. However, because the IP-SAN storage switch is in the network layer, it can create and maintain mirrored members/ volumes anywhere within the network, indifferent to traditional physical limitation such as enclosures and distance. Local synchronous mirroring can now be performed simultaneously to two or more independent storage systems. IP-SANs can write directly and simultaneously to all the attached storage systems. So performance is increased and data is never compromised. Like a RAID controller, if one of the mirrored partners goes off-line or experiences a failure, the IP-SAN switch will automatically remove the failed member from operation and continue to service application server requests with the remaining mirror member. With a storage utility service using an IP-SAN, application server operation is not interrupted in any way if a storage system goes offline. The failover to the surviving storage systems and rebuilding of downed storage systems is completely invisible to any application servers using the IP-SAN utility storage solution.
Storage switch clustering is another key to enabling high availability and fail-over paths with the IP-SAN storage switch. In the event an IP-SAN storage switch is temporarily off-line or the network connections to the switch is lost, other IP-SAN storage switches attached to the same storage and the same host network can take over the IP addresses and data communication for the off-line switch. All IP-SAN storage switches are “active/active” servicing their assigned application servers, but they can also provide a “passive” failover path for other application servers within the network. This is because all IP-SAN storage switches maintain the configuration information of other IP-SAN storage switches within the same network topology and monitor the health and availability of their designated partner or partners. When a switch goes off-line, iSCSI will terminate the application server connection with the problematic switch or path and search for a new connection. While the new connection is being sought, the iSCSI session layer maintains the application while a new connection is sought. Within a few seconds of an IP-SAN storage switch going offline, another IP-SAN storage switch will sense the condition and exposes the IP addresses from the down storage switch. In doing so it also generates an ARP to update any Ethernet switches. iSCSI will discover the re-exposed IP addresses and immediately create a new connection, thus enabling the application servers to proceed with data communication through a new IP-SAN storage switch to the storage systems. The IP-SAN storage switch will continue to service its own application servers and the application servers of the off-line storage switch until the original off-line switch is brought back on-line and the connection paths or sites repaired. During fail-back, the IP addresses will be automatically restored to the returning switch and the original connections will be re-established, thus providing continuous data availability. The entire failover and re-connect sequence occurs within a matter of seconds and has no effect on application server operation.
This technique is called IP take-over and is used in conjunction with host level connection failover within a single data center to provide multiple failover paths to storage systems. It can also be used across multiple data centers to automatically provide a new route to storage resources in the event of sectional site failure.
Diagram of highly available IP-SAN including MPIO, mirroring and IP-SAN storage switch clustering
Host-site creation is very simple when combining real-time data mirroring and storage switch clustering. As mentioned in the previous section, when used in conjunction with real-time data mirroring and FC or third party FC tunneling techniques, it can provide an off-site disaster recovery facility which re-connects hosts with mirrored partners within seconds of primary site failure. The entire primary site is 100% replicated by placing an IP-SAN storage switch at the secondary site, in front of the storage systems receiving the mirrored data. The secondary site will have all the data which 100% matches the data at the primary site, and the second intelligent switch at the secondary site provides the mount point needed for application servers. Moreover, servers can also be clustered across the network so that the entire site (storage, data and servers) failover in under a minute in the event of complete site failure. www.rad-direct.com
Diagram of IP-SAN using data mirroring across two sties within a campus environment
IP-SAN storage switches can also replicate data across long distances using iSCSI over IP networks (LAN, MAN, WAN). As mentioned earlier, intelligent switches can mirror data between volumes on storage systems. This mirroring capability also exists between intelligent switches, one to one, many to one, or many to many. This mirroring can be done in a synchronous (real-time) manner, but is usually done in an asynchronous manner because IP links between sites are usually slower than internal LAN speeds. To keep replication as efficient as possible the IT professional can specify which volumes need to be replicated and which volumes are not mission critical and do not require replication. During replication, a copy of all write commands are cached locally into consistency groups on non-volatile local media (disk dives). Using PIT (point in time) technology, these groups of write commands are compiled into PIT files, which are transmitted to other IP-SAN storage switches. PIT technology provides a time/data consistent application mount point that eliminates data corruption issues. www.rad-direct.com
Diagram of IP-SAN using remote data replication across IP LAN, MAN or WAN
Guaranteed Data Availability Conclusion
IP-SANs use clustering technology to create system fail-over and fail-back scenarios among IP-SAN storage switches. In addition to system redundancy, real-time (synchronous) data mirroring to independent storage systems provides a highly persistent and fault tolerant data environment. Host-site creation is very simple when combining real-time data mirroring and clustering found in IP-SAN storage switches. IP-SAN storage switches can also replicate data across long distances using iSCSI over IP networks (LAN, MAN, WAN). Using PIT (point in time) technology, groups of write commands are compiled into PIT files, which are transmitted to other IP-SAN storage switches. PIT technology provides time/data consistent application mount points for application servers.
Cost-Effective Scalability
IP-SANs, by their nature, use an Open architecture and are much more cost effective to manage and scale.
IP-SANs do not limit the number of application servers that can connect to the architecture. IP-SANs that use IP-SAN storage switches do not require agents or license fees for each added server. So the IT professional does not have to pay fees in the future as they add more and more application servers - year after year to the IP-SAN. Large scale IP-SANs can scale to 250 servers without requiring any additional IP-SAN infrastructure expenditures.
Adding storage capacity is similar to adding servers. The IP-SAN storage switch has no practical limit to the number of custom volumes it can create and capacity it can manage. Leading IP-SAN storage switches allow IT professionals to scale their capacity to over 4000TB (4 Petabytes) and can create and manage over 50000 unique volumes. Since the IP-SAN storage switch is in the network layer, there is no cost associated with the IP-SAN storage switch when adding more disk drives, disk enclosures or arrays to increase the available capacity pool. Because the IP-SAN storage switch is agnostic to the storage brand and has FC, SCSI and IP ports it can support most storage interface protocols and drive types (FC, SCSI, SATA, ATA, iSCSI) the user has the freedom to select any disk storage supplier he wants for increasing the storage capacity and can take full advantage of the continuing reductions in drive / capacity prices allowing you to maintain a consistent CAPEX.
Two common applications are Microsoft Sequel and Exchange. The following test results demonstrate the performance of an IP SAN relative to these popular applications.
Diagram for of IP-SAN using different classes of storage for different applications
Microsoft Sequel and Exchange using IP-SAN
Users are concerned that network latencies will degrade application server performance. Two common data center applications among large business are Microsoft Sequel and Exchange. The following test results demonstrate the performance of an IP-SAN relative to these popular applications.
Microsoft Sequel Test
This Microsoft Sequel test demonstrates the performance differences between an FC-SAN and an IP-SAN. The test environment included a Windows 2003 server running SQL 2000. The server was attached to the IP-SAN using a regular 1 GB NIC and the standard MS iSCSI initiator software. The same server was attached to the FC-SAN using a FC HBA. In the case of the IP-SAN the server had a connection to an intelligent IP-SAN storage switch by SANRAD, the iSCSI V-Switch, which was using storage from an attached mid-range FC disk array. The disk array was configured with two LUNs. On the SANRAD V-Switch two volumes were created, one for the database and one for the transaction log. In the case of the FC-SAN test, the server was connected to the same disk array thru a fibre-channel switch. In both cases performance was tested using the Microsoft Database Hammer utility from the SQL 2000 resource kit. This utility created a database with 10 million rows and simulated SQL 2000 activities with different numbers of connections. The test was first conducted on the FC-SAN and then the IP-SAN. As you can see from the results, the FC-SAN did perform better than the IP-SAN, but the difference in latency and latch time was negligible and became relatively less as the number of connections increased. This test demonstrates not only that the performance of an IP-SAN is adequate for Microsoft SQL, but it also shows that IP-SANs and FC-SANs perform at nearly the same speed.
IOPs
Average Disc Latency(Database Volume)(ms)
SQL Server Avg Latch(ms)
Number of Connections
FC-SAN
Sanrad V-Switch
FC-SAN
Sanrad V-Switch
FC-SAN
Sanrad V-Switch
100
934
879
21
27
47
51
200
1009
976
29
35
51
55
300
1134
1118
39
43
62
69
400
1401
1355
48
55
78
81
500
1603
1546
59
66
96
103
Microsoft Exchange Test
The Microsoft Exchange test demonstrates the ability of an IP-SAN to perform within the performance boundaries recommended by Microsoft. The test environment included a Windows 2003 server running Exchange. The server was attached to the IP-SAN using a regular 1 Gb NIC and the standard MS iSCSI initiator software. On the intelligent IP-SAN switch two volumes were created from an attached mid-range FC disk array, one for the database and one for the transaction log. The following table describes the “JetStress” stress mode results for Exchange supporting different numbers of users. The important parameter to look for is average disk latency in milliseconds. The table presents latency in milliseconds for read IO and Write IO for the database and transaction log volumes. The test results show IP-SAN Exchange performance well beyond the performance limits recommended by Microsoft.
| Volume | 500 Users | 1000 Users | 1500 Users | 2000 Users | Maximum Recommended by Microsoft |
| Database Volume Read | 15 ms | 17 ms | 18 ms | 20 ms | 40 ms |
| Database Volume Write | 7 ms | 8 ms | 9 ms | 11 ms | 40 ms |
| Log Volume Raed | 0 ms | 0 ms | 0 ms | 1 ms | 20 ms |
| Log Volume Raed | 2 ms | 2 ms | 2 ms | 4 ms | 20 ms |
Conclusion
Today, whether we’re at work or home, we don’t think twice about plugging our appliances into electrical outlets to instantly receive the power we need. Electricity, water and phones are utilities, they are consistent wherever you go, are delivered with varying degrees of reliability and you can have access to a lot or a little depending on your needs. The concept of “utility” services has not escaped IT professionals within large organizations. Using iSCSI and an IP-SAN architecture, storage becomes a utility…simply plug your servers into the network and you will receive consistent and reliable storage capacity for any application. Like a utility, storage is consistent and compatible with all applications; it can be delivered with varying degrees of reliability and can be increased and recycled to meet diverse business needs.
By Zophar Sante
VP of Market Development
SANRAD Inc.
Send us your question and we will be happy to reply: