This leading security company improved database replication and analysis server performance by deploying Fusion ioMemory PCIe cards.
Cloudmark, a messaging security provider, is a veteran in the arms race against spam, phishing, and viruses. Spammers are constantly looking for ways to beat the system, and Cloudmark’s reaction time to new threats is of paramount importance to fighting new attacks. It invested in a high-performance disk solution, but data growth ate up every performance upgrade within a few months.
Always on the lookout for innovative ways to improve service at a lower cost, Ryan White, director of operations, began looking for a cost-effective and scalable performance solution that did not require continual outlays on disks.
Cloudmark had a couple of systems that required perpetual upgrading. The first system hosted its messaging security application and required database replication. As its data load grew, so did its performance requirements and disk investment. In order to get the top-notch performance Cloudmark’s customers required, continuous investments in capital were needed. The second system was an analysis server for which they wanted a more scalable solution to analyze ever-growing amounts of data.
After speaking with SanDisk®, Ryan decided to give some Fusion ioMemoryTM ioDrive® cards a try.
The SanDisk® Solution
Before implementing the Fusion ioMemory solution, Ryan ran benchmarks for various load patterns. Following are the results of a benchmark with 8KB records and a MySQL simulation.
|2x 160GB ioDrive cards||4x10K SAS (512MB write cache)||24x15K SAS (512MB write cache)||ioDrive card’s Improvement over 10K SAS||ioDrive card’s Improvement over 15K SAS|
|2x 160GB ioDrive cards||4x10K SAS (512MB write cache)||ioDrive card’s Improvement over 10K SAS|
Ryan also ran some benchmark tests on Cloudmark’s analysis server.
|1x 320GB ioDrive||4x10K SAS (512MB write cache)||24x15K SAS (512MB write cache)||ioDrive card’s Improvement over 10K SAS||ioDrive card’s Improvement over 15K SAS|
Director of Operations, Cloudmark
Supercharging Database Replication
The system supporting Cloudmark’s back-end consisted of a MySQL master server that replicated to slave (or mirrored) databases at three sites. The slaves at these sites were supported by massive disk arrays taking up a whopping 49U of rack space (see page 6 for detailed system information).
Just two ioDrive cards in each slave server completely eliminated the replication I/O bottleneck, improving Cloudmark’s average system reads and writes over five times. It also eliminated 210 spinning disks supporting the slave servers and recovered 27U of rack space.
Needless to say, Ryan was thrilled. “SanDisk has given us the edge in the security/ spammer arms race,” he said. “The ioDrive card performance improvements improved our replication times immeasurably, effectively shortening the window spammers have to work in before we block them.”
Creating Opportunities to Improve Application Performance
The ioDrive card performance improvements had a side-effect—it allowed Cloudmark’s messaging security application to fully utilize the CPUs. After adding the ioDrive cards, the application maxed the system’s processors. “Once we installed Fusion ioMemory products, we found that we ran into CPU walls before we ran into I/O walls,” Ryan said. “I’ve never had a database system that hit the CPU wall.”
The Cloudmark team began digging into ways to more efficiently use its system’s processors and found the application was making some inefficient queries whose impact wasn’t felt when disk I/O was the bottleneck. For example, one of its main applications had been issuing inserts and updates one query at a time. With a small amount of effort, fixing this problem produced amazing results. Ryan told us, “Without the I/O bottleneck, we could now update 1,000 or more records per query. This work improved the performance of our data-intensive queries by about five times. We went from our disk-based slaves having constant trouble keeping up, with constant 90-100% disk I/O saturation on the P800 controller with MSA70’s, to the SanDisk-based slaves always keeping up at 5% or less I/O utilization.”
Improving Analyst Productivity and Problem Turnaround Time
In addition to improving its data replication performance, Cloudmark was also interested in finding a more scalable solution for problem data analysis. This analysis consisted of investigating spam messages that should have been caught as well as authentic messages getting blocked. The investigation often involved searching massive amounts of data, including many large files (500MB-1GB in size), and could take hours.
The ideal solution would improve analyst productivity by reducing the time the system took to read the data from its sources, analyze it, manipulate it, and update the results in the master database to be propagated to customers with the next update.
SanDisk answered the call. The new system consisted of a server with two RAID0- striped 320GB ioDrive cards that provided sufficient capacity to hold the last three to four days worth of data, which covered well over 90% of the problem data cases. Cloudmark relegated its legacy 4TB system to searches of older data.
“People love it because they can do whatever they want, just bash away and it never slows down,” Ryan told us. “Our analysts are much more productive and we can get customer fixes for problem data much more quickly. This freed our analysts to improve analysis routines to leverage ioDrive cards to crunch through data nearly five times faster.”
Enhancing the Datacenter
In addition to achieving Cloudmark’s primary goal of improving data replication and analysis performance while curbing future outlay on disks, it also improved the reliability and availability of the system, reduced its server footprint, and lowered its operational costs.
Improving Availability and Reliability
Cloudmark implemented full system redundancy, improving both data reliability and availability.
“We RAID0 striped the ioDrive cards to get the capacity we needed, which meant that if we lost the data on one card, we would lose it on both.” Ryan told us, “However, we saved so much money on disks that we were able to purchase additional SanDisk powered servers to give us system-level clustering and redundancy.”
Lowering Operational Costs
As if these benefits weren’t enough, Cloudmark lowered future operating costs on several fronts.
|Master at Headquarters (9U)
||1 Slave at Replication Site 1 (22U)
Supermicro Whitebox, 1x Quad Core Xeon E5345, 4GB RAM (3U), 3Ware 9650SE RAID Controller with battery-backed write cache
HP DL380 G5, 1x Quad Core Xeon E5430, 14GB RAM (2U)
SanDisk helped Cloudmark stop the continual outlay on disks previously necessary to provide its customers with industry leading performance. This investment improved its data center as follows:
Ryan has the following to say about SanDisk: “After about a year and a half of using Fusion ioMemory products, when anyone asks me for a disk array, my first question is ‘Why?’ There is no reason for a disk array here. It’s just a box that I have to support with 25 individual failure points.”
While the previous statement is a testament to SanDisk solutions’ superiority to disk, the following is the most ringing endorsement: “We spend a lot less time now worrying about disk I/O and our databases keeping up, and we can focus on other things in the business.”
About the Customer
Cloudmark provides comprehensive messaging security solutions that protect subscribers and messaging infrastructure from spam, phishing, viruses and other harmful content. Founded by pioneers in messaging anti-abuse, Cloudmark’s solutions were architected from the ground up to stop current and emerging threat vectors with high efficiency and scalability.
As a result of its unrivaled accuracy and performance, Cloudmark has emerged as the most widely deployed messaging security solution in the world today. Cloudmark’s customers include over 100 of the world’s largest service providers, including the majority of tier-1 operators in North America, Japan and Europe.