• 家用
    • 移动存储
      • 至尊高速 Type-C USB 3.1双接口 OTG 闪存盘 | 闪迪
      • 适用于 iPhone 和 iPad 的欢欣i享™ 闪存盘 | 闪迪
        • 闪迪欢欣i享闪存盘
      • 闪迪至尊高速 OTG USB 3.0 闪存盘 | 闪迪
      • 欢欣畅享™无线闪存盘 | 闪迪
      • 闪迪至尊高速 Type-C 薄型USB闪存盘 | 闪迪
      • 欢欣i享迷你
      • 闪迪至尊高速™酷捷 OTG USB 3.0闪存盘 | 闪迪
      • iPhone专用的闪迪欣享自动备份充电座 | 闪迪
      • 闪迪双接口闪存盘 | 闪迪
      • 至尊™ C 型 USB 闪存盘 | 闪迪
    • 存储卡和读卡器
      • SD Cards
        • 闪迪至尊超极速移动™microSD
        • 闪迪至尊超极速™SDHC™/SDXC™ UHS-I 存储卡
        • 闪迪至尊极速™ SDHC™/SDXC™ UHS-I 存储卡
        • 至尊高速 SDHC/SDXC 存储卡,读取速度高达 80MB/s | 闪迪
        • SDHC/SDXC 存储卡 | 闪迪
      • microSD 存储卡 | 闪迪
        • 闪迪至尊超极速移动 microSD UHS-I 存储卡
        • 至尊极速移动 microSD UHS-I 存储卡加强版 A1 | 闪迪
        • 至尊极速移动microSDXC/microSDHC UHS-I 存储卡 | 闪迪
        • 至尊极速移动 microSD 存储卡| 闪迪
        • 适用于运动相机的闪迪至尊极速移动 microSD 存储卡 | 闪迪
        • 至尊高速移动 microSD 存储卡,读取速度高达 95MB/s | 闪迪
        • 适用于相机的闪迪至尊高速移动 microSD UHS-I 存储卡
        • 闪迪移动microSD 存储卡
        • 闪迪高度耐用的视频监控 microSD 存储卡
        • 至尊高速移动 microSD UHS-I 存储卡,读取速度高达 48MB/s | 闪迪
        • 至尊高速移动 microSD 存储卡(适用于智能手机) | 闪迪
        • 至尊超极速移动 microSDXC UHS-II 存储卡 | 闪迪
        • 闪迪至尊高速移动 microSDXC 存储卡加强版 | 闪迪
        • 至尊高速移动 microSDXC | 闪迪
        • Nintendo Switch 专用闪迪 microSDXC 存储卡
        • 闪迪至尊极速 microSDXC 存储卡 | 闪迪
      • 紧凑型闪存卡
        • 闪迪至尊超极速™ CompactFlash® 存储卡
        • 闪迪至尊极速™ CompactFlash 存储卡
        • 闪迪至尊高速™ CompactFlash 存储卡
      • CFast 2.0 卡
        • 闪迪至尊超极速™ CFast 2D 存储卡
      • 读卡器
        • 至尊超极速 CFast 2.0 读卡器/写卡器 | 闪迪
        • ImageMate All-in-One USB 3.0 读卡器 | 闪迪
        • MobileMate Duo 适配器和读卡器 | 闪迪
        • MobileMate USB 读卡器 | 闪迪
        • 至尊超极速 SD UHS-II 读卡器 | 闪迪
        • 至尊超极速 SD UHS-II USB-C 读卡器 | 闪迪
        • ImageMate Pro USB 3.0 读卡器 | 闪迪
    • USB 闪存盘
      • 闪迪至尊极速USB 3.1 闪存盘
      • 闪迪至尊超极速 USB 3.1 固态闪存盘
      • 闪迪酷圈 USB闪存盘
      • 闪迪至尊高速酷铄™ USB 3.0 闪存盘
      • 闪迪至尊高速 USB 3.0 闪存盘
      • 闪迪至尊高速酷豆™ USB 3.0 闪存盘
      • 闪迪酷悠™ 3.0 USB 闪存盘
      • Cruzer Force USB 闪存盘 | SanDisk
      • 闪迪酷悠™ USB 闪存盘
      • 酷轮 USB 闪存盘
      • 酷扭™ USB 闪存盘
      • 闪迪酷锁™ USB 闪存盘
      • 酷刃 USB 闪存盘
      • 酷捷 USB 闪存盘
      • 闪迪酷豆 USB 闪存盘
      • 闪迪至尊高速酷豆™ USB 3.1 闪存盘
    • 固态硬盘
      • 闪迪至尊极速™ 900 型移动固态硬盘
      • 闪迪至尊极速 510移动固态硬盘
      • 闪迪至尊极速™ 500 型移动固态硬盘
      • 闪迪至尊高速mSATA固态硬盘
      • 闪迪至尊超极速™ 固态硬盘
      • 闪迪至尊高速™固态硬盘
      • 闪迪固态硬盘加强版
      • 闪迪固态硬盘笔记本电脑升级工具套件
      • 闪迪至尊高速 3D 固态硬盘
      • 闪迪至尊高速 3D 固态硬盘重新定向
      • 闪迪至尊极速 便携式固态硬盘
      • 闪迪至尊超极速™ M.2 NVMe 3D 固态硬盘
    • 至尊极速团队
      • Steve McCurry
      • Akito Mizutani
      • Walter Iooss
      • Elizabeth Kreutz
      • Tom Bol
      • Lucas Gilman
      • Marcelo Maragni
      • Jeff Lewis
      • Corey Rich
      • Christian Pondella
      • Caio Guatelli
      • Terrell Lloyd
      • Michael Grecco
      • Akash Das
      • Kaustav Saikia
      • Mahryska 和 Tinayums
      • Amy Tierney
      • Alex Liu
      • Scott A. Woodward
      • Steve Simon
      • Kike Calvo
      • C.S. Ling
      • Peter Eastway
      • Cliff Mautner
      • Bambi Cantrell
      • Bob 和 Dawn Davis
      • Zach 和 Jody Gray
      • Marcus Bell
      • Sam Nicholson
      • Richard Finn Gregory
      • Vincent Laforet
      • Sebastien Devaud
      • Paolo Baccolo
      • Phil Coates
      • David Newton
      • Daniel Fox
      • Fred Pompermayer
      • Christian Nørgaard
      • Kirill Umrikhin
      • Matthew Jordan Smith
      • Richard Walch
      • Marcel Laemmerhirt
      • Dom Daher
      • Joao Carlos
      • Claudia Goetzelmann
      • Patrick Bellair
      • Wendell Phillips
      • Tyler Stableford
      • Jonathan & Angela Scott
      • Ellen Anon
      • Daisy Gilardini
      • Georg Tappeiner
      • George Karbus
      • Renan Ozturk
      • Lars Schwellnus
      • Tim Laman
      • Sean Scott
      • Nick Didlick
      • Dave Black
    • Stories
    • 购买地点
  • 商用
    • 数据中心
      • 产品
        • 闪存设备(非页面)
          • PCIe 闪存(非页面)
          • SSD (非页面)
        • 闪存软件(非页面)
          • 开源
          • ioVDI 软件
      • 解决方案
        • 数据库
        • 虚拟化
        • 大数据
        • 云解决方案
      • 合作伙伴
      • 资源库
        • Case Studies
        • Data Sheets
        • Overviews
        • FAQ
        • Report
        • Articles
        • Video
        • Solution Briefs
          • All-Flash Storage System for Ceph and OpenStack
          • Red Hat Ceph Storage on the InfiniFlash All-flash Storage Platform
          • The Solution for Cassandra at Scale | SanDisk
          • Storage for MongoDB Analytics
          • Optimizing Ceph Deployments Solution Brief
      • 博客
    • 计算
      • X600
      • X400
      • Z400s
      • sn720
      • sn520
    • 专业照片和视频
  • OEM 设计
    • 数据中心
    • 计算
    • 移动设备
      • iNAND 嵌入式闪存驱动器
    • Commercial
    • 工业原始设备制造商 (OEM) 设计解决方案
      • iNAND(内部存储器)
      • 工业存储卡
      • OEM 卡
    • 车载
      • 车载 SD 存储卡
      • 车载 iNAND
      • 车载 SD 存储卡
    • Connected Home
    • Blog
  • 关于闪迪
    • 备受信赖的品牌
      • 里程发展
      • 品牌优势
        • 企业
        • 消费类
      • 品牌多元化
      • 投资项目
    • 媒体中心
      • 新闻稿
        • 2017
          • SanDisk® Launches its Fastest, High-Capacity USB Flash Drive Ever
          • SanDisk® Unveils World’s First microSD Card Designed to Deliver a New Dimension of Mobile Application Performance
          • Western Digital Transforms the Mobile Experience with New ‘Smart’ iNAND 7350 Storage Solution Built on 3D NAND
          • SanDisk® Teams with Leading Manufacturers to Offer Consumers a Faster Mobile App Experience
          • SanDisk Now Offers 256GB of Extra Storage for iPhone and iPad
          • Western Digital Introduces iNand 7250a Embedded Storage Device For Evolving Data Demands Of Connected Automotive Technologies
          • Western Digital To Deliver World’s First Client Solid State Drives With 64-layer 3d Nand Technology
          • Western Digital gives consumers a unique, hassle-free solution for daily iPhone backup
      • Awards & Accolades
        • 太平洋电脑网 (pconline.com)
        • 太平洋电脑网 (pconline.com)
        • 电脑之家 (PCHome)
        • 电脑之家 (PCHome)
        • 中关村在线 (ZOL)
        • 中关村在线 (ZOL)
        • IT168
        • 天极网 (Yesky)
        • 天极网 (Yesky)
        • 天极网 (Yesky)
        • 天极网 (Yesky)
        • 电玩巴士 (TGBUS.com)
        • PCEVA
        • 三好街网 (Sanhaojie.com)
        • 52硬件网 (52hardware.com)
        • 电脑报 (PCW)
        • 电脑报 (PCW)
        • 电脑报 (PCW)
        • 电脑报 (PCW)
        • 电脑报 (PCW)
        • 新潮电子E-fashion
        • 新潮电子E-fashion
        • 数码影像时代 (Digital Video Times Magazine)
        • 数码摄影 (Foto Video)
        • 新电脑 (CHIP)
        • 新电脑 (CHIP)
        • 微型计算机 (Micro-computer)
        • 热点科技
        • 手机中国
        • 手机中国
      • 活动
        • Microsoft World Partner Conference
        • International Super Computing
        • Mobile World Congress Shanghai
        • Flash Memory Summit
        • Intel Developer Forum 2015
        • VMworld US
        • LinuxCon North America
        • SpiceWorld
        • VMworld Europe
        • VMworld Europe
        • GTEC
        • Oracle OpenWorld
        • HP Discover London
        • Gartner Data Center Conference
        • Photo Plus Expo 2015
        • Big Data Expo
        • SQL PASS Summit 2015
        • TigerDirect Innovation – IT Conference and Expo
        • Mobile World Conference
        • Embedded World
        • Open Compute
        • Linux Open Vault Conference
        • LinuxCon North America 2016
        • Western Digital: The Future of Data
      • 媒体资源
        • 产品
        • 企业
          • 管理团队
      • 洞察报告
    • 招聘信息
      • 为什么选择闪迪
        • 薪资和福利
      • 校园招聘
      • 职业发展
    • 世界公民
      • 社会
      • 环境
      • 社区
      • 奖学金
        • 奖学金获得者
      • 供应商管理
    • 投资者关系
    • 联系我们
  • 支持
    • iXpand Compatibility
  • 家用
  • 商用
  • OEM 设计
  • 关于闪迪
  • 支持
 
  • 数据中心
    • 计算
    • 专业照片和视频
  • 产品
  • 解决方案
  • 合作伙伴
  • 资源库
  • 博客

白皮书

Accelerating Enterprise Search with Fusion ioMemory PCIe Application Accelerators

This paper shows how Fusion ioMemory accelerates performance results for an Apache Lucene search.

Abstract

With enterprise data sets growing larger, the need to provide accessibility to that data is more important than ever. Analytics are important, but they are only a part of your overall data ecosystem. More and more, companies are finding the need to make their content searchable, and they are turning to enterprise search applications, such as Apache Lucene, to make that happen.

However, techniques that work on traditional data sets often fail when applied to today’s big data problems. Large indexes can quickly overwhelm traditional search platforms, forcing users to invest in expensive scale-out architectures in order to meet their performance requirements. In this paper we describe how Fusion ioMemory PCIe solutions from SanDisk® can be used to dramatically increase search performance, without a massive investment in infrastructure.

 

Performance Considerations for Apache Lucene

Apache Lucene is an open source Java library, which provides full text search for a variety of different content types. It is managed by the Apache Software Foundation and released under a Creative Commons license. Because of its powerful text search capabilities, Lucene has been used to create a number of enterprise search engines, such as Solr, ElasticSearch, and Blur.

The hardware architectures for these applications have traditionally relied on locally attached hard drives to store their indexes. While this works well for smaller indexes, larger indexes can quickly overwhelm these drives, resulting in degraded performance. To mitigate this problem, search applications store as much of their indexes in memory as possible. This approach is not always sufficient, however, as larger indexes can quickly exceed the amount of available memory in the server, forcing indexes to be read from disk.

Search applications provide scalability by distributing the index across multiple servers in a cloud configuration. This makes it possible to store the entire index in memory, using the combined memory capacity of the cluster. However, the "RAM cloud" approach requires a significant investment in infrastructure and may be cost prohibitive for many organizations.

 

Solving the Scalability Problem

Traditional storage devices rely on disk controllers and RAID controllers for access. This imposes additional latency and overhead, as data is serialized, copied and re-copied through multiple layers of controllers and embedded processors. The Fusion ioMemory PCIe devices from SanDisk use a virtual memory architecture, whereby the CPU accesses the NAND directly, as though it were second tier of server memory. The Virtual Storage Layer (VSL) presents the device to the host operating system as a virtual disk, creating a storage device, which has the performance characteristics of memory. By storing Lucene’s indexes on a Fusion ioMemory device rather than traditional memory, it is possible to support much larger data sets, allowing users to expand their search capabilities without a massive investment in infrastructure.

 

Performance Testing

To evaluate the performance characteristics of this approach, a test system was built with the following configuration1:

  • Dual Intel® Xeon® E5-2600 Processors
  • 64 GB memory
  • 6x 1TB 7200 RPM HDDs in RAID 0
  • 6x ioScale 1.6 TB PCIe card in RAID 0
  • CentOS 6.4
  • Apache Solr 5.0

Two instances of Apache Solr were created, using the default configuration. The first was installed on the HDD array, while the second was stored on an array of six Fusion ioMemory devices. An identical index was loaded onto each instance, and several tests were conducted in order to compare their performance.

The data set used for testing was derived from Wikipedia's database, which is available for download under the Creative Commons license. A single database dump contains roughly 50 GB of data. This data set is not large enough for a meaningful test on a system with 64 GB of memory, as the majority of data would be cached in memory. Therefore, the test data set consisted of six Wikipedia dumps from six different months, totaling 246 GB, and creating an index of 202 GB.

At the time of this writing, there were no benchmarking tools available for Apache Solr 5.0, so a custom benchmarking tool was written. The test tool generated random search terms, consisting of one, two, or three dictionary words. The tool spawned multiple threads, which submitted search requests using Solr's RESTful interface. Before each test, the OS buffer cache was cleared so as not to skew the results. The test was run for a total of 120 minutes, and statistics were collected on the number of searches performed and the latency of each request.

 

Test Results

Comparing the HDD-based solution to the Fusion ioMemory flash-based solution the results were as follows:

 

 

The average response time on disk was 1.3 seconds, with many request taking 30 seconds or more. The Fusion ioMemory based instance showed an average response time of 24 milliseconds, with sub-second responses for every request. Overall, the Fusion ioMemory-based instance showed a 54x improvement in response times over the disk-based system.

 

 

In the overall performance test, the Fusion ioMemory-based system was able to service 328 searches per second, as opposed to 23 searches per second on disk. The overall performance improvement was more than 14x.

 

Conclusion

The Fusion ioMemory-based solution showed a significant performance improvement over disk. It was able to maintain sub-second response times with the larger index, while the disk-based solution experienced degraded performance. Overall, the Fusion ioMemory-powered system offers a much more cost effective solution to scaling with DRAM, delivering more than ten times the performance of disk within a single server.

Disclosures

1. Based on internal testing.

Accelerating Enterprise Search with Fusion ioMemory PCIe Application Accelerators
Download

准备好借助闪存技术继续前进了吗?

不论贵公司是财务五百强企业还是五人小创业公司,闪迪都能提供适合您的解决方案,帮助您充分利用现有基础架构。

在线
咨询

欢迎向我们提问,我们会尽快回复您。

全球销售
联系人

查找全球办事处的联系方式。

客户
关怀

对您购买的产品有疑问? 我们可以提供帮助!

销售咨询

如果您有任何疑问或者想要讨论根据贵公司需求定制的闪迪解决方案,闪迪销售团队可随时为您提供帮助。

我们会竭诚回答您的问题。请填写下表,以便我们提供帮助。

字段不能为空。
字段不能为空。
请输入有效的电子邮件地址。
字段不能为空。
字段不能为空。
字段不能为空。
字段不能为空。

请指出您感兴趣的领域:

您必须选择一个选项。

问题或评论:

隐私政策

谢谢!我们已收到您的请求。

网站菜单

  • 家用
    • 移动存储
    • 读卡器
    • USB
    • 固态硬盘
    • MP3播放器
  • 商用
    • 数据中心
    • 计算
    • 合作伙伴
    • 专业相片 & 视频
  • OEM 设计
    • 移动设备
    • 计算
    • 车载
    • 互连家庭
    • 工业 & 其他
    • 数据中心
  • 媒体中心
    • 新闻稿
    • 奖项和荣誉
    • 媒体资源
  • 招聘信息
  • 投资者关系
  • 联系我们

支持部

  • 业务支持
  • 零售支持

快速链接

  • IT 博客
  • 渠道合作伙伴门户
  • 闪迪故事
  • 至尊极速团队

IT 博客

Post Thumbnail

{{feed.title}} | {{getFormattedDate(feed.publishedDate) | date:''}}

阅读更多信息 »
  • 法律事务
  • 使用条款
  • 商标
  • 隐私声明
  • Cookie
  • Taiwan BSMI RoHS
  • 关于广告
  •  
闪迪
© 2018 Western Digital Corporation 或其关联公司版权所有。保留所有权利。