Big Insurance Data: Drawing Lessons from Amazon, Google, and Facebook
Abstract
There has been much written about the emergence of Big Data, both as a phenomenon and as a set of practices, infrastructure, and algorithms designed to allow modern computing to analyse ever-increasing data in fast and efficient ways. Celent uses the open source Hadoop solution to provide insight into what Big Data is, what it means to the insurance industry, and the kinds of solutions it provides to new and old insurance problems.
In a new report, Big Insurance Data: Drawing Lessons from Amazon, Google, and Facebook, Celent reviews how Big Data is now relevant and accessible to the insurance industry. In this report, Celent offers a discussion on the following topics:
- Grid computing or the use of many distributed, connected machines to undertake large-scale work.
- Distributed file systems, which store large amounts of data across many machines.
- MapReduce, which is an algorithm to split workloads over grid computing.
- Hadoop, which is a specific implementation of the MapReduce algorithm.
“The simple fact is that, with little more than a credit card, today you can rent an infrastructure from Amazon similar to that used by Yahoo to index and search the entire Internet,” says Craig Beattie, Analyst with Celent’s Insurance group and author of the report. “It is important that insurers understand just how much these frameworks do on their behalf and how easy it is to leverage this infrastructure to ask big questions of our data.”
The report draws together recent developments in cloud infrastructures, grid computing, and distributed computing to address issues regarding large volumes of structured and unstructured data. While there are many approaches to leveraging grid infrastructures, this report details one of the most popular approaches and perhaps one of the simpler approaches to implement. While the topic is necessarily technical in nature, the report addresses it in an accessible manner.
Celent is a research and advisory firm dedicated to helping financial institutions formulate comprehensive business and technology strategies. Celent publishes reports identifying trends and best practices in financial services technology and conducts consulting engagements for financial institutions looking to use technology to enhance existing business processes or launch new business strategies. With a team of internationally based analysts, Celent is uniquely positioned to offer strategic advice and market insights on a global basis. Celent is a member of the Oliver Wyman Group, which is part of Marsh & McLennan Companies [NYSE: MMC].
Media Contacts
North America (New York)
Dana Greenstein
dgreenstein@celent.com
Tel.: +1 646 364 8254
Europe (Stockholm)
Nick Bockh
nbockh@celent.com
Tel.: +46.8.650.4401
Asia (Tokyo)
Yumi Nagaoka
ynagaoka@celent.com
Tel.: +81.3500.3023
Table of Contents
Executive Summary | 3 | |
Introduction | 4 | |
| An Overview of the Solution | 5 |
An Overview of Grid Computing | 6 | |
Storing Data in Distributed Storage | 8 | |
| Distributed Storage Isn’t Simple | 8 |
| Distributed File Systems | 9 |
| Alternative Solutions | 11 |
The MapReduce Approach | 13 | |
| Build: The Map Then Reduce Approach | 13 |
| Distribution and Sharing | 14 |
| Execution | 14 |
Example Applications of Hadoop | 16 | |
| Sorting Data | 16 |
| Finding Recommendations | 16 |
| Analysing Social Data | 17 |
| Analysing Telematics Data | 18 |
| Programming Hadoop | 19 |
| Considerations | 20 |
Adopting Hadoop | 21 | |
| Hadoop Grids as a Data Analysis Resource | 21 |
| Hadoop Grids for Real Time Log Capture | 21 |
| Hadoop Offerings | 21 |
Conclusion | 23 | |
Leveraging Celent’s Expertise | 24 | |
| Support for Financial Institutions | 24 |
| Support for Vendors | 24 |
Related Celent Research | 25 | |