Search:
Computing and Library Services - delivering an inspiring information environment

HPC and the Big Data challenge

Holmes, Violeta and Newall, Matthew (2016) HPC and the Big Data challenge. Safety and Reliability: SaRS Journal, 36 (3). pp. 213-224. ISSN 0961-7353

Metadata only available from this repository.

Abstract

High performance computing (HPC) and Big Data are technologies vital for advancement in science, business and industry. HPC combines computing power of supercomputers and computer clusters, and parallel and distributed processing techniques for solving complex computational problems. The term Big Data refers to the fact that more data are being produced, consumed and stored than ever before. This is resulting in datasets that are too large, complex, and/or dynamic to be managed and analysed by traditional methods. Access to HPC systems and the ability to model, simulate and manipulate massive and dynamic data, is now critical for research, business and innovation. In this paper an overview of HPC and Big Data technology is presented. The paper outlines the advances in computer technology enabling Peta and Exa scale and energy efficient computing, and Big Data challenges of extracting meaning and new information from the data. As an example of HPC and Big Data synergy in risk analysis, a case study of processing close-call data is conducted using HPC resources at the University of Huddersfield. A parallel program was designed and implemented on the university's Hadoop cluster to speed up processing of unstructured free form text records pertaining to close call railway events, in order to identify potential risks and incidents. This case study demonstrates the benefits of using HPC with parallel programming techniques, and the improvements achieved compared to serial processing on a standard workstation computer system. However, it also highlights the challenges in risk analysis of Big Data that require novel approaches in HPC system and software design.

Item Type: Article
Subjects: Q Science > QA Mathematics > QA75 Electronic computers. Computer science
Q Science > QA Mathematics > QA76 Computer software
T Technology > T Technology (General)
Schools: School of Computing and Engineering
School of Computing and Engineering > High-Performance Intelligent Computing > High Performance Computing Research Group
Related URLs:
Depositing User: Violeta Holmes
Date Deposited: 03 Jan 2017 15:12
Last Modified: 21 Aug 2017 12:05
URI: http://eprints.hud.ac.uk/id/eprint/30559

Downloads

Downloads per month over past year

Repository Staff Only: item control page

View Item View Item

University of Huddersfield, Queensgate, Huddersfield, HD1 3DH Copyright and Disclaimer All rights reserved ©