Real-Time Big Data Processing Framework: Challenges and Solutions

Authors

Zhigao Zheng, School of Software and Microelectronics, Peking University, Beijing 100260, China\\ National Engineering Research Center for E-Learning, Central China Normal University, Wuhan 430079, ChinaFollow
Ping Wang, School of Software and Microelectronics, Peking University, Beijing 100260, China\\ School of Electronics Engineering and Computer Science, Peking University, Beijing 100791, China\\ National Engineering Research Center of Software Engineering, Peking University, Beijing 100791, ChinaFollow
Jing Liu, School of Electronics Engineering and Computer Science, Peking University, Beijing 100791, ChinaFollow
Shengli Sun, School of Software and Microelectronics, Peking University, Beijing 100260, ChinaFollow

Author Country (or Countries)

China

Abstract

Data type and amount in human society is growing in amazing speed which is caused by emerging new services as cloud computing, internet of things and location-based services, the era of big data has arrived. As data has been fundamental resource, how to manage and utilize big data better has attracted much attention. Especially, with the development of internet of things, how to processing large amount real-time data has become a great challenge in research and applications. Recently, cloud computing technology has attracted much attention with high-performance, but how to use cloud computing technology for large-scale real-time data processing has not been studied. This paper studied the challenges of big data firstly and concludes all these challenges into six issues. In order to improve the performance of real-time processing of large data, this paper builds a kind of real-time big data processing (RTDP) architecture based on the cloud computing technology and then proposed the four layers of the architecture, and hierarchical computing model. This paper proposed a multi-level storage model and the LMA-based application deployment method to meet the real-time and heterogeneity requirements of RTDP system. We use DSMS, CEP, batch-based MapReduce and other processing mode and FPGA, GPU, CPU, ASIC technologies differently to processing the data at the terminal of data collection. We structured the data and then upload to the cloud server and MapReduce the data combined with the powerful computing capabilities cloud architecture. This paper points out the general framework for future RTDP system and calculation methods, is currently the general method RTDP system design.

Recommended Citation

Zheng, Zhigao; Wang, Ping; Liu, Jing; and Sun, Shengli (2015) "Real-Time Big Data Processing Framework: Challenges and Solutions," Applied Mathematics & Information Sciences: Vol. 09: Iss. 6, Article 46.
Available at: https://digitalcommons.aaru.edu.jo/amis/vol09/iss6/46