TY - GEN
T1 - Real-time scheduling in MapReduce clusters
AU - He, Chen
AU - Lu, Ying
AU - Swanson, David
PY - 2014
Y1 - 2014
N2 - MapReduce has been widely used as a Big Data processing platform. As it gets popular, its scheduling becomes increasingly important. In particular, since many MapReduce applications require real-time data processing, scheduling real time applications in MapReduce environments has become a significant problem. In this paper, we create a novel real-time scheduler for MapReduce, which overcomes the deficiencies of an existing scheduler. It avoids accepting jobs that will lead to deadline misses and improves the cluster utilization. We implement our scheduler in Hadoop system and experimental results show that our scheduler provides deadline guarantees for accepted jobs and achieves good cluster utilization.
AB - MapReduce has been widely used as a Big Data processing platform. As it gets popular, its scheduling becomes increasingly important. In particular, since many MapReduce applications require real-time data processing, scheduling real time applications in MapReduce environments has become a significant problem. In this paper, we create a novel real-time scheduler for MapReduce, which overcomes the deficiencies of an existing scheduler. It avoids accepting jobs that will lead to deadline misses and improves the cluster utilization. We implement our scheduler in Hadoop system and experimental results show that our scheduler provides deadline guarantees for accepted jobs and achieves good cluster utilization.
KW - MapReduce
KW - cluster utilization
KW - real-time scheduling
UR - http://www.scopus.com/inward/record.url?scp=84903973687&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=84903973687&partnerID=8YFLogxK
U2 - 10.1109/HPCC.and.EUC.2013.216
DO - 10.1109/HPCC.and.EUC.2013.216
M3 - Conference contribution
AN - SCOPUS:84903973687
SN - 9780769550886
T3 - Proceedings - 2013 IEEE International Conference on High Performance Computing and Communications, HPCC 2013 and 2013 IEEE International Conference on Embedded and Ubiquitous Computing, EUC 2013
SP - 1536
EP - 1544
BT - Proceedings - 2013 IEEE International Conference on High Performance Computing and Communications, HPCC 2013 and 2013 IEEE International Conference on Embedded and Ubiquitous Computing, EUC 2013
PB - IEEE Computer Society
T2 - 15th IEEE International Conference on High Performance Computing and Communications, HPCC 2013 and 11th IEEE/IFIP International Conference on Embedded and Ubiquitous Computing, EUC 2013
Y2 - 13 November 2013 through 15 November 2013
ER -