Scholarly open access journals, Peer-reviewed, and Refereed Journals, Impact factor 8.14 (Calculate by google scholar and Semantic Scholar | AI-Powered Research Tool) , Multidisciplinary, Monthly, Indexing in all major database & Metadata, Citation Generator, Digital Object Identifier(DOI)
— In the realm of big data processing, Hadoop clusters play a pivotal role. However, the increasing energy demand and the need for efficient resource utilization in these clusters remain a challenge. This research introduces a novel resource-aware scheduling algorithm aimed at enhancing the performance and energy efficiency of heterogeneous Hadoop clusters. Unlike conventional scheduling algorithms that overlook the varied capabilities of individual nodes, our algorithm dynamically assigns tasks based on real-time resource availability and node-specific performance metrics. The algorithm profiles each node for CPU, memory, disk, and network capabilities, and tasks are characterized by their resource demands to ensure a precise match during scheduling. Experimental results, derived from a series of benchmarks on a simulated Hadoop environment, indicate a significant improvement in resource utilization and energy efficiency. The proposed algorithm not only reduces the overall energy consumption by optimizing task allocation but also decreases the task completion time, thereby increasing throughput. These advancements contribute to a sustainable and cost-effective operation of Hadoop clusters, which is crucial for energy-intensive big data processes. The implications of this research are profound, offering a scalable solution that can adapt to the evolving landscape of data analytics infrastructure.
Keywords:
Hadoop Clusters, Big Data Processing, Resource-Aware Scheduling, Heterogeneous Computing, Energy Efficiency, Performance Optimization, Task Allocation, Node Profiling.
Cite Article:
"Dynamic Resource-Aware Scheduling for Enhanced Performance in Heterogeneous Hadoop Clusters", International Journal of Science & Engineering Development Research (www.ijrti.org), ISSN:2455-2631, Vol.8, Issue 12, page no.601 - 610, December-2023, Available :http://www.ijrti.org/papers/IJRTI2312085.pdf
Downloads:
000205257
ISSN:
2456-3315 | IMPACT FACTOR: 8.14 Calculated By Google Scholar| ESTD YEAR: 2016
An International Scholarly Open Access Journal, Peer-Reviewed, Refereed Journal Impact Factor 8.14 Calculate by Google Scholar and Semantic Scholar | AI-Powered Research Tool, Multidisciplinary, Monthly, Multilanguage Journal Indexing in All Major Database & Metadata, Citation Generator