Task runtime prediction in scientific workflows using an online incremental learning approach

Muhammad Hafizhuddin Hilman, Maria A. Rodriguez, Rajkumar Buyya

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

17 Citations (Scopus)

Abstract

Many algorithms in workflow scheduling and resource provisioning rely on the performance estimation of tasks to produce a scheduling plan. A profiler that is capable of modeling the execution of tasks and predicting their runtime accurately, therefore, becomes an essential part of any Workflow Management System (WMS). With the emergence of multi-tenant Workflow as a Service (WaaS) platforms that use clouds for deploying scientific workflows, task runtime prediction becomes more challenging because it requires the processing of a significant amount of data in a near real-time scenario while dealing with the performance variability of cloud resources. Hence, relying on methods such as profiling tasks' execution data using basic statistical description (e.g., mean, standard deviation) or batch offline regression techniques to estimate the runtime may not be suitable for such environments. In this paper, we propose an online incremental learning approach to predict the runtime of tasks in scientific workflows in clouds. To improve the performance of the predictions, we harness fine-grained resources monitoring data in the form of time-series records of CPU utilization, memory usage, and I/O activities that are reflecting the unique characteristics of a task's execution. We compare our solution to a state-of-the-art approach that exploits the resources monitoring data based on regression machine learning technique. From our experiments, the proposed strategy improves the performance, in terms of the error, up to 29.89%, compared to the state-of-the-art solutions.

Original languageEnglish
Title of host publicationProceedings - 11th IEEE/ACM International Conference on Utility and Cloud Computing, UCC 2018
EditorsJosef Spillner, Alan Sill
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages93-102
Number of pages10
ISBN (Electronic)9781538655047
DOIs
Publication statusPublished - 4 Jan 2019
Event11th IEEE/ACM International Conference on Utility and Cloud Computing, UCC 2018 - Zurich, Switzerland
Duration: 17 Dec 201820 Dec 2018

Publication series

NameProceedings - 11th IEEE/ACM International Conference on Utility and Cloud Computing, UCC 2018

Conference

Conference11th IEEE/ACM International Conference on Utility and Cloud Computing, UCC 2018
Country/TerritorySwitzerland
CityZurich
Period17/12/1820/12/18

Keywords

  • Online incremental learning
  • Scientific workflow
  • Task runtime prediction

Fingerprint

Dive into the research topics of 'Task runtime prediction in scientific workflows using an online incremental learning approach'. Together they form a unique fingerprint.

Cite this