Machine learning based dynamic reconfiguration of distributed data management systems

A core problem affecting distributed data management systems relates to deciding the optimal system configuration in terms of, e.g., computing resources to be allocated. In the cloud computing era, this issue has become a compelling one, since cloud technologies enable elastic resource provisioning, based on dynamic ac- quisition and release of resources according to the "pay-per-use" pricing model. To take advantage of this resource management model, methods enabling the estima- tion of the minimum amount of resources that are required to sustain the application workload, while guaranteeing adequate system performance and availability (as es- tablished, e.g., in a Service Level Agreement - SLA), would be highly desirable. In this chapter, an overview of such a kind of methods is provided, particularly focusing on black-box system modeling approaches based on machine learning. Also, a case study where the automatic tuning of the configuration of a distributed in-memory data platform, deployed on top of a cloud infrastructure, is actuated via neural net- works is presented. In this case study, a controller, which exploits a neural-network based performance predictor, has been integrated with a mainstream open-source data platform and has been tested on top of the Amazon EC2 cloud platform.