Control-based Scheduling in a Distributed Stream Processing System

Abstract

Stream processing systems receive continuous streams of messages with raw information and produce streams of messages with processed information. The utility of a stream-processing system depends, in part, on the accuracy and timeliness of the output. Streams in complex event processing systems are processed on distributed systems; several steps are taken on different processors to process each incoming message, and messages may be enqueued between steps. This paper deals with the problems of distributed dynamic control of streams to optimize the total utility provided by the system. A challenge of distributed control is that timeliness of output depends only on the total end-toend time and is otherwise independent of the delays at each separate processor whereas the controller for each processor takes action to control only the steps on that processor and cannot directly control the entire network. This paper identifies key problems in distributed control and analyzes two scheduling algorithms that help in an initial analysis of a difficult problem.