Stream Processing Algorithms that model behavior changes

Abstract

This paper presents algorithms that fuse information in multiple event streams to update models that represent system behavior. System behaviors vary over time; for example, an information network varies from heavily loaded to lightly loaded conditions; patterns of incidence of disease change at the onset of pandemics; file access patterns change from proper usage to improper use that may signify insider threat. The models that represent behavior must be updated frequently to adapt to changes rapidly; in the limit, models must be updated continuously with each new event. Algorithms that adapt to change in behavior must depend on the appropriate length of history: Algorithms that give too much weight to the distant past will not adapt to changes in behavior rapidly; algorithms that don't consider enough past information may conclude incorrectly, from noisy data, that behavior has changed while the actual behavior remains unchanged. Efficient algorithms are incremental -- the computational time required to incorporate each new event should be small and ideally independent of the length of the history.