We propose a syscall-based anomaly detection system that incorporates both deterministic and stochastic models. We analyze in detail two alternative approaches for anomaly detection over system call sequences and arguments, and propose a number of modifications that significantly improve their performance. We begin by comparing them and analyzing their respective performance in terms of detection accuracy. Then, we outline their major shortcomings, and propose various changes in the models that can address them: we show how targeted modifications of their anomaly models, as opposed to the redesign of the global system, can noticeably improve the overall detection accuracy. Finally, the impact of these modifications are discussed by comparing the performance of the two original implementations with two modified versions complemented with our models.
@inproceedings{Frossi2009Selecting_and,
title = {{Selecting and Improving System Call Models for Anomaly Detection}},
author = {Frossi, Alessandro and Maggi, Federico and Rizzo, Gian∼Luigi and Zanero, Stefano},
booktitle = {Proceedings of the 6th International Conference on Detection of Intrusions and Malware, and Vulnerability Assessment (DIMVA)},
month = {July},
year = {2009}
}