Log File Analysis with Bayesian Content Filters
My thesis research relates to using off-the-shelf Bayesian Spam filters to filter syslog and application log files. Just as a spam filter can be trained with spam and ham messages then used to filter other messages, this same filter can be trained with "outage-related" and "non-outage-related" log entries, then used to filter other log entries. With a minimum of manipulation, these filters proved to be quite effective, even showing 4 previous outages of which I was previously unaware. The greatest challenge lay in finding multiple similar outages with their associated log entries. I was lucky enough to find two such sets. The Spring set were known to be related; the Fall set were not known to be related (and indeed were not).
Here are the graphs used for the thesis, both of my ICADIWT papers (conference paper and published paper) as well as my NOMS short paper. Cutting down my 100+ pages of text and 40 images to 8 pages of text and 5 images was painful, but cutting that down to the 4 page NOMS paper was even more painful. With that said, the shorter versions are much easier to read, though they are lacking the following graphs which make the point most clearly. This is the full set of graphs, showing the effectiveness of SpamAssassin, SpamBayes and Bogofilter for log file outage-correlation.
The first two graphs are simulated "perfectly effective" and "perfectly ineffective" filter score sets. The vertical lines in the graphs represent approximate times of of outages (the thesis explains why these are approximate). Note in the Spring images that the filter scores for outage records form vertical dot patterns in the graphs. I have circled them to make them more plain to see in the printed documents.
Here are the graphs used for the thesis, both of my ICADIWT papers (conference paper and published paper) as well as my NOMS short paper. Cutting down my 100+ pages of text and 40 images to 8 pages of text and 5 images was painful, but cutting that down to the 4 page NOMS paper was even more painful. With that said, the shorter versions are much easier to read, though they are lacking the following graphs which make the point most clearly. This is the full set of graphs, showing the effectiveness of SpamAssassin, SpamBayes and Bogofilter for log file outage-correlation.
The first two graphs are simulated "perfectly effective" and "perfectly ineffective" filter score sets. The vertical lines in the graphs represent approximate times of of outages (the thesis explains why these are approximate). Note in the Spring images that the filter scores for outage records form vertical dot patterns in the graphs. I have circled them to make them more plain to see in the printed documents.