Behavior-Based Malware Detection Using Machine Learning

The main goal of this project was to prove that behavioral patterns—such as anomalous system calls, unusual file manipulations, or unexpected network activity—can provide a more resilient detection mechanism against zero-day malware and polymorphic threats. By focusing on how malware behaves rather than what it looks like, the system can detect previously unseen threats with higher accuracy.

Project Methodology: I began by collecting an extensive dataset comprising both benign and malicious executables. For each sample, dynamic analysis was performed in a controlled sandbox environment to log real-time behavioral indicators. Key features extracted included system call frequency, memory usage fluctuations, file I/O operations, registry modifications, and network packet counts.

To manage this high-dimensional data effectively, I applied Uniform Manifold Approximation and Projection (UMAP) for dimensionality reduction. UMAP helped preserve the intrinsic structure of the data while making it computationally feasible to perform clustering on complex behavioral patterns.

For the unsupervised detection engine, I utilized Gaussian Mixture Models (GMM). GMMs are well-suited for probabilistic modeling of clusters, which is ideal for detecting subtle variations in behavioral traits across different malware families. By fitting the GMM to the reduced dataset, the system could effectively classify samples into distinct behavioral clusters, differentiating malicious actions from benign baseline activity.

Rigorous testing was carried out using a test set that included both common malware variants and newly modified samples to simulate zero-day conditions. The system achieved a 94% classification accuracy, demonstrating robust detection performance even when facing obfuscated or previously unseen malware samples.