A Hybrid CNN–LSTM Framework for Behavioral Malware Detection and Dynamic YARA Rule Generation

Abstract

The rapid evolution of malware through the use of obfuscation techniques and continuous runtime behavior mutation has made traditional signature-based detection mechanisms much less effective, making there is a dire need for adaptive and deployable malware detection solutions. In response, behavioral analysis based on API call sequences has received more and more attention, especially with the use of deep learning models, such as Convolutional Neural Networks (CNNs) and Long Short-Term Memory (LSTMs) networks. Although these models are shown to have a strong ability in modeling the sequential execution behavior, a lot of the existing approaches are limited to API level features and are not strongly linked to the practical detection tools used in real-world environments. This paper proposes a hybrid malware detection framework using CNN-LSTM-based behavioral modelling and contextual intelligence using the Hybrid Analysis platform. The system takes execution level API call sequences and augments them with light-weight external features such as threat score and antivirus detection counts to make the classification robust and reduce ambiguity in decision making. Furthermore, the learned behavioral patterns are then translated into dynamically generated YARA rules for interpretable and practical deployment, not limited to black box classification. The proposed framework is evaluated with a well-established academic data set created by combining the MalBehavD-V1 and Oliveira API call sequence datasets with 3500 samples. Experimental results show that the accuracy of hybrid CNN and LSTM reach 95.43% with only API sequences, and reach 97.49% when incorporating Hybrid Analysis features and combining the two sets of accuracy will be clearly improved, the discriminative effect will be improved as shown in the AUC metric. These results show that the fusion of deep learning-based behavioral analysis with external contextual intelligence is an effective and deployable malware detection solution which supports dynamic YARA rule generation.