The detection of vulnerability exploitation programs has always been a research hotspot in the field of network security. The existing detection techniques have suffered from the problems of accuracy and efficiency in the network traffic environment.

In order to develop a more effective method for detecting and classifying vulnerability exploitation programs in network traffic, this project first studies the honeypot technology based on active defense strategy to collect network traffic, so as to solve the problem of data set imbalance caused by abnormal traffic data being far smaller than normal traffic data.

Then, the traffic data is filtered by kernel principal component analysis and the linear discriminant analysis method is used to extract the secondary features in the new feature space to obtain more accurate feature attributes. After the feature attributes are extracted, the generated features are combined with the asymmetric convolutional autoencoder and random forest algorithm to build a detection model.

A vulnerability attack database construction algorithm based on similarity measurement is studied, and a network traffic-oriented vulnerability attack database is constructed. The data in the database is used as a training set, and a decision tree is constructed to form a classification model through the fast decision tree algorithm, so as to classify the identified attack programs and improve the error and accuracy of other classification methods.

In addition, considering that existing exploitation programs may transmit binary malicious files to carry out network traffic attacks, binary files generating skip-based control flow diagram is studied. According to the characteristics of the exploitation programs of abnormal jump, the relevant definition and the detection method of exploitation programs based on control flow diagram are proposed.

Further, the project studied the classification method of malicious network traffic based on deep learning and the detection method of vulnerability exploitation program. By studying the characteristics of sequential convolutional networks, the project learned the byte sequence characteristics of traffic packets and extracted the key features of the data set, and the trained detection model could obtain higher accuracy in a short training time. Considering the importance of encrypted traffic classification, the project studied the characteristics of TLS certificates of encrypted traffic analyzed by deep belief network, and realized the classification of encrypted traffic by vulnerability exploitation programs.

Finally, a prototype system of vulnerability exploitation program detection in network traffic is developed by integrating the above research techniques. The prototype system can effectively detect typical vulnerability exploitation programs in network traffic.