Automated State-Machine-Based Analysis of Hostname Verification in IPsec Implementations

Abstract

Owing to the advent and rapid development of Internet communication technology, network security protocols with cryptography as their core have gradually become an important means of ensuring secure communications. Among numerous security protocols, certificate authentication is a common method of identity authentication, and hostname verification is a critical but easily neglected process in certificate authentication. Hostname verification validates the identity of a remote target by checking whether the hostname of the communication partner matches any name in the X.509 certificate. Notably, errors in hostname verification may cause security problems with regard to identity authentication. In this study, we use a model-learning method to conduct security testing for hostname verification in internet protocol security (IPsec). This method can analyze the problems entailed in implementing hostname verification in IPsec by effectively inferring the deterministic finite automaton model that can describe the matching situation between the certificate subject name and the hostname for different rules. We analyze two popular IPsec implementations, Strongswan and Libreswan, and find five violations. We use some of these violations to conduct actual attack tests on the IPsec implementation. The results show that under certain conditions, attackers can use these flaws to carry out identity impersonation attacks and man-in-the-middle attacks.