Security auditing today is usually done manually by expert security researchers. Despite the fact that security auditing has been done on Internet products for many years, few automated tools have been created for this purpose. Today, these products are usually in-house tools created by security researchers to assist them during their audits. Products like these usually require a high expertise level in usage, and convey very little machine "intelligence." Most of those tools have no automated functionality beyond assistance to the manual analysis nor do they do a full and comprehensive analysis of the protocol.
Other alternatives include source code audit software. These solutions search the source code for what appears to be bad code that could indicate a potential security hole. And, there are other security testing tools that rely on a relatively small number of case studies (thousands or tens of thousands) that are known to trigger buffer overflows in certain protocol implementations.