Recent research conducted by two scholars from Salus Security, a blockchain security firm operating across North America, Europe, and Asia, showcases the capabilities of GPT-4 in parsing and auditing smart contracts. While artificial intelligence (AI) proves adept at generating and analyzing code, caution is advised against solely relying on it as a security auditor.
The study underscores that while GPT-4 can serve as a valuable asset in smart contract audits, particularly in tasks like code parsing and offering vulnerability hints, it falls short in comprehensive vulnerability detection. Despite its potential, the AI is unable to fully replace professional audit tools and the expertise of experienced auditors.
Salus researchers employed a dataset comprising 35 smart contracts, referred to as the SolidiFI Benchmark Vulnerability Library, containing a collective tally of 732 vulnerabilities. This dataset served as the basis for evaluating GPT-4's efficacy in identifying potential security risks across seven prevalent vulnerability types.
Results indicate that ChatGPT demonstrates proficiency in detecting genuine positives, or actual vulnerabilities meriting further investigation outside of controlled test environments, achieving a test accuracy of over 80%. However, the AI exhibits a notable issue in producing false negatives, as reflected in its low recall rate, which in the Salus team's experiments plummeted to as low as 11% for GPT-4.
Consequently, the researchers assert that GPT-4's vulnerability detection capabilities are deficient, with a maximum accuracy rate of only 33%. Hence, they advocate for the continued utilization of dedicated audit tools and the indispensable involvement of human expertise in smart contract audits until AI systems like GPT-4 can bridge this gap.
In summary, while GPT-4 can play a supportive role in smart contract audits, particularly in tasks such as code parsing and offering vulnerability hints, it should be employed in tandem with other audit methods and tools to enhance overall accuracy and efficiency in auditing endeavors.
