Crypto exchange Coinbase has tested Openai’s Chatgpt as a token verification tool comparing it with its standard security procedure. In over half of the cases, the AI platform produced the same results as the manual review, but it also failed to recognize some high-risk assets.
Chatgpt Approves 5 High-Risk Tokens, Coinbase May Use It for Secondary Control
Digital asset exchange Coinbase has tried the artificial intelligence (AI) chatbot developed by Openai to conduct automated token reviews. The U.S.-based trading platform said that while Chatgpt was not accurate enough to be immediately integrated into its asset review process, it showed enough potential to deserve further investigation.
The experiment is part of Coinbase’s efforts to apply efficient and effective methods to review token contracts before deciding to list the assets. The exchange pointed out that its Blockchain Security team employs in-house automation tools developed to help security engineers in reviewing ERC20/721 smart contracts and explained the AI initiative, saying:
With the emergence of ChatGPT by OpenAI and the buzz around its ability to detect security vulnerabilities, we wanted to test how well this would work as a frontline tool applied at scale rather than just a one off code reviewer.
“Chatgpt has shown promise to be beneficial at improving productivity across a wide range of development and engineering tasks,” Coinbase elaborated. Furthermore, the AI tool can be used to optimize code and identify vulnerabilities.
The leading American crypto exchange carried out the experiment to compare the accuracy of a token security review conducted by Chatgpt to that of a standard review performed by a blockchain security engineer using internal tools. To produce comparable risk scores, the chatbot had to be taught how to identify risks as defined by the platform’s own security review framework.
The researchers compared 20 smart contract risk scores between Chatgpt and a manual security review. While the AI tool produced the same results as the manual review 12 times, out of the eight misses, five were cases in which Chatgpt incorrectly labeled a high-risk asset as low-risk one. “Underestimating a risk score is far more detrimental than overestimating,” the exchange noted in a blog post.
Despite this “worst case failure” and the tool’s tendency to be inconsistent in its answers, when asked the same question multiple times, Coinbase says that the efficiency of the Chatgpt review has been remarkable. The company expects that with further prompt engineering, the accuracy of the tool can be improved.
Currently, the bot cannot be solely relied upon to perform a security review, Coinbase concluded. However it also pointed out that if its team is able to increase the accuracy, a “good first use case for the tool would be to serve as a secondary QA check.” That means that its engineers can potentially leverage it for additional control checks to identify risks that may have gone undetected.
Openai’s Chatgpt platform has been in the spotlight this year amid growing popularity of artificial intelligence applications. Earlier in March, the world’s largest cryptocurrency exchange, Binance, announced the launch of a new, AI-centric, non-fungible token (NFT) platform.
Do you think other crypto exchanges will soon consider employing AI tools like Chatgpt for their risk assessment procedures? Share your thoughts on the subject in the comments section below.
