How effective are smart contract analysis tools? evaluating smart contract static analysis tools using bug injection

Security attacks targeting smart contracts have been on the rise, which have\nled to financial loss and erosion of trust. Therefore, it is important to\nenable developers to discover security vulnerabilities in smart contracts\nbefore deployment. A number of static analysis tools have been developed for\nfinding security bugs in smart contracts. However, despite the numerous\nbug-finding tools, there is no systematic approach to evaluate the proposed\ntools and gauge their effectiveness. This paper proposes SolidiFI, an automated\nand systematic approach for evaluating smart contract static analysis tools.\nSolidiFI is based on injecting bugs (i.e., code defects) into all potential\nlocations in a smart contract to introduce targeted security vulnerabilities.\nSolidiFI then checks the generated buggy contract using the static analysis\ntools, and identifies the bugs that the tools are unable to detect\n(false-negatives) along with identifying the bugs reported as false-positives.\nSolidiFI is used to evaluate six widely-used static analysis tools, namely,\nOyente, Securify, Mythril, SmartCheck, Manticore and Slither, using a set of 50\ncontracts injected by 9369 distinct bugs. It finds several instances of bugs\nthat are not detected by the evaluated tools despite their claims of being able\nto detect such bugs, and all the tools report many false positives\n