Automatic Smart Contract Generation Through LLMs: When The Stochastic Parrot Fails

Blockchain and Artificial Intelligence (AI) have revolutionized the technology landscape, evolving predominantly along parallel trajectories. This research investigates the feasibility of integrating these two domains, specifically examining the potential of the latest advancements in Large Language Models (LLMs) to assist non-experts in the development of production-ready Solidity smart contracts. Utilizing a lease agreement as a case study, we employ GPT-4 to translate the document into smart contract code. To ensure consistency, we design several distinct prompts with analogous objectives, each employed multiple times. Our evaluation methodology encompasses both automated analysis and expert manual review. The findings indicate a clear limitation: the current iteration of GPT-4 is incapable of generating production-ready smart contracts, primarily due to undetected coding flaws and discrepancies between the prompts and the generated code. This study underscores the challenges and limitations inherent in leveraging LLMs for the autonomous generation of complex, real-world applicable smart contracts.