Unit Test Case Generation with Transformers and Focal Context

Speaker: Michele Tufano, Senior Research Scientist at Microsoft

Abstract:
Software testing is a critical part of software development life cycle which helps identify potential regressions and reduce maintenance costs, yet it is often neglected by developers. Automated unit test case generation tools facilitate test-driven development and support developers by suggesting tests intended to identify flaws in their code. Existing approaches are usually guided by the test coverage criteria, generating synthetic test cases that are often difficult for developers to read or understand. In this talk we'll present AthenaTest, an approach that aims to generate unit test cases by learning from real-world focal methods and developer-written test cases. We formulate unit test case generation as a sequence-to-sequence learning task, adopting a two-step training procedure consisting of denoising pretraining on a large unsupervised Java corpus, and supervised finetuning for a downstream translation task of generating unit tests. We investigate the impact of natural language and source code pretraining, as well as the focal context information surrounding the focal method. We found that both techniques provide improvements in terms of validation loss, with pretraining yielding 25% relative improvement and focal context providing additional 11.1% improvement. We also introduce Methods2Test, the largest publicly available supervised parallel corpus of unit test case methods and corresponding focal methods in Java, which comprises 780K test cases mined from 91K open-source repositories hosted on GitHub. We evaluated AthenaTest on five defects4j projects, generating ~25K passing test cases covering 43.7% of the focal methods with only 30 attempts. We execute the test cases, collect test coverage information, and compare them with test cases generated by EvoSuite and GPT-3, finding that our approach outperforms GPT-3 and has comparable coverage w.r.t. EvoSuite. Finally, we survey professional developers on their preference in terms of readability, understandability, and testing effectiveness of the generated test cases. The results show that developers overwhelmingly prefer test cases generated by AthenaTest over standard EvoSuite's, suggesting that our approach could significantly improve developer productivity.

Biography:
Michele Tufano is a Senior Research Scientist at Microsoft in the Data & AI team, building AI-assisted developer tools. He graduated with his B.A in Computer Science in 2012, and with his M.S. in Computer Science in 2014 at the University of Salerno. He obtained his Ph.D. in Computer Science from William & Mary in 2019, advised by Dr. Denys Poshyvanyk. His main research interests include software engineering and applications of machine learning, with a focus on automated software testing. His current effort involves the design, training, and deployment of Transformer-based models to automate software testing activities for developers.

Host: Prof. Gabriele Bavota