Impact case study database
Sapienz deployment at Facebook
1. Summary of the impact
Research at UCL led by Professor Mark Harman since 2010 has revolutionised the way that “bugs” are identified in mobile Apps, improving the everyday experience of mobile phone Apps for millions of users. The research led to a spin-out company, Majicke, to commercialise the testing tool designed and developed by Harman, Jia and Mao – called Sapienz. – which was then acquired by Facebook in 2017. Since then, Harman, Jie and Mao started working at Facebook, where Harman founded the Facebook Sapienz team. The Facebook Sapienz team deployed the Sapienz tool into Facebook’s infrastructure, where it was applied at the largest scales experienced in the software industry by any software testing technology. The Sapienz tool remains in full deployment at Facebook where it directly impact on the user experience of 2,600,000,000 people every day who rely on the communications and social networking apps it tests, such as Facebook, Instagram, Messenger and WhatsApp.
2. Underpinning research
Mark Harman co-founded the research field of Search Based Software Engineering (SBSE), an engineering approach—now widely studied across the software sector—which applies metaheuristic search techniques to software engineering problems ( R1). Of particular relevance is his research on a subset of this discipline: Search Based Software Testing (SBST), which concerns software testing and uses computational search techniques to tackle software engineering problems involving large, complex search spaces.
In this approach, test objectives find natural counterparts as the fitness functions used by SBSE to guide automated search, thereby facilitating SBSE formulations of many diverse testing problems. As a result, SBST has proved to be a widely applicable and effective way of generating test data, in addition to optimising the testing process ( R2).
The approach to search-based testing developed by Harman used a novel multi-objective approach to testing that includes two important innovations: firstly, it minimises the size of test cases used in the approach, and secondly, it simultaneously maximises the coverage achieved by the test cases. The first of these innovations—minimising test case size in comparison to other approaches—is important because it maximises the actionability of any faults found in the process (as shorter fault-revealing tests are easier to debug). The second of these innovations (maximising coverage) is significant because it elevates the number of faults that can be discovered. This work built on Harman’s long-standing advocacy for Pareto-optimal approaches being well-suited to software engineering problems, and here used a multi-objective formulation and the well-known NSGA-II algorithms to find Pareto-optimal solutions. This suitability is based on the observation that most software engineering measurements (including all those that later turned out to be relevant to Sapienz) are ordinal scale measurements, and this makes weighted approaches to multi-objective optimisation inappropriate.
In 2015, Harman began working at UCL with former PhD student and current Associate Professor, Yue Jia, and new PhD student Ke Mao (supervised by Harman and Jia) on the problem of applying Search Based Software Engineering to the automated generation of fault-revealing test cases for Android apps to improve their functionality. Based on this research, the same UCL team developed the prototype search-based testing tool, Sapienz, which was released as open source (and which has been further developed by other research teams).
In 2016, Harman and the team published the algorithm and approach to search-based testing at the first-tier software testing conference, ISSTA ( R3). The Sapienz tool was the first technology to target simultaneously both of the competing and conflicting objectives of test size and coverage and has subsequently been taken up by Facebook.
3. References to the research
R1. [CSUV paper] Harman M, Mansouri SA, Zhang Y. (2012) Search-based software engineering: Trends, techniques and applications. ACM Comput. Surv. 45(1): 11:1-11:61
R2. [ ICST 2015] Harman M, Jia Y, Zhang Y. (2015) Achievements, Open Problems and Challenges for Search Based Software Testing. ICST 2015: 1-12
R3. [ISSTA 2016] Mao K, Harman M, Jia Y. (2016) Sapienz: multi-objective automated testing for Android applications. ISSTA 2016: 94-105
4. Details of the impact
Research at UCL, led by Professor Mark Harman has revolutionised the way that software bugs are identified in mobile Apps. The UCL team’s software – Sapienz – has been deployed by Facebook and is improving the user experience of more than 2,600,000,000 users of Facebook, Instagram, Messenger and Whatsapp every day.
During a keynote presentation at the International Conference on Software Testing (ICST) in April 2015, Harman set out a vision for search-based testing ( S1). The UCL team developed a software tool – Sapienz – that uses SBSE to automatically design tests that reveal faults. In September 2016, Harman, Mao and Jia co-founded the start-up company Majicke ltd to ensure their search-based testing methods and their ground-breaking Sapienz testing tool would be applied more widely ( S2).
In February 2017, Harman, Mao and Jia took up full-time positions at Facebook ( S3), founding a new Facebook Sapienz team within the company’s Developer Infrastructure organisation ( S3, S4). At the same time, Harman continued his research and development work at UCL, supported by the European Research Council advance fellowship grant EPIC (Evolutionary Program Improvement Collaborators; ERC grant no.741278), for which he is Principal Investigator.
The first prototype of Sapienz technology was deployed at Facebook in September 2017 and used to test continuous master builds of the main Facebook app and the workplace application ( S5). The research prototype reported on at ISSTA in 2016 had already found 558 bugs in the top 1,000 apps ( S2). When the Facebook Sapienz team was established in 2017 it deployed the Sapienz tool into the Facebook infrastructure and the tool went on to find thousands more bugs ( S5). Tested apps now include the key members of Facebook’s family of apps, including Facebook, Instagram, Messenger and WhatsApp, which, in December 2020, had approximately 2,600,000,000 users each day ( S6).
By February 2018, the prototype had been extended to handle continuous testing of each and every change submitted by developers to the central code repository, rather than simply continuously testing master builds ( S5). In 2018, the Facebook Sapienz team reported that over 700 bugs had already been found and fixed by developers by January 2018 ( S5). By 2018, the Facebook Sapienz team had grown from three to eight staff members ( S5).
The work also generated significant media attention. For example, a blog post that the Sapienz team wrote went viral, and it was covered by SD Times, CNET, SiliconANGLE (further picked up by SlashDot), Startup World, The Register, TechCrunch (further picked up by the Verge), ZDNet, The Next Web (further picked up by Wheaton Business Journal), Fossbytes, and JAXenter. It was also the basis of a high profile Forbes article ( S7). In addition to this media attention, members of the Facebook Sapienz team have also given public lectures on the impact of their work ( S8).
At Facebook, Harman collaborated with Peter O’Hearn to launch two calls for funding to further develop the testing and verification research agenda, supported by funding from the Facebook Research Operations and Academic Relations (ROAR) team ( S9). Through Facebook, Harman and O’Hearn ran three successful symposia, drawing industry and academia together on testing and verification ( S10).
In May 2019, in recognition of this impact, Harman received both the IEEE Harlan Mills award and the ACM SIGSOFT Outstanding Research Award — the first time in 20 years that both these awards had been simultaneously given to the same researcher. This was partly in recognition of Harman’s co-founding of the field of Search Based Software Engineering itself, and partly a reflection of the impact that this research work has had, at Facebook and elsewhere.
In July 2019, Harman was invited to give the opening keynote at the International Symposium on Software Testing and Analysis (ISSTA 2019), on the deployment of search-based software engineering research at Facebook; three years after the publication of the underpinning research at ISSTA 2016. The General Chair of ISSTA 2019, said: “Mark received the IEEE Computer Society’s 2019 Harlan D. Mills Award for his fundamental contributions throughout software engineering, most notably on his seminal contributions in establishing search-based software engineering. In recent years, Mark led the Sapienz team in Facebook to deploy Sapienz to continuously test Facebook’s suite of Android and iOS apps, which has made significant impact in practice. Mark’s great accomplishments in both research and practice made him a perfect keynote speaker at ISSTA 2019, the flagship conference of software testing in the software engineering community” ( S11).
Similarly, the Chair of Committee, IEEE (Harlan Mills Award) said: ”[o]ne of the key reasons Professor Mark Harman received the Harlan Mills award in 2019 was his successful deployment of the Sapienz technology, resulting from many years of research at UCL, into a Facebook technology enabling efficient automated testing” ( S11).
5. Sources to corroborate the impact
S1. [ ICST 2015] Harman M, Jia Y, Zhang Y: Achievements, Open Problems and Challenges for Search Based Software Testing. ICST 2015: 1-12
S2. [ISSTA 2016] Mao K, Harman M, Jia Y. (2016) Sapienz: multi-objective automated testing for Android applications. ISSTA 2016: 94-105
S3. Facebook Research post https://m.facebook.com/academics/posts/1326609954057075
S4. Colligated news feature of the acquisition of Majicke Ltd.
S5. Alshahwan N, Gao X, Harman M, Jia Y, Mao K, Mols A, Tei T, Zorin I. (2018).Deploying Search Based Software Engineering with Sapienz at Facebook. SSBSE 2018: 3-45
S6. Facebook 2020 Q4 Investor Report https://investor.fb.com/home/default.aspx
S7. The Sapienz blog post was covered by SD Times, CNET, SiliconANGLE, Startup World, The Register, TechCrunch, ZDNet, The Next Web, Fossbytes, JAXenter, and Forbes.
S8. Publicly available talks about the impact of the team’s research: Ke Mao gave a talk at the @scale conference, Nadia Alshahwan give an FMATS talk. Mark and Ke gave a joint talk at the F8 developers’ conference (video: https://developers.facebook.com/videos/f8-2018/friction-free-fault-finding-with-sapienz/).
S9. Facebook public announcement of the research award 2019: https://research.fb.com/blog/2019/10/announcing-the-winners-of-the-2019-testing-and-verification-research-awards/
S10. Website of the FaceTAV symposium: https://www.facebook.com/groups/FaceTAV/
S11. Testimonials from the General Chair, ISSTA 2019 and the Canada Research Chair (Tier 1), Chair of Committee, IEEE (Harlan Mills Award).
Additional contextual information
Grant funding
| Grant number | Value of grant |
|---|---|
| EP/J017515/1 | £6,834,903 |