Impact case study database
Enhanced image processing techniques to enable the expansion of "street view” technology across China
1. Summary of the impact
Online mapping services can offer street view panoramic images of locations but require accurate and large-scale image manipulation techniques. Cardiff’s research on efficient image processing algorithms, including image graph matching and meta-filters, enabled efficient processing of street view images. In partnership with Tencent Holdings Ltd, the primary street view provider in China, Cardiff’s research enabled the rapid expansion of street view technology to cover 300 cities across China. The street view service is now accessed over six billion times a day through integration with instant messaging and commercial platforms. Cardiff’s integral role in research-led impact with Tencent was recognised by the Chinese government’s highest science and technology honours.
2. Underpinning research
In 2011, multinational conglomerate Tencent launched the first version of its own Street View technology, developed by collecting images of streets using sets of cameras mounted on a fleet of vehicles. Although Street View technology is ubiquitous across the USA, Europe, and Australia through Google, coverage in China remained limited for two years after Tencent’s launch, with restrictions preventing external companies from accessing the Street View sector within China.
To expand the Street View service, Tencent sought the expertise of researchers at the Cardiff University School of Computer Science and Informatics following Cardiff’s strong collaborative history with Tsinghua University since 2001. The Cardiff team were co-led by Hu after starting a joint-professorship between Cardiff and Tsinghua in 2013, and by Martin, who has been a guest professor at Tsinghua University since 2010. The continuing collaborative research with Tsinghua and Tencent began in 2013 and to date has been underpinned by £335,316 in research and travel grants [G3.1 – G3.4] to facilitate co-operative work.
Cardiff’s expertise in the development of image processing and manipulation tools was instrumental in overcoming key technical challenges in four main areas:
2.1 Identifying boundaries and stitching images together
Cardiff researchers developed techniques based on graph matching allowing images captured by individual cameras to be stitched together robustly and efficiently to form Street View panoramas, while research in image extrapolation [3.1] ensured boundary consistency in the composite panoramas.
2.2 Novel image enhancement to improve panoramas
The individual images used to construct panoramas are usually captured from a wide range of viewpoints and under differing lighting conditions, making them challenging to align and blend together. Cardiff research proposed techniques to improve the quality of the generated panoramas consistently across a wide range of input images with minimal manual input, including the use of meta-filters that approximate multiple image filters [3.2] and PatchNets, a hierarchical representation of structural and appearance characteristics of image regions [3.3].
2.3 Panorama completion and blending
Limitations of the capture devices leads to missing content at the bottom of panoramas. For Street View applications, pedestrians and sensitive text also need to be removed. Existing techniques for image completion do not work well due to the large distortion introduced when combining individual images into panoramas [3.4]. Cardiff introduced a novel technique that balances speed and quality by morphing sections of an image to regular dimensions, applying traditional completion methods, before warping the image back to the original perspective [3.5].
2.4 Identifying useful geographic and street information
Although techniques to detect and classify traffic signs exist, this is challenging when the area of interest occupies a small fraction of a larger image. Cardiff research provided a key step in improving accuracy by determining a tighter bounding box around objects of interest [3.6], leading to the creation of a new public dataset of 25,000 panoramas from Tencent Street View. This image information provided the basis for Tencent location services that correct inaccurate GPS location data.
The combined research led to a Cardiff-developed suite of key image processing techniques that underpinned the production, expansion, and use of Street Views in Tencent products and supported its massive expansion and use in recent years.
3. References to the research
[3.1] Miao Wang, Yu-Kun Lai, Yuan Liang, Ralph R. Martin, Shi-Min Hu. BiggerPicture: data-driven image extrapolation using graph matching. ACM Transactions on Graphics, vol. 33, no. 6, pp. 173:1-13 (2014) DOI: 10.1145/2661229.2661278
[3.2] Shi-Sheng Huang, Guo-Xin Zhang, Yu-Kun Lai, Johannes Kopf, Daniel Cohen-Or, Shi-Min Hu. Parametric meta-filter modeling from a single example pair, The Visual Computer, vol. 30, no. 6-8, pp. 67A3-684 (2014) DOI: 10.1007/s00371-014-0973-y
[3.3] Shi-Min Hu, Fang-Lue Zhang, Miao Wang, Ralph R. Martin, Jue Wang. PatchNet: A Patch-based Image Representation for Interactive Library-driven Image Editing, ACM Transactions on Graphics, vol. 32, no. 6, Article No. 196 (2013) DOI: 10.1145/2508363.2508381
[3.4] Zhe Zhu, Jiaming Lu, Minxuan Wang, Songhai Zhang, Ralph R. Martin, Hantao Liu, and Shi-Min Hu. A Comparative Study of Algorithms for Realtime Panoramic Video Blending, IEEE Transactions on Image Processing, vol. 27, no. 6, pp. 2952-2965, (2018) DOI: 10.1109/TIP.2018.2808766
[3.5] Zhe Zhu, Ralph R. Martin, Shi-Min Hu. Panorama completion for Street Views, Computational Visual Media, vol. 1, no. 1, pp. 49-57 (2015) DOI: 10.1007/s41095-015-0008-2
[3.6] Zhe Zhu, Jiaming Lu, Ralph R. Martin, and Shi-Min Hu. An Optimization Approach for Localization Refinement of Candidate Traffic Signs, IEEE Transactions on Intelligent Transportation System, vol. 18, no. 11, pp. 3006-3016 (2017) DOI: 10.1109/TITS.2017.2665647
Selected grants:
[G3.1] Intelligent Processing of Visual Media. Awarded by Engineering and Physical Sciences Research Council £78,802. (01/01/2007 – 31/12/2011).
[G3.2] Structural analysis and interactive composition of visual media. Awarded by Engineering and Physical Sciences Research Council £95,804. (15/01/2012 – 21/04/2017).
[G3.3] Intrinsic voronol/delaunay structure on 2-manifold mesh and its applications in visual computing - Newton Advanced Fellowship that supports collaborations between Cardiff and Tsinghua University. Awarded by Royal Society £110,710. (01/03/2016 - 28/02/2020).
[G3.4] Video scene analysis for virtual and augmented reality. Awarded by Shenzhen Tencent Comp System Co Ltd £50,000. (20/12/2017 – 19/12/23).
4. Details of the impact
Cardiff University has been key to enabling the rapid expansion of Tencent’s Street View technology across China, providing quicker, higher-quality, and more accurate images and location services. The Tencent Street View and location service is now accessed over six billion times a day, widely employed by commercial organisations including DiDi and Mobike, and has received multiple awards from the highest levels of the Chinese government for enabling the successful growth of the service.
4.1 Expanding the coverage of Street View within China
Tencent’s Street View service was first launched in 2011 but was still within the early stages of development by mid-2013, providing panoramic views of only five cities in China, covering 24,000 kilometres (about 15,000 miles). Juliet Wang, Vice President and Director of Tencent University Relations noted that “ combining such images is a highly technical challenge” that required significant resources to ensure accuracy [5.1].
Cardiff’s research provided Tencent with new highly effective methods of “stitching” the images together which “ significantly reduced the need for manual work to compile and check images, saving considerable time and economic costs” [5.1]. Cardiff’s research also provided image-recognition algorithms applied to landmarks and automatically referenced against existing location information. This provision enabled the Street View panoramas to reference the correct location on online maps, significantly enhancing accuracy [5.1].
The improved speed and reliability of combining images “ was a key factor in Tencent's strategic decision to expand the Street View service and Tencent have invested significant resources to expand the number of camera cars and the resources to widen the capture of street images” [5.1]. As of 2020, Tencent’s Street View service now covers over 1 million kilometres of roads, footpaths, and significant locations, providing accessible views of nearly 300 cities across China [5.1].
Tencent stated that “ the expansion of the Street View service has been greatly supported by the contributions of Prof Shi-Min Hu and colleagues at Tsinghua University and Cardiff University”, and cited the contributions of the Cardiff team “ since 2010 including Prof Shi-Min Hu, as well as Prof Ralph Martin, Dr Yukun Lai, and Dr Hantao Liu of Cardiff University”. Tencent confirmed “ the technical insights provided are now integrated within the core of the Street View service and continued to be utilised” [5.1].
- Supporting wider application of Street View technology
The growth of Street View also supported its integration into commercial applications within Tencent and beyond. This includes Tencent’s flagship product WeChat, the most widely used mobile instant messaging app in China with an estimated one billion unique monthly users, and QQ, Tencent’s PC-based messaging app. Street View allows users to quickly share images of their location, scan environments to pinpoint their location, and improve accuracy of a user’s location.
The Street View service has also been integrated into third-party commercial users, with Tencent stating that over 30 external products “ rely on our service” [5.1]. These include:
DiDi, the leading ride sharing service in China;
Meituan-Dianping, the world's largest online and on-demand delivery service;
Mobike, a leading global bike-sharing service headquartered in Beijing.
The improved location accuracy enabled by Cardiff’s research provided commercial users with a greater level of accuracy and quality that enhance the delivery of their products, as well as providing the companies with more accurate location data [5.1]. Wang comments: “ External users of the Street View service enjoy the ability to help customers to experience the views of streets and hotels to promote their sales for hotel booking, and more accurately map the location of their users to provide more efficient localisation for their customers” [5.1].
Tencent highlighted the wider impact of Street View’s third-party integration, stating: “ By providing the Tencent Street View and location service to the external users who are leaders in their sectors, Street View has contributed significantly to the growth of the Internet industry in China as they integrate the service into their own platforms” [5.1]. With widespread integration across multiple Tencent products, and additional value for a range of third-party users, Tencent stated that the Street View service is “ currently accessed over six billion times a day” [5.1].
Enhanced image and location data, enabled by Cardiff research, delivered additional benefits for Tencent’s development of autonomous driving vehicles [5.1]. High-quality panoramic views have been integrated into the Tencent Autonomous Driving 16K dataset, comprising 100,000 images. These images contain 30,000 traffic-sign instances that provide the key learning set to train autonomous vehicle detection systems within China [5.2]. Highlighting their growth in this area, Tencent confirmed that “ the expansion of Street View is continuing to have wider benefits by facilitating Tencent's expansion into new areas” [5.1].
- Recognition and accolades for contributions to Street View technology
Cardiff’s contribution to the successful growth of Street View technology in China was recognised in 2018 by the Chinese Government. Hu was awarded the State Science and Technology Progress Award: the highest award presented by the Chinese Government for research and innovation [5.3]. Hu’s award was specifically cited for his work on the “ Large-scale Street View System and its Key Technical Services” [5.3].
In 2014, Martin was presented with the highly prestigious Friendship Award: the highest award by the Chinese government for " foreign experts who have made outstanding contributions to the country's economic and social progress" [5.4].
5. Sources to corroborate the impact
[5.1] Testimonial from Juliet Wang, Vice President and Director of Tencent University Relations.
[5.2] Li et al., (2017) TAD16K: An enhanced benchmark for autonomous driving. DOI: 10.1109/ICIP.2017.8296701
[5.3] Award certificate: State Science and Technology Progress Award of China for Prof. Shi-Min Hu on “Large-scale Street View System and its location-based services”
[5.4] Award certificate: Friendship award for Prof. Ralph Martin
Additional contextual information
Grant funding
| Grant number | Value of grant |
|---|---|
| N/A | £50,000 |
| NA150431 | £110,710 |
| EP/J009830/1 | £95,804 |
| EP/E034357/1 | £78,802 |