Are bugs in some web browsers holding people back from completing government services?
I recently set out to address this question using data reported to the Performance Platform showing the percentage of people who start and complete all GOV.UK services. Such data is collected to make sure all services are as good as they can be, and that any problems with their performance are caught straight away.
We continuously test all GOV.UK services and make sure they work on different devices and browsers but I wanted to see whether the data we collect could be used to flag up problems inbetween testing.
The investigation centred on analysing the characteristics of those users who start but fail to complete services. For this analysis, I wrote a little Python script called Dropout, which works on all websites running Google Analytics.
Dropout runs a series of queries about users who successfully complete an online task and those who fail. Then it automatically flags up the characteristics (known as dimensions) of each group, eg the device, browser, or operating system people use to access a service.
Afterwards, Dropout runs a chi-squared test to make sure the differences are not purely down to chance before ranking each characteristic in order of significance. Here's a (not very glamorous) screenshot of an output:
I ran Dropout across 8 different GDS exemplar services, looking through the past 60 days for significant results. One of these exemplars - the view your driving licence information beta - flagged up significantly lower completion rates on some mobile browsers.
The completion rate for the service was 30.1% on desktop Internet Explorer, but just 3% (only 14 users succeeded, while 481 failed) on the mobile browser Opera Mini. Similarly, on the mobile UC Browser, just 3 users succeeded, while 242 failed. Unsurprisingly, these results for mobile browsers stood out as highly significant.
Putting these findings to a quick test, I tried to access the service on a few mobile browsers, and quickly found a bug. You can see the bug in action here:
https://www.youtube.com/watch?v=3kjg-oeuGQY
So what's causing the problem? It turned out to be a client-side validation bug. If the user left a space at the end of the postcode field, then the lookup always failed. This probably affected mobile users because phone keyboards often automatically insert spaces after words, as shown above.
The service's bug is now fixed. Of course, bugs would ideally never occur, but anyone who’s ever been a coder knows that's unrealistic and the crucial thing is to spot problems quickly. Google Analytics dashboards can help you spot some problems, but this Dropout script speeds up the process.
A key point about Dropout is that it’s agnostic about your goal - it doesn’t care whether you’re trying to get users to watch a video, use a government service, or buy a car. So potentially, it’s useful for any site manager who wants to find problems affecting completion rates.
You can see the open-source Dropout library on GitHub. Please do contribute suggestions and improvements.
If work like this sounds good to you, take a look at working for GDS – we’re usually in search of talented people to come and join the team.
Sign up now for email updates from this blog or subscribe to the feed.
1 comment
Comment by Phil Wilson posted on
Great tactic, brilliant work.