https://dataingovernment.blog.gov.uk/e-referrals-service-assessment/
e-Referrals - Service Assessment
The NHS e-Referral Service will replace the current Choose and Book service, which has been in use by NHS in England for 10 years, with over 50 million patient referrals having been completed. The service is designed to support electronic referral in the NHS. The service will make the NHS more efficient, reduce waiting times, make it safer, more convenient, and more secure for patients. e-Referral is looking to support patients booking follow ups, of which there are 60m a year.
Currently there are about 40,000 individual patient referrals completed on Choose and Book each working day with around 45,000 available services published.
Users are GPs and their Practices, NHS Hospital Providers (consultants and their support teams), Independent Sector providers, NHS Mental Health and Community providers, NHS Commissioners and Allied Health Professionals. Patients can only use the service once a clinical decision to refer has been made.
The initial build will replicate the Choose and Book functionality to minimise the impact on the NHS and Independent Sector users. This change will enable many more users in future.
Department / Agency:
HO / NHS England
Date of Assessment:
2/7/2014
Assessment stage:
Alpha
Result of Assessment:
Not passed
Lead Assessor:
J. Hughes
Service Manager:
B. Gildersleve
Digital Leader:
W. Cavendish
Assessment Report
The e Referrals service was assessed the full 26 points of the Digital by Default service standard criteria for a beta service.
The purpose of the assessment is to establish whether the service is on track to meet the criteria for a Live Digital by Default service.
Outcome of service assessment
The overall result of the assessment is that the e-Referrals service is not on track to meet the required standards, and has not passed this assessment.
Reasons and Recommendations
A more detailed breakdown of the points where the assessment panel did not see sufficient evidence that the service is on track to meet the required standards follows, with recommendations for action to remedy these.
1. Understand user needs. Research to develop a deep knowledge of who the service users are and what that means for digital and assisted digital service design.
The service team has clearly carried out extensive stakeholder consultation to help inform important decisions about the service. This has given the team information about what some users and stakeholders want from the service, and has helped the team to cultivate support for the project in a context of highly complex stakeholder relationships and prevailing resistance to change.
However in order to meet the required standard the team should now develop a plan to conduct qualitative and quantitative user research in order to understand more thoroughly the full range of users now and in the future, their needs from the service, and the extent to which the design and build of the service will meet those needs. This should include separate research with patients as distinct from ‘professional’ users, and with a wide range of users within each segment, including people who don’t take part in formal governance mechanisms or organised events. It should include investigating, from the point of view of defined and evidence-based user needs rather than stakeholder opinions, design issues about the choice between optimising for efficiency and optimising for accuracy and completeness in search results.
2. Put in place a sustainable multidisciplinary team that can design, build and operate the service, led by a suitably skilled and senior service manager with decision-making responsibility.
and
19. Build a service that can be iterated on a frequent basis and make sure resources are in place to do so.
There are 2 people who make decisions together about the development of the service, in consultation with various governance groups. The product owner does not have the necessary capacity (being part-time) or authority to make decisions. The product owner is constrained by multiple governance and consultation layers and a lack of ability to change business processes within the service. For example, there are decisions being made about the design of the service on the basis of voting on options by stakeholders, rather than being made by the product owner on the basis of evidence about user needs drawn from user research. This constrains the team’s ability to iterate on a frequent basis.
The service should establish a single individual who is empowered to make decisions on a day-to-day basis. This should include the ability to make decisions about the relationships between the user interface and the business processes that support the service.
The service does not yet have a plan to support and maintain the service once it is live. The service should develop a plan for ongoing support once the service is operational.
9. Create a service that is simple and intuitive enough that users succeed first time, unaided
The team is planning to achieve this by replicating an existing system and fixing defects and making minor improvements, so that further training for users will not be required. However the existing system requires training, and this has partly driven the pressure to replicate it rather than building something different and potentially more intuitive. This has the potential to result in a self-fulfilling requirement to continue the existing user interface.
The team showed some screenshots of alternative designs for the user interface that could be implemented at a later stage, but there are no plans in place to implement them and it was not clear from the information the panel were given how these alternative designs meet specific user needs derived from evidence from user research. The relationship between needs, features / functionality and interaction design was not clearly reflected in the team’s approach to the alternative designs.
To meet the requirement of this standard the team would need to either:
- stop development of the current solution and revisit the discovery phase for the whole project, elucidate the user needs and develop a new service design and user interface, or
- proceed with the release in November but develop a plan, with appropriate resources and authority, significantly to develop the service beyond its initial release in November.
In either case the next phase of work needs to be based on a thorough understanding of user needs (as distinct from ‘wants’ expressed in consultation and focus group type exercises), and an approach to designing and building the service in a way that ensures it is capable of developing and iterating to meet those needs. The team should develop a plan to make sure it has the capacity and authority to do this work.
10. Put appropriate assisted digital support in place that’s aimed towards those who genuinely need it
The service team did not demonstrate that they have a full understanding of assisted digital principles and did not provide any evidence of assisted digital user research. Although they were able to give volumes based on the current service and they plan to continue to provide support through an existing contact centre, there were no plans in place to test that support to ensure that it meets user needs and the assisted digital standard (as detailed in the Service Design Manual).
The service must develop a plan to test their assisted digital support, based on user research and user needs and in line with GDS guidance in the Service Design Manual. GDS will provide a point of contact in the assisted digital team to support this.
7 and 21-24 - Performance measurement
To meet the required standard the service team would need to demonstrate a more fully developed plan to meet the requirements in these 5 points.
The team does not have a plan in place to measure the performance of the service, seeing this as something that can be put in place after the service goes live in November.
The team should bring forward its plan to recruit some data analytics expertise, so that it can establish performance benchmarks and ensure the right data is being collected and reported, and take-up of the service properly extrapolated.
16. Make all new source code open and reuseable, and publish it under appropriate licences (or provide a convincing explanation as to why this cannot be done for specific subsets of the source code)
The service team should analyse what opportunities there are to publish some of the software created during the development of the service. This should be assessed on the potential value to the wider public, rather than just benefits to the service itself. The team should develop a plan to release at least some code having carried out this analysis.
Summary
The service is being developed in a highly complex stakeholder environment. The service team has successfully introduced, through its use of external suppliers, some elements of agile working, and is working towards being able to expand this as it introduces an ability to release more frequently and cost-effectively. The team has achieved some positive outcomes so far in respect of stakeholder engagement, using open source platforms, getting away from vendor lock-in, and contributing to open standards.
As outlined under item 9, there are 2 strategic options available to the team to meet the service standard, given the stage the project is currently at. Our expectation is that the team will want to continue with its planned release in November, in which case the team will need to develop a fully resourced plan for further work beyond November. Beyond November, the team may need to carry out significant additional work to demonstrate compliance with the required standards.
Digital by Default Service Standard criteria
| Criteria | Passed | Criteria | Passed |
| 1 | No | 2 | No |
| 3 | Yes | 4 | Yes |
| 5 | Yes | 6 | Yes |
| 7 | No | 8 | Yes |
| 9 | No | 10 | No |
| 11 | Yes | 12 | Yes |
| 13 | Yes | 14 | Yes |
| 15 | Yes | 16 | No |
| 17 | Yes | 18 | Yes |
| 19 | No | 20 | Yes |
| 21 | No | 22 | No |
| 23 | No | 24 | No |
| 25 | Yes | 26 | Yes |
