https://dataingovernment.blog.gov.uk/civil-service-fast-stream-service-assessment/
Civil Service Fast Stream - Service Assessment
This is a digital service to support a multi-stage recruitment and selection process for candidates wishing to apply for the Civil Service Fast Stream and Early Talent Programmes. The Fast Stream is an annual programme recruiting around 1,000 graduates to join the Civil Service as a talent pipeline for future SCS roles, plus outreach and positive action internship activities to encourage applications from under-represented groups such as those from lower socio-economic backgrounds, black and ethnic minority backgrounds and those with registered disabilities.
Department / Agency:
HMRC/CO - Civil Service Resourcing
Date of Original Assessment:
13/2/2015
Date of Reassessment:
30/3/2015
Assessment stage:
Alpha Review
Result of Original Assessment:
Not passed
Result of Reassessment:
Pass
Lead Assessor:
J. Barlow
Service Manager:
S. Morton
Digital Leader:
M. Dearnley / P. Maltby
Reassessment Report
The Civil Service Fast Stream has been reassessed against points 1, 2, 20 and 10 of the Service Standard at the end of the Alpha development.
Outcome of service assessment
After consideration, the assessment panel have concluded that the Civil Service Fast Stream service is on track to meet the Digital by Default Service Standard at this early stage of development.
Reasons
The service team were able to demonstrate to the panel that they had adapted their approach to user research and assisted digital and had made sufficient progress against the recommendations made following the alpha assessment.
User Research - Points 1 and 20
The panel found that the service team had responded energetically and positively to the initial alpha assessment feedback on research. A significant body of research fieldwork was conducted in the six weeks since the original assessment. Three weeks after, a research plan was produced outlining how the team would begin to address the research issues raised at the assessment. It was clear at the reassessment that the initial research outlined in that plan had been carried out.
The 24 user research interviews conducted in that period represent a significant body of work. Key diversity-related user groups - ethnic minorities and people from lower socio-economic backgrounds - have been targeted and accessed (79% of the sample was from ethnic minorities, 46% were from lower socio-economic backgrounds).
Importantly, the respondents were not existing fast-streamers or people already in the Civil Service: they were all people outside, and hence unfamiliar with Civil Service culture and terminology.
Useful findings were obtained in this research, some of which were generic and some of which were directly related to diversity issues (e.g. the need for applicants to feel confident that the diversity questionnaire was unconnected to the application form - that diversity information will not influence chances of success).
The plans in place for continued research in Beta seemed to maintain a focus on the issues raised in the original assessment - namely, ensuring that the bulk of respondents are not from within the Civil Service, and that there is effective targeting of people in the key diversity groups.
The Team - Point 2
The service team informed the panel that a full time user researcher had been appointed until the end of May. To ensure that this is sustained, this role will be included in the broader procurement process which was being planned for beta.
Assisted Digital - Point 10
The service team have completed research to better understand demand and assisted digital needs for their service. The service is working with partners to use a broader range of research techniques to understand assisted digital needs. While more research is required in beta to test initial findings, this research is being used to inform the assisted digital support that is required. It is important that this work continues during beta development.
Although initial findings suggest that assisted digital demand will be low, the panel believe it is important that the service complete further research to ensure that the needs of potentially diverse and hard to reach groups have been fully understood and met. This should include thoroughly testing, and challenging where required, any previous assumptions on assisted digital needs of users.
Recommendations
User Research and the Team
Building on the plan already produced for Beta, the assessment panel felt that there needed to be additional focus on:
- Testing the full integrated journey - registration and online tests (including any e-mails or other comms which feature in this) in one sitting with the same respondent (or in more than one sitting if multiple sittings are also realistic)
- Broadening out the research audience to include disability more broadly - not just in relation to ‘adjustments’ - to ensure that diversity is addressed in a wider sense. (e.g. in relation to general accessibility)
- Ensuring that the bulk of respondents are people who have not been seen before in the research>
Assisted Digital
- Continue to work with specialist partners to make sure the service has understood the assisted digital support needs of harder to reach groups
- Carry out further research with AD users to confirm demand for AD support and test initial findings
- Adapt plans for assisted digital support to meet assisted digital users’ demand and needs on all required channels
Summary
The digital service will play an important role to ensure that the Fast Stream programme attracts a broad and diverse range of people. The panel believe that the service team have made some significant changes to their approach to user research and assisted digital since the last assessment. The service has passed the alpha assessment and is ready to move to beta development. The panel believe that it is important that this work is sustained in beta to ensure the service meets the potentially broad and complex range of user needs.
Original Assessment Report
The Civil Service Fast Stream service has been reviewed against the 26 points of the Service Standard at the end of the Alpha development.
Outcome of service assessment
In spite of many positives and being generally headed in the right direction, the Civil Service Fast Stream service is not yet meeting the Digital by Default Service Standard at this early stage of development.
Reasons
User Research and the Team - Points 1, 2 and 20
The user research conducted in Discovery and Alpha is not sufficient for the service to move onto Beta development.
Given the need to ensure the service is accessible to a more diverse user group, a significantly broader, deeper and more deliberate research effort is required.
The service team should conduct research with a larger, more balanced and more thorough sample. The weighting of the existing sample towards people already in the Civil Service, and towards people who have successfully passed through the existing system (i.e. existing fast-streamers) means that it is likely to be skewed. The experience of people in the target audience who do not (or probably would not) get through the application process should also be properly understood. There may be some of these people among the students who have been interviewed during Discovery and Alpha. However, only three students during Discovery and 10 during Alpha is not enough to ensure the needs of a wide audience are fully understood.
There has been no segmentation of the target audience in relation to specific Diversity issues. Other than contact with three fast streamers with dyslexia, there had been no targeted diversity related research (e.g. focussing on ethnic or socio-economic background).
More generally there has not been enough user research. With three professional researchers on the team, a return of 17 respondents across 6 sprints is relatively low for this stage in the development.
None of the researchers are currently working full time in this role. It may be necessary to consider a full time user researcher. The lack of focus in the research and the low volume of user contact mean there are likely to be significant gaps in the team’s understanding of the issues, both in relation to user needs and to interface issues.
Assisted Digital - Point 10
The service team have focused on the accessibility of the digital service and have made assumptions about potential users’ skills and access based on candidates who currently complete the fast stream application (which is 100% digital). There is a reliance on existing surveys which ask users set questions on whether any adjustments are required.
The service team needs to carry out research to understand whether there are any broader assisted digital needs or barriers to using the service and to plan for developing assisted digital support in Beta. This requirement is only unnecessary if there is demonstrable proof from user research that there are no users with assisted digital needs.
Note also that assisted digital support is only required for the public facing parts of the service, not for internal candidates.
Recommendations for Alpha Review reassessment
User Research
- Increase the focus on user research, taking into account the issues raised above
- Produce a research plan which addresses the research issues raised above. The plan should include reference to how research will be used to explore and resolve the diversity questions issue. i.e. different design approaches explored with a cross section of the audience to identify the approach which best balances user needs (objectivity, trust) with business needs (understanding the impact of the online application process on diversity objectives).
- Consider increasing the user research resource as necessary
Assisted Digital
- Complete user research to understand the demand and needs of assisted digital users
- Use broader range of research techniques to ensure the service team fully understand potential barriers to using the digital service
- Develop a plan based on feedback to provide assisted digital support which meets user needs
Recommendations for Beta
While these are not required for the alpha reassessment, they should be considered during beta development and before coming in for a beta assessment.
Technical
- There are issues with the nature and depth of questions asked of users and the corresponding implications on the data the system would hold. The service team should consider whether the application could capture these anonymously and divorce the socio-economic data from the application itself.
- When it comes to designing and implementing the beta platform there will need to be team member(s) with the necessary skills to own a high level system architecture of a beta system, with all its back office interactions, ahead of the procurement. The concern is that there is no technical architect on the team to perform this piece of work, and especially heading into a procurement this may be problematic in ensuring the correct thing is purchased.
- The program needs to be able to articulate the correct security level and ensure the system meets those requirements and does not include security features that go beyond this.
- There was some outdated requirements on the system architecture diagram (GSi / IL levels) which the team need to ensure they are able to migrate to the current structures (PSN / Official / etc).
Design
We will provide more detailed recommendations on design separately. High-level recommendations are:
- Remove the Diversity and Socio-economic sections to a separate, genuinely anonymous survey. This will address users’ concerns that diversity and socio-economic data is not being used to assess their application, as it won’t be possible for you to do so
- Split everything into individual questions – approx one question per screen. Keep the 5 broad stages for navigation, but have several screens in each. Test this approach with users. We saw it test very well for Register to vote. It makes each question and any validation messages clearer and more focused. It also makes it easier to branch questions and remove steps that are not applicable
- Replace check-boxes with questions – for example ‘Do you have a registered disability? Yes/No’ Do this unless the checkbox is required to accept T&Cs or similar. In this case, use a simple statement ‘By registering, you agree to accept the terms and conditions’. This is legally equivalent but requires one fewer click
- Rather than explaining eligibility criteria up-front, ask the user a question. If the user is eligible, you don’t need to explain the criteria. If they’re not eligible, explain the criteria to show how they failed. If you need more information to decide, reorder the questions so you have the information in the order you require. Ask eligibility questions up-front to reduce the number of users who put in significant effort before getting screened out
- Welsh language: consider instead adding a single page in Welsh which allows Welsh-speaking users to request Welsh-language support. Assess the takeup of this support and overall cost against the cost of providing a Welsh translation
User Research
There may be other issues which are relevant to the Diversity agenda and to the service generally which should be addressed in the research plan. For example, in relation to the online tests which feature in the application:
- How will the team use research to explore how the tests play out in the application process in relation to diversity? Are some groups more likely to fail because of the format?
- How will the team use research to disentangle ‘genuine’ failure in these tests from failure which is an artifact of poor online implementation?
- Which parts of the audience would be addressed in researching these issues, bringing diversity (learning differences, ethnic and socio-economic background etc. ) and other relevant audience dimensions into play?
Next Steps
The service team should follow any recommendations made in this report and see the Government Service Design Manual for further guidance. In order for the service to proceed the assessment panel require a reassessment against the not passed criteria.
Summary
Overall the panel were impressed with the way that the team were working at this early stage in the development of the service. While more user research is required, the team demonstrated that they were able to work in an agile way and were able to quickly respond to user needs.
The panel also felt that the service had taken the right approach technically. The service demonstrated that it has the technical flexibility to deploy code quickly. Security and privacy risks have been carefully evaluated and appropriate steps had been taken to minimise risk. The service was also built using open source and open standards throughout.
The panel felt the service were taking the right approach to the design of the alpha and demonstrated that they were improving the service in line with user needs. However, more in depth user research is required to ensure that the service meets user needs, particularly those of from diverse user groups.
The current design asks users to provide large amounts of socio-economic information. It is important that the service continues to challenge and push back on this where user research demonstrates that the current design does not meet user needs.
The service is heading in the right direction but more work is required on user research and assisted digital support.
Digital by Default Service Standard criteria
| Criteria | Passed | Criteria | Passed |
| 1 | Yes | 2 | Yes |
| 3 | Yes | 4 | Yes |
| 5 | Yes | 6 | Yes |
| 7 | Yes | 8 | Yes |
| 9 | Yes | 10 | Yes |
| 11 | Yes | 12 | Yes |
| 13 | Yes | 14 | Yes |
| 15 | Yes | 16 | Yes |
| 17 | Yes | 18 | Yes |
| 19 | Yes | 20 | Yes |
| 21 | Yes | 22 | Yes |
| 23 | Yes | 24 | Yes |
| 25 | Yes | 26 | Yes |
