Homes England is an executive non-departmental public body,
sponsored by the Ministry of Housing, Communities and Local
Government. The mission at Homes England is to accelerate the pace
of house building to provide affordable, quality homes that improve
people’s lives. A team of data engineers are enabling more accurate
and timely decision making through building and maintaining a robust
data platform that empowers the entire organisation.
Why Every Great Data Strategy Starts with a Data Engineering Team
A machine learning model required by a data scientist, or a data
visualisation used by a decision maker, has a common denominator –
an unsung data engineer! There is no technology nor fancy tool that
can replace an accurate, timely dataset that has securely been ingested
through a reliable and robust data platform. Data engineers are the
foundation building block that enable others to do their job well. A
summary is provided below:
- Automation: Replace outdated, manual processes with automated, modern technology, to save time and reduce human error.
- Improved Data Quality: Implement automated validation checks to help catch errors early and ensure consistency across datasets.
- Security and Compliance: Embed security best practices to ensure sensitive data is handled appropriately and access is controlled.
From Raw Data to Real Impact: What Is Homes England’s Data Platform?
Imagine a user who wants to analyse a dataset. They might log into a
website, download a file, and then create graphs or run some statistics
to uncover insights. At first glance, this seems like a reasonable
approach. But if they need to repeat this process daily, share the data
with colleagues, or combine it with other datasets, the workflow quickly
becomes time consuming, error prone, and potentially
insecure.
This is where the power of Homes England’s data platform comes into
play. It provides a centralised, secure, and scalable environment that
automatically ingests, processes, and stores data across the
organisation - eliminating repetitive manual tasks and enabling faster, more reliable access to trusted data.
The platform currently supports the ingestion of data from a wide range
of sources, including databases such as SQL Server, PostgreSQL and
Oracle. In addition to databases, it can also pull data from: Application Programming Interfaces (APIs), Microsoft Dataverse, Microsoft SharePoint sites and Secure File Transfer Protocol (SFTP). The platform is scalable and adaptable to accommodate new data types as needs evolve.
Now the data has been ingested, it needs to be stored. Homes England
uses a cloud based storage solution, which ensures the platform
remains scalable as data volumes increase. The data is stored in Delta
format, a powerful structure that offers several advantages: it supports
efficient querying, simplifies version control through built in delta history
tracking, and integrates seamlessly with analytics tools. Within
Databricks, this data is presented as relational tables, making it easy for
users to explore and work with.
This first stage of storage is known as the “raw” layer, as it captures the
data exactly as it was received from the source. To prepare this data for
meaningful analysis, the data engineering team collaborates closely
with data architects and agile project teams. Together, they transform
the raw data into structured fact and dimension tables, forming what’s
known as the “curated” layer. This refined layer is optimised for analysis
and is what end users interact with to generate insights and drive
decisions. What starts as complex, varied data sources, is transformed
into a consistent, tabular format - clear rows and columns that are
intuitive to navigate. This makes the data not only accessible but also
reliable for analysis and decision making.
Under the Hood: The Tech Powering Homes England’s Data Platform
Homes England leverages a modern suite of technologies to power its
data platform:
- Azure Data Factory orchestrates data pipelines, managing the
ingest of data from various sources. - Azure Data Lake Storage ensures reliable, ACID-compliant
storage. - Databricks provides a scalable environment for data processing
and collaborative analytics. - Terraform is used for infrastructure as code (IaC), supporting
consistent and automated deployment through CI/CD practices.
This technology stack ensures the platform is robust, scalable, and well
suited for handling complex data workflows.
Looking Ahead: Future Improvements
We’re continuously enhancing our data platform to stay aligned with the
latest technological advancements. One of our upcoming initiatives is to
pilot asset bundles in Databricks, aiming to streamline and simplify our
data ingestion processes.
If you’re interested in sharing your own data engineering journey, or if
you have questions about how data engineering can support your team,
we’d love to hear from you. Please get in touch at pippa.harflett@homesengland.gov.uk.
Our mission is to highlight the vital role of data engineering in delivering
timely, accurate data that supports the public good. We hope you’ll join
us in spreading that message.
Leave a comment