The clean-up before the transformation
Across social housing, organisations are undertaking significant digital transformation. Housing management systems are being replaced or upgraded, integrations are being developed, reporting capabilities are expanding, and interest in automation and artificial intelligence continues to grow.
These ambitions are bringing renewed attention to something considerably less exciting, but equally important: the quality of the underlying data.
Many organisations are discovering that the information supporting their existing systems and processes is not as consistent, reliable, or well understood as they need it to be.
Over time, housing systems have accumulated:
- Historic classifications, inherited from previous systems or organisational arrangements.
- Inconsistent definitions, with different departments interpreting similar information in different ways.
- Duplicated or redundant codes, introduced to meet particular operational requirements.
- Undocumented processes and workarounds, developed to overcome system limitations or changing business needs.
Understandably, organisations are investing time and resources into addressing these weaknesses.
Property types are being standardised. Ownership classifications are being reviewed. Redundant statuses are being retired. Records are being corrected and data structures redesigned to support future requirements.
This is necessary work, and often long overdue. However, when working deep within a particular project or departmental process, it is not always easy to see the bigger organisational picture.
A change that makes perfect sense to the team implementing it may have consequences for people, processes, and systems they have never interacted with.
The challenge is not simply improving the data. It is understanding what else depends on the information being changed.
A reasonable improvement, with unexpected consequences
Consider an organisation reviewing the codes used to classify its housing stock.
Over time, its systems have accumulated:
- Overlapping ownership classifications, often reflecting different historical arrangements.
- Multiple property statuses, some with unclear or inconsistent meanings.
- Legacy codes, introduced for specific operational needs that may no longer exist.
- Poorly documented definitions, leaving uncertainty about how values should be interpreted.
A project team works with the relevant department to rationalise these classifications, remove redundant values and improve consistency.
The new structure is clearer, easier to maintain and better suited to future requirements. From the department's perspective, the exercise has been successful.
But where else are those classifications being used?
The same values might underpin:
- Finance: Ownership classifications used to identify stock for financial reporting.
- Asset management and compliance: Property types and statuses used to establish which homes fall within particular programmes.
- Performance reporting: Void classifications used to distinguish routine re-lets from properties undergoing major works.
- Regulatory submissions: Business rules determining which properties are included, excluded, or grouped within reported figures.
These dependencies may exist within SQL queries, reporting models, interfaces, or automated processes developed years earlier.
If a code is changed or retired without updating that underlying logic, the consequences may not be immediately obvious.
The report might still run successfully. It might simply produce the wrong figure.
The original data improvement may have been entirely appropriate. The problem arises when the change is considered only within the boundaries of one department, despite the information supporting activities across the organisation.
It is unrealistic to expect every individual to understand every downstream dependency.
That is precisely why organisations need governance processes that make those dependencies visible before changes are implemented.
When data changes affect regulatory reporting
Regulatory reporting provides a useful illustration of these dependencies.
For local authorities in England, the Local Authority Data Return (LADR) and Local Authority Housing Statistics (LAHS) require accurate information about ownership, tenure, and property status, amongst other characteristics.
Although both returns use housing information, they do not always apply the same definitions, reporting populations, or calculations.
Changes to underlying classifications can therefore influence:
- Which properties are included in reported stock.
- How accommodation is categorised.
- Which records contribute to calculations and performance measures.
- Whether figures reconcile with previous returns and other reporting outputs.
The actual reporting requirements demonstrate how easily these dependencies can be overlooked.
Example 1: Ownership is not the same as management
The LADR distinguishes between the ownership and management of social housing.
Its definition of ownership requires a qualifying freehold or leasehold interest and a direct legal relationship with the occupants.
Properties owned by a local authority but managed through arrangements such as an Arm's Length Management Organisation (ALMO) or Tenant Management Organisation (TMO) must still be reported by the owner.
Relevant reporting requirements include:
- LADR, Stock Information, Questions 1–3 (pp. 9–12): Owned housing units classified by accommodation category.
- LADR, Glossary, Ownership (p. 53): The distinction between legal ownership, landlord responsibility, and management arrangements.
- LAHS, Section A, item a2ia (p. 13): Total housing stock owned within the local authority's Housing Revenue Account (HRA).
Imagine an organisation rationalising ownership codes, consolidating several historic classifications into a simpler structure. A code previously used to identify council-owned properties managed by another organisation might be retired or replaced.
However, an existing reporting query may rely on that code to identify properties that should be included in a regulatory return.
Without reviewing the reporting logic, the change could result in qualifying properties being excluded. The underlying ownership arrangements remain unchanged, but the reported stock figure changes.
A classification change has created an apparent change in the number of homes owned.
Example 2: A vacant property is not always just a vacant property
Property status presents another risk.
An organisation may have several classifications for vacant homes, distinguishing those ready for letting from properties undergoing works or awaiting disposal.
During a clean-up, simplifying these categories may appear sensible.
However, LAHS, Section E, items e2aa–e2dc (pp. 41–42) requires vacant HRA dwellings to be categorised as:
- Available for letting: Including properties undergoing normal maintenance or redecoration between tenancies.
- Temporarily unavailable: Including properties requiring substantive works before they can be re-let.
- Permanently unavailable: Including certain properties awaiting sale or demolition.
A related dependency exists within LAHS, Section G, item g1a (p. 57), which collects average re-let times.
The calculation excludes certain properties, including those undergoing major works, scheduled for demolition or used as temporary accommodation.
Suppose several operational statuses are consolidated into a general vacant category, without preserving the information required to distinguish these circumstances.
An existing reporting process may no longer identify the correct properties for inclusion or exclusion. The resulting figure could affect the regulatory return, internal performance dashboards, and operational targets.
The calculation itself may be technically correct. The problem is that the underlying population no longer reflects the intended definition.
Example 3: Different returns can require different classifications
Not every reporting discrepancy indicates that the underlying data is incorrect.
Different reporting requirements can legitimately apply different classifications to the same property.
London Affordable Rent provides an example.
- LADR, Stock Information, Question 2 (p. 11), and Glossary (p. 52): London Affordable Rent units should be recorded as Affordable Rent.
- LAHS, Section A, items a2ia, a2iaa and a2iab (p. 13): For London boroughs, Greater London Authority tenures such as London Affordable Rent are included in the overall HRA stock figure, but not in the separate Affordable Rent or Social Rent columns.
An organisation might develop a single standardised tenure mapping intended to support multiple reporting processes.
That mapping could be correct for LADR but inappropriate for LAHS.
The underlying tenure information does not necessarily need to be recorded differently. Rather, the reporting processes need to apply the appropriate rules.
Standardisation should establish consistent meaning, not assume that every report requires identical treatment.
These examples demonstrate that the consequences of changing data depend on how the information is interpreted and used, not simply on whether the replacement values are valid.
The same risks apply to private registered providers
These challenges are not confined to local authorities.
Private registered providers, including housing associations, submit the Statistical Data Return (SDR) to the Regulator of Social Housing.
The SDR also depends on accurate classifications of ownership, management, accommodation, and property availability.
For example, the 2025/26 SDR short-form guidance includes:
- Stock and Rent Data, Question 1 (pp. 12–14): Distinguishes low cost rental stock owned and directly managed, owned but managed by other organisations, and managed on behalf of others. It also requires different accommodation types to be identified.
- Stock and Rent Data, Question 9 (pp. 19–20): Distinguishes vacant owned rental units that are available for letting, temporarily unavailable or permanently unavailable.
Larger providers complete a more detailed SDR, introducing further reporting requirements and dependencies.
A provider simplifying its management classifications or vacant-property statuses could encounter similar problems if the changes are not reflected in its regulatory reporting logic.
The broader point applies across the sector.
Information maintained for housing operations may also support asset investment, compliance programmes, financial planning, contractor services, and performance reporting.
A classification introduced for one purpose can gradually become an important dependency within several others.
The reporting logic hiding in plain sight
One practical starting point for identifying these dependencies is to examine how existing reports are produced.
Behind many operational, financial, and regulatory reports sit reusable SQL queries, database views, stored procedures, transformation models, and shared reporting datasets.
These can contain years of accumulated business knowledge, often developed by different analysts and technical teams.
For example, reporting logic might:
- Filter properties using specific ownership, tenure, or status codes.
- Derive classifications by combining information from several fields or systems.
- Apply exceptions for particular accommodation types or management arrangements.
- Calculate reporting populations using rules that are not explicitly recorded in the source system.
Some of these rules will be documented. Others may exist only within the underlying SQL, reporting model, or transformation process.
Consider a query using a fixed list of status codes to identify properties available for letting.
If a clean-up retires those codes and replaces them with new values, the query may continue running without error. It may simply stop selecting some of the properties it previously included.
This can be more difficult to detect than an outright technical failure.
The report still works. The figures look plausible. But the business logic no longer reflects the source data.
Reusable queries are valuable evidence
Rather than examining every report independently, organisations can begin by identifying the shared queries and transformations on which those reports depend.
These are particularly useful during impact assessments because one reusable query may support several reports, dashboards, or downstream datasets.
Before changing critical classifications, it is worth establishing:
- Which queries, views, and stored procedures reference the affected fields or codes?
- Are values hard-coded, maintained in reference tables or derived through calculations?
- Which reports, interfaces and processes depend on that logic?
- Would the proposed change affect filters, joins, aggregations, or historical comparisons?
- Can the existing and revised outputs be compared and reconciled?
Reviewing this logic can also reveal undocumented assumptions, outdated calculations and rules introduced to compensate for weaknesses in historical data.
Those findings should be challenged.
The objective is not to preserve every existing query or workaround. Some may need correcting, redesigning, or retiring as part of the transformation.
But understanding what the existing logic does is essential before deciding what should replace it.
Making the wider organisational picture visible
A change-impact assessment becomes easier when an organisation already understands its data, processes, and reporting dependencies.
This is where established data management practices become useful.
Three resources are particularly important.
Process
Process documentation should explain how information is created, maintained, and changed.
For critical data, this includes:
- The business events that trigger updates.
- The people and systems responsible.
- The rules governing changes, corrections, and exceptions.
Understanding the process helps distinguish how a field is intended to work from how it is actually maintained.
Data dictionaries
A useful data dictionary connects business meaning with technical implementation.
It should establish:
- What a field represents and how it should be interpreted.
- Where it is held and which values are permitted.
- Whether values are entered, imported, or derived.
- Who is accountable for its definition and quality.
- Which codes have been replaced or retired, and when.
This information becomes particularly valuable when classifications are being rationalised or moved between systems.
For a more detailed exploration of data dictionaries, their role in governance and their practical implementation, see my five-part series, Implementing HACT’s UK Housing Data Standards.
Data lineage and reporting dependencies
Data lineage connects source information with the systems, transformations and outputs that depend upon it.
For example, an ownership classification may originate in a housing management system, pass through an integration, be transformed within a reporting database and ultimately contribute to several regulatory and financial reports.
Documenting that relationship allows the organisation to identify where a proposed change may have consequences.
Bringing the information together
Consider the earlier example of retiring ownership codes.
An organisation could use:
- The data dictionary to establish the meaning of the existing codes and their proposed replacements.
- Process documentation to understand how those classifications are created and maintained.
- Reporting queries and lineage to identify the calculations, integrations and outputs that depend on them.
- Governance arrangements to involve the appropriate business owners and agree the change.
The result is a more informed assessment of what needs to change, where, and why.
Importantly, this does not necessarily require expensive specialist technology.
Organisations can begin with controlled documentation, accessible data dictionaries, existing code repositories, and a structured approach to recording dependencies.
The value comes from maintaining and using that information, rather than simply producing it.
Governance should enable change, not obstruct it
Improving data foundations should not become an unnecessarily bureaucratic exercise. Not every field amendment, code replacement or correction requires formal committee approval.
The level of oversight should reflect the importance of the information and the potential consequences of changing it.
For critical data, a proportionate change process should address four questions.
1. What exactly is changing?
Is the organisation correcting incorrect records, renaming a classification, changing its definition, or redesigning the underlying data structure?
A change to a label may have different implications from a change to the meaning of a value.
2. What depends on it?
Identify the relevant systems, processes, integrations, reusable queries, and reporting outputs.
This should include dependencies outside the department initiating the change.
3. Who needs to be involved?
Technical teams may implement changes, but they should not automatically become responsible for deciding their business meaning.
Relevant data owners, operational specialists, and reporting leads should be consulted where their processes are affected.
4. How will the change be tested and recorded?
Depending on its significance, this may include:
- Comparing stock populations and key calculations before and after implementation.
- Testing affected queries, interfaces, and reporting outputs.
- Preserving historical classifications, mappings, and effective dates.
- Recording approvals, exceptions, and outstanding issues.
A minor correction might need little more than a documented update. A change affecting regulatory reporting, compliance populations or significant financial calculations may require formal assessment and approval.
The objective is not to prevent departments from improving their information. It is to ensure that significant changes are made with an appropriate understanding of the wider implications.
Improving the foundations without losing sight of the organisation
The growing interest in automation, analytics and AI has given social housing organisations another reason to examine the quality of their data.
That is a positive development.
But reliable foundations are not created simply by cleansing records, introducing consistent classifications, or moving information into a more modern platform.
They also require an understanding of the processes, calculations and decisions that depend on that information.
When working within a particular team or transformation project, it is natural to focus on the immediate task. The wider consequences may sit several departments, systems, or reporting processes away.
The people undertaking the work cannot reasonably be expected to know every dependency.
That is why effective governance matters. It provides the structures through which organisational knowledge can be connected, responsibilities established and significant changes assessed.
Transformation should challenge outdated data structures and inefficient processes, not preserve them indefinitely.
But there is a considerable difference between deliberately replacing a dependency because it is no longer appropriate and unintentionally breaking one because nobody knew it existed.
A successful data clean-up should improve the information available to the organisation without introducing avoidable problems elsewhere.
Before changing an important piece of data, perhaps we should ask two questions:
What are we trying to improve?
And who else relies on what we are about to change?