Background and inspiration
Housing organisations hold large amounts of useful information within reports, assessments, and narrative records, but turning that information into structured data can require substantial manual effort. This article explores a proof of concept that combines deterministic rules, locally hosted language models, validation, and human review to make that process more controlled, auditable, and useful.
Although this is a standalone article, it follows naturally from my recent five-part series on the practical implementation of HACT’s UK Housing Data Standards. The same principles carry into this proof of concept: define the output, retain the evidence, validate the result, and keep people responsible for judgement and approval.
Current AI usage
Much of the current conversation about AI in housing understandably focuses on highly visible applications. At one level, people are using tools such as Copilot to help produce reports, document requirements, write SQL, or develop code. At another, organisations are exploring more substantial applications such as predictive maintenance, intelligent triage, forecasting, tenant-facing services, and arrears models.
Many of these applications have genuine potential. But AI does not always need to mean a large platform, an organisation-wide transformation, or a significant technology investment.
There is also a useful space for smaller, tightly defined applications built around problems that already exist: repetitive analysis, difficult-to-access information, manual interpretation, reconciliation, classification, and review.
What can housing practitioners build around the real problems already in front of them?
That question emerged from a practical housing data problem: gaps and uncertainty within stock information, and the need to understand what other evidence might already exist that could help bridge them.
Fire risk assessments presented one potentially valuable source. They are prepared by competent assessors who have inspected, considered, and described buildings for a specific safety purpose. Although they are not intended to function as asset data records, they can contain detailed descriptions of homes, layouts, floors, access arrangements, construction, communal areas, and the relationship between different parts of a building.
This represents a potentially significant source of stock intelligence. The difficulty is that the information is held within narrative documents rather than in a consistent dataset.
Using it conventionally could require someone to open each assessment, locate the relevant passage, interpret the description, record the answer, retain the supporting evidence, and decide whether the result can be relied upon.
Before suggesting this as a practical route for future work, I wanted to know whether the idea was actually viable.
So I built a small proof of concept using created test data and representative examples.
The first question was deliberately narrow: could the process identify the property-type breakdown within a building, including the number of flats, maisonettes, or a mixture of both?
That quickly opened the door to a much wider set of questions about the physical structure of the stock:
- How many homes are described?
- How are they distributed across floors?
- How are blocks, sub-blocks, cores, and entrances related?
- How many storeys does the building have?
- What height or construction materials are recorded?
- What entrances, exits, or escape arrangements are described?
- Does the evidence support or challenge the information held in the asset system?
The proof of concept takes prepared text from fire risk assessments and attempts to convert it into structured, defensible data. In this context, defensible does not mean unquestionable or automatically authoritative. It means that each proposed value has a clear definition, traceable source evidence, a known extraction method, documented validation results, and an explicit review status.
The elements that make candidate data defensible.
The workflow combines conventional rules, including regex, with locally hosted language models. The results are generated independently, compared, checked against validation rules, linked back to the supporting evidence, and separated into those that appear straightforward and those that require human review.
This is not a finished AI product, and it is not intended to be one. Nor is the premise that AI simply fixes the underlying data problem.
The aim is to evaluate whether a language model can be used as one component within a structured and auditable workflow. If the straightforward records can be extracted consistently, supported with evidence, and separated from the questionable cases, the process could save significant manual effort while keeping people responsible for interpretation and approval.
The resulting data is therefore not presented as truth simply because a model produced it. It is presented as a candidate output that can be traced, tested, challenged, and, where appropriate, approved for use.
That leads to the more practical question behind the work:
- What can analysts, business analysts, developers, data engineers, and subject specialists build by combining established analytical methods with language models?
Turning the idea into a testable workflow
A simplified workflow from prepared text to human-reviewed candidate data.
For this proof of concept, I have started from the point at which the relevant text has already been extracted from the fire risk assessment.
In many cases, that can be achieved through standard PDF text extraction. Scanned documents may require OCR, while more complex document layouts may need additional processing to preserve the correct reading order. Those are important considerations for a larger solution, but they are not being tested at this stage.
The immediate question is: Once the relevant text is available, can it be converted into structured, testable, and evidence-backed information?
For the initial proof of concept, that means identifying:
- Whether the building contains flats, maisonettes, or a mixture of both
- The number of flats described
- The number of maisonettes described
- The total number of homes
- Whether the answer is explicitly stated or inferred from the layout
- The evidence supporting the proposed answer
- Whether the result requires human review
Defining the required output in advance is important. The task is not to ask a model to read the text and provide a general summary of the building. It is to ask narrow questions, restrict the possible answers, and require the results in a consistent structure.
For example, the property-type mix must be returned as one of four values:
- Flats only
- Maisonettes only
- Mixed
- Unclear
Counts must be returned as integers or left blank where the evidence is insufficient. The relevant passage must be retained, and the process must record whether the answer was explicitly stated, inferred from a layout, or could not be established safely.
This turns the exercise into a data extraction problem that can be tested and compared, rather than a demonstration of whether a model can produce a convincing description.
It also provides the foundation for expanding the proof of concept.
Once the approach has been tested against property types and dwelling counts, the same structure can be applied to questions about floors, building height, construction materials, entrances, exits, escape arrangements, and the relationships between blocks, sub-blocks, cores, and other physical sections.
Start with conventional methods
The first extraction stage does not use a language model. It uses regex and other deterministic rules to identify explicit statements and clear numerical ranges.
For example, it may detect wording such as:
- “30 two-storey maisonettes”
- “10 self-contained flats”
- “Ground floor: flats 28 to 30”
- “Fourth floor: flats 39 to 42”
This creates a transparent baseline. A deterministic rule can be inspected, tested, corrected, and applied consistently. When it produces an incorrect result, it is normally possible to understand why.
This is particularly important because fire risk assessments contain many numbers that do not represent dwelling totals.
A number could refer to:
- A flat number
- An address range
- A floor
- A building height
- A year
- A date
- A room
- A measurement
For example, flats 21 to 30 represent ten flats within that range. They do not necessarily indicate that the building contains 30 flats in total.
A document may also describe several separate ranges across different floors or building sections. The process may need to calculate each range, avoid double counting, and determine whether the numbering is continuous.
Deterministic rules can manage many of the clearer examples. They can identify explicit quantities, calculate straightforward ranges, and reject numbers that appear to represent heights, dates, or floor numbers.
However, rules will not solve every case. Assessors use different language and different ways of describing building layouts. A total may be implied across several sentences, property types may share one numbering sequence, and the document may not clearly distinguish between the whole block and the section being assessed.
This is where language models may add value. The principle is not to replace conventional analysis with AI. It is to use the simplest reliable method first, then use language models where interpretation is genuinely required. The objective is not to maximise the amount of AI in the process. It is to use it where it adds something that simpler methods cannot provide.
Use language models where interpretation is needed
The more difficult cases are those where the answer is spread across several sentences, implied through the layout of the building, or described using language that varies between assessors.
The same prepared text is therefore passed independently to locally hosted language models. The models do not receive the regex result and are not asked to confirm an answer that has already been proposed. Each model attempts to answer the question directly from the source text.
Local hosting is an implementation choice for this proof of concept rather than a requirement of the approach. The same workflow could use an organisation’s approved cloud or enterprise language-model service through an API. The important part is the controlled process around the model: what information it receives, what it is asked to return, how the result is validated, and how evidence and review are retained.
This separation is important. If the model were shown the regex result first, it could simply repeat or rationalise that answer. Running the methods independently makes the later comparison more meaningful.
Each request is built around a narrowly defined question, a restricted set of permitted answers, and a required output structure. Counts must be returned as numbers or left blank, and the model is not permitted to replace a missing value with an estimate simply because an answer appears likely.
The output structure can also be validated before the response enters the wider process. If a model returns prose instead of the required structure, produces incompatible values, or fails to provide a usable answer, the result can be rejected or flagged rather than silently accepted.
This is a different use of a language model from asking it to summarise a document. The aim is not to generate a persuasive description, but to use the model as a controlled extraction method that converts variable narrative language into candidate data.
That distinction matters because language models are very good at producing plausible answers. A response can be clearly written, confidently expressed, and correctly formatted while still being wrong.
For example, a model may see a reference to flats 1 to 30 and conclude that the building contains 30 homes. That may be correct, but it may also ignore gaps in the numbering, properties outside the assessment scope, or wording that covers only one section of a larger building.
Requiring source evidence makes the interpretation more visible, but it does not prove that the answer is correct. The model output is therefore treated as one proposed interpretation of the source text, rather than as authoritative data.
The purpose of using more than one method is not to create a vote in which the most common answer automatically wins. It is to produce independently generated results that can be examined for agreement, disagreement, and recognisable failure patterns.
Compare the methods, but do not confuse agreement with accuracy
Once the deterministic baseline and the language-model outputs have been produced, the next stage is to compare them.
The simplest cases are those where the methods return the same answer and point to the same supporting evidence. For example, the regex baseline may identify an explicit statement that the building contains 24 flats, while both language models return the same total and cite the same passage.
That agreement increases confidence, but it does not automatically prove that the answer is correct. Several methods can make the same mistake. A rule and two models may all interpret the highest flat number as the total number of homes. Two models may rely on the same ambiguous wording. A clearly stated figure may itself relate to only one section of the building rather than the full assessment scope.
The comparison therefore needs to look beyond whether the final values match. It should also consider:
- Whether the total is stated explicitly or calculated from a range
- Whether the flat and maisonette counts add up to the total
- Whether the same address range has been counted more than once
- Whether the numbering contains gaps
- Whether the document describes the whole block or only one section
- Whether the supporting passage genuinely supports the proposed answer
- Whether any result appears to have confused a property number, floor number, height, or date with a dwelling count
Where the methods agree and the validation checks are satisfied, the result can be presented as a stronger candidate answer. Where they disagree, the process records the result from each method, the fields that differ, the evidence used, any validation warnings, and whether the record should be reviewed.
This makes disagreement useful. It may reveal a weakness in the regex patterns, a recurring model error, an ambiguous source passage, or a question that has not been defined tightly enough.
A document may, for example, clearly support a total of 30 homes without providing enough evidence to divide them reliably between flats and maisonettes. In that case, the total can be retained while the component counts remain blank and the record is flagged for review.
Similarly, one method may identify 20 homes from two address ranges, while another returns 30 because it has used the highest property number. The comparison process can identify the conflict and retain the evidence needed for a person to resolve it.
The intention is not to create a majority vote between methods. It is to produce a controlled reconciliation process that accepts supported results, preserves uncertainty, and makes difficult cases visible.
The value of the process does not depend on automating every record. It depends on handling straightforward cases consistently and directing uncertain cases to someone who can review them properly.
Use people where judgement is needed
A worked example showing how disagreement between methods triggers human review.
Fire risk assessments were not written to create an asset dataset, and an extracted answer should not automatically become an authoritative system value simply because several methods agree.
There will always be records where the source is incomplete, the wording is ambiguous, the assessment scope is unclear, or the document conflicts with information held elsewhere. Those cases still require judgement, but the potential benefit comes from changing how that judgement is applied.
Without a structured extraction process, someone may need to open and review every document in the same way. They must locate the relevant passage, interpret it, record the result, and decide whether it can be relied upon.
Across hundreds or thousands of assessments, much of that work may involve relatively straightforward records. A controlled workflow could separate those from cases where:
- The extraction methods disagree
- The property-type split is unclear
- The component counts do not match the total
- An address range may have been mistaken for a quantity
- The document describes only part of the building
- The evidence is incomplete or contradictory
- The extracted text appears damaged or out of order
- The assessment conflicts with the existing asset record
- The model fails to return a valid structured response
The proof of concept is therefore designed around review, rather than automation alone. Its outputs bring together the proposed answer, source evidence, regex result, model outputs, level of agreement, validation warnings, reconciliation notes, and review status.
A reviewer can see not only the proposed value, but also how it was produced and why it has been presented for attention. They are no longer starting from a blank document because the relevant passage, candidate answers, and reason for review have already been assembled.
The aim is not to claim that hundreds or thousands of assessments can be processed without human involvement. It is to avoid using the same level of human effort on every record.
Straightforward cases can be managed consistently, questionable cases can be prioritised, and subject specialists can concentrate on the records where their judgement adds the most value.
Directing human judgement towards uncertain cases.
The success of the workflow should therefore be measured by:
- How accurately it identifies straightforward cases
- How effectively it catches questionable results
- How much evidence it provides to the reviewer
- How quickly a reviewer can resolve an exception
- How much total manual effort is avoided
That is a more realistic form of human-in-the-loop automation. The workflow performs repetitive extraction and comparison, while people remain responsible for interpretation, challenge, and approval.
Challenge the proof of concept properly
A proof of concept should not be judged by whether it produces a few convincing examples. Language models are good at generating answers that look plausible, even where the interpretation is wrong.
The process therefore needs to be assessed against a manually reviewed set of agreed answers. For the current proof of concept, this includes the expected property-type mix, flat count, maisonette count, total number of homes, and whether review should be required.
The regex baseline, each language model, and the final reconciled result can then be assessed against the same reference data.
The important questions are practical:
- Is the property-type mix correct?
- Is the total number of homes correct?
- Are the component counts correct?
- Does combining methods improve the result?
- Does the process recognise uncertainty?
- Does it send the right records for review?
Evaluating accuracy, review efficiency, and time saved.
A single accuracy figure is not enough. A result might identify the correct total but the wrong split between flats and maisonettes. It might return the correct answer using weak evidence, or several methods might agree on the same incorrect interpretation.
Two errors matter. A false acceptance occurs when an incorrect result is presented as dependable, while unnecessary review occurs when a straightforward and correct result is still sent to a person.
The first creates a data-quality risk, while the second reduces the potential time saving.
Reviewed examples can also reveal recurring failure patterns, such as confusing the highest flat number with the total, double counting ranges, assuming numbering is continuous, combining separate building sections, or forcing an answer where the evidence is unclear.
Those findings can then be used to improve the rules, prompts, validation checks, and review criteria.
The same challenge should be applied to the use of multiple models. Adding another model increases processing time and complexity, so it is only worthwhile where it produces a measurable improvement in accuracy, uncertainty detection, or review efficiency.
Initial testing has included more complex examples where the required counts are not stated as a single total but must be derived from the way properties are described. In these cases, the workflow has successfully identified the total number of homes and the split between flats and maisonettes, including examples where the distinction is expressed through property-number ranges, such as flats 1 to 10 and maisonettes 11 to 20.
These results are encouraging, but they are still preliminary. A larger manually reviewed sample is needed before making broader claims about accuracy, reliability, or the amount of manual effort that could be avoided.
The real test is not whether the workflow is technically impressive. It is whether it can process straightforward records reliably, catch questionable results, produce a manageable review queue, and save enough manual effort to justify the approach.
Build governance into the workflow
The current proof of concept uses locally hosted models and file-based processing. This means that the source text can remain within the local processing environment rather than being sent to an external model service.
That is one deployment option rather than an automatic governance advantage. An organisation might instead use an approved enterprise or cloud-hosted model where appropriate controls, contractual arrangements, access management, and data protections are already in place.
The appropriate approach will depend on the sensitivity of the information, existing technology estate, required scale, model performance, cost, and organisational controls.
Local processing should therefore not be confused with automatic security. It changes where processing takes place, but the same governance questions still apply:
- Who can access the source documents?
- Where are extracted texts and model responses stored?
- How long are they retained?
- Which models and versions were used?
- Which prompts and validation rules produced the result?
- Who is permitted to approve or amend the output?
- Can the process be reproduced and audited?
- What happens when an incorrect answer is identified?
A controlled implementation would require proportionate access controls, encryption, logging, retention rules, backups, and separation between development, testing, and live data.
It should also retain the relationship between every extracted value and its source evidence. A reviewer should be able to identify the document, section, method, model, prompt, and rule that contributed to a proposed answer.
Changes to models, prompts, regex patterns, or reconciliation rules should be versioned and assessed against the reviewed dataset before they are introduced. Otherwise, a small technical change could alter the results without its effect being understood.
The intended use of the output also matters. Candidate data used to support an investigation may require a different level of control from information being written back into an operational asset or compliance system.
Running locally provides deployment options, but the location of the model is only one part of governance. A trustworthy process also requires defined ownership, documented rules, controlled access, evidence retention, testing, human approval, and a clear route for correcting mistakes.
Governance should therefore shape the workflow from the beginning, rather than being added after the proof of concept succeeds.
Consider scale, time, and infrastructure
A successful proof of concept still needs to be considered at the scale of the full document set.
For context, the current proof of concept has been developed using Qwen3 8B, Llama 3.1 8B, and Gemma4 12B models running on a consumer laptop with an RTX 4060 GPU. The machine cost roughly £1,500 rather than being specialist AI infrastructure.
That is not a recommendation for how a production service should be deployed. It demonstrates that organisations can test whether an approach has value before making significant infrastructure or platform investments.
For other organisations, an existing enterprise AI service or API may be simpler, particularly where processing volumes are low or the relevant infrastructure and governance are already in place.
At larger volumes, the economics may change again. Local or internally hosted models can reduce marginal processing costs, while managed services can reduce the infrastructure and maintenance burden. The appropriate choice depends on the workload rather than on a preference for one deployment model.
Larger document collections can be processed in controlled batches, with outputs retained between runs and questionable records brought together for review. This makes it possible to scale the work without requiring every document to be processed immediately or every model to run concurrently.
The practical considerations include:
- The number of documents
- The number of questions asked of each document
- Whether multiple models provide enough additional value to justify their use
- How the work is divided into batches
- How failures and partial runs are handled
- How much intermediate evidence is retained
- The likely size of the review dataset
More powerful equipment, an internal model server, or additional GPUs could reduce processing times and support greater parallelism. These may become useful where rapid turnaround or frequent processing is required, but they are not prerequisites for the approach.
The aim should not be to use the largest available model, the most expensive infrastructure, or the greatest possible amount of AI. It should be to identify the simplest combination of rules, models, infrastructure, and human review that produces the required accuracy and throughput at a proportionate cost.
The business case should therefore consider the entire process. The relevant comparison is between the cost of extracting, validating, and reviewing the information through the workflow and the manual effort required to locate and interpret the same evidence across thousands of documents.
From proof of concept to reusable workflow
This proof of concept began with a question: can information already held within fire risk assessments be turned into structured, testable, and evidence-backed data?
Initial testing suggests that the answer is yes, but not because a language model can simply read a document and provide the truth. The value comes from the wider workflow: defining the question, applying deterministic rules, constraining the models, comparing methods, retaining the evidence, assessing the results, and directing uncertainty to a person for review.
There is still work to do. The questions and regex patterns need further refinement, the value of using multiple models needs to be evaluated, and a larger reviewed dataset is required to assess the process properly across a wider range of documents and descriptions.
The next technical challenge is to make the workflow more adaptable. The underlying structure should remain consistent, while the source text, questions, permitted answers, validation rules, and outputs can be changed for different datasets and use cases.
That could extend well beyond fire risk assessments. The same approach could be used to extract structured intelligence from surveys, financial reports, regulatory documents, news articles, policies, or almost any other source where useful information remains embedded within narrative text.
The individual technologies used here are not new. What has changed is their accessibility. Housing practitioners can now combine established analytical methods with language models and relatively accessible computing or managed AI services to build controlled workflows around real organisational problems.
The important point is not whether the model runs locally or through an enterprise platform. It is whether the technology is proportionate to the problem and whether the surrounding process makes its outputs testable, traceable, and reviewable.
That accessibility makes experimentation possible, but it does not remove the need for evidence, validation, governance, and human judgement. A locally hosted model can still produce an incorrect answer. A structured output can still misrepresent the source. A confident response can still be unsupported.
I am not particularly attached to the term AI. A language model is another analytical tool, and one that can introduce significant risk when it is used without clear questions, controlled outputs, retained evidence, validation, and accountable review.
Used within a properly designed process, however, it can help turn information that is difficult and time-consuming to access into candidate data that is more consistent, traceable, auditable, and useful.
The real test is not whether a model can produce an answer. It is whether the wider workflow can show where that answer came from, how it was tested, where uncertainty remains, and why it should or should not be relied upon.
The model produces an answer. The workflow determines whether that answer is defensible.
This article provides a standalone overview of the proof of concept. Future technical articles will follow the development of the workflow, including document preparation, deterministic extraction, local model evaluation, validation, review tooling, and the route from experiment to a controlled production process. The intention is to show not only what works, but what fails, where uncertainty remains, and where human judgement is still required.
Have a housing data problem?
One aim of these articles is to make the route into difficult housing data problems more visible.
If there is a problem your organisation is struggling to understand, you are welcome to reach out. I will select suitable examples and, with permission, use them as the basis for future articles explaining how I would begin to approach the problem, including the questions I would ask, the data and evidence I would look for, the likely risks and constraints, and how the work could be structured.
Related reading:* My five-part series on the practical implementation of HACT’s UK Housing Data Standards, covering data definitions, data dictionaries, governance, data quality, and standards implementation.