The UK statistics agency’s failure to produce reliable jobs market data was caused by “systematic” under-investment and “recurring over-optimism”, according to an internal review.
The review, published by the Office for National Statistics on Tuesday, revealed issues with funding, strategy and internal culture that has left the UK without a reliable unemployment figure for more than a year.
The report said the ONS had pressed ahead with cost-saving plans to decommission the survey underpinning the UK’s labour market data despite concerns about its replacement.
The document, which the Financial Times previously sought disclosure of under transparency rules, was made public as the ONS admitted the new “transformed labour force survey” could be delayed until 2027.
The agency also warned on Tuesday of continued “instability” in data based on the old survey that would “take time” to improve, leaving policymakers and investors without a clear picture of the UK jobs market.
Dubbed a “lessons learned” report, the internal review said “significant” budget cuts and “resulting under-investment” have driven problems at the ONS and had a “wider impact on the quality of social surveys”.
Even after “systematic and compounding under-investment in surveys”, the review cited requests for “substantial” budget reductions in 2023-24.
It also pointed to wider cultural issues at the ONS, noting a “pervasive and recurring over-optimism at different levels of the organisation”, “many unchallenged assumptions” that things would turn out right, and a “lack of planning or consideration of worst-case scenarios”.
The ONS, which is run by Sir Ian Diamond, has long used paper-based surveys of the public to find out how many people are employed or unemployed, along with other key statistics about the UK jobs market.
The agency has sought to replace an interviewer-led survey with an online questionnaire. The review found this was initially “a strategy driven by reducing costs” and a desire to exploit new administrative data sources.
The nature of the project changed with the Covid-19 pandemic, which forced the ONS to stop knocking on doors and start conducting the old survey by phone — a change that both reduced and skewed responses.
The response rate slid further after the agency focused its resources on rolling out the new online survey. But last year, it became clear that the data the ONS was producing via phone was no longer reliable.
The review found the ONS lacks the administrative data that was meant to underpin the new survey and avoid the need for long lists of questions that hamper response rates.
The full version of the new survey takes up to an hour for a four-person household to complete, the review found. The survey has high rates of incomplete responses and the ONS is still trying to fix its design.
Senior policymakers have warned of “major concerns” about setting monetary and fiscal policy “in the absence of reliable data”.
The review, dated September 2024, said there was still disagreement within the ONS about its plans for a new survey. “The vision is still not fully clear, shared and accepted at all levels of the organisation,” it said.
The ONS is run by the UK Statistics Agency, an independent government body that receives its core funding from the Treasury.
The review cited “the mandating of hard dates” to decommission the old survey and “budget reductions” as contributing to problems with the labour market data.
Staff interviewed for the review also said that cultural issues within the organisation were “still a live issue”, with “a perceived preference for delivery over quality” and a reluctance to hear concerns.
Sarah Henry, the new head of the ONS’ labour market transformation team, said in a briefing on Tuesday the review had led the agency to put a “comprehensive” action plan in place.
Asked whether even the 2027 delivery date might prove over-optimistic, she said: “We’re definitely not giving a heroic promise here.”
Staff interviewed for the report by the ONS methodology and quality team warned that “the complexity and scale of the transformation were systematically underestimated throughout its lifetime” with milestones set according to “funding availability and cut-offs”.
The review found the project had a “profoundly negative impact on morale and wellbeing” at the ONS, and warned improving the body’s internal culture was “the most critical and pressing need”.
The findings were presented to board members in September. Minutes published in late November said “anxiety” and “a perception that positive messaging was needed at a senior level” had “impacted the ability of colleagues to escalate risks” about the delivery of new employment data.
UKSA board members discussed the need to “encourage candour” at all levels of seniority in response to the findings, and said some concerns about delivery may reflect resistance to change within the organisation, according to the minutes.
Diamond in a statement on Tuesday said: “Our full focus now is on high quality and sustainable statistics for our users and stakeholders.”
The ONS said it accepted the findings of the review, while noting that statistical agencies in many other countries had similar issues with falling responses to surveys.
“We always keep the quality of our surveys and outputs under review to ensure they meet high standards. As the report makes clear, ONS teams are dedicated to making the project a success,” it said.
