Will higher taxes stifle UK job creation? Will employers still hire young people if they have to pay a higher minimum wage? Are health problems really shutting nearly 3mn people out of the workforce?
UK ministers need urgent answers to these questions as they press ahead with changes to taxes and benefits, wages and workers’ rights.
But if the Labour government’s policies are undermining the jobs market, it will be hard to tell — because for two years or more, the UK’s official labour force survey has been misfiring.
Declining response rates to the LFS have made the numbers so volatile that it is impossible to be sure whether employment is rising or falling from one quarter to the next — let alone how the labour market has evolved in the years since the pandemic.
“The UK’s main source of labour market data is unusable — it will take another two years to fully resolve the problems,” says Andrew Goodwin, chief UK economist at the consultancy Oxford Economics.
The failure of the UK’s largest household study to provide reliable data leaves policymakers with “major blind spots”, he says, at a time when developments in the jobs market are crucial to the UK’s economic performance and the Bank of England’s decisions on interest rates.
“It is a substantial problem — and not just for monetary policy — when we don’t know how many people are participating in the economy,” BoE governor Andrew Bailey told a gathering of City dignitaries in November.
No permanent fix is guaranteed before 2027. “If the proposed timetable holds, the UK will endure a four-year period without reliable data for its key labour market series,” Goodwin notes.
The labour force survey is one of the most critical outputs of the Office for National Statistics, the government agency that constitutes the UK’s main source of official data on the economy, population and society.
The survey’s collapse is turning into a cautionary tale for statistics agencies around the world that share a common problem: people’s growing reluctance to take cold calls, share personal data or spend time filling in surveys.
It also highlights the risks of wider problems emerging with the national statistics that underpin policymaking, just as the Labour government has made economic growth its core mission.
The ONS is struggling to run other household surveys; it has made big revisions to closely watched figures on migration and earnings; and it is approaching a critical decision on how far it can replace the UK’s once-a-decade nationwide census with real-time data held by government.
Finally, it exposes the degree to which sustained funding cuts and low staff morale have strained the capacity of UK institutions such as the ONS — which has had a much steeper fall in survey responses than international peers.
Sir Charlie Bean, the former BoE deputy governor who led a review of UK economic statistics in 2016 after a similar series of mishaps, says the ONS acted at the time to become more “proactive, on the ball and self-questioning”. Now, he says, “there is a sense they have slipped back”.
The labour force survey saga “has done quite a bit of damage to the ONS’s reputation”, he adds.
The labour force survey is designed to give a comprehensive picture of employment in the UK. Every three months, it gathers details from around 25,000 households about people’s working hours and earnings, their family and health, the prevalence of second jobs and zero-hour contracts, and much more besides.
Response rates to the survey were already slipping a decade ago. But the Covid-19 pandemic turned a chronic problem into an acute one: it disrupted face-to-face interviews, just as lockdowns shook up working life and Brexit changed the pattern of international migration.
The combination made it hard to tell whether new trends — such as a big rise in the number of people saying that health conditions stopped them working — were real, or a statistical mirage caused by some groups becoming less likely than others to respond.
Initially, the ONS managed to stave off disaster: it switched to telephone interviews, bolstered its field force and reweighted the data, when it found, for example, that it was reaching too many homeowners and not enough renters. At the same time, it was accelerating work to replace the LFS with a new online-first survey intended to supplement other data sources.
But in mid-2023, officials made a critical mistake: pulling resources from the old survey to focus on the new one. Responses to the LFS plunged. In October that year, it became clear it was producing results that could not reflect reality, forcing the ONS to pause its monthly publication.
Sir Ian Diamond, the UK’s national statistician, whose wide-ranging role includes strategic oversight of the ONS, says he did not know how fragile the survey was until it reached this tipping point.
“The issue is bias — bias that you are not able to control for . . . We believed we could manage it,” he told the Treasury committee this week.
Yet efforts to repair and replace the LFS remain beset by delays.
The ONS has resumed publication of the figures, but they are badged as “official statistics under development”. Although Diamond says repairs will start to bear fruit from May onwards, it may take until 2027 until the successor to the LFS — the online-first “transformed” labour force survey — is up and running.
Although alternative sources can fill some gaps, that leaves the UK with no reliable count of unemployment, economic inactivity or self-employment — let alone detailed data on local jobs markets, youth employment, precarious work or a host of other topics.
The faulty data has already created problems. For several years, it led ministers to focus on an apparent surge in the number of people shut out of the workforce by poor health.
Many think that now looks like a misdiagnosis. The BoE no longer believes there has been any significant rise in economic inactivity, although there has been a sharp increase in the number of people who claim health-related benefits.
In January, Lord George Bridges, then chair of the House of Lords economic affairs committee, said that the “flawed” data had led ministers to focus resources on the wrong problem — seeking to cut NHS waiting lists in specific local areas where economic inactivity was high, rather than fixing skewed incentives in the welfare system.
Alison McGovern, the employment minister, earlier told the committee that, while the data gaps were “very worrying”, they had not led to any policy mis-steps. Diamond, meanwhile, says he remains confident this trend is real.
But the absence of reliable data also means the government cannot tell how its policies are affecting employment. It is pressing ahead with double-digit increases in the youth rates of the minimum wage, for example, without a clear view of whether this is driving up youth unemployment.
The Low Pay Commission, which advises government on where to set the wage floor, warned ministers before the Budget that the data problems “make our job more difficult and our advice less credible”.
At the BoE, Bailey and other rate-setters have warned that flawed data makes it harder to tell whether high inflation (now driven largely by wage growth) is becoming entrenched, increasing the risk that the central bank will keep interest rates high longer than needed and destroy jobs.
The BoE, government departments and agencies, and independent researchers are now all developing their own internal data as delays in mending and replacing the LFS lengthen.
MPs worry the labour force survey might be the tip of an iceberg. In a letter to the Treasury committee in the House of Commons, Diamond described wider, systemic issues that contributed to the debacle — including funding constraints that left managers overloaded, an internal culture where staff felt unable to raise concerns, and failures of strategic direction.
Dame Meg Hillier, the committee chair, tells the Financial Times that she and her fellow MPs had been “dumbfounded” by the letter, which suggested there could be other data gaps that policymakers were not yet aware of.
Some potential risks are already visible. The ONS has had to take emergency action to shore up responses to its living costs and food survey, an important input to GDP and inflation data. It is reviewing the way it produces earnings data used to calculate the minimum wage, after the Low Pay Commission flagged concerns over its quality.
And it has repeatedly made big revisions to politically sensitive estimates of international migration — a sign of how difficult it has proved to move away from old-style passenger surveys to better measures of migration based on administrative data held by government.
“It is a real worry,” Hillier says of the LFS issue. “If you have a data gap here, what other data gaps might exist? What could the implications of that be for forecasts?”
The problems with data collection are mirrored by problems within the ONS itself. An internal review of what went wrong with the labour market data, published in December, says the quality of its output has been undermined by a broader institutional malaise.
Under Diamond’s leadership, the overarching strategic goal has been to shift away from surveys, which have grown more expensive, to more comprehensive, timely administrative data held by HMRC, the Department for Work and Pensions and other parts of government.
This led the ONS to make significant cuts to the budget for surveys in 2022-23, at a time when surging inflation was driving up costs and it faced a need to make efficiency savings totalling more than £40mn by 2025.
But at the same time, the ONS found it much harder than expected to gain access to data sets held by other government departments. This left the department overburdened as the cuts kicked in.
These pressures weighed heavily on staff. The internal review found people working on the labour market project were reluctant to alert managers to problems, because of “a perceived preference for delivery over quality” and a fear it would be seen as “a pessimistic response to the ambitious organisational objective”.
Diamond says the agency is addressing these “historical cultural issues”, encouraging staff to raise concerns and inviting “constructive challenge”.
But unions say concerns over culture have not been resolved and there is widespread disquiet about repeated changes in strategic direction and the long-term objective of replacing surveys with administrative data.
Fran Heathcote, general secretary of the PCS union, says members working at the ONS “tell us the constant changes to management programmes make their work feel chaotic”.
She adds: “There’s a sense within senior leadership that [the shift to using] administrative data solves everything, but it doesn’t.” They have been a “bit too gung-ho”, she adds.
Several ONS employees who spoke to the FT anonymously say there is discontent over pay — lower at the ONS than at almost any other government agency — and over new attendance rules at its Newport headquarters and other offices.
Many middle managers also feel a lack of agency, they say, and people in technical roles grappling with the difficulties of sharing data across government departments often feel their concerns are not recognised by senior leaders.
Some employees are now voting with their feet. About 30 per cent of the ONS’s force of field interviewers left the organisation in 2023, according to the union Prospect, which represents many of them. This is a much higher rate of churn than in the past, and the union says there has been no noticeable improvement over the past year.
The ONS has also seen a steady outflow of mid-ranking analysts who represent the core of its workforce.
Figures seen by the FT show the number of government economists working at the agency fell by almost 30 per cent in the year to October, while that of government statisticians fell by 6 per cent and of social researchers by 14 per cent. The number working in the policy profession fell by 36 per cent. The ONS said these numbers reflected its return “to a sustainable long-term operating model” after it expanded its headcount to deal with the 2021 census.
Senior figures in the tight-knit statistical community are reluctant to blame Diamond openly. But several noted that it had always been difficult to fill the role of national statistician, as few people had the combination of qualities it required: academic credibility, experience of managing a large organisation and communication skills.
A senior government official also acknowledges Diamond had been too ready to downplay the difficulties of the shift to administrative data.
“It hasn’t been the golden bullet people thought it might be,” the official says, adding that Diamond had shown “a hesitancy to pivot when the challenges . . . became clear”.
Diamond made clear this week that the pressures of repairing the LFS and ensuring the quality of the ONS’s output weighed on him too. “Please, please, please don’t think I am being complacent,” he told MPs. “I lie awake at night worrying about this the whole time.”
Against this backdrop, there is growing concern among statisticians over the next big decision looming at the ONS: on the future of the census.
For more than 200 years, this nationwide poll has produced population estimates and documented changes in society, helping determine how councils, charities and central government provide services.
The agency ran a consultation in 2023 on plans to scale back the once-a-decade exercise, relying instead on a range of government data sets to compile the official count of the population.
But it has become clear that a survey of some kind will still be needed to collect detailed, local data unavailable elsewhere — on people’s ethnicity and language preferences, for example.
The ONS will soon publish a recommendation to government on how best to achieve this, but in the meantime it is doing what it can to fill the data gaps and restore its reputation.
It is restoring funds for surveys, recruiting more interviewers and investigating ways to overcome people’s aversion to filling them in — whether through digital innovation, or by making it a legal obligation.
It is bolstering its senior team, and urging the Treasury to fund a pay rise for its field force. Crucially, given funding pressures, it is scaling back its output to focus on essentials.
“We are not going to reduce on quality,” Diamond said this week. “We are so clear that policymakers need to make decisions based on the best data.”
Additional reporting by Sam Fleming
