Historic England is commemorating the American abolitionist Frederick Douglass’s time in Newcastle in a project that links the legacy of black abolitionists with listed buildings.

The story of how black Americans came to Britain to fight slavery has still not been fully recognised. The Missing Pieces Project aims to shed new light on the struggle by charting the locations on the lecture tours of 19th-century activists.

In church halls, factories and theatres across Britain, Christians, workers, radicals and liberals came to hear African American abolitionists talk and show solidarity with the cause. Now, buildings in 189 cities, towns and villages have been added to Historic England’s Missing Pieces Project, which uncovers overlooked stories behind historic sites with an interactive online map.

Douglass, a writer, reformer, orator and a seminal figure in American civil rights who escaped enslavement, travelled to Britain and Ireland three times. Among the buildings Douglass visited was the music hall at Nelson Street, Newcastle, which, testifying to Tyneside’s radical past, was also visited by the activists William Wells Brown, William Craft, Henry Highland Garnet and Moses Roper.

Buckingham Palace is on the map – having been visited by the “Black Swan”, the singer and activist Elizabeth Taylor Greenfield, who performed for Queen Victoria in 1854. Greenfield was born in enslavement and worked with British aristocrats to end slavery.

Sarah Parker Remond, the free-born feminist who refused to conceal the sexual violence of slavery, played a leading role in securing support in Manchester for a boycott of Confederate cotton, telling an audience: “When I walk through the streets of Manchester and meet load after load of cotton, I think of those 80,000 cotton plantations on which was grown the $125m worth of cotton which supply your market, and I remember that not one cent of that money ever reached the hands of the labourers.”

Now, 165 years later, Parker Remond’s visit to Leeds town hall, on the same tour, is recorded by the Missing Pieces Project.

The project reveals that the fight against American slavery was not confined to urban centres. Moses Roper went from Cornwall to the Scottish Highlands, resolving to “tell the truth” of the system’s brutality, with his autobiography selling thousands of copies in Welsh. His lectures at Wattisham Baptist chapel, Suffolk, on tours between 1838 and 1844, are charted on the Missing Pieces Project map.

Dr Hannah-Rose Murray, a lecturer in history at the University of Suffolk who created the new entries, said: “If the walls of churches, chapels or town halls could talk, they would tell powerful, emotional and hard-hitting stories of Black life, liberty and love that have been deliberately erased from our landscape.”

Historic England said: “From rural communities to large industrial cities, the map tracks a historical trail of Black activists who had been championing the anti-racist and anti-slavery cause for decades. Between 1833 and 1899, more than 50 African American activists spoke at venues across England, from churches and theatres to schools and factories (and) spoke to millions of people.

“They published bestselling autobiographies; composed and recited poetry; exhibited panoramas and paintings; encouraged boycotts of goods produced by enslaved labour (cotton, rice, sugar) that also shaped Britain’s wealth and challenged the racism they experienced in England.

“These activists spoke at landmarks across the country and worked with all sections of society, influencing every aspect of Victorian Britain.”

Historic England, which is inviting the public to contribute stories to the Missing Pieces Project, said: “Sharing your story will add a unique piece to the picture and help people understand what makes these places so significant.”