The search operation was launched when a life jacket was spotted by a member of the public out at sea, authorities said.
Some 48 people were recovered in the search and first aid was given to two unconscious people brought on board a rescue ship.
They were later confirmed dead in Calais.
A ferry recovered a third body and a fourth person had been injured, a later update from the French coastguard said.
Two boats and a helicopter were used in the rescue mission, alongside a medical team deployed to Calais.
Those rescued were being taken care of by land rescue services and border police while the search for others who may remain lost at sea continued, authorities added.
The year 2024 has already become the deadliest for migrant crossings in the English Channel, according to a United Nations agency.
On 3 September, six children and a pregnant woman were among 12 people who died after a boat carrying dozens of people sank off the French coast.
A month later, four people, including a two-year-old boy, died after seemingly being “trampled to death” on two separate boats.
The latest sinking brings the total number of deaths from migrants attempting to cross the Channel this year to at least 56.
On 18 October, 647 migrants crossed the English Channel in 10 small boats, bringing the total number of people arriving in 2024 to 28,204.
The figure is almost 8% higher than at the same point last year, but 25% down on the peak year of 2022.
Prime Minister Sir Keir Starmer has said he will take tougher measures to “smash” the gangs smuggling people to the UK.
The top stories and transfer rumours from Friday's newspapers...DAILY MIRROR Man Utd have reservations over triggering Alvaro Carr
Manchester United captain Bruno Fernandes was shown a red card during their defeat to Wolves19:30, 26 Dec 2024Updated 20:32, 26 Dec 2024Bruno Fernandes of Manch
Liverpool will look to maintain their excellent form in the Premier League when they welcome Leicester City on Thursday.Arne Slot's
Prince William invited son George to play a festive football game at Sandringham (Picture: AFP) Prince George lined up alongside his father Will