Data experts raise fresh questions on BBC’s probe into Twitter abuse

After glaring errors were pointed out by the Sunday National, the BBC made multiple edits to documents which it published alongside a report claiming to reveal the “scale of abuse of politicians on Twitter”.

The BBC also deleted six full pages from a document detailing key information about its headline findings, which had come from an AI programme scanning “all tweets mentioning MPs from March to mid-April” and categorising them as “toxic” or not.

READ MORE: Drag queen kids show at Dundee's DCA cancelled after 'abusive threats'

Dr Liam McLoughlin, an academic from the University of Liverpool who has conducted research into abuse aimed at MPs online, told the Sunday National that the BBC’s data was “fundamentally methodologically wrong”.

He said the AI software the BBC team had used – called Perspective API – could not truly detect toxicity as it cannot read context or understand nuance.

“What these journalists have done is taken a system that’s not fit for purpose and decided it fits their needs perfectly when it clearly doesn’t,” McLoughlin said.

“Actual racism and Nazi dog whistles and things like that weren’t being detected as toxic. I don’t think they’ve rectified that whatsoever.”

The academic added that the BBC’s article – which one of the three journalists who worked on it described as an “eight-month labour of love” – had been “effectively trashed an hour after it got published”. He suggested that the post-publication deletions and edits from the BBC were because they wanted “to put that under the rug”.

“I’m not saying that abuse doesn’t happen to MPs,” McLoughlin went on. “We know it does. My own research has highlighted that it does happen.

“It is a key democratic concern if you’re scaring away people from standing in politics, but you’re not going to help with dodgy data, which is my key gripe with this.”

Published on November 9, the BBC investigation led to headlines around the UK about which MPs received the most abuse on Twitter. Tory MP Ben Bradley was named the most targeted, and a BBC interview with him was published that same day.

However, background information published by the corporation – which was deleted hours later – made clear that “almost all” of the flagged tweets aimed at Bradley had come off the back of a single post in which he falsely claimed that Channel 4 was in receipt of public money in an attempt to argue for its privatisation.

The BBC data further claimed that the MP who saw the second-highest rate of toxic tweets was Labour MP for Ealing North James Murray.

However, the corporation deleted background information that revealed that “three-quarters”

of all the tweets marked as toxic aimed at Murray had occurred on a single day, the same day he told Times Radio: “I believe trans women are women.”

Further information deleted by the BBC revealed that Labour’s deputy leader Angela Rayner and Tory MP Jamie Wallis both saw massive spikes in the number of toxic tweets aimed at them after they spoke about trans rights or came out as trans respectively.

Also removed from the BBC document was a table that it claimed showed “examples of ‘you are a’ preceding extreme profanities” in toxic tweets. The now-deleted table listed such “extreme profanities” as “you are a Tory” and “you are a fool”.

Elsewhere, the BBC deleted analysis which claimed to show that four key topics for toxic tweets were trans rights, the Partygate scandal, the Rwanda deportation plan, and the war in Ukraine. Words which it had flagged as likely to appear in “toxic” tweets on these topics included “woman”, “women”, “man”, “like”, “government”, “corrupt”, “children”, “silly”, “war”, and “shame”.

Dr Katie Spalding, a journalist with IFLScience, noticed the BBC had deleted the pages of key information and asked the corporation to provide its raw data for a report she was looking into. They refused.

“I cover science every day and I have never seen a study where, if I have asked for the data after a press release, I’ve been told no,” Spalding told the Sunday National. “It’s never happened to me.”

“I finally got a response from the BBC [two weeks after first trying] and it is also the first time anyone has ever said this to me: ‘I wanted to ask if you could provide details around the angle of your story’.

“Again, I’m a science journalist, I’m used to being told, ‘here are our results’.

“With science, it’s generally the case that if someone can poke holes in your working you kind of want them to because if you’ve done something wrong, you need to know about it.”

The BBC also declined to comment when approached by the Sunday National but has since had to make multiple changes to its published data after we highlighted errors in the data.

These came in a spreadsheet which purported to show which MPs in each “region” of the UK were in the top ten in terms of toxic tweets received.

The Scotland list had nine SNP MPs and one Tory MP. However, the ranking did not even closely agree with another under a different tab on the same spreadsheet. The BBC quietly edited it, but the spreadsheet’s data was still not consistent.

After the Sunday National highlighted that the BBC’s corrections still contained inconsistencies and errors, the corporation again went and quietly changed the document.

SNP president Michael Russell

SNP president Michael Russell said that the BBC’s “slipshod” analysis and presentation of data was a “worrying indication of how casually it regards facts”.

McLoughlin queried why external experts had not been consulted.

“There [are] so many experts out there that have published on this, researched on this, that know the data and how the data works, and they haven’t been asked at any point in a fact-checking process,” he said. “Is there a certain level of arrogance there?”

Now apparently correct, the BBC’s spreadsheet states that the Scottish MPs who received the most toxic tweets were, from first to tenth: SNP MP Steven Bonnar, Tory MP Andrew Bowie, Tory MP David Mundell, SNP MP Deidre Brock, SNP MP Stephen Flynn, LibDem MP Alistair Carmichael, SNP MP Carol Monaghan, SNP MP Alison Thewliss, now Independent MP Patrick Grady, and SNP MP Kirsty Blackman.

However, both Spalding and McLoughlin told the Sunday National that the BBC’s report “doesn’t show what we’re being told it shows”.

“[The AI] was very good at picking out swear words and saying it was toxic,” McLoughlin explained. “But what do we know about people in the United Kingdom when they talk about politics? They use colourful language. This is normal.

“It’s not necessarily highlighting that MPs are being abused because of the way the system’s designed, it’s highlighting that people swear when they talk about politics. Well, no shit.”

Spalding added: “I’ve seen a lot of people say ‘this wouldn’t get you past sixth form sociology’, and it’s very difficult to say that those comments are wrong.

“I think the nicest thing I can say about this report is it shows something, but it doesn’t show what we’re being told it shows. It’s basically a report into the things that may or may not make MPs feel bad when they read it on Twitter.”

The BBC declined to comment.