Enlarge /. The Facebook app is displayed on the screen of an iPhone.
Most spammers and scammers create accounts in bulk and can easily search for differences between the blocked and under-the-radar. These differences can allow them to evade automated algorithms by structuring new accounts to avoid the functions that trigger bans. The end result is an arms race between algorithms and spammers and scammers trying to guess their rules.
Facebook believes it has found a way to avoid being involved in this arms race while still using automated tools to monitor its users, and has decided this week to tell the press about it. The result was an interesting window on how to keep AI-based moderation useful in the face of controversial behavior, an approach that could be applied well beyond Facebook.
Facebook sees billions of active users in a month, and only a small portion of it falls into what the company calls abusive: fake and compromised accounts, spammers, and those who use the social network to do fraud. While the company can (and does) use human moderators, the problem is just big enough that they can't be expected to intercept everything. This means that an automated system is required if the service does not want to be flooded with content that it does not want to see.
Facebook (or another social network operator) obviously has access to a lot of data that can be used by an automated system: an account's booking history, login details, friend networks, etc., and an algorithm could easily use this data to identify problematic accounts identify, including neural networks that are trained using the data and a human-curated list of problematic and acceptable behaviors.
As mentioned above, the problem is that the people who keep abusive accounts also have access to all of this data and may be able to find out what features are causing accounts to be locked out. Alternatively, they can change their behavior to such an extent that no suspicion is raised. This increases the risk of an arms race because the scammers are always one step ahead of the algorithms they are supposed to catch.
To avoid this, the Facebook researchers switched from using account data to so-called account metadata. Instead of using the number of posts that a particular account may create, it shows the number of posts from a typical friend account. Similar values can be generated for the average number of friends the friends of the account are connected to, the number of times friend requests are sent, and so on. A number of such values are summarized in a profile that the company's researchers call a "deep unit".
This assumes that the typical account creates relationships with accounts that are also closer to the typical account. In the meantime, a spammer is likely to have fewer connections to real accounts and more things like bot accounts that also have unusual behaviors and connections. The deep entity profile captures these differences altogether and offers two main advantages: abusive account holders are much more difficult to understand which aspects of a deep entity are used by an algorithm, and account holders are even more difficult to do so change if you could understand.
Deep entity classification
Deep entity classification is fairly straightforward, albeit somewhat computationally. It's just about crawling a particular user's network diagram and aggregating data from all of their connections. Things are entering the realm of computer science in how these classifications are used to actually identify problematic accounts.
Facebook engineers chose a neural network to do the classification. To do this, the network must have training data: deep entity profiles that contain information as to whether the account is problematic or not. The engineers had two options here. Working with other classification algorithms had produced a large amount of relatively uncertain data that identified different accounts as problematic or not problematic. In the meantime, human moderators had gone through a much smaller collection of accounts but made much higher quality calls to see if the account was abusive.
Of course, the people on Facebook decided to do both. They produced a two-stage system. In the outer layer, a multi-layer neural network used the low quality training data to identify accounts with deep entity profiles that were typically associated with unusual behavior. While this neural network would of course process the data until a binary decision was made – abusive or not – the researchers actually stopped the analysis at the level just before the binary decisions.
At that point, the network had processed the original deep entity information into a limited number of values that could be used to determine whether an account's connections were unusual or not. These values could be extracted as a 32-number vector that captures the characteristics that are normally associated with unusual accounts.
These values were then passed to a second form of processing using a machine learning approach called the decision tree. This decision tree was trained using human-identified account data. What mattered was that the Facebook engineers trained several decision trees: one for spammers, one for kidnapped accounts and so on. These decision trees make the last call on whether an account is a problem and needs to be deactivated.
Computer science meets politics
The system has been in production for some time and has proven to be quite successful. It blocks at least half a billion accounts per quarter, peaking at over 2 billion blocks in the first quarter of last year. Locked accounts can also be used to continuously retrain the system in the background, and it can evaluate its own metrics to determine when retraining has progressed enough to replace the production system productively.
While the system may be effective, deciding how to deploy the system (and how to incorporate it into a broader strategy for acceptable content) is more a matter of policy than IT. Human moderators offer a higher level of accuracy in their calls about whether content is abusive, and a Facebook communications manager told Ars that the company is expanding the use of human moderators. However, people can only respond to content that has been reported, while the algorithms can work preventively. Finding the right balance between investing in both aspects of moderation will ultimately be a decision.
The other problem suggested by this technology is whether it can be used against accounts that spread misinformation on issues such as climate change and health information. The latter problem arises more and more with the spread of corona viruses. Here, the company has taken an uncomfortable line and is trying to avoid becoming, according to its communications manager, "the arbiter of truth" – particularly including the refusal to monitor the factual content of political notices. His approach to outsourcing fact checking has caught fire as websites with a questionable history of facts can act as fact verifiers.
Facebook's communications manager informed Ars that certain health claims exposed by the WHO or CDC could be removed. However, there is no indication that groups who make repeated claims will ever block their accounts – even if tools like the one described here should make identification much easier. In other words, while Facebook engineers have done a masterly job of developing a system that can identify problematic accounts, deciding how to use this technology remains a political decision.