how Facebook AI works
In a paper highlighted today in a Facebook blog post, engineers describe an algorithm — SybilEdge — to detect fake accounts that evade Facebook’s anti-abuse filters at registration time but that haven’t friended enough people to perpetuate abuse. The goal is to mitigate the accounts’ ability to launch attacks against other users, in part by comparing the way users add friends to their extended social networks.
SybilEdge — which can detect fake Facebook accounts less than a week old with fewer than 20 friend requests — has immediate application for platforms dealing with a wave of misleading information about the coronavirus pandemic. An analysis published by the Reuters Institute for the Study of Journalism at the University of Oxford found that 33% of people have seen some form of misinformation about COVID-19 on social networks like Twitter, Facebook, and YouTube.
In architecting SybilEdge, the development team noted that abusers need to connect to targets in order to launch abuse — that is, they need to find targets, send them a friend request, and have the request accepted. Perhaps unsurprisingly, internal Facebook studies revealed that non-abusers differ in both their selection of friends and those friends’ responses to their friend requests: Fake accounts’ requests were rejected more often than real users’ requests. In addition, fake accounts were often careful when picking their friend request targets, likely to maximize the probability of their requests being accepted.
Facebook created a corpus with which to train SybilEdge by segmenting users into two groups: those more likely to accept friend requests from real accounts and those likely to accept fake account requests. If the former rejects an incoming request, it serves to signal that the requester is a legitimate user. On the other hand, if the users who accept more fake requests accept a request, it indicates that the requester was likely fake.
SybilEdge works in two stages. First, it’s trained by observing the aforementioned samples over time, after which it leverages outputs from Facebook’s behavioral and content classifiers that flag accounts based on actual abuse. This training phase provides the model with all the necessary parameters (i.e., configuration variables estimated from data and required by the model when making predictions) to run in real-time for each friend request and response and update the probability of the requester being fake.