First experimental results about bot identification

We just got our first results from an experiment (n=65) dealing with the question whether accounts that are labeled as bots are also identified as bots by human users and what effects this has on how human users perceive these accounts. In this experiment, we were concerned with the perception of bots in general and not with social bots (social bots as they are typically defined in the literature hide their true identity and would not identify themselves as bots).

In addition to the control group (no declared bot account) we used two different labels for accounts to enable users to identify them as bots (“RainerBot” in one condition and “(!)Bot-Account Rainer Schmitt-Sasse” in the other). We showed participants a section of an online political discussion the account contributed to (the discussion had a total of 9 participants) . Participants of the experiment were then asked a set of questions, including whether they thought that any of the accounts contributing to the discussion was a bot.
T-tests were calculated to examine if participants were able to identify the account as a bot or not. Interestingly we found that “RainerBot” and “(!)Bot-Account Rainer Schmitt-Sasse” did not get significantly identified as bot or non-bot, p > .208, while participants in the control group mostly identified the account as a non-bot-account, p < .002.
These results show that a name such as “RainerBot” is an indication to some users that they are dealing with a bot but no clear indication to the majority of users that they are dealing with a bot. This suggests that users by default identify accounts as human and that additional means are necessary for users to be able to identify bots in online discussions.

Leave a comment

Your email address will not be published. Required fields are marked *