Bad Bots: What Do They Actually Do Online?

Since the 2016 presidential elections in America, most people are aware of the existence of malicious bots on the internet. Hackers, shoppers, corporations and even governments use bots every day. Many bots have perfectly legitimate uses, but the prevalence of malicious bots has increased noticeably in recent years.

For example, the limited-edition sneaker market is massively distorted by the presence of bots, as is the live music ticket industry. Usually, proxy networks are a huge part of this process, so there are geeks out there who test, review and write about proxy providers (e.g. Proxyway) However, these automatic purchasing bots are nothing compared to some of the bad bots that are out there.

Akamai and Distil networks issue a Bad Bots Report each at least annually, which reveal a great deal about the current state of the bot arms race.

The Scale of Bad Bots

Since Akamai and Distil Networks are some of the most respected names in the field of bot mitigation, there is no one better to really tell us about how ‘the bot arms race continues’. In reality, hundreds of billions of bot requests get recorded throughout the year, targeting tens of thousands of separate domains. By analysing these requests, we can learn a great deal about the types of attacks that bots launch every day and what can be done to stop them.

Who Uses Bad Bots?

Bots are used by a variety of different people and for a variety of different purposes (scraping, social media automation, research purposes, etc.). In some industries, they have become an unmitigated nuisance. However, other industries have accepted bots as an inevitable evil and have grown to accept them to varying degrees. For example, some commercial airlines now allow bots to scrape their websites for data about flights for customers who are using price comparison websites.

But whether an industry wants them there or not, bots are spreading their influence far and wide. There is now no industry or type of business that is unaffected by the actions of bots. Because bots are used by both legitimate users and criminals, many businesses have been hesitant to take heavy action against them, such as banning them entirely from their services.

Key Findings

Distil’s report made a number of significant findings that are of interest to both industries analysts and observers. In 2018, bad bots were responsible for around 1 in 5 of every 360 websites requests made. All told, this accounts for 20.4% of all the internet traffic on the web. On the other hand, good bots decreased their share of overall internet traffic to just 17.5%.

A vast majority of 73.6% of all bad bots were classified as advanced persistent bots. This type of bot is characterized by its ability to cycle through different IP addresses, or in some cases residential proxies, making them very difficult to detect. If they are also coupled with human-like behaviour, it can be all but impossible to tell them apart from a human user.

Almost 50% of all the bad bots studied were reporting their user agent as Chrome. Wild mobile browsers accounted for a higher proportion of user agents than in previous years, but the number is still far below the figure of 50% for Chrome.

One of the most surprising findings in the report was that Amazon is the ISP responsible for the highest level of bad bot traffic. 18% of all bad bot traffic originated from Amazon, which is an 8% increase on last year.

The report also noted that while 53.4% of all bot traffic originates from the US, Ukraine and Russia and account for 48.2% of the country-specific IP block requests that are made against bots.

What Are Bad Bots Doing?

Bad bots are now active in most types of industry and are causing some level of disruption. For example, the financial services industry accounted for 42.2% of all bad traffic. Finance industry businesses were subject to credential stuffing, which enabled attackers to take over user accounts and manipulate services.

The ticketing industry is another prominent casualty of the rise of bad bots. Scalping bots are a widespread issue, one that we still do not have any good solutions for. However, these ticketing bots are doing much more than just buying tickets. They will also check seat inventory and can also use credit stuffing to gain unauthorized access to accounts.

Education services have been targeted by bots, which are used to scrape their service for research papers, class inventories, or probe for weaknesses that might enable unauthorised account access.

We are all familiar with how bots can be used to interfere with political campaigns. If this is an issue that is not addressed soon, bots will become a staple of elections like they have become a staple of buying tickets.

Bad bots have become just another part of the online experience, unfortunately. Despite the valiant actions that have been taken by some organisations and businesses, malicious bots are a problem that is going to take a long time to solve.

Whereas these bots used to be restricted to just a few platforms, they are now prevalent across a wide range of industries. In fact, anyone can buy a bot pre-programmed and ready to go if they so choose. For industries like the ticketing industry, it may be too late to remove bots from the equation. However, for other sectors such as financial services and government, the stakes are far too high to let bots run rampant.

If the problem of bad bots is ever going to be adequately addressed, we are going to need to take a multi-pronged approach and encourage all affected industries to get involved. Without the backing of national governments, there is only so much that we will ever be able to do to address this problem.