Tiffany Olson Kleemann, vice president of bot mitigation, Imperva
Alan Turing, one of the founding fathers of modern computing, famously devised his eponymous Turing Test as a way to determine whether a machine was capable of thinking like a human. CAPTCHAs are one example of how this thought experiment has been applied in the real world to distinguish real people from hackers and the automated programs they use to scrape content and insert malicious code.
As bots have grown in scope and sophistication, businesses across all industries are finding it increasingly challenging to contend with them and perhaps no industry has more at stake than the burgeoning ecommerce market. With Black Friday and Cyber Monday just around the corner, ecommerce providers are once again preparing themselves for battle.
According to Deloitte’s annual holiday forecast report, online sales for the 2019 holiday season are “poised to grow in the range of 14% to 18% compared with 2018… Deloitte expects ecommerce sales to hit $144 billion to $149 billion during the holiday period, which it defines as November through January, up from $126.4 billion spent online in 2018.”
Recently, Imperva conducted the first research report dedicated exclusively to the ecommerce sector to learn more about the impact that bad bots are having on businesses and their customers. Over the course of six weeks, our research team analyzed more 16.4 billion requests across 231 ecommerce websites, APIs and mobile apps. Here’s a broad overview of what we learned.
Who’s Behind Bad Bots?
The bot operator ecosystem is more complex and diverse than one might think. The following four constituents account for the bulk of bad bots in the ecommerce market:
- Fraudsters: Criminal hacking syndicates employ bad bots to perpetrate a variety of fraudulent schemes, including account takeover attacks, gift card fraud, brute force credential stuffing to name but a few.
- Competitors: Web scraping of content by competitors has become a common practice as a way to ensure prices are in line with the market and determine what products are most in-demand. Technologically savvy competitors also leverage scraped content to boost SEO page results and expand into other markets and geographies.
- Unauthorized Resellers: The secondary reseller market has quickly grown to become a multi-billion-dollar cottage industry due in large part to ‘sneaker bots,’ ‘Grinch bots’ and other specialized bots that focus on products with limited inventory such as collector items.
- Investment Companies: Hedge funds and other investment firms are increasingly using bots to track market indicator data, such as inventory and pricing data to build more accurate market forecasts to guide their investment decisions. A 2019 report from Opimas estimates that 5% of all web traffic is attributable to investment-scraping bots.
The Scourge of Bad Bots in Ecommerce
Of course, not every bot is bad. Ecommerce sites rely on good bots such as web search crawlers to help customers find their products and scrape their content to enhance their SEO efforts. Our research estimates that within the domain of ecommerce sites, good bots account for 13% while bad bots represent almost 18% of traffic.
Source: Imperva
While the volume of bad bots in ecommerce is lower than industries such as airlines and ticketing, the functionality of bots that abuse ecommerce systems is more diverse due to the fact that there is a broader swath of potential targets, including loyalty reward programs, gift cards, and sensitive customer account information.
Our research also shows that bots are growing in sophistication, with nearly a quarter (23.5%) of the bots analyzed showing advanced levels of complexity, such as learning to better mimic human-like behavior to avoid detection.
Of course, the impact of bad bots in the ecommerce sector hit more than just the bottom line. At a broad level, these risks include:
- Poor Customer Experience: Bots aren’t just a business problem. They can fundamentally degrade the customer experience. Sneaker bots and Grinch bots are perhaps the two most notable examples of shopping bots causing frustration and friction to the customer experience by denying access to limited inventory which in turn fuel the creation of artificial secondary markets.
- Targeted Fraud Attacks: In addition to the threats of brute force credential stuffing and account takeover attacks common in the ecommerce industry, bot operators have built and deployed specialized bots that exploit gift cards and customer loyalty reward programs, driving down net margins and further diminishing the lifetime value of customers.
- Operational Disruptions: Our research found that for a third of the ecommerce sites we analyzed, 30% of their traffic originated from bad bots (compared to an industry average of 17.7%), slowing site performance and forcing IT staff to overprovision capacity to ensure site availability. Bad bots also distort the key traffic metrics operations teams rely on to make informed business decisions.
- Brand Reputational Risk: When coveted items become scarce due to inventory denial bots, customer frustration can cause irreparable harm to a brand’s reputation, both for the product brands themselves as well as their authorized reseller channels.
Keeping Bad Bots at Bay
Like other aspects of cybersecurity, bots represent an escalating arms race with no end in sight. Not only are we seeing bots evolve in sophistication, but the authors of these bots are also increasingly selling their wares on Dark Web forums, democratizing their use to a broader audience of less sophisticated users.
While there is no one-size-fits-all solution, there are steps you can take to proactively identify the presence of bots. These include blocking known outdated user agents/browsers, blocking known hosting providers and proxy services, protecting exposed APIs and mobile apps, evaluating and analyzing traffic sources and spikes, and monitoring failed login attempts.
A comprehensive bot mitigation strategy will help protect your customers, your brand reputation, and ensure your site can weather the busy holiday season.
Imperva is a provider of cyber security software and services.
Favorite