The online-only retailer’s system engineering team used to spend 20% of its time trying to stop malicious bots. A new system from e-commerce vendor Distil Networks now automatically stops them.

Battling bad bots is an issue that online retailer Hayneedle.com has been dealing with for years, says Brian Gress, director of information technology systems and governance for the merchant of home decor and furniture.

Some of those malicious bots attempt to flood Hayneedle.com with traffic so that legitimate shoppers encounter a sluggish site, other bots attempt to make fraudulent transactions. Bots from competitors also attempt to scrape the Hayneedle site for prices or product data to use for their own competitive advantage, Gress says.

“We would see a really heavy amount of traffic from one IP address and block it,” Gress says. Or Hayneedle would encounter bots with stolen credit card numbers trying to guess the CVV number many times over. “We were in a cycle of back and forth, trying to stay on top of them,” Gress says. CVV numbers refer to card verification values, typically a three-digit code on the back of a payment card. The number helps e-retailers verify that a payment card is in a consumer’s possession during a card-not-present transaction.

What’s more, Gress and his team had to be cautious not to block good bots, such as those from Pinterest or Google Inc.’s search bots that crawl the site for content to use in search results, says Hayneedle, No. 91 in the Internet Retailer 2016 Top 500 Guide. Hayneedle in February was purchased by online marketplace Jet.com Inc., which is now owned by Wal-Mart Stores Inc. (No. 4).

Gress and his team had long been manually trying to curb bad bots, but the process was time consuming, cumbersome and, in short, not working. By spring, the bots were clearly getting the better of Hayneedle. Traffic surges from bots were taking a toll on the retailer’s servers and hardware, which were aging and needed to be replaced.

advertisement

“Things were coming to a head,” he says. “Our hardware had reached the end of its life cycle. It was highly utilized and pressed to its limits.” What’s more, stopping bots was eating up about 20% of Hayneedle’s system engineering team’s time, Gress says. And even with all that time spent managing bots, Hayneedle was stopping only about 30% of the bad ones.

To fight the bots, Hayneedle earlier this year began using a cloud-based service from Distil Networks. Distil sits in front of Hayneedle’s content delivery network (CDN) so that all traffic goes through Distil before reaching Hayneeedle’s CDN, which serves traffic to Hayneedle.com. A CDN is a network of servers that delivers web content to consumers based on their location, the origin of the webpage and the location of the server.

Distil can check to see if a bot is legitimate or flag it if it looks suspicious. Hayneedle has set parameters in Distil. For example, it may require a visitor enter a captcha, which are fuzzy letters or numbers not distinguishable by a bot, or Distil can block the IP address altogether, Gress says. “We still make manual decisions, but Distil makes a lot of automated decisions for us,” he says.

Gress says he chose Distil because it allowed him to test the service before committing and because it was web-based, meaning Hayneedle staff didn’t have to implement and manage software or run it in-house.

advertisement

The system is also easy to control, Gress says. For example, earlier this month, Gress was examining traffic records from Distil and noticed a large amount of traffic from a single IP address. He decided to block it and later learned the traffic was legitimate, so he immediately unblocked the IP address. Additionally, Distil’s false bad bot positive rate for Hayneedle is a low 0.00046%. Distil charges Hayneedle a fee based on the amount of traffic it intercepts, Gress says.

“Some bot operators interject scripts into their code to mimic human behavior,” Gress says. “Others will circumvent the web product page and go directly to the API behind it. Distil lets us granularly control questionable traffic or simply block it all.”

A telltale sign that Distil has saved Hayneedle ample time occurred last month, when Gress looked at traffic logs. “From Sept. 26 to Oct. 2 we had a massive amount of bots hitting our site—about 30 million a day when it’s usually between 3 and 4 million,” Gress says. “Before, I was on the phone on weekends during my kid’s baseball games trying to manage bots like these and determine the steps we needed to take,” Gress says. “But this time I didn’t even notice.”

 

advertisement

 

 

 

 

advertisement
Favorite