Support: 1-800-961-4454
Sales Chat

Web Bots: What Every Website Owner Needs To Know

By Charlie Minesinger, Director of Sales, Distil Networks

If you or your company operates a website or utilizes online advertising, then web bots and spiders are a cause of serious concern. No one is immune from bot traffic, so the question is: how much bad bot traffic is acceptable? Can you measure your bad bot traffic? Malicious bots can threaten the viability of your ad campaigns, affect your KPI tracking and analytics, and negatively impact your sales, conversions and overall customer experience. To get an idea of their true impact, let’s first discuss the kinds of web bots to which your site could fall victim.

Theft Bots

Theft bots aim to steal content, pricing, images and other information from sites on the web. Sometimes, they do this to sell the content to rival websites or content farms; at other times, they simply use it to post directly to their own sites. In some cases, a competitor may even use your data to get an inside look at your content and pricing, so they can get a leg up in the marketplace. In addition to scraping your content and data, theft bots can also harvest lists of email addresses for spam purposes.

In addition to these negative effects, theft bots can also cause SEO problems for site owners. If content is stolen and duplicated elsewhere on the web, it severely downgrades the original site’s search engine rankings and could even result in getting a site blacklisted.

A great example of a theft bot can be seen in a recent court case. Last year, the Metropolitan Regional Information Systems Inc., or MRIS, filed suit against the American Home Realty Network for copyright infringement. MRIS alleged that AHRN had used web bots to effectively steal content from various real estate MLS listing sites and then pass the information off as its own. While the case is still pending, MRIS was granted an injunction, which likely means that AHRN will be found guilty.

Fraud Bots

Fraud bots create fraudulent traffic by clicking on paid advertisements and display ads throughout the web, and sometimes they even post spam to websites, blogs and various web forms. These bots can create fraudulent accounts on databases or networks to gain access to directories of customer information. Fraud bots mean bad news all around for your site, your customers and your sales.

The biggest issue with fraud bots is the problem of click fraud. In fact, according to AdWeek, web bots steal an average of $6 billion from advertisers every year through fraudulent traffic and clicks. Furthermore, according to Medialink president Wenda Millard, about 25 percent of all traffic attracted by online ads is likely fraudulent. That means one out of every four clicks your site gets on its PPC or display ad campaigns could be a fake. When you consider the costs of those fraudulent clicks over a week, a month or even a year, it can result in substantial financial losses.

More on Bad Bots

In addition to the above repercussions, web bots also lower the quality of the user’s experience as a whole. Bots can cause slow moving, poorly performing websites; infect users’ devices, computers and mobile phones; and diminish the overall web-surfing experience. Businesses and website owners may see their bandwidth decreased, their KPIs and analytics become hard to decipher, and their intellectual property, content and data stolen.

The number of both fraud and theft bots has grown exponentially over the last few years. According to Distil Network’s most recent Bad Bot Landscape Report, the number of damaging web bots doubled in 2013. Bad bots accounted for just 12.25 percent of all web traffic in the first quarter, but climbed to a whopping 23.6 percent by the end of the fourth quarter. Furthermore, though many bad bots come from overseas, the Landscape Report reveals that more bad bots originate in the United States than any other country in the world.

On May 15, 2014, the Executive Director and President of the Online Trust Alliance (OTA), Craig Spiezle, testified to the U.S. Senate about the risks of malicious ads and web bots. OTA research reports that malicious advertising, or “malvertising,” rose by more than 200 percent in 2013. In addition to threatening the effectiveness of your website, malicious ads can be used to infect personal computers, steal personal information, or manipulate devices into becoming part of a DDoS attack. In addition, one botnet called “Pushdo” was detected on more than 4 million IP addresses, including IP addresses of most every major Internet Provider in North America and Europe.

Fortunately, there are technologies and strategies for blocking bots and protecting your website and your ads from fraud and theft without negatively affecting SEO and your (human) customer experience. We’ll be discussing these strategies, as well as more on web bots, in an upcoming post, so stay tuned! In the meantime, take your traffic readings and PPC results with a grain of salt—you never know when web bots may strike.

About the Author

This is a post written and contributed by Charlie Minesinger.

Charlie Minesinger is Director of Sales at Distil Networks and headquartered in Virgnia. Charlie works with Rackers who need to protect their clients’ websites from the various challenges of malicious bot traffic – price scraping, content theft, fraudulent registrations, form spam, click-fraud and server slowdowns. Charlie joined Distil as the first sales person early in 2013 and then signed the first Fortune 500 accounts, the first private cloud clients and the first reseller deals.... And Charlie likes challenges - he served in the Peace Corps, speaks two African languages, worked in a refugee camp and completed the Comrades Marathon and Coeur d'Alene Ironman Triathlon.

Racker Powered
©2015 Rackspace, US Inc.