• Skip to primary navigation
  • Skip to footer navigation
  • Skip to main content
  • Skip to primary sidebar
  • Skip to footer

WP Fangirl

WordPress Consultant Sallie Goetsch

  • speakerdeck icon
  • Home
  • About
  • Why WordPress?
  • How I Work
  • Portfolio
  • Services
  • Blog
  • Contact

How to Tell Bots to Go Crawl Themselves

April 13, 2016 by Sallie Goetsch 1 Comment

padlock and circuitry

A couple of weeks ago SiteGround notified me that I had reached my allowed daily executions. Since I have almost two dozen sites on my account, half of which are dev sites locked behind coming soon pages, I asked for some help in pinpointing the problem.

George M. was extremely helpful in showing me what was going on and suggesting what to do about it. Overall, I was suffering a bot infestation (more than six million visits from unknown bots), and my login page for the Author-izer site was particularly hard hit.

He also provided me with a handy trick for deterring bots. At first, when he said the code would “make bots crawl themselves,” I thought he was joking, but in fact that’s exactly what the provided code does. It directs all bots except those specifically identified to 127.0.0.1, which is localhost. This is wonderfully sneaky and elegant, and I have added it to the .htaccess files for all of my sites.

# BEGIN Tell bots to go crawl themselves
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^$ [OR]
RewriteCond %{HTTP_USER_AGENT} (bot|crawl|robot)
RewriteCond %{HTTP_USER_AGENT} !(bing|Google|msn|MSR|Twitter) [NC]
RewriteRule ^/?.*$ "http\:\/\/127\.0\.0\.1" [R,L]
</IfModule>
# END Tell bots to go crawl themselves

I also took a closer look at all of my sites and discovered that I had deactivated iThemes Security Pro on the Author-izer site and not actually installed it on some of the other sites, so I spent some time fixing that problem and then added Let’s Encrypt SSL certificates to all the sites for good measure.

I’m looking forward to inspecting my statistics again in a few weeks to see how it’s working.

Related Items

  • screenshot of CloudFlare's Crypto page with SSL set to strict
    Adventures in SSL, CloudFlare, and WooCommerce
  • find replace SQL query in phpMyAdmin
    PreviewDNS Can Bork Your WordPress Install—But There’s a Fix
  • Manage domain aliases at Pagely (2012)
    Setting Up DNS and Email on Page.ly

Share this post:

Share on Twitter Share on Facebook Share on Pinterest Share on LinkedIn Share on Email

Filed Under: Hosting and Servers Tagged With: Tutorial, Security

Reader Interactions

Comments

  1. maxbet says

    June 1, 2016 at 2:04 am

    And political spenders, who often outbid brands on targeted inventory, are uniquely vulnerable to digital fraud and bots.

    Reply

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Primary Sidebar

What I Write About

  • Book Reviews
  • Content Strategy
  • Design
  • Hosting and Servers
  • Most Valuable Plugins
  • There's a Plugin for That
  • Using WordPress
  • Widgets
  • WordPress Consulting
  • WordPress Events

Series

  • Interviews (5)
  • Checking Up on Your Website (4)
  • Client from Hell (5)
  • WordCamps (17)
  • WP-Tonic Roundtable (30)
  • Modern Tribe Tutorials (13)

Follow Sallie on Twitter

    Sorry, no Tweets were found.

RSS Latest News from the East Bay WordPress Meetup

  • Does It Work? Using The New CSS Layout with Rachel Andrew
    Things change rapidly in the WordPress world. The content in this post is more than a year old and may no longer represent best practices.Description Over the past two years, […] The post Does It Work? Using The New CSS Layout with Rachel Andrew appeared first on East Bay WordPress Meetup.
  • Speaker Training
    Get the workbook and slides for the October 2019 speaker training, plus background and pro tips. The post Speaker Training appeared first on East Bay WordPress Meetup.
  • SEO Audit Template & Resources
    Our November speaker, John Locke, graciously provided a template for an SEO audit report. You can download it as a Microsoft Word or PDF document. The post SEO Audit Template & Resources appeared first on East Bay WordPress Meetup.

Footer

Contact Info

2063 Main St #133 · Oakley, CA 94561

+1 (510) 969-9947

author-izer

sallie [at] wpfangirl [dot] com

Location

Map of East Contra Costa County

I live in Oakley, CA and run a WordPress Meetup in Oakland, CA. Don't confuse them!

Subscribe for New Posts

  • Since I blog on an unpredictable schedule, you might want to subscribe by email. I'll also send out occasional announcements about events.

  • Privacy Policy: I will never sell or rent your contact information.

  • This field is for validation purposes and should be left unchanged.
  • Contact
  • Colophon
  • Comment Policy
  • Privacy Policy
  • Five for the Future

Copyright © 2023 · Utility Pro on Genesis Framework · WordPress · Log in

MENU
  • Home
  • About
  • Why WordPress?
  • How I Work
  • Portfolio
  • Services
  • Blog
  • Contact