SEO Tips

Sally’s SEO Tip: Switch User-Agent When Crawling A Site

SEO Tip: Switch User-Agent When Crawling A Website

Hello friends!

Happy Friday to you all 🙂 In this blog post today I just wanted to share a quick tip with you.

 

SEO tip:

When running a crawl of a site, switch your user-agent from the default (crawler) to a browser (such as Chrome).

For example, if you’re crawling a site with Screaming Frog, the default user-agent is Screaming-Frog. I recommend you change this!

 

Why?

Websites can treat crawler user-agents differently to users (usually less sympathetically!). By switching your user-agent to a browser, you look like a real user to the website you are crawling.

 

Tip in action:

I discovered this recently when carrying out a crawl for a client site. I had a list of over 2,000 URLs and I was crawling them to see which URLs were status 200 pages and which were JavaScript redirecting.

I was flummoxed when I saw the crawl registering all the URLs as live, status 200 pages! I knew this wasn’t the case, as when I accessed some of these in the browser, they JavaScript redirected to a different URL.

It was only when I changed my user-agent from Screaming Frog to Chrome that my crawl data reflected the real-world behaviour of these URLs.

 

How do I change user-agent?

In this example I’m showing you how to change user-agent in Screaming Frog, as this is the crawl software I use the most. I’m sure it’s also possible to change user-agent if you’re using other crawler software, such as DeepCrawl or Sitebulb.

  1. Before starting your crawl, simply go to Configuration –> User-Agent.
User Agent Configuration in Screaming Frog
 

2. You will see the default user-agent is Screaming Frog. Click on ‘Custom‘.

Default User-Agent Configuration in Screaming Frog
 

3. Choose a different user-agent (in this example I chose ‘Chrome’) & click ‘OK‘.

Switching User-Agent in Screaming Frog To A Browser
 

I hope that helps! Have a wonderful day 🙂

Sally xXx

Leave a Reply

Your email address will not be published. Required fields are marked *