Hello friends!
Happy Friday to you all 🙂 In this blog post today I just wanted to share a quick tip with you.
SEO tip:
When running a crawl of a site, switch your user-agent from the default (crawler) to a browser (such as Chrome).
For example, if you’re crawling a site with Screaming Frog, the default user-agent is Screaming-Frog. I recommend you change this!
Why?
Websites can treat crawler user-agents differently to users (usually less sympathetically!). By switching your user-agent to a browser, you look like a real user to the website you are crawling.
Tip in action:
I discovered this recently when carrying out a crawl for a client site. I had a list of over 2,000 URLs and I was crawling them to see which URLs were status 200 pages and which were JavaScript redirecting.
I was flummoxed when I saw the crawl registering all the URLs as live, status 200 pages! I knew this wasn’t the case, as when I accessed some of these in the browser, they JavaScript redirected to a different URL.
It was only when I changed my user-agent from Screaming Frog to Chrome that my crawl data reflected the real-world behaviour of these URLs.
How do I change user-agent?
In this example I’m showing you how to change user-agent in Screaming Frog, as this is the crawl software I use the most. I’m sure it’s also possible to change user-agent if you’re using other crawler software, such as DeepCrawl or Sitebulb.
- Before starting your crawl, simply go to Configuration –> User-Agent.
2. You will see the default user-agent is Screaming Frog. Click on ‘Custom‘.
3. Choose a different user-agent (in this example I chose ‘Chrome’) & click ‘OK‘.
I hope that helps! Have a wonderful day 🙂
Sally xXx
