Happy Friday to you all 🙂 In this blog post today I just wanted to share a quick tip with you.
When running a crawl of a site, switch your user-agent from the default (crawler) to a browser (such as Chrome).
For example, if you’re crawling a site with Screaming Frog, the default user-agent is Screaming-Frog. I recommend you change this!
Websites can treat crawler user-agents differently to users (usually less sympathetically!). By switching your user-agent to a browser, you look like a real user to the website you are crawling.
Tip in action:
It was only when I changed my user-agent from Screaming Frog to Chrome that my crawl data reflected the real-world behaviour of these URLs.
How do I change user-agent?
In this example I’m showing you how to change user-agent in Screaming Frog, as this is the crawl software I use the most. I’m sure it’s also possible to change user-agent if you’re using other crawler software, such as DeepCrawl or Sitebulb.
- Before starting your crawl, simply go to Configuration –> User-Agent.
2. You will see the default user-agent is Screaming Frog. Click on ‘Custom‘.
3. Choose a different user-agent (in this example I chose ‘Chrome’) & click ‘OK‘.
I hope that helps! Have a wonderful day 🙂