Sit down in your developer’s swivel chair as we explain some simple but effective technical SEO tweaks you can make yourself.
Below are two techie tweaks for SEO that you can implement without bringing in the developers. Grab your insect repellent and bug box because you’re going to hunt for spiders.
Tech tweak One: Implementing robots.txt
As with all web crawlers, spider bots have an insatiable appetite for exploring. If your website has a few corners that you’d prefer went unnoticed, knowing how to stop the bots is a useful tweak. And we’re going to show you how to do it without dipping into developer time.
The first thing a spider will look for when visiting a web page is robots.txt. This gives the spider instructions on what to look for and where. If you’re using robots.txt for the first time, you need to know which instructions to give to the spider bot so it understands what to do. This is a text file and is not HTML-based. It needs to be accessed from a server’s root location i.e. http://yourdomainname.com/robots.txt
Key features of robots.txt files:
This is the name of the robot given by directories and search engines. For example, Google’s robot is called Googlebot. Your user-agent goes at the top of the robots.txt file and will be read first from left to right by the visiting bot.
using an asterisk (*) indicates that the instruction is to be read by all.
user-agent: * indicates instructions in the txt.file are for all robots.
user-agent: Googlebot explains instructions are only for Google’s benefit.
writing disallow under the user-agent line tells spiders where you don’t want them to crawl. If you have a page(s) or a folder(s) that you’d prefer wasn’t indexed by search engines, you would write:
Disallow: /About us page/
Disallow: /animals pics/
Not aware of anywhere you don’t want the bots to visit? It’s still best practice to include an empty/default robots.txt file. This cuts down on ‘404 page not found’ errors that can occur when bots are searching for txt.file instructions.
Write empty/default as follows:
Tech tweak 2:URL parameter handling in Webmaster Tools
First, you’ll need to visit Webmaster Tools to find out whether you actually need to tweak your parameters (NB: As of August 2019, this is only available in the older version of Google Search Console but should be heading to the new one soon). If Google says everything is fine, don’t start tinkering just for the sake of it. If you do, you risk pages being dropped from search results. If you don’t find a message from Google saying that everything is fine, it’s time to get involved.
Using the URL parameter handling in Webmaster Tools you can tell Googlebot where to crawl and which pages to leave alone. This cuts the amount of time the bot spends crawling to check over duplicate URLs, duplicate content and smaller sub-sections of your site. The result? The bot can spend more time indexing site pages that matter.
There are two main categories of URL parameters:
- URL parameters that don’t change the content. These are commonly used for tracking and referring purposes
- URL parameters that do change the content. These are used to separate clusters and help narrow down searches for specific items/colours/sizes etc.
Parameter handling in Webmaster Tools also allows you to tell Google exactly what different parameters do on your site. This way, it’s clear what needs to be ignored and the risk of duplicate content issues is reduced.
Why you should tell Google what your parameters do
The URLs below point to the same thing but one URL has come from an email campaign and includes a tracking code for Google Analytics:
We’d want Google to ignore the ?utm_medium and ?utm_source parameters, as they don’t change the page’s content. This is just extra information for Google Analytics.
In the case below as another example, you’ll see that all three URLs point to exactly the same thing – plaid trousers for men. The URLs are just sorted and filtered differently.
Having a good developer with sound SEO knowledge in your team is a great start for getting your technical SEO right. But by developing your digital skills as a marketer, you’ll have a better understanding and more control of your site. That can only be a positive thing in our eyes.