This guide breaks down the more technical but on-page side of search engine optimization. The guide covers all the aspects of technical SEO, listing out the issues and solutions along with it. The guide aims to help you understand technical SEO to optimize your content and Rank better.
Beginner Guide to Technical SEO
Technical SEO refers to the behind the scene work that helps search engine crawlers identify and index your site more effectively. It is an applied knowledge that organizations and individuals use to optimize the technical elements of their websites.
Search engineer crawlers are instructed in a way that gives websites that display specific characteristics a higher ranking. These may include a secure connection, fast servers, and loading times. They all fall under tech SEO.
Importance of Technical SEO
- Helps your content Rank better by making sure your website is easy to crawl
- Ensures quick load of your content on the website
- It helps in identifying and removing duplicate content
- Includes guidelines for crawlers on what content to Rank and index
- Proper management of your website, even with thousands of pages
- Keeps your website safe and secure
Terms Related to Technical SEO Glossary
If you want to learn about technical SEO, It is essential to be well acquainted with its general terms. It is not complete but aims to help familiarize users with the terms related to technical SEO.
- Crawler / Spider / Bot
A crawler is a bot designed to retrieve information when a user makes a search query. The bots systematically browse webpages, analyzing what the webpage is about and keeping note of it. If a user wants information that might be on the webpage, it indexes it and shows it to the user.People can also use crawlers to identify any potential problems with the website.
- Sitemap
Sitemaps are files where you can store information about the website’s content. In simple terms, it is a mother tree, and all the content on your website is branched out. Sitemaps help search engine bots understand your website structure easily, making them crawl your site more efficiently.
- 404 Error
A 404 error occurs when a user is led to a dead end. The response status indicates that a server cannot find the requested resource.
- 301 Redirect
A 301 Redirect redirects users to a new destination URL if the previous URL is no longer in use. The redirect has authority over rankings; even if you click the old URL, 301 redirects will take you to a new one regardless of the Search Engine Rankings.
- Anchor Text
The Anchor text is the front end of a URL that redirects the page to another URL. Anchor Texts are visible; clickable text usually links in a different color (blue) and is underlined. Upon clicking the anchor texts, users are taken to another page. A good anchor text tells the reader what to expect if they click the link.
- Backlinks
A backlink links a website to another through anchor texts. A link in a website that connects it to the source or another website is known as a backlink. Backlinks are valuable for search engine optimization as they act as a vouch for your content by other websites.
- SSL Certification
SL or Secure Sockets Layer is a technology that keeps your internet connection secure and protects any shared data from leaking out. SSL certificates ensure that your website is safe and that no personal details are being modified or stolen. It is a big part of SEO as having a secure connection automatically gains favor from the search engine crawlers.
- Follow and Nofollow Links
SL or Secure Sockets Layer is a technology that keeps your internet connection secure and protects any shared data from leaking out. SSL certificates ensure that your website is safe and that no personal details are being modified or stolen. It is a big part of SEO as having a secure connection automatically gains favor from the search engine crawlers.
- Indexing
Indexing is a method by which a bot assumes your content ranking. Search engines process your content and store it in a database displayed to users according to their needs. It is how search engines organize information and enable super-fast responses that they deem good enough to query.
- 200 Status Code
Consider a 200 status code response from the server that gives a thumbs up on your request. This means that the HTML code can be loaded successfully and your site can show up.
Crawling And Indexing
Get Started,
Robots.txt
Robots.txt is a text file with a list of rules instructing a web crawler how to access and crawl your website. Robots.txt falls under the Robots Exclusion Protocol (REP), which sets web standards regulating how robots crawl the web and index your content. REPs are just instructions; it is entirely up to the crawlers to obey the rules. Not all search engines follow the protocol, but significant search engines like Google and Bing follow it.
In theory, users can use the to keep crawlers out of your site for indexing. Users can use the file to exclude domains, files, and directories from the search engine. However, Google could still index your content URL without visiting the page, as the bots can index it through your backlinks.
Use of Robots.txt
Robotx.txt file can be used for webpages to avoid crawling unimportant or similar pages from your site. Crawlers scour the internet for content and analyze it. There may be many duplicates or less important content on your site that you don’t want crawlers to waste their time on. Robots.txt file instructs a crawler on how to work, and its effect can be seen with different file types to prevent the server from being overwhelmed by crawlers.
A crawler can be blocked from accessing unimportant resource files such as scripts, styles, images, and audio files. So your crawl budget is well-spent on meaningful content.
A List of Terms Used In Robots.tx
- User-agent: Name of the crawler (Each crawler has its own username)
- Disallow: Prevent the crawling of your directories, files, and webpages
- Allow: Allows crawling of your directories, files, and webpages. (Strong command and overwrites disallow)
- Sitemap: Shows the location of the sitemap (Is totally optional)
- *: Includes all search engines to follow the directives
- $:To indicate the end of an URL
Website owners can place their sitemap directive on the robots.txt file. The following example shows how a sitemap is added to the robots.txt file.
| User-agent: Bot1 Disallow: /User-agent: * Allow: / Sitemap: https://www.example.com/sitemap.xml |
The following command disallows Bot1 from crawling any pages at all. The second line includes all bots except Bot1 for crawling the site.
Disallow crawlers to crawl a single site
| User-agent: * Disallow: /hotdog_pictures.html |
Disallow crawlers to crawl a specific file type
| User-agent: * Disallow: /*.gif$ |
The following command disallows all the bots from crawling all the gifs on the site.
What Happens If You Block Your Website From Crawling?
A robots.txt file prevents crawlers from crawling sites and pages a user has disallowed. This can include media, resource files, entire pages, or directories.
If your site does not have a robots.txt file, search engine crawlers assume that all the pages of a particular website that is public are indexable. Crawlers are free to crawl, and index pages of the website that they think are suitable.
Crawl Inefficiency
Crawl inefficiency can happen due to a multitude of reasons.Google bot has set its own crawling budget that depends on the server’s response time and errors. In simple terms, you will want google to crawl the most important and content-rich sites.
Crawling problems are displayed when a google bot tries to crawl your website, but the robots.txt file is blocking its crawlers. This can lead to bigger problems, as google can give up crawling your website after a few unsuccessful tries.
How To Prevent Robots.txt Errors: Basic Guidelines and Tips
- If you encounter problems such as your robots.txt file blocking URLs, you can use the google search console to find errors. Go to the coverage tab in the google console and look for the error section.
- Google robots tester to check files and ensure no directives block google Bots from site access.
- Robots.txt files are independent of the agent line and can be placed anywhere.
- Take care while writing lines for your robots.txt file to avoid wasting time in unnecessary debugging. Use proper wild cards for proper situations.
- If you made your website before 2019, you should recheck your robots.txt file. Google removed the indexed rules in 2019, so your no-index rules are no longer valid. Users can have their website indexed even if they don’t have it.
- The Robotx.txt file is case-sensitive and must be lowercase. The file must also be located directly in the root directory of your domain.
- Users should address each bot with a different record, as all bots on the web might not be compatible with a single record type of command. For Example,
User-agent: Googlebot
User-agent: Bingbot
Disallow: /
The following lines may be interpreted by bots differently, giving rise to errors and blocking URLs. It is recommended that users use different records while addressing different bots.
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /
Robots Meta and NoIndex
While Robots.txt and Robots meta sound similar, they each carry out different tasks. Robots.txt manages the accessibility of your content to crawlers. They block the crawlers from crawling your site but ultimately are not a solution to stopping your site from getting indexed.
This is where Robots’ meta directives come in. Robots meta directives are the definitive guidelines that order the crawler what to do and what not to do on a website. If you want to stop a site from being indexed, you use Noindex Robots Meta Tags.
Robots Meta tags are placed in the head section of a webpage.
<meta name=”robots” content=”noindex”>Essential Parameters for Robots Meta Directives
- Noindex: Prevents crawlers from indexing a page
- Follow: Instructs a crawler to follow the links in the page even if it’s not indexed (follow links advantages already discussed above)
- Nofollow: Instructs a crawler not to follow any links on a page
- Noimageindex: Prevents a crawler from indexing any images on the site
- None: Can be used instead of noindex and nofollow at once
- Unavailable_after: to remove the following site from indexation after a certain time period
- Nositelinkssearchbox: prevents Google from showing a search bar in SERP
 An Example of a Sitelink |
Why is the Robots meta tag important for SEO?
We already know how each site has a limited crawling budget. Admins should remove sites that have no content for users to read. This saves a search engine crawler’s budget and helps it become more efficient.There are various pages on a website, like admin login pages, that have no meaning to a visitor. By preventing pages from being indexed, the page will stop showing up in the SERPs.
The bigger a website is, the more the admins will have to control indexation and manage the crawling budget. This is why sitemaps are also important. An example of a robot’s meta tag is given below:
| <html><head><meta name=”robots” content=”noindex”> → meta tag</head></html> |
The following example of robots in the meta name. This includes all the crawling bots in the search engine. If you want to address a specific bot, you should replace “robots” with “<name of the bot>.”
If you want to block multiple bots but not all, simply copy and paste the meta name again.
| <html><head><meta name=”bot1″ content=”noindex”> <meta name=”bot2″ content=”noindex”></head></html> |
X-Robots Tag
X-robots tag is another way to control indexation. Users can use a robot meta tag to stop crawlers from indexing your HTML-based sites, which is straightforward. X-robots tag is used to manage the indexation of your content that is non-HTML based.
Non-HTML-based file types include PDFs, Microsoft word documents, mac pages files, etc. The x tag is not placed in the site’s header.
An example of an X-robot tag is given below:
| Header set X-Robots-Tag “noindex, nofollow” |
Note: Meta Robots Tags and X-robots-tag are supported by Google but not all the search engines.
Common Indexing Errors
- Always check that you do not disallow the crawling of content you want to get through robots.txt.
- If you want to deindex your content using the robot directive tags, do not remove the tags from your sitemap until your content is fully deindexed. This helps later on if you want to index your site again.
- Most of the error mistakes with indexation are due to human errors. It is best to take proper care in the spacings of your lines of codes and wildcards and properly audit them. Debugging is essential.
Things To Know
In order to remove a page from the index, a “noindex” meta tag must be added to the page. When Googlebot crawls and processes the page again, Google will process the “noindex” tag and will drop the page from the index in time.
For the “noindex” tags to be effective, the page must not be blocked by the robots.txt file, otherwise, Googlebot will not be able to view the “noindex” tags, and the page will not be removed. Wait for the page to be removed first, and then block it in the robots.txt file. More on the following is covered later on in the guide.
<meta name=”robots” content=”noindex”>
You should not block important resource files and media to avoid server overload. Crawling bots depend on such files for a proper analysis of the site and, without out, cannot do an excellent job of analyzing it for indexing.
Redirection
A redirect is a response from the server to the user. These Redirects can take users to other URLs depending on what the admin has set up.
If your request for the URL is successful, the server will send a response status code 200. If the URL has moved, it issues a redirect.
Depending on the time frame the redirect has been set for, Redirects are of two types:
- Permanent (301 redirects)
- Temporary (302 redirects)
Each carries its role in SEO and indexation. Here we discuss the role of 302 Temporary redirects and how it affects your search engine indexation and rankings.
A 302 redirect is temporary. This means that 302 issues a temporary URL to redirect the users. Since it’s a temporary redirect, Google will consider the original URL canonical and won’t change its indexation or rankings. There’s no point to it since it’s not permanent.
When to Use 302 Redirects?
Administrators can use the 302 redirects if they want to send visitors to another temporary site for a certain period. This is helpful if you are updating your website or want to test website updates without hampering the rankings of the older site.
Only use 302 redirects if it’s temporary. You will not be able to rank the newer site as google considers it quick.
How to Issue 302 Redirects?
Issuing a 302 redirect is very simple. There are a few methods to create the redirects, but the simplest one is editing your website’s .htaccess file. The file is present in your root directory.
| Please add a screenshot of your site’s root directory |
You can go ahead and add the following line of code to your .htaccess file.
| Redirect 302/ <old page>.html/ <new site of redirection>.html You can remove the line whenever you are done with your projects. |
If you own a WordPress Website use seo tools such as RankMath to help redirect easier.
You should employ constituent redirects when redirecting to another URL. If a non-canonical URL redirects to the canonical URL and then the canonical URL redirects back to the non-canonical URL, search engines may become confused and SEO may suffer as a result. It causes redirection chains.
Problems With Temporary Redirects
- The ERR_TOO_MANY_REDIRECTS error
Temporary Redirects generally don’t have issues that need experts to resolve. Most of the errors are due to faulty mistakes. The page can show a redirect error if the redirects are not valid. It would be best if you navigate your websites to see where the 302 error has been issued.
Another problem can be due to the plugins you use for your website. There are cases where you can incorrectly configure your redirects resulting in an error.
If you use WordPress, make sure your site address and WordPress address URL are the same, with or without the www. portion.
 |
Site Navigation
Website navigation is the links present in your website that connect other pages. Just as the name suggests, the purpose of this feature is so that users can navigate through your site efficiently.
Search engines also use the same feature. Search engines use your web navigation to discover and index new pages. It’s a sitemap that helps crawlers understand your site better.
Site Navigation is essential in eCommerce websites where user-generated URLs can lead to clusters. We’ll discuss more importance of Site Navigation for E-commerce businesses and how to improve it.
Navigation for E-Commerce
E-commerce widely uses dynamic URLs as they are based more on user-generated content. The UI that you see in such businesses are generally designed in such a way that it helps users discover what they’re looking for. In short, they’re ‘filters.’
These UI designs are also labeled Faceted navigation. Their purpose is for quick and efficient navigation that helps users find what they need by using multiple filters. For Example,
 The Category section you see on your left side is faceted navigation. It helps users shorten their options by sorting out the products based on their needs. In short, it is very important. The URL is updated to reflect the selection as users shorten their queries. This way, dynamic parameters work with Faceted navigation to show you the results. |
Problems with Faceted Navigations
- Duplicated content with different URLs
We have discussed how duplicate content affects the rankings as your contents compete with each other for being indexed. It causes keyword cannibalization, where many of your URLs compete in organic search. Big no-no. - Faceted Navigation causes index bloating.
With so many user-generated indexable URLs with no unique content, it provides no value for search engines. Faceted Navigation tends to be more specific and does not contain broad keywords.
This is the laptop a user finalized buying after filtering through so many options. The generated URL from this query is insanely long and has no value to search engines. No user will search for such a specific laptop through organic search.
Having such a page indexed will provide no actual value to the site or the user.
This is the laptop a user finalized buying after filtering through so many options. The generated URL from this query is insanely long and has no value to search engines. No user will search for such a specific laptop through organic search.
Having such a page indexed will provide no actual value to the site or the user.Fixing Faceted Navigation Problems
If you have followed up with the guide, the solution to such problems should be apparent. Here are a few solutions to solve errors with such navigation.- Using Canonical Tags
Canonical tags can fix problems of duplicated content and help in maintaining the crawl budget. Canonicalize the non-facet page so that google identifies it as the original and indexes it.
https://hotdogstore.com/gaming-laptops/Asus/?ratings=5&color=black
If this is your faceted URL, you will have to add a canonical tag indicating the main category of the page.
<link rel=” canonical” href=”https://hotdogstore.com/gaming-laptops/asus/”/> - Use of Meta Tags
If problems with indexing and rankings for faceted URLs persist, you should resolve them using robots’ meta tags. Setting up Meta Tags is simple; you can paste the code below in the head of your faceted URL.<meta name=”robots” content=”noindex”>
For X-Robots, simply use
X-Robots-Tag:noindex - Using Robots.txt
Canonical tags can be directly ignored by google for various reasons. While it may not be regular, it still could. This is why using robots.txt is another method to fix such problems.Users can disallow the crawling of a URL with robots.txt. To block the crawling of a URL with robots.txt, users can use the following code:User-agent: * Disallow: *size=*
Setting Up Images and Videos
Plain content without any rich multimedia like pictures and videos is not interactive. It enhances user experience and makes a website visually appealing. But trying to set media on your site can also lead to a few problems. Here are a few things to keep in mind when it comes to images and videos in technical SEO.
Things to Know
- If you have very high-quality images and videos on your site, it can take a while to load. Visitors on your site can bounce off from your site if it takes a long time to load. For this reason, you should optimize your images and videos by compressing and reducing their file size. Be sure to not compromise quality!
- Structured Data:
- Structured data provides additional information about the media that you are uploading to your site. It includes media descriptions, titles, alt texts, and file names.
- Alternative text also known as alt text is a text description of an image or a video. This text helps in SEO by helping search engines understand your media. Follow a rule of thumb for setting up alt texts: should be simple enough so that even blind people can understand it.
- The best way to upload videos on your platform is to upload none at all. Upload your videos to a video platform like youtube and simply link your video to your site. Likewise, youtube videos can also provide you with an additional source of traffic and helps reduce server load.
- Content Delivery Network
CDN refers to a group of distributed servers that work together rot provide fast delivery of internet content. It distributes the weight of images and videos and makes them load faster for users, which can help improve the user experience and search engine rankings.CDN servers need to be hosted, and for that, you’ll need to look into your options to purchase them. - Video Sitemaps
A video sitemap is an XML file that contains all of the videos on your website, as well as information about each video, such as the title, description, and thumbnail picture. Video sitemaps are just sitemaps but for videos.
Here’s an example of a video sitemap:
| <url> <loc>http://www.example.com/videos/how-to-cook-pasta..html</loc> <video: video> <video: thumbnail_loc>http://www.example.com/thumbs/some-video.jpg</video:thumbnail_loc> <video: title>How to cook Pasta</video: title> <video: description>Here’s how to cook Pasta</video: description> </video: video> </url> |
Add the sitemap to your robots.txt file and submit your sitemap to Google or other engines for crawling.
Some Commands and Terms used in Video Sitemaps
| <url> URL included in the sitemap <loc> Indicates the URL where the video can be found <video: video> Information about the video in sitemap <video: thumbnail_loc> Includes information on where the thumbnail is located <video: title> Title of the video <video: description> This is the meta description for the video. Max 2048 characters <video: content_loc> The actual video file must be in a supported format, not HTML <video: player_loc> Video player in which the video is played |
Site Improvement
Canonical URL
Canonical URLs are an essential part of SEO. While the word might seem intimidating and hard to understand, it is as simple as a learning table of 5. But before we talk about canonical URLs, we’ll have to understand a canonical tag.
A canonical tag is located in the source code that tells search engines which URL is the mother version of the page. If you have multiple URLs for the same page, you will use a canonical link to tell the search engine which page they should consider as the master version and index it.
The link to the page considered canonical is known as a canonical URL. Canonical URLs ensure that your SEO is strong and don’t get confused when there are different URLs to point to the exact content of the webpage.
Why Should There Be Multiple URLs for the Same Page?
Multiple URLs for the same page can be useful for a variety of reasons.
- You can have the same type of content across a few different pages to support multiple device types:
- https://www.facebook.com/growfore
- https://mobile.facebook.com/growfore
- To enable dynamic URLs for things like sorting or filtering parameters or session ID based on custom input characters. Any URL with characters such as ?, =, & are considered Dynamic.
- https://www.shopwithme.com/products?category=laptop
- https://www.twitter.com/1029321?ref_src=twsrc%5Et
- Your system positions the same post under multiple sections under multiple URLs.
- https://blog.example.com/recipes/how-to-make-a-pasta
- https://blog.example.com/pasta/how-to
- If your server is set up to offer the same content for http://www. and non-www://http://https/protocol/port variations:
- https://example.com/razer-laptops
- https://www.example.com/razer-laptops
- Syndicated content is the publishing of owned content to other websites. This way, many sites for the same content can exist. Users can set to help the search engine determine the site to index.
- https://blog.example/best-costumes-to-wear-for-haloween
- https://news.example/what-to-wear-for-haloween
Why Should You Use Canonical URLs?
There can be various circumstances where multiple URLs for the same content can happen. Canonical URLs can help Google understand your site better.
Canonical URLs inform the search engine of the mother URL and separate it from its many duplicates. They help google crawlers to crawl and index the real site properly.
How to Use Canonical Tag?
Specifying a canonical URL is very easy. Specify the canonical URL in your HTML code.
Use the tag in your page’s header.
The Canonical URLs Issues
Canonical URLs are very easy to recreate and are easy to have issues with it. Canonical issues are generally seen with e-commerce business sites as they have URLs that change with user interactions.
If your site has an SSL certificate, you can access your site by using both HTTPS and HTTP. The two versions can create duplicates of every single page of your website. This can also happen with www and non-www URLs.
There are multiple URLs for the same content for different types of devices; this can create canonical issues. The same goes for syndicated content.
Basic Tips To Prevent Any Canonical URL Errors
- If you have any problems with SSL Certificates and WWW and non-WWW canonicals, consider implementing sitewide 301 redirects for the duplicate pages. By redirecting users to the correct version of the URL, canonical issues will no longer occur.
- Syndicating content is not a violation, and users are free to do so. Remember to add a rel=canonical tag to your secondary site to redirect the content to your URL. By doing this, google recognizes the canonical site and gives it a priority in the rankings.
- Refer duplicate pages to your original content so that google identifies the original site to prioritize it.
- Do not use the robots.txt file for any canonical actions.
Canonicalization Problem in E-Commerce Sites
There are many problems regarding duplicate content in E-commerce sites, where there are multiple URLs for the same product.
For example, if two URLs:
- www.example.com/category/fruits.html
- www.example.com/category/fruits.html?page=1
Exist, these URLs are identical in content as both display fruits in the same category. It is important for users to use the rel=canonical link element to indicate to search engines which URL should be considered the original and indexed.
URL Parameters
URL parameters are additional elements present in your URLs that help you filter, organize or track information on your website. They serve as a way to add additional information to a URL structurally.
Parameters lie at the end of a URL after a ‘?’ symbol. Multiple parameters in a URL are separated by a ‘&’ sign. These parameters contain a key and value separated by a ‘=’ sign.
| https://www.pictureofhotdog.com/browse?</span >style=chicago&sort=rating |
The above URL that you can see is a Dynamic URL. Dynamic URLs are the user-generated URLs that are generated the moment a user submits a search query. If you have a URL parameter in your URL it’s dynamic.
On the other hand, we have a static URL. It’s a URL that is stored as a whole on the relevant server. For example,
| https://www.growfore.com/blogs/seo-services The following link does not have any visible parameters, and its data is stored in the system. No amount of user interactions can change the URL. It’s static. |
There’s a huge confusion among young SEO enthusiasts about whether they should practice using dynamic URLs or not. Let’s review the positive and negative aspects of using URL parameters.
A basic guide to URL parameters and how they impact SEO
- URL parameters are best used for e-commerce websites or sites in general that need high user interactions. If you are a service-based site with limited services, you should stick to static URLs. Static URLs help more in SEO as it has a higher click rate.
- Users should always canonicalize the main static version of the URL so that it stays on the top. They should set up canonical tags on the parameterized URLs so that they always reference the preferred URL for the site.
- This is where robots.txt comes in. You can choose to block the crawling of endless URLs with non-unique content across your website.
| User-agent: *Disallow:/*?tag=* The following tag will block crawlers from crawling URL parameters. Make sure all your important URLs do not have parameters to avoid the resourceful content from being crawled. |
- Google controls more than 80% of the search engine. According to google developers, Dynamic URLs are as indexable as Static URLs with their optimized search engine crawlers. While this may be true, It does not mean you shouldn’t structure your parameters.
- Users should remove unnecessary parameters and limit the dynamic rewrites so that they can maintain them. You should create static content equivalent to the original dynamic content. This, however, does not apply much to e-commerce websites.
Types of URL Parameters
There are two types of URL parameters.
- Content-modifying parameters: Parameters that modify the content displayed on the page.
example.com?productid=xyz
To take users to a product named XYZ - Tracking parameters: Tracks user network or how the user was referred to the site.
example.com/?utm_source=newsletter&utm_medium=email
To track where users came from to your newsletter.
Cons of URL Parameters
- Dynamic URLs are not informative. Since the URLs are generated with user interactions and filled with random characters, users don’t know where the link is headed.
Take this, for example,
https://www.gonzalez-shop.co/index.php?id=42
Users have no idea what this link possibly could be about. Without many ideas, users are less likely to click on your link. This brings fewer clicks to your websites, even if it pops up in the SERPs. - They cannot be shared. Well yeah, you could copy-paste it, but how about trying to share this link physically? It’s almost impossible to remember strings of random characters without any correlation. In short, Static URLs are better quality of life adjustment.
- They cannot be shared. Well yeah, you could copy-paste it, but how about trying to share this link physically? It’s almost impossible to remember strings of random characters without any correlation. In short, Static URLs are better quality of life adjustment.
- The same content with different links can clash with your SEO rankings. Why fight with your own self for the spot on #1?
If it’s so disadvantageous, why use it?
Sitemaps
Sitemaps are files where you can store information about the website’s content. In simple terms, it is a mother tree, and all the content on your website is branched out. It is a file where you provide information about the content on your site; pages, videos, pictures, and other files and the relationship between them. Sitemaps are a source of information for search engine crawlers and provide them with valuable information about files.
Understanding Key Terms
- <url>: opening line to add your URL
- <loc>: add URL
- <lastmod>: When was the page last modified?
- <changefreq>: How often does the page update?
– Never, yearly, monthly, weekly, daily, hourly - <priority>: How important is this page compared to your other pages on a scale of 0.1 to 1? (1 is the highest priority)
- <urlset>: open or close sitemap
Why are Sitemaps important?
We have already discussed how sitemaps act as a roadmap to your site. Sitemaps make search engine crawling more efficient as it knows which site to navigate.
A crawler will only spend a short time trying to crawl your website. Each site has a specific crawling budget allocated to it, and if your website is large, a crawler might waste its crawl budget on figuring out which pages exist on your site. XML sitemaps help search engine crawlers analyze your website correctly for faster indexation. Crawlers might overlook the newer pages if you update them occasionally.
Note: Sitemaps do not affect your SE rankings; they make it easier for crawlers to crawl your website.
Types of Sitemaps
There is two types of sitemaps and they are:
- HTML Sitemap
HTML sitemaps are sitemaps made for the viewers. These are made to serve website visitors and help them navigate to a specific page. In short, they are internal links across your web pages. HTML sitemaps are clickable lists of pages on a website that reform and organize your website, generally found linked in website footers.
An HTML Sitemap of a E-commerce Business.
- XML Sitemaps
XML sitemaps are dedicated to search engine crawlers. These sitemaps act as a roadmap to tell search engines about the list of content on your website and directions on how to access it. It’s like you are being given a map of a place. Keeps things simple and convenient.HTML sitemaps, on the other hand, do not contribute anything to search engine bots or SEO bots. They exist to interlink pages and maintain them. It’s the site’s blueprint and oversees the structure and connection between pages. HTML Sitemaps help direct the bots and show them the structure of your website.
An XML Sitemap of Growfore
How to build an XML sitemap?
While you can create your own XML site, you will have to add each URL manually. You’ll need to gather it first. This can be done with the help of bots too. But automated XML Sitemap is the way to go.
- You can use google’s own tool to search for your URLs and add them to a sitemap. Users can also use a secondary online tool to have a bot make a sitemap for them. The process is easy and simple, with no extra processes required.
- Submit your sitemap through the Google search console
Google Search Console
What Pages to Exclude in your XML Sitemap?
- What Pages to Exclude in your XML Sitemap?
- Duplicate pages
- Paginated pages
- Parameterized URLs
- Site search result pages
- URLs created by filtering options
- Archive pages
- Any redirections (3xx), missing pages (4xx), or server error pages (5xx)
- Pages blocked by robots.txt
- Pages blocked by robots.txt
- Pages blocked by robots.txt
Pages blocked by robots.txt
While Sitemaps aren’t necessary for all websites, it is recommended by most SEO experts. Sitemaps help sort out your sites so that bots can crawl your site efficiently. Sitemaps can also help in the indexing and ranking of your older content through the help of links. Sitemaps are necessary:
- If you have a big website, navigating through it can become difficult.
- If you have a bad interlinking strategy,
- If your website is new,
- If your website has a lesser number of backlinks,
- If your site has a lot of rich media (videos, gifs, pictures),
Additionally, you might not require Sitemaps if
- Your site is small and will remain that way (500 pages or less than that)
- Your site is well interlinked
- Your site does not have additional media like pictures and videos.
Here are a few tips to help you avoid errors:
If your sitemap generates a 404 error code by any chance, it can be due to your URL being blocked by Robots.txt files. Give it a look.
Many sitemap errors can be resolved by clearing the cache. Clear the cache from your plugin, Cloudflare, and browser followed by a refresh.
You will need to delete the sitemap in google and submit it again if errors persist. Do give proper auditing to your sitemap.
A single sitemap can only support up to fifty thousand (50,000) URLs. Any more than this, and you will have to make multiple sitemaps. You can split up your XML sitemap into smaller files to help speed up the crawling process. We recommend a maximum of 1500 pages on a single sitemap as the google search console has a limit to export only 1000 URLs.
Using self-referential canonical URLs in sitemaps can help ensure that search engines understand which version of a page should be considered the original and indexed. Users should also make sure that their URLs in the sitemap return a 200 HTTP code. If not, the page in the sitemap might not be indexed by search engines.
Backlinking Issues
Google tries to ensure the safety of SEO enthusiasts by cracking down on spammy SEO tactics. While most of the efforts have been successful with newer and improved algorithms and security, some black hats still escape Google’s watch.
Here is a short guide on avoiding linking issues and technical SEO spam that make you lose rankings altogether.
What is SEO backlink spam?
Backlinks are good; they give search engine algorithms the thumbs-up sign and build trust, right? Most of the backlinks to your site are good. However there are spamming backlinks from shady or harmful sites that redirect to your site. This can leave a negative impression of your site on the search engine algorithm.
And trust me, you do not want a penalty from Google. Getting a penalty from Google is having your content removed from SERPs. Spam backlinks is a shady black hat technique that falls under negative SEO. In most cases, negative SEO is done by your competitor to harm your rankings.
How to detect Backlink Spam?
Link spam generally comes at once. You’ll notice an unusual rise in backlinks to your sites, most coming from different countries. The contents from your site also are found in other languages you don’t recall adding.
- Detect low-quality sites that backlink to you
Users can use various SEO tools available to them to identify how solid or trustworthy the backlinking site is. It’s generally called a domain rating or a domain score. You can then select the sites you feel are scummy and record them.
Learn More about SEO Tools and how it works.
Getting Rid of Spam Backlinks
You now have a list of suspicious sites you want to block. All that’s left is to tell Google to blacklist these sites so that you do not get penalized.
This feature is known as disavow. Head up to the google search console and search for a Disavow category. Submit the file to disavow the sites, and then you’re done! Good riddance to all those spam links!
The following guide works to stop spammy links and out-of-context links referencing the site, a large number of unnatural links, and staging/production site links referencing the site. As the search engine bot itself gets smarter, it is learning to ignore spam backlinks altogether. Here’s what John Mueller, a Senior Webmaster Trends Analyst at Google had to say about spammy backlinks.
 |
By any chance, if your site gets spammed by backlinks and spammy references, you should not be disheartened. According to John Mueller, the number of backlinks Does NOT affect your search engine rankings at all. It’s quality over quantity, so even if you only have one strong backlink reputation, Google will not decrease your rankings.
Error Pages
You aren’t alone if you have ever encountered a 404 error while accessing a site. 404 Errors are the most common HTTP error that exists today.
A 404 error or a “header response code” is an error displayed when the crawler or user request cannot access the page. It’s just not there where it’s supposed to be. A 404 error generally appears when users make minor errors when typing a URL. It can also pop up when admins do not issue 301 or 302 redirects.
Do Error Pages Harm the SEO?
Users cannot access the site anymore. While Error pages sound grave to SEO, that’s not quite the case. The broken links do disturb a visitor’s impression, but a few 404 errors on a page do not decrease the rankings of a site overall. Google themselves have plenty of 404 mistakes now and then.
But you still lose visitors to your site, which may affect you somehow.
“404 errors won’t impact your site’s search performance, and you can safely ignore them if you’re certain that the URLs should not exist on your site”
HOWEVER,
There is one case where 404 errors hurt the rankings of a page. Remember inbound links?
Inbound links are essential for a well-optimized SEO as google bot deems inbound links as a ranking factor. If you have an inbound link that still brings in much traffic, you should issue a 301 redirect.
With a redirect, all your rankings from those inbound links will be recovered. It’s like losing valuable pieces in chess to aid you. Keep it in mind whenever a 404 error page pops up.
The Other Type of Error Page
Soft 404 is another type of error that Google issues. The error does not have a header response code, but the Google bot thinks something might be wrong with the site. The server gives a thumbs up and sends a 200 response code, but Google goes, “nope, something’s wrong, I can feel it,” and issues a soft 404 instead.
There isn’t anything wrong with your site; it’s alright. You’ll just have to go to your google search engine console to determine the problem.
Setting Up Your Own Error Page
Error pages aren’t good at all. But if you’re going out anyways, why not go out with a bang?
Error pages can become excellent landing pages. Landing pages are static pages that users land directly. It can become a great chance to advertise yourself. Here are a few tips for setting up your 404 error page.
- It should have an error message (obviously).
- Since error pages can be a great way of advertising, drop links to some of the best content the site offers. This way, users stay on your site longer.
- Have methods given on the page to contact the webmaster so that they can fix the broken link.
- Adding a bit of humor to your 404 error page keeps things fun.
- Remember branding. Even if it’s an error page, it should be yours.
- Add a search box on the page where users can search for other content on your site. It reduces bounce rates significantly.
- Most importantly, please keep it simple. Avoid cluttering your page.
Most web-building sites have a built-in feature for 404 error pages. All you have to do is design a 404 page and configure the server for it.
Structured Data
Ever searched for recipes online and had something like this come up? This isn’t like a normal search result, is it? This result has user ratings, recipe timings, and attractive pictures. Such search results are also known as rich results.
How do you get rich results? This is all thanks to structured data.
Structured data are codes written in a certain order or structure. They usually refer to implementing markups (markup: ways to write code) on a webpage to provide additional detail about the content. Its goal is to help search engine bots to help understand the page.
Issues with Structured Data
Issues with your structured data can easily be found with Google’s Rich Results Test.
Google structured data testing tools have made it easier to find bugs in your structured data and fix any potential errors in it.
You should ensure your security protocols are not blocking Google’s tool from rendering the content. There are also free plugins like Schema Tester that help locate any errors within your code.
False Practices of Structured Data That Lead To Penalty
- Applying a page-specific markup sitewide
If a fast food chain of a certain location has gotten good reviews, the management could update their sites so that a good review is assigned to all the chains belonging to them. This is a bad practice and can lead to a penalty from Google. Avoid any spammy tactics when practicing Structured data markups. - Marking up false reviews
Another dirty tactic is marking up false good reviews by companies without any reviews or low reviews. Marking up reviews puts the reviews in rich results, creating a false impression. This also applies to only marking individual reviews, not the average. - Delivering different data based on Users
There are instances where companies show different markups based on users’ locations. While this could fall under targeted ads, it is perceived as a manipulative action by Google. Doing so might lead you to get a penalty.
Schema.org
Schema.org is a project that contains all structured data markups supported by search engines. Users can use Schema.org to find the markup they need for their particular page.
The schema supports three different markups:
- JSON-LD (preferred by google)
- Microdata
- RDFa
Many in-depth studies need to be done to understand Schema markups fully. Head over to Schema.org to learn more about the markups.
Here’s an example of how Schema works. The above text is understandable by human brains, but how about bots? By changing the above code into JSON-LD, we have made the code understandable by a search engine bot. This helps add structured data to your SEO. |
HTML Rendering
The following topic should be easier for Gamers to understand. Rendering is a process where the bot retrieves your pages, runs your code, and tries to analyze your content. This is essential for Google as rendering is used to examine your site content against many others and rank it accordingly.
Web Page Rendering
The term is referred to the process of building a webpage. It doesn’t build it from scratch but individually collects materials to give the final result. Let’s give a good example.
You have every ingredient in the kitchen to make a good hotdog. You prepare the ingredients, lay it out, start boiling your sausage, cut up your greens, ready your sauces, heat your bread, and finally compile it.
The process of building your hotdog to the end is called rendering. It’s the start to finish of loading a webpage.
Importance of Web Page Rendering
The Google bot needs to index your content to rank and appear on the SERPs. And for indexing, your content first needs to be rendered. The faster you render the page, the better for you.
Role of JavaScript in Rendering
JavaScript plays an important role in SEO. This is also true for rendering. The animated images you see in an online shop or animated buttons are done with the help of JavaScript. Remove it, and users will no longer be able to see media present on the site.
JavaScript execution is just a part of the rendering process.
Expensive JavaScripts
The more files or scripts you have on your site, the longer it’ll take for devices to render your website fully. If you add expensive (time-consuming and resource-expensive) JavaScript to your site, older devices might take longer to render your website. The following also applies to newer devices. So the more expensive your rendering is, the worse the user experience can be.
Solutions For a Better User Experience with Rendering
- The more resource heavy your site becomes, the more the users will have to wait to access your content. Keep your site clean and less resource extensive. Try reducing the size of the media you upload to your site.
- Only use the scripts you need. Shipping unnecessary scripts increase the load and slow execution time.
Tips for Finding Unnecessary Scripts- Open Developer’s Tools.
- Click the 3 dots in the upper right corner.
- Select more tools, Coverage.
- Reload the page.
Picture: Search engine journal
The sweet spot is below 1MB of used space. Try splitting your scripts if it’s too much. - Yeah, your site is pretty, pretty clustered. Avoid littering your site with tools and scripts that are definitely unnecessary. Especially third-party scripts, which have higher security and privacy risks.
Picture: Search engine journal
The sweet spot is below 1MB of used space. Try splitting your scripts if it’s too much.
HTTPs
HTTPS (Hypertext Transfer Protocol Secure) is a protocol for secure communication on the internet. It is essentially an extension of the standard HTTP protocol, with the addition of a security layer through the use of SSL/TLS certificates.
HTTPS is just an upgraded version of HTTP which is more secure. While Google has stated that HTTPS sites will get a tiny SEO boost, trying to set it up can cause a lot of SEO problems in return. Let us look at technical issues while setting up HTTPS.
 An example of HTTP vs HTTPS site.  |
Errors in HTTPS affecting SEO
- Warnings for Mixed Content
A mixed content warning happens when a website is loaded via HTTPS but contains elements that are loaded over HTTP, such as pictures, scripts, or stylesheets. This might result in a warning notice being shown in the user’s browser, which can have a negative influence on the user’s experience and trust in your site. Check that all site components are loaded using HTTPS or relative protocol URLs. - Redirect Chains
When migrating a website to HTTPS, it’s common to redirect all pages from HTTP to HTTPS. However, this can create a chain of redirects, which can slow down page load times, affecting the user experience and search engine rankings.
Use the 301 redirects, redirecting all traffic from HTTP to HTTPS, as well as updating internal links and URLs to use the new HTTPS URLs. Do not use 302 redirects to redirect your HTTP sites to HTTPS sites as Google will not index your HTTPS site if it’s temporary. - SSL/TLS Certificate An SSL/TLS certificate is necessary for HTTPS to work, but the certificate must be valid, installed correctly, and not expired, this way the site can establish a secure connection with the user’s browser.
Make sure the certificate is from a reputable provider and that you have followed best practices for installation and configuration. A good tool to help you test your SSL is Qualys SSL Labs. SSL Report for Growfore, A being the best.
SSL Report for Growfore, A being the best.Indexation
Another problem of duplicate content also arises from two different site versions. This wastes the crawl budget and has link dilution problems.
Google has stated that the HTTPS version of the site will be leveraged more in terms of indexing but the bot will still crawl your HTTP version of the site wasting your crawl budget.
There also arises a problem of link dilution. Even though Google favors HTTPS sites, your HTTP site can still be accessed by users, diluting your traffic into two different sites.
To fix this, you can use Search Console to submit a sitemap, or you can use the ‘Fetch as Google’ tool, to request that Google crawl specific URLs.
Use canonical tags properly to address your indexation and crawling budget problems. Ensure all your internal links, social and other links lead to your HTTPS site so that google knows which one is the better version of your site.
Analytics tracking
If you’re using analytics tools such as Google Analytics, make sure that you update the tracking code to use the new HTTPS URLs, or it will not track the data correctly.
It’s important to keep in mind that switching a website from HTTP to HTTPS can have some impact on the website’s SEO, for example, all internal links should be updated to HTTPS, and all external links pointing to the website too.
Also, there may be temporary drops in ranking and traffic in the short term, but in most cases, if the site is properly configured, that’s only a temporary situation.
Mobile SEO
General SEO and mobile SEO are indeed different things. While desktop SEO has no geographic focus, mobile SEO is typically optimized for local results. It makes sense, too; mobile users generally search for local results. Mobile searches alone make up more than half of search engine queries. Myriad errors can come up while trying to work on a mobile site. We’ve compiled these errors and devised solutions to help you optimize your site.
Issues
- Non-Responsive Design
The biggest mobile SEO flaw anyone can think of is a non-responsive design. If your site was a responsive design, it ensures that your site displays content properly on a wide range of device sizes. Example: facebook.com can open on multiple devices without distorting content; everything is visible, just smaller. - Slow Page Load Speed
The more content-heavy your site is, the longer it can take to load. If your site takes a while to load, users can easily lose interest. Slow page load speed can lead to high bounce rates and lower visibility in mobile search results. - Lack of Mobile-Specific Meta Data
Not using meta tags, such as the “viewport” meta tag, to control how the content of your website is scaled and displayed on different mobile devices can lead to a poor user experience and visibility in mobile search results. - Blocked Resources
By blocking access to certain resources (CSS, javascript, images, etc), it can negatively impact the way the site is rendered on mobile.
Mobile App Interstitials
Ever had a pop-up full-screen ad when browsing your phone? They are called Interstitial Ads. They are full-screen ads that usually cover your entire screen. These types of pop-up ads are considered highly engaging with higher click ratios. Although it can be annoying, google has approved the use of such ads.
There are a few things that should be considered before setting up Interstitials.
The advertiser must set an AD that covers only 50 and 40% of the screen width and height. It’s a requirement set by Google itself. Besides that, interstitials can also cause a variety of secondhand SEO problems.
- Bad User Experience
Ad Interstitials are effective but interrupt a user’s experience too. Since the Ads generally cover a lot of space, they can lead to high bounce rates and lower engagement. - Affects Mobile Rankings
The use of intrusive interstitials will definitely impact a website’s search rankings. Google has demoted sites that use full-screen ads without a proper way to close the ads ever since 2017. - Reduces Conversions
Ad interstitials can reduce conversions by distracting users from the main call to action or making it harder for them to complete a task, such as filling out a form or making a purchase.
Best Practices for Mobile SEO:
- Have a responsive design: A responsive design leads to good user conversions and lower bounce rates. This can be easily done by adding a line of code to your HTML head portion. It can be found in Growfore’s source code too.
<meta name=”viewport” content=”width=device-width, initial-scale=1, maximum-scale=1“> - A fast-loading website is essential for a good mobile user experience. Google has a tool called PageSpeed Insights that can help you analyze and optimize the speed of your website on both mobile and desktop devices.
Google’s PageSpeed Insights for Mobile and Desktop. - Ensure that the content of your website is easily readable and navigable on a small screen. Use large, easy-to-read fonts and keep the content concise and to the point. Optimize image file sizes, minify CSS and JavaScript, and use a content delivery network (CDN) to minimize server response time to increase website load speed.
- Also, check your robots.txt file to ensure you are not blocking important resources for your mobile site. Administrators should also use lower-resolution images to help load your site faster.
- Desktop sites should be made canonical to mobile sites for better SEO.
Internalization
If you ever plan to expand your site to many different regions, It’s called internalization. Internationalization (i18n) in SEO refers to the process of optimizing a website to reach and engage an international audience.
SEO Checklist for Internalization:
- Hire a language translator to translate your content. AI is not fully competent enough to translate all but simple phrases. Chances are your content is a bit complex for other international users to understand. A faulty translation can be a huge problem for international users. Create a proper target audience list for different countries.
- Use the tools you have at your disposal. Webmaster tools like Google Search Console can help you configure your website to target the proper audience and countries.
- The problem of duplicate content can arise as multiple language versions of the same site exist. Use canonical tags to indicate the original version of the site to avoid impacting your search rankings.
- Create content according to the region you want to target. Have content tailored to a specific culture, language, and interests of the community to improve engagement and win hearts.
Hreflang Tags
Hreflang tags are the HTML elements that inform search engines to know which language version of your website is meant for which audience. This helps a lot in internalization, as the tags help minimize confusion for both users and engines.
Admins can follow the code below and add it to the header section of their site source code. <link rel=”alternate” hreflang=”x” href=”http://example.com/page” />
X = language or regional code
<link rel=”alternate” hreflang=”en-us” href=”http://howtodo.com”/how-to-approach>
Things to Remember While Using Hreflang Tags
- Use them on all versions of the page: Hreflang tags should be included on all versions of the page, not just the default version.
- Use the correct language and regional codes: Use the correct language and regional codes to indicate which version of the page is intended for which audience.
- Use the correct URL: Use the appropriate URL for each version of the page, as this will help search engines understand which version of the page should be served to users.
- While Google has the majority of the search engine market, people should also know that not all users use google. Yandex in Russia and Baidu in China are some examples. This is why users should try and cater to optimizing sites for such search engines.
Content Optimization
It is essential that you optimize your content properly for a better user experience. Here are a few errors you can make while writing and publishing your content.
| Errors In Title Segment |
| Missing Titles |
| Titles missing brand name |
| Titles have inconsistent brand names |
| Titles contain pipes |
| Titles are too long |
| Titles are too short |
| Titles for paginated pages do not contain page numbers |
| Unique pages (pages containing self-referential canonical tags) contain duplicate titles |
| Titles are generic |
| Pages missing H1 tags |
| H1s are irrelevant to the content displayed on pages |
| Errors in Meta Descriptions |
| Missing meta descriptions |
| Meta descriptions are too short |
| Meta descriptions do not contain a clear call to action |
| Meta descriptions contain pipes |
| Meta descriptions for paginated pages do not contain page numbers |
| Unique pages (pages containing self-referential canonical tags) contain duplicate meta descriptions |
| Meta descriptions are generic |
Hopefully, you’ve grasped the basics of Technical SEO.
Content Optimizing to Avoid Stale Contents
If you still have outdated content on your website, you better update it! Any content that is no longer up-to-date, irrelevant, or accurate can be considered stale.
Problems with Stale Content
- Decreases user engagement as stale content can be misleading with higher bounce rates. No one wants incorrect information.
- Stale content is regarded as lower quality by search engines, leaving your site with a negative impression. Lower-quality content will rank lower by default.
- The more users trust your site the higher search engine bots will rank your site. False information will decrease user engagement and trust in your site.
- Stale content also has no value to a server, so it is a waste of server space.
Addressing Stale Contents
Solutions to stale content are generally direct.
Be sure to conduct regular content audits and review them. Update any content on your website if necessary. Content that is stale and useless should be removed from your site to free up server space and for a better server reputation. It’s always better to create fresh new content for the site than to plagiarize someone else’s.
