Things to consider before adding a disallow to your robots.txt file
The robots.txt file is a core tool in any technical SEO’s arsenal. Auditing it is paramount in any standard technical SEO review due to the hard-line nature that typically comes with the file. Unlike other SEO components, directives within the file are usually more respected (for search engines, anyway) as opposed to just being hints […]