One study found that organic search traffic increased by 22% after removing duplicate web pages, while Moz reported a 13.7% increase in organic search traffic after removing low-value pages. Oftentimes, it is necessary to deliberately prevent search engines from indexing certain pages from your website to boost SEO. However, that may not always be the case. The common misconception is that doing so could result in better SEO rankings. This does not prevent the rest of the page from being crawled.In search engine optimization, the typical goal is to get as many pages in your website indexed and crawled by search engines like Google. Non-essential resources, such as the scripts that load the HubSpot sprocket menu, may prompt blocked resources errors.Links for RSS feeds and blog listing pages expire when new blog posts are published, which can generate blocked resources errors.Auditing a root domain, rather than the subdomain connected to HubSpot, is causing a timeout error.A noindex meta tag in the head HTML of your pages is preventing them from being indexed or crawled.The inclusion of your pages in the robots.txt file is preventing them from being indexed or crawled.If you have attempted to crawl your HubSpot pages using an external SEO tool such as Moz or Semrush, you may find that you are unable to crawl your pages successfully. Reach out to your site administrator and request that they add our crawler's user agent, "HubSpot Crawler," to the allow list as an exemption.Īn external SEO tool crawling a HubSpot page.Learn more about resolving DNS errors in Google's documentation. Verify that the page being crawled is currently live.
Verify that the URL has been entered correctly.The crawler isn't able to scan this URL : if HubSpot's crawlers can't crawl a specific URL, try the following troubleshooting steps:.Learn more about working with a robots.txt file here. Robots.txt file couldn't be retrieved: if HubSpot's crawlers can't access your site's robots.txt file, verify that the robots.txt file is accessible and in the top-level directory of your site.Scan blocked by robots.txt file: if your external page is excluded from indexing by your robots.txt file, add our crawler’s user agent “HubSpot Crawler” as an exemption.If you have attempted to crawl external pages using HubSpot's SEO tools or are importing external content to HubSpot, you may encounter one of these errors: HubSpot's SEO tools crawling an external page Crawl of blocked by robots.txt - a robots.txt file is blocking the content from being indexed.
You can view SEO recommendations on the Optimization tab of a page or post's performance details. HubSpot's SEO tools crawling a HubSpot page The steps for resolving a crawling error depend on the error and where the page is hosted. This can happen with the crawlers in HubSpot's SEO and import tools, as well as external crawlers like Semrush.
If an SEO crawler can't index a page, it will return a crawling error.
0 kommentar(er)
