You can now choose to configure the crawler to stay within the sub-page it’s given so it no longer crawls back to the home page. If you point it at a sub-page, it won’t follow links leading back to the home page or a previous page.
You can now choose to configure the crawler to stay within the sub-page it’s given so it no longer crawls back to the home page. If you point it at a sub-page, it won’t follow links leading back to the home page or a previous page.