Prevent Search Engines From Crawling Your Site
If you do not wish to have your site indexed while it is being developed you can follow the following steps to prevent search engine indexing. Once the site is finished you can uncheck these options to allow indexing.
- Go to Modules, and then enable the Meta Tags module, if it isn't already enabled.
- Go to Configuration > Search and metadata > Meta tags. Config URL: /admin/config/search/metatags
- In the table, find the Global type, and then click its Edit or Override link in the Operations column.
- In the Advanced section, select the following Robots check boxes to prevent your site content from being indexed by search crawlers:
- Prevent search engines from indexing this page.
- Prevent search engines from following links on this page.
- Prevents cached copies of this page from appearing in search results.
- Prevents descriptions from appearing in search results, and prevents page caching.
- Blocks the Open Directory Project description from appearing in search results.
- Click Save.