n order to add news articles to Google News, our crawler must be able to access the content on your site. Since crawlers can't currently fill out registration forms, nor do they support cookies, we need to be able to circumvent those pages in order to successfully crawl those sites.
The easiest way to do this is to configure your webservers to not serve the registration when our crawler visits your pages (when the User-Agent is "Googlebot-News"). It is equally important that your robots.txt file allows access by Googlebot-News. You may wish to allow other Google User-Agents to access your content as well, such as Google Web Preview (you can find the full User-Agent string for Google Web Preview in the Google Web Preview FAQ).
JIMPER 2011 |
KERALA CEE 2011 |
MHT CET 2011 |
MHT CET |
NLU Orissa
