What is meant by Robots.txt? custom robots.txt for blogger
Robots.txt is a text file on server where you can write some thing about directress and webpages or links should not be included in search engines. By this you can get some idea that you can restrict search engine to crawl directories and web pages or links of your website or blog.
custom robots.txt for blogger and robots.txt in Blogger.
- custom robots.txt for blogger
A good news for blogger sites that the google is continuously upadating blogger plat from. And now Google included robots.txt part into blogger. This means that a person who runs a blogger blog can also restrict search engine crawls.
Sitemap is the file that is located on the server where all the blog posts permalinks are present. Blogger is reading site map through feeds.
DIFFERENT TYPES custom robots.txt for blogger: robots.txt in Blogger | robots.txt for blogger
Robots.txt for blogger type 1
This type of robots.txt allows search engine to crawl most recent 25 posts. If you really want to crawl most recent 25 post you can prefer robots.txt 1. Google ad-sense bot is allowed to crawl the whole website for best ad-sense performance.
Robots.txt for blogger Type 2
If you want search engine to crawl most recent 500 posts then you can do follow the Robots.txt Type 2
This type of robots.txt allows search engine to crawl most recent 25 posts. If you really want to crawl most recent 25 post you can prefer robots.txt 1. Google ad-sense bot is allowed to crawl the whole website for best ad-sense performance.
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Disallow: /b
Allow: /
Sitemap: http://mybloggerground.blogspot.com/feeds/posts/default?orderby=updated
Robots.txt for blogger Type 2
If you want search engine to crawl most recent 500 posts then you can do follow the Robots.txt Type 2
If you posts more then 500 then you have to add another site map that is given blow.
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Disallow: /b
Allow: /
Sitemap: http://mybloggerground.blogspot.in/atom.xml?redirect=false&start-index=1&max-results=500
Robots.txt for blogger type 3:
If you more than 500 post please use following robots.txt
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Disallow: /b
Allow: /
Sitemap: http://mybloggerground.blogspot.in/atom.xml?redirect=false&start-index=1&max-results=500
Sitemap: http://mybloggerground.blogspot.in/atom.xml?redirect=false&start-index=501&max-results=1000
If you have organized post labels in a well format and good experience of search engine optimization (SEO) then you can remove following line -
Disallow: /search
If you don't want to submit any blogger post or page to search engines then you can add them like that:
For Post add a line like that -
Disallow: /yyyy/mm/post-name.html
For Page add a line like that -
For this please follow these steps carefully. robots.txt in Blogger.
Disallow: /p/page-name.html
For this please follow these steps carefully. robots.txt in Blogger.
Dashboard ›› Blog's Settings ›› Search Preferences ›› Crawlers and indexing ›› Custom robots.txt ›› Edit ›› Yes
To get better understanding of this you can take reference of image given below:
- robots.txt in Blogger.
Customize accordingly...
No comments:
Post a Comment