What is Robots.txt ?
Robot.txt is the file in the [ wwwroot ]of a server to define the Bots the Functionality on the website.
There are many Bots on the internet like the most famous are Google Search Engine BOT aka Google Spider , Yahoo Search Engine Bot and many others.
What robots.txt do is that it gives orders to the Bot on how to spider on the website
what’s the use of Robots.txt file?
It is used by webmasters to manage the upcoming bots on their website and to hide the directories in the website and it also describes where the Bot and spider should not go.
Analyzing Robots.txt
Where is robots.txt Located ?
= its in the [ WWWROOT ] that is in the main Directory.
Lets Get More knowledge about robot.txt:
type on address bar, http://www.anydomain.com/robots.txt
Now lets Analyze this Robots.txt
First Line :-
User-agent: Mediapartners-Google
This means that the above commands are given for the Google Search Engine Bot i.e. Google Spider.
Second Line :-
Disallow:
This mean that nothing is disallowed to the Google Bot, Remember these Orders are given to the Google bot only not other bots.
Third Line :-
User-agent: *
This means that now all the bots coming to the site will follow these rules.note that previous rules were for only Google Bot.
Fourth Line :-
Disallow: /search
This means that all the bots will not spider the files under the directory.
Did you get it? If you have any more confusion, please share with us.
Incoming search terms:
- app mig33 robot (2)
- what does user agent * mean (2)
- whats the use of robots txt (2)
- swifty migv4 (2)
- create free mig33 bot (2)
- mig33 robot (2)
- swifty migv4 2 (2)
- what does it means robots txt file (1)
- what does Disallow: in robot mean (1)
- what does Disallow: / mean in robot (1)
- what does disallow media partners mean (1)
- what does disallow mean in robots txt (1)
- what does disallow /*? robots mean (1)
- What does a Robot mean? pdf (1)
- what does * txt mean? (1)
- what does * mean in robots txt (1)
- what does mean by spiders in robots txt file (1)
- what does mean of robot txt (1)
- what does pc robots txt file (1)
- where is robots txt (1)
- what`s means robot txt (1)
- whats this mean User-agent: Mediapartners-Google Disallow: User-agent: * Disallow: /search (1)
- whats means by robotics (1)
- whats a robot txt file (1)
- what-does-robots-txt-mean-tips-to-make-robots-txt-file (1)
- what is robot txt mean (1)
- what does user-agent disallow mean (1)
- what does User agent: * mean (1)
- what does user agent mean in robots txt (1)
- what does robots txt mean (1)
- What does robots text mean (1)
- trick to make robots (1)
- tips and tricks for making robot (1)
- tips and trick to make line robots (1)
- Intext:User-agent: * inurl:robots txt intext:Mediapartners-Google (1)