It is a request, not a protection: honest robots respect it, malicious ones don't. Don't hide addresses that are meant to be secret in it, because the file is public. It is good to point to the sitemap in it and not to block files needed to display the page.
robots.txt
A file in which you tell search engine robots what they may and may not access.
Ready to start?
Let’s start with a conversation
Describe what you want to build. The reply contains the scope, the price and the timeline. The inquiry costs nothing and commits you to nothing.