What is Robots.txt file?
Robots.txt file inform to the search engines to allow website for index. Once our website is ready to live we need to enable search engines for online search visibility.
For WordPress Websites:
if you are using WordPress website here are WordPress dashboard settings – go to settings next click on reading, enable search engine visibility. Check the below screenshot.
For HTML and other Websites:
Open notepad and update the following line User-agent: *
Next save your notepad with the following name robots. Now Go to your website control panel and open file manager and click on public_html then find upload button on top then upload notepad (robots.txt file). after uploading your robots.txt file, check your robtos.txt link in the browser it will show like this http://www.digitalmediarole.com/robots.txt.