Lisda Ikhwantini Digital Marketing Consultation

I want to do SEO audit for a website using one of free SEO tools, Seoptimier but unfortunately the website have server issue that Seoptimier can’t do the SEO audit. This is happen also when I check using Pagespeed insight. When you use Seoptimier, you can see robots txt file url and sitemap url, if any and to make sure the robots txt and sitemap is correct, you just copy-paste the url and go to them.

How to check Robots txt file?

Robots txt url usually in this format: http://example.com/robots.txt or http://www.example.com/robots.txt. Change example dot com with your website url. There is a free robots txt tester tool from Google, you can check your robots txt there, if it’s already good to go or not.

1. Make sure you have a property of the website on search console.
2. Go to google.com/webmasters/tools/robots-testing-tool.
3. You will see you robots txt file there. Google will tell you if there is any error and warning on the robots txt.
4. You can test any sub folder to test if it’s blocked or not. My url is allowed.

Do we need robots txt file for small website or a blog?

Google said no, but I think it depends. Google said if Google bot coming to a website without a robots txt file, Google bot still can crawling and indexing the website, as usual. But I suggest, if you want ‘some rules’ for a search engine bot when crawling and indexing, you can create a robots txt file, such as if you don’t want some folders to be crawl or indexed or you want to completely block search engine bot to crawl and index your website.

How to create robots txt? Do we need robots txt generator?

It depends. Google already give a sample here developers.google.com/search/docs/advanced/robots/create-robots-txt but for a beginner SEO like me, it’s still too general. Google said we can read the full syntax for robots txt, but still I search for a robots txt generator.

There are a lot SEO tools to generate robots txt file, and come out with txt file. You can open it with notepad or any word apps. I try this en.ryte.com/free-tools/robots-txt-generator/ and it’s quite simple to use.

Now, how do you think? Please comment below. Do you have more question about robots txt file checker? You can see more article from Neil Patel, I like the way he educate his reader on his blog, way more human for this topic. You can see interesting article and tips here, SEO tips here, and Google Analytics articles here.