An address ending in /robots.txt does not necessarily deliver a usable control file. An authentication redirect, HTML error document or temporary server failure can alter the response. Record what is actually delivered before inspecting individual Disallow rules.
Compare 200, 404 and 503
A 200 containing a text file differs from a 404 reporting that the file is missing. A 503 instead signals temporary unavailability. Google documents separate robots.txt handling for these situations; a missing file is not equivalent to a server failure.
curl -sS -D - https://example.invalid/robots.txt
Replace the placeholder with your hostname. Inspect the status, any redirect destination and the beginning of the response body. If it contains an HTML sign-in page, correct the delivery route. Editing a text file that the crawler never receives will not resolve that problem.
Test the canonical hostname using the actual protocol. A working robots.txt on a different subdomain does not confirm your shop’s configuration.
Recheck after an incident
Record whether the failure happens only during a deployment or persists afterward. Inspect affected content pages separately: their availability cannot be established from the robots.txt status. After the change, repeat the same hostname and content-type checks. The indexability checker assists with individual URLs; use Search Console for Google’s recorded state. Live delivery and a saved crawl report can refer to different times.