Skip to content
Uzman Tescil
TR

Robots.txt Checker

Read robots.txt, list sitemaps and review critical crawl rules.

Example: uzmantescil.com.tr

robots.txt bulundu HTTP 200

google.com

Sitemap
1
Disallow
179
Allow
67

Bildirilen sitemap’ler

Dosya içeriği

User-agent: *
User-agent: Yandex
Disallow: /search
Allow: /search/about
Allow: /search/howsearchworks
Disallow: /sdch
Disallow: /groups
Disallow: /index.html?
Disallow: /?
Disallow: /goto?
Allow: /?hl=
Disallow: /?hl=*&
Allow: /?hl=*&gws_rd=ssl$
Disallow: /?hl=*&*&gws_rd=ssl
Allow: /?gws_rd=ssl$
Allow: /?pt1=true$
Disallow: /imgres
Disallow: /u/
Disallow: /setprefs
Disallow: /m?
Disallow: /m/
Allow:    /m/finance
Disallow: /wml?
Disallow: /wml/?
Disallow: /wml/search?
Disallow: /xhtml?
Disallow: /xhtml/?
Disallow: /xhtml/search?
Disallow: /xml?
Disallow: /imode?
Disallow: /imode/?
Disallow: /imode/search?
Disallow: /jsky?
Disallow: /jsky/?
Disallow: /jsky/search?
Disallow: /pda?
Disallow: /pda/?
Disallow: /pda/search?
Disallow: /local?
Disallow: /local_url
Disallow: /products?
Disallow: /product_
Disallow: /products_
Disallow: /products;
Disallow: /print
Disallow: /books/
Disallow: /bkshp?*dq=
Disallow: /bkshp?*q=
Disallow: /books?*dq=
Disallow: /books?*q=
Disallow: /books?*qtid=
Disallow: /books?*output=
Disallow: /books?*pg=
Disallow: /books?*jtp=
Disallow: /books?*jscmd=
Disallow: /books?*buy=
Disallow: /books?*zoom=
Allow: /books/about
Allow: /books?*zoom=1
Allow: /books?*zoom=5
Allow: /books/content?*zoom=1
Allow: /books/content?*zoom=5
Disallow: /patents?
Disallow: /patents/download/
Disallow: /patents/pdf/
Disallow: /patents/related/
Disallow: /scholar
Disallow: /citations?
Allow: /citations?user=
Allow: /citations?view_op=new_profile
Allow: /citations?view_op=top_venues
Allow: /scholar_share
Disallow: /s?
Disallow: /maps?
Allow: /maps?daddr=
Allow: /maps?entry=wc
Allow: /maps?f=
Allow: /maps?hl=
Allow: /maps?q=
Allow: /maps?saddr=
Allow: /maps?sid=
Allow: /maps?*output=classic
Allow: /maps?*file=
Disallow: /mapslt?
Disallow: /maphp?
Disallow: /maps/
Allow: /maps/$
Allow: /maps/@
Allow: /maps/?daddr=
Allow: /maps/?entry=wc
Allow: /maps/?f=
Allow: /maps/?hl=
Allow: /maps/?q=
Allow: /maps/?saddr=
Allow: /maps/?sid=
Allow: /maps/api/staticmap?
Allow: /maps/search/
Allow: /maps/sitemap.xml
Allow: /maps/sitemaps/
Allow: /maps/dir/
Allow: /maps/d/
Allow: /maps/reserve
Allow: /maps/about
Allow: /maps/contrib/
Allow: /maps/match
Allow: /maps/ms?
Allow: /maps/place/
Allow: /maps/_/
Allow: /search?*tbm=map
Allow: /maps/vt?
Allow: /maps/preview
Disallow: /maps/api/js/
Allow: /maps/api/js
Disallow: /mld?
Disallow: /staticmap?
Disallow: /help/maps/streetview/partners/welcome/
Disallow: /help/maps/indoormaps/partners/
Disallow: /lochp?
Disallow: /ie?
Disallow: /uds/
Disallow: /transit?
Disallow: /trends?
Disallow: /trends/music?
Disallow: /trends/hottrends?
Disallow: /trends/viz?
Disallow: /trends/embed.js?
Disallow: /trends/fetchComponent?
Disallow: /trends/beta
Disallow: /trends/topics
Disallow: /trends/explore?
Disallow: /trends/api
Disallow: /musica
Disallow: /musicl
Disallow: /musics
Disallow: /urchin_test/
Disallow: /movies?
Disallow: /wapsearch?
Disallow: /reviews/search?
Disallow: /cbk
Disallow: /profiles/me
Disallow: /s2/profiles/me
Allow: /s2/profiles
Allow: /s2/oz
Allow: /s2/photos
Allow: /s2/search/social
Allow: /s2/static
Disallow: /s2
Disallow: /transconsole/portal/
Disallow: /aclk
Disallow: /tbproxy/
Disallow: /support/forum/search?
Disallow: /reviews/polls/
Disallow: /hosted/images/
Disallow: /accounts/ClientLogin
Disallow: /accounts/ClientAuth
Disallow: /accounts/o8
Allow: /accounts/o8/id
Disallow: /quality_form?
Disallow: /labs/popgadget/search
Disallow: /compressiontest/
Disallow: /analytics/feeds/
Disallow: /analytics/partners/comments/
Disallow: /analytics/portal/
Disallow: /analytics/uploads/
Allow: /alerts/manage
Allow: /alerts/remove
Disallow: /alerts/
Allow: /alerts/$
Disallow: /phone/compare/?
Disallow: /travel/clk
Disallow: /travel/lodging/clk
Disallow: /travel/lodging/imp
Disallow: /travel/entity
Disallow: /travel/search
Disallow: /travel/flights/booking
Disallow: /travel/flights/s/
Disallow: /travel/flights/search
Disallow: /travel/hotels/stories
Disallow: /travel/hotels/*/stories
Disallow: /travel/story
Disallow: /hotelfinder/rpc
Disallow: /hotels/rpc
Disallow: /evaluation/
Disallow: /forms/perks/
Disallow: /shopping/suppliers/search
Disallow: /edu/cs4hs/
Disallow: /trustedstores/s/
Disallow: /trustedstores/tm2
Disallow: /trustedstores/verify
Disallow: /shopping?
Disallow: /shopping/product/
Disallow: /shopping/seller
Disallow: /shopping/ratings/account/metrics
Disallow: /shopping/ratings/merchant/immersivedetails
Disallow: /shopping/reviewer
Disallow: /shopping/search
Disallow: /shopping/deals
Allow: /shopping?udm=28$
Disallow: /storefront
Disallow: /storepicker
Disallow: /about/careers/applications/candidate-prep
Disallow: /about/careers/applications/connect-with-a-googler
Disallow: /about/careers/applications/jobs/results?page=
Disallow: /about/careers/applications/jobs/results/?page=
Disallow: /about/careers/applications/jobs/results?*&page=
Disallow: /about/careers/applications/jobs/results/?*&page=
Disallow: /landing/signout.html
Disallow: /gallery/
Disallow: /landing/now/ontap/
Allow: /maps/reserve
Allow: /maps/reserve/partners
Disallow: /maps/reserve/api/
Disallow: /maps/reserve/search
Disallow: /maps/reserve/bookings
Disallow: /maps/reserve/settings
Disallow: /maps/reserve/manage
Disallow: /maps/reserve/payment
Disallow: /maps/reserve/receipt
Disallow: /maps/reserve/sellersignup
Disallow: /maps/reserve/feedback
Disallow: /maps/reserve/terms
Disallow: /maps/reserve/m/
Disallow: /maps/reserve/b/
Disallow: /maps/reserve/partner-dashboard
Disallow: /local/cars
Disallow: /local/dealership/
Disallow: /local/dining/
Disallow: /local/place/products/
Disallow: /local/place/reviews/
Disallow: /local/place/rap/
Disallow: /local/tab/
Disallow: /localservices/
Allow: /localservices/coupon/
Disallow: /nonprofits/account/
Disallow: /uviewer
Disallow: /landing/cmsnext-root/

# AdsBot
User-agent: AdsBot-Google
Disallow: /maps/api/js/
Allow: /maps/api/js
Disallow: /maps/api/place/js/
Disallow: /maps/api/staticmap
Disallow: /maps/api/streetview

# New user agent groups must also have a user agent reference in the global (*)
# group. See "Order of precedence" section in
# https://goo.gle/rep#order-of-precedence-for-user-agents
User-agent: Yandex
Disallow: /about/careers/applications/jobs/results
Disallow: /about/careers/applications-a/jobs/results

# Crawlers of certain social media sites are allowed to access page markup when
# google.com/imgres* links are shared. To learn more, please contact
# images-robots-allowlist@google.com.
User-agent: facebookexternalhit
User-agent: Twitterbot
Allow: /imgres
Allow: /search
Disallow: /groups
Disallow: /hosted/images/
Disallow: /m/

Sitemap: https://www.google.com/sitemap.xml
All tools Look Up

What the robots.txt check shows

robots.txt tells crawlers which paths they may fetch. The tool downloads the file and shows the HTTP status, Sitemap lines, Allow and Disallow counts, and the raw text. A mistaken Disallow: / removes the whole site from crawling. A missing file means crawlers may fetch every path, which is not automatically an error.

  • Hiding an admin panel in robots is not protection. The panel needs a password. robots.txt advertises the path.
  • A Sitemap line tells crawlers where the map is. Missing it slows discovery. It does not by itself drop rankings.
  • Do not confuse Googlebot with AdsBot. Blocking the ads crawler affects campaigns.
  • After an edit, also test the file in Search Console. This tool reads the live file.

Disallow does not delete an indexed URL

Disallow reduces crawling. Removing an already indexed URL needs noindex or a removal request in Search Console. Writing a line in robots.txt is not the same as deleting a result. If a sensitive URL was crawled before, a copy can remain.

Frequently asked questions

What is robots.txt for?

It tells bots which paths they may crawl. The tool shows the body, sitemap lines and Allow/Disallow counts.

Is a sitemap required?

No. On large sites it speeds discovery. The line can live inside robots.txt.

Is a missing file a problem?

Bots may crawl by default. Use authentication, not robots.txt, for anything private.

Does Disallow lower rankings?

A blocked page is not crawled. Blocking the homepage by mistake removes visibility. A correct Disallow keeps junk parameters out of the crawl.

Why show the raw file?

Comments, typos and extra User-agent groups disappear inside the counts. Read the text itself.

Other website tools

WhatsApp Message us now Call Now 0850 309 1 889