# Should You Let AI Bots Crawl Your Website? A Plain-English Answer for 2026 > Blocking every AI bot to protect your content can quietly erase you from ChatGPT and Gemini answers. Here is the 2026 decision, in plain owner language, and exactly what to do. *Roald, Founder Fonzy · Aug 22, 2026 · 8 min read* Source: https://www.fonzy.ai/blog/should-i-let-ai-crawl-my-site --- **Short answer:** For almost every small business, yes, let the AI bots that answer live questions read your site. Block them and you disappear from ChatGPT and Gemini when customers ask for a recommendation. Opt out of the training bots only if you have a specific reason to. Picture a neighborhood bakery. The owner reads a headline about "AI stealing your content," panics about the sourdough recipe, and blocks every AI bot she can find. Her site is now protected. It is also gone. A month later, someone in her town opens ChatGPT and types "best bakery near me for fresh sourdough." ChatGPT names three shops. Hers is not one of them, because she told the bot that fetches pages to answer that exact question to stay out. She traded a recipe nobody was going to lift for a customer who was ready to walk in the door. That trade is the whole point of this article. There is one idea to hold onto, and everything else follows from it. ## Two jobs AI bots do, and only one of them is a threat AI bots come to your site for two completely different reasons, and people blur them together. The first job is training. A bot reads huge amounts of the web to help teach a model how language works. Your page becomes one grain of sand on a very large beach. This is the job people mean when they say "AI is stealing my content." The second job is answering a live question. When someone asks ChatGPT or Gemini "who does gluten-free birthday cakes in Portland," a different bot goes out, fetches current pages, and pulls an answer from what it finds right now. This is not training on your recipe. This is a customer standing at the counter, and the bot is deciding whether to point them at you. Most small business owners want to be invisible to the first job and wide open to the second. The mistake is blocking both because they look the same from the outside. They are not the same. One turns your page into training data. The other turns your page into a recommendation. ## What robots.txt is, in one plain sentence Robots.txt is a tiny text file at the root of your website that tells automated bots which ones are allowed in and which are not. That is it. It is a list of doormen instructions, not a lock. Well-behaved bots read it and obey; you name a specific bot and either wave it through or turn it away. You almost never touch this file by hand, and you should not need to. But it is the single place where the block-or-allow decision actually happens, so it helps to know that this one small file is doing the deciding. If you run on a platform like Shopify, Wix, or WordPress, there is usually a setting or a plugin that edits it for you. ## OpenAI runs two different bots, and blocking the wrong one is the costly mistake OpenAI is the clearest example of why the two jobs matter, because it uses two separate bots with two separate purposes. GPTBot is the training bot. It gathers content to help train OpenAI's models. If your worry is your work being used as training material, this is the one you would turn away. OAI-SearchBot is the search bot. It fetches pages so ChatGPT can answer live questions with current information. This is the bot that decides whether your bakery shows up when someone asks ChatGPT for a recommendation right now. Here is the part owners miss. OpenAI's own documentation says each crawler setting is independent: a site can allow OAI-SearchBot so it appears in ChatGPT's search results while disallowing GPTBot to keep its content out of training. You do not have to choose all-in or all-out. You can keep your content out of the training pile and still show up in the answers. And the cost of getting it backwards is spelled out plainly. OpenAI states that sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers." Read that again. Block the search bot and you are not protecting anything valuable. You are removing yourself from the exact moment a customer is asking a machine who to buy from. That is the bakery, gone from "best bakery near me," for nothing. If you want to understand how ChatGPT actually picks which businesses to name once it can read you, we walk through that in [how ChatGPT recommends businesses](/blog/how-chatgpt-recommends-businesses). ## Google-Extended will not touch your Google ranking Google gives you a similar control, and this is where a lot of fear comes from, so it is worth being exact. Google-Extended is a robots.txt token that controls whether your content is used to train Google's Gemini models. It is the training switch, the same category as GPTBot. And Google documents that Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search." So you can opt out of Gemini training and keep every bit of your normal Google Search presence. Your rankings do not move. Your traffic does not drop. The switch that controls AI training is a different switch from the one that controls whether you show up on Google, and Google says so directly. If you also want to be present in Gemini's answers themselves, the work there is about being readable and relevant, which we cover in [optimizing for Google Gemini](/blog/optimize-for-google-gemini), not about blocking a crawler. ## What you actually lose when you block the answer bots The scary headline gets the risk backwards. For a small local business, the real risk in 2026 is not that AI reads your content. It is that AI cannot find you when a customer asks. BrightLocal's 2026 Local Consumer Review Survey reports that AI is quickly becoming a go-to for business recommendations, and that more consumers now trust AI for recommendations than fear it. That shifts the math. Every customer who asks an assistant "who should I call" instead of scrolling Google is a customer you either show up for or you do not. Block the answer bots and you have opted out of that entire line of people at the counter. In the work of getting owner-run shops found in AI answers, the same pattern shows up over and over. Someone reads a headline, adds a blanket block, and months later cannot understand why a competitor keeps getting named by ChatGPT while they never do. The block is silent. Nothing breaks, no error appears, the site still ranks fine on Google, so nobody connects the missing recommendations back to one line in a text file. It is the quietest way to disappear. Practitioners have landed in the same place. On r/TechSEO, a recurring question is literally "AI bots (GPTBot, Perplexity, etc.), block all or allow?" and the emerging consensus is a selective robots.txt: block the training-only crawlers if you feel you must, but keep the retrieval and search crawlers open so you stay visible in AI answers. It is not a fringe take. It is where the people who do this all day have settled. If you are not sure whether ChatGPT already names you, you can test it in a few minutes using [this walkthrough on checking if ChatGPT recommends your business](/blog/check-if-chatgpt-recommends-your-business). ## When blocking an AI bot actually makes sense Allowing the answer bots is the default, not a law. There are real reasons to turn a bot away, and none of them apply to a typical bakery, plumber, or salon. Server strain is the most legitimate one. If a crawler is hitting your site so hard that pages slow down for real visitors, blocking or slowing that bot is a reasonable defense. This is a hosting problem more than a marketing one, and most small sites never feel it. Premium or paywalled content is the other real case. If your business is selling access to the words themselves, a paid course, a members-only research library, a recipe archive people pay for, then keeping those specific pages out of training data can be a deliberate choice. Note the word specific. You can protect the paid archive and still let the answer bots read your homepage, your service pages, and your location, which is what customers actually search on. And every one of these choices is reversible. Blocking a crawler is done by editing that robots.txt file, and you can undo it just as easily. OpenAI notes it can take about 24 hours for a robots.txt update to take effect for its search systems, so this is not instant, but it is not permanent either. You are never locked into a decision you made in a panic after reading one headline. ## So what should you do this week? For the ordinary small business, the honest answer is: probably nothing, and definitely do not block everything. Leave the search and answer bots alone so ChatGPT, Gemini, and the rest can read you and recommend you. If you have a genuine reason to stay out of AI training, opt out of the training bots only, GPTBot and Google-Extended, and leave the answer bots open. If a crawler is hammering your server, deal with that specific bot. Beyond that, the more useful work is not blocking anything. It is making your pages clear enough that when a bot does read you, it can tell a customer exactly what you do and where. A plain, well-structured page is what gets you named, and a simple file that spells out your business for AI can help, which we cover in [the small-business guide to llms.txt](/blog/llms-txt-small-business). If your goal is to actively get picked, start with [getting AI to recommend your business](/blog/get-ai-to-recommend-your-business). ## Frequently asked questions ### Will letting AI bots read my site hurt my Google ranking? No. The bot that controls AI training, Google-Extended, is separate from Google Search. Google documents that Google-Extended does not affect your inclusion in Google Search and is not a ranking signal. Letting AI answer bots read you does not cost you anything on regular Google. ### If I block GPTBot, do I still show up in ChatGPT? You can. GPTBot is the training bot, and OAI-SearchBot is the one that fetches pages to answer live questions. OpenAI says the two settings are independent, so you can block GPTBot to stay out of training and still allow OAI-SearchBot so you appear in ChatGPT's search answers. ### I already blocked everything in a panic. Can I undo it? Yes. Blocking is just an edit to your robots.txt file, and reversing it is the same kind of edit. OpenAI notes it can take about 24 hours for a robots.txt change to take effect for its search systems, so you will not see the change the same minute, but nothing you did is permanent. ### Is AI really sending me customers, or is this hype? It is becoming real. BrightLocal's 2026 survey found AI is quickly turning into a go-to for business recommendations, with more consumers trusting it than fearing it. You do not have to bet your whole marketing on it. You just have to not lock the door when a customer knocks. That bakery lost a walk-in customer to protect a recipe nobody was taking. The fix was one line, not a new strategy. Fonzy keeps the answer bots reading you and your pages clear enough to get named, so when someone asks a machine "who's the best near me," the honest answer is you. ## Sources - [OpenAI, Overview of OpenAI Crawlers: GPTBot trains models, OAI-SearchBot powers ChatGPT search, the settings are independent, opting out of the search bot removes you from ChatGPT search answers, and robots.txt changes are reversible in about 24 hours](https://developers.openai.com/api/docs/bots) - [Google Search Central: Google-Extended controls Gemini training and does not affect Google Search inclusion or ranking](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) - [BrightLocal, Local Consumer Review Survey 2026: AI is becoming a go-to for business recommendations and more consumers trust it than fear it](https://www.brightlocal.com/research/local-consumer-review-survey/) - [r/TechSEO discussion: practitioner consensus favors a selective robots.txt, keeping search and retrieval crawlers open](https://www.reddit.com/r/TechSEO/comments/1ladbhr/ai_bots_gptbot_perplexity_etc_block_all_or_allow/) --- Published by [Fonzy](https://www.fonzy.ai) — expert articles that get you found on Google and AI search.