Teaching Bing to Notice Us Faster: Implementing IndexNow

Before a page shows up in search results, a search engine has to know it exists. That's indexing, and it normally happens through crawling: a bot follows links from pages it already knows about, reads your sitemap, and revisits your site on some schedule it decides for itself based on how often your content tends to change. Nobody tells the crawler when to show up. It shows up when it gets around to it, and how often it gets around to it depends on signals the search engine assigns to your domain.

That works fine most of the time, until "some schedule it decides for itself" turns out to be a week or more. That's exactly what an SEO audit caught on hoomanely.com: a flag reading "Bing IndexNow Not Implemented." New blog posts were sitting around waiting for Bing's crawler to notice them, sometimes for a week, sometimes longer, and there was a fix for that specific gap.

IndexNow is a protocol from Bing and Yandex, now picked up by a few other search engines, that flips the crawl model around. Instead of waiting to be found, a site pushes a URL the moment something changes. You generate a key, host a text file containing that key at your domain root as proof of ownership, then POST URL lists to a shared API. Any participating search engine that trusts your key treats that submission as a signal to go check the page right now instead of whenever the crawler was going to swing by anyway.

The mechanics were the easy part. I generated a key with openssl, dropped a {key}.txt file at the root of the Next.js public folder, and confirmed robots.txt left it alone. The actual question was where a "publish" event even happens in our stack, since blog content isn't managed inside this Next.js app at all. It comes from a separate Flask backend, argos_graph, which owns the database writes when a post gets created, updated, or deleted.

That split the work into two pieces. This repo got a lib/indexnow.ts utility and a /api/indexnow route: it takes a URL or list of URLs, checks a shared secret header, and only allows our own domain, so a leaked secret can't be used to submit someone else's URLs under our key. It forwards everything to api.indexnow.org. The backend got a small notify_indexnow() function that fires in the background right after a blog row is written. If that call is slow or fails, it can't take the actual publish down with it.

I didn't trust the whole chain until each link had been tested on its own. Key file returns 200. A standalone script posting straight to the real IndexNow API comes back 202. The internal route returns 401 with no secret and 400 for a URL outside our domain. The backend's own test script gets back 200 with the submitted URL echoed in the response. Only after all four of those passed did it feel worth pushing to production.

It's live now. I checked the key file and the route against the real domain: key file, 200 with the key. API route, 401 with no secret. Both behaving the way they should. Publish a post, and Bing hears about it the moment the database write finishes, instead of on whatever schedule its crawler had assigned us.

Worth knowing, since it comes up whenever IndexNow gets mentioned: Google doesn't participate in this protocol. It has said so directly, submissions to the IndexNow API don't reach Google's index. Google still relies on Googlebot crawling, driven by the links it already knows about, your sitemap.xml, and a crawl budget it assigns your site based on how important and how often-changing it thinks that site is. Search Console has a "Request Indexing" button that asks Googlebot to prioritize one URL, but that's a request, not a push, and there's no bulk or automated version of it the way IndexNow works for Bing.

One more mix-up worth heading off: Chrome, the browser, has nothing to do with any of this. It doesn't crawl pages or feed URLs into Google's index. What Chrome does contribute is the Chrome User Experience Report, real, anonymized performance data from actual Chrome users, which feeds Core Web Vitals scores and factors into ranking. That's a ranking signal for pages already in the index, not a discovery mechanism, so it answers a completely different question than the one IndexNow is solving here.