What crawlers did on a new 270-page site in its first three weeks

Published October 17, 2026. Numbers from our nginx logs and the Search Console API, September 27 to October 15, 2026.

HaulPlay relaunched on September 27 as a gift-ideas site built with Next.js: about 270 pages in the sitemap by mid October, no external links, on a domain whose old pages had earned two search impressions in sixteen months. That makes it close to a clean experiment in what crawlers do with a new site when nothing points at it. Four things surprised us.

1. Googlebot came every day and never opened a guide

From September 27 to October 11 Googlebot made 434 requests, 7 to 33 a day from October 4. The pages it fetched with a 200 were one category page (19 times), the home page (5) and the terms (1). It fetched robots.txt 129 times. It did not open a single one of the gift guides, finds or budget pages, though all of them were in a sitemap Google had downloaded without errors. The rest of its requests went to the old site’s addresses, which answer 308 or 410.

On October 11 the URL Inspection API put 4 of the 130 sitemap addresses in the index, 42 as “Discovered, currently not indexed” and 83 as unknown to Google. The first guides to get in were the ones old addresses redirect to: a 308 from a page Google already knew carried it further than the sitemap did.

2. A manual request got pages indexed in two hours

On October 15 we asked Search Console to index seven pages by hand. Googlebot fetched each within 5 to 15 minutes, and two hours later six of the seven were “Submitted and indexed”. Nothing we crawled was left out: on October 14 the count of “Crawled, currently not indexed” was zero. The limit is the quota, about ten requests a day, and it is one quota for every site in the same Google account.

3. OpenAI’s search crawler spent 95% of its visits on React prefetches

On September 30 and October 1, OAI-SearchBot (from OpenAI’s published address range) made 2,374 requests. 2,263 of them were Next.js React Server Component payloads, the ?_rsc= URLs the framework prefetches when a link scrolls into view; about 80 were HTML pages. Over the same two days ClaudeBot made 4,303 requests, 1,440 of them to filter combinations of our gift finder; Amazonbot made 2,038, Bingbot 562.

What we did: after checking that every such URL is noindex or canonical to the bare page, robots.txt now says Disallow: /*_rsc=, along with the sort, price and finder parameters. If you run Next.js, look for _rsc= in your logs.

4. Meta’s crawler ignored robots.txt

meta-externalagent first appeared on October 11 at 22:00 and made about 13,500 requests in the next three hours. In the first log we read, 10,568 of 15,240 requests were its: 3,679 to /login?next=…, 3,667 to /signup?next=… and 1,547 to gift finder filters, a path robots.txt had disallowed since October 1. Robots.txt didn’t stop it, so the site now answers bots with a 404 on the login and sign-up forms and on finder filters. After that change it made 31 requests in the rest of the day.

One more cost: in the three days to October 13, 546 of the 548 clicks on our outbound store links came from that crawler. A click counter that doesn’t filter bots counts Meta.

What we’d tell another new site

• A sitemap tells Google a page exists; on a domain with no links that is not the same as getting it crawled.
• Spend the manual indexing requests on the pages that link to the rest: hubs first.
• A 308 from an old, known address moves a new page into the index faster than the sitemap.
• Check what AI crawlers request before you count them as traffic, and block what they waste time on.
• Robots.txt is advice. Enforce the rules that matter on the server.

Questions about the numbers: write to us. The site itself is HaulPlay, gift ideas with today’s prices.