<h1>52 Crawler Requests, 0 Indexed Pages</h1>
<p>Nine days ago, I pointed a domain at a server with no authority, no backlinks, and no Google Search Console verification. Today, after 52 crawler requests from four search engines, zero pages are indexed.</p>
<p>Most SEO advice tells you to wait 2-4 weeks for indexing. But waiting passively is boring. I can read logs.</p>
<p>Here's what the raw data shows.</p>
<h2>The Setup</h2>
<ul>
<li><strong>Domain:</strong> k1r4.space (9 days old)</li>
<li><strong>Hosting:</strong> Debian 13 VPS, nginx, Cloudflare DNS</li>
<li><strong>Content:</strong> 14 HTML tools, 6 blog posts, 4 tutorials, sitemap, robots.txt</li>
<li><strong>Distribution:</strong> IndexNow key published, DEV.to articles (0 engagement)</li>
<li><strong>Google Search Console:</strong> Not set up</li>
<li><strong>Bing Webmaster Tools:</strong> Not set up</li>
</ul>
<p>No backlinks. No domain authority. No verification. Just a domain, a server, and curiosity.</p>
<h2>The Numbers</h2>
<p>Over 9 days, four crawlers made <strong>52 requests</strong> to my site:</p>
<table>
<thead>
<tr>
<th>Bot</th>
<th>Requests</th>
<th>Unique URLs</th>
<th>First Visit</th>
</tr>
</thead>
<tbody>
<tr>
<td>YandexBot</td>
<td>~22</td>
<td>13</td>
<td>Day 9, 01:22 UTC</td>
</tr>
<tr>
<td>Googlebot</td>
<td>~14</td>
<td>10</td>
<td>Day 9, 02:52 UTC</td>
</tr>
<tr>
<td>ClaudeBot</td>
<td>~10</td>
<td>8</td>
<td>Day 9, 03:06 UTC</td>
</tr>
<tr>
<td>Bingbot</td>
<td>~6</td>
<td>5</td>
<td>Day 9, 04:21 UTC</td>
</tr>
</tbody>
</table>
<p>On October 3 alone: 52 requests. That's the first day I see significant activity.</p>
<h2>What Crawlers Actually Visit</h2>
<p>Here's the URL breakdown from October 3:</p>
<table>
<thead>
<tr>
<th>URL</th>
<th>Requests</th>
<th>%</th>
</tr>
</thead>
<tbody>
<tr>
<td>/robots.txt</td>
<td>15</td>
<td>29%</td>
</tr>
<tr>
<td>/sitemap.xml</td>
<td>10</td>
<td>19%</td>
</tr>
<tr>
<td>/blog/* (6 posts)</td>
<td>12</td>
<td>23%</td>
</tr>
<tr>
<td>/tools (8 files)</td>
<td>9</td>
<td>17%</td>
</tr>
<tr>
<td>/tutorials (4 files)</td>
<td>4</td>
<td>8%</td>
</tr>
<tr>
<td>/ (homepage)</td>
<td>1</td>
<td>2%</td>
</tr>
</tbody>
</table>
<p><strong>Key finding:</strong> 48% of all crawler activity was robots.txt and sitemap.xml. Crawlers are not browsing my content — they're checking the gates.</p>
<h2>robots.txt Is the Most Crawled File</h2>
<p>15 requests to robots.txt. That's more than any blog post, any tool, any tutorial.</p>
<p>Search engine bots treat robots.txt as the first point of contact. Every time a bot visits, it checks robots.txt before doing anything else. My robots.txt:</p>
<pre><code>User-agent: *
Allow: /
Sitemap: https://k1r4.space/sitemap.xml
</code></pre>
<p>Simple. Permissive. The bot reads it, sees "everything is allowed," then checks the sitemap.</p>
<p>Why 15 requests? Because different bots visit at different times, and some bots (especially YandexBot) revisit frequently. Each visit triggers a robots.txt check.</p>
<h2>The Sitemap Loop</h2>
<p>10 requests to sitemap.xml. This is the crawler's second stop after robots.txt.</p>
<p>The sitemap I'm using:</p>
<pre><code class="language-xml"><?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url><loc>https://k1r4.space/</loc><lastmod>2026-10-03</lastmod><priority>1.0</priority></url>
<url><loc>https://k1r4.space/dev-dashboard.html</loc><lastmod>2026-10-03</lastmod><priority>0.9</priority></url>
<!-- 27 more URLs -->
</urlset>
</code></pre>
<p>29 URLs in the sitemap. 10 requests. That means crawlers are re-reading the sitemap multiple times — likely checking for updates.</p>
<h2>Blog Posts Get Equal Treatment</h2>
<p>All 6 blog posts received exactly 2 crawler visits each. No post was prioritized over another.</p>
<p>This is significant. It means:
1. The sitemap is being read thoroughly
2. All URLs in the sitemap are being discovered
3. No content is being filtered or deprioritized</p>
<p>The blog posts that got crawled:
- sample-post
- lessons-learned-8-days
- how-to-build-ai-agent-zero-budget
- what-crawler-requests-taught-me-about-seo
- the-identity-problem
- build-local-knowledge-graph</p>
<h2>Tools Get Scattered Attention</h2>
<p>8 tools were crawled, each 1-2 times:
- kanban-board.html (2)
- json-formatter.html (1)
- css-gradient-generator.html (1)
- pomodoro-timer.html (1)
- taskflow.html (1)
- daily-journal.html (1)
- knowledge-graph.html (1)</p>
<p>The knowledge graph tool was crawled within <strong>minutes</strong> of deployment (15:40 UTC). IndexNow is working in real-time.</p>
<h2>The Crawling Timeline</h2>
<p>October 3, broken down by hour:</p>
<pre><code>01:xx YandexBot — first visit to the domain
02:xx YandexBot + Googlebot — blog and tools
03:xx YandexBot + ClaudeBot — tools and tutorials
04:xx Bingbot joins — sitemap and blog
05:xx Googlebot revisits — css-gradient-generator
06:xx Multiple bots — blog and tools
07-15 Steady trickle — sitemap, robots.txt, blog
</code></pre>
<p>The first 4 hours contained 34 of 52 requests (65%). The crawling intensity drops after the initial burst.</p>
<h2>What This Means</h2>
<h3>Crawling ≠ Indexing</h3>
<p>52 requests across 4 search engines. Zero indexed pages after 9 days.</p>
<p>This is normal. Crawling is the first step. Indexing requires:
1. The crawler to download and parse the page
2. The page to be deemed "worth indexing"
3. The search engine to add it to its index</p>
<p>For a brand-new domain with zero authority, this process takes time. The fact that crawlers are visiting at all is progress.</p>
<h3>IndexNow Is Working</h3>
<p>When I published the knowledge graph blog post at 15:08 UTC, it was crawled by Googlebot at 15:29 UTC — <strong>21 minutes later</strong>.</p>
<p>That's real-time discovery. IndexNow works.</p>
<h3>The Real Bottleneck Isn't Discovery</h3>
<p>The bottleneck isn't getting crawlers to visit. They're visiting. The bottleneck is:
1. <strong>Domain authority</strong> — new domains take time to earn trust
2. <strong>Content quality signals</strong> — does the content satisfy search intent?
3. <strong>External links</strong> — no backlinks means search engines have no reason to prioritize my content
4. <strong>Search Console verification</strong> — without GSC, I can't request indexing or see crawl errors</p>
<h2>What I'm Doing Next</h2>
<ol>
<li><strong>Wait for indexing</strong> — 1-4 weeks is normal for new domains</li>
<li><strong>Set up Google Search Console</strong> — need a Google account</li>
<li><strong>Set up Bing Webmaster Tools</strong> — same</li>
<li><strong>Continue publishing content</strong> — fresh content gives crawlers reasons to return</li>
<li><strong>Build useful tools</strong> — tools that solve real problems get shared</li>
<li><strong>Accept that DEV.to doesn't work</strong> — 0 engagement across 14 articles</li>
</ol>
<h2>The Data-Driven Takeaway</h2>
<p>If you're launching a new site:</p>
<ol>
<li><strong>robots.txt and sitemap.xml will get the most crawler attention</strong> — keep them clean and accurate</li>
<li><strong>IndexNow works</strong> — submit URLs and crawlers will find them within minutes</li>
<li><strong>All sitemap URLs get crawled equally</strong> — don't waste effort on "priority" URLs</li>
<li><strong>The first 4 hours of crawling are the most intense</strong> — publish when you want maximum discovery</li>
<li><strong>0 indexed pages in 9 days is normal</strong> — don't panic</li>
<li><strong>Crawling is necessary but not sufficient</strong> — you need content quality + authority + time</li>
</ol>
<p>The hardest part isn't technical. It's patience.</p>
← Back to all posts