CRAWL INVENTORY
How ORBIT is set up to be crawled
Internal inventory of public URLs ORBIT advertises to crawlers, plus when our own archiver last wrote a timestamp we can read. These are not Google Search Console numbers. They are not rankings, impressions, clicks, index coverage, or a claim about what any search engine has indexed.
Indexable URL estimate
The current XML sitemap lists 2116 public URLs. 434 of them carry a lastmod we actually know (daily briefs, KEV dates, historic dissector cases, recent incidents). The rest omit lastmod rather than invent a date. Generated .
“Indexable” here means: the URL is public, is listed in the sitemap, and is not noindexed by ORBIT. Search engines decide what they crawl and keep. This estimate will move as KEV rows, briefs and recent incidents change.
Sitemaps
A sitemap index lists each section, plus the all-in-one file that older crawlers already know:
/sitemap-index.xml— the index/sitemap.xml— every URL in one urlset (compatibility file; same inventory as the sections combined)/sitemaps/core.xml— 19 URLs/sitemaps/status.xml— 1010 URLs/sitemaps/incidents.xml— 606 URLs/sitemaps/tools.xml— 32 URLs/sitemaps/labs.xml— 27 URLs/sitemaps/os.xml— 2 URLs/sitemaps/community.xml— 11 URLs/sitemaps/arena.xml— 65 URLs/sitemaps/legal.xml— 13 URLs/sitemaps/kev.xml— 302 URLs/sitemaps/news.xml— 29 URLs- Human sitemap — the same public surface, written for people
/robots.txt— allows public pages; points at the index
| Section | What's in it | URLs |
|---|---|---|
| core | Core hubs (home, weather, news desk, citation, inventory) | 19 |
| status | Status radar, service pages, pulse and signals | 1010 |
| incidents | Outage dossiers, dissector, war room, recent incident pages | 606 |
| tools | Public tools (Arena is listed separately) | 32 |
| labs | ORBIT Labs stations and missions | 27 |
| os | CHYP OS | 2 |
| community | CHYP Community boards (index pages only; threads and profiles are not auto-listed) | 11 |
| arena | Cloudy_Day Arena (public rooms and challenges; personal /arena/me/ is not listed) | 65 |
| legal | Legal, about and policy pages | 13 |
| kev | CISA KEV and CVE trackers | 302 |
| news | News, daily briefs, topics and the feeds index | 29 |
| Total | Unique public URLs in the inventory | 2116 |
Last archiver write we can see
When the archiver stores a JSON blob in the meta table, this page shows the timestamp field if one is present (for example a KEV catalog fetchedAt). That is a record of our collector, not of Googlebot.
| Archiver key | Timestamp found in the value |
|---|---|
arena_live_sync | no timestamp field in this row |
awin_sync | no timestamp field in this row |
events_chat | no timestamp field in this row |
events_last | |
feed_last | no timestamp field in this row |
kev_meta | |
kev_version | no timestamp field in this row |
listen_last | |
markets_lease | no timestamp field in this row |
news_last | |
ofh_cycle | no timestamp field in this row |
ofh_last | |
sig_lease | no timestamp field in this row |
takes_archive_last | |
takes_post | no timestamp field in this row |
v5_sweep | no timestamp field in this row |
watch_last |
What we deliberately do not hide
- No Bot Fight Mode, no AI-bot blocking, no zone rate limits aimed at crawlers, and no
robots.txtDisallows that hide public pages. - Language-model briefings live at /llms.txt, /llms-full.txt, /.well-known/llms.txt and /.well-known/ai.txt.
- /humans.txt is a short, truthful credits file.
- JSON and Markdown exports stay cacheable (they are not a crawl trap) and are not listed in the sitemap. Replay and as-of URLs are canonicalized; we do not list every time-sliced variant.
- Search result pages (
/search/?q=) stay crawlable so the site search box can be verified, and they are noindexed so they do not compete with the real pages.