# composal.com serves two things from one origin: the public marketing site and The # Reflog at /reflog, both of which we want indexed, and the authenticated # product app, which no crawler can see. Without this file every request for # /robots.txt fell through to the app's catch-all and answered 302 -> the # sign-in page, so a crawler asking "may I fetch this site" got an HTML login # form instead of a policy. Social card fetchers are the ones that notice. User-agent: * Allow: / # Signed-out these only ever render the sign-in form, so crawling them spends # budget to index the same page many times over. Disallow: /auth/ Disallow: /__manifest # The marketing site and The Reflog publish separate sitemaps from the same # origin. Sitemap: https://composal.com/sitemap.xml Sitemap: https://composal.com/reflog/sitemap-index.xml