Documents and website crawling

1. Select your organization and database, then open Documents.
2. Choose Upload files to select one or more files. Follow upload progress and inspect any failures.
3. Use the document editor to create a new entry or edit an existing one. Review the saved content and path so you can identify it later.

Documents belong to the selected database. Confirm the database before uploading or editing shared material.

Use the AI visibility control beside a document to include or exclude it from AI retrieval. Selection controls support bulk visibility changes and deletion. Review any failed items after a bulk operation.

AI visibility is a retrieval setting, not a substitute for database access control or a public sharing permission. Keep structured application records in their domain tables and use documents for supporting material.

1. In Website Crawler, enter the starting website URL.
2. Choose whether to restrict the crawl to the same host, then select Crawl website.
3. Review the resulting documents in the selected database and check their contents and AI visibility.

The crawler runs when requested. Rerun it when source pages change; the Documents screen does not provide an automatic recurring crawl schedule. Pages requiring sign-in or unavailable to the crawler may not be captured.

Replace or edit outdated content and remove entries that are no longer useful. Check source dates and the retrieved text when evaluating an AI answer. Upload completion or crawl completion does not guarantee that every expected passage was extracted or used in an answer.

For programmatic uploads and retrieval, use the document API.