Enrichment
When you drop a URL, a post or a person onto the board, DeepState does more than store the link. A background enrichment queue fetches, extracts, archives and summarizes so the board holds the material itself rather than a pointer to it. This page lists what runs, when, and what it needs.
What runs, and when
| Trigger | Job | What it does |
|---|---|---|
| Embed created or URL changed; Event source added | Link preview | Fetches OpenGraph / Twitter-card metadata: title, description, image, author, site |
| News Article created (by you, the AI or a feed) | Article scrape | Loads the page in a hidden browser, extracts the readable article, then summarizes it with the summarizer model |
| Social Post created | Media archive | Downloads the post's images and video into the board's media folder |
| Social Post that links to an article | Link expansion | Fetches the linked page's preview and stores it on the post |
| Actor created | Actor profile | Wikipedia lookup by name; attaches summary and thumbnail if found |
| Organization created | Organization profile | Same, for organizations |
| Media created from a URL | Store media | Copies the file into the board's media folder |
| RSS Feed due | Feed poll | Fetches the feed and creates nodes for new items (RSS feeds) |
Everything runs inside the app on your machine. The only network traffic is to the sites being fetched (and to your AI provider for summaries).
Article extraction in detail
- The URL is opened in a hidden browser window so JavaScript-rendered pages and consent walls behave as they would for a reader.
- Mozilla's Readability algorithm extracts the article body, byline, publish date and hero image.
- For a list of major paywalled sites (NYT, WSJ, Washington Post, FT, The Atlantic, Bloomberg, The New Yorker, The Economist, Business Insider, Wired, The Information, Foreign Affairs, HBR, Medium) the extractor goes through archive.today first, which usually has a snapshot.
- The text is summarized: a short summary, key points, sentiment and topics, produced by your summarizer model. See Summarizer.
If extraction fails the node shows scrape failed with the reason; Re-scrape article in the editor tries again. Some sites block headless browsers entirely - for those, paste the text into a Note.
Queue behaviour
- The queue is stored in the database, so jobs survive a restart.
- It is checked every 5 seconds and runs jobs in priority order; jobs you trigger directly (a paste, a Refresh preview) are higher priority than feed polls.
- Each job gets 3 attempts before it is marked failed.
- Duplicate jobs for the same node collapse into one.
- Completed jobs update the node live on every open window.
Seeing status
Cards reflect enrichment state directly: a News Article shows a scraping… state until the article lands and a small AI summarized mark when the summary is in; Social Posts show their media once archived; Actors gain a Wikipedia icon and summary. Failures appear as an error line on the card with the message from the fetch.
Re-running
| Node | Button |
|---|---|
| News Article | Re-scrape article (editor) |
| Embed | Refresh preview (editor) |
| Social Post | Fetch post data from URL (editor) |
| Event | Refresh map (editor, when coordinates are set) |
| RSS Feed | Refresh feed (on the card) |
| Media | Analyze image (editor) PRO - runs OCR and vision analysis on demand |
Not automatic (yet)
To set expectations: DeepState does not currently extract text from Document files, does not
automatically take screenshots or Wayback snapshots of Embeds (use the assistant's
wayback_lookup / wayback_fetch tools or paste an archive URL into the node's archived URL
field), and does not run entity extraction over article text on its own. Ask the assistant for any
of these - "read these three articles and create actor nodes for everyone named" works well.