Browsing and procedures

scrape_url pulls the readable article out of a page. Plenty of sources do not work that way: registries behind a search form, portals that render everything with JavaScript, result lists that paginate, sites that want a session before they show anything. For those the assistant has a browser it can drive step by step, and a way to remember what worked on a site.

The browse tool

browse controls a hidden Chromium window that belongs to the current board. It stays open between calls, so the assistant can search, read, follow a link, go back and read again without losing its place. Each call is one action:

Action	What it does
open	Load an http(s) URL.
read	The rendered page as text: headings, bullet lists, tables, and every link inline as [url] text. Long pages are returned in chunks with a nextOffset to continue from. An optional selector reads just one element.
links	Every link on the page, optionally filtered by a substring or regex.
forms	Every visible input, select, button and control, with a selector the assistant can use, its label and current value.
click	Click by CSS selector or by visible text ("Next", "Search", "Dockets").
fill	Type into an input (optionally press Enter to submit).
select	Choose an option in a <select>.
press	Send a key: Enter, Escape, Tab, PageDown.
scroll	Top, bottom, or a screen up/down - for pages that load on scroll.
wait	Wait for a selector or a piece of text to appear, or a fixed delay.
back	Browser back.
history	The steps taken so far in this session - the raw material for a procedure.
handoff / resume	Show the window to you, then take it back (below).
close	Discard the session.

After any action that changes the page the assistant gets back the new URL, title and the first part of the text, so it can see what happened without an extra call.

Sessions close by themselves after five minutes idle, and when DeepState quits.

Logins and CAPTCHAs

The assistant never types passwords, one-time codes or card numbers, and refuses if asked. When a site wants a sign-in, the assistant calls handoff: the browser window appears on your screen, and the assistant asks you (through the normal question prompt) to log in and confirm. When you do, it calls resume, the window hides again and it carries on where you were.

Cookies are kept in a dedicated browser profile that only the assistant uses, separate from your own browsers. A site you signed in to once stays signed in for later sessions, on any board.

What it will not do

• Open anything other than http and https URLs.
• Download files (downloads are cancelled).
• Open pop-ups or new tabs (links that would are followed in the same window).
• Run arbitrary JavaScript: there is no eval action.

Procedures

Working out a site takes a few tries: which selector is the search box, whether Enter submits or a button does, what to wait for before results are trustworthy. Once the assistant has a sequence that works, it saves it with store_procedure: a name, a description, the ordered steps, and parameters for the parts that change between runs.

Parameters are written into steps as {{name}}; {{name|url}} URL-encodes the value for query strings. A procedure for a court registry might be:

{
  "name": "courtlistener_party",
  "description": "Search CourtListener dockets by party name and read the results table.",
  "params": [
    { "name": "party", "description": "party name", "required": true },
    { "name": "type", "default": "r" }
  ],
  "steps": [
    { "action": "open", "url": "https://www.courtlistener.com/?q={{party|url}}&type={{type}}" },
    { "action": "wait", "selector": "#result-count" },
    { "action": "read", "selector": "#search-results" }
  ]
}

Next time you ask about a different party, the assistant calls run_procedure with { "party": "Acme Holdings" } and gets the results table back - one tool call instead of five. Called without a name, run_procedure lists what has been saved.

If a replay fails part-way (the site changed, a selector no longer matches) the browser is left on the page where it stopped, so the assistant can finish by hand and then correct the saved steps with edit_procedure, which can also rename or delete a procedure.

Managing procedures

Procedures are global - they belong to you, not to a board - and are stored in DeepState's database. Settings → Agent → Browse procedures lists them with their site, parameters, how many times they have run and whether the last run failed, and lets you delete any you no longer want. To change one, ask the assistant.

Ask for it

If you know you will hit the same source repeatedly, say so: "Work out how to search the land registry for a lot number, then save it as a procedure." The assistant will explore with browse, confirm the results, and store the sequence with a lot parameter.