Execution boundaries and HTTP access
The skill gathers recent public evidence through local tools, direct HTTP services, browser sessions, and an optional hosted API. It keeps these paths separate so a search can use the right access method while limiting credential reads, command execution, network calls, and data sent to remote systems.
This page is part of Runtime configuration and safety. Read Setup, credentials, and preflight for credential setup and Health, diagnostics, and recovery for source health and recovery guidance.
Boundary model
The runtime has two main external boundaries. Subprocesses reach installed tools or a selected SSH host. HTTP clients reach source services or a configured hosted service. Policy at the command entry point decides which boundaries are allowed before source work starts.
| Boundary | Main purpose | Safety property |
|---|---|---|
| Local subprocess | Run tools such as yt-dlp, Node-based X search, xurl, and macOS helpers | Uses argument lists, captures output, and bounds runtime |
| HTTP client | Call source APIs and public web endpoints | Adds deadlines, retries, error classes, throttling, redirect guards, and secret scrubbing |
| Browser cookie read | Reuse an existing signed-in session | Off by default for library calls and inspection modes |
| Hosted API | Submit a complete search to a configured remote service | Requires both an API key and an explicit API base |
References: skills/…/subproc.py• L1-16 skills/…/http.py• L97-170 skills/…/env.py• L98-112 skills/…/last30days.py• L2591-2623
Subprocess execution
The subprocess boundary gives source adapters access to external command-line tools without letting a slow or broken child control the run. The shared process runner captures standard output and error, returns normal exit codes, and turns a timeout or cancellation into a controlled failure.
Process lifetime
Each child runs in its own process group on POSIX systems. Windows children run in a kill-on-close job. On timeout, cancellation, shutdown, or an operation deadline, the runner stops the child and its descendants, then escalates when needed. This prevents helpers spawned by a command from remaining after the parent has ended.
The runner can report the child PID to the top-level process. The X search path uses this to track children for wider cleanup. A missing executable remains a spawn error, while a non-zero exit remains data for the caller to classify. The runner can also cap captured output by using temporary files instead of unbounded pipes.
References: skills/…/subproc.py• L20-43 skills/…/subproc.py• L121-153 skills/…/subproc.py• L156-207 skills/…/subproc.py• L275-365 skills/…/bird_x.py• L261-306
Command routing
The skill uses subprocesses for tools that are not simple Python library calls. YouTube work can run locally or through an SSH alias when a residential network path is needed. X search can run through the vendored Node client or the official xurl command.
SSH routing accepts only a plain host or SSH alias. The host is placed after an option terminator and the remote command is quoted as one command string. Batch mode makes a bad key or host fail instead of waiting for an interactive password prompt.
References: skills/…/youtube_yt.py• L285-353 skills/…/xurl_x.py• L57-87 skills/…/xurl_x.py• L158-200
HTTP access
The shared HTTP boundary gives source-specific calls one request model. It supplies request defaults and encodes request data. It applies per-attempt timeouts and bounded retries, then reports failures with source-health states such as rate limited, authentication failed, timed out, unreachable, or schema drift.
The caller can provide one wall-clock deadline for all attempts. The client stops retry waits and in-progress reads when that deadline expires. A bounded GET transport can also stop waiting during DNS or response-body reads. The client removes authentication headers when a redirect crosses to a different origin, and a caller can reject a redirect before it reaches a private or unrelated destination.
DNS failures receive a small recovery budget even when a caller requests few retries. Other network errors and HTTP errors keep the caller's normal retry budget. Rate-limit waits use server reset hints when available and remain bounded so a worker does not wait for a long server reset.
Keyless Reddit access adds a shared rate limit across its public request tiers. A short-lived run memo serves repeated successful requests, while concurrent requests for the same URL share one in-flight fetch. A fetched body can be checked before it enters that memo, so a challenge page is not reused as a valid result. A rate-limited fetch gets one short retry through the shared limiter instead of holding a worker for a long server reset. Together these controls reduce duplicate traffic and prevent a broad search from creating a request burst.
References: skills/…/http.py• L33-170 skills/…/http.py• L917-1301 skills/…/http.py• L1392-1553 skills/…/http.py• L1579-1695
Request isolation
HTTP failures can be captured in the current retrieval context without shared mutable state between worker threads. Expected probe misses can be excluded from the run's failure record, while terminal failures remain available to the pipeline even when a source adapter returns an empty result.
The HTTP layer also supports recorded exchanges for offline evaluation. Replay fails closed when code makes an unrecorded request. Recorded URLs, request data, source results, and response values are scrubbed so credentials do not become fixture data. Credential-bearing URLs can be excluded from diagnostics and fixture capture. The recording path also creates its temporary fixture with user-only permissions before writing and then replaces the target file, which prevents a credential-adjacent fixture from being briefly exposed through a wider file mode.
References: skills/…/http.py• L173-205 skills/…/http.py• L217-343 skills/…/http.py• L367-433 skills/…/http.py• L447-643 skills/…/http.py• L917-1007
Keyless page access
Some public pages use a fallback reader rather than a direct page fetch. The reader sends the public URL to Jina's hosted endpoint and returns clean markdown or a typed failure. This path is best effort because the reader can be rate limited or return a cached snapshot.
Only public-research URLs should use this third-party path. It must not carry private pages or session credentials.
References: skills/…/web_fetch_keyless.py• L1-15 skills/…/web_fetch_keyless.py• L48-105
Session-bound page access
An opt-in Reddit session path uses a browser cookie for pages that need a signed-in session. It accepts only HTTPS Reddit destinations and keeps session pages in a separate rate limit and memo, so a logged-in response cannot cross into the keyless path or be returned to a logged-out caller.
The session path returns the response status and body without raising into the source pipeline. It memoizes successful pages only. Error pages remain visible for session diagnosis and are not reused as valid content.
References: skills/…/http.py• L1698-1776
Hosted runtime
Hosted mode moves the search work to a configured remote API. It activates only when both LAST30DAYS_API_KEY and LAST30DAYS_API_BASE are present, when no local corpus is selected, and when the run is not a mock or a local-only deep-research path. There is no built-in remote endpoint. The configured base must use HTTPS, except for allowed loopback HTTP addresses.
The local client submits the topic once, polls the returned search ID, shows new progress narration, and renders the server report when the run reaches a terminal state. The submit call is not blindly retried because a repeated POST could create a second search. Poll requests are safe to retry because they are reads. It does not reproduce pricing or billing rules. It displays balance data only when the API returns it, and it never logs or saves the API key.
The hosted result can include the engine version that produced it. When its major version differs from the local skill, the client warns that hosted and local results may differ. Saved hosted output uses a path-safe suffix and an exclusive filename so a user-selected suffix cannot add path components or overwrite an existing result.
This diagram shows the hosted search lifecycle and the point where local output replaces remote progress.
flowchart TD
A[Topic and runtime settings] --> B{Hosted mode allowed?}
B -- No --> C[Run local sources]
B -- Yes --> D[POST topic to configured API]
D --> E{Clarification needed?}
E -- Yes --> F[Show choices and stop]
E -- No --> G[Poll search status]
G --> H{Terminal status?}
H -- No --> G
H -- Error --> I[Show server error]
H -- Complete --> J[Render server report]
The hosted path has distinct handling for invalid credentials, insufficient credits, rate limits, clarification requests, network loss, and a poll that exceeds its overall time limit. A local corpus takes precedence so its files remain on the local machine.
References: skills/…/last30days.py• L3226-3280 skills/…/hosted.py• L1-30 skills/…/hosted.py• L63-105 skills/…/hosted.py• L179-223 skills/…/hosted.py• L247-312 skills/…/hosted.py• L339-407
Browser cookie extraction
Cookie extraction lets the skill use an existing browser login for services that need session access. The policy separates permission to inspect browser stores from the act of using the resulting values. A normal research run can read selected cookies, while setup needs an explicit allow flag and inspection commands use a plan-only mode that does not read values.
The X path needs both auth_token and ct0. Truth Social uses _session_id. Existing explicit credentials take priority, and a browser result is used only when the matching configuration value is not already set.
| Context | Browser-cookie behavior |
|---|---|
| Normal research | Read when browser selection is configured; --no-browser-cookies disables it |
| Setup | Read only with --allow-browser-cookies |
| Diagnose, preflight, doctor | Resolve possible browsers without reading cookie values |
| Library and queue commands | Do not read browser cookies |
| Unattended watchlist | Passes --no-browser-cookies to every search |
References: skills/…/env.py• L98-112 skills/…/env.py• L657-757 skills/…/last30days.py• L2522-2528 skills/…/last30days.py• L2591-2623 skills/…/watchlist.py• L170-197
Browser data handling
The extractor reads only named cookies for a named domain. It checks the default Firefox profile first, then other profiles, and prefers a profile that contains the complete requested cookie set over a partial match. macOS supports Firefox, Safari, and the Chromium browser family. Linux auto mode uses Firefox. WSL can fall back to a Windows Firefox profile when native Linux Firefox has no matching session.
Locked browser databases are copied to temporary files with user-only permissions and removed after use. On macOS, Chromium-family cookies use the platform's protected store and Safari cookies come from its binary cookie store. Safari values are accepted only for the requested host or a subdomain. Decrypted cookie values stay in memory for the current run. A failed browser read becomes an unavailable source rather than a reason to stop unrelated sources.
References: skills/…/cookie_extract.py• L27-49 skills/…/cookie_extract.py• L97-125 skills/…/cookie_extract.py• L252-317 skills/…/cookie_extract.py• L319-430 skills/…/cookie_extract.py• L433-549 skills/…/cookie_extract.py• L552-685 skills/…/safari_cookies.py• L94-154 skills/…/safari_cookies.py• L157-208
Credential handoff
The Python runtime passes selected X session values to the Node search client through a child environment. The child process disables its own browser-cookie fallback, so a normal run does not create a second hidden cookie read or a Keychain prompt. The official xurl path instead uses its own stored app credential or local token store according to the selected backend.
Inspection uses local evidence for xurl authentication. It checks the token store and does not run xurl whoami, because that command would make a live authenticated network request during a safe diagnostic.
References: skills/…/bird_x.py• L54-83 skills/…/xurl_x.py• L90-155 skills/…/xurl_x.py• L158-200
Security-sensitive helpers
The boundary helpers protect the points where user-controlled configuration can become a command, a network destination, or a stored credential. Their value is in preserving a clear decision before the external action starts.
Key controls include:
- Rejecting unsafe SSH host values before they reach
ssh. - Using argument lists for local commands instead of interpreting user text as a shell command.
- Removing authentication headers when an HTTP redirect crosses to another origin.
- Rejecting redirects that leave an approved destination, such as the HTTPS Reddit site boundary.
- Keeping hosted credentials in request headers and out of logs, output, and saved files.
- Redacting secret-shaped fields and known secret values from HTTP fixtures. This includes password, passphrase, credential, bearer, JWT, and related key names.
- Excluding sensitive URLs from request diagnostics and fixture capture when a request must carry a credential in its address.
- Creating recorded HTTP fixture files through a user-only temporary file before replacing the final file.
- Avoiding cookie reads during safe inspection and unattended runs.
- Treating third-party page readers as public-only paths.
- Restricting temporary cookie copies to the current user while they are read.
- Rejecting incomplete browser-cookie matches when a complete session pair is available in another profile.
- Requiring the hosted API base to use a permitted scheme and keeping the API key in an authorization header.
- Sanitizing hosted save suffixes before using them in output filenames.
References: skills/…/youtube_yt.py• L296-353 skills/…/http.py• L90-170 skills/…/http.py• L217-343 skills/…/http.py• L367-433 skills/…/http.py• L917-1025 skills/…/http.py• L1719-1764 skills/…/hosted.py• L63-89 skills/…/hosted.py• L274-312 skills/…/cookie_extract.py• L27-49 tests/test_security_boundaries.py• L17-155
Change checklist
When these files or sections are changed, remember to consider these:
- Keep the command-entry policy and setup policy aligned so
--no-browser-cookies, inspection modes, and unattended watchlist runs cannot regain browser access through a second path. - Keep the X credential handoff aligned with the Node client's browser-cookie disable flag and the cookie names mapped in env.py.
- Keep HTTP redirect handling aligned with the set of headers that can carry credentials across origins and with any destination-specific redirect guard.
- Keep HTTP fixture key matching, value redaction, sensitive-URL exclusion, and user-only file creation aligned when a new secret-bearing field or recording path is added.
- Keep the shared keyless Reddit limiter, run memo, body validation, and batch wait allowance aligned when adding or changing a public Reddit request tier.
- Keep the keyless and session Reddit limiters and memos separate when changing signed-in page access.
- Keep hosted-mode activation aligned with local-corpus precedence and the API contract for submit, poll, clarification, and terminal responses.
- Keep hosted API-base validation, authorization-header use, engine-version reporting, and save-suffix sanitization aligned when the remote contract or output options change.
- Keep subprocess timeout cleanup, deadline handling, output capture, and any child-PID registration aligned when adding a command that can spawn descendants.
- Keep Safari and Chromium cookie-store paths, host matching, profile selection, and requested cookie names aligned with the common browser-cookie policy when browser support changes.
Important failure modes
| Trigger | Consequence | Guard |
|---|---|---|
| A subprocess exceeds its deadline | The command and its child group may hang or remain orphaned | Dedicated POSIX groups, Windows jobs, and bounded TERM-to-KILL cleanup in subproc |
| A subprocess emits more output than the run can safely hold | Memory use can grow while the caller waits | Optional bounded capture through temporary files |
| An SSH host setting contains flags or shell characters | ssh could interpret configuration as options or a proxy command | Plain alias validation and the -- option terminator |
| An HTTP redirect crosses to a different origin | An API credential could be sent to the redirected host | Remove authentication headers on cross-origin redirects |
| A fetched URL redirects to a private or unapproved destination | The request could reach a host outside the source's trust boundary | Apply the redirect guard before following the redirect |
| DNS resolution fails for a short period | A healthy source could be marked unreachable before DNS recovers | Dedicated DNS retries with increasing waits |
| A request deadline expires during DNS or body reading | A worker can remain blocked after the search deadline | Deadline-aware GET transport and cancellation checks |
| A browser cookie store is locked or permission is missing | The session-backed source is unavailable | Copy-and-read for locked databases; return no credentials and continue |
| A default browser profile has only part of a required cookie pair | A logged-out or partial session can shadow a complete session in another profile | Prefer a complete requested cookie set while scanning profiles |
| Safari is unavailable, unreadable, or has no matching named cookie | A macOS session-backed source cannot authenticate the related source | Restrict extraction to macOS, validate the binary store, match the requested host, and return no credentials on failure |
| Diagnostic code calls a live auth command | A supposedly safe inspection performs a network request or prompts for auth | Use plan-only cookie policy and local xurl token-store evidence |
| Keyless Reddit tiers fan out without shared throttling or reuse | Rate limits and duplicate requests can make source work exceed its result window | Shared token bucket, same-URL in-flight sharing, response validation, and run-scoped memo |
| A logged-in Reddit response enters the keyless path | A session-bound page can leak across access modes | Separate session limiter and memo, plus an HTTPS Reddit-only redirect guard |
| A hosted API key or fixture secret is included in normal output | A credential can leak through logs or offline artifacts | Header-only key use, secret-value redaction, sensitive-URL exclusion, and user-only fixture creation |
| A hosted base uses an unsafe scheme or a save suffix contains path syntax | The client can contact an unapproved endpoint or write outside the intended output name | HTTPS-or-loopback validation and path-safe suffix sanitization |
| A keyless reader receives a private URL | The URL and its page content go to a third party | Restrict the helper to public-research fetches |
| Hosted polling loses the network or exceeds its poll window | The local run cannot produce a remote report | Retry idempotent GET polls and stop with a user-facing failure |
Key files
| File | Why to read it |
|---|---|
| skills/…/last30days.py | See how command intent selects cookie policy and how local and hosted execution are separated. |
| skills/…/env.py | See the central rules for browser access, credential precedence, and pending browser auth. |
| skills/…/subproc.py | See the shared process lifetime, deadline, output, and timeout boundary. |
| skills/…/http.py | See request deadlines, retries, redirect destination and credential protection, source failure states, public-source throttling, session separation, response validation, and secret-safe fixture handling. |
| skills/…/hosted.py | See the remote submit, poll, clarification, version notice, output-save, and report lifecycle. |
| skills/…/cookie_extract.py | See the common browser-cookie boundary, complete-session selection, secure temporary reads, and platform selection. |
| skills/…/safari_cookies.py | See the macOS Safari cookie-store boundary and host-scoped cookie extraction. |
| skills/…/bird_x.py | See how session credentials cross into the Node-backed X search child. |
| skills/…/youtube_yt.py | See the controlled local and SSH-routed command boundary. |
| skills/…/web_fetch_keyless.py | See the public-only third-party page-reader fallback. |