Fetching

Most pages need no special setup. analog get uses a browser on your machine, runs the page’s JavaScript, and captures what the page renders. Start with the ordinary command and read its preview before adding fetch options.

If you have not chosen between structured extraction and local Markdown, start with analog assess. This guide begins after you have chosen how to use the page.

Choose a fetch path

The built-in browser is the default from both the command line and Python. Use it for pages that render with JavaScript or place content behind interactive controls.

Python also supports two direct paths:

• Pass fetcher=HttpFetcher() when a plain HTTP response contains everything you need and no JavaScript or interaction is required.
• Pass html= when you already have the page content. Analog performs no fetch in this path. A custom Fetcher can supply content from an environment you control.

from analog import HttpFetcher, analog

result = analog("<url>", fetcher=HttpFetcher())

The Python API reference lists the fetcher interfaces. If you provide signed-in page content through html= or a custom fetcher, read the boundary in Privacy.

Capture more of a page

Run once with the defaults. If the preview reports content the browser could have opened or followed, rerun with the matching option:

• The page opens on a filtered subset. Analog looks for a show-everything control by default. Pass --no-reveal-all only when you want the page’s initial view unchanged. In Python, use reveal_all=False.
• A Load more or Show more button reveals additional items. Pass --load-all, or load_all=True in Python.
• Each item has a disclosure, accordion, or detail toggle. Pass --expand-all, or expand_all=True in Python.
• The collection continues across pages or numbered batches. Pass --pages N, or pages=N in Python, with the number of pages you want Analog to follow.

The options can be combined when the page requires more than one kind of interaction:

analog get <url> --load-all --expand-all --pages 5

--load-all is off by default because a large page may require many button presses. --expand-all is off because opening every item’s details changes what the page presents. Both operations are bounded; expansion is also paced politely.

Check what was captured

The result reports interactive controls the browser found and whether it used them. An unused load-more or expansion control appears under browse_actions with the option that enables it.

A one-page fetch also reports when the page appears to continue and states that the current records cover only this page. When the page publishes a total, Analog places that number beside the count it captured rather than presenting a partial result as the whole collection.

If the capture still looks wrong, run the command with --headed to watch the browser. The Troubleshooting guide’s Markdown check then distinguishes content the browser never captured from content that was captured but not returned as records.

Pagination reference

Pagination sweeps are experimental and accept between 1 and 50 pages. The default is 1, which follows nothing.

Links to another page

With --pages N, Analog follows the next link each page renders. It does not construct page URLs, stays on the page’s own host, and pauses politely between fetches. The result reports the pages covered, why the sweep stopped, and how many duplicate records were dropped.

Each retained record also carries the URL of the page that supplied it. page_sweep.source_page_field names that field, normally source_page_url; if the page already uses that name, Analog chooses a collision-safe suffix.

Structured and auto modes make one extraction request per page. Local mode makes no extraction requests.

Numbered batches on one page

Some sites replace the visible batch without changing the URL. The built-in browser can drive those numbered controls with the same --pages N option and merge the batches into one result.

Because every batch shares one URL, records do not receive a source-page field. page_sweep.mechanism states whether the sweep followed links or used in-page controls.

Merging pages and batches

Compatible sections are combined. When a page lacks a field found on another page, its records receive a null value for that field and the sweep note names it. Sections that cannot be combined stay separate, and the note says which ones.

When the page states its own total in pagination text or a count heading, the result preserves both the total and where it appeared as page_sweep.stated_total and page_sweep.stated_total_source. The sweep summary places that number beside the merged record count so partial coverage remains visible.

Fetching boundaries

The built-in browser has a stable identity and respects robots.txt, including feed-verification requests made by analog assess. Analog does not use proxies, fingerprint spoofing, or CAPTCHA solving. A site declining automated visitors is a result Analog reports, not a block it tries to evade.

Analog clicks controls available to every visitor, follows links the page renders, and reports what it chose not to do. Private-network and local addresses are refused by default; Security documents that boundary.