page-keeper/ archiver
Unattended website archiver

Keep any website.
Offline, browsable, verifiable.

Give page-keeper one URL. It crawls the whole site, strips the tracking and consent noise, and hands you a self-contained offline snapshot you can open in any browser — years from now.

Get started → See how it works
No per-site selectors Safe to run from cron A–D quality grades
Why it exists

A screenshot isn't a record. A live site is a moving target.

Websites get redesigned, migrated, or quietly disappear. When you need a faithful copy — before a CMS migration, for a compliance archive, or as an offline reference — you want the real, clickable pages, working offline, and a way to know exactly how good the capture is.

The live page

  • Changes the moment you look away
  • Buried under cookie walls & trackers
  • Client-rendered menus, lazy media
  • Gone for good after a migration

The kept snapshot

  • Frozen at a timestamp, never mutated
  • Consent dismissed, tracking stripped
  • Menus & carousels still work offline
  • One folder — or one ZIP — that outlives the site
How it works

One URL in. A trustworthy archive out.

Five stages, fully generic — no per-site scripting. Crawlee drives a real Chromium browser through every page, so what you keep is what a visitor actually saw.

STAGE 01

Discover

Reads robots.txt for sitemaps, probes common sitemap paths, and seeds the crawl. Falls back to the base URL.

STAGE 02

Crawl

Real Chromium per page: dismisses consent overlays, blocks trackers & heavy media, retries transient 5xx/429, follows links, nav, footers & pagination.

STAGE 03

Snapshot

Captures each page self-contained — CSS, images and fonts inlined via SingleFile — so a single .html opens anywhere.

STAGE 04

Post-process

Rewrites in-archive links to local files, injects a tiny offline-UI script, and grays out dead links so you can see what's missing.

STAGE 05

Report

Writes index.html plus a self-contained dashboard: per-page A–D grades, coverage, and every action it took.

What makes it different

Generic by design. It just works on sites it's never seen.

No selectors to write, no per-site config to maintain. The hard parts are handled by behaviour, not brittle rules.

Consent walls, dismissed

Detects cookie & consent overlays by behaviour — high z-index, dialog role, consent wording — clicks accept to reveal gated content, then removes stragglers and restores scroll.

Interactive, offline, without the original JS

SingleFile strips the site's scripts; page-keeper injects a tiny one back. Carousels, sliders and dropdown menus keep working offline — --no-offline-ui to skip.

Quality you can verify

Every page gets an A–D grade and 0–100 fidelity score — images inlined, no page errors, real content — so you know the copy is faithful, not just that it exists.

Built for cron

Fully non-interactive and exits 0 even when some pages fail. Each run is an isolated, timestamped folder — previous snapshots are never touched.

Smart crawl budget

Canonical dedup, rel=next pagination discovery, tracker & media blocking, and tunable concurrency mean the budget goes to unique, real content.

Never archives a broken page

Transient errors are retried with backoff; a page that keeps failing is recorded as failed — never saved as a "502 Bad Gateway" snapshot masquerading as content.

The report

Every run ships a dashboard you can open offline.

Fidelity measures how good the capture is; coverage measures how much of the site you got. page-keeper gives you both — plus a log of every action it performed — in one self-contained report.html.

  • Coverage breakdown — found vs. archived vs. failed vs. skipped vs. found-but-not-captured.
  • Every automated action — navigations, network-idle waits, autoscroll passes, consent buttons clicked, ad iframes removed, requests blocked.
  • Per-page detail — grade, HTTP status, size, images, links, blocked requests, capture format.
  • A visual gallery — a screenshot of every page with --thumbnails.
  • Machine-readable too — report.json and manifest.json for every URL you crawl.
Run it your way

A one-line CLI, or a local dashboard with a live log.

Drive it from a terminal and cron, or open the web UI, tune every option, and watch each page get exported in real time.

CLI

archive-site

Point it at a URL and walk away. Flags override an optional JSON config; the exit code is cron-friendly.

# one-time setup
npm install && npm run build
npx playwright install chromium

# archive a whole site
archive-site https://example.com \
  --output ./archives --max-requests 200 \
  --thumbnails --extract-text

# daily, from cron
0 3 * * * node dist/cli.js https://example.com
Web UI

archive-serve

A local single-user Next.js dashboard, served from one port. Everything the CLI can do, with a progress bar.

  • Enter a URL and tune every option — depth, budget, concurrency, snapshot format, domain scope.
  • Watch a live log stream over SSE, with counters and a progress bar.
  • See the real export quality the moment it finishes — coverage, grades, score.
  • Open the preview, open the report, or download the whole snapshot as a ZIP.
npm run build:frontend
npm run serve   # → http://127.0.0.1:4321
Snapshot formats

Pick your fidelity trade-off.

singlefiledefault

One portable .html per page with CSS, images and fonts inlined. Opens in any browser; falls back to MHTML if the inliner is unavailable.

asset-mirror

Rendered HTML plus a shared, deduplicated assets/ folder — lighter on disk when pages share resources.

mhtml

Chromium's native MHTML capture — robust and complete, opens mainly in Chrome.

Start keeping pages

Before the site changes, keep it.

Turn a single URL into an archive that's browsable, self-contained, and verifiable — and outlives the original.

Install page-keeper Read how it works
npm run dev -- https://example.com  ·  no build required