OliverCrawl

The problem

I was building applications that depended on collecting information from a growing number of websites and keeping that information current over time. What started as a straightforward retrieval task became an increasingly complex operational responsibility. Each source behaved differently, pages changed, some content required browser rendering, and others exposed structured data that could be collected more directly. As the number of sources grew, so did the effort required to retrieve information reliably, maintain it, and handle exceptions.

The cost extended beyond collection itself. Recurring crawls often retrieved information that hadn't changed, consuming resources without adding value. When that content fed AI workflows, unnecessary retrieval also meant unnecessary downstream processing. Accepting URLs from external sources introduced security and reliability risks that required consistent controls across applications.

I needed a common foundation that could manage these concerns once and be reused across projects. I built OliverCrawl to centralize retrieval, change detection, and safety controls, giving applications a consistent way to collect and maintain web data without repeatedly solving the same problems or surrendering control of their own workflows.

The approach

OliverCrawl is a TypeScript library that handles retrieval, safety, and change detection while leaving application-specific decisions with the application. It doesn't require a hosted crawling service, prescribe a database, or take ownership of the data pipeline.

• Safety built into retrieval. Destinations are validated throughout the request process, including redirects. The library also screens collected content for instructions that could compromise downstream AI workflows, quarantining suspicious results rather than passing them through as ordinary page content. Automated tests cover destination validation, redirect handling, and content screening.

• Change-aware collection. Multiple signals help determine whether a page needs to be fetched or processed again. Unchanged content can be skipped before expensive downstream work begins. Local crawl and recrawl benchmarks track throughput, request volume, and transferred data, making it possible to evaluate the effects of changes to retrieval and caching.

• Retrieval that escalates only when needed. The library starts with simpler collection methods and moves to browser rendering or external services when a source requires them. Successful retrieval approaches are reused to avoid repeating unnecessary work. It supports standard HTTP requests, local browser rendering, configured proxy and external retrieval services, and archived content.

• Cost under application control. The default retrieval path requires no API key or account. Paid providers are optional, explicitly configured, and subject to application-defined usage and budget controls.

• More than HTML. The library handles structured data, feeds, calendars, CSV, JSON, and other common web formats. It preserves useful page structure in Markdown and exposes structured source data so applications can use exact information without automatically involving a language model. A fixed test corpus measures extraction accuracy and helps identify regressions as the library evolves.

• Built for recurring work. Crawls can preserve progress, resume interrupted runs, respect source restrictions and request pacing, and distinguish temporary failures from sources that need attention.

• Independent of the consuming application. Storage, scheduling, schemas, and downstream processing remain outside the library. Applications connect their existing systems through a small integration layer rather than adopting a new platform.

The library is also validated beyond its development environment. Continuous integration builds, tests, and packages each release, then installs it in a clean project environment to verify that the published library works as expected.

The result

OliverCrawl is a versioned, reusable library for applications that depend on reliable web collection.

It provides a consistent retrieval and safety layer without requiring applications to surrender control of their data or infrastructure. Its default functionality runs without a paid provider, while optional retrieval methods are available when a source demands more.

The library powers web collection within Fallow, where information is gathered from a large and varied set of sources and maintained over time. It gives that system a shared foundation for retrieving content, recognizing changes, and avoiding unnecessary downstream processing.

Screenshots

Stack

TypeScript · Node.js 20+ · Playwright · Vitest

Access

oliver-chase/olivercrawl (public).