> ## Documentation Index
> Fetch the complete documentation index at: https://composio-preview.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# WebCrawlerAPI

> WebCrawlerAPI provides crawling, page scraping, AI web agents, and monitored website feeds through a unified API.

![WebCrawlerAPI logo](https://mintcdn.com/composio-preview/DMB8UrgJmE7r-ZQ5/images/toolkits/webcrawlerapi.svg?fit=max\&auto=format\&n=DMB8UrgJmE7r-ZQ5\&q=85\&s=a9d88b1b688757157018b123520532f5)

* **Category:** ai web scraping
* **Auth:** API\_KEY
* **Composio-managed OAuth available?** N/A
* **Tools:** 10
* **Triggers:** 0
* **Slug:** `WEBCRAWLERAPI`
* **Version:** 20261030\_00

## Tools

> [accordion] Create Crawl
>
> **Slug:** `WEBCRAWLERAPI_CREATE_CRAWL`
>
> Start a metered asynchronous crawl over a site and return its job ID. The crawl may visit and charge for up to items\_limit pages; use GET\_CRAWL\_JOB to monitor it.
>
> #### Input Parameters
>
> **url** `string` *required*
>
> Seed URL where crawling starts.
>
> **max\_age** `integer`
>
> Maximum cached-result age in seconds. Defaults to allowing cached content up to seven days old; use 0 to disable cache reuse.
>
> **max\_depth** `integer`
>
> Maximum link depth; 0 crawls only the seed URL, while omission leaves the depth unrestricted.
>
> **items\_limit** `integer`
>
> Maximum number of pages to crawl and potentially charge for.
>
> **webhook\_url** `string`
>
> Public HTTPS URL to notify when the crawl finishes.
>
> **output\_formats** `array`
>
> Representations to generate for each crawled page.
>
> **blacklist\_regexp** `string`
>
> Exclude discovered URLs matching this regular expression.
>
> **whitelist\_regexp** `string`
>
> Only crawl discovered URLs matching this regular expression.
>
> **keep\_query\_params** `boolean`
>
> Keep query parameters when deduplicating discovered URLs.
>
> **main\_content\_only** `boolean`
>
> Exclude navigation and peripheral content from page output.
>
> **respect\_robots\_txt** `boolean`
>
> Honor the target site's robots.txt rules.
>
> #### Output
>
> **data** `string` *required*
>
> Data from the action execution
>
> **error** `string`
>
> Error if any occurred during the execution of the action
>
> **successful** `boolean` *required*
>
> Whether or not the action execution was successful or not

> [accordion] Create Feed
>
> **Slug:** `WEBCRAWLERAPI_CREATE_FEED`
>
> Create a recurring website-change feed. Creation immediately schedules an initial crawl that may consume metered credits, and future scheduled runs may also consume credits; delete the feed when it is no longer needed.
>
> #### Input Parameters
>
> **url** `string` *required*
>
> Website URL to monitor.
>
> **name** `string`
>
> Human-readable feed name.
>
> **max\_depth** `integer`
>
> Maximum link depth for each run; 0 checks only the seed URL.
>
> **items\_limit** `integer`
>
> Maximum number of pages checked per feed run.
>
> **webhook\_url** `string`
>
> Public HTTPS URL to receive feed-run notifications.
>
> **output\_format** `string ("markdown" | "cleaned" | "html")`
>
> Content representation stored for changed pages.
>
> **include\_errors** `boolean`
>
> Whether unavailable or error pages should be included in feed output.
>
> **blacklist\_regexp** `string`
>
> Exclude URLs matching this regular expression.
>
> **whitelist\_regexp** `string`
>
> Only monitor URLs matching this regular expression.
>
> **main\_content\_only** `boolean`
>
> Whether to exclude navigation and peripheral content when detecting changes.
>
> **respect\_robots\_txt** `boolean`
>
> Whether to honor the target site's robots.txt rules.
>
> #### Output
>
> **data** `string` *required*
>
> Data from the action execution
>
> **error** `string`
>
> Error if any occurred during the execution of the action
>
> **successful** `boolean` *required*
>
> Whether or not the action execution was successful or not

> [accordion] Delete Feed
>
> **Slug:** `WEBCRAWLERAPI_DELETE_FEED`
>
> Permanently cancel a feed so it cannot be resumed. The feed is removed from feed listings, but retained history may remain directly readable for 30 days.
>
> #### Input Parameters
>
> **feed\_id** `string` *required*
>
> Feed ID to permanently cancel.
>
> #### Output
>
> **data** `string` *required*
>
> Data from the action execution
>
> **error** `string`
>
> Error if any occurred during the execution of the action
>
> **successful** `boolean` *required*
>
> Whether or not the action execution was successful or not

> [accordion] Get Crawl Job
>
> **Slug:** `WEBCRAWLERAPI_GET_CRAWL_JOB`
>
> Get a crawl job's status, configuration, per-page results, content URLs, errors, and observed costs. Respect recommended\_pull\_delay\_ms when polling.
>
> #### Input Parameters
>
> **crawl\_id** `string` *required*
>
> Crawl job ID returned by CREATE\_CRAWL.
>
> #### Output
>
> **data** `string` *required*
>
> Data from the action execution
>
> **error** `string`
>
> Error if any occurred during the execution of the action
>
> **successful** `boolean` *required*
>
> Whether or not the action execution was successful or not

> [accordion] Get Feed
>
> **Slug:** `WEBCRAWLERAPI_GET_FEED`
>
> Get one feed's configuration, lifecycle status, and recent run history, including per-run crawl counts and cost. Canceled feeds remain directly readable while retained by the provider.
>
> #### Input Parameters
>
> **feed\_id** `string` *required*
>
> Feed ID returned by CREATE\_FEED or LIST\_FEEDS.
>
> #### Output
>
> **data** `string` *required*
>
> Data from the action execution
>
> **error** `string`
>
> Error if any occurred during the execution of the action
>
> **successful** `boolean` *required*
>
> Whether or not the action execution was successful or not

> [accordion] Get Organization Costs
>
> **Slug:** `WEBCRAWLERAPI_GET_ORGANIZATION_COSTS`
>
> Return current spendable balance plus request count and USD usage for a date range. The connected API key is used for both organization routes.
>
> #### Input Parameters
>
> **to\_date** `string`
>
> Inclusive usage end date in YYYY-MM-DD format; defaults to the current UTC date.
>
> **from\_date** `string`
>
> Inclusive usage start date in YYYY-MM-DD format; defaults to 30 days before to\_date.
>
> **include\_daily** `boolean`
>
> Include one usage row per day, including dates with no usage.
>
> #### Output
>
> **data** `string` *required*
>
> Data from the action execution
>
> **error** `string`
>
> Error if any occurred during the execution of the action
>
> **successful** `boolean` *required*
>
> Whether or not the action execution was successful or not

> [accordion] List Feed Changes
>
> **Slug:** `WEBCRAWLERAPI_LIST_FEED_CHANGES`
>
> Return one page of detected feed changes as structured JSON, with an opaque continuation cursor for older pages.
>
> #### Input Parameters
>
> **feed\_id** `string` *required*
>
> Feed ID whose detected changes should be returned.
>
> **page\_size** `integer`
>
> Maximum changes to return on this page (1-1000).
>
> **next\_cursor** `string`
>
> Opaque cursor returned by the previous call; omit for the newest page.
>
> #### Output
>
> **data** `string` *required*
>
> Data from the action execution
>
> **error** `string`
>
> Error if any occurred during the execution of the action
>
> **successful** `boolean` *required*
>
> Whether or not the action execution was successful or not

> [accordion] List Feeds
>
> **Slug:** `WEBCRAWLERAPI_LIST_FEEDS`
>
> List active and paused feeds for the connected organization, newest first. Logically canceled feeds are omitted.
>
> #### Output
>
> **data** `string` *required*
>
> Data from the action execution
>
> **error** `string`
>
> Error if any occurred during the execution of the action
>
> **successful** `boolean` *required*
>
> Whether or not the action execution was successful or not

> [accordion] Scrape Page
>
> **Slug:** `WEBCRAWLERAPI_SCRAPE_PAGE`
>
> Scrape one web page synchronously and return requested content or structured extraction. This consumes metered credits; supplying a prompt adds \$0.002 per request.
>
> #### Input Parameters
>
> **url** `string` *required*
>
> Absolute HTTP or HTTPS URL of the page to scrape.
>
> **prompt** `string`
>
> AI extraction instruction. Supplying this adds \$0.002 to the request.
>
> **max\_age** `integer`
>
> Maximum cached-result age in seconds. Defaults to allowing cached content up to seven days old; use 0 to force a fresh scrape.
>
> **output\_formats** `array`
>
> Content representations to return.
>
> **clean\_selectors** `string`
>
> Comma-separated CSS selectors to remove before content extraction.
>
> **response\_schema** `object`
>
> JSON Schema for structured\_data when prompt is supplied.
>
> **keep\_query\_params** `boolean`
>
> Preserve query parameters when identifying the target URL.
>
> **main\_content\_only** `boolean`
>
> Exclude navigation and other peripheral page content.
>
> **respect\_robots\_txt** `boolean`
>
> Honor the target site's robots.txt rules.
>
> #### Output
>
> **data** `string` *required*
>
> Data from the action execution
>
> **error** `string`
>
> Error if any occurred during the execution of the action
>
> **successful** `boolean` *required*
>
> Whether or not the action execution was successful or not

> [accordion] Set Feed State
>
> **Slug:** `WEBCRAWLERAPI_SET_FEED_STATE`
>
> Pause an active feed's future scheduled runs or resume a paused feed. Pausing does not stop a run that has already started, and resuming schedules future crawls that may consume metered credits.
>
> #### Input Parameters
>
> **feed\_id** `string` *required*
>
> Feed ID to pause or resume.
>
> **desired\_state** `string ("paused" | "active")` *required*
>
> Set paused to stop future schedules or active to resume scheduling.
>
> #### Output
>
> **data** `string` *required*
>
> Data from the action execution
>
> **error** `string`
>
> Error if any occurred during the execution of the action
>
> **successful** `boolean` *required*
>
> Whether or not the action execution was successful or not
