Skip to main content
WebCrawlerAPI logo
  • Category: ai web scraping
  • Auth: API_KEY
  • Composio-managed OAuth available? N/A
  • Tools: 10
  • Triggers: 0
  • Slug: WEBCRAWLERAPI
  • Version: 20261030_00

Tools

Slug: WEBCRAWLERAPI_CREATE_CRAWLStart a metered asynchronous crawl over a site and return its job ID. The crawl may visit and charge for up to items_limit pages; use GET_CRAWL_JOB to monitor it.

Input Parameters

string
required
Seed URL where crawling starts.
integer
Maximum cached-result age in seconds. Defaults to allowing cached content up to seven days old; use 0 to disable cache reuse.
integer
Maximum link depth; 0 crawls only the seed URL, while omission leaves the depth unrestricted.
integer
Maximum number of pages to crawl and potentially charge for.
string
Public HTTPS URL to notify when the crawl finishes.
array
Representations to generate for each crawled page.
string
Exclude discovered URLs matching this regular expression.
string
Only crawl discovered URLs matching this regular expression.
boolean
Keep query parameters when deduplicating discovered URLs.
boolean
Exclude navigation and peripheral content from page output.
boolean
Honor the target site’s robots.txt rules.

Output

string
required
Data from the action execution
string
Error if any occurred during the execution of the action
boolean
required
Whether or not the action execution was successful or not
Slug: WEBCRAWLERAPI_CREATE_FEEDCreate a recurring website-change feed. Creation immediately schedules an initial crawl that may consume metered credits, and future scheduled runs may also consume credits; delete the feed when it is no longer needed.

Input Parameters

string
required
Website URL to monitor.
string
Human-readable feed name.
integer
Maximum link depth for each run; 0 checks only the seed URL.
integer
Maximum number of pages checked per feed run.
string
Public HTTPS URL to receive feed-run notifications.
string ("markdown" | "cleaned" | "html")
Content representation stored for changed pages.
boolean
Whether unavailable or error pages should be included in feed output.
string
Exclude URLs matching this regular expression.
string
Only monitor URLs matching this regular expression.
boolean
Whether to exclude navigation and peripheral content when detecting changes.
boolean
Whether to honor the target site’s robots.txt rules.

Output

string
required
Data from the action execution
string
Error if any occurred during the execution of the action
boolean
required
Whether or not the action execution was successful or not
Slug: WEBCRAWLERAPI_DELETE_FEEDPermanently cancel a feed so it cannot be resumed. The feed is removed from feed listings, but retained history may remain directly readable for 30 days.

Input Parameters

string
required
Feed ID to permanently cancel.

Output

string
required
Data from the action execution
string
Error if any occurred during the execution of the action
boolean
required
Whether or not the action execution was successful or not
Slug: WEBCRAWLERAPI_GET_CRAWL_JOBGet a crawl job’s status, configuration, per-page results, content URLs, errors, and observed costs. Respect recommended_pull_delay_ms when polling.

Input Parameters

string
required
Crawl job ID returned by CREATE_CRAWL.

Output

string
required
Data from the action execution
string
Error if any occurred during the execution of the action
boolean
required
Whether or not the action execution was successful or not
Slug: WEBCRAWLERAPI_GET_FEEDGet one feed’s configuration, lifecycle status, and recent run history, including per-run crawl counts and cost. Canceled feeds remain directly readable while retained by the provider.

Input Parameters

string
required
Feed ID returned by CREATE_FEED or LIST_FEEDS.

Output

string
required
Data from the action execution
string
Error if any occurred during the execution of the action
boolean
required
Whether or not the action execution was successful or not
Slug: WEBCRAWLERAPI_GET_ORGANIZATION_COSTSReturn current spendable balance plus request count and USD usage for a date range. The connected API key is used for both organization routes.

Input Parameters

string
Inclusive usage end date in YYYY-MM-DD format; defaults to the current UTC date.
string
Inclusive usage start date in YYYY-MM-DD format; defaults to 30 days before to_date.
boolean
Include one usage row per day, including dates with no usage.

Output

string
required
Data from the action execution
string
Error if any occurred during the execution of the action
boolean
required
Whether or not the action execution was successful or not
Slug: WEBCRAWLERAPI_LIST_FEED_CHANGESReturn one page of detected feed changes as structured JSON, with an opaque continuation cursor for older pages.

Input Parameters

string
required
Feed ID whose detected changes should be returned.
integer
Maximum changes to return on this page (1-1000).
string
Opaque cursor returned by the previous call; omit for the newest page.

Output

string
required
Data from the action execution
string
Error if any occurred during the execution of the action
boolean
required
Whether or not the action execution was successful or not
Slug: WEBCRAWLERAPI_LIST_FEEDSList active and paused feeds for the connected organization, newest first. Logically canceled feeds are omitted.

Output

string
required
Data from the action execution
string
Error if any occurred during the execution of the action
boolean
required
Whether or not the action execution was successful or not
Slug: WEBCRAWLERAPI_SCRAPE_PAGEScrape one web page synchronously and return requested content or structured extraction. This consumes metered credits; supplying a prompt adds $0.002 per request.

Input Parameters

string
required
Absolute HTTP or HTTPS URL of the page to scrape.
string
AI extraction instruction. Supplying this adds $0.002 to the request.
integer
Maximum cached-result age in seconds. Defaults to allowing cached content up to seven days old; use 0 to force a fresh scrape.
array
Content representations to return.
string
Comma-separated CSS selectors to remove before content extraction.
object
JSON Schema for structured_data when prompt is supplied.
boolean
Preserve query parameters when identifying the target URL.
boolean
Exclude navigation and other peripheral page content.
boolean
Honor the target site’s robots.txt rules.

Output

string
required
Data from the action execution
string
Error if any occurred during the execution of the action
boolean
required
Whether or not the action execution was successful or not
Slug: WEBCRAWLERAPI_SET_FEED_STATEPause an active feed’s future scheduled runs or resume a paused feed. Pausing does not stop a run that has already started, and resuming schedules future crawls that may consume metered credits.

Input Parameters

string
required
Feed ID to pause or resume.
string ("paused" | "active")
required
Set paused to stop future schedules or active to resume scheduling.

Output

string
required
Data from the action execution
string
Error if any occurred during the execution of the action
boolean
required
Whether or not the action execution was successful or not