- Category: ai web scraping
- Auth: API_KEY
- Composio-managed OAuth available? N/A
- Tools: 10
- Triggers: 0
- Slug:
WEBCRAWLERAPI - Version: 20261030_00
Tools
Create Crawl
Create Crawl
Slug:
WEBCRAWLERAPI_CREATE_CRAWLStart a metered asynchronous crawl over a site and return its job ID. The crawl may visit and charge for up to items_limit pages; use GET_CRAWL_JOB to monitor it.Input Parameters
Maximum cached-result age in seconds. Defaults to allowing cached content up to seven days old; use 0 to disable cache reuse.
Maximum link depth; 0 crawls only the seed URL, while omission leaves the depth unrestricted.
Output
Create Feed
Create Feed
Delete Feed
Delete Feed
Get Crawl Job
Get Crawl Job
Get Feed
Get Feed
Get Organization Costs
Get Organization Costs
List Feed Changes
List Feed Changes
List Feeds
List Feeds
Scrape Page
Scrape Page
Slug:
WEBCRAWLERAPI_SCRAPE_PAGEScrape one web page synchronously and return requested content or structured extraction. This consumes metered credits; supplying a prompt adds $0.002 per request.Input Parameters
Maximum cached-result age in seconds. Defaults to allowing cached content up to seven days old; use 0 to force a fresh scrape.
Output
Set Feed State
Set Feed State
Slug:
WEBCRAWLERAPI_SET_FEED_STATEPause an active feed’s future scheduled runs or resume a paused feed. Pausing does not stop a run that has already started, and resuming schedules future crawls that may consume metered credits.Input Parameters
Set paused to stop future schedules or active to resume scheduling.