Skip to main content
Crawlbase logo
  • Category: web scraping & browser
  • Auth: API_KEY
  • Composio-managed OAuth available? N/A
  • Tools: 8
  • Triggers: 0
  • Slug: CRAWLBASE
  • Version: 20261030_00

Tools

Slug: CRAWLBASE_CRAWL_URLFetch a public HTTP or HTTPS URL through Crawlbase using the connected Normal token and return its content with separate target and Crawlbase statuses.

Input Parameters

string
required
Public HTTP or HTTPS URL to crawl.
boolean
Store the crawl result in Crawlbase Cloud Storage when true.
string ("desktop" | "tablet" | "mobile")
Device profile used for the request.
string
Optional cookie string to send to the target.
string
Optional country code for geolocated crawling.
string
Optional Crawlbase scraper name for structured extraction.
boolean
Ask Crawlbase to automatically parse supported pages.
string
Optional User-Agent value to send to the target.
boolean
Route through the Tor network when true.
string ("html" | "json" | "markdown")
Content representation to request. Markdown uses Crawlbase’s md format.
Optional Crawlbase cookie-session identifier, at most 32 characters.
object
Optional headers to send to the target URL.
array
Additional target HTTP status codes Crawlbase should treat as successful.

Output

string
required
Data from the action execution
string
Error if any occurred during the execution of the action
boolean
required
Whether or not the action execution was successful or not
Slug: CRAWLBASE_DELETE_STORED_PAGESIrreversibly delete explicitly selected pages from Crawlbase Cloud Storage by request ID and return each page’s deletion outcome.

Input Parameters

array
required
One to 100 unique request IDs of stored pages to delete irreversibly.

Output

string
required
Data from the action execution
string
Error if any occurred during the execution of the action
boolean
required
Whether or not the action execution was successful or not
Slug: CRAWLBASE_GENERATE_USER_AGENTSGenerate one to ten realistic randomized User-Agent strings for a selected device class.

Input Parameters

integer
Number of User-Agent strings to return, from 1 to 10.
string ("desktop" | "mobile")
Device class for generated User-Agent strings.

Output

string
required
Data from the action execution
string
Error if any occurred during the execution of the action
boolean
required
Whether or not the action execution was successful or not
Slug: CRAWLBASE_GET_CRAWLING_USAGEReturn current Crawling API usage and optionally previous-month statistics for the connected Crawlbase account.

Input Parameters

boolean
Include previous-month crawling usage statistics when available.

Output

string
required
Data from the action execution
string
Error if any occurred during the execution of the action
boolean
required
Whether or not the action execution was successful or not
Slug: CRAWLBASE_GET_STORAGE_COUNTReturn the number of pages currently held in Crawlbase Cloud Storage.

Output

string
required
Data from the action execution
string
Error if any occurred during the execution of the action
boolean
required
Whether or not the action execution was successful or not
Slug: CRAWLBASE_GET_STORED_PAGERetrieve the latest Crawlbase Cloud Storage page by exactly one request ID or original URL.

Input Parameters

string
Stored request ID. Provide exactly one of rid or url.
string
Original HTTP or HTTPS URL. Provide exactly one of url or rid.

Output

string
required
Data from the action execution
string
Error if any occurred during the execution of the action
boolean
required
Whether or not the action execution was successful or not
Slug: CRAWLBASE_GET_STORED_PAGESRetrieve up to 100 Crawlbase Cloud Storage pages by request ID without deleting them. Returns decoded page content and identifies requested IDs that were not found.

Input Parameters

array
required
One to 100 unique request IDs for stored pages to retrieve.

Output

string
required
Data from the action execution
string
Error if any occurred during the execution of the action
boolean
required
Whether or not the action execution was successful or not
Slug: CRAWLBASE_LIST_STORED_PAGE_IDSReturn one page of Crawlbase Cloud Storage request IDs and a short-lived continuation cursor.

Input Parameters

integer
Maximum number of stored request IDs to return, from 1 to 1000.
string ("asc" | "desc")
Order stored request IDs from oldest to newest (asc) or newest to oldest (desc).
string
Short-lived cursor from the previous response. Omit for the first page and use a returned cursor within about 15 seconds.

Output

string
required
Data from the action execution
string
Error if any occurred during the execution of the action
boolean
required
Whether or not the action execution was successful or not