Skip to main content
WebScraping.AI logo
  • Category: ai web scraping
  • Auth: API_KEY
  • Composio-managed OAuth available? N/A
  • Tools: 7
  • Triggers: 0
  • Slug: WEBSCRAPING_AI
  • Version: 20261101_00

Tools

Slug: WEBSCRAPING_AI_ACCOUNT_INFOTool to retrieve account API call quota and usage. Use when checking remaining requests and subscription details.

Output

string
required
Data from the action execution
string
Error if any occurred during the execution of the action
boolean
required
Whether or not the action execution was successful or not
Slug: WEBSCRAPING_AI_ASK_QUESTIONGet an LLM answer to a question about a web page for AI-powered analysis or extraction. Page retrieval supports proxies and optional JavaScript rendering.

Input Parameters

boolean
Execute on-page JavaScript using a headless browser. Default: true.
string
required
URL of the target page to analyze.
string ("datacenter" | "residential")
Proxy type to use for the request.
string ("desktop" | "mobile" | "tablet")
Device emulation type. Default: desktop.
string ("json" | "text")
Response format. Default: json.
string
Proxy country code. Default: us.
object
HTTP headers to pass to the target page.
integer
Maximum web page retrieval time in milliseconds. Default: 10000, Max: 30000.
string
required
Question or instructions to ask the LLM model about the target page.
string
CSS selector to wait for before returning page content.
string
Custom JavaScript code to execute on the target page.
integer
Maximum JavaScript rendering time in milliseconds. Default: 2000, Max: 20000.
string
Custom proxy URL in http://user:password@host:port format.
boolean
Return error on 404 HTTP status on the target page. Default: false.
boolean
Return error on redirect on the target page. Default: false.

Output

string
required
Data from the action execution
string
Error if any occurred during the execution of the action
boolean
required
Whether or not the action execution was successful or not
Slug: WEBSCRAPING_AI_EXTRACT_FIELDSTool to extract structured data fields from a web page using AI. Returns extracted fields as JSON. Uses proxies and Chromium JavaScript rendering for page retrieval and processing.

Input Parameters

string
required
The target website URL to scrape and extract structured data from.
object
required
Object describing fields to extract from the page. Keys are field names, values are descriptions of what to extract. Example: {‘title’: ‘Main product title’, ‘price’: ‘Current product price’}
string
Optional parameter for handling unknown or additional configuration.

Output

string
required
Data from the action execution
string
Error if any occurred during the execution of the action
boolean
required
Whether or not the action execution was successful or not
Slug: WEBSCRAPING_AI_GET_RENDERED_HTMLTool to retrieve fully rendered HTML of a webpage. Use when JS-generated content must be included.

Input Parameters

string
Base64-encoded JavaScript to execute after rendering. Must be Base64-encoded; plain script strings are ignored.
string
required
The target URL to render and fetch HTML.
integer
Wait time before capture, in milliseconds. Increase for deferred JS execution or slow-loading pages to ensure complete content capture.
string ("desktop" | "mobile")
Browser device mode to simulate.
string
Browser locale (RFC5646 code).
string
Cookies in ‘key1=value1; key2=value2;’ format.
object
Extra HTTP headers as JSON object.
string
Referer header value.
integer
Request timeout, in milliseconds.
string ("datacenter" | "residential")
Proxy type to use for the request.
string
Custom User-Agent string.
boolean
Whether to disable image loading.

Output

string
required
Data from the action execution
string
Error if any occurred during the execution of the action
boolean
required
Whether or not the action execution was successful or not
Slug: WEBSCRAPING_AI_GET_SELECTED_HTMLTool to extract HTML from specific page elements using CSS selectors. Use when you need HTML from a particular section rather than the entire page.

Input Parameters

boolean
Execute JavaScript via headless browser. Default: true.
string
required
URL of the target page to scrape.
string ("datacenter" | "residential")
Proxy type options.
string ("desktop" | "mobile" | "tablet")
Device type options.
string
Proxy country code (2-letter ISO code). Default: us.
object
HTTP headers to pass to the target page as a JSON object.
integer
Maximum retrieval time in milliseconds. Default: 10000, max: 30000.
string
CSS selector to extract specific page elements. If not provided, returns the entire page HTML.
string
CSS selector to wait for before returning content.
string
Custom JavaScript code to execute on the page before extraction.
integer
JavaScript rendering time in milliseconds. Default: 2000, max: 20000.
string
Custom proxy URL to use instead of built-in proxies.
boolean
Return error when target page responds with 404 status. Default: false.
boolean
Return error when target page redirects. Default: false.

Output

string
required
Data from the action execution
string
Error if any occurred during the execution of the action
boolean
required
Whether or not the action execution was successful or not
Slug: WEBSCRAPING_AI_GET_SELECTED_MULTIPLEExtract HTML from multiple areas of a web page using its URL and CSS selectors, without parsing the page yourself. Optional settings control JavaScript rendering, retrieval timeouts, and the proxy type and country.

Input Parameters

boolean
Execute on-page JavaScript using a headless browser. True by default.
string
required
The target URL to scrape.
string ("datacenter" | "residential")
Proxy type options.
string
Country of the proxy to use. US by default.
integer
Maximum web page retrieval time in milliseconds. Default is 15000, maximum is 30000.
array
Multiple CSS selectors to extract. If null, returns whole page HTML.
integer
Maximum JavaScript rendering time in milliseconds. Default is 2000.

Output

string
required
Data from the action execution
string
Error if any occurred during the execution of the action
boolean
required
Whether or not the action execution was successful or not
Slug: WEBSCRAPING_AI_GET_TEXTRetrieve text content from a specified web page. Non-JSON responses are returned as raw text.

Input Parameters

string
required
The target URL to scrape text from.
string ("us" | "eu")
Proxy region to use for the request (e.g., ‘us’ or ‘eu’).
string
Browser locale/language (e.g., ‘en-US’).
string
Session ID for preserving cookies across multiple calls.
integer
Request timeout in seconds (must be >= 1).
boolean
Whether to render JavaScript on the page before extracting text.

Output

string
required
Data from the action execution
string
Error if any occurred during the execution of the action
boolean
required
Whether or not the action execution was successful or not