Skip to main content
Scrapfly logo
  • Category: web scraping & browser
  • Auth: API_KEY
  • Composio-managed OAuth available? N/A
  • Tools: 12
  • Triggers: 0
  • Slug: SCRAPFLY
  • Version: 20261101_00

Tools

Slug: SCRAPFLY_CAPTURE_SCREENSHOTCapture a full-page, viewport, or selected-element screenshot of a website, with options such as custom resolution and dark mode. Returns a downloadable screenshot image.

Input Parameters

string
JavaScript code to execute on the page before capturing screenshot.
string
required
Target URL to capture a screenshot of.
boolean
Enable caching of the screenshot result.
string ("jpg" | "png" | "webp" | "gif")
Screenshot image format (jpg, png, webp, or gif). Default is jpg.
string
Area to capture: ‘viewport’ for visible area, ‘fullpage’ for entire page, or a CSS/XPath selector for a specific element. Default is viewport.
string
Proxy geolocation country code (e.g., ‘US’, ‘FR’, ‘DE’) to capture screenshot from that region.
string
Comma-separated screenshot options: ‘dark_mode’ for dark theme, ‘block_banners’ to hide cookie/privacy banners, ‘print_media_format’ for print styles.
integer
Request timeout in milliseconds. Default is 30000 (30 seconds).
integer
Cache time-to-live in seconds. Only applies when cache is enabled.
string
Screen resolution in WIDTHxHEIGHT format (e.g., ‘1920x1080’). Default is 1920x1080.
boolean
Automatically scroll down the page before capturing. Useful for lazy-loaded content.
boolean
Clear and refresh cache for this request.
integer
Time to wait after page load before capturing screenshot, in milliseconds. Useful for waiting for dynamic content.
string
CSS selector or XPath to wait for before capturing. Screenshot is taken after this element appears.

Output

string
required
Data from the action execution
string
Error if any occurred during the execution of the action
boolean
required
Whether or not the action execution was successful or not
Slug: SCRAPFLY_CAPTURE_SCREENSHOT_HEADTool to capture screenshot metadata without downloading the image body. Use this for async screenshot workflows where you need the URL to retrieve the image later. Returns the screenshot URL in response, saving bandwidth compared to full screenshot retrieval.

Input Parameters

string
Base64-encoded JavaScript to execute on the page before screenshot (max 16KB).
string
required
Target URL to capture screenshot of.
boolean
Enable caching for repeated requests. Default is false.
string ("jpg" | "png" | "webp" | "gif")
Image format options for screenshot.
string
Capture area: ‘viewport’ for visible area, ‘fullpage’ for entire page, or a CSS selector/XPath for specific element.
string
Proxy location using ISO 3166 alpha-2 country code (e.g., ‘US’, ‘FR’, ‘DE’). Default is ‘us’.
string
Comma-separated flags: ‘dark_mode’ for dark theme, ‘block_banners’ to hide ads/popups, ‘print_media_format’ for print styles.
integer
Maximum request time in milliseconds (range: 60000-120000). Default is 60000.
integer
Cache duration in seconds. Default is 86400 (24 hours).
string
Screen dimensions as widthxheight (e.g., ‘1920x1080’). Default is 1920x1080.
boolean
Auto-scroll to bottom of page to trigger lazy-loaded content. Default is false.
boolean
Force cache refresh by clearing existing cached result. Default is false.
integer
Delay after page load before capturing screenshot, in milliseconds. Default is 1000.
string ("none" | "deuteranopia" | "protanopia" | "tritanopia" | "achromatopsia" | "blurredVision")
Accessibility simulation options for vision deficiency.
string
CSS selector or XPath to wait for before capturing screenshot.

Output

string
required
Data from the action execution
string
Error if any occurred during the execution of the action
boolean
required
Whether or not the action execution was successful or not
Slug: SCRAPFLY_CREATE_CRAWLERCreate a web crawler to recursively crawl pages from a starting URL, with configurable limits and content formats. Returns a crawler UUID for tracking progress.

Input Parameters

boolean
Enable Anti Scraping Protection bypass
string
required
Starting URL for the crawl. Must be a valid HTTP/HTTPS URL
integer
Delay between requests in milliseconds. Range: 0-15000ms
integer
Maximum crawl depth - controls how many levels deep the crawler will follow links
boolean
Enable JavaScript rendering
integer
Maximum number of pages to crawl. Set to 0 for unlimited (subject to subscription limits)
integer
Maximum concurrent scrape requests. Set 0 for account default
number
Automatically stop the crawl when reaching a credit limit
array
Exclude URLs matching these patterns. Mutually exclusive with include_only_paths
array
Array of desired output formats (markdown, extracted_data, page_metadata, etc.)
array
Only crawl URLs matching these patterns. Supports wildcards (*). Max 100 paths
Allow crawler to follow links to external domains

Output

string
required
Data from the action execution
string
Error if any occurred during the execution of the action
boolean
required
Whether or not the action execution was successful or not
Slug: SCRAPFLY_EXTRACT_DATATool to extract structured data from HTML or other content using AI models, LLM prompts, or custom templates. Use when you need to parse web pages or documents into structured JSON data. Supports predefined extraction models for common types (articles, products, events) or custom extraction via prompts/templates.

Input Parameters

string
Base URL for converting relative URLs to absolute URLs in the extracted data. Useful when extracting links and images.
string
Document charset encoding. Use ‘auto’ for automatic detection or specify a charset like ‘utf-8’, ‘iso-8859-1’, etc.
string
required
HTML, text, or structured content to extract data from. This will be sent as the request body.
string ("text/html" | "text/plain" | "text/markdown" | "text/csv" | "application/json" | "application/ld+json" | "application/xml" | "application/xhtml+xml")
required
Content type of the document body. Required to properly parse the input content.
string
Webhook name for asynchronous processing. If provided, the extraction will be processed asynchronously and results sent to the webhook.
string ("article" | "event" | "food_recipe" | "hotel" | "hotel_listing" | "job_listing" | "job_posting" | "organization" | "product" | "product_listing" | "real_estate_property" | "real_estate_property_listing" | "review_list" | "search_engine_results" | "social_media_post" | "software" | "stock" | "vehicle_ad" | "vehicle_ad_listing")
AI extraction models for structured data extraction.
string
Custom LLM prompt for extraction. Use this when you need custom extraction logic beyond predefined models. Cannot be used together with extraction_model or extraction_template.
string
JSON extraction template defining custom extraction rules. Cannot be used together with extraction_model or extraction_prompt.

Output

string
required
Data from the action execution
string
Error if any occurred during the execution of the action
boolean
required
Whether or not the action execution was successful or not
Slug: SCRAPFLY_GET_ACCOUNT_INFORetrieve Scrapfly account information, including subscription, usage, billing, and project settings. Use it to review account status and usage limits.

Output

string
required
Data from the action execution
string
Error if any occurred during the execution of the action
boolean
required
Whether or not the action execution was successful or not
Slug: SCRAPFLY_GET_CRAWLER_ARTIFACTTool to download crawler artifact files in WARC or HAR format. Use when you need to retrieve the complete crawl results as an archive file. WARC format is recommended for large crawls as it includes gzip compression.

Input Parameters

string ("warc" | "har")
required
The artifact format type. WARC format is recommended for large crawls with gzip compression.
string
required
The unique identifier (UUID) of the crawler whose artifact you want to download.

Output

string
required
Data from the action execution
string
Error if any occurred during the execution of the action
boolean
required
Whether or not the action execution was successful or not
Slug: SCRAPFLY_GET_CRAWLER_CONTENTSTool to retrieve extracted content from crawled pages. Supports multiple output formats including markdown, text, HTML, and JSON. Use when you need to access the actual content extracted during a crawl, with optional filtering by URL and format selection.

Input Parameters

string
Specific URL to query for single-URL retrieval. When specified, returns content only for this URL.
integer
Number of items per page (default: 10, max: 50).
boolean
Return raw content without JSON wrapper. Only works with single URL queries (when ‘url’ parameter is specified).
integer
Number of results to skip for pagination (default: 0).
string
Comma-separated list of content formats to retrieve. Available formats: html (raw HTML), clean_html (HTML with boilerplate removed), markdown (LLM-optimized markdown), text (plain text only), json (structured JSON), extracted_data (AI-extracted structured data), page_metadata (page metadata like title, description).
string
required
The unique identifier (UUID) of the crawler whose contents you want to retrieve.

Output

string
required
Data from the action execution
string
Error if any occurred during the execution of the action
boolean
required
Whether or not the action execution was successful or not
Slug: SCRAPFLY_GET_CRAWLER_STATUSTool to get the current status of a crawler including progress, pages crawled, and completion state. Use for polling workflow to monitor crawl progress.

Input Parameters

string
required
The unique identifier (UUID) of the crawler job to check status for.

Output

string
required
Data from the action execution
string
Error if any occurred during the execution of the action
boolean
required
Whether or not the action execution was successful or not
Slug: SCRAPFLY_GET_CRAWLER_URLSRetrieve discovered and crawled URLs from a crawler, optionally filtered by status. Supports pagination for large result sets.

Input Parameters

integer
Page number for pagination (default: 1).
string
Filter results by status. Use ‘visited’ for successfully crawled URLs or ‘failed’ to get URLs that failed with error codes.
integer
Number of URLs to return per page (default: 100, max: 1000).
string
required
The unique identifier (UUID) of the crawler whose URLs you want to retrieve.

Output

string
required
Data from the action execution
string
Error if any occurred during the execution of the action
boolean
required
Whether or not the action execution was successful or not
Slug: SCRAPFLY_SCRAPERequest a web scrape with options such as JavaScript rendering, proxy location, and extraction. The action requires a Bearer API key in metadata headers and reads results from the response’s data wrapper.

Input Parameters

boolean
Enable anti-scraping protection bypass.
string
required
The URL to scrape.
string
Request body payload for POST/PUT requests.
array
Custom labels for analytics.
boolean
Enable result caching.
integer
Number of retries on failure (default is 2).
string ("GET" | "POST" | "PUT" | "PATCH" | "DELETE" | "HEAD" | "OPTIONS")
HTTP method to use for request, default is GET.
string
Target country for the proxy (e.g., ‘US’, ‘FR’, ‘DE’). Matching the target site’s region can improve success rates against geo-restricted or anti-bot measures.
object
Custom HTTP request headers to send.
string
Proxy session ID for sticky sessions.
integer
Timeout for the scrape request in seconds (default is 60).
boolean
Enable JavaScript rendering for sites requiring it. Default returns static HTML only; dynamic/SPA content requires this set to true.
object
Extraction rules (JMESPath, CSS selector, or JSONPath).

Output

string
required
Data from the action execution
string
Error if any occurred during the execution of the action
boolean
required
Whether or not the action execution was successful or not
Slug: SCRAPFLY_SCRAPE_POSTTool to scrape web pages using POST method to send data in the request body. Use when you need to scrape endpoints that require POST requests, such as form submissions or APIs that expect data payload.

Input Parameters

boolean
Enable anti-scraping protection bypass.
string
required
The target URL to scrape with POST request.
object
required
POST request body payload as JSON object. This data will be sent in the request body to the target URL.
array
Custom labels for analytics and monitoring.
boolean
Enable result caching to avoid repeated requests.
integer
Number of retries on failure (default is 2).
string
Target country for the proxy using ISO 3166-1 alpha-2 codes (e.g., ‘US’, ‘FR’, ‘DE’).
object
Custom HTTP request headers to send to the target URL. Note: Content-Type defaults to application/json for POST requests.
string
Proxy session ID for sticky sessions and persistent cookies.
integer
Timeout for the scrape request in seconds (default is 60).
boolean
Enable JavaScript rendering for sites requiring browser execution.
object
Extraction rules using JMESPath, CSS selector, or JSONPath for structured data extraction.

Output

string
required
Data from the action execution
string
Error if any occurred during the execution of the action
boolean
required
Whether or not the action execution was successful or not
Slug: SCRAPFLY_SCRAPE_WITH_PUTTool to scrape web pages using PUT method with body payload. Use when the target API requires PUT requests with data in the request body. Forwards PUT request with custom body to the target URL. If not specified, content-type defaults to application/x-www-form-urlencoded.

Input Parameters

boolean
Enable anti-scraping protection bypass.
string
required
The target URL to scrape with PUT request.
string
required
Request body payload to send with the PUT request. Should be JSON string for JSON payloads.
array
Custom labels for analytics.
boolean
Enable result caching.
integer
Number of retries on failure (default is 2).
string
Target country for the proxy (e.g., ‘US’, ‘FR’, ‘DE’).
object
Custom HTTP request headers to send. For JSON body, include ‘content-type’: ‘application/json’.
string
Proxy session ID for sticky sessions.
integer
Timeout for the scrape request in seconds (default is 60).
boolean
Enable JavaScript rendering for sites requiring it.
object
Extraction rules (JMESPath, CSS selector, or JSONPath).

Output

string
required
Data from the action execution
string
Error if any occurred during the execution of the action
boolean
required
Whether or not the action execution was successful or not