Skip to main content

Scrape Endpoint

Note: Easy Scrape API goes beyond basic scraping! You can truly interact with web pages—click buttons, fill forms, navigate, and extract dynamic content that loads after user actions. Most scrapers only capture the initial page load, but here you can automate real browser workflows for advanced use cases.

The main endpoint for executing web scraping scripts with Puppeteer and Cheerio.

Endpoint Details​

POST /api/scrape

Execute custom JavaScript code in a browser environment with full access to Puppeteer v24 and Cheerio libraries.

Request Body​

Required Parameters​

ParameterTypeDescription
urlstringTarget URL to scrape (must be valid HTTP/HTTPS URL)

Optional Parameters​

ParameterTypeDefaultDescription
scriptstringnullJavaScript code to execute (for simple scripts)
scriptBase64stringnullBase64-encoded JavaScript code (for complex scripts)
outputFormatstring"json"Response format: "json" or "html"
Script Parameter Options

You can provide your JavaScript code in one of two ways:

Option 1: script parameter (recommended for simple scripts)

  • Use for straightforward scripts without complex quotes or formatting
  • Easier to read and debug during development
  • Works well with most HTTP clients

Option 2: scriptBase64 parameter (recommended for complex scripts)

  • Use when your script contains multiple quotes (" and '), newlines, or special characters
  • Prevents JSON parsing errors and script corruption
  • Essential for scripts with complex string manipulations or multi-line templates
  • Required when using certain no-code platforms that struggle with escaped characters
tip

Use our Online Code Parser Tool to convert your JavaScript code to either escaped strings or Base64 format depending on your needs.

When to Use scriptBase64

Use scriptBase64 instead of script when your code contains:

  • Multiple nested quotes: console.log("He said 'Hello' to me")
  • Template literals with backticks: const template = `<div>${data}</div>`
  • Regular expressions with special characters: /[^a-zA-Z0-9]/g
  • Multi-line strings or complex formatting
  • Code that breaks when escaped for JSON

Note: You cannot use both script and scriptBase64 in the same request. Choose the appropriate parameter for your use case.

Response Formats​

JSON Success Response​

note

The following response fields are only included when outputFormat is set to "json".

FieldTypeDescription
messagestringStatus message about script execution
dataobjectThe returned data from your script
executionTimenumberTime taken to execute the script (milliseconds)
logsstring[]Array of debug messages from console.log()
warning

The examples below are shown in JavaScript format for readability. When sending a script to the Easy Scrape API, the format requirements depend on your HTTP client:

  • Most HTTP clients (Postman, cURL, fetch, etc.): Provide the script as a string inside the script field of your JSON request
  • Some platforms (certain no-code tools): May require the JavaScript code unescaped directly in the field

If you need help converting your JavaScript code to a properly escaped string, check out our Online Code Parser Tool to easily format your code for the API.

tip

Remember that your script always returns something. If you don't explicitly return a value, your script will return undefined. Always include a return statement with meaningful data.

{
  "message": "Script executed successfully",
  "data": {
    "title": "Example Domain",
    "products": [
      {
        "name": "Product 1",
        "price": "$29.99",
        "image": "https://example.com/img1.jpg"
      }
    ]
  },
  "executionTime": 2450,
  "logs": [
    "Starting extraction...",
    "Found 5 products",
    "Extraction complete"
  ]
}

HTML Response​

When outputFormat is set to "html", the API returns raw HTML:

<html>
  <head><title>Scraped Results</title></head>
  <body>
    <h1>Example Domain</h1>
    <p>Scraped content here...</p>
  </body>
</html>

Error Responses​

400 - Bad Request​

Missing URL​

{
  "error": "Missing url",
  "code": "MISSING_REQUIRED_FIELD",
  "statusCode": 400
}

Script Execution Failed​

{
  "error": "Script execution failed: TypeError: Cannot read property 'textContent' of null",
  "code": "SCRAPING_FAILED", 
  "statusCode": 400
}

Invalid URL​

{
  "error": "Invalid URL format",
  "code": "INVALID_URL_FORMAT",
  "statusCode": 400
}

401 - Unauthorized​

Missing API Key​

{
  "error": "API key is required",
  "code": "MISSING_API_KEY",
  "statusCode": 401
}

Invalid API Key​

{
  "error": "Invalid API key",
  "code": "INVALID_API_KEY",
  "statusCode": 401
}

429 - Rate Limited​

{
  "error": "Rate limit exceeded",
  "code": "RATE_LIMIT_EXCEEDED",
  "statusCode": 429
}

500 - Server Error​

{
  "error": "Internal server error",
  "code": "INTERNAL_SERVER_ERROR",
  "statusCode": 500
}

Script Guidelines​

Available Objects​

Limitations​

  • Memory: Reasonable limits for typical scraping tasks
  • Network: No access to localhost or internal networks
  • File System: No file system access
  • Security: Malicious activities monitored and blocked

Testing Tips​

  1. Start simple: Test with basic URL validation first
  2. Use console.log: Debug your scripts with logging
  3. Test incrementally: Build your script step by step
  4. Handle errors gracefully: Always return useful error information

Common Use Cases​

  • Price Monitoring: Track product prices and availability
  • Content Aggregation: Collect articles, posts, or listings
  • Market Research: Gather competitive intelligence
  • Data Migration: Extract data from old systems
  • Monitoring: Track changes on websites
  • Lead Generation: Collect contact information from directories