How it works

To scrape a webpage with AI-Scraper, follow these steps:

1. Provide the webpage URL you want to scrape.
2. Describe the data to extract in natural language (e.g. “Get all product names and prices”).
3. Select the output format – structured JSON or Markdown.
4. (Optional) Define a schema – Let AI-Scraper generate one automatically, or provide your own OpenAPI schema for the exact structure you desire.

Installation

To begin, make sure you have access to an AI Studio API key (or get a free trial with 1000 credits) and Python ver. 3.10 or above installed. You can install the oxylabs-ai-studio package using pip:

pip install oxylabs-ai-studio

Code examples (Python)

The following examples show how to use AiScraper to extract data from a sample page.

from oxylabs_ai_studio.apps.ai_scraper import AiScraper
import json

# Initialize the AI Scraper with your API key
scraper = AiScraper(api_key="YOUR_API_KEY")

# Generate a schema automatically from natural language
schema = scraper.generate_schema(prompt="want to parse developer, platform, type, price game title, and genre (array)")
print(f"Generated schema: {schema}")

# Scrape a webpage and extract structured data
url = "https://sandbox.oxylabs.io/products/3"
result = scraper.scrape(
    url=url,
    output_format="json",
    schema=schema,
    render_javascript=False,
    geo_location="US",
)
# Print the scrape output as JSON
print("Results:")
print(json.dumps(result.data, indent=2))

Learn more about AI-Scraper and Oxylabs AI Studio Python SDK in our PyPI repository. You can also check out our AI Studio JavaScript SDK guide for JS users.

Request parameters

| Parameter | Description | Default Value | |---------------------|----------------------------------------------------------------|---------------| | url* | Target URL to scrape | – | | output_format | Output format (json, markdown) | markdown | | schema | OpenAPI schema for structured extraction (mandatory for JSON) | – | | render_javascript | Enable render JavaScript | False | | geo_location | Proxy location in ISO2 format | – |

* – mandatory parameters

Output samples

The AI-Scraper can return parsed, ready-to-use output that is easy to integrate into your applications.

This is a structured JSON of the response output:

Results:
{
  "games": [
    {
      "developer": "Nintendo EAD Tokyo",
      "platform": "wii",
      "type": "singleplayer",
      "price": 91.99,
      "title": "Super Mario Galaxy 2",
      "genre": [
        "Action",
        "Platformer"
      ]
    },
    {
      "developer": "Eidos Interactive",
      "platform": "wii",
      "type": null,
      "price": 80.99,
      "title": "Death Jr.: Root of Evil",
      "genre": [
        "Action",
        "Platformer",
        "3D"
      ]
    }...

Alternatively, you can use output_format=”markdown” to receive Markdown results instead of parsed JSON.

Practical use cases

Oxylabs AI-Scraper can be applied to a wide variety of data collection tasks:

1. Extract product details – Gather product names, descriptions, and prices from e-commerce sites.
2. Parse news articles – Retrieve article titles, dates, authors, and body text.
3. Scrape pricing pages – Collect structured pricing information for competitor or market research.
4. Extract job postings – Capture job titles, locations, salaries, and posting dates from recruitment portals.

FAQ

How does AI-Scraper differ from normal scrapers?

AI-Scraper doesn’t rely on CSS/XPath selectors or custom parsing logic. Instead, it uses natural language prompts and AI-powered extraction, making it more adaptable to layout changes and much easier to set up.

Can I scrape any webpage?

Yes, you can scrape any public webpage as long as the page is publicly accessible. AI-Scraper also supports JavaScript rendering for dynamic pages. Private or login-protected content isn’t supported out of the box.

Is schema mandatory for AI-Scraper?

No, schema is not mandatory, but it’s required if you want structured JSON output. If you don’t provide one, AI-Scraper can generate a schema automatically based on your prompt.

What happens if the page structure changes?

Unlike traditional scrapers, AI-Scraper is more resilient to layout changes because it interprets content with AI. However, major changes may require you to adjust either your prompt or the schema.

Is AI-Scraper free?

Oxylabs AI Studio AI-Scraper is free to try by signing up for a free trial that includes 1,000 credits. After the trial, the monthly plans start at just $12/month with 3000 credits and 1 request/s, with higher plans offering more credits and higher request rates.

Learn more

For a deeper dive into available parameters, advanced integrations, and additional examples, check out the AI Studio documentation.

Contact us

If you have questions or need support, reach out to us at support@oxylabs.io, or through live chat, accessible via Oxylabs Dashboard, or join our Discord community. For enterprise-related inquiries, contact your dedicated account manager.