Python SDK

The Scrapybara Python library provides convenient access to the Scrapybara API from Python.

Installation

1
pip install scrapybara

Reference

Please only refer to this documentation site for reference. The GitHub reference here is generated programmatically and incomplete.

Requirements

• Python >= 3.8
• requests >= 2.25.1
• anthropic ^0.39.0
• pydantic ^2.0.0

License

This project is licensed under the MIT License - see the LICENSE file for details.

Usage

Instantiate and use the client with the following:

1
from scrapybara import Scrapybara
2
3
client = Scrapybara(
4
    api_key="YOUR_API_KEY",
5
)
6
client.start()
7
8
# Pause an instance to save resources
9
client.instance.pause(instance_id="your_instance_id")
10
11
# Resume a paused instance
12
client.instance.resume(
13
    instance_id="your_instance_id",
14
    timeout_hours=1  # Optional: specify how long the instance should run
15
)

Async Client

The SDK also exports an async client so that you can make non-blocking calls to our API.

1
import asyncio
2
3
from scrapybara import AsyncScrapybara
4
5
client = AsyncScrapybara(
6
    api_key="YOUR_API_KEY",
7
)
8
9
10
async def main() -> None:
11
    await client.start()
12
13
14
asyncio.run(main())

Exception Handling

When the API returns a non-success status code (4xx or 5xx response), a subclass of the following error will be thrown.

1
from scrapybara.core.api_error import ApiError
2
3
try:
4
    client.start(...)
5
except ApiError as e:
6
    print(e.status_code)
7
    print(e.body)

Advanced

Retries

The SDK is instrumented with automatic retries with exponential backoff. A request will be retried as long as the request is deemed retriable and the number of retry attempts has not grown larger than the configured retry limit (default: 2).

A request is deemed retriable when any of the following HTTP status codes is returned:

• 408 (Timeout)
• 429 (Too Many Requests)
• 5XX (Internal Server Errors)

Use the max_retries request option to configure this behavior.

1
client.start(..., request_options={
2
    "max_retries": 1
3
})

Timeouts

The SDK defaults to a 60 second timeout. You can configure this with a timeout option at the client or request level.

1
from scrapybara import Scrapybara
2
3
client = Scrapybara(
4
    ...,
5
    timeout=20.0,
6
)
7
8
9
# Override timeout for a specific method
10
client.start(..., request_options={
11
    "timeout_in_seconds": 1
12
})

Custom Client

You can override the httpx client to customize it for your use-case. Some common use-cases include support for proxies and transports.

1
import httpx
2
from scrapybara import Scrapybara
3
4
client = Scrapybara(
5
    ...,
6
    httpx_client=httpx.Client(
7
        proxies="http://my.test.proxy.example.com",
8
        transport=httpx.HTTPTransport(local_address="0.0.0.0"),
9
    ),
10
)

Contributing

While we value open-source contributions to this SDK, this library is generated programmatically. Additions made directly to this library would have to be moved over to our generation code, otherwise they would be overwritten upon the next generated release. Feel free to open a PR as a proof of concept, but know that we will not be able to merge it as-is. We suggest opening an issue first to discuss with us!

On the other hand, contributions to the README are always very welcome!