Apify Automation
Automate web scraping and data extraction with Apify -- run Actors, manage datasets, create reusable tasks, and retrieve crawl results through the Composio Apify integration.
Get the complete skill folder, including its scripts and reference files.
When to use this skill
Use Apify Automation when an AI agent needs a reusable SKILL.md workflow for this job: Automate web scraping and data extraction with Apify -- run Actors, manage datasets, create reusable tasks, and retrieve crawl results through the Composio Apify integration.
When not to use it
Skip Apify Automation when the task is outside the coding category, or when a more specific skill in this directory already covers the same workflow with clearer triggers.
How to install
Where will you use it?
mkdir -p ~/.claude/skills/apify-automation # Save SKILL.md to ~/.claude/skills/apify-automation/SKILL.md
- Claude Code personal install: place the skill and its bundled files at ~/.claude/skills/apify-automation/SKILL.md. For a shared project, commit the folder at .claude/skills/apify-automation/ instead.
- Current Claude Code detects SKILL.md changes in watched directories during the session. If you just created a top-level skills directory, run /reload-skills. Check that the skill is listed before testing its trigger.
- For Claude chat or Cowork in the Desktop app, manage uploaded skills in Customize > Skills; copying files into a local Claude Code folder is not a Desktop install. Other agents may use different install locations.
Full install guide for Claude, Cursor, and Codex →
Try a starter prompt
After installing the skill, paste this starter prompt into your agent to verify it triggers on a small task.
Use the Apify Automation skill for this task: Use Apify Automation when this documented workflow matches the task. Confirm any required inputs or files before making changes.
What supports this recommendation?
63/100 provisional content score. This measures documentation, practical detail and source signals. It is not a measured task success rate or a security certification.
- Documentation
- 67/100
- Practical detail
- 50/100
- Source evidence
- 63/100
- Source signals
- 80/100
Model compatibility evidence
- claudeinferred
The skill text mentions Claude or a closely associated term. This is keyword-derived compatibility; no hands-on model test is recorded.
- chatgptuntested
No model-specific signal or recorded compatibility test was found.
- geminiuntested
No model-specific signal or recorded compatibility test was found.
- copilotuntested
No model-specific signal or recorded compatibility test was found.
- llamauntested
No model-specific signal or recorded compatibility test was found.
- perplexityuntested
No model-specific signal or recorded compatibility test was found.
- mistraluntested
No model-specific signal or recorded compatibility test was found.
- grokuntested
No model-specific signal or recorded compatibility test was found.
Inferred compatibility is based on content signals. Check the app, its available tools and setup requirements before running a skill.
What this skill does
# Apify Automation
Run **Apify** web scraping Actors and manage datasets directly from Claude Code. Execute crawlers synchronously or asynchronously, retrieve structured data, create reusable tasks, and inspect run logs without leaving your terminal.
**Toolkit docs:** [composio.dev/toolkits/apify](https://composio.dev/toolkits/apify)
---
## Setup
1. Add the Composio MCP server to your configuration: ``` https://rube.app/mcp ``` 2. Connect your Apify account when prompted. The agent will provide an authentication link. 3. Browse available Actors at [apify.com/store](https://apify.com/store). Each Actor has its own unique input schema -- always check the Actor's documentation before running.
---
## Core Workflows
### 1. Run an Actor Synchronously and Get Results
Execute an Actor and immediately retrieve its dataset items in a single call. Best for quick scraping jobs.
**Tool:** `APIFY_RUN_ACTOR_SYNC_GET_DATASET_ITEMS`
Key parameters: - `actorId` (required) -- Actor ID in format `username/actor-name` (e.g., `compass/crawler-google-places`) - `input` -- JSON input object matching the Actor's schema. Each Actor has unique field names -- check [apify.com/store](https://apify.com/store) for the exact schema. - `limit` -- max items to return - `offset` -- skip items for pagination - `format` -- `json` (default), `csv`, `jsonl`, `html`, `xlsx`, `xml` - `timeout` -- run timeout in seconds - `waitForFinish` -- max wait time (0-300 seconds) - `fields` -- comma-separated list of fields to include - `omit` -- comma-separated list of fields to exclude
Example prompt: *"Run the Google Places scraper for 'restaurants in New York' and return the first 50 results"*
---
### 2. Run an Actor Asynchronously
Trigger an Actor run without waiting for completion. Use for long-running scraping jobs.
**Tool:** `APIFY_RUN_ACTOR`
Key parameters: - `actorId` (required) -- Actor slug or ID - `body` -- JSON input object for the Actor - `memory` -- memory limit in MB (must be power of 2, minimum 128) - `timeout` -- run timeout in seconds - `maxItems` -- cap on returned items - `build` -- specific build tag (e.g., `latest`, `beta`)
Follow up with `APIFY_GET_DATASET_ITEMS` to retrieve results using the run's `datasetId`.
Example prompt: *"Start the web scraper Actor for example.com asynchronously with 1024MB memory"*
---
### 3. Retrieve Dataset Items
Fetch data from a specific dataset with pagination, field selection, and filtering.
**Tool:** `APIFY_GET_DATASET_ITEMS`
Key parameters: - `datasetId` (required) -- dataset identifier - `limit` (default/max 1000) -- items per page - `offset` (default 0) -- pagination offset - `format` -- `json` (recommended), `csv`, `xlsx` - `fields` -- include only specific fields - `omit` -- exclude specific fields - `clean` -- remove Apify-specific metadata - `desc` -- reverse order (newest first)
Example prompt: *"Get the first 500 items from dataset myDatasetId in JSON format"*
---
### 4. Inspect Actor Details
View Actor metadata, input schema, and configuration before running it.
**Tool:** `APIFY_GET_ACTOR`
Key parameters: - `actorId` (required) -- Actor ID in format `username/actor-name` or hex ID
Example prompt: *"Show me the details and input schema for the apify/web-scraper Actor"*
---
### 5. Create Reusable Tasks
Configure reusable Actor tasks with preset inputs for recurring scraping jobs.
**Tool:** `APIFY_CREATE_TASK`
Configure a task once, then trigger it repeatedly with consistent input parameters. Useful for scheduled or recurring data collection workflows.
Example prompt: *"Create an Apify task for the Google Search scraper with default query 'AI startups' and US location"*
---
### 6. Manage Runs and Datasets
List Actor runs, browse datasets, and inspect run details for monitoring and debugging.
**Tools:** `APIFY_GET_LIST_OF_RUNS`, `APIFY_DATASETS_GET`, `APIFY_DATASET_GET`, `APIFY_GET_LOG`
For listing runs: - Filter by Actor and optionally by status - Get `datasetId` from run details for data retrieval
For dataset management: - `APIFY_DATASETS_GET` -- list all your datasets with pagination - `APIFY_DATASET_GET` -- get metadata for a specific dataset
For debugging: - `APIFY_GET_LOG` -- retrieve execution logs for a run or build
Example prompt: *"List the last 10 runs for the web scraper Actor and show logs for the most recent one"*
---
## Known Pitfalls
- **Actor input schemas vary wildly:** Every Actor has its own unique input fields. Generic field names like `queries` or `search_terms` will be rejected. Always check the Actor's page on [apify.com/store](https://apify.com/store) for exact field names (e.g., `searchStringsArray` for Google Maps, `startUrls` for web scrapers). - **URL format requirements:** Always include the full protocol (`https://` or `http://`) in URLs. Many Actors require URLs as objects with a `url` property: `{"startUrls": [{"url": "https://example.com"}]}`. - **Dataset pagination cap:** `APIFY_GET_DATASET_ITEMS` has a max `limit` of 1000 per call. For large datasets, loop with `offset` to collect all items. - **Enum values are lowercase:** Most Actors expect lowercase enum values (e.g., `relevance` not `RELEVANCE`, `all` not `ALL`). - **Sync timeout at 5 minutes:** `APIFY_RUN_ACTOR_SYNC_GET_DATASET_ITEMS` has a maximum `waitForFinish` of 300 seconds. For longer runs, use `APIFY_RUN_ACTOR` (async) and poll with `APIFY_GET_DATASET_ITEMS`. - **Data volume costs:** Large datasets can be expensive to fetch. Prefer moderate limits and incremental processing to avoid timeouts or memory pressure. - **JSON format recommended:** While CSV/XLSX formats are available, JSON is the most reliable for automated processing. Avoid CSV/XLSX for downstream automation.
---
## Quick Reference
| Tool Slug | Description | |---|---| | `APIFY_RUN_ACTOR_SYNC_GET_DATASET_ITEMS` | Run Actor synchronously and get results immediately | | `APIFY_RUN_ACTOR` | Run Actor asynchronously (trigger and return) | | `APIFY_RUN_ACTOR_SYNC` | Run Actor synchronously, return output record | | `APIFY_GET_ACTOR` | Get Actor metadata and input schema | | `APIFY_GET_DATASET_ITEMS` | Retrieve items from a dataset (paginated) | | `APIFY_DATASET_GET` | Get dataset metadata (item count, etc.) | | `APIFY_DATASETS_GET` | List all user datasets | | `APIFY_CREATE_TASK` | Create a reusable Actor task | | `APIFY_GET_TASK_INPUT` | Inspect a task's stored input | | `APIFY_GET_LIST_OF_RUNS` | List runs for an Actor | | `APIFY_GET_LOG` | Get execution logs for a run |
---
*Powered by [Composio](https://composio.dev)*
Intended uses
- Use Apify Automation when this documented workflow matches the task.
Implementation notes
- Review the source instructions and adapt inputs before running the workflow.
Try it, then tell us how it went
A copied prompt does not tell us whether the task worked. Report your result after checking the output. We collect your tool and outcome, never your files or prompt.