Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 10 additions & 10 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

## Overview

This is a standalone OpenClaw plugin (`@apify/apify-openclaw-integration`) that provides web scraping and data extraction via Apify's API. It registers **1 agent tool** (`apify_scraper`) — a universal scraper with 3 actions: `discover`, `start`, and `collect`.
This is a standalone OpenClaw plugin (`@apify/apify-openclaw-plugin`) that provides web scraping and data extraction via Apify's API. It registers **1 agent tool** (`apify`) — a universal scraper with 3 actions: `discover`, `start`, and `collect`.

- **Upstream repo:** https://github.com/openclaw/openclaw
- **Plugin docs:** https://docs.openclaw.ai/plugins/community
Expand All @@ -15,7 +15,7 @@ This is a standalone OpenClaw plugin (`@apify/apify-openclaw-integration`) that

```
src/
index.ts # Plugin entry point — registers apify_scraper + CLI
index.ts # Plugin entry point — registers apify + CLI
apify-client.ts # Shared Apify client factory, config helpers
cli.ts # openclaw apify setup|status|test commands
util.ts # Inlined utilities (not exported by openclaw/plugin-sdk)
Expand All @@ -28,7 +28,7 @@ openclaw.plugin.json # Plugin manifest (configSchema + uiHints) — REQ
package.json # npm package config
```

## The `apify_scraper` Tool
## The `apify` Tool

Single tool with 3 actions:

Expand Down Expand Up @@ -57,7 +57,7 @@ The tool description includes instructions for the agent:

## Key Architecture Decisions

- **Single tool, multiple actions:** All scraping goes through `apify_scraper` with `discover`/`start`/`collect` actions.
- **Single tool, multiple actions:** All scraping goes through `apify` with `discover`/`start`/`collect` actions.
- **Async two-phase pattern:** `start` returns immediately with run references. `collect` polls and fetches results. The agent does other work between calls.
- **`apify-client` SDK:** Uses the official `apify-client` npm package (not raw HTTP). Client created via `createApifyClient(apiKey, baseUrl)`.
- **Inlined utilities (`util.ts`):** `ToolInputError`, cache helpers, and `wrapExternalContent` are NOT exported from `openclaw/plugin-sdk`. We carry local copies.
Expand Down Expand Up @@ -92,8 +92,8 @@ The wizard merges safely: preserves existing config, adds to `tools.alsoAllow` w
## Coding Style

- TypeScript (ESM). Prefer strict typing; avoid `any`.
- Tool names: `snake_case` (e.g., `apify_scraper`).
- Plugin id / config keys: `kebab-case` (e.g., `apify-openclaw-integration`).
- Tool names: `snake_case` (e.g., `apify`).
- Plugin id / config keys: `kebab-case` (e.g., `apify`).
- Keep files concise. Add comments for non-obvious logic.
- Tool schema guardrails: avoid `Type.Union`. Use `stringEnum` for string enums, `Type.Optional(...)` instead of nullable types.

Expand All @@ -120,7 +120,7 @@ OpenClaw scans for plugins in strict precedence order:

For npm-installed plugins: `openclaw plugins install <npm-spec>` runs `npm pack`, extracts the tarball into `~/.openclaw/extensions/<id>/`, and runs `npm install --ignore-scripts` for dependencies.

The plugin id is derived from the **unscoped** npm package name. For `@apify/apify-openclaw-integration`, the id = `apify-openclaw-integration`.
The plugin id is derived from the **unscoped** npm package name. For `@apify/apify-openclaw-plugin`, the id = `apify`.

#### 2. Manifest Loading

Expand All @@ -142,7 +142,7 @@ OpenClaw uses [Jiti](https://github.com/unjs/jiti) to import the plugin entry fi

#### 5. Registration

Our plugin calls `api.registerTool(tool)` for `apify_scraper` and `api.registerCli(...)` for the `apify` CLI subcommand.
Our plugin calls `api.registerTool(tool)` for `apify` and `api.registerCli(...)` for the `apify` CLI subcommand.

#### 6. Tool Resolution at Runtime

Expand All @@ -155,7 +155,7 @@ Tool names that collide with core tool names are silently dropped. Plugin tools
plugins: {
enabled: true,
entries: {
"apify-openclaw-integration": {
"apify": {
enabled: true,
config: {
apiKey: "apify_api_...", // or use APIFY_API_KEY env var
Expand All @@ -168,7 +168,7 @@ Tool names that collide with core tool names are silently dropped. Plugin tools
},
},
tools: {
alsoAllow: ["group:plugins"], // or "apify-openclaw-integration" or "apify_scraper"
alsoAllow: ["group:plugins"], // or "apify" or "apify"
},
}
```
Expand Down
24 changes: 12 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,14 +5,14 @@ Universal web scraping and data extraction via [Apify](https://apify.com) — 57
## Install

```bash
openclaw plugins install @apify/apify-openclaw-integration
openclaw plugins install @apify/apify-openclaw-plugin
```

Restart the Gateway after installation.

## How it works

The plugin registers a single tool — `apify_scraper` — with three actions:
The plugin registers a single tool — `apify` — with three actions:

| Action | Purpose |
|--------|---------|
Expand All @@ -35,7 +35,7 @@ The tool uses a **two-phase async pattern**: `start` fires off a run and returns
{
plugins: {
entries: {
"apify-openclaw-integration": {
"apify": {
config: {
apiKey: "apify_api_...", // optional if APIFY_API_KEY env var is set
baseUrl: "https://api.apify.com",
Expand All @@ -47,7 +47,7 @@ The tool uses a **two-phase async pattern**: `start` fires off a run and returns
},
// Make the tool available to agents:
tools: {
alsoAllow: ["apify_scraper"], // or "apify-openclaw-integration" or "group:plugins"
alsoAllow: ["apify"], // or "apify" or "group:plugins"
},
}
```
Expand All @@ -58,7 +58,7 @@ Or use the interactive setup wizard:
openclaw apify setup
```

## apify_scraper
## apify

### Workflow

Expand Down Expand Up @@ -94,19 +94,19 @@ Most Actors accept arrays of URLs/queries in their input (e.g., `startUrls`, `qu

```javascript
// 1. Search the Apify Store
const search = await apify_scraper({
const search = await apify({
action: "discover",
query: "linkedin company scraper",
});

// 2. Get an Actor's input schema
const schema = await apify_scraper({
const schema = await apify({
action: "discover",
actorId: "compass~crawler-google-places",
});

// 3. Start a Google Search scrape
const started = await apify_scraper({
const started = await apify({
action: "start",
actorId: "apify~google-search-scraper",
input: { queries: ["OpenAI", "Anthropic"], maxPagesPerQuery: 1 },
Expand All @@ -115,21 +115,21 @@ const started = await apify_scraper({
// -> { runs: [{ runId, actorId, datasetId, status }] }

// 4. Collect results
const results = await apify_scraper({
const results = await apify({
action: "collect",
runs: started.runs,
});
// -> { completed: [...], pending: [...] }

// Instagram profile scraping
await apify_scraper({
await apify({
action: "start",
actorId: "apify~instagram-profile-scraper",
input: { usernames: ["natgeo", "nasa"] },
});

// TikTok search
await apify_scraper({
await apify({
action: "start",
actorId: "clockworks~tiktok-scraper",
input: { searchQueries: ["AI tools"], resultsPerPage: 20 },
Expand All @@ -138,7 +138,7 @@ await apify_scraper({

### Sub-agent delegation

The tool description instructs agents to delegate `apify_scraper` calls to a sub-agent. The sub-agent handles the full discover → start → collect workflow and returns only the relevant extracted data — not raw API responses or run metadata.
The tool description instructs agents to delegate `apify` calls to a sub-agent. The sub-agent handles the full discover → start → collect workflow and returns only the relevant extracted data — not raw API responses or run metadata.

## Security

Expand Down
4 changes: 2 additions & 2 deletions openclaw.plugin.json
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"id": "apify-openclaw-integration",
"id": "apify",
"name": "Apify",
"description": "Universal web scraping and data extraction via Apify — scrape any platform using 57+ Actors across social media, maps, search, e-commerce, and more.",
"configSchema": {
Expand Down Expand Up @@ -28,7 +28,7 @@
"type": "array",
"items": {
"type": "string",
"enum": ["apify_scraper"]
"enum": ["apify"]
},
"description": "Which tools to register (default: all tools enabled)."
}
Expand Down
8 changes: 4 additions & 4 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

10 changes: 5 additions & 5 deletions package.json
Original file line number Diff line number Diff line change
@@ -1,18 +1,18 @@
{
"name": "@apify/apify-openclaw-integration",
"version": "0.2.0",
"name": "@apify/apify-openclaw-plugin",
"version": "0.1.0",
"description": "Web scraping and AI-powered data extraction via Apify for OpenClaw — market research, competitor intelligence, trend analysis, lead generation, e-commerce, social media analytics, and more.",
"type": "module",
"license": "MIT",
"author": "Apify Technologies s.r.o.",
"repository": {
"type": "git",
"url": "git+https://github.com/apify/apify-openclaw-integration.git"
"url": "git+https://github.com/apify/apify-openclaw-plugin.git"
},
"bugs": {
"url": "https://github.com/apify/apify-openclaw-integration/issues"
"url": "https://github.com/apify/apify-openclaw-plugin/issues"
},
"homepage": "https://github.com/apify/apify-openclaw-integration#readme",
"homepage": "https://github.com/apify/apify-openclaw-plugin#readme",
"keywords": [
"openclaw",
"apify",
Expand Down
8 changes: 4 additions & 4 deletions src/cli.ts
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ function getBaseUrl(api: OpenClawPluginApi): string {
}

const ALL_TOOLS: { name: string; desc: string }[] = [
{ name: "apify_scraper", desc: "Universal scraper — any Apify Actor (57+ Actors)" },
{ name: "apify", desc: "Universal scraper — any Apify Actor (15k+ Actors)" },
];

// ---------------------------------------------------------------------------
Expand All @@ -97,12 +97,12 @@ async function applyConfigChanges(
// Merge plugin entry
if (!cfg.plugins) cfg.plugins = {};
if (!cfg.plugins.entries) cfg.plugins.entries = {};
const existing = cfg.plugins.entries["apify-openclaw-integration"] ?? {};
const existing = cfg.plugins.entries["apify"] ?? {};
const existingPluginConfig =
typeof existing.config === "object" && existing.config !== null
? (existing.config as Record<string, unknown>)
: {};
cfg.plugins.entries["apify-openclaw-integration"] = {
cfg.plugins.entries["apify"] = {
...existing,
enabled: true,
config: {
Expand Down Expand Up @@ -135,7 +135,7 @@ function printManualConfig(apiKey: string, selectedTools: string[], allSelected:
console.log(" Add this to your OpenClaw config:\n");
console.log(" plugins:");
console.log(" entries:");
console.log(" apify-openclaw-integration:");
console.log(" apify:");
console.log(" enabled: true");
console.log(" config:");
console.log(` apiKey: "${apiKey}"`);
Expand Down
2 changes: 1 addition & 1 deletion src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ import { createApifyScraperTool } from "./tools/apify-scraper-tool.js";
import { registerCli } from "./cli.js";

export default {
id: "apify-openclaw-integration",
id: "apify",
name: "Apify",
description:
"Web scraping and data extraction via Apify — scrape any platform using 57+ actors across social media, maps, search, e-commerce, and more.",
Expand Down
12 changes: 6 additions & 6 deletions src/tools/apify-scraper-tool.ts
Original file line number Diff line number Diff line change
Expand Up @@ -211,7 +211,7 @@ async function handleCollect(params: {

const rawText = truncateResults(JSON.stringify(items, null, 2));
const wrapped = wrapExternalContent(rawText, {
source: `apify_scraper:${actorId}`,
source: `apify:${actorId}`,
includeWarning: false,
});

Expand All @@ -222,7 +222,7 @@ async function handleCollect(params: {
status: "SUCCEEDED",
resultCount: items.length,
text: wrapped,
externalContent: { untrusted: true, source: "apify_scraper", wrapped: true },
externalContent: { untrusted: true, source: "apify", wrapped: true },
fetchedAt: new Date().toISOString(),
...(label ? { label } : {}),
} as Record<string, unknown>;
Expand Down Expand Up @@ -300,14 +300,14 @@ export function createApifyScraperTool(options?: {
const config = parsePluginConfig(options?.pluginConfig);
const apiKey = resolveApiKey(config);
if (!resolveEnabled({ config, apiKey })) return null;
if (!isToolEnabled(config, "apify_scraper")) return null;
if (!isToolEnabled(config, "apify")) return null;

const baseUrl = resolveBaseUrl(config);
const client = options?.client ?? createApifyClient(apiKey!, baseUrl);

return {
label: "Apify Scraper",
name: "apify_scraper",
label: "Apify",
name: "apify",
description: TOOL_DESCRIPTION,
parameters: ApifyScraperSchema,
execute: async (_toolCallId, args) => {
Expand All @@ -317,7 +317,7 @@ export function createApifyScraperTool(options?: {
if (!apiKey) {
return jsonResult({
error: "missing_api_key",
message: "Set APIFY_API_KEY env var or configure apiKey in the apify-openclaw-integration plugin config.",
message: "Set APIFY_API_KEY env var or configure apiKey in the apify plugin config.",
});
}

Expand Down
8 changes: 4 additions & 4 deletions test/apify-scraper.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,15 @@ import { describe, it, expect } from "vitest";
import { createApifyScraperTool } from "../src/tools/apify-scraper-tool.js";
import { makeMockClient, TEST_CONFIG } from "./helpers.js";

describe("apify_scraper tool", () => {
describe("apify tool", () => {
it("returns null when no API key", () => {
expect(createApifyScraperTool({ pluginConfig: {} })).toBeNull();
});

it("registers with correct name", () => {
const tool = createApifyScraperTool({ ...TEST_CONFIG, client: makeMockClient() });
expect(tool!.name).toBe("apify_scraper");
expect(tool!.label).toBe("Apify Scraper");
expect(tool!.name).toBe("apify");
expect(tool!.label).toBe("Apify");
});

it("discover action — store search", async () => {
Expand Down Expand Up @@ -121,7 +121,7 @@ describe("apify_scraper tool", () => {
await expect(tool.execute("t1", { action: "unknown" })).rejects.toThrow();
});

it("returns null when apify_scraper is excluded from enabledTools", () => {
it("returns null when apify is excluded from enabledTools", () => {
const tool = createApifyScraperTool({
pluginConfig: { apiKey: "test-key", enabledTools: ["other_tool"] as string[] },
});
Expand Down
Loading