Setting Up SearXNG as a Private Search Backend for OpenClaw Agents

Search is fundamental to AI agents. Whether researching topics, gathering context, or finding documentation, agents rely on search engines constantly. But commercial search engines track queries, build profiles, and limit API access. For agents handling sensitive information or operating at scale, privacy and control matter.

SearXNG is a privacy-respecting metasearch engine that aggregates results from multiple sources without tracking. This guide shows you how to set up SearXNG as a private search backend for OpenClaw agents.

Why SearXNG for Agents?

Privacy

Commercial search engines track everything:

• Query history


• IP addresses


• User profiles


• Behavioral patterns

For agents working with proprietary information, client data, or personal details, this tracking is a liability. SearXNG eliminates it:

• No query logging


• No tracking cookies


• No user profiling


• Queries don't leak to third parties

Cost

Search API pricing adds up fast:

• Google Custom Search: $5 per 1000 queries


• Bing Search API: Similar pricing tiers


• Brave Search: Free tier limited to 2000 queries/month

SearXNG is free and self-hosted. Once set up, query volume is limited only by your server capacity.

Control

With SearXNG, you control:

• Which engines to query


• Result ranking algorithms


• Rate limiting


• Content filtering


• Result formatting

No vendor lock-in, no arbitrary API changes.

Aggregation

SearXNG queries multiple engines simultaneously and merges results. One query to SearXNG can pull from:

• Google


• Bing


• DuckDuckGo


• Wikipedia


• GitHub


• Reddit


• Stack Overflow


• YouTube


• And 70+ other engines

Better coverage than any single commercial API.

Installation

Option 1: Docker (Recommended)

The easiest way to run SearXNG is via Docker:

# Clone the SearXNG Docker repo
git clone https://github.com/searxng/searxng-docker.git
cd searxng-docker

# Generate secret key
sed -i "s|ultrasecretkey|$(openssl rand -hex 32)|g" searxng/settings.yml

# Start services
docker-compose up -d

SearXNG is now running on http://localhost:8080.

Option 2: Manual Installation

For production or custom setups:

# Install dependencies (Ubuntu/Debian)
sudo apt update
sudo apt install -y python3-dev python3-pip python3-venv \
  git build-essential libxml2-dev libxslt1-dev \
  zlib1g-dev libffi-dev libssl-dev

# Clone SearXNG
git clone https://github.com/searxng/searxng.git /opt/searxng
cd /opt/searxng

# Create virtual environment
python3 -m venv venv
source venv/bin/activate

# Install SearXNG
pip install -U pip setuptools wheel
pip install -r requirements.txt

# Run
export SEARXNG_SECRET=$(openssl rand -hex 32)
python searx/webapp.py

SearXNG runs on http://127.0.0.1:8888.

Option 3: Public Instances

For testing, use a public SearXNG instance:

• https://searx.be


• https://search.bus-hit.me


• https://searx.work

Full list: https://searx.space/

Warning: Public instances may log queries or have rate limits. Only use for non-sensitive testing.

Configuration

Basic Configuration

Edit searxng/settings.yml:

general:
  instance_name: "OpenClaw Search"
  privacypolicy_url: false
  donation_url: false
  contact_url: false
  enable_metrics: false

search:
  safe_search: 0  # 0=off, 1=moderate, 2=strict
  autocomplete: "google"
  default_lang: "en"
  formats:
    - html
    - json

server:
  port: 8080
  bind_address: "0.0.0.0"
  secret_key: "YOUR_SECRET_KEY_HERE"
  limiter: false  # Disable rate limiting for agents
  image_proxy: true

Engine Configuration

Enable/disable specific engines in settings.yml:

engines:
  - name: google
    engine: google
    shortcut: g
    weight: 1.0

  - name: bing
    engine: bing
    shortcut: b
    weight: 0.8

  - name: duckduckgo
    engine: duckduckgo
    shortcut: ddg
    weight: 0.9

  - name: github
    engine: github
    shortcut: gh
    weight: 1.0

  - name: stackoverflow
    engine: stackoverflow
    shortcut: so
    weight: 1.0

  - name: wikipedia
    engine: wikipedia
    shortcut: wp
    weight: 1.0
    language: en

Adjust weight to prioritize certain engines in result ranking.

JSON API Configuration

For programmatic access, enable JSON format:

search:
  formats:
    - json
    - html

Query via:

http://localhost:8080/search?q=query&format=json

Integrating with OpenClaw

Method 1: Direct HTTP Requests

Simplest approach:

async function searchSearXNG(query: string) {
  const url = new URL("http://localhost:8080/search");
  url.searchParams.set("q", query);
  url.searchParams.set("format", "json");
  url.searchParams.set("language", "en");
  
  const response = await fetch(url.toString());
  const data = await response.json();
  
  return data.results.map(r => ({
    title: r.title,
    url: r.url,
    content: r.content,
    engine: r.engine
  }));
}

Method 2: OpenClaw Skill

Create a reusable skill:

mkdir -p skills/searxng

skills/searxng/SKILL.md:

# SearXNG Search

Privacy-respecting search via self-hosted SearXNG.

## Commands

- `searxng search <query>` - Search and return results
- `searxng config` - Show current configuration

## Configuration

Set `SEARXNG_URL` environment variable:

bash export SEARXNG_URL="http://localhost:8080"

skills/searxng/scripts/search.ts:

import fetch from "node-fetch";

const SEARXNG_URL = process.env.SEARXNG_URL || "http://localhost:8080";

interface SearchResult {
  title: string;
  url: string;
  content: string;
  engine: string;
}

async function search(query: string, options = {}) {
  const url = new URL(`${SEARXNG_URL}/search`);
  url.searchParams.set("q", query);
  url.searchParams.set("format", "json");
  url.searchParams.set("language", options.language || "en");
  
  if (options.engines) {
    url.searchParams.set("engines", options.engines.join(","));
  }
  
  if (options.categories) {
    url.searchParams.set("categories", options.categories.join(","));
  }
  
  const response = await fetch(url.toString());
  
  if (!response.ok) {
    throw new Error(`SearXNG search failed: ${response.statusText}`);
  }
  
  const data = await response.json();
  
  return {
    query: data.query,
    results: data.results.map(r => ({
      title: r.title,
      url: r.url,
      content: r.content || "",
      engine: r.engine
    })),
    suggestions: data.suggestions || [],
    answersCount: data.answers?.length || 0
  };
}

if (require.main === module) {
  const query = process.argv.slice(2).join(" ");
  
  search(query)
    .then(results => {
      console.log(JSON.stringify(results, null, 2));
    })
    .catch(console.error);
}

export { search };

Use in OpenClaw:

node skills/searxng/scripts/search.ts "best practices for typescript"

Method 3: Replace web_search Tool

If OpenClaw's web_search tool uses a commercial API, you can swap it out for SearXNG:

Locate the web_search tool implementation

Replace the API call with SearXNG

Maintain the same interface so agents don't need changes

This gives you privacy without changing agent code.

Advanced Features

Category-Specific Search

SearXNG supports categories:

await search("machine learning", { categories: ["science"] });
await search("react hooks", { categories: ["it"] });
await search("pizza recipe", { categories: ["general"] });

Categories: general, images, videos, news, music, files, it, science, social media.

Engine-Specific Search

Query specific engines only:

// Search GitHub only
await search("openclaw", { engines: ["github"] });

// Search Stack Overflow + Reddit
await search("python async", { engines: ["stackoverflow", "reddit"] });

Image Search

const images = await search("golden retriever", { categories: ["images"] });

for (const img of images.results) {
  console.log(img.title);
  console.log(img.img_src);  // Direct image URL
  console.log(img.thumbnail_src);  // Thumbnail
}

Autocomplete

Get search suggestions:

async function autocomplete(query: string) {
  const url = new URL(`${SEARXNG_URL}/autocompleter`);
  url.searchParams.set("q", query);
  url.searchParams.set("format", "json");
  
  const response = await fetch(url.toString());
  return await response.json();
}

const suggestions = await autocomplete("python as");
// ["python asyncio", "python async", "python assert", ...]

Production Deployment

Security

Use HTTPS: Put SearXNG behind nginx with SSL

server {
  listen 443 ssl;
  server_name search.yourdomain.com;
  
  ssl_certificate /path/to/cert.pem;
  ssl_certificate_key /path/to/key.pem;
  
  location / {
    proxy_pass http://localhost:8080;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
  }
}

Firewall: Only allow access from your OpenClaw server

sudo ufw allow from YOUR_OPENCLAW_IP to any port 8080

Authentication: Add basic auth if exposing to internet

server:
  limiter: true
  public_instance: false

Performance Tuning

Redis caching for faster repeated queries:

redis:
  url: redis://localhost:6379/0

Adjust timeouts for slow engines:

outgoing:
  request_timeout: 10.0  # seconds
  max_request_timeout: 20.0

Enable result pooling:

search:
  default_pool: true

Monitoring

Enable metrics:

general:
  enable_metrics: true

Metrics available at http://localhost:8080/stats:

• Total queries


• Engine response times


• Error rates


• Cache hit rates

Use Cases

Use Case 1: Research Agent

Agent that researches topics and compiles reports:

async function researchTopic(topic: string) {
  const results = await search(topic, { 
    categories: ["general", "science", "it"] 
  });
  
  const sources = results.results.slice(0, 10);
  
  const report = {
    topic,
    sources: sources.map(s => ({
      title: s.title,
      url: s.url,
      summary: s.content
    })),
    generatedAt: new Date().toISOString()
  };
  
  await saveReport(topic, report);
  return report;
}

Use Case 2: Documentation Finder

Find docs for libraries and frameworks:

async function findDocs(library: string) {
  const results = await search(`${library} documentation`, {
    engines: ["google", "duckduckgo"]
  });
  
  const officialDocs = results.results.find(r =>
    r.url.includes("docs") || r.url.includes("documentation")
  );
  
  return officialDocs || results.results[0];
}

const typescriptDocs = await findDocs("typescript");
console.log(typescriptDocs.url);  // https://www.typescriptlang.org/docs/

Use Case 3: Code Example Search

Find code examples on GitHub:

async function findCodeExamples(query: string) {
  const results = await search(query, { engines: ["github"] });
  
  return results.results.filter(r =>
    r.url.includes("/blob/") || r.url.includes("/tree/")
  ).map(r => ({
    repo: extractRepoName(r.url),
    file: extractFileName(r.url),
    url: r.url,
    snippet: r.content
  }));
}

const examples = await findCodeExamples("react useEffect example");

Use Case 4: News Monitoring

Track news on specific topics:

async function monitorNews(topic: string) {
  const results = await search(topic, { 
    categories: ["news"],
    language: "en"
  });
  
  const articles = results.results.map(r => ({
    headline: r.title,
    url: r.url,
    source: r.engine,
    publishedAt: r.publishedDate || "unknown"
  }));
  
  // Store for comparison with previous runs
  await storeNewsSnapshot(topic, articles);
  
  return articles;
}

// Run periodically
setInterval(() => monitorNews("AI regulation"), 3600000); // Hourly

Comparison with Commercial Search APIs

Google Custom Search

Pros:

• High quality results


• Large index


• Advanced features (autocomplete, knowledge graph)

Cons:

• $5 per 1000 queries


• 10K query/day limit


• Tracks queries


• API key required

Bing Search API

Pros:

• Good coverage


• Reasonable pricing

Cons:

• 3K free queries/month, then $5/1000


• Requires API key


• Microsoft ecosystem lock-in

Brave Search API

Pros:

• Privacy-focused


• 2K free queries/month


• Independent index

Cons:

• Smaller index than Google/Bing


• Free tier limited


• Still centralized

SearXNG

Pros:

• Unlimited queries


• No tracking


• Self-hosted (full control)


• Aggregates multiple engines


• Free and open source

Cons:

• Requires hosting/maintenance


• Result quality depends on upstream engines


• No official support

For agents with high query volume or privacy requirements, SearXNG wins.

Troubleshooting

Results are slow

Enable Redis caching

Disable slow engines

Reduce timeout values

Use a more powerful server

No results for some queries

Check which engines are enabled

Try different engines for that query type

Verify engines aren't blocking your IP

Rate limiting errors

Reduce query frequency

Use more engines to distribute load

Add delays between queries

Consider multiple SearXNG instances

Docker container crashes

Check logs: docker-compose logs -f

Increase memory limit in docker-compose.yml

Verify settings.yml syntax

Wrapping Up

SearXNG gives OpenClaw agents privacy-respecting, cost-effective, and powerful search capabilities. Whether you're handling sensitive information, running high query volumes, or just want control over your search infrastructure, self-hosting SearXNG is worth the setup effort.

Start with Docker, get it working, then tune for your workload. Your agents will thank you for the privacy and unlimited queries.

Search is fundamental to agent intelligence. Make sure yours is private.