portainer-core/docs/services/core-api.md

Core API Service

OpenAPI-compatible functions for Open WebUI and infrastructure management, providing web scraping, AI orchestration, and Portainer automation capabilities.

Features

Web Scraper

• Intelligent content extraction using Trafilatura
• BeautifulSoup fallback for complex pages
• Configurable content length limits
• Optional link extraction
• Perfect for feeding webpage content to LLMs

Infrastructure Management

• Portainer stack control (start/stop services)
• Service status monitoring
• Container health checks
• Service group management
• Read/write REST API

AI Orchestration

• OpenAI-compatible API endpoints
• Model routing and management
• Streaming responses
• Function calling support
• Multi-phase enhancement roadmap

Architecture

src/
├── config.py              # Global application settings
├── logging_config.py      # Logging configuration
├── base_schema.py         # Base Pydantic models
├── main.py               # FastAPI application entry point
└── modules/
    ├── web_scraper/      # Web scraper module
    │   ├── config.py
    │   ├── schemas.py
    │   ├── service.py
    │   ├── router.py
    │   └── exceptions.py
    └── infrastructure/   # Infrastructure management
        ├── config.py
        ├── schemas.py
        ├── service.py
        └── router.py

Deployment

Portainer Stack

1. Navigate to Portainer UI
2. Go to Stacks → Add Stack
3. Name: core-api
4. Upload stacks/core-api.yml or paste contents
5. Deploy

Environment Variables

See .env.example in the service directory for all available configuration options.

Key variables:

• PORTAINER_URL - Portainer API endpoint
• PORTAINER_API_KEY - API key for Portainer authentication
• LOG_LEVEL - Logging verbosity (DEBUG, INFO, WARNING, ERROR)
• CORS_ORIGINS - Allowed CORS origins

API Documentation

Once deployed, access documentation at:

• Swagger UI: http://localhost:8083/docs
• ReDoc: http://localhost:8083/redoc
• OpenAPI Spec: http://localhost:8083/openapi.json

API Endpoints

Web Scraper

POST /web-scraper/scrape

Scrape and extract content from a website.

Request:

{
  "url": "https://example.com/article",
  "extract_main_content": true,
  "include_links": false,
  "max_length": 10000
}

Response:

{
  "url": "https://example.com/article",
  "title": "Article Title",
  "content": "Extracted article content...",
  "extracted_at": "2025-11-12T19:30:00Z",
  "content_length": 5432,
  "links": null
}

Infrastructure Management

GET /infrastructure/services

List all Portainer stacks with status.

Response:

[
  {
    "name": "jellyfin",
    "status": "running",
    "containers": 1,
    "running_containers": 1
  }
]

POST /infrastructure/services/{name}/start

Start a service stack.

POST /infrastructure/services/{name}/stop

Stop a service stack.

GET /infrastructure/service-groups

Get service groupings and always-on services.

Response:

{
  "service_groups": {
    "jellyfin": ["jellyfin"],
    "nextcloud": ["nextcloud"],
    "ai-stack": ["open-webui", "ollama", "qdrant"]
  },
  "always_on": ["portainer", "nginx-proxy-manager", "core-api"]
}

Health Check

GET /health

Service health check endpoint.

Response:

{
  "status": "healthy"
}

Integration with Open WebUI

Method 1: Functions (OpenAPI Import)

1. In Open WebUI, navigate to Functions
2. Import from OpenAPI spec: http://localhost:8083/openapi.json
3. Use functions directly in chat

Method 2: Pipelines

1. Create a pipeline that calls Core API endpoints
2. Use as data source for LLM workflows

Method 3: Direct API Calls

import httpx

async with httpx.AsyncClient() as client:
    response = await client.post(
        "http://localhost:8083/web-scraper/scrape",
        json={
            "url": "https://example.com",
            "extract_main_content": True
        }
    )
    data = response.json()

Development

Requirements

• Python 3.12+
• Docker (for containerized deployment)

Local Development

# Install dependencies
pip install -r requirements.txt

# Run locally
uvicorn src.main:app --reload --host 0.0.0.0 --port 8083

Docker Build

# Build image
docker build -t core-api:latest .

# Run container
docker run -p 8083:8083 core-api:latest

Logging

Logs are written to:

• Console: stdout (captured by Docker)
• File: /app/logs/app.log (persisted via volume mount)

Log format:

2025-11-12 19:30:00 | INFO     | src.web_scraper.service:scrape_url:45 | Starting scrape for URL: https://example.com

Security

• Runs as non-root user (uid 1000)
• No authentication required (internal network only)
• CORS configured for same-network access
• Rate limiting: Not implemented (internal use only)
• Always-on service - Cannot be stopped via infrastructure management

Future Enhancements

See AI Orchestrator Plan for upcoming features:

Phase 2: Memory Systems (In Progress)

• Ephemeral, short-term, and long-term memory
• Vector embeddings with Qdrant
• Memory search and retrieval

Phase 3: Multi-Model Management

• Dynamic model routing
• Cost optimization
• Fallback strategies

Phase 4: Reasoning & Chain-of-Thought

• Structured reasoning
• Multi-step problem solving
• Verification and validation

Phase 5: Agentic Workflows

• Tool integration
• Multi-agent orchestration
• Autonomous task execution

Phase 6: Production Optimization

• Caching strategies
• Performance tuning
• Monitoring and metrics

Troubleshooting

Container won't start

docker logs core-api

API not responding

curl http://localhost:8083/health

Check OpenAPI spec

curl http://localhost:8083/openapi.json | jq

Portainer connection issues

1. Verify PORTAINER_URL is correct
2. Check PORTAINER_API_KEY is valid
3. Ensure Portainer is accessible from core-api container
4. Check Docker network connectivity

Related Documentation

• Stacks Reference - All Docker Compose stacks
• Automation Reference - Portainer REST API details
• AI Orchestrator Plan - Feature roadmap
• Organizr Widget - Service control UI integration