# Core API Service

OpenAPI-compatible functions for Open WebUI and infrastructure management, providing web scraping, AI orchestration, and Portainer automation capabilities.

## Features

### Web Scraper
- Intelligent content extraction using Trafilatura
- BeautifulSoup fallback for complex pages
- Configurable content length limits
- Optional link extraction
- Perfect for feeding webpage content to LLMs

### Infrastructure Management
- Portainer stack control (start/stop services)
- Service status monitoring
- Container health checks
- Service group management
- Read/write REST API

### AI Orchestration
- OpenAI-compatible API endpoints
- Model routing and management
- Streaming responses
- Function calling support
- Multi-phase enhancement roadmap

## Architecture

```
src/
├── config.py              # Global application settings
├── logging_config.py      # Logging configuration
├── base_schema.py         # Base Pydantic models
├── main.py               # FastAPI application entry point
└── modules/
    ├── web_scraper/      # Web scraper module
    │   ├── config.py
    │   ├── schemas.py
    │   ├── service.py
    │   ├── router.py
    │   └── exceptions.py
    └── infrastructure/   # Infrastructure management
        ├── config.py
        ├── schemas.py
        ├── service.py
        └── router.py
```

## Deployment

### Portainer Stack

1. Navigate to Portainer UI
2. Go to **Stacks** → **Add Stack**
3. Name: `core-api`
4. Upload `stacks/core-api.yml` or paste contents
5. Deploy

### Environment Variables

See `.env.example` in the service directory for all available configuration options.

Key variables:
- `PORTAINER_URL` - Portainer API endpoint
- `PORTAINER_API_KEY` - API key for Portainer authentication
- `LOG_LEVEL` - Logging verbosity (DEBUG, INFO, WARNING, ERROR)
- `CORS_ORIGINS` - Allowed CORS origins

## API Documentation

Once deployed, access documentation at:
- **Swagger UI**: http://localhost:8083/docs
- **ReDoc**: http://localhost:8083/redoc
- **OpenAPI Spec**: http://localhost:8083/openapi.json

## API Endpoints

### Web Scraper

**POST /web-scraper/scrape**

Scrape and extract content from a website.

Request:
```json
{
  "url": "https://example.com/article",
  "extract_main_content": true,
  "include_links": false,
  "max_length": 10000
}
```

Response:
```json
{
  "url": "https://example.com/article",
  "title": "Article Title",
  "content": "Extracted article content...",
  "extracted_at": "2025-11-12T19:30:00Z",
  "content_length": 5432,
  "links": null
}
```

### Infrastructure Management

**GET /infrastructure/services**

List all Portainer stacks with status.

Response:
```json
[
  {
    "name": "jellyfin",
    "status": "running",
    "containers": 1,
    "running_containers": 1
  }
]
```

**POST /infrastructure/services/{name}/start**

Start a service stack.

**POST /infrastructure/services/{name}/stop**

Stop a service stack.

**GET /infrastructure/service-groups**

Get service groupings and always-on services.

Response:
```json
{
  "service_groups": {
    "jellyfin": ["jellyfin"],
    "nextcloud": ["nextcloud"],
    "ai-stack": ["open-webui", "ollama", "qdrant"]
  },
  "always_on": ["portainer", "nginx-proxy-manager", "core-api"]
}
```

### Health Check

**GET /health**

Service health check endpoint.

Response:
```json
{
  "status": "healthy"
}
```

## Integration with Open WebUI

### Method 1: Functions (OpenAPI Import)
1. In Open WebUI, navigate to Functions
2. Import from OpenAPI spec: `http://localhost:8083/openapi.json`
3. Use functions directly in chat

### Method 2: Pipelines
1. Create a pipeline that calls Core API endpoints
2. Use as data source for LLM workflows

### Method 3: Direct API Calls
```python
import httpx

async with httpx.AsyncClient() as client:
    response = await client.post(
        "http://localhost:8083/web-scraper/scrape",
        json={
            "url": "https://example.com",
            "extract_main_content": True
        }
    )
    data = response.json()
```

## Development

### Requirements
- Python 3.12+
- Docker (for containerized deployment)

### Local Development

```bash
# Install dependencies
pip install -r requirements.txt

# Run locally
uvicorn src.main:app --reload --host 0.0.0.0 --port 8083
```

### Docker Build

```bash
# Build image
docker build -t core-api:latest .

# Run container
docker run -p 8083:8083 core-api:latest
```

## Logging

Logs are written to:
- **Console**: stdout (captured by Docker)
- **File**: `/app/logs/app.log` (persisted via volume mount)

Log format:
```
2025-11-12 19:30:00 | INFO     | src.web_scraper.service:scrape_url:45 | Starting scrape for URL: https://example.com
```

## Security

- Runs as non-root user (uid 1000)
- No authentication required (internal network only)
- CORS configured for same-network access
- Rate limiting: Not implemented (internal use only)
- **Always-on service** - Cannot be stopped via infrastructure management

## Future Enhancements

See [AI Orchestrator Plan](../../plans/active/ai-orchestrator-plan.md) for upcoming features:

### Phase 2: Memory Systems (In Progress)
- Ephemeral, short-term, and long-term memory
- Vector embeddings with Qdrant
- Memory search and retrieval

### Phase 3: Multi-Model Management
- Dynamic model routing
- Cost optimization
- Fallback strategies

### Phase 4: Reasoning & Chain-of-Thought
- Structured reasoning
- Multi-step problem solving
- Verification and validation

### Phase 5: Agentic Workflows
- Tool integration
- Multi-agent orchestration
- Autonomous task execution

### Phase 6: Production Optimization
- Caching strategies
- Performance tuning
- Monitoring and metrics

## Troubleshooting

### Container won't start
```bash
docker logs core-api
```

### API not responding
```bash
curl http://localhost:8083/health
```

### Check OpenAPI spec
```bash
curl http://localhost:8083/openapi.json | jq
```

### Portainer connection issues
1. Verify `PORTAINER_URL` is correct
2. Check `PORTAINER_API_KEY` is valid
3. Ensure Portainer is accessible from core-api container
4. Check Docker network connectivity

## Related Documentation

- [Stacks Reference](../reference/stacks.md) - All Docker Compose stacks
- [Automation Reference](../reference/AUTOMATION.md) - Portainer REST API details
- [AI Orchestrator Plan](../../plans/active/ai-orchestrator-plan.md) - Feature roadmap
- [Organizr Widget](organizr-widgets.md) - Service control UI integration