Core Code API

OpenAPI-compatible functions for Open WebUI, providing web scraping and data processing capabilities.

Features

Web Scraper

• Intelligent content extraction using Trafilatura
• BeautifulSoup fallback for complex pages
• Configurable content length limits
• Optional link extraction
• Perfect for feeding webpage content to LLMs

Architecture

src/
├── config.py              # Global application settings
├── logging_config.py      # Logging configuration
├── base_schema.py         # Base Pydantic models
├── main.py               # FastAPI application entry point
└── web_scraper/          # Web scraper module
    ├── __init__.py
    ├── config.py         # Module-specific settings
    ├── schemas.py        # Pydantic request/response models
    ├── service.py        # Business logic
    ├── router.py         # API routes
    └── exceptions.py     # Custom exceptions

Development

Requirements

• Python 3.12+
• Docker (for containerized deployment)

Local Development

# Install dependencies
pip install -r requirements.txt

# Run locally
uvicorn src.main:app --reload --host 0.0.0.0 --port 8083

Adding New Dependencies

Important: Dependencies use major version pinning (~=) for automatic patch updates while preventing breaking changes.

1. Add package to requirements.txt with major version constraint:

   package-name~=1.2.0  # Allows 1.2.x, blocks 1.3.0
   

2. Restart the container to install:

   docker restart core-api
   

The container automatically runs pip install -r requirements.txt on every boot, so new dependencies are installed immediately on restart.

Version Pinning Best Practices:

• Use ~= (compatible release) for most packages: fastapi~=0.115.0
• Use >=X,<Y for complex constraints: langchain-core>=0.3.17,<0.4.0
• Allows automatic security patches without breaking changes
• Documented in PEP 440

Docker Build

# Build image
docker build -t core-code:latest .

# Run container
docker run -p 8083:8083 core-code:latest

Deployment

Portainer Stack

1. Navigate to Portainer UI
2. Go to Stacks → Add Stack
3. Name: core-code
4. Upload stacks/core-code.yml or paste contents
5. Deploy

Environment Variables

See .env.example for all available configuration options.

API Documentation

Once deployed, access documentation at:

• Swagger UI: http://192.168.86.149:8083/docs
• ReDoc: http://192.168.86.149:8083/redoc
• OpenAPI Spec: http://192.168.86.149:8083/openapi.json

Integration with Open WebUI

Method 1: Functions (OpenAPI Import)

1. In Open WebUI, navigate to Functions
2. Import from OpenAPI spec: http://192.168.86.149:8083/openapi.json
3. Use functions directly in chat

Method 2: Pipelines

1. Create a pipeline that calls Core Code API endpoints
2. Use as data source for LLM workflows

Method 3: Direct API Calls

import httpx

async with httpx.AsyncClient() as client:
    response = await client.post(
        "http://192.168.86.149:8083/web-scraper/scrape",
        json={
            "url": "https://example.com",
            "extract_main_content": True
        }
    )
    data = response.json()

API Endpoints

Web Scraper

POST /web-scraper/scrape

Scrape and extract content from a website.

Request:

{
  "url": "https://example.com/article",
  "extract_main_content": true,
  "include_links": false,
  "max_length": 10000
}

Response:

{
  "url": "https://example.com/article",
  "title": "Article Title",
  "content": "Extracted article content...",
  "extracted_at": "2025-11-12T19:30:00Z",
  "content_length": 5432,
  "links": null
}

Logging

Logs are written to:

• Console: stdout (captured by Docker)
• File: /app/logs/app.log (persisted via volume mount)

Log format:

2025-11-12 19:30:00 | INFO     | src.web_scraper.service:scrape_url:45 | Starting scrape for URL: https://example.com

Health Checks

• Endpoint: GET /health
• Docker: Automatic health checks configured
• Response: {"status": "healthy"}

Security

• Runs as non-root user (uid 1000)
• No authentication required (internal network only)
• CORS configured for same-network access
• Rate limiting: Not implemented (internal use only)

Future Modules

The architecture supports adding new modules:

• Data transformation functions
• API integrations
• File processing
• Database queries

Each module follows the same structure:

src/
└── module_name/
    ├── config.py
    ├── schemas.py
    ├── service.py
    ├── router.py
    └── exceptions.py

Troubleshooting

Container won't start

docker logs core-code

API not responding

curl http://192.168.86.149:8083/health

Check OpenAPI spec

curl http://192.168.86.149:8083/openapi.json | jq

License

Internal use only.