restructure documentation
This commit is contained in:
@@ -0,0 +1,290 @@
|
||||
# Core API Service
|
||||
|
||||
OpenAPI-compatible functions for Open WebUI and infrastructure management, providing web scraping, AI orchestration, and Portainer automation capabilities.
|
||||
|
||||
## Features
|
||||
|
||||
### Web Scraper
|
||||
- Intelligent content extraction using Trafilatura
|
||||
- BeautifulSoup fallback for complex pages
|
||||
- Configurable content length limits
|
||||
- Optional link extraction
|
||||
- Perfect for feeding webpage content to LLMs
|
||||
|
||||
### Infrastructure Management
|
||||
- Portainer stack control (start/stop services)
|
||||
- Service status monitoring
|
||||
- Container health checks
|
||||
- Service group management
|
||||
- Read/write REST API
|
||||
|
||||
### AI Orchestration
|
||||
- OpenAI-compatible API endpoints
|
||||
- Model routing and management
|
||||
- Streaming responses
|
||||
- Function calling support
|
||||
- Multi-phase enhancement roadmap
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
src/
|
||||
├── config.py # Global application settings
|
||||
├── logging_config.py # Logging configuration
|
||||
├── base_schema.py # Base Pydantic models
|
||||
├── main.py # FastAPI application entry point
|
||||
└── modules/
|
||||
├── web_scraper/ # Web scraper module
|
||||
│ ├── config.py
|
||||
│ ├── schemas.py
|
||||
│ ├── service.py
|
||||
│ ├── router.py
|
||||
│ └── exceptions.py
|
||||
└── infrastructure/ # Infrastructure management
|
||||
├── config.py
|
||||
├── schemas.py
|
||||
├── service.py
|
||||
└── router.py
|
||||
```
|
||||
|
||||
## Deployment
|
||||
|
||||
### Portainer Stack
|
||||
|
||||
1. Navigate to Portainer UI
|
||||
2. Go to **Stacks** → **Add Stack**
|
||||
3. Name: `core-api`
|
||||
4. Upload `stacks/core-api.yml` or paste contents
|
||||
5. Deploy
|
||||
|
||||
### Environment Variables
|
||||
|
||||
See `.env.example` in the service directory for all available configuration options.
|
||||
|
||||
Key variables:
|
||||
- `PORTAINER_URL` - Portainer API endpoint
|
||||
- `PORTAINER_API_KEY` - API key for Portainer authentication
|
||||
- `LOG_LEVEL` - Logging verbosity (DEBUG, INFO, WARNING, ERROR)
|
||||
- `CORS_ORIGINS` - Allowed CORS origins
|
||||
|
||||
## API Documentation
|
||||
|
||||
Once deployed, access documentation at:
|
||||
- **Swagger UI**: http://localhost:8083/docs
|
||||
- **ReDoc**: http://localhost:8083/redoc
|
||||
- **OpenAPI Spec**: http://localhost:8083/openapi.json
|
||||
|
||||
## API Endpoints
|
||||
|
||||
### Web Scraper
|
||||
|
||||
**POST /web-scraper/scrape**
|
||||
|
||||
Scrape and extract content from a website.
|
||||
|
||||
Request:
|
||||
```json
|
||||
{
|
||||
"url": "https://example.com/article",
|
||||
"extract_main_content": true,
|
||||
"include_links": false,
|
||||
"max_length": 10000
|
||||
}
|
||||
```
|
||||
|
||||
Response:
|
||||
```json
|
||||
{
|
||||
"url": "https://example.com/article",
|
||||
"title": "Article Title",
|
||||
"content": "Extracted article content...",
|
||||
"extracted_at": "2025-11-12T19:30:00Z",
|
||||
"content_length": 5432,
|
||||
"links": null
|
||||
}
|
||||
```
|
||||
|
||||
### Infrastructure Management
|
||||
|
||||
**GET /infrastructure/services**
|
||||
|
||||
List all Portainer stacks with status.
|
||||
|
||||
Response:
|
||||
```json
|
||||
[
|
||||
{
|
||||
"name": "jellyfin",
|
||||
"status": "running",
|
||||
"containers": 1,
|
||||
"running_containers": 1
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
**POST /infrastructure/services/{name}/start**
|
||||
|
||||
Start a service stack.
|
||||
|
||||
**POST /infrastructure/services/{name}/stop**
|
||||
|
||||
Stop a service stack.
|
||||
|
||||
**GET /infrastructure/service-groups**
|
||||
|
||||
Get service groupings and always-on services.
|
||||
|
||||
Response:
|
||||
```json
|
||||
{
|
||||
"service_groups": {
|
||||
"jellyfin": ["jellyfin"],
|
||||
"nextcloud": ["nextcloud"],
|
||||
"ai-stack": ["open-webui", "ollama", "qdrant"]
|
||||
},
|
||||
"always_on": ["portainer", "nginx-proxy-manager", "core-api"]
|
||||
}
|
||||
```
|
||||
|
||||
### Health Check
|
||||
|
||||
**GET /health**
|
||||
|
||||
Service health check endpoint.
|
||||
|
||||
Response:
|
||||
```json
|
||||
{
|
||||
"status": "healthy"
|
||||
}
|
||||
```
|
||||
|
||||
## Integration with Open WebUI
|
||||
|
||||
### Method 1: Functions (OpenAPI Import)
|
||||
1. In Open WebUI, navigate to Functions
|
||||
2. Import from OpenAPI spec: `http://localhost:8083/openapi.json`
|
||||
3. Use functions directly in chat
|
||||
|
||||
### Method 2: Pipelines
|
||||
1. Create a pipeline that calls Core API endpoints
|
||||
2. Use as data source for LLM workflows
|
||||
|
||||
### Method 3: Direct API Calls
|
||||
```python
|
||||
import httpx
|
||||
|
||||
async with httpx.AsyncClient() as client:
|
||||
response = await client.post(
|
||||
"http://localhost:8083/web-scraper/scrape",
|
||||
json={
|
||||
"url": "https://example.com",
|
||||
"extract_main_content": True
|
||||
}
|
||||
)
|
||||
data = response.json()
|
||||
```
|
||||
|
||||
## Development
|
||||
|
||||
### Requirements
|
||||
- Python 3.12+
|
||||
- Docker (for containerized deployment)
|
||||
|
||||
### Local Development
|
||||
|
||||
```bash
|
||||
# Install dependencies
|
||||
pip install -r requirements.txt
|
||||
|
||||
# Run locally
|
||||
uvicorn src.main:app --reload --host 0.0.0.0 --port 8083
|
||||
```
|
||||
|
||||
### Docker Build
|
||||
|
||||
```bash
|
||||
# Build image
|
||||
docker build -t core-api:latest .
|
||||
|
||||
# Run container
|
||||
docker run -p 8083:8083 core-api:latest
|
||||
```
|
||||
|
||||
## Logging
|
||||
|
||||
Logs are written to:
|
||||
- **Console**: stdout (captured by Docker)
|
||||
- **File**: `/app/logs/app.log` (persisted via volume mount)
|
||||
|
||||
Log format:
|
||||
```
|
||||
2025-11-12 19:30:00 | INFO | src.web_scraper.service:scrape_url:45 | Starting scrape for URL: https://example.com
|
||||
```
|
||||
|
||||
## Security
|
||||
|
||||
- Runs as non-root user (uid 1000)
|
||||
- No authentication required (internal network only)
|
||||
- CORS configured for same-network access
|
||||
- Rate limiting: Not implemented (internal use only)
|
||||
- **Always-on service** - Cannot be stopped via infrastructure management
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
See [AI Orchestrator Plan](../../plans/active/ai-orchestrator-plan.md) for upcoming features:
|
||||
|
||||
### Phase 2: Memory Systems (In Progress)
|
||||
- Ephemeral, short-term, and long-term memory
|
||||
- Vector embeddings with Qdrant
|
||||
- Memory search and retrieval
|
||||
|
||||
### Phase 3: Multi-Model Management
|
||||
- Dynamic model routing
|
||||
- Cost optimization
|
||||
- Fallback strategies
|
||||
|
||||
### Phase 4: Reasoning & Chain-of-Thought
|
||||
- Structured reasoning
|
||||
- Multi-step problem solving
|
||||
- Verification and validation
|
||||
|
||||
### Phase 5: Agentic Workflows
|
||||
- Tool integration
|
||||
- Multi-agent orchestration
|
||||
- Autonomous task execution
|
||||
|
||||
### Phase 6: Production Optimization
|
||||
- Caching strategies
|
||||
- Performance tuning
|
||||
- Monitoring and metrics
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Container won't start
|
||||
```bash
|
||||
docker logs core-api
|
||||
```
|
||||
|
||||
### API not responding
|
||||
```bash
|
||||
curl http://localhost:8083/health
|
||||
```
|
||||
|
||||
### Check OpenAPI spec
|
||||
```bash
|
||||
curl http://localhost:8083/openapi.json | jq
|
||||
```
|
||||
|
||||
### Portainer connection issues
|
||||
1. Verify `PORTAINER_URL` is correct
|
||||
2. Check `PORTAINER_API_KEY` is valid
|
||||
3. Ensure Portainer is accessible from core-api container
|
||||
4. Check Docker network connectivity
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Stacks Reference](../reference/stacks.md) - All Docker Compose stacks
|
||||
- [Automation Reference](../reference/AUTOMATION.md) - Portainer REST API details
|
||||
- [AI Orchestrator Plan](../../plans/active/ai-orchestrator-plan.md) - Feature roadmap
|
||||
- [Organizr Widget](organizr-widgets.md) - Service control UI integration
|
||||
@@ -0,0 +1,215 @@
|
||||
# Organizr Service Control Widget
|
||||
|
||||
A beautiful, responsive widget for managing on-demand services from your Organizr dashboard.
|
||||
|
||||
## Features
|
||||
|
||||
- ✨ **Real-time Status** - Live service status with container counts
|
||||
- 🎮 **One-Click Control** - Start/Stop services with a single click
|
||||
- 🔒 **Safety First** - Always-on services are protected and clearly marked
|
||||
- 🎨 **Beautiful UI** - Dark theme that matches Organizr
|
||||
- ⚡ **Auto-Refresh** - Updates every 10 seconds
|
||||
- 📱 **Responsive** - Works on desktop, tablet, and mobile
|
||||
|
||||
## Installation
|
||||
|
||||
### Method 1: Organizr Custom Homepage Item (Recommended)
|
||||
|
||||
1. **Copy the widget file** to a web-accessible location:
|
||||
```bash
|
||||
# If you have a web server serving files from /var/www/html:
|
||||
sudo cp organizr-widgets/service-control.html /var/www/html/widgets/
|
||||
|
||||
# Or use Organizr's public directory:
|
||||
cp organizr-widgets/service-control.html /path/to/organizr/plugins/widgets/
|
||||
```
|
||||
|
||||
2. **Add to Organizr Homepage**:
|
||||
- Open Organizr
|
||||
- Go to **Settings** → **Customize** → **Homepage Items**
|
||||
- Click **Add New Item**
|
||||
- Configure:
|
||||
- **Name**: "Service Control"
|
||||
- **Category**: Custom
|
||||
- **Type**: iFrame
|
||||
- **URL**: `http://localhost/widgets/service-control.html` (adjust path)
|
||||
- **Minimum Authentication**: User
|
||||
- **Enabled**: Yes
|
||||
- Save
|
||||
|
||||
3. **Add to Homepage**:
|
||||
- Go to **Settings** → **Customize** → **Appearance**
|
||||
- Edit your homepage layout
|
||||
- Add the "Service Control" item to desired location
|
||||
- Save
|
||||
|
||||
### Method 2: Organizr Custom HTML Tab
|
||||
|
||||
1. **Open Organizr Settings**:
|
||||
- Settings → **Tab Editor**
|
||||
|
||||
2. **Add New Tab**:
|
||||
- Click **Add Tab**
|
||||
- Configure:
|
||||
- **Tab Name**: "Services"
|
||||
- **Tab URL**: Leave empty
|
||||
- **Category**: Custom
|
||||
- **Type**: iFrame
|
||||
- **Image**: `images/tabs/services.png` (or your choice)
|
||||
|
||||
3. **Add Custom HTML**:
|
||||
- In the same tab configuration, find **Custom HTML** section
|
||||
- Copy and paste the entire contents of `service-control.html`
|
||||
- Save
|
||||
|
||||
4. **Access the Tab**:
|
||||
- The "Services" tab will now appear in your Organizr sidebar
|
||||
|
||||
### Method 3: Nginx Reverse Proxy Integration
|
||||
|
||||
If you want to serve the widget through Nginx Proxy Manager:
|
||||
|
||||
1. **Create a location** in your Organizr proxy host:
|
||||
```nginx
|
||||
location /widgets/ {
|
||||
alias /path/to/portainer-core/organizr-widgets/;
|
||||
autoindex off;
|
||||
}
|
||||
```
|
||||
|
||||
2. **Access via**: `https://your-organizr-domain.com/widgets/service-control.html`
|
||||
|
||||
## Configuration
|
||||
|
||||
### Changing API Endpoint
|
||||
|
||||
If your core-api is not on `localhost:8083`, edit the widget file:
|
||||
|
||||
```javascript
|
||||
const API_BASE = 'http://your-server:8083'; // Change this line
|
||||
```
|
||||
|
||||
### Adjusting Auto-Refresh Interval
|
||||
|
||||
Default is 10 seconds. To change:
|
||||
|
||||
```javascript
|
||||
setInterval(fetchServices, 10000); // Change 10000 to desired milliseconds
|
||||
```
|
||||
|
||||
### Customizing Displayed Services
|
||||
|
||||
By default, the widget shows all stoppable services (excludes always-on infrastructure).
|
||||
|
||||
To filter specific services, modify the `renderServices()` function:
|
||||
|
||||
```javascript
|
||||
const stoppableServices = services.filter(s =>
|
||||
!isAlwaysOn(s.name) &&
|
||||
['jellyfin', 'nextcloud', 'gitea', 'ai-stack'].includes(s.name) // Add this line
|
||||
);
|
||||
```
|
||||
|
||||
## Service Groups
|
||||
|
||||
The following service groups are defined (stopping one stops all in group):
|
||||
|
||||
- **jellyfin**: jellyfin
|
||||
- **nextcloud**: nextcloud (uses shared postgres-shared + redis-shared)
|
||||
- **gitea**: gitea, gitea-db
|
||||
- **ai-stack**: open-webui, ollama, qdrant
|
||||
- **samba**: samba
|
||||
|
||||
## Always-On Services (Cannot be stopped)
|
||||
|
||||
These infrastructure services are protected:
|
||||
- portainer
|
||||
- nginx-proxy-manager
|
||||
- core-api
|
||||
- uptime-kuma
|
||||
- organizr
|
||||
- headscale
|
||||
- watchtower
|
||||
- netdata
|
||||
- maintenance
|
||||
- postgres-shared (shared database infrastructure)
|
||||
- redis-shared (shared cache infrastructure)
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "Failed to connect to API"
|
||||
|
||||
**Problem**: Widget shows red error message
|
||||
|
||||
**Solutions**:
|
||||
1. Verify core-api is running: `docker ps | grep core-api`
|
||||
2. Check core-api URL is correct (localhost vs IP address)
|
||||
3. If accessing from remote, change `API_BASE` to full URL
|
||||
4. Check browser console for CORS errors
|
||||
|
||||
### CORS Issues
|
||||
|
||||
If accessing widget from a different domain than core-api:
|
||||
|
||||
**Option 1**: Update core-api CORS settings in `services/core-api/src/config.py`:
|
||||
```python
|
||||
cors_origins: list[str] = ["http://your-organizr-domain.com"]
|
||||
```
|
||||
|
||||
**Option 2**: Proxy the API through same domain using Nginx
|
||||
|
||||
### Services Not Appearing
|
||||
|
||||
**Check**:
|
||||
1. Services are deployed as Portainer stacks
|
||||
2. Services have proper labels: `com.docker.compose.project`
|
||||
3. Core-API can connect to Portainer
|
||||
4. Check browser console for errors
|
||||
|
||||
### Buttons Disabled
|
||||
|
||||
**Expected Behavior**:
|
||||
- Start button disabled when service is running
|
||||
- Stop button disabled when service is stopped
|
||||
- All buttons disabled for always-on services
|
||||
|
||||
## API Endpoints Used
|
||||
|
||||
The widget consumes these core-api endpoints:
|
||||
|
||||
- `GET /infrastructure/services` - Fetch service list with status
|
||||
- `GET /infrastructure/service-groups` - Fetch service groups and always-on list
|
||||
- `POST /infrastructure/services/{name}/start` - Start a service
|
||||
- `POST /infrastructure/services/{name}/stop` - Stop a service
|
||||
|
||||
See [Core API Documentation](core-api.md) for full API reference.
|
||||
|
||||
## Advanced Customization
|
||||
|
||||
### Colors
|
||||
|
||||
Edit the CSS variables in the `<style>` section:
|
||||
|
||||
```css
|
||||
.status-running {
|
||||
background: rgba(72, 187, 120, 0.2); /* Green background */
|
||||
color: #48bb78; /* Green text */
|
||||
}
|
||||
```
|
||||
|
||||
### Card Size
|
||||
|
||||
Adjust grid columns:
|
||||
|
||||
```css
|
||||
.service-grid {
|
||||
grid-template-columns: repeat(auto-fill, minmax(300px, 1fr));
|
||||
/* Change 300px to make cards wider/narrower */
|
||||
}
|
||||
```
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Core API Service](core-api.md) - Infrastructure management API
|
||||
- [Stacks Reference](../reference/stacks.md) - All deployed services
|
||||
- [Automation Reference](../reference/AUTOMATION.md) - Portainer REST API
|
||||
Reference in New Issue
Block a user