chore(publication): close pre-integration release blockers

This commit is contained in:
Alexandre Teixeira
2026-10-05 01:37:49 +01:00
parent 3d3aee2093
commit dab660543b
77 changed files with 4074 additions and 83103 deletions
+37
View File
@@ -0,0 +1,37 @@
# Runtime model catalogs
The shipped `hf_models.json` and `mlx_community_models.json` contain independently
authored empty JSON lists (`[]`). They contain no copied upstream rows,
descriptions, weights or metadata. A fresh offline installation has no catalog
recommendations until user data has been populated.
Use **Rescan** in Cookbook while online (the models API accepts
`refresh_catalog=1`). Existing discovery code fetches selected Hugging Face
organization collections and MLX community collections into `DATA_DIR/hwfit/`:
`hf_collection_models.json` and `mlx_community_models.json`. These runtime cache
files retain source/fetch timestamps and model rows, with the existing 24-hour
freshness policy. Forced refresh bypasses freshness, invalidates the merged
in-memory catalog, and preserves the existing cache/network failure behavior.
Previously populated caches can supply offline results; a failed cold refresh
leaves an explicit empty-state message. Refresh does not write shipped lists.
Maintenance commands from the repository root:
```
python scripts/add_hwfit_models.py
python scripts/backfill_model_release_dates.py --dry-run
python scripts/import_from_vllm_recipes.py --dry-run
```
Their catalog path is `DATA_DIR/hwfit/hf_models.json`, merged before runtime
collection caches. The add/import commands can initialize a missing runtime
catalog; backfill requires one. These commands use Hub metadata and, for recipe
import, vLLM recipe inputs. Runtime metadata is user data, not an approved
redistributable snapshot. Publishing any populated catalog requires its own
source/rights review; metadata, copied descriptions and model-weight licenses
are distinct. Do not copy runtime results into these repository lists.
Tests use `tests/fixtures/hwfit_publication_models.json` through the scoped
`tests/hwfit_publication_fixtures.py` fixture. The rows are independently authored
synthetic test inputs. No production snapshot is required to test platform,
quantization and GGUF behavior.
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+5 -2
View File
@@ -282,7 +282,7 @@ def reset_model_cache():
def refresh_dynamic_catalogs(force=False):
"""Refresh API-backed model catalogs and invalidate the merged cache.
The bundled JSON files remain the offline fallback. Dynamic catalogs live
The bundled JSON lists are intentionally empty. Dynamic catalogs live
under DATA_DIR so runtime refreshes do not dirty the source tree.
"""
from services.hwfit.hf_discovery import (
@@ -324,6 +324,7 @@ def get_models():
seen.add(name)
rows.append(_normalize_model_entry(model))
_append_models(_load_model_file(model_catalog_path()))
for model in _load_model_file(data_path):
if not isinstance(model, dict):
continue
@@ -340,4 +341,6 @@ def get_models():
def model_catalog_path():
return os.path.join(os.path.dirname(__file__), "data", "hf_models.json")
"""Mutable user catalog populated by the maintenance scripts."""
from src.constants import DATA_DIR
return os.path.join(DATA_DIR, "hwfit", "hf_models.json")