If I put several AI models on a 128GB Mac, who assigns the work?

Why do I want to talk to just one AI?

I am developing Cookly, KnitLens and other apps. I want to add local LLMs and image/video work to that environment. Using each tool separately would mean explaining the previous work again more often.

I want one conversation. Ask, “Analyze and fix the error in Cookly,” and have the agent find the project, split the work, finish it and bring back the result. I would rather not choose the model every time. That is also the kind of experience I would want from a personal AI such as ChatGPT Dots.

This follows from EP05. I wanted a development environment that could handle several AI workloads alongside my apps, rather than only the fastest single LLM. The ordered M5 Max 128GB / 1TB has not arrived, but I have already tried a small tiered pipeline on the 8GB iMac.

Here I want to separate what that experiment established from the system I hope to build with Hermes at the front.

Hermes as the main personal agent

Hermes is my candidate for the main agent that talks to me and manages the work. Telegram is already connected. I want to combine project context, ongoing tasks, tools and delegated agents behind it.

For an error-fixing request, Hermes would retrieve relevant memory and code, then divide the work into investigation, changes and tests. It would pass context and permissions to workers, collect results, and report what changed and which checks passed. Hermes itself still needs a language model for planning and tool calls.

EP06 / Target design

Behind one conversation

User — one conversation

Telegram · Paseo · Mac

↓

Hermes Agent

Intent / planning · Tasks / sessions · Long-term memory

Tool / agent delegation · Progress · Result aggregation

↓

Policy

Privacy · Cloud permission · Action permissions

Router

Task type / difficulty · Choose execution target

↓

Local

General / coding LLM Small decision model

OpenCode Go

Permitted cloud work Check model and plan

Codex

Complex coding work CLI tested · ACP pending

Specialist workers / tools

Vision · ComfyUI · STT / TTS · Documents · Files · Code

↓

Judge / Verification

Deterministic checks → Semantic assessment → Quality policy

Retry / escalation returns to Policy / Router

↓

Hermes → User

Result · Verification status · Execution path · Open issues

This is a target architecture, not a claim that Hermes supplies every box. Its documented delegation feature provides a basis for child-agent work and returned results. Memory, session search and Kanban task state are also documented. Connecting these to my projects and adding approval and verification policies remains work to do.

In particular, a configured model for delegate_task is not a task-specific model router. Current documentation distinguishes a shared child-model configuration from paths such as Kanban for per-task overrides. Delegation support alone does not complete automatic model selection.

The router chooses where work goes; the judge checks what comes back. Policy supplies the privacy, permission and cost constraints for both. Combining these roles makes it too easy to skip verification because the system supposedly picked a good model.

A tiered system starting with a small local model

The baseline is Local First → Verify → Escalate Only When Necessary. Choose a local worker for suitable task types and difficulty, check its result, then retry or change execution targets if needed. Verify again after handing off.

The path I am considering is Local → OpenCode Go → Codex. It is neither a compulsory sequence nor a performance ranking. OpenCode Go is a coding-model provider and plan, not one model or an agent runtime. Running OpenCode as a coding worker and calling a Go model through its API also need distinct integrations.

EP06 / Target design

Local First → Verify → Escalate

Policy → Router

Check privacy / permission, then choose starting route

LOCAL_ONLY / CLOUD_OK

↓

Local → Judge

PASS: return with verification status

RETRY: local retry within attempt / time budget

ESCALATE: recheck cloud permission

↓

LOCAL_ONLY

Block cloud Local retry, stop or ask user

CLOUD_OK

Inspect outgoing material Check delegated task and permissions

↓ Only if permitted

OpenCode Go → Judge

PASS → return / failure → consider next route

↓ If needed and permitted

Codex → Judge

Return if checks pass · Otherwise stop or seek review

Final result → Hermes → User

Report execution route and unresolved issues

A clearly difficult code change can start with Codex. Repeated local failures before attempting the same work elsewhere can cost more time and money. Conversely, LOCAL_ONLY blocks cloud escalation even when the result is inadequate. The choices then are local retry, asking me, or stopping.

This policy cannot apply only to workers. If the main Hermes model is hosted in the cloud, the request has already left the computer. For sensitive work, LOCAL_ONLY must cover the main model, judge, search tools and exported logs. CLOUD_OK does not permit sending every file either. I want to select outgoing code and logs and apply task-specific permissions to file changes or external-service calls.

I reviewed Weave Router (weave-os/router) as a routing candidate. Its official repository and proxy implementation cover model selection for API actions, provider-format translation and compatible endpoints. That does not establish automatic Mac memory monitoring, model loading/unloading or escalation driven by my failed quality checks.

My router would ideally consider task type and complexity, model success on previous tasks, loaded models and memory, context length, cost and delay. Privacy policy should filter candidates first. I still need to establish which parts Weave can handle and which memory or quality policies need integration. Other projects also use the name Weave; the repository identity matters.

How do I decide a result has failed?

If a local model says it does not know, handing off is easy. Confidently wrong answers are harder. Asking the model whether it did a good job is not enough.

I want to check mechanically verifiable requirements first: JSON Schema, mandatory fields and output format, generated files, compilation and unit tests. Then assess requirements, relevance, logic, evidence and missing parts of the plan.

Check or policy	What it examines	Next action
Deterministic checks	Schema, fields, files, compilation and tests	Fix, retry, escalate or stop after failure
Semantic checks	Requirements, relevance, logic, evidence and plan completeness	Record the shortfall and seek more work or review
Execution policy	Complexity, likely success, remaining time, cost, context and risk	Choose within permitted targets and retry budgets

JevMLX is an experimental candidate for lightweight classification and semantic assessment. More precisely, it is a tool for evaluating predefined choices with MLX models and assembling structured decisions, rather than the name of a standalone judge model. Its underlying model and decision schema matter. Choice scores cannot simply be read as the probability that an answer is true.

I could separate questions such as whether requirements are met, whether evidence is missing and whether more review is needed. But a semantic PASS must not override a failed test. Tests can also pass while the implementation addresses the wrong requirement. Semantic assessments are fallible too; PASS is not a correctness guarantee.

A handoff needs failed-check results and relevant context. If files were already changed, retain the diff and prevent repeated side effects. I want attempt, time and cost limits rather than endless retries, with unresolved work reported as unresolved.

The PoC already tried on the 8GB iMac

This is not an entirely new idea. As briefly mentioned in EP00, I ran a limited PoC in September 2026 on an M1 iMac with 8GB unified memory. Ollama served Qwen2.5 1.5B Q4 as the local starting point; the setup also used OpenCode Go, Codex CLI, QwenPaw and Paseo.

I checked these three routes under controlled tasks and verification conditions. This is a record of working paths, not a newly measured performance table.

Route	Observed behavior
A · Local PASS	The small local model completed a task, passed verification and returned the result
B · Local → OpenCode Go	Failed local verification led to a real Go call
C · Local → OpenCode Go → Codex CLI	The first two tiers failed and a real Codex CLI invocation performed the final work

All three worked in those tests. I also separately checked a QwenPaw command-plugin invocation and a remote iPhone request through Paseo. That gave me a starting point for connecting tiers even on 8GB.

It did not complete a general personal agent for arbitrary requests. JevMLX semantic verification was not applied, and Codex was called directly through CLI rather than ACP. QwenPaw native ACP delegation remained unfinished. Cloud routes B and C were not tested from the iPhone, and the whole pipeline has not been tested with Hermes.

The PoC established that controlled verification could drive real provider calls and route changes. It did not prove general answer quality or the ability to diagnose unfamiliar app errors.

What changes with the 128GB Mac Studio?

Initially I wanted to know how large a model I could leave running. Now I am more interested in using a small, sufficiently good model frequently and loading heavier models when needed. App development needs its share of memory too.

Resident candidates are a medium general LLM, a small judge/router model, and STT or embeddings if needed. Large Qwen or DeepSeek models, vision, image, video and music models are on-demand candidates. Versions and quantization are not fixed. I do not plan to load them all simultaneously.

EP06 / Target design

Resident models and on-demand models

Resident candidates

• Medium general LLM
• Small judge / router
• STT or embeddings when needed

Load on demand

• Large Qwen / DeepSeek candidates
• Vision · Image · Video · Music

↓

Check memory and active tasks first

macOS · Xcode · Simulator · KV Cache · Runtime buffers

↓

1. Finish and unload less-used models if necessary
2. Load needed model → execute → verify
3. Return to baseline after the task

The weights, KV Cache and execution space discussed in EP02 must share the budget with development tools. Task priority and active sessions should guide unloading less-used models and loading the required worker. Ending a process does not mean every other buffer or the OS file cache disappears.

After arrival I want to record peak unified memory, KV Cache, swap and concurrent sessions, together with model load/unload latency. I will measure router and judge time separately, and the time from sending a request to receiving a verified result.

EP01 separates token performance into Prefill and Decode. Fast Decode does not guarantee fast task completion. Even with acceleration from EP03 or streaming from EP04, loading, tools, verification and retries belong in the end-to-end measurement.

Long-term memory, projects and specialist workers

I do not want to explain Cookly from scratch whenever I ask for a fix. Nor can I keep every project and conversation in one growing prompt. I would divide memory like this.

Memory layer	Contents
User Memory	Preferences, recurring workflows, approval policy and frequently used tools
Project Memory	Relevant context for Cookly, KnitLens, Delsa, Sobok, Sound Travel and Personal AI Lab
Session Memory	Current goal, progress, results, failures and retries
Knowledge Base	Obsidian Vault, development journal, technical documents and benchmark records

These are my logical categories, not four built-in Hermes stores with those names. Its official memory feature provides curated notes, user information and past-session search. I plan to retrieve relevant records into the current task and separate session context when changing projects.

Memory needs provenance and scope, with ways to edit, delete and search it. Approval policy must also be enforced by permission settings rather than only summarized as a memory. Searching old conversations implies retained history, so retention and log cleanup need decisions too. Keys and credentials belong outside prompts and general memory. Obsidian is one knowledge store.

Specialist workers follow the same separation. Coding candidates include a local coding LLM, OpenCode and Codex; images and video can use ComfyUI workflows; speech can use STT/TTS; information processing can use document, search, summary and file tools. I do not intend to replace every worker with Hermes’ conversation model.

EP06 / Target design

Split one request, bring results back together

“Build a Cookly feature and make its launch video”

Clarify feature, public material and action permissions

↓

Hermes → Task decomposition

Identify dependencies between code and video work

↓

Coding worker

Local Coding LLM / OpenCode / Codex

Investigate / implement → build / test → change summary

Production worker

Vision / ComfyUI / STT / TTS

Approved screens / script → image / video / audio

↓

Judge / Human review

Check functionality / requirements and video screens / wording

↓

Hermes → Aggregate results

Changes · Tests · Media files · Actions needing approval

For “Build a new Cookly feature and make its launch video,” first clarify the feature and what material may be public. A coding worker handles implementation and tests; a production worker creates video and audio from approved screens and scripts. A video needing the actual feature screen must wait for the code result. Parallelizing everything would ignore those dependencies.

Hermes would bring back changes, tests, media files and remaining approval requests. The same system might organize development notes or draft a blog post. Those are individual use cases, rather than the purpose of its memory system.

What remains to build?

First I want Hermes to send a task to a local worker and receive its result after deterministic checks. Then reproduce PoC routes A, B and C through that integration before adding JevMLX semantic decisions and task-dependent routing. If the main model cannot call tools reliably, changing only worker models will not fix it.

The entry points I am considering are Telegram, Paseo and the Mac. Paseo iPhone requests worked in the small PoC, but direct Hermes integration is separate. Paseo provider documentation covers Codex, OpenCode and custom ACP providers; that does not establish access to the same Hermes session. Authentication, project permissions, session isolation and protected logs still need checks.

Codex connections need distinction too. Direct CLI invocation is the PoC path. The Codex ACP adapter is a separate integration, and is not the same as Hermes’ documented Codex app-server runtime. Cancellation, permission prompts, resumption and error returns need real tests before I call ACP delegation complete.

For cloud use I want outgoing-material and action-permission checks, plus records of routes, models, checks, retries and usage. Logs must avoid retaining sensitive material. Signing into Codex with a ChatGPT subscription is separate from metered OpenAI API billing. Go subscription allowances and additional-usage policies also need separate tracking. I have no evidence yet that this architecture saves money.

Several integrations remain unverified: Hermes with the tiered pipeline, JevMLX as a judge, Weave with my task policy, Codex ACP, iPhone cloud escalation and automatic memory switching. Adding more models does not need to be the first step.

I want to begin with one “Analyze the Cookly error” request reaching the correct project, completing work within its permissions and returning the checks. Whether it finishes locally or reaches Codex, my question is the same: did the task actually get done?

Sources and verification scope

Official sources checked on 2026-10-10. Features were checked within the documented/source scope and will be checked against installed versions. The PoC account describes my September 2026 tests, not a rerun during this edit or measurements from the Mac Studio.

• Hermes: delegation, Kanban, memory, Codex app-server runtime.
• Weave Router: official repository, action semantics, configuration, proxy implementation. Hermes/Mac residency integration is untested.
• JevMLX: official repository, architecture. It is not yet integrated as my semantic judge.
• Connections and providers: OpenCode Go, OpenCode ACP, Paseo providers, Codex ACP adapter.