Over the last several weeks, I have spent a fair amount of time rebuilding parts of my homelab around local language models.
The original goal was straightforward. Run useful models locally, keep more work private, and reduce how often routine tasks needed a cloud API. I started where most people do. I compared benchmarks, quantization levels, context windows, memory use, and tokens per second. I wanted to know which model was best.
That question became less useful the longer I worked on the system. The interesting question stopped being which model to call and became where the work should run at all.
Every request is a workload. Workloads compete for memory, latency, privacy, network access, and human attention. In my system, that creates two separate placement decisions: where the task executes, and where each inference request is served. Model routing answers only the second.
A model that was useful for exploring a repository was not always the one I wanted for a detailed implementation plan. A larger reasoning model could produce a better answer, but not always enough better to justify a cold start, higher memory pressure, or a remote API call. The operational system around the model decides what is allowed to happen next.
The harness is not the placement layer
One early version let each tool choose its own endpoint. That worked until a scheduled task and an interactive coding session competed for the same local resources. Policy had spread across shell scripts, prompts, provider settings, scheduler entries, and environment variables. There was no single answer to why one request stayed local while another went remote.
Modern agent harnesses already do a lot. They can inspect repositories, call tools, manage context, break work into steps, retry operations, generate patches, and review results. Rebuilding those capabilities inside a routing service would mostly create a worse harness.
The layer I care about sits underneath or beside those tools. It makes operational decisions without taking over the task.
Task-execution placement decides where the harness runs, where its tools execute, and where the artifact lands. Inference placement decides which runtime serves a model request. A harness can run locally while calling remote inference. A queue-backed worker can execute the entire task through its own configured inference path. Some workloads need only the inference decision because there is no separate task to place.
Those decisions can share policy without pretending that a model endpoint and a place where an entire task runs are the same kind of resource. That is a narrower responsibility than agent orchestration, but it is also the part an individual harness cannot solve consistently when several tools and workflows share the same infrastructure.
Where managed platforms stop
Managed agent platforms already provide gateways, runtimes, workflow execution, sandboxes, identity, and observability inside their own boundaries. That is useful infrastructure, but it does not answer the earlier question for a system spread across a laptop, a homelab, private workers, and cloud APIs.
Should this workload stay local, wait for local capacity, use an approved cloud inference path, execute on a remote worker, or be rejected because no permitted target exists? That decision happens before the work enters any single platform boundary.
None of this is new to anyone who has run a cluster scheduler. The difference is applying those familiar placement concerns to AI workloads that cross personal, private, and cloud environments, where task execution and inference can be placed independently.
What is running today
The current system has two main operational paths. Both use shared policy mechanisms today; neither is a general-purpose automated scheduler.
The synchronous path handles model calls that can finish inside a bounded request window. A local llm-proxy exposes one API surface and selects among llama.cpp, MLX, and approved cloud inference using caller location, inferred request complexity, WAN availability, explicit overrides, and backend health.
The asynchronous path handles work that should not depend on an HTTP timeout or an open terminal. Scheduled jobs and queue feeders submit work to a shared hub. A remote coding worker handles longer frontier tasks through its configured model path, while a separate private worker handles simpler tasks against local inference. Workers publish run and audit records to a shared ledger.
Today, the platform consists of the following pieces.
- an 18GB M3 MacBook as the operator console and local trigger surface
- always-on llama.cpp inference on a homelab LXC
- an auxiliary MLX endpoint
- a proxy that exposes local and cloud inference through one API
- a queue-backed remote coding worker
- scheduled private agents for mail and homelab automation
- a shared ledger for execution visibility
That is enough to demonstrate the boundary, but not enough to claim a mature placement service. Capacity-aware admission, automated model-residency decisions, and a unified request-level telemetry record do not exist yet.
Think in placement targets, not model names
A harness may request a capability such as coding, summarization, or deeper reasoning. The placement layer should not answer with a model name alone. It should select eligible placement targets that satisfy the workload's operational constraints.
A placement target is the contract around a place where work or inference may run. It combines these operational properties.
locality and data boundary
runtime and execution mode
validated model profile
resource envelope
latency and timeout limits
cost policy
current health and availabilityThe contract can describe a target for the harness, a target for inference, or a target capable of both. Those capabilities remain explicit so task execution and inference do not collapse back into one ambiguous decision.
This matters because the same capability may be served differently depending on the workload. A short private task may stay local end to end. A longer coding task may enter the queue and run on the remote worker through its configured frontier path. An interactive request may keep task execution local while using an approved cloud inference target when local capacity is unavailable.
The harness asks for what the work needs. Policy narrows the targets that are allowed to satisfy it.
Constrained hardware makes placement visible
My local machine is an 18GB M3 MacBook. That is enough unified memory to run useful quantized models, but not enough to pretend capacity does not matter.
A model competes with the operating system, the editor, the browser, containers, and every other process using unified memory. Loading another model may displace one already serving interactive work. A poor placement decision can turn a quick request into a long load followed by an answer that never needed the larger model.
The constraint changed the questions I asked.
- Is enough capacity available to admit this work locally?
- Is a suitable target already ready?
- Is the expected quality improvement worth the startup cost?
- Should background work wait for an interactive session?
- Does the task belong on a queue instead of a synchronous endpoint?
- May the local path degrade to an approved remote target?
Answering them has little to do with how the agent reasons. It has to do with where the work runs and what it is allowed to use.
What placement will need
Memory fit is necessary, but it is not a qualification test. The next step is to attach runtime profiles to eligible inference targets, covering runtime version, quantization, context budget, and observed latency and timeout behavior. Placement can then choose from combinations that have actually been validated.
That deserves its own article. The architectural point is simpler: placement should select qualified runtime-and-model combinations, not names from a catalog.
Admission and degradation deserve their own treatment as well. A caller should eventually receive a clear operational result:
admitted
queued
redirected to another eligible target
rejected by capacity or policyThe policy mechanics behind those outcomes (priority, capacity, privacy, cost, queue depth, and explicit refusal) belong in the next placement article.
Observability is about the operational decision
The infrastructure layer does not need to record every reasoning step inside an agent. The harness already has its own execution logs.
What I need from the policy and placement layer is enough evidence to answer a different set of questions.
- Why did this workload run locally or remotely?
- Why did it wait?
- Which policy and runtime profile were selected?
- What was queue time versus startup and generation time?
- How did the request change local capacity or remote cost?
- Did the platform use a degraded path?
Some of this exists today. The proxy reports the selected backend, model, route reason, inferred complexity, latency, and per-backend request and token counts. Queue-backed workers and scheduled jobs write run records to the shared ledger.
The missing piece is a unified request record connecting the original workload identity, placement decision, runtime state, and final result across both paths. The implementation belongs in a later observability article; the architectural point is that the decision should be inspectable.
The boundary is what makes the system durable
The model landscape changes quickly. Agent harnesses will keep absorbing more planning, tool use, context management, and workflow logic. That is a reason to keep the placement layer narrow, not to expand it into another harness.
The durable parts are straightforward.
- harnesses declare workload requirements
- policy determines eligible execution and inference targets
- admission protects shared capacity
- placement accounts for qualified profiles and current state
- degradation stays inside locality and cost boundaries
- observability records the decision and its impact
An 18GB machine makes these concerns visible early, but they are not unique to a laptop. The same boundary appears anywhere multiple workloads share inference or execution capacity: a workstation, a homelab cluster, a small GPU pool, or a larger internal platform.
I still compare models. Benchmarks, quantization, context limits, and runtime performance all matter. They just do not answer the placement question.
The harness runs the task. The model performs the inference. The runtime hosts the model or executes the workload. Policy governs which eligible targets may be used.
Routing picks a model. Placement decides where the work is allowed to run.
Coming next
This article intentionally stays at the architectural level. The next posts will go deeper into the operational pieces.
- Measured Systems, Not Benchmark Systems: runtime profiles, deployment qualification, and why fitting in memory is not enough
- Placement Policy in Practice: admission, priority, locality, cost, capacity, and policy-governed degradation
- Operating a Remote AI Runtime: queues, workers, isolation, lifecycle, artifact delivery, and verification
- Observability for AI Infrastructure: measuring placement, capacity, latency, reliability, and cost rather than agent reasoning
- Applying SRE Principles to AI Infrastructure: SLOs, error budgets, failure modes, recovery, and production readiness