I’m so over agent harnesses, prompting, and hand-holding AI agents. Now I have a team of agents that use meta-harnesses, that use agents harnessed on my behalf and only bring me in on escalations.
This is Agent Lab. It’s a system of autonomous teams, each with a mandate to build an application for me. The apps are all home-cooked meal style software personalized to how I do things, with the first one being Agent Lab itself.
Who’s on the Teams
Each team has a lead and two coder agents. The lead decides what to build next, triages tickets, updates processes, and escalates blockers. The coders run coding sessions, make pull requests, review each-others work, and merge code.
Separate from any team is the Principal agent (using Fable) who is the avatar of my will within the system. It resolves blockers, makes decisions on my behalf, and helps decide what needs my call. Its mandate is to ensure the agents work smoothly.
These are the two current teams. They started getting names after coder-1 and coder-2 were instantiated:
Every agent gets:
- a name, a role, and a system prompt
- a Matrix user to chat with other agents and me
- a Forgejo user to pull tickets, open pull requests, and merge them
- an Omnigent user for coding sessions
- a Hindsight memory bank
- a token for Agent Lab’s MCP tools
Each team has it’s own Matrix channel and is responsible for a repository, but can make pull requests anywhere. The Principal agent also has a channel shared with me.
How the agents work
Here’s a real bug ticket going through the team. I don’t do anything.
Stanbly spotted an issue with Agent Lab and created a Forgejo issue noting the problem. Jenda ran a short coding session to investigate and write up acceptance criteria on the issue, then changed the issues status to “ready for development”. Coder-2 woke-up on the Forgejo event and starts working on the ticket in a coding session. Coder-1 reviews, asks for changes, etc then eventually merges. The agents decided that reviewers merge the pull requests once all the fixes are in.
How the system works
There are several components wired together:
- Agent Lab web orchestrator, agent loops, and web app
- Matrix server for chat
- Forgejo for tickets and code
- Omnigent for coding sessions
- Kubernetes for ephemeral sandbox pods
- Hindsight for memory
- LiteLLM for routing inference
Agent Lab is a Hanami web app that does a few things:
- Orchestrator listens for events (from Matrix, Forgejo, etc) and wakes up agents to handle them
- MCP server gives the agents and coding sessions tools to use
- Agent loops run one loop for each agent in the teams. They wake up on a schedule or when the orchestrator gives them an event.
- Web UI for me to see what’s going on and configure the agents
There’s a distinction here between two types of agent turns:
- Reactive turns are in the AgentLab loops and react to the schedule or events.
- Coding turns are in the coding sessions that run within Omnigent on agent harnesses like OpenCode.
Tools
Agent Lab defines tools that can be used for MCP or within the reactive turns directly through RubyLLM. Agents can write their own tools and include them (through a pull request). Currently they have tools such as
- Forgejo: manage issues, pull requests, labels, merge PRs. They included some guardrails such as merging requires CI to be green.
- Matrix: send/read messages, create rooms, invite users to rooms.
- Omnigent: manage coding sessions, share coding sessions with me.
- Agent Management: create new agents, create teams, add agents to teams, read recent agent turns.
- Network Access: request time-limited access to specific domain (I approve in the Agent Lab UI).
The agents are expected to unblock themselves: check whether they have the tools to do what they want, and if they don’t, build them. An agent asked me to create a Matrix room and invite someone (one of their agent buddies) to it. I told them to “do it yourself” and they built the matrix_create_room and matrix_invite_user tools.
Coding sessions
The coding sessions are started in and run using the Agent Lab tools. They use the Omnigent API which is pretty great. There’s a custom Omnigent sandbox provider plugin that will create ephemeral Kubernetes pods for the session to run within. It’s a slightly customized version of the built-in k8s provider plugin which allows pods to run userspace pods within them.
The “brief” is a prompt that gives info on identity, environment, credentials, best practices, how to use git with Forgejo, how to access memory, and why the session was started.
Network egress
Sandbox pods are locked down so they can’t access the network by default. This done with:
- Allowlist DNS server answers with the egress proxy’s IP for approved hostnames only.
- NetworkPolicy lets the pod reach only that DNS server, the egress proxy, and the Omnigent server.
- Nginx egress proxy denies/allows network access and also transparently swaps fake credentials within the pod for real credentials in HTTP headers.
Git pushes work a little differently with token swapping. The egress proxy gets the key on-demand from AgentLab.
If an agent needs to access a domain it can use a network request tool in AgentLab to provide a domain and a rationale. The Agent Lab UI will let me approve or deny the network access and it automatically expires. The egress proxy will dynamically start allowing that domain when I approve it.
Memory
In the reactive loops the agents are not running between turns and Agent Lab restarts a lot (when the agents make changes). The agents get their context on new turns from:
- Room history: team agents will receive the latest messages from their team chat room.
- Compact context: the previous sessions context is compacted and stored in the database
- Hindsight: Each agent has it’s own memory bank where it chooses what to retain and recall. Coding sessions also have per-repository memory banks that are shared across sessions.
Agents can also use tools to gather more information from Forgejo, Hindsight, Matrix, etc.
After Bootstrapping
After Agent Lab had bootstrapped itself I asked Stanbly to insantiate a new team for building a personal budgeting app “FinanceBuddy” for me.
Crispopher turned a diatribe about what I wanted (more on this below) into a series of stories for the agents to build. Willaim and Robart soon built several new features like adding transactions manually, searching, CSV export, editing categories, inline editing, budgeting goals, reports, bank imports.
What do I do
There is a few things where I still pickup an agent harness and collaborate directly with an agent:
- Vision: My diatribes on what I want turns into building a
VISION.mddocument that describes my north star goals for the app and what they need to align with whenever they decide on a new feature. - Design: I have specific tastes in the design of the applications. I’ll write a
DESIGN.mdand create mock-ups/prototypes to define what I want it to look and feel like. - Research: Sometimes I have to do research myself with an agent. To research my bank export, I wasn’t going to give the agents access to my bank. I downloaded them myself and produced anonymized documentation on the format.
- Escalations: For example agents noted that CI was slow, investigated, and asked me to add more runners.
Key pieces
These are what I think are the important bits right now:
- Agent identities: each agent gets it’s own accounts and long-term memory
- Kanban board: pull-based workflow where I can see what’s going on
- Chat room: Both for the agents to chat and for me to direct them.
- Coding sandboxes: I don’t have to worry about secrets or network access.
- Memory: Both individual banks for agents to build their own expertise. And shared application banks to know how the project works.