I was annoyed at how Claude Code ignores my CLAUDE.md and skills, so I was looking for ways to expand type checking to them. So I wrote a wrapper on top of claude-agents-sdk that reads my CLAUDE.md and skills, and compiles them into rules - could be linter rules or custom checking scripts. Then it hooks up to all tools and runs the checks. The self improving part comes if some rule doesn't work: I run the tool with the session id in review mode, it proposes the fixes and improves the rule checkers. (not the md files) So it's kinda like vibe coding rules, definitely lowers the bar for me to maintain them. Repo: https://github.com/chebykinn/agent-ruler
This, IMO is the biggest insight into where we're at and where we're going:
> Because both evaluation and self-modification are coding tasks, gains in coding ability can translate into gains in self-improvement ability.
There's a thing that I've noticed early into LLMs: once they unlock one capability, you can use that capability to compose stuff and improve on other, related or not, capabilities. For example "reflexion" goes into coding - hey, this didn't work, let me try ... Then "tools". Then "reflxion" + "tools". And so on.
You can get workflows that have individual parts that aren't so precise become better by composing them, and letting one component influence the other. Like e2e coding gets better by checking with "gof" tools (linters, compilers, etc). Then it gets even better by adding a coding review stage. Then it gets even better by adding a static analysis phase.
Now we're seeing this all converge on "self improving" by combining "improving" components. And so on. This is really cool.
OTOH, there's loads you can do for evaluation before a human even sees the artifact. Things like does the site load, does it behave the same, did anything major change on the happy path, etc etc. There's a recent-ish paper where instead of classic "LLM as a judge" they used LLMs to come up with rubrics, and other instances check original prompt + rubrics on a binary scale. Saw improvements in a lot of evaluations.
Then there's "evaluate by having an agent do it" for any documentation tracking. Say you have a project, you implement a feature, and document the changes. Then you can have an agent take that documentation and "try it out". Should give you much faster feedback loops.
Another thing that get quantized is video preferences to maximize engagement.
Larger composition, though, starts to run into typical software design problems, like dependency graphs, shared state, how to upgrade, etc.
I've been working on this front for over two years now too: https://github.com/smartcomputer-ai/agent-os/
Its is the core of any and all learning/exellency; exposure to chaotic perturbations allow selection of solutions that are then generalized to further, ever more straining problems; producing increasingly applicable solutions.
This is the core of evolution, and is actually derivable from just a single rule.
On this view, learning in general operates via selection under uncertainty. This is less visible in individual cognition, where we tend to over-attribute agency, but it is explicit in science: hypotheses are proposed, subjected to tests, and selectively retained, precisely because the future cannot be deduced from the present.
In that sense, generation/discrimination is a particular implementation of this broader principle (a way of instantiating variation and selection) not the primitive itself.
I've always felt that the most important part of engineering was feedback loops.
Maybe nature is the greatest engineer ever?
The great oxygenation was a time of near catyclismsic scarcity for most complex organisms, as resources scale to food/energy requirements imply the most complex organisms were the most dependent on the environment, and were most impacted by changes.
Inversely, oxygenation was our most crucial abundancy pre cursor, as it provides a large substrate chemically for life to exhibit
Abstract:
Self-improving AI systems aim to reduce reliance on human engineering by learning to improve their own learning and problem-solving processes. Existing approaches to self-improvement rely on fixed, handcrafted meta-level mechanisms, fundamentally limiting how fast such systems can improve. The Darwin Gödel Machine (DGM) demonstrates open-ended self-improvement in coding by repeatedly generating and evaluating self-modified variants. Because both evaluation and self-modification are coding tasks, gains in coding ability can translate into gains in self-improvement ability. However, this alignment does not generally hold beyond coding domains. We introduce \textbf{hyperagents}, self-referential agents that integrate a task agent (which solves the target task) and a meta agent (which modifies itself and the task agent) into a single editable program. Crucially, the meta-level modification procedure is itself editable, enabling metacognitive self-modification, improving not only the task-solving behavior, but also the mechanism that generates future improvements. We instantiate this framework by extending DGM to create DGM-Hyperagents (DGM-H), eliminating the assumption of domain-specific alignment between task performance and self-modification skill to potentially support self-accelerating progress on any computable task. Across diverse domains, the DGM-H improves performance over time and outperforms baselines without self-improvement or open-ended exploration, as well as prior self-improving systems. Furthermore, the DGM-H improves the process by which it generates new agents (e.g., persistent memory, performance tracking), and these meta-level improvements transfer across domains and accumulate across runs. DGM-Hyperagents offer a glimpse of open-ended AI systems that do not merely search for better solutions, but continually improve their search for how to improve.
Here is a breakdown - https://vectree.io/c/plant-self-incompatibility-logic
But this idea of having a task agent & meta agent maybe has wings. Neat submission.
I've had to really shift how I think about building code bases, alot of logic can go into claude skills and sub agents. Requires essentially relearning software engineering