> The enterprise mindset dictates that you need an out-of-process database server. But the truth is, a local SQLite file communicating over the C-interface or memory is orders of magnitude faster than making a TCP network hop to a remote Postgres server.
I don't want to diss SQLite because it is awesome and more than adequate for many/most web apps but you can connect to Postgres (or any DB really) on localhost over a Unix domain socket and avoid nearly all of the overhead.
It's not much harder to use than SQLite, you get all of the Postgres features, it's easier to run reports or whatever on the live db from a different box, and much easier if it comes time to setup a read replica, HA, or run the DB on a different box from the app.
I don't think running Postgres on the same box as your app is the same class of optimistic over provisioning as setting up a kubernetes cluster.
Sqlite smokes postgres on the same machine even with domain sockets [1]. This is before you get into using multiple sqlite database.
What features postgres offers over sqlite in the context of running on a single machine with a monolithic app? Application functions [2] means you can extend it however you need with the same language you use to build your application. It also has a much better backup and replication story thanks to litestream [3].
The main problem with sqlite is the defaults are not great and you should really use it with separate read and write connections where the application manages the write queue rather than letting sqlite handle it.
Actually, there are no inserts in this example each transaction in 2 updates with a logical transaction that can be rolled back (savepoint). So in raw terms you are talking 200k updates per second and 600k reads per second (as there's a 75%/25% read/write mix in that example). Also worth keeping in mind updates are slower than inserts.
> no indexes.
The tables have an index on the primary key with a billion rows. More indexes would add write amplification which would affect both databases negatively (likely PG more).
> Also, I didn't get why sqlite was allowed to do batching and pgsql was not.
Interactive transactions [1] are very hard to batch over a network. To get the same effect you'd have to limit PG to a single connection (deafeating the point of MVCC).
- [1] An interactive transaction is a transaction where you intermingle database queries and application logic (running on the application).
Not the person you are responding to, but sqlite is single threaded (even in multi process, you get one write transaction at a time).
So, if you have a network server that does BEGIN TRANSACTION (process 1000 requests) COMMIT (send 1000 acks to clients), with sqlite, your rollback rate from conflicts will be zero.
For PG with multiple clients, it’ll tend to 100% rollbacks if the transactions can conflict at all.
You could configure PG to only allow one network connection at a time, and get a similar effect, but then you’re paying for MVCC, and a bunch of other stuff that you don’t need.
Sqlite supports nested transactions with SAVEPOINT so each client can have their own logical transaction that can be rolled back. The outer transaction just batches the fsync effectively. So an individual client failing a transaction doesn't cause the batch to fail. But, a crash would cause the batch to fail. Because, it's a single writer, there's no rollback/retries from contention/MVCC.
You could try to imitate this in postgresql but the problem is the outer transaction does not eliminate the network hops for each inner/client transaction so you don't gain anything doing it and you still have the contention problem which will cause rollbacks/retries. You could reduce your number of connections to one to eliminate contention. But, then you are just playing sqlite's game.
An interactive transaction works like this in pseudo code.
beginTx
// query to get some data (network hop)
result = exec(query1)
// application code that needs to run in the application
safeResult = transformAndValidate(result)
// query to write the data (network hop)
exec(query2, safeResult)
endTx
How would you batch this in postgres and get any value? You can nest them all in a single transaction. But, because they are interactive transactions that doesn't reduce your number of network hops.
The only thing you can batch in postgres to avoid network hops is bulk inserts/updates.
But, the minute you have interactive transactions you cannot batch and gain anything when there is a network.
Your best bet is to not have an interactive transaction and port all of that application code to a stored procedure.
Thing is though - either of those options is still multiple orders of magnitude faster than running on a remote host. Either will work, either will scale way farther than you reasonably expect it to.
> What features postgres offers over sqlite in the context of running on a single machine with a monolithic app
The same thing SQL itself buys you: flexibility for unforeseen use cases and growth.
Your SQLite benchmark is based in having just one write connection for SQLite but all eight writable connections for Postgres. Even in the context of a single app, not everyone wants to be tied down that way, particularly when thinking how it might evolve.
If we know our app would not need to evolve we could really maximize performance and use a bespoke database instead of an rdbms.
It seems a little aggressive for you to jump on a comment about how it’s reasonable to run Postgres sometimes with “SQLite smokes it in performance.” That’s true, when you can accept its serious constraints.
As a wise man once said, “Postgres is great and there's nothing wrong with using it!”
> Sqlite smokes postgres on the same machine even with domain sockets [1].
SQLite on the same machine is akin to calling fwrite. That's fine. This is also a system constraint as it forces a one-database-per-instance design, with no data shared across nodes. This is fine if you're putting together a site for your neighborhood's mom and pop shop, but once you need to handle a request baseline beyond a few hundreds TPS and you need to serve traffic beyond your local region then you have no alternative other than to have more than one instance of your service running in parallel. You can continue to shoehorn your one-database-per-service pattern onto the design, but you're now compelled to find "clever" strategies to sync state across nodes.
Those who know better to not do "clever" simply slap a Postgres node and call it a day.
> SQLite on the same machine is akin to calling fwrite.
Actually 35% faster than fwrite [1].
> This is also a system constraint as it forces a one-database-per-instance design
You can scale incredibly far on a single node and have much better up time than github or anthropic. At this rate maybe even AWS/cloudflare.
> you need to serve traffic beyond your local region
Postgres still has a single node that can write. So most of the time you end up region sharding anyway. Sharding SQLite is straight forward.
> This is fine if you're putting together a site for your neighborhood's mom and pop shop, but once you need to handle a request baseline beyond a few hundreds TPS
It's actually pretty good for running a real time multiplayer app with a billion datapoints on a 5$ VPS [2]. There's nothing clever going on here, all the state is on the server and the backend is fast.
> but you're now compelled to find "clever" strategies to sync state across nodes.
That's the neat part you don't. Because, for most things that are not uplink limited (being a CDN, Netflix, Dropbox) a single node is all you need.
May be an "out" there question, but any tech book suggestions you'd recommend that can teach an average dev on how to build highly performant software with minimal systems?
I feel like the advice from people with your experience is worth way way way way more than what you'd hear from big tech. Like what you said yourself, big tech tends to recommend extremely complicated systems that only seem worth maintaining if you have a trillion dollar monopoly behind it.
Not specific books per say. Though I'd advise starting with some constraints. As that really helps you focus.
Your reading/learning material can spin out of those constraints.
So for me my recent constraints were:
1. Multiplayer/collaborative web apps built by small teams.
2. Single box.
3. I like writing lisp.
So single box pushes me towards a faster language, and something that's easy to deploy. Go would be the natural choice here, but I want a lisp so Clojure is probably the best option here (helps that I already know it). JVM is fast enough and has a pretty good deployment story. Multiplayer web apps, pushed me to explore distributed state vs streaming with centralised state. This became a whole journey which ended with Datastar [1]. Thing is immediate mode streaming HTML needs your database queries to be fast and that's how I ended up on SQLite (I was already a fan, and had used it in production before), but the constraints of streaming HTML forced me to revisit it in anger.
Your constraints could be completely different. They could be:
1. Fast to market.
2. Minimise risk.
3. Mobile + Web
4. Try something new.
Fast to market might mean you go with something like Rails/Django. Minimise risk might mean you go with Rails because you have a load of experience with it. Mobile + web means you read up on Hotwire. Try something new might mean you push more logic into stored procedures and SQL queries so you can get the most out of Postgres and make your Rails app faster. So you read The Art of Postgresql [2] (great book). Or maybe you try hosting rails on a VPS and set up/manage your own postgres instance.
A few companies back mine were:
1. JVM but with a more ruby/rails like development experience.
2. Mobile but not separate iOS/Android projects.
3. Avoid the pain of app store releases.
4. You can't innovate everywhere.
That meant Clojure. React native. Minimal clients with as much driven from the backend as possible. Sticking to postgres and Heroku because it's what we knew and worked well enough.
Backups, litestream gives you streaming replication to the second.
Deployment, caddy holds open incoming connections whilst your app drains the current request queue and restarts. This is all sub second and imperceptible. You can do fancier things than this with two version of the app running on the same box if that's your thing. In my case I can also hot patch the running app as it's the JVM.
Server hard drive failing etc you have a few options:
1. Spin up a new server/VPS and litestream the backup (the application automatically does this on start).
2. If your data is truly colossal have a warm backup VPS with a snapshot of the data so litestream has to stream less data.
Pretty easy to have 3 to 4 9s of availability this way (which is more than github, anthropic etc).
My understanding is litestream can lose data if a crash occurs before the backup replication to object storage. This makes it an unfair comparison to a Postgres in RDS for example?
Last I checked RDS uploads transaction logs for DB instances to Amazon S3 every five minutes. Litestream by default does it every second (you can go sub second with litestream if you want).
> Backups, litestream gives you streaming replication to the second.
You seem terribly confused. Backups don't buy you high availability. At best, they buy you disaster recovery. If your node goes down in flames, your users don't continue to get service because you have an external HD with last week's db snapshots.
If anything backups are the key to high availability.
Streaming replication lets you spin up new nodes quickly with sub second dataloss in the event of anything happening to your server. It makes having a warm standby/failover trivial (if your dataset is large enough to warrant it).
If your backups are a week old snapshots, you have bigger problems to worry about than HA.
This is a disingenuous scenario. SQLite doesn't buy you uptime if you deploy your app to AWS/GCP, and you can just as easily deploy a proper RDBMS such as postgres to a small provider/self-host.
Do you actually have any concrete scenario that supports your belief?
> SQLite doesn't buy you uptime if you deploy your app to AWS/GCP
This is...not true of many hyperscaler outages? Frequently, outages will leave individual VMs running but affect only higher-order services typically used in more complex architectures. Folks running an SQLite on a EC2 often will not be affected.
And obviously, don't use us-east-1. This One Simple Trick can improve your HA story.
Nonsense. You can't outrun physics. The latency across the Atlantic is already ~100ms, and from the US to Asia Pacific can be ~300ms. If you are interested in performance and you need to shave off ~200ms in latency, you deploy an instance closer to your users. It makes absolutely no sense to frame the rationale around performance if your systems architecture imposes a massive performance penalty in networking just to shave a couple of ms in roundtrips to a data store. Absurd.
https://antonz.org/sqlite-is-not-a-toy-database/ — 240K inserts per second on a single machine in 2021. The problem you describe is real, but the TPS ceiling is wrong by three orders of magnitude on modern hardware.
Do you know why it is a toy? Because in a real prod environment after inserting 240k rows per second for a while you have to deal with the fact that schema evolution is required. Good luck migrating those huge tables with Sqlite ALTER table implementation
This doesn't seem like a toy but you know... realizing different systems will have different constraints.
Not everyone needs monopolistic tech to do their work. There's probably less than 10,000 companies on earth that truly need to write 240k rows/second. For everyone else, we can focus on better things.
Try doing that on a “real” DB with hundreds of millions of rows too. Anything more than adding a column is a massive risk, especially once you’ve started sharding.
Yes it might be risky. But most schema evolution changes can be done with no or minimal downtime even if you have to do then in multiple steps. When is a simple ALTER going to be totally unacetable if youare using Sqlite?
I’ve seen multimillion dollar “enterprise” projects get no where close to that. Of course, they all run on scalable, cloud native infrastructure costing at least a few grand a month.
It's funny, we're now trained to see these things where they can't possibly ever have been (like in this case with the 11 year old blog). It's as if we all collectively forgot that whatever the LLMs are doing comes from somewhere, so it's obviously going to be found out in the wild.
I love them both too but that might not be the best metric unless you’re planning to run lots of little read queries. If you’re doing CRUD, simulating that workflow may favor Postgres given the transactional read/write work that needs to take place across multiple concurrent connections.
Not sure that’s always right either though. For example Mapbox used to use an SQLite database as the disk cache for map tile info. You cannot possibly store that amount of data in memory, so it’s a great use case.
This is mostly about thread communication. With SQLite you can guarantee no context switching. Postgres running on the same box gets you close but not all the way. It's still in a different process.
The original claim being discussed is about the overhead of an in-process database vs. a database server in a separate process, not about whether SQLite or PostgreSQL have a faster database engine.
I have used SQLite with extensions in extreme throughput scenarios. We’re talking running through it millions of documents per second in order to do disambiguation.
I won’t say this wouldn’t have been possible with a remote server, but it would have been a significant technical challenge.
Instead we packed up the database on S3, and each instance got a fresh copy and hammered away at the task. SQLite is the time tested alternative for when you need performance, not features
> It's not much harder to use than SQLite, you get all of the Postgres features, it's easier to run reports or whatever on the live db from a different box, and much easier if it comes time to setup a read replica, HA, or run the DB on a different box from the app.
Isn't this idea to spend a bit more effort and overhead to get YAGNI features exactly what TFA argues against?
> It's not much harder to use than SQLite, you get all of the Postgres features [..]
More features is a net negative if you don't need those features. Ideally you want your DB to support exactly what you need and nothing more. Not typically realistic but the closer you can get the better.
A feature you don't think you need today, might be one you actually need tomorrow. It would be short-sighted to choose some tech based only on what you need today. If the extra features don't cost you anything, I can't see that as a "net negative".
You can't simply copy/paste a Postgres database though...also you'd be surprised how fast SQLite can be...I've used SQLite for projects where I just couldn't get the performance elsewhere. For example, I had a names database with over 100 million rows in it for converting names to diminutives (e.g. David to Dave) and the inverse...after I precomputed a metric ton of indices it went like a rocket. Sure the file was quite big but oh boy was it quick.
Thats just swapping another enterprise focused concern into the mix. Your database connection latency is absolutely not a concerning part of your system.
I mean, you’re not wrong about the facts, but it’s also pretty trivial to migrate the data from SQLite into a separate Postgres server later, if it turns out you do need those features after all. But most of the time, you don’t.
So you are migrating from Sqlite to Postgres because you need it. What is the state of your product when you need to do this migration? Is your product non trivial? Are you now dependent on particular performance characteristics of Sqlite? Do you now need to keep your service running 24/7? Accounting for all of that takes way more than 5 minutes. The only way to beat that is if you still have a toy product and you can just export the database and import it and pray that it all works as a migration strategy.
IIRC TCP/IP through localhost actually benchmarked faster than Unix sockets because it was optimized harder. Might've been fixed now. Unix sockets gives you the advantage of authentication based on the user ID of who's connecting.
My experience with sqlite for server-based apps has been that as your app grows, you almost always eventually need something bigger than sqlite and need to migrate anyway. For a server-based app, where minimizing deployment complexity isn't an extremely important concern, and with mixed reads and writes, it's rarely a bad idea to use Postgres or MariaDB from the start. Yes there are niche scenarios where sqlite on the server might be better, but they're niche.
If this sounds like basic advice, consider there are a lot of people out there that believe they have to start with serverless, kubernetes, fleets of servers, planet-scale databases, multi-zone high-availability setups, and many other "best practices".
Saying "you can just run things on a cheap VPS" sounds amateurish: people are immediately out with "Yeah but scaling", "Yeah but high availability", "Yeah but backups", "Yeah but now you have to maintain it" arguments, that are basically regurgitated sales pitches for various cloud platforms. It's learned helplessness.
When I was a consultant we would plan out 25 piece cloud deployments for little pie in the sky apps that would never see more than 200 users. Everyone has been trained that 'cloud' means a lot of expensive moving parts and doesn't stop to plan their deployments beyond that.
It's all of five minutes to write a deployment yaml and ingress and have literally anything on the web for a handful of dollars a month.
I've written rust services doing 5k QPS on DO's cheapest kube setup.
It's not rocket science.
Serverless node buns with vite reacts are more complicated than this.
Ten lines of static, repeatable, versioned yaml config vs a web based click by click deploy installer with JavaScript build pipelines and magical well wishes that the pathing and vendor specific config are correct.
And don't tell me VPS FTP PHP or sshing into a box to special snowflake your own process runner are better than simple vanilla managed kube.
You can be live on the web from zero in 5 minutes with Digital Ocean kube, and that's counting their onboarding.
More and more, I'm seeing this issue with agents-based workflows as well. The training set is full (in quantity and in proportion) of codebases that are organized for very large teams, so that's what most prompted architectures lead to.
In my case I'm seeing it a lot on the front-end side. My clients end up with single-page apps that install Shadcn, Tailwind, React, React Router, Axios, Zod, React Form and Vite, all to center a some input elements and perform a few in-browser API calls. It's a huge maintenance burden even before they start getting value out of it.
These large setups are often a correct answer, but not the right one for the situation.
Don’t forget that people involved in information technology procurement will pay very large sums of the company’s money to not have to understand anything.
Why go live if you don't have a reasonable expectation of users?
Worrying about HA when you don't have customers that need it is one thing, but I wouldn't want to be in a place where I have to put a banner on the website asking users to please make a new account because we had an oopsie.
I do like this: cron to run the backup and then rsync to https://www.rsync.net, then an after script that check it was run and post to my telegram the analysis.
Another good option is Restic, since snapshots let you go back in time. That is useful in case you accidentally delete/break something and you're not quite fast enough to restore from backup before the next cron runs.
And also incredibly trivial to fix. Most VPS providers include their own backup services, and for the rest there's rsnapshot and some other cheaper VPS somewhere else to keep it "off site."
Too many have forgotten what it means to administrate a single system. You can do a lot with very simple tooling.
> If this sounds like basic advice, consider there are a lot of people out there that believe they have to start with serverless, kubernetes, fleets of servers, planet-scale databases, multi-zone high-availability setups, and many other "best practices".
I currently work in a small b2c startup with 200 active users (and targeting 5000 by the end of the year) and we're already paying AWS $1000/month on infra and it drives me crazy…
And the deployment process is also over-engineered in a way that makes it hard to change anything (if you want to release without changing things too much that's fine, but changing the deployment process is already a nightmare).
“But best practices”, “but scalability”, “but 99.999% uptime” …
The better availability and scalability of “the cloud” always relied on so many things being done and maintained just right by just the right people that I don’t think it’s ever been broadly true.
You get such a large performance malus and increase in complexity right from the start with The Cloud that it’ starts at a serious deficit, and only eventually maybe overcomes that to be overall beneficial with the right workload, people, and processes. Most companies are lacking minimum two of those to justify “the cloud”.
And that’s without even considering the cost.
What I think it actually is, is a way for companies that can’t competently (I mean at an organizational/managerial level) maintain and adequately make-available computing resources, to pay someone else to do it. They’re so bad at that, that they’re willing to pay large costs in money, performance, and maybe uptime to get it.
Not just stats. Configuration changes take around a day to take effect as well. Figuring out how to do authentication and permissions was such a pain. A half-assed integration with google cloud doesn't quite behave like the normal google cloud. Vague error messages. And every time you changed something you couldn't be certain your new setting was incorrect until you waited for an approximate day.
I end up explaining this term to every junior developer that doesn't know it sooner or later, the same way I explain bike shedding to all PMs that don't know it... often sooner, rather than later.
It seems to really help if you can put a term to it.
Heh, I was gonna say cargo cult might mean something different in today’s programming landscape but then I thought about it for a second and it actually reinforces th meaning.
I don't know what to say. People keep saying these engineers exist and here I am not having seen a single, and I follow many indie hackers communities.
A devops coworker found my blog and asked me how I host it, is it Kubernetes. I told him it's a dedicated server and he seemed amazed. And this was just a blog. It's real
Devops engineers did not know 101 of cable management or what even a cage nut is and being amazed to see a small office running 3 used dell servers bought dirt cheap, and shocked when it sounded like a air raid when they booted up, thought hot swapping was just magic.
It is always the case - earlier in the 80s-90s programmers were shaking their heads when people stopped learning assembly and trusted the compilers fully
This is nothing and hardly is shocking? new skills are learnt only if valuable otherwise one layer below seems like magic.
My point is that none of these coworkers have ever been at that stage. He was surprised about me hosting something because he seems to think hosting is expensive and for companies. Straight in at the top end of k8s and microservices
There's plenty of people that got a CS degree and went to work and this is only a job for them, they have no interest outside of work. Unfortunately I'm not one of those people so I get off work troubleshooting issues to troubleshoot issues at home lol though there aren't that many just my choice to self host cameras through HomeKit sometimes falls apart somehow but im also squeezing every KB or RAM out of that beelink I can.
Don't get me wrong I don't think a homelab is necessary, but I think people who have only done this in a big corporate environment are doing themselves a disservice - either a small company or a homelab can fix that itch, but like you say a lot of people don't have the interest
It's like a developer who went straight from knowing nothing about programming to JavaScript and never looked back. They missed C, they missed assembly, they missed cycle counting, they missed knowing what your memory footprint is at all times in your application, they missed keeping your inner loops tight and in the cache... It's not just "oh this person doesn't have a nerdy hobby." These are real skill holes in [many] developers' backgrounds, just like knowing how to host something on bare metal+OS is a real skill hole for some devops people.
I once interviewed for a small print shop that was proudly throwing out every AWS product name when describing their stack. They serve a few hundred customers and their previous system worked for decades entirely over email and a web form. I decided I wasn't interested around the point where he explained how they're migrating to lambdas
hey - devs aren' the only ones who fall in the premature optimization trap! Everyone from the CTO envisioning the scale of their future startup down to the IT intern is influenced by this, plus it's in the best interest of a dedicated infra guy to have a lot of dedicated infra. If you don't manage people K8s can become your kingdom and the size a badge of importance.
In this case I think it was a bit of CTO envisioning scale, then a bit of CTO genuinely overestimating what is needed, plus a good amount of CTO just being the average nerdy dev who likes the idea of shiny toys and cool sounding stuff - "we're running on k8s!".
A year or so after I left they ran out of money. They would've lasted longer if the infra guy would've just stayed the backend guy and helped get projects done more quickly instead of shiny k8s setups for projects with a dozen end-users per day. Recently I saw that the CTO has started a new startup - and ironically the only guy who he took with him onto the new team looks to have been the infra guy!
I don't blame infra guy, he genuinely believed he was doing the right thing.
> I use Linode or DigitalOcean. Pay no more than $5 to $10 a month. 1GB of RAM sounds terrifying to modern web developers, but it is plenty if you know what you are doing.
If you get one dedicated server for multiple separate projects, you can still keep the costs down but relax those constraints.
I put Proxmox on it and can have as many VMs as the IO pressure of the OSes will permit: https://www.proxmox.com/en/ (I cared mostly about storage so got HDDs in RAID 0, others might just get a server with SSDs)
You could have 15 VMs each with 4 GB of RAM and it would still come out to around 2.66 EUR per month per VM. It's just way more cost efficient at any sort of scale (number of projects) when compared to regular VPSes, and as long as you don't put any trash on it, Proxmox itself is fairly stable, being a single point of failure aside.
Of course, with refurbished gear you'd want backups, but you really need those anyways.
Aside from that, Hetzner and Contabo (opinions vary about that one though) are going to be more affordable even when it comes to regular VPS hosting. I think Scaleway also had those small Stardust instances if you want something really cheap, but they go out of stock pretty quickly as well.
Mostly to have stronger separation, I'm sure the person who prefers VM-per-project also has their own reasons.
I just have a few large VMs, each a different environment with slightly different ways how I treat them - the prod ones get more due diligence and being careful, whereas all of the dev ones (including where I host Gitea, Woodpecker CI, Nextcloud, Kanboard, Uptime Kuma etc.) I mess around with the configuration in and do restarts more often. I personally used to run a Docker Swarm cluster, but now just use Docker Compose with Ansible directly, still multiple stacks per each of those servers, dead simple
So my setup ended up being:
* VPS / VMs - an environment, since don't really need replication/distributed systems at my scale
* container stack (Compose/Swarm) - a project, with all its dependencies, though ingress is a shared web server container per environment
* single container - the applications I build, my own are built on top of a common Ubuntu LTS base more often than not, external ones (like Nextcloud and tbh most DBs) are just run directly
Works very well, plus containers allow me to easily have consistent configuration management, networking, resource limits and persistent storage.
What do you do about ipv4 ? Do you also use a routing VM to manage all that ?
It’s very interesting how people rent large VMs with a hypervisor. I’m wondering if licenses for VPS have any clauses preventing this for commercial scale.
I help my dad run a proxmox setup on a server he's got from a local craigslist analog and put on a co-location in a datacenter. It only uses a single public IP. All VMs are in a "virtual intranet", and the host itself acts like a router (giving local IP addresses to VMs via dnsmasq, routing VM internet access via NAT, forwarding specific outside ports to specific VMs). For example ports 80, 443 are given to a dedicated "nginx vm" which then will route a request to a specific VM depending on the hostname.
Since I only needed about 3 VMs (though each being a bit beefier, running containers on them, a web server sitting in front of those with vhosts as ingress), I could give each VM its own IPv4 address and it didn’t end up being too expensive for my use case. Would be a bit different for someone who wants many small VMs.
The exploits themselves may be simple, but that's precisely the point. The paper isn't claiming novel cybersecurity research – it's demonstrating that the benchmarks we use to evaluate multi-billion dollar investment decisions can be gamed with trivially simple tricks.
If a benchmark can be topped by checking if a test ID is even/odd rather than actually solving the task, the benchmark is measuring something fundamentally different from what we think it measures. The simplicity of the exploit amplifies the critique, it doesn't diminish it.
This is basically Goodhart's Law with a proof of concept attached.
Nice list! I'd say the SQLite with WAL is the biggest money saver mentioned.
One note: you can absolutely use Python or Node just as well as Go. There's Hetzner that offers 4GB RAM, 10TB network (then 1$/TB egress), 2CPUs machines for 5$.
Two disclaimers for VPS:
If you're using a dedicated server instead of a cloud server, just don't forget to backup DB to a Storage box often (3$ /mo for 1TB, use rsync). It's a good practice either way, but cloud instances seem more reliable to hardware faults. Also avoid their object store.
You are responsible for security. I saw good devs skipping basic SSH hardening and get infected by bots in <1hr. My go-to move when I spin up servers is a two-stage Terraform setup: first, I set up SSH with only my IP allowed, set up Tailscale and then shutdown the public SSH IP entrypoint completely.
Personally for backups I’d avoid using a product provided by the same company as the VM I’m backing up. You should be defending against the individual VM suffering corruption of some kind, needing to roll back to a previous version because of an error you made, and finally your VM provider taking a dislike to you (rationally or otherwise) and shutting down your account.
If you’re backing up to a third party losing your account isn’t a disaster, bring up a VM somewhere else, restore from backups, redirect DNS and you’re up and running again. If the backups are on a disk you can’t access anymore then a minor issue has just escalated to an existential threat to your company.
Personally I use Backblaze B2 for my offsite backups because they’re ridiculously cheap, but other options exist and Restic will write to all of them near identically.
> You are responsible for security. I saw good devs skipping basic SSH hardening and get infected by bots in <1hr. My go-to move when I spin up servers is a two-stage Terraform setup: first, I set up SSH with only my IP allowed, set up Tailscale and then shutdown the public SSH IP entrypoint completely.
Note that you don't need all of that to keep your SSH server secure. Just having a good password (ideally on a non-root account) is more than enough.
I'd call it unnecessary exposure. Under both modern threat models and classic cybernetic models (check out law of requisite variety) removing as much surface attack area as possible is optimal. Especially disabling passwords in SSH is infosec 1o1 these days. No need to worry about brute force attacks, credential stuffing, or simple human error, which was the cause of all attacks I've seen directly.
It's easier to add a small config to Terraform to make your config at least key-based.
Once I had Postgresql db with default password on a new vps, and forgetting to disable password based login, on a server with no domain. And it got hacked in a day, and was being used as bot server. And that was 10 years ago.
Recently deployed server, and was getting ssh login attempts within an hour, and it didn't had a domain. Fortunately, I've learned my lesson, and turned of password based login as soon as the server was up and running.
And similar attempts bogged down my desktop to halt.
Having an machine open to the world is now very scary. Thanks God for service like tailscale exists.
Yeah that sounds quite annoying, but has nothing to do with ssh log noise. Maybe investigate what's causing the OOM. I have multiple 1GB vps with ssh open to the world and they never OOM, and they're obviously not just running ssh. It sounds like you've been compromised.
I need more info about devs getting infected over ssh in less than an hour. Unless they had a comically weak root password or left VNC I don't believe it at all
Yes, <1h was a weak root password. All attacks I've seen directly were always user error. The point is effectively removing attack surfaces rather than enhancing security in needlessly exposed internet-facing protocols.
> Nice list! I'd say the SQLite with WAL is the biggest money saver mentioned.
Funny you said that. I migrated an old, Django web site to a slightly more modern architecture (docker compose with uvicorn instead of bare metal uWSGI) the other day, and while doing that I noticed that it doesn't need PostgreSQL at all. The old server had it already installed, so it was the lazy choice.
I just dumped all data and loaded it into an SQLite database with WAL and it's much easier to maintain and back up now.
Does WAL really offer multiple concurrent writers? I know little about DBs and I've done a couple of Google searches and people say it allows concurrent reads while a write is happening, but no concurrent writers?
Not everybody says so... So, can anyone explain what's the right way to think about WAL?
No, it does not allow concurrent writes (with some exceptions if you get into it [0]). You should generally use it only if write serialisation is acceptable. Reads and writes are concurrent except for the commit stage of writes, which SQLite tries to keep short but is workload- and storage-dependent.
Now this is more controversial take and you should always benchmark on your own traffic projections, but:
consider that if you don't have a ton of indexes, the raw throughput of SQLite is so good that on many access patterns you'd already have to shard a Postgres instance anyway to surpass where SQLite single-write limitation would become the bottleneck.
Thanks! even I run a sqlite in "production" (is it production if you have no visitors?) and WAL mode is enabled, but I had to work around concurrent writes, so I was really confused. I may have misunderstood the comments.
First step is to get ssh setup correctly, and second step is to enable a firewall to block incoming connections on everything except the key ports (ssh but on a different port/web/ssl). This immediately eliminates a swathe of issues!
Historical reliability and compatibility. They claimed they were S3 compatible, but they were requiring deprecated S3 SDKs, plus S3 advanced features are unimplemented (but at least they document it [0]). There was constant timeouts for object creation and updates, very slow speeds and overall instability. Even now, if you check out r/hetzner on reddit, you'll see it's a reliability nightmare (but take it with a grain of salt, nobody reports lack of problems). Not as relevant for DB backups, but billing is dumb, even if you upload a 1KB file, they charge you for 64KB.
At least with Storage Box you know it's just a dumb storage box. And you can SSH, SFTP, Samba and rsync to it reliably.
I quite like the websequencediagram. looks like a cool product!
He's mainly talking about the tech implementation which is the easy part.
the hard part of creating a business is finding a problem valuable enough to solve and reaching the users who need that problem solved. that's where the real value is.
This is the most frustrating problem I have. I do my 40 hours per week, play with my kids, relax with the wife, and play some video games. I don't really have any other problems besides not enough time in the day. And yet when I learn about some domain specific problem, it is blindingly obvious.
There are zero reasons to limit yourself to 1GB of RAM. By paying $20 instead of $5 you can get at least 8gb of RAM. You can use it for caches or a database that supports concurrent writes. The $15 difference won’t make any financial difference if you are trying to run a small business.
Thinking about on how to fit everything on a $5 VPS does not help your business.
$15 is not exactly zero, is it? If you don't need more than 1GB, why pay anything for more than 1GB?
I recall running LAMP stacks on something like 128MB about 20 years ago and not really having problems with memory. Most current website backends are not really much more complicated than they were back then if you don't haul in bloat.
It is. With 10k MRR it represents 0.15% of the revenue. Having the whole backend costing that much for a company selling web apps is like it’s costing zero.
You probably don't make 10k MMR on day one. If you make many small apps, it can make sense to learn how to run things lean to have 4x longer runway per app.
The runway is going to be your time and attention span, not $10/mo.
I don't know what you value your time or opportunity cost as... but the $10/mo doesn't need to save very many minutes of your time deferring dealing with a resource constraint or add too much reliability to pay off.
If resource limitations end up upsetting one end user, that costs more than $10.
Given how much revenue depends on the experience of a web app and loading times, I’d be happy to pay 100$ a month on that revenue if I don’t have to sacrifice a second of additional loading time no matter how clever I was optimizing it.
if the project already has positive revenue then arguably the ability to capture new users is worth a lot, which requires acceptable performance even when a big traffic surge is happening (like a HN hug of attention)
if the scalability is in the number of "zero cost" projects to start, then 5 vs 15 is a 3x factor.
NVME read latency is around 100usec, a SQLite3 database in the low terabytes needs somewhere between 3-5 random IOs per point lookup, so you're talking worst case for an already meaningful amount of data about 0.5ms per cold lookup. Say your app is complex and makes 10 of these per request, 5 ms. That leaves you serving 200 requests/sec before ever needing any kind of cache.
That's 17 million hits per day in about 3.9 MiB/sec sustained disk IO, before factoring in the parallelism that almost any bargain bucket NVME drive already offers (allowing you to at least 4x these numbers). But already you're talking about quadrupling the infrastructure spend before serving a single request, which is the entire point of the article.
> There are zero reasons to limit yourself to 1GB of RAM
There is a good reason: teaching yourself not to over-engineer, over-provision, or overthink, and instead to focus on generating business value to customers and getting more paying customers. I think it’s what many engineers are keen to overlook behind fun technical details.
> There is a good reason: teaching yourself not to over-engineer, over-provision, or overthink, (...)
This is specious reasoning. You don't prevent anything by adding artificial constraints. To put things in perspective, Hetzner's cheapest vCPU plan comes with 4GB of RAM.
If I give you a box with 1 GiB of RAM, you are literally forced to either optimize your code to run in it, or accept the slowdown from paging. How is this specious?
> If I give you a box with 1 GiB of RAM, you are literally forced to either optimize your code to run in it, or accept the slowdown from paging. How is this specious?
It is specious reasoning. Self-imposing arbitrary constraints don't make you write good, performant code. At most it makes your apps run slower because they will needlessly hit your self-impose arbitrary constraints.
If you put any value on performant code you just write performance-oriented code, regardless of your constraints. It's silly to pile on absurd constraints and expect performance to be an outcome. It's like going to the gym and work out with a hand tied behind your back, and expect this silly constraints to somehow improve the outcome of your workout. Complete nonsense.
And to drive the point home, this whole concern is even more perplexing as you are somehow targeting computational resources that fall below free tiers of some cloud providers. Sheer lunacy.
Constraints provide feedback. Real-world example from my job: we have no real financial constraints for dev teams. If their poor schema or query design results in SLO breaches, and they opt to upsize their DB instead of spending the effort to fix the root problem, that is accepted. They have no incentive to do otherwise, because there are no constraints.
I think your analogy is flawed; a more apt one would be training with deliberately reduced oxygen levels, which trains your body to perform with fewer resources. Once you lift that constraint, you’ll perform better.
You’re correct that you can write performant code without being required to do so, but in practice, that is a rare trait.
The gym analogy fails. Isolation exercises are almost exactly what you described. They target individual muscles to maximize hypertrophy, i.e. "improve the outcome of your workout."
I think we have to re-think and re-evaluate RAM usage on modern systems that use swapping with CPU-assisted page compression and fast, modern NVMe drives.
The Macbook Neo with 8GB RAM is a showcase of how people underistimated its capabilities due to low amount of RAM before launch, yet after release all the reviewers point to a larger set of capabilities without any issues that people didn't predict pre-launch.
Memory compression sounds like going back to DOS days. I think we're better off with writing tighter more performant code with no YAGNI. Alas, vibe coding will probably not get us there anytime soon.
Apple laptop CPUs have hardware memory compression and exceptionally high memory bandwidth for a CPU, and with their latest devices, very high storage bandwidth for a consumer SSD, so the equation is very different from the old DOS days.
I do agree that the overall tendency towards cloud has made things much more complicated and expensive than they need to be in many cases. Cloud has its place, but so do simple server instances. Many projects won't reach any kind of scale that would exceed the capabilities of a medium-sized VPS. We're running a page with 600k users at work that could easily fit on a 30€ VPS. Instead, we moved to AWS and are now paying 800€ for it. No benefits whatsoever.
So yea, stick with what worked for decades if you don't see a reason not to.
Also, I remember reading that StackOverflow runs on a bunch of super powerful root servers?
> I bought a GitHub Copilot subscription in 2023, plugged it into standard VS Code, and never left. I tried Cursor and the other fancy forks when they briefly surpassed it with agentic coding, but Copilot Chat always catches up.
> Here is the trick that you might have missed: somehow, Microsoft is able to charge per request, not per token. And a "request" is simply what I type into the chat box. Even if the agent spends the next 30 minutes chewing through my entire codebase, mapping dependencies, and changing hundreds of files, I still pay roughly $0.04.
> The optimal strategy is simple: write brutally detailed prompts with strict success criteria (which is best practice anyway), tell the agent to "keep going until all errors are fixed," hit enter, and go make a coffee while Satya Nadella subsidizes your compute costs.
People get banned abusing this per request strategy so be careful. This guy was running super long prompts per request and is somehow surprised why they got banned.
It works with all models, some have a cost multiplier like Opus 4.6 ”charges” 3 requests per prompt, but its still only for the prompts you send yourself - even if it works on the issue for hours. GPT-5.4 has no multiplier i.e. costs 0.04$ per prompt.
Worth noting however that they are starting to introduce rate limits lately so you might struggle to run multiple concurrent sessions, though this is very inconsistent for me. Some days I can run 3-4 sessions concurrently all day, other times I get rate limited if I run one non-stop..
If you feel like it: start a blog! You have knowledge that you consider basic and a certain other subset of the population is interested in it and doesn't know it exists.
I think it's good. I've definitely seen resource inflation exactly that OP is alluding to in enterprise. A desire to have some huge cloud based solution with AWS, spark bla bla when a python script with pandas in a cron job was faster.
Not only that, his whole business model seems to be "profit off the AI bubble and get the big techs to indirectly subsidize you"
Which obviously works, it's not like there aren't tons of multi-million startups ultimately doing the exact same thing, and yet. It feels a bit... trite?
Honestly, yes. I'm on HN for tech content, I don't really care about startups and the business side of things, even though sometimes there are interesting reads on this side as well. Also, it may very well be the case that I rediscover the meaning of MRR for the second or third time in sixteen years :).
I was also curious about that, I would've thought especially in 2010 the startup ethos would've been more prevalent on HN whereas these days it's more about AI and big tech.
There was always tech content. I'd say it was even a more important part back in the days, and it was more diverse. There were always some trends (Ruby on Rails, Rust, etc.) but it was never like these days with LLM-related content which is almost all of the tech content. Because of that I've gone back to Reddit like two years ago, and now spend even more time there than here, which hadn't been the case in almost 15 years before that.
French subreddits, coffee subreddits, free software and free culture subreddits, specific programming languages subreddits, computer science subreddits, and yes, general tech subreddits which are also flooded with AI stuff, but I feel that it makes a significant difference that users can still downvote links there, which hasn't been possible in years on HN (not saying that it's a bad thing, just that it's a significant difference).
There is also ARR which is "annual recurring revenue" and you should know that when people use ARR they usually are just making up numbers based on their current MRR (so lying). I've seen people announce their ARR after running their business for two whole months!
That's not really "lying" — ARR is usually understood as your projected "Annual Run Rate". It's a useful metric, as long as it is understood that it is an estimate.
But, in all honesty, all RR numbers are estimates. MRR is also a "made up number" from a certain point of view: it is not equivalent to cash received every month, because of annual subscriptions, cancelations, etc.
>But, in all honesty, all RR numbers are estimates.
Sure, but I would expect you to have at least one data point or at least near it, before making any estimates for that timescale.
I don't see many people make MRR projections based on 2 days of of sales, it's just something I've noticed with startups and ARR.
This is supposed to be a contrarian opinion yet this is a retoric yapped non-stop in the “build in public” community. Of course lean is a good approach, it makes sense, and most engineers know this. Is not a new concept, we’ve been doing this for years in every branch of engineering.
The invented “people start with a k8s cluster for 5 users” doesn’t really exist. This is just a story repeated ad nauseam to fit a narrative that help them justify their choices. This position is just as dogmatic, if not more, than the alleged dogma it attempts to disrupt.
Smart technical leaders knows that technical decisions only matter in context never in absolutes. The right answer is always “it depends”.
I can agree that there is a tendency to prematurely optimize infra, as a direct consequence of lack of measuring especially in young busy startups. One could argue that premature optimization might be the smart choice when you don’t have enough data, as in the best case scenario (your startup do well) you’ve saved some time, worst case scenario you’ve lost some money that depending on the situation might be less valuable than time spent in maintaining, and later refactoring, infra.
I've seen A LOT of public sector projects starting out with loads of Azure services and >$3000 montly bills for applications that could've easily run on a single VM.
This a structural problem not an awareness one. Is not like they don’t know they can run it on a 5$ VPS, the problem is that there are no incentives to do so. You’d be surprised to know how much of engineering is there to address organisational challenges rather than technical ones (ie: micro-services)
I recall reading multiple AskHN posts about people trying to get attention from a cloud provider because they ran up thousands of dollars in charges accidentally. I've seen large companies do this too, even if you think something is just a dev environment, its the cloud provider's production environment and they will charge you per their ToS for everything you use, doesn't matter what the customer usage profile looks like.
Experienced dev with limited hands-on big tech infrastructure experience. Based on the results I get from LLMs in domains I understand how get they even get this stuff running using AI?
“people start with a k8s cluster for 5 users” doesn’t really exist
Most people in the BiP these days barely know how to deploy a database or host something using nginx. it's all vercel, supabase, aws, clerk, yada yada. Cost aside, I think that people are addicted to complexity.
I know this article is about the stack, but I'd like to point out that the success of the author has probably more to do with their marketing/sales strategy than their choice of technical infrastructure.
What has your career looked like? I'm interested because I've spent 20 years in applied research and I've only more recently realized the continual stress that I've felt for 20 years from trying (and mostly failing) to innovate in the "what to build" space.
Working for a salary on other people's ideas, LOL. Mostly incredibly boring ones. One interesting ones that fizzled due to too-low investment and too-safe management (odds are it would have fizzled anyway, of course, though I do think if that one had had the eyes of the right investors it probably could have done a "successful exit"—this was like 15 years ago though)
For me it has been just saying "yes" when I was offered a job and when that one was getting a bit annoying someone happened to offer me another and I said "yes" too. I have ended up a bit underemployed and underpaid, but life's comfortable and safe and I have ample time to stress over hobbies instead of work.
So comfortable that lately I have declined offers for interesting and much much better paid work, because I can no longer be bothered to take any risks or alter my lifestyle.
But sometimes I wish I could have been the guy managing to get 10k MMR using knowledge I've got in spades.
The text feels incoherent to me and lacks some nuance.
It starts about cutting costs by the choice of infrastructure and goes further to less resource hungry tools and cheaper services. But never compares the cost of these things. Do I save actually the upgrade to a bigger server by using Go and sqlite over let's say Python and postgres? Or does it not even matter when you have just n many users.
Then I do not understand why at one point the convenience of using OpenRouter is preferred over managing multiple API keys, when that should be cheaper and a cost point that could increase faster than your infrastructure costs.
There are some more points, but I do not want to write a long comment.
It actually starts with a completely unrelated anecdote:
"What do you even need funding for?"
I agree. The author claims to have multiple $10K MRR websites running on $20 costs. I also don't understand what he needs money for — shouldn't the $x0,000 be able to fund the $20 for the next project? It doesn't make any sense at all.
Then the author trails off and tells us how he runs on $20/month.
Building a $10K MRR website is hard. Building multiple (assuming "multiple" here means >= 3) $10K MRR websites is extremely hard.
I don't know which investors they pitched to, but most investors seeing that number will write a 100-200K check to invest in THE PERSON pretty immediately; unless there was strong red flags in their business model (porn, drug, gambling, etc...)
The author says he wants funding to grow the businesses. Presumably he wants funding and the help from investors to enable quicker growth than what is possible organically.
Just because you start this lean doesn’t mean you should stay that way. Perhaps he’s now spending too much time managing his stack and not enough time on product development, customer service, a/o growth.
In other words, what gets you to $10k MRR isn’t the same thing(s) for 2x, 5x, or 10x that.
SQLite is fine, but I have ran Postgresql on a $20 server without any issues, and I would suggest if you have to deal with concurrent users and tasks, Postgresql is the way to go. SQLite WAL works, but sometimes it caused some issues, when you have a lot of concurrent tasks running continuously.
And, not sure I'm correct, but I felt Postgresql has more optimized storage if you have large text data than SQLite, at least for me I had storage full with SQLite, but same application on Postgresql never had this issue
Great stack! I'm doing a similar approach for my latest project (kavla.dev) but using fly.io and their suspend feature.
Scaling to zero with database persistence using litestream has cut my bill down to $0.1 per month for my backend+database.
Granted I still don't have that many users, and they get 200ms of extra latency if the backend needs to wake up. But it's nice to never have to worry about accidental costs!
I think his argument is that the functionality is unnecessary. You don’t need dynamic service scaling because your single-instance service has such high capacity to begin with.
I guess it’s all about knowing when to re-engineer the solution for scale. And the answer is rarely ”up front”.
Thats true. The reason I like k8s is once you've gone up the learning curve you can apply that knowledge to cloud deployments, on prem, or in this case VPS.
The authors stack left me thinking about how will he re-start the app if it crashes, versioning, containers, infra as code.
I've seen these articles before... the Ruby on Rails guys had the same idea and built https://kamal-deploy.org/
Which starts to look more and more like K3s as time goes on.
I’m thinking even simple containers have automatic restarts. I wouldn’t deploy to prod using ”docker start” but I wouldn’t look askance at someone using “docker compose” for that purpose.
Lots of debate about SQLite vs Postgres. One thing to note is you can certainly run Postgres (Kubernetes even if you want) and your app for $20 / month.
I'm taking the opposite approach - managed services all the way, and my monthly infrastructure costs are higher than what's described here.
No regrets. Infrastructure isn't the problem I'm trying to solve. The problem is: who's actually going to pay for this?
Optimizing infrastructure before you have customers is like designing a kitchen before you've written the menu. I launched within 72 hours of starting development and went straight to customer validation. The market feedback started coming in immediately.
Infrastructure costs show up in your bill. The cost of slow customer validation doesn't show up anywhere - until it's too late. That's the number I watch.
Some of this will depend on what experience you’ve got. Someone with lots of experience running Linux servers can probably stand up the sort of thing described in this article in a couple of hours from a starting point of being given the Go application source and a credit card.
It doesn’t sound like OP was optimizing anything; it sounds like they just knew how to use that stack, and so are able to get customer validation while also spending very little per month.
Fair point. Stack selection is mostly about what you already know. I chose managed services not because I optimized for it, but because that's the stack I'm comfortable with.
That said, my real point was simpler: whatever stack you pick, figure out who's going to pay for it before you spend time on infrastructure decisions.
which approach works better depends on your financial situation and your existing setup. if you have money you can invest, then your approach works. if you have more time than money then invest the time instead. when you have built up your servers over the years, when building a new product, you can also do it quickly because the services you need are already running, and firing up a new database or a new server takes just as long as it takes to set up a managed service. but it doesn't add any cost.
The basic premise, try to be lean, is a good one. The implementation will clearly be debated with everyone having their own opinion on it but the core point is sound. I'd argue a different version of this though: keeping things lean forces simplicity and focus which is incredibly important early on. I have stepped into several startups and seen a mess of old/broken/I don't know what it does so leave it/etc etc. All of that, beyond the cost, slows you down because of the complexity. Regular gardening of your tech stack matters and has a lot of benefits.
Do these things actually work? I've seen way too many gurus on twitter claiming to make 10K+ MRR every month. And then they quietly start applying for jobs. or selling courses instead of cashing in.
Anyone doing per tenant database with SQLITE + Litestream? Please share your experiences and pain points. I know migrations are one. The other challenge is locating the correct database from incoming request. What else?
Solid writeup and impressive experience. You can try Caddy instead of nginx. Automatic HTTPS, dead-simple config, and it proxies to your Go binary in about 4 lines. If you're still manually managing Let's Encrypt certs in 2026, stop. Caddyfile for a Go backend is literally: reverse_proxy localhost:8080 — that's the whole thing.
Always good to challenge the narrative - but I don't pay for RDS Postgres because of the WAL, replication, all the beauty of pg etc. I pay RDS because it's largely set and forget. I am gladly paying AWS to think about it for me. I think at a certain scale, this is a really good tradeoff. At the very beginning it could be overkill, and at the top end obviously its unsuitable - but for most of us those tradeoffs are why it's successful.
I was writing about this recently [0]. In the 2000s, we were bragging about how cheap our services are and are getting. Today, a graduate with an idea is paying $200 amounts in AWS after the student discounts. They break the bank and go broke before they have tested the idea. Programming is literally free today.
The biggest cost when bootstrapping always seemed to be your salary to me, not infra costs. How long can you pay your mortgage and feed your kids off what should be your retirement or rainy day funds?
Similar approach here. I run a side project on Next.js + Vercel (free tier) + Neon Postgres (free tier). Total hosting cost: $0/month.
The one place I'd push back on SQLite: if your app has any write concurrency from external processes (cron jobs, webhooks), WAL mode helps but you still hit lock contention. I have data collection scripts running every 30 minutes that write to the same DB the web app reads from. Postgres handled that cleanly from day one. Neon's free tier is 512MB with connection pooling — more than enough for a side project with real data.
The most interesting thing in here is https://github.com/smhanov/laconic which is the author's "agentic research orchestrator for Go that is optimized to use free search & low-cost limited context window llms".
I have been doing this kind of thing with Cursor and Codex subscriptions, but they do have annoying rate limits, and Cursor on the Auto model seems to perform poorly if you ask it to do too much work, so I am keen to try out laconic on my local GPU.
EDIT:
Having tried it out, this may be a false economy.
The way it works is it has a bunch of different prompts for the LLMs (Planner, Synthesizer, Finalizer).
The "Planner" is given your input question and the "scratchpad" and has to come up with DuckDuckGo search terms.
Then the harness runs the DuckDuckGo search and gives the question, results, and scratchpad to the Synthesizer. The Synthesizer updates the scratchpad with new information that is learnt.
This continues in a loop, with the Planner coming up with new search queries and the Synthesizer updating the scratchpad, until eventually the Planner decides to give a final answer, at which point the Finalizer summarises the information in a user-friendly final answer.
That is a pretty clever design! It allows you to do relatively complex research with only a very small amount of context window. So I love that.
However I have found that the Synthesizer step is extremely slow on my RTX3060, and also I think it would cost me about £1/day extra to run the RTX3060 flat out vs idle. For the amount of work laconic can do in a day (not a lot!), I think I am better off just sending the money to OpenAI and getting the results more quickly.
But I still love the design, this is a very creative way to use a very small context window. And has the obvious privacy and freedom advantages over depending on OpenAI.
>To manage all this, I built laconic, an agentic researcher specifically optimized for running in a constrained 8K context window. It manages the LLM context like an operating system's virtual memory manager—it "pages out" the irrelevant baggage of a conversation, keeping only the absolute most critical facts in the active LLM context window.
The 8K part is the most startling to me. Is that still a thing? I worked under that constraint in 2023 in the early GPT-4 days. I believe Ollama still has the default context window set to 8K for some reason. But the model mentioned on laconic GitHub (Qwen3:4B) should support 32K. (Still pretty small, but.. ;)
I'll have to take a proper look at the architecture, extreme context engineering is a special interest of mine :) Back when Auto-GPT was a thing (think OpenClaw but in 2023), I realized that what most people were using it for was just internet research, and that you could get better results, cheaper, faster, and deterministically, by just writing a 30 line Python script.
Google search (or DDG) -> Scrape top N results -> Shove into LLM for summarization (with optional user query) -> Meta-summary.
In such straightforward, specialized scenarios, letting the LLM drive was, and still is, "swatting a fly with a plasma cannon."
(The analog these days would be that many people would be better off asking Claw to write a scraper for them, than having it drive Chromium 24/7...)
> (The analog these days would be that many people would be better off asking Claw to write a scraper for them, than having it drive Chromium 24/7...)
Possibly. But possibly you have a very long tail of sites that you hardly ever look at, and that change more frequently than you use them, and maintaining the scraper is harder work than just using Chromium.
The dream is that the Claw would judge for itself whether to write a scraper or hand-drive the browser.
That might happen more easily if LLMs were a bit lazier. If they didn't like doing drudgery they would be motivated to automate it away. Unfortunately they are much too willing to do long, boring, repetitive tasks.
Not sure if top model should be the biggest one though. I hear opposite opinions there. Small model which delegates coding to bigger models, vs big model which delegates coding to small models.
The issue is you don't want the main driver to be big, but it needs to be big enough to have common sense w.r.t. delegating both up[0] and down...
[0] i.e. "too hard for me, I will ping Opus ..." :) do models have that level of self awareness? I wanna say it can be after a failed attempt, but my failure mode is that the model "succeeds" but the solution is total ass.
I think newer developers really need to learn that you can actually do production stuff using bare tools. It is not crazy, especially in the beginning, and it will save you a ton of money and time.
Strongly agree. Forget the savings. Learning the basic tools and understanding how and why the complexity is added (what problems does it solve) is a big one.
I get that the focus of this article is on the tech portion, but I don't know anyone pitching today (aside from OpenAI) who is asking for big funding for the tech costs. It doesn't really matter if you built a system that costs you $200/month or $20/month if your lifetime value is $1000 and CAC is only $10 but you've got no money. That's what people want to fund. VC funding is gasoline you pour on a fire (or fuel for you rocket if you're being charitable) - it makes you go faster; a pitch that focuses on "slightly lower monthly op costs" is not attractive.
I love SQLite and have ran it even on networked drives with queued writes for read-heavy applications. It’s an incredibly robust piece of software that’s often cost me pennies per month to serve 100k+ monthly users. But there’s definitely a time and place for solid, dedicated database servers like Postgres.
This is my life goal right now. I have a bajillion ideas, know how to code them (even faster now), and just not enough time due to day job. A few questions:
How do you market them?
Is customer support an issue?
Do you see risk since ai makes it so easy to build/copy?
>>The optimal strategy is simple: write brutally detailed prompts with strict success criteria (which is best practice anyway), tell the agent to "keep going until all errors are fixed," hit enter, and go make a coffee while Satya Nadella subsidizes your compute costs.
This is similar to what I do. Linode, Debian, Go, HTMX, SQLite (with modernc.org/SQLite so I have no CGO dependency) and Caddy. If I have apps that need a lot of storage, I just add an S3 bucket.
> If you need a little breathing room, just use a swapfile.
You should always use a swap file/partition, even if you don't want any swapping. That's because there are always cold pages and if you have no swap space that memory cannot be used for apps or buffers, it's just wasted.
I always thought I had to add a swap file to avoid crashing with OOM. I wasn't aware of the cold pages overhead.
Sometimes that crashing is what I want: a dedicated server running one (micro)service in a system that'll restart new servers on such crashes (e.g. Kubernetes-alike). I'd rather have it crash immediately rather than chugging along in degraded state.
But on a shared setup like OP shows, or the old LAMP-on-a-vps, i'd prefer the system to start swapping and have a chance to recover. IME it quite often does. Will take a few minutes (of near downtime) but will avoid data corruption or crash-loops much easier.
Basically, letting Linux handle recovery vs letting a monitoring system handle recovery
The biggest risk to cloud revenues is that everyone wakes up and realizes they could slash their cloud bills by 60+% quite quickly with just some minimal leaning.
I do appreciate the technical simplicity argument and I'm always advocating for it. And the few neat tricks i.e. Copilot.
That being said, I'd much rather read a few ideas for good recurring passive income. Instead, the author kind of flexes on that, then says "I get refused VC money because they don't see how their money would be useful for me" -- which is one more flex -- and moves on to the technical bits.
It isn’t 10k MRR from day one. It also doesn’t make sense to think “well, now that I’m a big boy let’s move to a fancy stack , even if there is no need for it”
Exactly. Deciding on some very expensive subscriptions that can cost 1k per month or so might be worth thinking about, but this is just meaningless optimisation.
not at all meaningless. unless you have money to invest, at the beginning you don't have an income. i could not afford to spend $300 a month to host a new product that doesn't make any money yet. i can afford the $20 however, but then once the product does make money, why should i change it if it works?
While I applaud the acumen, this reads like watching a kid standing on the 3rd floor balcony shouting "look what I can do!"
$20/month. Yeah. Great, but why? You get a lot of peace of mind with "real" HA setup with real backups and real recovery, for not much more than $20, if you are careful.
Another half of article is about running "free, unlimited" local AI on a GPU (Santa brought it) with, apparently, free electricity (Santa pays for it).
AWS is not value for money, I do have a DO account that is great but my development is mostly hosted locally with tunnels from cloudflare, it is remarkable how far you (I) can get with that setup.
Last I saw, AWS has way better peering agreements than DO. Lots of problems with terrible throughput and lots of dropped packets for various clients (in several cities in North America, not just overseas or in the middle of nowhere) that vanished instantly on switching to AWS (including overseas ones that were also having problems)
Unfortunately, this isn’t something that shows up on spec sheets when you’re choosing a service. :-/
Source was we used it, and that's what we saw, ~20% of clients on three continents (about half in North America) consistently had terrible connectivity to DO (not none, but it was really bad) and we spent a lot of time trying to fix it. Vanished through nothing but shifting that to AWS. It was clearly DO's peering network.
You probably won't see this unless both the following are true for your situation:
1) You have a workload that makes this issue noticeable. Long-lived connections and large transfer sizes make it more likely you'll notice. Loading 20kb of static html over the connection likely won't seem to have any problems (unless you run repeated trials and network analysis tools). Of course, modern websites can be pretty large...
2) Your users are long-term enough and in communication with you so these issues can even be noticed in the first place. Also helps if they're technical. If you're not hearing the story and aware of the situation on the other end of the line, all you see is a slow connection, could be anything causing it, and there are plenty of them for reasons that have to do with things closer to the client's end.
So all e.g. an e-commerce site might see is a somewhat higher bounce rate than necessary (due to some fraction of their users experiencing the site like it's on a somewhat-jittery ISDN line) without even knowing they're leaving money on the table because they likely have no way of even being aware of the problem.
[EDIT] Yes, we tried shifting around a bunch of ways on DO's side trying all kinds of ways to fix this, I'm quite sure it wasn't that we were unlucky with our hardware draw there or just one of their datacenters had this problem. It was something past the edge of their network.
One thing that I noticed was the mention of Claude 3.5 Sonnet or GPT-4o as cutting-edge models when the blog was written 25 days ago. This sadly makes me suspect that this was written by a LLM instead of a person...
One thing that I noticed was the mention of Claude 3.5 Sonnet or GPT-4o when the blog was written 25 days ago. This sadly makes me suspect that this was written by a LLM instead of a person...
I run a python flask app on Docker on AWS EC2. Including Dozzle (logs), Beszel (monitoring) and Unami (analytics). Runs on three t4g instances. One being a dedicated caddy exposed to the internet, one being a NAT gateway (self managed) and the main Docker worker host (private). All in cost is $35 a month. You’re splitting hairs and being the typical overly frugal developer instead of founder mode if you optimizing dollars when you have $20k / mo revenue.
What do I get as an advantage being on AWS? S3 (literally like a $1 month) SQS (free tier) and Lambda (async jobs; free tier). Capacity if needed, just scale up t4g instances.
Where do you get your eh-trade.ca stock price data? Given the licensing fees, that seems like one of the greater challenges of bootstrapping anything with market data.
While I agree with your points, this one could be more nuanced:
> Infrastructure: Bare Server > Containers > Kubernetes
The problem with recommending a bare server first is that bare metal fails. Usually every couple of years a component fails - a PSU, a controller, a drive. Also, a bare metal server is more expensive than VPS.
Paradoxically, a k3s distro with 3 small nodes and a load balancer at Hetzner may cost you less than a bare metal server and will definitely give you much better availability in the long run, albeit with less performance for the same money.
In 5 years of running 3x Dell R620s 24/7 - which were already 9 years old when I got them - I had two sticks of RAM have ECC errors, and one PSU fail. The RAM technically didn’t have to be replaced, but I chose to. The PSU of course had a hot spare, so the system switched over and informed me without issue.
IME, hardware is much more reliable than people think.
The pricing is so good that it's the only way I do agentic coding now. I've never spent more than $40 in a month on Opus, and I give it large specs to work on. I usually spend $20 or so.
This is really what 10k mrr can get you? A badly designed AI slop website that isn't even mobile correctly compatible. The logo is white background on black website like a university project.
I can't believe that people are willingly spending money on this.
You'd be surprised at the amounts household name companies spend on broken software. I've personally seen multiple companies spend tens of thousands paying just for the opportunity to evaluate the broken software. And I don't mean the time taken for their own employees to spend doing the evaluation. I mean that plus forking over large piles of cash.
I have worked with healthcare clients paying gobs of money to completely broken sites that they have to call support about twice a week. I don't pretend to know why people spend money on things anymore.
Agree with a lot of this except sqlite, and the fact that if I commit to building something I typically put it in the cloud with basic scaling from the start for efficiency.
Dude you are my spirit animal. Ive been building lean like this forever and thought I was the only nutcase out there...I too have had the same pitch problems as you, when a platform is hella efficient and there is no spending to be had on fixed assets, its not very appealing to investors.
> Here is the trick that you might have missed: somehow, Microsoft is able to charge per request, not per token. And a "request" is simply what I type into the chat box. Even if the agent spends the next 30 minutes chewing through my entire codebase, mapping dependencies, and changing hundreds of files, I still pay roughly $0.04.
Really? Lol. If it's true why would you publish it? To ensure Microsoft will patch it up and fuck up your workflow?
>Really? Lol. If it's true why would you publish it? To ensure Microsoft will patch it up and fuck up your workflow?
It's true and it's their official pricing, so talking about it won't change anything.
People are spending way too much money with Claude Code while they could simply pay for GitHub Copilot and fire up OpenCode to get the same results but way cheaper.
It always make me both roll my eyes and smile a little when i see someone daft enough to think they need some obscene setup - you dont. You never have. You are not Amazon, Microsoft, Google, etc. If you get to the point where you need that kind of setup you're already employing a dev ops team thats telling you that.
Stick whatever you're working on onto a ~$5/mo cheapo vps from someone like Hetzner, Digitalocean, etc and just get on with building your thing.
For single-person companies infra can be the single largest expense (especially if you aren't paying yourself yet!). The day you bring a full-time employee onboard, I have a hard time seeing infra costs ever exceeding salaries for most shops
I was wondering this as well: Why did OP look for VC?
In my case, I've used a similar strategy of keeping costs under €100/month. (But have sold, or stopped my ventures before hitting such MRRs as OP reports).
I raised some capital to pay my own bills during development. But mostly to hire freelancers to work on parts that I'm bad at, or didn't have time for: advertising, a specific feature, a library, rewrite-in-rust (wink) or deep research into functional improvements.
>The feedback was simply: "What do you even need funding for?"
Not clear from the text, but what was your plan using the funding on? If you did not have a plan, what did you expect? VCs want to see how adding more money results in asymmetric returns.
Eh-trade.ca eh? The name spells the exit strategy this is seeking. Awesome idea and a great execution. Vertical scaling will take this simple setup far and probably far enough.
eh the super low cost only comes from low complexity. If complex, people pay more, features cost more, infra costs aren’t that big of a cost compared to dev time.
nice article, validates some of the things i already thought. although im sure things like aws and database servers etc are still useful for big companies
What a fascinating article. I especially love the part about writing extremely detailed requests which only cost $0.04 versus the token approach most “vibe code” devs use. Fortunately his tactic is almost impossible to emulate for 90% of the YCombinator audience / HN commentators.
Why do I know this? Because there had to be a declaration here to stop using ChatGPT and other Agents to write YOUR OWN GODDAMN POSTS. Thinking isn’t your strong suit, Greed is, and taking the time to learn the power of English doesn’t satisfy the latter, so you minimize it to your own detriment.
A lot of this advice is good or at least interesting. A lot of it is questionable. Python is completely fine for the backend. And using SQLite for your prod database is a bad idea, just use Postgres or similar.
There’s a lot to be said about his approach with go for simplicity. Python needs virtual environments, package managers, dependencies on disk, a wsgi/asgi server to run forked copies of the server, and all of that uses 4x-20x the ram usage of go. Docker usually gets involved around here and before you know it you’re neck deep in helm charts and cursing CNI configs in an EKS cluster.
The go equivalent of just coping one file across to a server a restarting its process has a lot of appeal and clearly works well for him.
Yes. It strikes me as odd how many people will put forward Python with the argument of "simplicity".
It is not. Simple. It may be "easy" but easy != simple (simple is hard, I tend to say).
I'm currently involved in a project that was initially layed out as microservices in rust and some go, to slowly replace a monolyth Django monstrosity of 12+ years tech debt.
But the new hires are pushing back and re-introducing python, eith that argument of simplicity. Sure, python is much easier than a rust equivalent. Esp in early phases. But to me, 25+ years developer/engineer, yet new to python, it's unbelievable complex.
Yes, uv solves some. As does ty and ruff. But, my goodness, what a mess to set up simple ci pipelines, a local development machine (that doesn't break my OS or other software on that machine). Hell, even the dockerfiles are magnitudes more complex than most others I've encountered.
Python will take you a long way, but its ceiling (both typical and absolute) is far lower than the likes of Go and Rust. For typical implementations, the difference may be a factor of ten. For careful implementations (of both), it can be a lot more than that.
Does the difference matter? You must decide that.
As for your dismissing SQLite: please justify why it’s a bad idea. Because I strongly disagree.
There are a myriad middle states in-between "frupid" (so frugal that it's stupid) and "Instagram scale".
Python requires much more hand-holding that many don't want to do for good reasons (I prefer to work on the product unimpeded and not feeling pride having the knowledge to babysit obsolete stacks carried by university nostalgia).
With Go, Rust, Zig, and a few others -- it's a single binary.
This is a post about keeping your infrastructure simple, so Instagram is not a good ceiling to pick. People do all kinds of hacks to scale Python before they hit Instagram levels
Unless your Cloudflare worker and the DB are scheduled onto the same physical server, they are not local to one another. I don’t know much about D1, but the overwhelming majority of cloud infra makes no such guarantees, nor are they likely to want to architect it in that manner.
Cloudflare's Durable Objects puts your Worker and SQLite DB on the same physical server (and lets you easily spawn millions of these pairs around the world).
D1 is a simplified wrapper around DO, but D1 does not put your DB on the same machine. You need to use DO directly to get local DBs.
I think the point is that your Python webapp will have more problems scaling to let's say 10,000 customers on a 5$ VPS tham Go. Of course you can always get beefier servers, but then that adds up for every project
I concur with some of the commenters that this read as a bit of a brain dump. It has a thread connecting several loosely-related topics.
Observation #1: You can also solve the tech stack problem with Heroku. I think the author's stack probably has a steeper learning curve, but is a cheaper option. I think it's a bit of an odd comparison (I won't say straw-man, as I don't doubt some people do this) to go from a fully-controlled simple setup to using AWS with a pile of extra crap. You can also, for example, run something similar to what he or she is describing on AWS, Heroku etc. (I.e. without the things in the AWS diagram he indicated like kubernetes and load balancers.)
Observation #2: I have not found WAL mode is an antidote to SQLite locks during multiple concurrent writes. (This is anecdotal)
I think regarding Go vs Python/Ruby etc. I completely get that. I would now like to check out Go on web. I use Rust for most of my software writing, but am still on Python for web servers, because there is nothing I can use for Rust that is as powerful and easy as Django.