Hello, I'm building a replacement for docker containers with a virtual machine with the ergonomics of containers + subsecond start times.
I worked in AWS previously in the container space + with firecracker. I realized the container is an unnecessary layer that slowed things down + firecracker was a technology designed for AWS org structure + usecase.
So I ended up building a hybrid taking the best of containers with the best of firecracker.
Hey this is super cool. I've been researching tech like this for my AI sandboxing solution, ended up with Lima+Incus: https://github.com/JanPokorny/locki
My problem with microVMs was that they usually won't run docker / kubernetes, I work on apps that consist of whole kubernetes clusters and want the sandbox to contain all that.
Does your solution support running k3s for example?
That's the one feature of similar systems that always gets left out. I understand why: it's not a priority for "cloud native" workloads. The world, however, has work loads that are not cloud native, because that comes at a high cost, and it always will. So if you'd like a real value-add differentiator for your micro-VM platform (beyond what I believe you already have,) there you go.
It helps if you offer a concrete use case, as in how large the heap is, what kinda of blackout period you can handle, and whether the app can handle all of it's open connections being destroyed, etc. The more an app can handle resetting some of it's own state, the easier LM is going to be to implement. If your workload jives with CRIU https://github.com/checkpoint-restore/criu you could do this already.
By what I assume is your definition, there are plenty of "non cloud native" workloads running on clouds that need live migration. Azure and GCP use LM behind the scenes to give the illusion of long uptime hosts. Guest VMs are moved around for host maintenance.
As does OCI, and (relatively recently) AWS. That's a lot of votes.
Use case: some legacy database VM needs to move because the host needs maintenance, the database storage (as opposed to the database software) is on a iSCSI/NFS/NVMe-oF array somewhere, and clients are just smart enough to transparently handle a brief disconnect/reconnect (which is built-in to essentially every such database connection pool stack today.)
Use case: a web app platform (node/spring/django/rails/whatever) with a bunch of cached client state needs to move because the host needs maintenance. The developers haven't done all the legwork to make the state survive restart, and they'll likely never get time needed to do that. That's essentially the same use case as previous. It's also rampant.
Use case: a long running batch process (training, etc.) needs to move because reasons, and ops can't wait for it to stop, and they can't kill it because time==money. It's doesn't matter that it takes an hour to move because big heap, as long as the previous 100 hours isn't lost.
"as in how large the heap is"
That's an undecidable moving target, so let the user worry about it. Trust them to figure out what is feasible given the capabilities of their hardware and talent. They'll do fine if you provide the mechanism. I've been shuffling live VMs between hosts for 10+ years successfully, and Qemu/KVM has been capable of it for nearly 20, never mind VMware.
"CRIU"
Dormant, and still containers. Also, it's re-solving solved problems once you're running in a VM, but with more steps.
Somewhat related: I have a branch of Ganeti that has first-class ZFS support baked in, including using ZFS snapshot replication to do live migration without shared storage or CEPH: https://github.com/linsomniac/ganeti
Current status is I'm looking for more feedback. In a few weeks when Ubuntu 26.04 comes out I'm going to set up my dev/stg clusters at work with it, at the moment I've only tested it in a test cluster at home.
It works this way: It creates a snapshot of the zvol, and replicates it to the secondary machine. When that's done, it does another snapshot and does a "catch up" replication (the first replication could take hours on large volumes). Pause the VM, do a final snapshot+replication. Replicate the working RAM. Start up the VM on the new host.
Live migrations and the tech powering it was the hardest thing I ever built. Its something that I think will come naturally to projects like smolVM as more of the hypervisors build it in, but its a deeply challenging task to do in userspace.
My team spent 4 months on our implementation of vm memory that let us do it and its still our biggest time suck. We also were able to make assumptions like RDMA that are not available.
All that to say — as someone not working on smolVMs — I am confident smolVMs and most other OSS sandbox implementations will get live migration via hypervisor upgrades in the next 12 months.
Until then there are enterprise-y providers like that have it and great OSS options that already solve this like cloud hypervisor.
I see. so right now smolvm can be stopped, and then "packed" (think of it as compressed), and restart on a different host. files in the disks are preserved, but memory snapshotting is still hard tbh
Ultimately the original does get stopped, but with additional techniques, we're talking milliseconds of downtime between when the old one stops and the new one resumes. (For live migration technology in general, no clue about smol machines.)
+1. i built something similar called shuru.run because i wanted an easy way to set up microVM sandboxes to run some of my AI apps, and firecracker wasn't available for macOS (and, as you said, it is just too heavy for normal user-level workloads).
Nice work on Shuru — I remember looking at it when I was researching this space. You went with a Rust wrapper on Apple’s Virtualization framework right?
I believe anyone with a spare linux box should be able to carve it into isolated programmable machines, without having to worry about provisioning them or their lifecycle.
The documentation’s still early but I have been using it for orchestrating parallel work (with deploy previews), offloading browser automation for my agents etc. An auction bought heztner server is serving me quite well :)
Yes, having a light-weight solution for local devices as well is one primary goal of the design. Another one is to make it easy for hosting, self or managed
I see the alpine and python:3.12-alpine images in your cli docs. Where does these come from?is it from a docker like registry or are these built in? Can I create my own images? Or this this purely done with the smolfile? Is there a Ubuntu image available?
Looks really nice btw. Hot resize mem/cpu would be nice. This could become a nice tech for a one-backend-per-customer infra orchestrator then.
Great job with the comparison table. Immediately I was like “neat sounds like firecracker” then saw your table to see where it was similar and different. Easy!
What are you actually doing on top of libkrun? Providing really small machine images that boot quickly? If I run the smolvm run --image alpine example, what is "alpine?" Where is that image coming from? Does this have some built-in default registry of machine images it pulls from? Does it need an Internet connection that allows outbound access to wherever this registry runs? Is it one of a default set of pre-built images that comes with the software itself and is stored on my own filesystem? Where are the builds for these images? Where do these machine images end up? ~/.local/share/smolvm/?
Basically any open source project nowadays run their software stack in containers often requiring docker compose. Unfortunatley Smol machines do not support Docker inside the microvms and they also do not support nested VMs for things that use Vagrant. I think this is a big drawback.
What I really like about containers is quickly being able to spin one up without having to specify resources (e.g. RAM limit). I hope this would let me do that also.
Hey this is pretty neat! I definitely would try using this for benchmarks and other places where I need strong isolation as Docker is just too bloated and slow, but sadly I don't think I can run this natively on my Windows laptop. I hope you extend to WSL! Good luck and congrats on launch.
We’re using smolmachines to create environments for our agents to execute code. It’s been great so far and the team is super responsive. The dev ergonomics are also great.
This project is very cool! One readme nit: "Pack a stateful virtual machine into a single file (.smolmachine) to rehydrate on any supported platform." For awhile I thought this meant that you could rehydrate a machine's memory like you can with a firecracker vm, but as far as I can tell you can't? It's stateful == disk?
smolvm is awesome. The team is highly responsive and very experienced. They clearly know what they’re doing.
I’m currently evaluating smolvm for my project, https://withcave.ai, where I’m using Incus for isolation. The initial integration results look very promising!
This is a very cool project and I'm happy to see it getting traction here. I stumbled upon it when I was looking to build something similar and surveying the state of the art...then I realized you built _exactly_ what I wanted!
im keen to check this out. since I've moved 100% to the Mac [1] I've been keen to move away from Docker to something like Apple Containers [2] which runs each "container" as an isolated vm. So I wanna try this out, too.
152 comments
I worked in AWS previously in the container space + with firecracker. I realized the container is an unnecessary layer that slowed things down + firecracker was a technology designed for AWS org structure + usecase.
So I ended up building a hybrid taking the best of containers with the best of firecracker.
Let me know your thoughts, thanks!
My problem with microVMs was that they usually won't run docker / kubernetes, I work on apps that consist of whole kubernetes clusters and want the sandbox to contain all that.
Does your solution support running k3s for example?
Really appreciate the feedback!
That's the one feature of similar systems that always gets left out. I understand why: it's not a priority for "cloud native" workloads. The world, however, has work loads that are not cloud native, because that comes at a high cost, and it always will. So if you'd like a real value-add differentiator for your micro-VM platform (beyond what I believe you already have,) there you go.
Otherwise this looks pretty compelling.
By what I assume is your definition, there are plenty of "non cloud native" workloads running on clouds that need live migration. Azure and GCP use LM behind the scenes to give the illusion of long uptime hosts. Guest VMs are moved around for host maintenance.
As does OCI, and (relatively recently) AWS. That's a lot of votes.
Use case: some legacy database VM needs to move because the host needs maintenance, the database storage (as opposed to the database software) is on a iSCSI/NFS/NVMe-oF array somewhere, and clients are just smart enough to transparently handle a brief disconnect/reconnect (which is built-in to essentially every such database connection pool stack today.)
Use case: a web app platform (node/spring/django/rails/whatever) with a bunch of cached client state needs to move because the host needs maintenance. The developers haven't done all the legwork to make the state survive restart, and they'll likely never get time needed to do that. That's essentially the same use case as previous. It's also rampant.
Use case: a long running batch process (training, etc.) needs to move because reasons, and ops can't wait for it to stop, and they can't kill it because time==money. It's doesn't matter that it takes an hour to move because big heap, as long as the previous 100 hours isn't lost.
"as in how large the heap is"
That's an undecidable moving target, so let the user worry about it. Trust them to figure out what is feasible given the capabilities of their hardware and talent. They'll do fine if you provide the mechanism. I've been shuffling live VMs between hosts for 10+ years successfully, and Qemu/KVM has been capable of it for nearly 20, never mind VMware.
"CRIU"
Dormant, and still containers. Also, it's re-solving solved problems once you're running in a VM, but with more steps.
Current status is I'm looking for more feedback. In a few weeks when Ubuntu 26.04 comes out I'm going to set up my dev/stg clusters at work with it, at the moment I've only tested it in a test cluster at home.
It works this way: It creates a snapshot of the zvol, and replicates it to the secondary machine. When that's done, it does another snapshot and does a "catch up" replication (the first replication could take hours on large volumes). Pause the VM, do a final snapshot+replication. Replicate the working RAM. Start up the VM on the new host.
Thanks
My team spent 4 months on our implementation of vm memory that let us do it and its still our biggest time suck. We also were able to make assumptions like RDMA that are not available.
All that to say — as someone not working on smolVMs — I am confident smolVMs and most other OSS sandbox implementations will get live migration via hypervisor upgrades in the next 12 months.
Until then there are enterprise-y providers like that have it and great OSS options that already solve this like cloud hypervisor.
I have been working on something similar but on top of firecracker, called it bhatti (https://github.com/sahil-shubham/bhatti).
I believe anyone with a spare linux box should be able to carve it into isolated programmable machines, without having to worry about provisioning them or their lifecycle.
The documentation’s still early but I have been using it for orchestrating parallel work (with deploy previews), offloading browser automation for my agents etc. An auction bought heztner server is serving me quite well :)
also, yes, shuru was (still) a wrapper over the Virtualization.framework, but it now supports Linux too (wrapper over KVM lol)
They are ext4 blocks which exist independent of sandboxes.
Probably a lot of other neat usecases for this, too
Looks really nice btw. Hot resize mem/cpu would be nice. This could become a nice tech for a one-backend-per-customer infra orchestrator then.
Nice job! This looks really cool
I’m currently evaluating smolvm for my project, https://withcave.ai, where I’m using Incus for isolation. The initial integration results look very promising!
Can you pipe into one? It would be cute if I could wget in machine 1 and send that result to offline machine 2 for processing.
[1] https://unikraft.org
Cheers!
Thank you, great work!
[1] shameful self plug: https://gigatexal.blog/pages/i-heart-my-macbook/i-heart-my-m...
[2] https://github.com/apple/container
https://docs.docker.com/reference/cli/sbx/
*yes, FreeBSD is specifically developed against Firecracker which is specifically avoided w "Smol machines", but interesting nonetheless
[0] https://github.com/NetBSDfr/smolBSD
[1] https://www.usenix.org/publications/loginonline/freebsd-fire...
question: why do you report that qemu is 15sthanks a lot