lola

A cloud you run yourself, that speaks AWS. S3, DynamoDB, Lambda and CloudFormation on one port, over the real wire protocols — so the AWS CLI, the SDKs and cdk deploy work against it unmodified.

Not a mock. Objects are stored durably and content-addressed; functions run as real processes under real cgroup limits, seccomp filters and network namespaces; conditional writes are genuinely atomic under a lock. Nothing here is a test double standing in for the real thing — it is the thing, at a scale you can run on one machine.

Two things it is built around:

Cold-start latency. Lambda invocations run in warm sandbox pools and nothing expensive is allowed on the invoke path. That is sub-millisecond on Linux, against roughly 10–25ms for a Rust function on real Lambda and hundreds of milliseconds to seconds for anything that starts a container per invoke.

Saying what it does not do. Every unimplemented feature returns a real AWS error rather than a plausible wrong answer, and stubbed CloudFormation resources announce themselves. Where lola diverges from AWS — replication lag, pagination, async invoke durability — it is written down below rather than discovered in production.

It is early: single node, no replication, and the isolation boundary is not yet one you should put untrusted code behind. Those gaps are enumerated further down, not glossed over.

Measured on this machine (M-series Mac, --release)

exec floor of the handler binary itself	1.4 ms (hyperfine, n=30)
lola cold start (spawn + init + invoke + respond)	2.1 – 3.5 ms
lola warm invoke, p50	0.159 ms
lola warm invoke, p95 / p99 / max	0.281 / 0.528 / 0.728 ms

So orchestration overhead over the raw exec floor is under 2ms. Reproduce it yourself with ./scripts/bench.sh — never take a claimed cold-start number on trust.

macOS quirk: the very first invoke after CreateFunction shows ~120–150ms. That is amfid validating a newly written binary's code signature, not lola. Every subsequent cold start of the same file is 2–3ms. bench.sh demonstrates this directly by timing a stable binary against a freshly copied one.

Cold start is a property of the runtime, not of lola

Same server, same sandbox pool, three handlers (./scripts/bench-runtimes.sh):

handler	cold start	warm p50
Rust, std-only	2.80 ms (2.48 – 3.30)	0.16 ms
Java 25 (Corretto), plain JVM	134 ms (127 – 153)	0.80 ms
Java 25 + AOT cache (Leyden, JEP 514)	157 ms (149 – 167)	0.74 ms

Two results worth not glossing over:

The AOT cache made cold start worse here — 157ms vs 134ms — even though the same cache is faster in isolation (375ms vs 427ms for a full standalone JVM lifecycle under hyperfine). The cache is 22MB and has to be mapped on every cold start, and the training run in examples/java/Handler.java --train exercises a different path from the real polling loop, so it pays the mapping cost without caching the classes that matter. Leyden earns its keep on large applications where class loading dominates startup; on a 100-line handler it is overhead. Retrain against the real invoke path before trusting it.

Standalone JVM startup (427ms) is much higher than lola's cold start (134ms) because hyperfine measures a full process lifecycle including JVM shutdown, whereas a cold start only has to reach the first /next response. Boot-to-first- response is the number that matters.

If you want Java with Rust-like cold starts, the answer is GraalVM native-image — it produces a binary with no JVM to start, so it lands in the same class as the Rust handler. AWS's own answer for managed Java runtimes is SnapStart, which is Firecracker snapshot/restore — the same mechanism as lola's M3 backend.

Quick start

cargo build --release
./scripts/package-example.sh              # builds build/hello.zip
./target/release/lola --data-dir ./data   # listens on :4566
./scripts/smoke.sh                        # full round-trip via the real AWS CLI
./scripts/bench.sh                        # the three latency numbers

Point any AWS tooling at it:

export AWS_ENDPOINT_URL=http://127.0.0.1:4566
export AWS_ACCESS_KEY_ID=test AWS_SECRET_ACCESS_KEY=test AWS_DEFAULT_REGION=us-east-1

aws s3 mb s3://demo
aws dynamodb list-tables
aws lambda create-function --function-name hello --runtime provided.al2023 \
  --handler bootstrap --role arn:aws:iam::000000000000:role/x \
  --zip-file fileb://build/hello.zip
aws lambda invoke --function-name hello --payload '{}' /dev/stdout

Keep the region us-east-1 and override the endpoint, not the region. A fake region breaks SigV4 signing, CDK's RegionInfo tables, and AZ lookups; an endpoint override breaks nothing. Using an IP endpoint also forces S3 path-style addressing, which is what you want.

GET /_lola/health reports the backend, counts, and live functions.

What works

S3 — CreateBucket, DeleteBucket, HeadBucket, ListBuckets, ListObjects(V1/V2) with prefix/delimiter/pagination, PutObject, CopyObject, GetObject, HeadObject, DeleteObject, DeleteObjects, GetBucketLocation. Objects are content-addressed blobs on disk, so identical bytes are stored once and object keys never touch the filesystem (no path-traversal surface). Persists across restarts.

DynamoDB — CreateTable, DescribeTable, ListTables, DeleteTable, PutItem, GetItem, UpdateItem, DeleteItem, Scan, Query, BatchWriteItem, BatchGetItem. Sorting is type-aware, so numeric sort keys order 1, 2, 10 rather than lexicographically. Persists across restarts.

Expressions are parsed properly (src/ddb_expr.rs, recursive descent, 17 unit tests) rather than string-matched:

• ConditionExpression on PutItem, UpdateItem and DeleteItem — comparators, attribute_exists / attribute_not_exists / attribute_type, begins_with, contains, size(), BETWEEN, IN, AND / OR / NOT with correct precedence, parentheses, and nested document paths (#p.langs[0])
• FilterExpression on Scan and Query, applied after the read, so ScannedCount exceeds Count exactly as on real DynamoDB
• UpdateExpression: SET / REMOVE / ADD / DELETE, with if_not_exists, list_append, arithmetic, and set union/difference. Nested paths are refused rather than half-applied.
• ReturnValues: NONE, ALL_OLD, ALL_NEW, UPDATED_OLD, UPDATED_NEW

Conditional writes are the point. The condition is evaluated while the table lock is held, so read-modify-write is atomic — which is what makes compare-and-swap usable, and hence queues with visibility timeouts, distributed locks, idempotency keys and state-machine transitions. ./scripts/smoke-ddb.sh proves it through the real AWS CLI, including 50 concurrent ADD operations landing on exactly 50 with no lost updates, and a stale-version write being rejected with ConditionalCheckFailedException.

Lambda — CreateFunction, GetFunction, GetFunctionConfiguration, ListFunctions, DeleteFunction, UpdateFunctionCode, Invoke (RequestResponse / Event / DryRun), plus the full Runtime API. Code comes from an inline base64 zip or from an S3Bucket/S3Key pointer into lola's own S3. Functions are not restored across restarts — a function is code plus a live Runtime API listener, so recreate it with CreateFunction.

STS — GetCallerIdentity, AssumeRole. Small but load-bearing: this is how the SDKs and the CDK CLI resolve the account id they stamp into ARNs.

Invoke responses carry two lola-specific headers for benchmarking: x-lola-cold: true|false and x-lola-duration-ms.

What does NOT work

Anything unimplemented returns a real AWS error shape (NotImplementedException, NotImplemented) rather than a wrong answer. Silently ignoring your FilterExpression and returning unfiltered rows would be worse than not implementing it at all: you would ship the bug to production.

• cdk deploy works, but the CDK CLI then exits with a TypeError. The stack reaches CREATE_COMPLETE and every resource is created; the CLI's progress monitor reads a response field lola does not yet return. Deploys correctly, exits ungracefully. cdk bootstrap is not implemented either — seed the staging bucket and the /cdk-bootstrap/hnb659fds/version SSM parameter by hand.
• CloudFormation has no rollback. A failed stack stays failed rather than reverting, and DeleteStack forgets the stack without deleting its resources — destroying data silently would be the worse failure. No drift detection, nested stacks or custom resources.
• No container-image functions. M1 is the provided.* fast lane only: the zip must contain an executable named bootstrap.
• Isolation is Linux-only, and still not a boundary for untrusted code. On Linux as root you get cgroup v2 limits, a per-function uid, no_new_privs, a seccomp allowlist, and pid/mount/network namespaces with egress blocked. What is missing is pivot_root, so the sandbox still sees the host filesystem, and a user namespace. On macOS there is no isolation at all.
• DynamoDB: no ProjectionExpression, secondary indexes, transactions, streams, or UpdateExpression on nested document paths. No pagination (LastEvaluatedKey). ConsistentRead is honoured — see the consistency section.
• S3: no multipart upload, versioning, ACLs, tagging, policies, encryption config, lifecycle, or range GETs.
• Lambda: no versions/aliases, layers, function URLs, event source mappings, provisioned concurrency, or UpdateFunctionConfiguration.
• No CloudWatch Logs surface — function stdout/stderr goes to lola's stderr.

Consistency: lola can be as unfaithful as AWS, on purpose

DynamoDB defaults ConsistentRead to false, so a read straight after a write can legitimately return stale data. Always returning fresh data would let that bug pass locally and fail in production — being more correct than AWS is a hazard, not a feature. If the use case is CI, it is precisely the failure mode that destroys trust in a test suite.

So lola models replication lag, in two modes:

A lint that is always on. Every read that omits ConsistentRead within ~1s of a write to the same key is counted, with no change to what is returned. Check it at GET /_lola/health:

{ "eventual_lag_ms": 0, "stale_read_warnings": 10 }

Nonzero means latent consistency bugs in the caller. This found ten of them in lola's own smoke tests the moment it was switched on.

Real staleness, opt-in. --eventual-lag-ms 800 serves what a lagging replica would, so those bugs fail here instead of in production:

	eventually consistent read	ConsistentRead: true
straight after an insert	item absent	item present
straight after an overwrite	the previous value	the new value
straight after a delete	item still present	absent
after the lag elapses	current value	current value

Honoured by GetItem, BatchGetItem (per-table), Query and Scan. The deleted-item case matters: a lagging replica has not seen the delete, and omitting that would make the simulation wrong in the direction that hides bugs.

Default is 0, because switching it on changes read results and would break existing tests confusingly. Turn it on in CI. ./scripts/smoke-consistency.sh proves all of the above through the real AWS CLI.

Worth knowing what this is not: lola still serializes everything under one lock, so it does not model concurrent-writer anomalies, only read staleness.

Still unfaithful in other ways

• Async Lambda invoke (InvocationType: Event) is a tokio::spawn: lost on restart, no retry, no DLQ, no ordering. A durable queue (M8b) comes before event sources for exactly this reason — and now that conditional writes exist, it can be built on DynamoDB the way real systems do it.
• No pagination: LastEvaluatedKey is never returned, so a caller that loops until it is absent will work here and truncate at 1MB on real AWS.

Networking: borrow a network, do not emulate one

The right move for the whole VPC/Route 53/ELB tier is not to implement it. Put lola on a Tailscale tailnet and the operator-facing networking surface is someone else's problem:

AWS thing	Tailscale equivalent	lola code
VPC, subnets, security groups	tailnet + ACLs	none
Route 53 private zones	MagicDNS (lola.<tailnet>.ts.net)	none
ALB + ACM certificates	tailscale serve (TLS terminated for you)	none
VPN, bastion, PrivateLink	the tailnet itself	none
Public endpoint	tailscale funnel	none

tailscale up
tailscale serve --bg 4566        # https://lola.<tailnet>.ts.net -> 127.0.0.1:4566
./target/release/lola --secret-key "$(openssl rand -hex 24)"

lola keeps listening on loopback only; Tailscale handles identity and TLS, and access control becomes an ACL file instead of a security-group implementation. Set --secret-key when you do this — the port is now reachable by your tailnet, and the default secret is public knowledge.

Two things this does not solve, so be clear-eyed:

1. Per-function egress. Tailscale is host-level. The sandbox network namespace currently blocks all outbound traffic, and giving functions controlled egress needs a veth pair plus nftables NAT inside the namespace — still lola's job.
2. The VPC resources a CDK stack declares. cdk deploy needs CreateVpc and friends to return plausible IDs even with nothing behind them. Tailscale gives real connectivity; the stubs exist only so stacks synthesize. Complementary, not alternatives.

Can I put this on a server?

Not yet. It is a real, working system and the right foundation, but today it is a development tool. Three things are disqualifying:

1. No authentication. Signatures are not verified. Anyone who can reach the port can CreateFunction + Invoke, which is arbitrary code execution as the user running lola. Do not expose this port — not to the internet, not to an untrusted network segment.
2. Isolation is incomplete. On Linux as root you get cgroup limits, a per-function uid, no_new_privs, a seccomp allowlist, and pid/mount/network namespaces — so a function cannot OOM the host, fork-bomb it, read another function's files, see host processes, or reach the network beyond lola's own API. What is still missing is pivot_root: it can read the host filesystem wherever uid permissions allow. That is not yet a boundary for untrusted code; M3 (Firecracker) is. On macOS there is no isolation whatsoever.
3. Resource limits only bind on Linux. MemorySize maps to memory.max under the namespace backend, and CPU follows Lambda's memory ratio. Under the process backend — the only option on macOS — every limit is ignored.

Also unfixed, and relevant even for trusted code:

• The index flush is not crash-safe. s3.rs/ddb.rs persist by rewriting the whole index with std::fs::write — truncate, then write, no fsync, no atomic rename. A crash mid-flush leaves a truncated JSON file and loses that bucket or table. Needs write-to-temp + fsync + rename.
• O(n) write amplification. Every PutObject rewrites the entire bucket index. Fine for thousands of keys, not for millions.
• Lambda functions do not survive a restart; the invoke queue is unbounded (no admission control); each service is behind one global mutex; single process, no HA.

Today it can reasonably serve trusted, first-party code on a private network, on Linux, if you accept the durability caveats. It cannot serve untrusted or multi-tenant code. The minimum for a real deployment is M2d + M4 (auth) + the crash-safe flush; the minimum for untrusted code is M3.

Architecture

                    one HTTP listener (:4566)
                              │
        ┌─────────────┬───────┴────────┬──────────────┐
        │ Lambda      │ DynamoDB       │ STS          │ S3
        │ REST paths  │ X-Amz-Target   │ Action= form │ (everything else)
        └──────┬──────┴────────────────┴──────────────┘
               │
               ▼
   one Runtime API listener per function version, on 127.0.0.1:0
               │  GET /2018-06-01/runtime/invocation/next
               │  POST .../invocation/{id}/response | /error
               ▼
        warm sandbox pool  ──▶  sandbox::Backend
                                  ├─ Process      (M1, no isolation)
                                  ├─ Namespace    (M2a-c: cgroup+uid+seccomp+pid/mnt/net ns)
                                  └─ Firecracker  (M3, microVM, ~5-30ms)

Two design decisions carry most of the weight:

One Runtime API listener per function, shared by all its sandboxes. Every warm sandbox long-polls the same GET /next, so invocations distribute themselves with no scheduler. It is also faithful to production — with real network namespaces each sandbox reaches its own function's endpoint and nothing else.

Nothing expensive may touch the invoke path. Namespace creation, cgroup setup, seccomp compilation and rootfs mounting all belong to pool fill. That rule is why the number above is 2ms and not 50ms, and it is the rule that has to survive contact with the Linux backends.

Source map:

file	role
src/runtime.rs	Runtime API + warm pool + invoke path — the core
src/sandbox.rs	the isolation seam; add a Backend variant, touch nothing else
src/lambda.rs	Lambda control plane, zip install
src/s3.rs, src/ddb.rs, src/sts.rs	service surfaces
src/main.rs	per-service request dispatch
src/bin/bootstrap.rs	example function; std-only, doubles as Runtime API docs

src/bin/bootstrap.rs is deliberately written with no async runtime and no HTTP crate. That is the entire reason a Rust handler starts in 1.4ms — there is nothing to initialise. It is also the shortest readable statement of the Runtime API contract.

Roadmap

Target: self-hosted production FaaS, with cdk deploy working against it so the same stacks move to real AWS unchanged.

"M1", "M2" etc. are this project's own labels, not AWS terminology.

	milestone	proves
M1	done — Runtime API + pool + S3/DDB/Lambda/STS	real SDKs work; sub-ms cold start on Linux
M2a	done — cgroup v2 limits + per-function uid + no_new_privs + seccomp allowlist	real resource limits, verified from /proc
M2b	done — clone3(CLONE_INTO_CGROUP) spawn path, own pid reaping	isolation costs ~0.4ms, not ~6.5ms
M2c	done — pid + mount + net namespaces, with lola's API bridged into the netns	egress blocked, host pids hidden, for ~0.65ms
M2d	pivot_root into a per-function rootfs	the sandbox stops seeing the host filesystem
M3	Firecracker backend, snapshot/restore over a userfaultfd memory backend	untrusted multi-tenant code — AWS's actual boundary
M4	SigV4 verification, per-invocation credential vending, quotas, CloudWatch Logs	production-shaped
M5	CFN-lite: changesets, intrinsics, resource graph, stack events + S3/SSM/ECR/IAM stubs for cdk bootstrap	cdk deploy works
M8a	done — ConditionExpression / FilterExpression / UpdateExpression	compare-and-swap, so coordination is possible
M8c	done — ConsistentRead + simulated replication lag + a stale-read lint	consistency bugs fail locally, not in production
M8b	a durable queue built on the CAS primitive	async invoke stops being fire-and-forget
M6	Event sources: Function URLs, API Gateway v2 payload shape, SQS, EventBridge	real apps run
M7	Zygote lane for Python/Node — fork from a pre-warmed interpreter	~3ms Python cold start, beating real Lambda

M2a/M2b: what landed, what it costs, and what it does not cover

Asserted from outside the process against /proc and /sys, not against lola's own logs — ./scripts/linux-isolation-test.sh:

PASS namespace backend active
PASS runs as unprivileged uid 61723 (lola runs as 0)
PASS seccomp filter mode active (Seccomp=2, filters=1)
PASS no_new_privs set
PASS in cgroup /lola/fn-hello
PASS memory.max == MemorySize (128MB)
PASS pids.max == --pids-max (64)
PASS cgroup is accounting the sandbox
PASS no zombies leaked after 30 cold starts

Cost of isolation, on Linux 6.19 / aarch64:

	cold start	warm p50
process backend (no isolation)	0.73 ms	0.109 ms
namespace backend	1.16 ms	0.120 ms

Warm latency is untouched, confirming the pool architecture is unaffected.

The cgroup join used to cost 7ms

The first cut of this backend joined the cgroup by writing the child's pid to cgroup.procs from a Command pre_exec hook, and cold start went from 0.84ms to 7.35ms — all of it in that one write:

step applied	migration path	clone3 path
none	0.84 ms	0.77 ms
cgroup join only	8.08 ms	0.90 ms
everything	7.35 ms	1.62 ms

Migrating a task takes percpu_down_write(&cgroup_threadgroup_rwsem), which goes through rcu_sync_enter and therefore synchronize_rcu(). Measured directly, with no lola involved:

back-to-back writes:      7113us 455us 440us 431us 371us 358us 391us 378us
after 3s idle each time:  8080us 14463us 8773us 17643us

Frequent writers keep the rcu_sync state in GP_PASSED and take a fast path; cold starts are infrequent by definition, so every one paid a full grace period. Raising MemorySize did not help, which is how CPU throttling was ruled out (128MB → 12ms, 3538MB → 9ms).

clone3(CLONE_INTO_CGROUP) (Linux 5.7+) creates the child already inside the target cgroup, so the migration path is never entered. Command cannot do this — it always fork+execs — so sandbox/linux.rs issues clone3 itself and reaps its own pids. The migration path remains as a fallback, selected by a runtime probe rather than a kernel-version check.

Verified on Amazon Linux 2023, not just in a VM

The same suite run against a t4g.small in EC2 — Amazon Linux 2023.12, kernel 6.18, arm64, i.e. exactly the provided.al2023 environment lola targets:

PASS using clone3(CLONE_INTO_CGROUP), no fallback
PASS handler is pid 1, so it is in its own pid namespace
PASS pid / mnt / net namespaces all differ from lola's
PASS only loopback exists in the sandbox netns
PASS external egress is blocked (network unreachable)
PASS lola's own API is reachable inside the netns (bridge works)
PASS uid 61723 / Seccomp=2 / no_new_privs / cgroup limits enforced
PASS no zombies leaked

on t4g.small (AL2023)	cold start	warm p50
process backend	1.20 ms	0.33 ms
namespace backend	2.67 ms	0.32 ms

Slower than the M-series numbers above, as expected from a burstable Graviton2 vCPU rather than an M5 core — and noisy with it: one warm sample hit 2.67ms against a 0.25ms p50, which is CPU steal, not lola. Treat any sub-millisecond figure from a t-series instance as indicative only. Provision the box with ./scripts/ec2-testbox.sh up; it creates one instance, a key pair and a security group allowing SSH from your address only, tags everything Project=lola, and down removes all of it.

Two bugs this exercise caught

Worth recording, because both produced passing tests while being wrong:

1. target_sh runs under sudo, target_put copies as the login user. With a relative remote path, $HOME resolved to /root for one and /home/ec2-user for the other. The remote project path is now resolved once, absolutely, at source time.
2. rsync --exclude target matches any path component named target, not just the top-level directory — so it silently excluded scripts/target/ too. The packaging step then failed, and the suite passed anyway using a build/create-*.json rsynced up from the dev machine. Same architecture, so it worked; it was still not a real verification. Excludes are now anchored (/target), and build/ is no longer synced at all so stale artifacts cannot travel.

Honest caveat on the per-step numbers

Below about 1ms the per-step attribution stops being resolvable on this VM. From ./scripts/linux-attribute-cost.sh (n=8, MemorySize=1769 so CPU is not a confounder):

steps applied              mean  p50   max   n
<none>                     0.77  0.77  1.54  8
cgroup                     0.90  0.84  1.34  8
uid                        1.34  1.31  1.70  8
seccomp                    1.73  1.75  1.95  8
cgroup uid                 1.25  1.23  1.47  8
all                        1.62  1.64  1.87  8

The individual steps imply 1.66ms of overhead, but all three together measure 0.85ms. Those cannot both be true, so at this scale the run-to-run noise on a 5-vCPU / 1.9GB VM exceeds the effect being measured. What is solid is the before/after on the one step that dominated: cgroup-only went 8.08ms → 0.90ms. Resolving the rest needs a quieter host, more samples, or direct syscall timing instead of end-to-end invoke timing.

M2c: pid, mount and network namespaces

The sandbox is now created with CLONE_NEWPID | CLONE_NEWNS (folded into the same clone3 call) and setns'd into a per-function network namespace. Asserted from outside:

PASS handler is pid 1, so it is in its own pid namespace
PASS pid namespace differs from lola's (pid:[4026532715] vs pid:[4026531836])
PASS mnt namespace differs from lola's (mnt:[4026532714] vs mnt:[4026531832])
PASS net namespace differs from lola's (net:[4026532548] vs net:[4026531833])
PASS only loopback exists in the sandbox netns
PASS external egress is blocked (network unreachable)
PASS lola's own API is reachable inside the netns (bridge works)

CLONE_NEWNS is not optional company for CLONE_NEWPID: procfs reports the pid namespace of whoever mounted it, so without a private mount namespace to stack a fresh /proc in, a new pid namespace would still show the host's processes.

The network namespace is the interesting one. A fresh netns has no route anywhere — including to the Runtime API, which is TCP on 127.0.0.1. The resolution is ordering: create the namespace at pool-fill time, bind the listeners inside it, and hand the namespace to the sandbox rather than the reverse. A socket bound inside a netns stays bound there no matter which thread later accepts on it, so lola serves it from ordinary runtime threads. There is no syscall to create a namespace without entering it, so a helper child unshares, brings up lo, and parks while the parent captures /proc/<pid>/ns/net; holding that fd keeps the namespace alive after the helper is killed. Binding then happens on a dedicated std::thread, because setns acts on the calling thread and moving a tokio runtime thread into the namespace would be both wrong and hard to undo.

Two listeners are bound inside each function's netns: the function's Runtime API, and lola's own API on its normal port, so an AWS SDK inside a sandboxed function can still reach this S3/DynamoDB/Lambda implementation. AWS_ENDPOINT_URL is set in the sandbox environment to match. Without that bridge, cutting egress would also cut functions off from the services they exist to use.

The egress claim is tested with controls in both directions — the host is confirmed reachable first (or the assertion is skipped as meaningless), and the same probe run with LOLA_SANDBOX_SKIP=netns reports connected:

	egress to 1.1.1.1:443	lola's API
VM itself (control)	301	—
netns enabled	network unreachable	HTTP/1.1 200 OK
netns disabled	connected	—

Cost of the whole isolation stack:

	cold start	warm p50
process backend (none)	0.85 ms	0.142 ms
namespace backend (all of M2a-c)	1.49 ms	0.128 ms

What is still missing

• No pivot_root — the sandbox still sees the host filesystem, restricted only by uid permissions. This is the last M2 gap (M2d) and the most significant one.

  M2d is half-landed. build_rootfs() and mount_ops() in src/sandbox/linux.rs assemble a per-function rootfs skeleton (/var/task, /tmp, /proc, six /dev nodes, a read-only slice of the host /usr and /etc, usr-merged symlinks) and plan the mount syscalls. They compile but are not wired up: Prepared does not hold a Rootfs, ExecPlan does not carry the mount ops, and the child does not call pivot_root. Finishing it means adding those three connections plus rewriting the exec path to /var/task/<entrypoint> and LAMBDA_TASK_ROOT=/var/task — which is also what makes lola more faithful to real Lambda, since those are the real paths. The same rootfs artifact is what a Firecracker guest needs, so it is not throwaway work.

• No user namespace — the sandbox's uid is a real host uid, not a mapped one.

• The seccomp default action is EPERM rather than KillProcess, matching Docker's default profile. Once the allowlist is validated per runtime, KillProcess is the stronger choice.

• M3 (Firecracker) is still what untrusted multi-tenant code needs. M2 is a shared-kernel boundary; AWS moved off exactly this model for exactly that reason.

Running the Linux scripts against any host

The sandbox backend only builds and runs on Linux, so every linux-*.sh script routes through scripts/lib/target.sh. Pick a target with LOLA_TARGET:

value	meaning
podman (default)	the podman machine VM; cross-builds in a rust container
local	this machine, which must be Linux; native cargo build
ssh:<host>	a remote box; rsyncs the project and builds natively

./scripts/linux-isolation-test.sh                              # podman VM
LOLA_TARGET=ssh:lola-box ./scripts/linux-isolation-test.sh     # remote box
LOLA_TARGET=local        ./scripts/linux-attribute-cost.sh     # already on Linux

For ssh: targets use an alias from your ~/.ssh/config — the scripts never take a password, host key or secret, so credentials stay in your own ssh agent. The project lands in $LOLA_REMOTE_DIR (default ~/lola), and data/, target/ and target-linux/ are excluded from the sync so remote state and build outputs are never clobbered.

The function zip is built on the target by scripts/target/package-fn.py, because for a remote target the Linux binary only exists there. That script needs nothing but python3, and sets the executable bit on the archive entry explicitly — without it the extracted bootstrap is not executable and the sandbox dies on execve.

Two things to keep in mind for M3, because they are real bugs and not footnotes: a restored snapshot is a clone, so every VM wakes with identical RNG state, stale clocks and dead TCP connections — re-seed from virtio-rng, fix time via a paravirt clock, reconnect sockets. And N VMs restored from one snapshot must share a page-cache-hot memory file, or you pay N copies of the same working set.