However my approach would be to use duckdb and S3 over lambda.
Leaving many of the concerns to the infrastructure. Like basically no OOM. No need to manage servers.