Stream S3 > zip > CSV > Postgres with Elixir

I was recently confronted with this situation: an S3 bucket contains a zip file the zip file contains a CSV I want to take the content of that CSV, transform it, and push it to a Postgres database Without streaming Without streaming, the steps to do that are the following: download the zip archive locally (maybe it is huge!) unzip the archive open the CSV file and load its content in memory (maybe it is huge!...

March 25, 2022 · 3 min · Francis Chabouis