whole setup in docker compose?

[MrClemRkz]

No description provided.

[sspaeti]

That is a good idea. In my recent big update, I switched spark with delta-rs. This means that no heavy Spark cluster is required for the pipeline part anymore. That helps, for sure. Docker might be another step. I will think more, but it should start with a couple of commands for now. I also made a video that helps with it.