Airflow 3 – the new beginning
Session Abstract
If you were living under the rock and have not heard that Airlfow 3 is out, and solves most of the pain points that you had with Airflow 2, this talk is for you. You will learn how you can boost your Data Engineering and AI/ML workflows (without having to rewrite your DAGs) with what Airlfow 3 community worked for last 12 months.
Session Description
Airflow 3 is out. Spring last year Airlfow community came to conclusion that in order to respond to a number of users of ours, we have to again reinvent ourselves and release Airflow 3. Berlin Buzzwords is at the right time and the right place to talk about it, as at the time of Buzzwords, Airflow 3 will already be out for a while and we will know not only what we planned, but how our users already use Airflow 3 and what benefit it brings for them.
In this talk you will learn from Jarek, one of the top maintainers of Airflow all about you want to know about Airflow 3:
* why we decided to change to Airflow 3
* what are the architectural changes and improvements that lay foundation under Airflow 3 being modern and applicable to more workflows
* what are the features you always wanted and you can use now: Versioning, Enterprise level security isolation, better dependency management, execution isolation, dataset as first class citizen, modern react-based UI, schedulable and UI-controlled backfills, ML/AI worfklows including inference workflows, almost streaming experience, and more
* what early users of Airlfow 3 say and how their workflow management improved
* what is coming next (yes! we are not nearly done yet and more things are coming!)
Last year, I presented “new orchestrator in town” – Airflow 2, but that was only a Prelude. If you want to hear the whole Symphony come to see the talk.
