AirFlow ETL Processes
Our client already had an existing ETL process workflow, while working and efficient. It was difficult to maintain and to audit. Our task was to implement a solution that would allow to run the same ETLs in a more auditable way, and make the DevOps pipelines more effective.
For this task we used Apache Airflow, and quickly established a cluster and server configuration using our client’s existing infrastructure. The deployment pipelines vary, but they are mostly are handled though GitHub and CircleCI.
We were able to deliver excellent results on-time, now our client is happy and porting all their ETLs to the new services, and creating new ones! :)
