Databricks is a market leader in big data and machine learning platforms used by thousands of companies. We are seeking a strategic technical leader who will be responsible for managing the engineering teams that ensure the performance of the Databricks Runtime (built on Apache Spark) is best in class.
This group currently consists of two quickly growing teams : works on the next generation execution engine for the Databricks Runtime, and develops performance improvements ranging from Spark’s optimizer (Catalyst) to its code generation framework (Tungsten).
They combine large amounts of data about performance bottlenecks with insights from the product management and field teams to improve performance and scalability where it matters most to our customers.
builds the infrastructure to fully automate the process of running a wide range of performance benchmarks on every change to the Databricks Runtime, so that the engines becomes faster all the time.
In this role, you’ll report to the head of our European R&D centre in Amsterdam, The Netherlands. Constant collaboration with the broader company is crucial.
In this role, you will work closely with engineering leaders across the company, as well as with product management, field engineering, recruiting and HR to ensure organizational alignment.
Continuous improved on performance and stability on the key metrics and benchmarks you’ll help identify, while not regressing on performance or correctness Grow a world class team of software engineers who work on Spark Runtime performance.
Increase headcount by 10+ engineers in next 18 months, with continued growth beyond that according to product objectives Ensure consistent delivery of milestones and goals Manage technical debt, including long term technical architecture decisions and balance product roadmap Previous experience building and leading teams in a complex technical domain, such as on distributed data systems or database internals Ability to attract, hire, and coach strong systems engineers who meet the Databricks hiring standards Management of growing team, including managing technical leads Ability to communicate across departments, with product management and customers