As a Senior Big Data Engineer you are responsible for the data content and applications within the IFRS17 big data platform.
The IFRS17 chain consists of several (modern) technologies, i.e. Big Data technology that ingests and exposes the data from over 15 source systems, Azure Cloud, Power BI, SAS and Oracle Financials.
The current Big Data Platform is on Hadoop (to be migrated to Cloudera Data Platform and Cloud technology in 2023) with Informatica Big Data for the ETL (to be migrated to PySQL and PySpark in 2023).
The majority of the data streams is fully automated in the IFSR17 Chain.
You do not only understand data, and big data technologies but also have hands-on programming experience with languages like Python, Spark and SQL.
You are part of a team that works agile in close cooperation with business and our development partners.
Perform and develop end-2-end additional ingestions of new source data in the data platform.
Implement data mappings for the IFRS17 contract data
Maintain the interfaces with the Big Data platform
Develop test scripts (unit, integration and system) and integrate automation of those tests in the CI / CD pipeline
Conduct GDPR analysis on data on new ingestions and new use cases, and advise the Security Architect on outcomes
Monitor the data loads and the jobs running within the big data platform
Keep the documentation up-to-date on the data catalogue, the Data architecture, Data flow diagrams and Data Lineage amongst others
Support, propose and execute Proof of Concepts for prospective tools and improvements to the data platform
Participate and provide input in future roadmap discussions
Provide guidance and be a sparring partner for continuous improvement throughout the IFSR17 Chain.
Skills and Experience :
Experience as big data engineer in middle to large companies >
5 years of work experience
Proficient understanding of distributed computing principles
In-depth knowledge of Hadoop, Cloudera Data Platform, Spark, and Databricks
Good knowledge of Big Data querying tools, such as Hive.
Good knowledge of Azure, Azure data engineering, Azure data services and modern data warehouses within Azure.
Experience with data modelling
Knowledge of various ETL techniques and frameworks
Experience with NoSQL and RDBMS databases
In-depth knowledge of scripting languages Python, Spark, SQL and R
Knowledge of automated testing and CI / CD, like Jenkins, GitHub and Azure pipelines
Experience with Agile development and working in a DevOps environment.
Experience with Jira and Confluence
Strong analytical and conceptual skills used to evaluate data gathered from multiple sources, while reconciling discrepancies
Thorough, analytical, and precise, but also big picture / helicopter view
Strong in structuring and organizing data.
Hands -on, proactive, and can work very focused at a fast pace.
Excellent communication skills (written and verbal) and stakeholder management
Effective IT data documentation skills.
Curious and thinking out-of-the-box
Ability to remove unneeded complexity without oversimplification
Strong customer focus and demonstrated ability to work in geographically dispersed teams.
Familiarity of orchestration / integration / API’s
Experience in working in an DevOps mode in an international team and setting
Ability to work in a team environment and efficiently under pressure
Fluent English, spoken and written
Willingness to travel
What do we offer?
A dynamic, international and challenging work environment
Training and support to reach your full potential including the opportunity for continuous professional development
Attractive terms and conditions, including competitive salary, pension package and a range of flexible benefits and rewards
Challenging tasks with individual development and training opportunities