Senior Big Data Engineer
Atradius
David Ricardostraat, 1066 JS, Amsterdam, Netherlands
5 dagen geleden

Job Description

As a Senior Big Data Engineer you are responsible for the data content and applications within the IFRS17 big data platform.

The IFRS17 chain consists of several (modern) technologies, i.e. Big Data technology that ingests and exposes the data from over 15 source systems, Azure Cloud, Power BI, SAS and Oracle Financials.

The current Big Data Platform is on Hadoop (to be migrated to Cloudera Data Platform and Cloud technology in 2023) with Informatica Big Data for the ETL (to be migrated to PySQL and PySpark in 2023).

The majority of the data streams is fully automated in the IFSR17 Chain.

You do not only understand data, and big data technologies but also have hands-on programming experience with languages like Python, Spark and SQL.

You are part of a team that works agile in close cooperation with business and our development partners.

Responsibilities :

  • Perform and develop end-2-end additional ingestions of new source data in the data platform.
  • Implement data mappings for the IFRS17 contract data
  • Maintain the interfaces with the Big Data platform
  • Develop test scripts (unit, integration and system) and integrate automation of those tests in the CI / CD pipeline
  • Conduct GDPR analysis on data on new ingestions and new use cases, and advise the Security Architect on outcomes
  • Monitor the data loads and the jobs running within the big data platform
  • Keep the documentation up-to-date on the data catalogue, the Data architecture, Data flow diagrams and Data Lineage amongst others
  • Support, propose and execute Proof of Concepts for prospective tools and improvements to the data platform
  • Participate and provide input in future roadmap discussions
  • Provide guidance and be a sparring partner for continuous improvement throughout the IFSR17 Chain.
  • Skills and Experience :

  • Experience as big data engineer in middle to large companies >
  • 5 years of work experience

  • Proficient understanding of distributed computing principles
  • In-depth knowledge of Hadoop, Cloudera Data Platform, Spark, and Databricks
  • Good knowledge of Big Data querying tools, such as Hive.
  • Good knowledge of Azure, Azure data engineering, Azure data services and modern data warehouses within Azure.
  • Experience with data modelling
  • Knowledge of various ETL techniques and frameworks
  • Experience with NoSQL and RDBMS databases
  • In-depth knowledge of scripting languages Python, Spark, SQL and R
  • Knowledge of automated testing and CI / CD, like Jenkins, GitHub and Azure pipelines
  • Experience with Agile development and working in a DevOps environment.
  • Experience with Jira and Confluence
  • Strong analytical and conceptual skills used to evaluate data gathered from multiple sources, while reconciling discrepancies
  • Thorough, analytical, and precise, but also big picture / helicopter view
  • Strong in structuring and organizing data.
  • Hands -on, proactive, and can work very focused at a fast pace.
  • Excellent communication skills (written and verbal) and stakeholder management
  • Effective IT data documentation skills.
  • Curious and thinking out-of-the-box
  • Ability to remove unneeded complexity without oversimplification
  • Strong customer focus and demonstrated ability to work in geographically dispersed teams.
  • Familiarity of orchestration / integration / API’s
  • Experience in working in an DevOps mode in an international team and setting
  • Ability to work in a team environment and efficiently under pressure
  • Fluent English, spoken and written
  • Willingness to travel
  • What do we offer?

  • A dynamic, international and challenging work environment
  • Training and support to reach your full potential including the opportunity for continuous professional development
  • Attractive terms and conditions, including competitive salary, pension package and a range of flexible benefits and rewards
  • Challenging tasks with individual development and training opportunities
  • deze vacature melden
    checkmark

    Thank you for reporting this job!

    Your feedback will help us improve the quality of our services.

    Solliciteren
    Mijn E-mail
    Door op "Doorgaan" te klikken, betekent dit dat je neuvoo toestemming geeft om je gegevens te verwerken en je e-mails met vacatures te sturen, zoals beschreven in neuvoo's -Privacybeleid . Je kunt je toestemming altijd intrekken
    Doorgaan
    Aanvraagformulier