As a key member of the Data team, you will ensure that data available from source systems is consolidated, accurate and available for BI/Analytics consumption across the European network of our client. Furthermore, you will work on making data easily discoverable and consumable while ensuring that these datasets conform to relevant compliance and data security policies and standards.
Your major duties will be to:
• Collaborate on data migration initiatives from on-premises to our cloud-hosted Enterprise Data Lake and automate processing capabilities
• Establish scalable, secure and automated data pipelines and infrastructure to enable data analysis and collection
• Support and maintain our legacy on-premises data warehouse environments
• Provide coaching and mentoring to others within the technical discipline
• Lead complex technical or product efforts across product families and be recognised as a technical advisor in the development of creative functionality to solve complex problems.
Skills we are looking for:
• 6+ years of experience with ETL, Data Modeling, Data Warehousing and working with large-scale datasets
• 6+ years of experience with Data Engineering and the respective tools and technologies (e.g. Spark, Hadoop, Airflow, Databricks, SQL, NoSQL, Data Lake concepts)
• 6+ years of experience designing, building and operating robust distributed systems
• Expert knowledge of Python and data-related interfaces/libraries like PySpark, Pandas, Koalas or NumPy
• Knowledge of AWS services such as Lambda, S3, API Gateway etc.
• Proven deep experience leveraging DevOps and lean development principles such as Continuous Integration, Continuous Delivery/Deployment using tools like Git, Jenkins, etc.
• Experience with Clean Code, TDD, Peer Reviews and Pair Programming
• Experience working in an agile team environment
• Fluency in English is essential (both spoken and written)
• Master’s degree in IT/Computer Science/Engineering or equivalent work experience
Interpersonal, Leadership and Teaming Competencies:
• Accountable, curious and collaborative
• Maniacal about product quality and takes pride in your craft
• A natural teacher of engineering best practices, and a culture carrier for the team.
• Capable of collaborating with multiple business stakeholders to provide information technology solutions
• Skilled in interpersonal communications, negotiation, and conflict resolution
• Ability to work in areas outside of your usual comfort zone and show motivation for personal growth