Lead AI Data Engineer

Lead Data Engineers deliver the data engineering vision for the organisation as set by the Principal Data Engineer. As a Lead Data Engineer, you will focus on designing and implementing numerous complex data flows to connect operational systems, data for analytics and our emerging AI-enabled services and systems, as part of the AI as a Service platform team

You will identify and deliver new data sources and develop ETL processes designed to enable both human and AI/machine consumption, collaborate with the Data Acquisition, Analytics and Infrastructure teams to deliver the data strategy through the design and implementation of technology systems. You will lead and assure on AI-focused data concerns including (meta) data processing within AI-agent flows, tracking, auditing and assuring AI-related data quality, and recording data provenance across lifecycles that include multiple interactions that bring legacy data sets and AI-based components together.

You will lead the data management and data engineering services and analytical projects, working collaboratively with other teams to acquire and ingest new data sources. You will also work with the Principal Data Engineer, Data Architects, Database Administrators, Technical Architects, Solution Architects and Data Analytics team to set the enterprise data strategy and standards and agree implementation and governance.

Main Responsibilities

Your main day to day responsibilities will be:

• Managing data sources in a way to improve data quality and conform them for analysis and processing by various AI and Machine Learning tools and systems.
• Developing new data models and ETL processes, working collaboratively within the AI as a Serviceteam to map data to an appropriate matching data model.
• Exercising key stakeholder management skills to manage expectations, listen and interpret business needs, communicate progress and facilitate difficult conversations, while building a 'service culture' around an organisational shift towards AI-ready data.
• Managing the performance of suppliers and the data team to improve capability, as well as creating the roadmap which balances immediate user needs with long-term investment.
• Working with AI and Machine Learning specialists on your team to develop processes that balance data availability, capacity and data archiving, while continually improving processes and efficiency in the acquisition and ingestion of data.
• Line managing staff, including coaching and professional development.
• Using agreed standards and tools to design, code, test, correct and document moderate to complex programs and scripts from agreed specifications, and deploying tested code to infrastructure hosted over multiple cloud providers including AWS and Azure.
• Facilitating the evaluation and selection of software products, data models, and design patterns by keeping abreast of best practice in industry and across government, to enable and assure enterprise-scale AI-data integrations.

UK residency and security requirements       

For meaningful security checks to be carried out, individuals need to have lived in the UK for a sufficient period of time. Learn more on our website. Security Checks - Home Office Careers. 

Please note that this role requires Security Check (SC) clearance, which would normally need 5 years’ UK residency in the past 5 years.  

However, in exceptional circumstances security clearance applications for candidates who have been present in the UK for at least 3 of the last 5 years may be considered. Failure to meet this residency requirement will result in your security clearance application being rejected.