Purpose of the role
This role provides technical support for the BI & analytics environment. This includes assisting with the design, development and maintenance of the Data Lake, Delta Lake, the Data Warehouses, reporting datasets and all data and analytical requirements (including dashboards and Data Science modelling) of the Inchcape Group.
The primary role will be to work with business users and analysts to design and develop effective technical solutions for our organisation’s data and information needs. Along with this the role is also responsible for the monitoring and maintenance of the analytics environment and will be required to support other specialists within the team.
Job role and responsibilities
- Design and build robust scalable data pipelines and data migration using the Azure Stack
- Responsible for creating reusable and scalable data pipelines
- Produce high-quality code to put solutions into production
- Execute high-performance data processing for structured and unstructured data, and data harmonization
- Schedule, orchestrate, and validate pipelines
- Design exception handling and log monitoring for debugging
- Work closely with the Architects to develop data pipelines on Azure Data Platforms
- Do data wrangling, manipulation, cleansing and blending from multiple data sources and exposure to various data types and storage paradigms, including structured and unstructured data.
- Prepare raw data from multiple sources and apply modelling techniques consistent with existing methodologies.
- Provision and configure effective Azure environments including resource creation and maintenance, platform security, and database management services.
- Conduct issue triage and investigation across the BI / Analytics stack.
- Develop ETL bug fixes using Azure Synapse Analytics, Azure Databricks, Azure Data Factory, Azure Data Lake v2 and Azure Purview.
Skills and Experience Required
- Demonstrated extensive experience working with Azure data solutions and supporting environments involving Azure Synapse Analytics, Azure Databricks, Azure Data Factory, Azure Data Lake v2 and Power BI and Power Apps.
- Experience with JIRA project management software is an advantage.
- Hands on experience in developing data pipelines using Azure Data Factory, Azure Logic Apps, Azure Functions, Azure Synapse, Azure Data Lake
- Design and develop the logic for end-to-end data pipeline including performance optimization
- ETL processing using Azure Databricks and Delta Lake including optimization
- Experience working with structured and unstructured data
- Design and develop dashboards using Power BI