Senior Data Engineer
Developing products that solve real problems for our customers is at the heart of everything we do. And we’re on the hunt for an exceptional Data Engineer to help shape Quest, our dynamic data platform that’s changing the game in commercial insurance.
Reporting to our Lead Data Scientist, the Data Engineer will enjoy working in a structured team environment. You will be inquisitive, and detail oriented, with strong experience building end-to-end ETL pipelines. You will enjoy working with large, messy datasets and transforming them into useable, structured, maintainable resources. On top of this, you will enjoy interrogating data to understand the best way to clean, merge, and derive maximum insight from the available data feeds.
This is an exciting and rewarding role requiring a smart, disciplined and experienced Data Engineer who is data savvy with a very good technical background in programming, the analysis of data and the design and creation of data pipelines.
You’re in your element getting hands on with messy data from multiple sources and transforming it into a useable resource. You’re motivated, inquisitive, and understand the potential of this data to the business. You enjoy a challenge and see the bigger picture of what is to be achieved.
You have extensive experience with scalable Data Pipelines and have created production grade code several times using range of approaches and tools.
You possess a good understanding of best practice software techniques, and have experience querying and processing real world data to derive insight and discuss results with the business.
- Carry out data assessment on new data from a variety of different sources
- Perform exploratory analysis to identify key data attributes and techniques for aggregating and matching data
- Design and implement data pipelines for receiving and transforming data
- Data and variable re-structuring, pre-processing and transformation across a wide range of data sources and Databases.
- Apply supervised and unsupervised algorithms to match, map, assign and cluster data
- From analysis and assessment of the data, develop features to determine behavioural risk which can be added to the Quest product and used in Machine Learning approaches
- Validation and monitoring of various aspects of Data Quality
- Working with Big Data tools and technologies (Spark, NiFi, Airflow) effectively manage the data ingest process into the Advanced Analytics Layer of Concirrus’ Big Data Platform.
- Strong Python Engineering skills
- Solid experience building ETL Data Pipelines
- Strong problem-solving skills
- Strong data wrangling and processing skills using Apache Spark, Hive/Pig or similar tools
- Degree in Computer Science or similar engineering degree
- Knowledge and experience of software engineering practices such as OOP, SOLID, TDD
- Proven ability to effectively communicate results.
- Very good interpersonal skills and ability to work closely with other members of the data science and engineering team.
- Python, and preferably exposure to other languages (SQL, Java)
- Apache Spark (preferably with PySpark) and Python Pandas, Numpy, SciKit
- Git version control
- Command line, Linux and bash scripting
Nice to have
- Experience of AWS and its ecosystem
- Experience of mapping tools such as geopandas or shapely
- Experience with Big Data would be a great asset!
- Experience with algorithm and model deployment (for example through AWS, Algorithmia or ML Flow)
- Track record of working with cluster computing and distributed systems.
- Experience with command-line scripting, data structures and algorithms.
- Ability to work in a Linux environment, processing large amounts of data in a cloud environment (AWS, Google Cloud Platform).
- Relational database experience
- Knowledge of system architecture best practices
As well as the opportunity to work on projects that you enjoy in an environment you’ll love, we like to look after our team members at Concirrus. Here are some of our perks:
- Employee share scheme so you get to own a piece of the pie
- We pay competitively with regular pay reviews
- 25 days holiday
- Pension scheme – generous matched contribution
- Learning and development fund to spend on courses or books
- Spacious, modern offices a stone’s throw from the culinary delights of Borough Market, and with a pub literally on our doorstep. The office is located on St Thomas Street, opposite The Shard
- Coffee and snacks are all on hand in the office to keep you fuelled
- Friday FED talks – like TED talks, but you get fed (on us)
- Regular team drinks, birthday cakes and social events
- Plus, we’re always on the lookout for creative ways to look after our employees and encourage them to come to us when they have an idea or needs