Senior Data Engineer

A financial institution is  looking to recruit a Senior Data Engineer to be responsible for the development and optismisation of the collection and data flow of the organisation. 

You will be expected to build relationships and collaborate with stakeholders at all levels across business, operations and technology teams. 

Duties and responsibilities

• Building of data sets and driving business decisions through the use of analytical and statistical approach 

• Designing, developing, testing and supporting data collection and integration with batched ETL and Real-time data processing

• Writing scalable code for managing Batch and Real-time data processing, using onsite/cloud tools and storage

• Improving existing batch based ETLs

• Helping in maintaining the existing infrastructure and its migration

• Maintaining records and documentation

• Building analytical tools that utilise the data pipeline

• Have a high interest in keeping up to date with latest best practices in the industry

Skills and Experience

• Experience in designing and implementing data streaming solutions involving technologies such as Kafka, NiFi and Spark Streaming

• Strong analytical and troubleshooting skills working in a Linux environment

• Experience with Linux based scripting tools such as shell and bash

• Experience with programming languages like Java and Python

• Good knowledge of cloud technologies and their tools in relation to data transformation

• Good knowledge of PostgreSQL streaming replication and RHEL based Linux 

• Familiarity with the use of Talend and Nifi 

• Familiarity with data structures such as JSON, XML and CSV

• Familiarity with data architecture and SQL querying on Oracle, MS SQL, PostgreSQL and HIVE

• Excellent verbal and written communication skills in English