The Data Engineer supports the design, implementation and maintenance of data flow channels and data processing systems that support the collection, storage, batch and real-time processing, and analysis of information in a scalable, repeatable and secure manner. The Data Engineer focuses on defining optimal solutions to data collection, processing and warehousing. He / She designs, codes and tests data systems and works on implementing those into the internal infrastructure.
Be Part of Our Success
- Work with stakeholders including customers, partners and colleagues on data-related technical issues and support their data infrastructure needs;
- Work closely with data scientists to solicit data requirements to support modeling works
- Design, develop, document, manage and maintain data models, ETL processes, data warehouse, data management and pipeline solutions for large volume of structured/unstructured data from disparate sources and with different latencies (e.g. on-demand, batch, real-time, near-real-time);
- Define, monitor and report SLAs for data pipelines and data products;
- Understand data security and governance standards or requirements to implement solutions that ensure adherence to these standards or meet such requirements;
- Drive/execute data quality assurance practices; and
- Support data management solutions pre-sales initiatives, proposal development and provide post-sales support.
Qualities We Value
- In-depth technical knowledge in:
- Data Modelling;
- Data Pipelines;
- OLAP;
- Data Ingestion & Integration techonlogies
- Query Optimisation
- Technical expertise in:
- Java, C/C++, Python, Scala, SQL etc
- Big data technologies e.g. Hadoop, Spark, Hive, HBase etc
- Experience in master data management, data governance, data lifecycle management etc.
- Experience in designing, documenting, implementing and supporting data management solutions
- Experience in using software engineering best practices in development, programming, testing, version control etc.
- Knowledge of data privacy and security assurance