Data Engineer - Singapore - SAKSOFT PTE LIMITED

SAKSOFT PTE LIMITED

Verified Company

Singapore

1 week ago

Posted by:

Wei Jie

beBee Recruiter

Experience: 1 to 9 Years

Role:
Data Engineer

Key Responsibilities:

Integrate Elasticsearch with Spark to enable efficient indexing, querying, and retrieval of data
Optimize and tune Spark jobs for performance and scalability, ensuring efficient data processing and indexing in Elasticsearch
Collaborate with data engineers, data scientists, and other stakeholders to understand requirements and translate them into technical specifications and solutions
Implement data transformations, aggregations, and computations using Spark RDDs, DataFrames, and Datasets, and integrate them with Elasticsearch
Troubleshoot and resolve issues related to data processing, performance, and data quality in the Spark-Elasticsearch integration
Monitor and analyze job performance metrics, identify bottlenecks, and propose optimizations in both Spark and Elasticsearch components
Stay updated with emerging trends and advancements in the big data technologies space to ensure continuous improvement and innovation.

Key Requirements:

Bachelor's or Master's degree in Computer Science, Software Engineering, or a related field
Proficiency in Scala programming language and familiarity with functional programming concepts
Indepth understanding of Apache Spark architecture, RDDs, DataFrames, and Spark SQL
Solid understanding of Elasticsearch data modeling, indexing strategies, and query optimization
Experience with distributed computing, parallel processing, and working with large datasets
Familiarity with big data technologies such as Hadoop, Hive, and HDFS
Strong problemsolving and analytical skills with the ability to debug and resolve complex issues
Familiarity with version control systems (e.g., Git) and collaborative development workflows
Excellent communication and teamwork skills with the ability to work effectively in crossfunctional teams