- Design, develop, and implement Spark Scala applications and data processing pipelines to process large volumes of structured and unstructured data
- Integrate Elasticsearch with Spark to enable efficient indexing, querying, and retrieval of data
- Optimize and tune Spark jobs for performance and scalability, ensuring efficient data processing and indexing in Elasticsearch
- Collaborate with data engineers, data scientists, and other stakeholders to understand requirements and translate them into technical specifications and solutions
- Implement data transformations, aggregations, and computations using Spark RDDs, DataFrames, and Datasets, and integrate them with Elasticsearch
- Develop and maintain scalable and fault-tolerant Spark applications, adhering to industry best practices and coding standards
- Troubleshoot and resolve issues related to data processing, performance, and data quality in the Spark-Elasticsearch integration
- Monitor and analyze job performance metrics, identify bottlenecks, and propose optimizations in both Spark and Elasticsearch components
- Stay updated with emerging trends and advancements in the big data technologies space to ensure continuous improvement and innovation Requirements
- Bachelor\'s or Master\'s degree in Computer Science, Software Engineering, or a related field
- Min 5+ Years experience in developing Spark & Scala applications. Experience with Spark Streaming is a plus.
- Proficiency in Scala programming language and familiarity with functional programming concepts
- In-depth understanding of Apache Spark architecture, RDDs, Data Frames, and Spark SQL
- Experience integrating and working with Elasticsearch for data indexing and search applications.
- Solid understanding of Elasticsearch data modeling, indexing strategies, and query optimization
- Experience with distributed computing, parallel processing, and working with large datasets
- Familiarity with big data technologies such as Hadoop, Hive, and HDFS
- Proficient in performance tuning and optimization techniques for Spark applications and Elasticsearch queries
- Strong problem-solving and analytical skills with the ability to debug and resolve complex issues.
- Familiarity with version control systems (e.g., Git) and collaborative development workflows -
Additional Information
Career Level
Not Specified
Qualification
Not Specified
Job Specializations
,
Company Overview
Headquartered in Singapore, Unison Consulting was founded in September 2012 with the core motive to provide cutting-edge solutions and services in a variety of areas. At Unison Consulting, our mission is to empower businesses with our financial and insurance services.
Our consulting services help our clients implement risk management, risk analytics system, systems integration, business intelligence, information management, and custom development of IT solutions. We surpass all standards of quality for financial, insurance institutions and other verticals.
Our team views our clients as our partners and derives its success through the success of our clients. We are highly focused on exceeding our clients\' expectations and offering them the best possible solutions for their businesses.
Unison Consulting has a team of young, energetic, imaginative, and innovative individuals who never hold back from facing their challenges head-on. They push all boundaries to help our clients and exceed their expectations. We help our team to upscale their knowledge as required in the market, thereby to increase their personal and professional capabilities.
Additional Company Information
Registration No.
C
Industry
Computer / Information Technology (Software)
Be careful
This job ad has not been subjected to our hirer verification process. Proceed cautiously and do your own checks before providing any personal information. Search Index -
Data Engineer, Data Engineering
6 days ago
Bank of Singapore SingaporeAt Bank of Singapore, we are constantly on the lookout for exceptional individuals to join our team. We promote a culture of openness, teamwork and fairness. Most importantly, we invest in our people through our programmes that develop them on both professional and personal level ...
-
Data Engineer
1 week ago
CXC Global SingaporeThis is a permanent role for a leading global technology organization with a strong APAC presence. Due to evolving business, the organization is undergoing an exciting transformation journey and is aiming to become a data-driven decision-making organization. This is an outstandin ...
-
Data Engineer
1 week ago
Singapore Land Authority Singapore**What the role is** · - The data engineer will work with stakeholders to execute the SLA Data Strategy · (3 Pillars: People, Process, Technology). · Job Scope: · - Plan and implement technology and policies to enhance organisational integration and responsiveness in data-driven ...
-
Data Engineer
1 day ago
NodeFlair Singapore**Job Summary**: · **Salary** · S$4,500 - S$5,800 / Monthly · **Job Type** · **Seniority** · Junior · **Years of Experience** · At least 2 years · **Tech Stacks** · ETL Informatica Analytics Talend MSSQL SQL PostgreSQL MySQL Python · **What is a Data Engineer?** · The role of a d ...
-
Data Engineer
1 week ago
Good Job Creations Pte Ltd SingaporeGrow and develop data infrastructure · Design and build efficient and resilient data pipelines for batch and streaming data · Manage data engineering solutions, through integration of data from a variety of data sources and delivering to downstream systems · Provide support to da ...
-
Data Engineer
1 week ago
Progression Search Singapore**CLIENT SUMMARY** · Our client is an innovative start-up specializing in ESG data collection & reporting. They offer a modular & customizable ESG Platform for the Financial Sector, Supply Chains, Corporate Sustainability and Investment Funds. · We are looking for a skilled Data ...
-
Data Engineer
1 week ago
SMRT Corporation Ltd SingaporeData Engineer · **Job Purpose** · - We are looking for an experienced data engineer to join our team. You will use various methods to transform raw data into useful data systems. For example, you will create and maintain optimal data pipelines architecture in cloud. Overall, you ...
-
Data Engineer
3 days ago
ASTEK SINGAPORE INNOVATION TECHNOLOGY PTE. LTD. SingaporeAstek is proposing an opportunity for **Data Engineer (Lexical Search) **to support our project based in **Singapore. · **Responsibilities** · - Acquire datasets that align with business needs · - Develop algorithms to transform data into useful, actionable information · - Build, ...
-
Data Engineer
1 day ago
ScienTec Consulting Singapore**Data Engineer** · **Job Brief** · We are looking for Data Engineer to join our team of analytics experts. The hire will be responsible for expanding and optimising data and data pipeline architecture, as well as optimising data flow and collect for cross-function teams. Data En ...
-
Data Engineer
1 day ago
NodeFlair Singapore**Job Summary**: · **Salary** · S$4,000 - S$5,500 / Monthly · **Job Type** · **Seniority** · Junior · **Years of Experience** · At least 1 year · **Tech Stacks** · ETL Informatica Analytics SSIS Talend MSSQL SQL PostgreSQL MySQL Python · **What is a Data Engineer?** · The role of ...
-
Data Engineer
3 days ago
H2 GAMES PTE. LTD. SingaporeWe are looking for data engineer who has great passion in the gaming industry, the responsibilities including but not limited to design, executing, and managing the data pipelines between source systems and Enterprise Data Platform. · **Job Responsiblities**: · - Responsible for ...
-
Data Engineer
1 week ago
Unison Consulting Pte Ltd SingaporeAs a Data Engineer, you will be responsible for designing, developing, and maintaining data pipelines, databases, and data warehouses to support the organization's data needs. Your expertise in Spark, Big Data technologies, Data Structures & Algorithms, Python, and SQL will be cr ...
-
Data Engineer
9 hours ago
NodeFlair Singapore**Job Summary**: · **Salary** · S$8,332 - S$16,664 / Monthly · **Job Type** · **Seniority** · Mid · **Years of Experience** · At least 5 years · **Tech Stacks** · ETL Snowflake play Realm Spring BigQuery Java Airflow Spark Flink Springboot Kotlin SQL Scala Hadoop Python · **The R ...
-
Data Engineer
3 days ago
1-FINITY CONSULTING PTE. LTD. SingaporeDevOps Batch engineer: · - Develop and maintain batch processing pipelines using Airflow · - Work with data engineers to design and implement data processing solutions · - Troubleshoot and debug batch processing pipelines · - Automate data processing tasks · - Monitor and report ...
-
Data Engineer
1 week ago
SEDHA CONSULTING PTE. LTD. Singapore**Reporting and Analytics Developer/Data Engineer**: · **Responsibilities** · - Analyse the company's data needs and document the requirements. · - Refine data collection/consumption by migrating data collection to more efficient channels. · - Plan, design and implement data engi ...
-
Data Engineer
1 week ago
Good Job Creations Pte Ltd SingaporeDevelop and support enterprise data services, and ensure operational stability and performance through continuous optimization. · - Work hand-in-glove the product owners and various stakeholders to deliver features, using best-of-breed technologies and robust engineering methodol ...
-
Data Engineer
4 days ago
Volt SingaporeLocation: · - Singapore- Job Type: · - Permanent- Salary: · - S$6000 - S$8000 per month- Reference: · - BBBH10722_ Contact: · - Darren Ou- **Data Engineer/Architect** · Company Overview: · Our client is globally established Multinational Company with a significant footprint withi ...
-
Data Engineer
1 week ago
ONYX SG PTE. LTD. Singapore**JOB DESCRIPTION** · Group Data team of a global asset manager with AUM of USD$35 Bn, with a diverse portfolio of alternative investments, currently seeking for a seasoned Data Engineer to join their expanding team. As a Data Engineer, you will play a crucial role in shaping the ...
-
Data Engineer
1 week ago
REGASK PTE. LTD. SingaporeWe, at RegASK are a high growth global RegTech company leveraging technology and a worldwide network of experts to revolutionize regulatory intelligence, focused on serving the LifeSciences (MedTech, CROs, Pharma) and FMCG industries (Food, Personal Care). The RegASK team is high ...
-
Data Engineer
2 days ago
Gravitas Recruitment Group SingaporeJob details · - Location · - Singapore · - Salary · - S$7500 - S$9500 per month · - Job Type · - Contract · - Ref · - BBBH145393_ · - Working Environment · - Hybrid · - Posted · - about 4 hours ago · Job summary · **Data Engineer - Finance Industry*** · Join our dynamic team as ...
Data Engineer(spark,scala And Elastic Search) - Singapore - search index
Description
We are seeking a talented and experienced Spark Scala Developer with expertise in Elasticsearch to join our team. As a Spark Scala Developer, you will play a crucial role in designing, developing, and optimizing big data solutions using Apache Spark, Scala, and Elasticsearch. You will collaborate with cross-functional teams to build scalable and efficient data processing pipelines and search applications. Knowledge and experience in the Compliance / AML domain will be a plus.
Responsibilities: