Position Details: Databricks Developer

Description:

• Work Location: **Hyderabad
• Title - Databricks Developer

Skill - ETL/ELT pipelines using Databricks and Apache Spark, PySpark ,
Advanced SQL Skillset, Automation & DevOps Integration,

• Analyst/Snr Analyst – Level 3
• A Pyspark and Databricks Developer with a good understanding of the entire
ETL/Azure lifecycle with a background of data projects.

• Work you’ll do
• *1.

Design and Build Data Pipelines

• - Develop robust, scalable ETL/ELT pipelines using Databricks and Apache
Spark.

• Integrate data from on-premises databases, cloud storage, APIs, and
third-party sources into the enterprise data lake.

• Automate both batch and real-time data ingestion to support enterprise
reporting and advanced analytics (with specific experience in SAP HANA
integration being highly valuable).

• Enhance pipeline designs by leveraging Azure Data Factory (ADF),
diverse connectors, and core cloud services to create seamless, end-to-end
data flows.

• 2.

Collaboration & Migration

• - Collaborate with onshore teams, architects, fellow developers, and
business analysts to align new data flows with overall enterprise data
warehouse (EDW) migration strategies.

• Support the migration of legacy and on-premises systems (especially
SAP HANA) to cloud-native services, including the use of Delta Lake on
Databricks.

• Optimize data modeling for analytical workloads, including the
development of star/snowflake schemas and dimensional models.

• 3.

Data Transformation & Preparation

• - Cleanse, transform, and join diverse datasets for analytics and
machine learning use cases.

• Use PySpark (mandatory) and, optionally, Scala within Databricks
notebooks and jobs for efficient data processing.

• Ensure high data quality through comprehensive validation, exception
handling, and logging practices.

• 4.

Performance Tuning & Cost Optimization

• - Optimize Spark jobs for efficiency, cost, and scalability (utilizing
partitioning, caching, and dynamic resource allocation).

• Monitor Databricks workloads and resolve bottlenecks affecting
compute, storage, or data skew.

• 5.

Data Governance & Security

• - Assist in enforcing security controls such as access management, data
masking, and encryption.

• Support enterprise-wide data governance policies, focusing on metadata
management, data lineage, and auditability.

• 6.

Advanced SQL Skillset

• - Build, optimize, and maintain SQL queries for ETL processes within
Databricks.

• Create and manage data models and tables with Databricks SQL or Spark
SQL.

• Write and automate high-performance SQL scripts for joining,
filtering, aggregating, and analyzing large datasets.

• Document SQL logic and data flows; convert and optimize SAP HANA SQL
code to Databricks-compatible SQL as part of migration.

• Collaborate with analysts and end-users to deliver accurate reporting
and dashboarding.

• 7.

Automation & DevOps Integration

• - Develop reusable components and orchestrate workflows; integrate
pipelines with DevOps and CI/CD practices.

• Utilize ADO (Azure DevOps) for version control and automated
deployment of data assets and Databricks notebooks.

• 8.

Documentation & Best Practices

• - Document all pipeline logic, ETL job definitions, and key system
processes with clarity and thoroughness.

• Champion coding standards, reusable code design, and ongoing knowledge
sharing within the team.

• 9.

SAFe Agile Teamwork

• - Work closely with business analysts, product owners, and data stewards
to understand evolving requirements and build jobs accordingly.

• Provide transparent updates and proactively escalate technical
challenges in alignment with SAFe Agile processes.

• The team
• Solutions Delivery-Canada is an integral part of the Information Technology
Services group.

The principle focus of this organization is the development
and maintenance of technology solutions that e-enable the delivery of
Function and Marketplace Services and Management Information Systems.

Solutions Delivery Canada develops and maintains solutions built on varied
technologies like Siebel, PeopleSoft Microsoft technologies and Lotus
Notes.

Solutions Delivery Canada has various groups which provide the best
of the breed solutions to the clients by following a streamlined system
development methodology.

Solutions Delivery Canada comprises of groups like
Usability, Application Architecture, Development and Quality Assurance and
Performance.

• Qualifications
•

Required:

• Computer Science University degree or equivalent work experience
• At least 1.5-2.5 years for Aanalyst and 4-5 years of total exp
• Experience in Databricks, Pyspark and ADF etc.

• Experience in Databricks end-to-end implementation.

• Expert in developing solutions and delivering the high-quality
projects using Pyspark and Databricks

• Proficiency with Azure Data Factory, Azure DevOps, and cloud-native
data services.

• Experience integrating and migrating data from SAP HANA to cloud
platforms (Good to have)

• Advanced SQL skills for data modeling and migration.

• Experience implementing data security and governance policies.

• Familiarity with DevOps, CI/CD, and infrastructure-as-code concepts
• Expert on data integration tools, such as Data Stage is good to have
• Knowledge of basic Python programming - Preferable
• Knowledge of Cloud based ETL services
• Excellent organizational and communication skills
• Must have strong interpersonal skills, and the ability to effectively
work with others in teams

• Strong problem-solving skills in identifying issues and resolution
• Adeptness at learning new technology and tools
• The ability to effectively manage multiple assignments and
responsibilities in a fast-paced environment

• Strong commitment to professional client service excellence
• Should be a strong individual contributor

Perform an action: