Job Description: Databricks + DataStage Engineer Role Summary Hands-on engineer responsible for migrating IBM DataStage workloads to Databricks (SparkSQL/PySpark), supporting framework-driven ETL modernization, validation, and performance tuning. Key Responsibilities Analyze existing DataStage server and parallel jobs Convert ETL logic to SparkSQL/PySpark on Databricks Implement metadata-driven ingestion patterns Support framework-based development and reusable templates Perform unit testing, integration testing, and reconciliation Optimize jobs using Delta Lake, partitioning, Z-ordering Support CI/CD pipelines (Git-based) Collaborate with architects and DataStage SMEs Required Skills Strong experience in Databricks (AWS/Azure), SparkSQL, PySpark Hands-on experience with IBM DataStage (server jobs mandatory; parallel preferred) Strong SQL skills (Teradata/Oracle experience preferred) Experience with Delta Lake and performance optimization Familiarity with CI/CD (GitLab, Azure DevOps, etc.) Good understanding of ETL/ELT patterns Good to Have Exposure to AI-assisted code conversion. Experience with metadata-driven frameworks Knowledge of orchestration tools (Airflow/Autosys/Control-M) Additional Information: SPOC - Kishore Mandatory Skills - under JD Nice to have skills - under JD Interview Mode - Virtual Interview Work Model - Hybrid Model Clean Room Project - No
Job Title
Databricks DataStage Engineer