Senior Data Engineer – Azure Databricks Migration & Optimization

We’re migrating complex SQL Server logic (stored procedures, views, SSIS packages) from legacy systems straight into Azure Databricks + ADF — lift-and-shift style , no full redesign. The real work starts after migration: aggressively optimize for performance and cost, understand Spark physical plans inside-out, avoid hidden cross-joins or skew surprises, master Delta Lake retention pitfalls, and make smart compute choices that balance speed and DBUs. If you love diving deep into Spark execution models, reading physical plans to fix subtle regressions, knowing exactly what VACUUM + OPTIMIZE does to time travel, and delivering production-grade cost savings without breaking business logic — this is your role. Key Responsibilities Migrate complex SQL Server stored procs/views/SSIS to Databricks notebooks (PySpark / Spark SQL) with zero functional breakage Analyze & optimize Spark physical plans — eliminate unintended cross-joins, skew, shuffles, inefficient join strategies Apply Delta Lake optimizations (OPTIMIZE, Z-ORDER, file sizing) while fully understanding vacuum/retention/time-travel trade-offs Implement medallion architecture during migration (bronze → silver → gold) for staging, cleansing & BI-ready layers Right-size compute (clusters vs SQL Warehouses vs Serverless) with strong cost-performance awareness Troubleshoot serverless compute jobs — access logs, diagnose long runs Orchestrate via ADF, handle incremental loads, ensure production reliability Document migration decisions, mappings, validations & optimization rationale Must-Have Depth Proven SQL Server → Databricks migration experience (complex stored procs → Spark SQL/PySpark) Deep Spark internals (stages/tasks, default join behavior, AQE, physical plan reading) Hands-on Delta Lake mastery (time travel pitfalls, VACUUM + OPTIMIZE interplay, retention config) Practical Unity Catalog usage (managed vs external table limitations) ADF + Databricks integration for production pipelines Strong cost-awareness — able to justify changes with measurable DBU/performance impact Nice-to-Have Databricks Certified Data Engineer Associate (or Professional) Experience with serverless compute troubleshooting & SQL Warehouse types (Classic/Pro/Serverless) 100% Remote - Global team, flexible in US hours If this sounds like your kind of challenge, Apply quick note on your favorite Spark optimization story. Serious applicants only — we move fast.

Apply Now