Data Engineer

Role Overview We are looking for a hands-on, senior Databricks Architect to design, build, and govern our Lakehouse data platform from the ground up. You will own the end-to-end architecture of our data infrastructure — from raw ingestion through the Medallion layers to serving — and establish the engineering standards that will guide the entire data organization. This is a highly strategic and technical role focused on driving adoption of Databricks, Unity Catalog, and modern Lakehouse patterns across all data products and pipelines. Key Responsibilities Lakehouse Architecture & Design - Design and implement a production-grade Medallion Architecture (Bronze / Silver / Gold) across all data pipelines. - Establish best practices for Delta Lake table design, partitioning strategies, Z-ordering, and optimization across large-scale datasets. - Define data modeling standards and schema evolution policies across the Lakehouse. - Architect end-to-end data flows from ingestion (streaming and batch) through transformation and serving layers. Unity Catalog & Data Governance - Lead the setup, configuration, and rollout of Unity Catalog as the centralized governance layer for all data assets. - Design metastore hierarchy, catalog/schema/table organization, and tagging standards. - Implement fine-grained access control (row-level, column-level), data masking policies, and audit logging. - Establish data lineage tracking and ensure end-to-end visibility across all pipelines. - Define and enforce data classification and sensitivity frameworks for PII and regulated data assets. Pipeline Development & Orchestration - Build and maintain production-grade data pipelines using PySpark, Delta Live Tables (DLT), and Databricks Workflows / Jobs. - Design modular, reusable pipeline patterns including incremental ingestion, CDC (Change Data Capture), and full-refresh strategies. - Implement robust pipeline observability: logging, alerting, lineage tracking, and SLA monitoring. - Leverage Databricks Repos for CI/CD integration, managing code promotion across dev / staging / production environments. Performance & Compute Optimization - Optimize Spark execution plans, identify and resolve performance bottlenecks across large-scale distributed workloads. - Right-size cluster configurations: Serverless warehouses, auto-scaling job clusters, and photon-enabled SQL warehouses. - Leverage Serverless Warehouses and SQL Warehouses for BI and ad hoc analytics workloads, minimizing cost and cold-start latency. - Manage cost governance for compute, storage, and DBU consumption across workspaces. Developer Experience & Standards - Set up and maintain Databricks Repos with standardized project structures and Git integration. - Define Python coding standards, notebook best practices, and modular library patterns for the data engineering team. - Build reusable Python utility libraries for common patterns: schema validation, data quality checks, Delta operations, and logging. - Establish unit testing and integration testing frameworks for Spark pipelines. Security, Compliance & Networking - Configure workspace-level and account-level security: Private Link, IP access lists, secrets management via Databricks Secrets or AWS Secrets Manager. - Design and enforce network isolation for sensitive data workloads. - Ensure compliance with data residency and access control requirements for customer data. Collaboration & Enablement - Partner with data engineers, data scientists, and analytics engineers to ensure the platform meets diverse workload needs. - Mentor the engineering team on Databricks, Spark optimization, and Lakehouse best practices. - Produce architectural documentation, runbooks, and internal knowledge bases. - Evaluate and recommend new Databricks features and third-party integrations relevant to the organization's data roadmap. Required Qualifications Core Databricks & Lakehouse - 5+ years of hands-on experience with Databricks, with at least 2 years in an architect or senior lead role. - Deep expertise in Unity Catalog: metastore setup, three-level namespace, ACL design, and data governance workflows. - Strong mastery of the Medallion Architecture and Delta Lake: ACID transactions, time travel, compaction, and OPTIMIZE/VACUUM strategies. - Proven experience designing and deploying production pipelines with Databricks Jobs and Workflows, including multi-task job DAGs, retry logic, and notifications. - Hands-on experience with Databricks Repos and CI/CD integration for notebook and Python library deployments. - Experience configuring and operating Serverless SQL Warehouses and Serverless compute for Jobs. Apache Spark - Expert-level PySpark development: DataFrames, Spark SQL, window functions, broadcast joins, and UDFs. - Strong understanding of Spark internals: DAG execution, shuffle optimization, memory management, and speculative execution. - Experience with structured streaming and micro-batch processing patterns. - Proven ability to diagnose and resolve Spark performance issues using Spark UI and event logs. Python & Software Engineering - Advanced Python skills with a strong software engineering background: packaging, testing (pytest), virtual environments, and dependency management. - Experience building modular Python libraries for data engineering use cases. - Familiarity with common data engineering libraries: pandas, pydantic, great_expectations or similar DQ frameworks. Cloud & Infrastructure - Experience deploying Databricks on AWS, including workspace provisioning, IAM integration, and VPC configuration. - Familiarity with cloud-native storage (S3/ADLS), external locations in Unity Catalog, and storage credentials management. - Exposure to infrastructure-as-code tooling (Terraform, Databricks Asset Bundles, or similar). Preferred Qualifications - Databricks Certified Data Engineer Professional or Databricks Certified Associate Developer for Apache Spark certifications. - Experience with Delta Live Tables (DLT) for declarative pipeline authoring. - Familiarity with dbt (data build tool) integrated with Databricks SQL. - Experience with Databricks Feature Store or MLflow for ML platform use cases. - Exposure to Databricks Marketplace and Partner Connect integrations. - Experience with Elasticsearch, Apache Kafka, or other streaming/search technologies complementary to the Lakehouse.

Apply Now