Principal Data Engineer

Job Description — Principal Data Engineer (15+ Years) | Location - BangaloreRole SummaryWe are looking for a senior technical leader and hands-on architect — a Principal Data Engineer — to own the design and evolution of an AWS + Databricks enterprise lakehouse spanning ingestion (batch/streaming), transformation (canonical/Silver), and curated insights (Gold), with strong governance, security, performance, and cost controls. This role sets engineering standards, mentors teams, and drives scalable delivery across multiple domains and source systems.Key ResponsibilitiesArchitecture & Platform LeadershipOwn end-to-end lakehouse architecture across Bronze/Silver/Gold and ensure best practices for:Raw landing on S3 + Glue CatalogCanonical modeling and transformationsCurated datasets for analytics/AI/consumptionDefine standards for data layout, partitioning, file sizing, compaction, and Iceberg table management.Establish platform patterns for batch + streaming ingestion, orchestration, and automated deployment.Data Engineering Delivery (Hands-On + Leadership)Lead implementation of scalable pipelines in Databricks (PySpark/Spark SQL) and AWS (Glue, Lambda, Step Functions where needed).Design and maintain robust data models (canonical/Silver) and curated marts (Gold) optimized for analytics and downstream use.Ensure pipelines meet SLA, reliability, security, and cost objectives.Streaming and Near Real-Time EnablementDefine approach for streaming ingestion and processing (e.g., Kafka/Kinesis patterns if applicable, structured streaming/micro-batch).Ensure correctness, idempotency, late arriving data handling, and replay strategies.Governance, Security & ComplianceImplement security architecture: IAM, least privilege, encryption (KMS), secrets management, network controls.Integrate catalog/lineage/governance (e.g., Atlan) with standardized metadata practices.Establish data access patterns including RBAC/ABAC and controlled data sharing with partners.Performance, Reliability & FinOpsDrive optimization on Databricks clusters, job tuning, caching strategies, and query performance.Implement observability: pipeline metrics, logs, lineage, and incident runbooks.Own cost optimization strategy: autoscaling, cluster policies, workload isolation, storage optimization.Engineering Excellence & Team EnablementCreate reference architectures, coding standards, reusable libraries, and delivery playbooks.Mentor data engineers and reviewers; lead design reviews, code reviews, and production readiness checks.Collaborate with stakeholders across Analytics/BI, AI/ML, and product teams.Must-Have Skills & Experience15+ years in Data Engineering with proven leadership owning enterprise-scale platforms.Expert-level Databricks: Spark architecture, PySpark optimization, Spark SQL, workflows, job orchestration, cluster policies.Deep AWS expertise: S3, Glue, Lake Formation (if used), IAM, CloudWatch, KMS, VPC/security controls.Strong experience with Lakehouse table formats: Iceberg (preferred) / Delta / Hudi and parquet optimization.Strong architecture skills for data ingestion, canonical modeling, and curated layer design.Strong hands-on coding in Python and advanced SQL.Experience implementing CI/CD for data (Git branching, deployment automation, environment promotion).Experience designing for analytics consumption: semantic layer readiness, BI/Power BI integration patterns.Nice-to-HaveExperience with Unity Catalog, multi-workspace governance, data sharing, and fine-grained access controls.Exposure to data virtualization patterns (semantic/virtualization layer) and federation strategies.AI/ML enablement experience (feature datasets, training data pipelines, governance for GenAI/LLM apps).Experience integrating enterprise apps (ERP, ServiceNow, Workday, factory systems like MES).QualificationsBachelor’s/Master’s in CS/Engineering or equivalent.Strong stakeholder management and ability to drive decisions across teams.

Apply Now