Data Engineer

Duration- 6 Months ContractDescription:What you'll do:Design and build the DKB entity schema on Delta Lake (entity tables governed by Unity Catalog; edge tables deferred to Phase 2 Graph Extension)Build ingestion connectors for Unity Catalog metadata, GitHub, Confluence, and dbt -- each producing a standard changeset with idempotent upsertsImplement graph traversal queries using Databricks SQL recursive CTEs on Serverless SQL WarehouseEnrich entity metadata from Unity Catalog Lineage API (table-to-table, column-to-column runtime lineage as entity properties)Build the Graph Repository abstraction layer (Python interface over Databricks SQL)Set up dev/staging/prod environments via UC catalog isolation, Terraform (IaC) and Databricks Asset Bundles (deployment)Own data quality: Pydantic validation at ingestion boundary, dead letter queues, graph integrity tests, connector integration tests with golden datasetsMust-have skills:4+ years with Databricks (Delta Lake, Unity Catalog, SQL Warehouse, Jobs Compute)Strong SQL including recursive CTEs, window functions, complex joinsPython for data engineering (Pydantic, pytest, API clients)Experience building data ingestion pipelines from REST APIs (GitHub, Confluence, Jira style)Comfortable with IaC (Terraform) and CI/CD for Databricks (Databricks Asset Bundles)Understanding of graph data modeling (entity-relationship, property graphs) -- does not need to be a graph DB expertNice-to-have:Experience with Unity Catalog Lineage API or External LineageExperience with dbt (manifest.json, lineage metadata)Familiarity with Mosaic AI Vector SearchPrior work with knowledge graphs or metadata platformsAbout US Tech Solutions:US Tech Solutions is a global staff augmentation firm providing a wide range of talent on-demand and total workforce solutions. To know more about US Tech Solutions, please visit .US Tech Solutions is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, colour, religion, sex, sexual orientation, gender identity, national origin, disability, or status as a protected veteranRecruiter DetailsName- Tarun KumarEmail Internal ID-26-06066

Apply Now