Lead Data Engineer Job at NAZZTEC, Cary, NC

  • NAZZTEC
  • Cary, NC

Job Description

READ CAREFULLY BEFORE APPLYING | URGENT HIRING

ROLE: Lead Data Engineer – Hands-On

Client: MetLife

Location: Cary, NC — On-site / Hybrid

Experience: 12–18 Years

Employment Type: Full-Time

Eligibility: US CITIZENS ONLY

ABOUT THE ENGAGEMENT

The engagement involves building a centralized, AI-first enterprise Data Hub for a global insurance and financial services organization using Azure Databricks .

The platform ingests 150+ inbound data feeds and distributes data to 35+ downstream systems , leveraging a Medallion Architecture — Bronze / Silver / Gold .

AI is embedded across ingestion, canonical mapping, data quality, reconciliation, and business-user access.

This is a senior hands-on technical leadership role . The selected candidate will own the end-to-end technical design of the data and AI layers , build reference implementations, mentor engineering teams, and continue writing/reviewing production-grade Python, Scala, and PySpark code every week .

⚠️ Candidates who have NOT written or reviewed production code within the past year will NOT be considered.

WHAT YOU WILL OWN

Data Platform Architecture & Engineering

  • Design and implement enterprise-scale Lakehouse architecture.
  • Define Bronze / Silver / Gold data contracts.
  • Design ADLS Gen2 zone layouts.
  • Delta Lake table design, partitioning, schema evolution and retention.
  • Build scalable and reusable data platform components.

Metadata-Driven Ingestion

  • Design parameterized ingestion frameworks for:
  • Batch files
  • Database extracts
  • CDC feeds
  • Streaming data
  • Implement streaming pipelines using Azure Event Hubs / Kafka / Spark Structured Streaming .
  • Build schema inference, data profiling, metadata and lineage capabilities.

Spark & Data Engineering

  • Develop production-grade Python, Scala and PySpark solutions.
  • Establish coding and testing standards.
  • Conduct PR/code reviews.
  • Debug production incidents.
  • Optimize Spark workloads, clusters and pipelines.
  • Implement performance and cost optimization strategies.

Databricks & Azure

  • Azure Databricks
  • Delta Lake
  • Unity Catalog
  • Databricks Jobs & Workflows
  • Cluster and pool management
  • Model Serving
  • ADLS Gen2
  • Azure Data Factory
  • Azure Event Hubs
  • Azure Monitor
  • Log Analytics

CI/CD & Infrastructure as Code

  • Implement CI/CD for Databricks and ADF using Azure DevOps .
  • Work with Databricks Asset Bundles .
  • Implement Infrastructure as Code using Terraform .
  • Automate data pipeline testing and deployments.
  • Establish observability and operational monitoring.

AI / LLM ENGINEERING

A major component of this role is designing and delivering production-grade AI/LLM solutions .

You will work on:

  • AI-augmented ingestion.
  • AI-assisted canonical mapping.
  • Auto-generated bridge documentation.
  • DML and canonical table generation.
  • Source-to-canonical mapping with human review gates.
  • AI-driven data quality.
  • Data drift and schema drift detection.
  • Volume anomaly detection.
  • Automated reconciliation.
  • Synthetic privacy-preserving test data generation.
  • Enterprise semantic layers.
  • Knowledge graphs.
  • GPT-powered conversational interfaces.
  • Text-to-SQL.
  • Semantic-layer retrieval.
  • Row-level and column-level security.

LLM / GENAI MUST-HAVES

Candidates must have 3+ years of hands-on experience designing and shipping LLM-based systems in production , including:

  • RAG pipelines
  • Agentic AI workflows
  • Tool-calling workflows
  • Chunking strategies
  • Embedding strategies
  • Vector retrieval
  • Hybrid retrieval
  • Prompt engineering
  • LLM evaluation
  • Golden datasets
  • Regression test suites
  • Accuracy tracking
  • Hallucination monitoring
  • Human-in-the-loop feedback

Hands-on experience with at least one of:

  • LangChain
  • LlamaIndex
  • LangGraph

Provider/platform experience:

  • Azure OpenAI
  • OpenAI
  • Databricks Model Serving

SECURITY, GOVERNANCE & DATA PROTECTION

Strong enterprise security and governance experience is required across:

  • Unity Catalog
  • Data lineage
  • Access control
  • PII standards
  • Entra ID
  • Managed Identities
  • RBAC
  • POSIX ACLs
  • Azure Key Vault
  • Private Endpoints
  • Data privacy and protection
  • Enterprise data governance

The candidate will also participate in:

  • Architecture Review Boards
  • AI Governance Forums
  • Technical design reviews
  • Enterprise governance discussions

MUST-HAVE TECHNICAL SKILLS

✅ 12–18 years of total experience in Data Engineering / Data Platform delivery

✅ Expert-level Python

✅ Expert-level Scala

✅ Expert-level PySpark / Spark

✅ Strong SQL

✅ Data modelling — dimensional & normalized

✅ Data contracts & schema design

✅ Azure Databricks

✅ Delta Lake

✅ Unity Catalog

✅ Azure ADLS Gen2

✅ Azure Data Factory

✅ Azure Event Hubs

✅ Kafka / Spark Structured Streaming

✅ Databricks Jobs & Workflows

✅ Spark performance tuning

✅ Azure DevOps

✅ Terraform

✅ Databricks Asset Bundles

✅ CI/CD

✅ Metadata-driven frameworks

✅ Data profiling & lineage

✅ Enterprise Lakehouse / Medallion Architecture

✅ Azure security & governance

✅ 3+ years production LLM/GenAI engineering

✅ RAG / Agentic AI / Tool Calling

✅ LangChain / LlamaIndex / LangGraph

✅ Azure OpenAI / OpenAI / Databricks Model Serving

⭐ STRONGLY PREFERRED

  • Knowledge graphs & ontologies
  • RDF / SPARQL
  • Neo4j
  • Graph modelling over Lakehouse
  • Text-to-SQL
  • Enterprise semantic layers
  • ML-based anomaly detection
  • Time-series / transactional financial data
  • Financial Services / Insurance domain
  • Finance Close
  • GL / General Ledger
  • Subledger
  • Reconciliation
  • Actuarial data
  • LLMOps / MLOps
  • Model & prompt versioning
  • LLM cost governance
  • AI observability
  • dbt
  • Great Expectations
  • Workday
  • Workday Prism
  • Accounting Center

CERTIFICATIONS — PLUS

  • Databricks Data Engineer Professional
  • Azure DP-203
  • Azure DP-700
  • Azure AZ-305

IDEAL CANDIDATE PROFILE

We are looking for a hands-on Lead Data Engineer / Data Architect / Principal Data Engineer who can operate at both architecture and engineering levels .

You should be comfortable designing the architecture, making technical decisions, writing production code, reviewing code, troubleshooting Spark workloads, mentoring engineers, and presenting architecture decisions to senior technical and business stakeholders.

⚠️ This is NOT a pure Data Architect, Solution Architect, Engineering Manager, or AI Strategy role.

⚠️ Recent hands-on production coding experience is mandatory.

⚠️ US CITIZENS ONLY.

#Nazztec #NazztecHiring #JoinNazztec #Hiring #UrgentHiring #LeadDataEngineer #DataEngineer #SeniorDataEngineer #PrincipalDataEngineer #DataArchitect #DataEngineering #DataPlatform #DataPlatformEngineer #BigData #BigDataEngineering #Azure #MicrosoftAzure #AzureDatabricks #Databricks #DeltaLake #UnityCatalog #AzureDataFactory #ADF #ADLS #ADLSGen2 #AzureEventHubs #Kafka #Spark #ApacheSpark #PySpark #Scala #Python #SQL #DataLake #Lakehouse #LakehouseArchitecture #MedallionArchitecture #BronzeSilverGold #DataArchitecture #DataModeling #DataEngineeringJobs #AzureJobs #DatabricksJobs #AI #ArtificialIntelligence #GenerativeAI #GenAI #LLM #LLMEngineering #LLMEngineer #LLMJobs #RAG #RAGPipeline #RetrievalAugmentedGeneration #AgenticAI #AgenticWorkflows #ToolCalling #LangChain #LlamaIndex #LangGraph #AzureOpenAI #OpenAI #DatabricksModelServing #PromptEngineering #VectorDatabase #VectorSearch #HybridSearch #Embeddings #SemanticSearch #TextToSQL #SemanticLayer #KnowledgeGraph #Neo4j #RDF #SPARQL #MLOps #LLMOps #AIEngineering #AIEngineer #MachineLearning #AnomalyDetection #DataQuality #DataGovernance #DataSecurity #DataLineage #DataCatalog #Metadata #MetadataDriven #DataContracts #SchemaEvolution #DataProfiling #DataObservability #DataOps #DevOps #AzureDevOps #Terraform #InfrastructureAsCode #IaC #CICD #ContinuousIntegration #ContinuousDeployment #SparkTuning #PerformanceTuning #CloudDataEngineering #CloudArchitecture #EnterpriseData #EnterpriseArchitecture #FinancialServices #Insurance #InsuranceTechnology #FinTech #MetLife #CaryNC #NorthCarolinaJobs #USJobs #USAJobs #USCitizen #USCitizenJobs #TechJobs #ITJobs #TechnologyJobs #DataScience #CloudJobs #DataProfessionals #TechHiring #ITHiring #DataHiring #AIJobs #RemoteJobs #HybridJobs #OnsiteJobs

Job Tags

Full time

Similar Jobs

Lutron Electronics

Lead Data Engineer Job at Lutron Electronics

OverviewLutron is seeking a Lead Data Engineer in Coopersburg, PA or Philadelphia, PA to help build and scale a modern, cloud-native data and AI platform powering analytics, real-time intelligence, and next-generation products. In this role, you will shape the architecture... 

Insight Global

Clinical Data Analyst Job at Insight Global

 ...job and more exclusive features.Direct message the job poster from Insight GlobalInsight Global is seeking a fully remote Healthcare Informatics Analyst to join the data exchange team of a large healthcare company. This person is responsible for creating and managing... 

Willis Knighton Health

Prior Authorization Specialist Job at Willis Knighton Health

 ...PharmacyShreveport, LANewly Created PositionFull-Time, DayThe Willis Knighton Department of Pharmacy is seeking a Pharmacy Prior Authorization Specialist to join our team. This individual will work directly with our Outpatient Infusion Units under the supervision of... 

Publicis Groupe

SEO Manager Job at Publicis Groupe

 ...all the Groupes tools and expertise around the world through the mantra "The Power of One."OverviewAs Search Engine Optimization Manager, youll analyze search data to provide tremendous insights into the motivations of consumers, leading a team thats on the cutting... 

Expedia

Mobile Engineer III, iOS Job at Expedia

At Expedia Group, we help travelers explore the world, one journey at a time. As a global travel company powered by passionate people, trusted partnerships, and leading technology, we connect travelers, partners, and advertisers through our consumer brands, B2B network,...