Databricks Data Engineer

Databricks Data Engineer

• Remote, US
• Information Technology

Build and operate the ingestion, bronze, silver, and gold pipelines on Databricks, migrating logic and data out of legacy. 

Responsibilities

• Build and maintain Databricks pipelines (60% of the time)
• Gather and analyze requirements (20% of the time)
• Monitor and troubleshoot data jobs (10% of the time)
• Document processes and workflows (10% of the time)
• Perform additional assigned tasks

Requirements

• US Citizenship
• Secret Clearance
• Excellent interpersonal skills and ability in overseeing/prioritizing work requests
• Excellent written and oral communication skills
• Strong data engineering, ETL/ELT development, or data warehousing
• Strong PySpark and/or Spark SQL skills; solid SQL fundamentals
• Solid understanding of the Medallion Architecture (Bronze/Silver/Gold layering)
• Proficiency with Delta Lake (ACID transactions, schema evolution, time travel, MERGE/Upsert patterns)
• Experience designing both batch and incremental/streaming ingestion pipelines
• Familiarity with the Agile process

Desired Skills

• Databricks Certification/experience/familiarity preferred
• Knowledge of and experience with the DoD
• Familiar with Customer and/or DoD IT architectures
• Federal experience 
• Experience in post go-live production support and break/fix
• Focus on customer service and responsiveness, with sound customer handling skills
• Experience working within AWS Gov Cloud
• Experience working migrating code from non-accredited to accredited environments within AWS

About Us
For more than 20 years, NewGen Technologies has solved our clients’ toughest IT challenges with integrity, security, and outstanding service by delivering both technology and talent. We have helped secure borders, have used artificial intelligence (AI) to fight terror, aided the identification of criminals, and have helped to prevent crime through the introduction of biometrics. Our team of Highly Cleared Specialists have hard-to-find skills and expertise in a wide spectrum of technologies to provide solutions that transform business processes and solve problems of national significance. #CJ