Data Engineer

Zachary Piper Solutions is seeking a highly skilled and experienced, security cleared, Data Engineer to join a mission critical program supporting the Defense Intelligence Agency . In this role, you'll work with a team of developers to help build and maintain ETL pipelines, along with interfacing with external systems using HTML and SFTP protocols. This role is Hybrid out of Annapolis Junction, MD, and requires an active TS/SCI with a CI Polygraph .

Responsibilities of the Data Engineer:

• Architect, develop, and maintain scalable ETL pipelines using Databricks, Apache Spark, Python, and SQL.
• Integrate and transform data from diverse sources to comply with IC ITE data standards.
• Collaborate with external systems and teams using protocols such as HTML and SFTP.
• Enhance the ETL infrastructure to support scalability and accelerate integration timelines.
• Partner with developers, data scientists, and stakeholders to ensure high data quality and usability.
• Create and maintain documentation for system architecture, development workflows, and platform improvements.

Qualifications for the Machine Learning Engineer:

• Active TS/SCI clearance with CI Polygraph.
• Proven experience using Databricks to build and manage data and AI solutions.
• Strong experience in ETL development and managing data pipelines for large-scale datasets.
• Proficiency in Apache Spark, Python, and SQL.
• Familiarity with Databricks features including Delta Lake, Delta Live Tables, and Workflows.

Preferred Qualifications (Nice to Haves):

• Experience working alongside data scientists on analytical and modeling projects.
• Knowledge of Advana or similar data platforms.
• Advanced proficiency in Python and SQL.
• Experience with cloud environments such as AWS or Azure.

Compensation and Benefits:

• $150,000 - $200,000 *Dependent on experience*
• Comprehensive Benefits: Medical, Dental, Vision, 401K, PTO, Sick Leave as required by law, and Holidays.

#LI-HYBRID

#LI-EA1

Data Engineer, ETL pipelines, Databricks, Apache Spark, Python, SQL, Delta Lake, Delta Live Tables, Databricks Workflows, IC ITE standards, HTML, SFTP, data integration, data transformation, scalable infrastructure, data quality, system architecture, documentation, TS/SCI clearance, CI Polygraph, Defense Intelligence Agency, DIA, hybrid role, Annapolis Junction, Maryland, cleared job, national security, big data, data analytics, data science collaboration, Advana, data platforms, cloud computing, AWS, Azure, machine learning, AI solutions, data pipeline management, large datasets, government contract, intelligence community, secure data, classified data, ETL development, workflow automation, data usability, technical documentation, Spark SQL, distributed computing, security cleared, mission critical, federal data systems.