Databricks Solution Architect

Description

We are seeking an experienced Databricks Architect Expert with a strong data engineering background to support the design and delivery of secure, monitored, cloud-hosted environments. The ideal candidate will possess hands-on experience with data science and analytics, extensive use of AI/ML and LLM models, AI/MLOps, DevSecOps, and GenAI tools and technologies, with expertise in deploying solutions to high-stakes production environments.

The successful candidate will join our platform team, which provides capabilities, tools, guardrails, and various patterns for our customers within a federal agency. Our goal is to support customers as they bring their applications/models into our environment, facilitating the training, building, tweaking, and productionalization of their data and models.

Key Responsibilities:

• Design and implement event-driven architecture to support scalable, resilient, and dynamic GenAI applications.
• Refactor and rehost existing applications onto cloud platforms, optimizing for performance and scalability.
• Collaborate on the development of Platform Architecture, ensuring seamless integration and efficient operation of cloud services.
• Develop and optimize Databricks Solution Architectures to handle large-scale data processing and analytics workloads.
• Engineer robust Bedrock Solution Architectures to support infrastructure provisioning and management at scale.
• Define and execute Deployment Architectures that allow for automated and reliable deployments of applications and services.
• Establish and enforce Security Architectures to protect data and maintain compliance with industry and regulatory standards.
• Create Use Case Solution Architectures, transforming business requirements into technical solutions.
• Proactively identify and address technical issues and challenges to improve the efficiency and reliability of cloud-based systems.
• Work with cross-functional teams to facilitate the smooth deployment of cloud services across various operational environments.
• Document architectural decisions and create best practice guidelines for cloud development and deployment.
• Stay abreast of emerging cloud technologies and recommend adaptations to keep ahead of industry trends.
• Utilize Agile and DevSecOps methodologies and practices.
• Coach and provide direction to junior engineers in the Databricks and data engineering areas.
• Lead efforts to integrate various data sources, ensuring seamless data flow and transformation. Develop strategies for data ingestion, cleansing, normalization, and integration.
• Leverage experience with MLFlow and general CI/CD principles for automating model deployment.
• Collaborate with customers and critical resources in an SME program to come up with solutions that align with long-term roadmaps and strategy.
• Execute Databricks upgrades, particularly in the federal space.

Qualifications

Qualifications:

• Bachelor's and 18 years or master's and 16 years of experience with a degree in computer science, Data Science, Engineering, or a related field.
• Must be a U.S. Citizen with the ability to obtain a public trust clearance.
• Proven experience in cloud engineering and architecture, with a focus on event-driven solutions.
• In-depth knowledge of public cloud platforms (e.g., AWS, Azure, Google Cloud, Oracle).
• Proficiency with Databricks and understanding of big data ecosystems.
• Experience with Infrastructure as Code (IaC) using tools like Terraform, AWS CloudFormation, etc.
• Strong understanding of deployment pipelines and automated testing frameworks.
• Demonstrated experience in implementing security practices in cloud architectures.
• Clear understanding of microservices architecture and containerization technologies (e.g., Docker, Kubernetes).
• Excellent verbal and written communication skills, with the ability to document and explain complex architectures to non-technical stakeholders.
• Databricks Certification and any other AI/ML or data engineering certifications (AWS, Azure, GCP).

Desired Skills:

• Previous experience working in a regulated environment, particularly within government agencies like the IRS.
• Certifications related to AWS or cloud architecture.
• Experience with AI/ML platforms and understanding of their integration with cloud environments.
• Prior experience with event-driven architectures and related technologies (e.g., Apache Kafka, RabbitMQ).
• Prior experience in a high-stakes production environment where uptime is critical.

Target salary range: $160,001 - $200,000. The estimate displayed represents the typical salary range for this position based on experience and other factors.