Splunk Infrastructure Engineer

Splunk Infrastructure Engineer

Position Overview:

We are seeking an experience Splunk Infrastructure Engineer to support a large-scale federal customer environment processing approximately 2.5TB of data per day. This role will focus on maintaining, optimizing and modernizing a complex Splunk deployment that spans multiple domain and supports critical security operations.

The ideal candidate will possess deep expertise in Splunk Enterprise architecture, distributed deployments, Search Head Clustering, indexer administration, configuration management, and infrastructure modernization efforts. While the immediate focus is operational support and stability, this position will play a key role in the long-term transformation of the Splunk environment toward a more scalable and cloud-ready architecture.

Responsibilities:

• Maintain and optimize a large-scale VMware-hosted Splunk deployment.
• Administer Search Head Clusters supporting Enterprise Security and ad-hoc search environments.
• Manage Splunk indexers, monitoring consoles, license servers, and supporting infrastructure components.
• Support complex data routing requirements across multiple security domains, including relay and IRDA tiers.
• Utilize btool to identify, troubleshoot, and remediate configuration inconsistencies across the environment.
• Manage and improve configuration control processes through Git repositories and version management practices.
• Support the ongoing implementation and integration of Cribl data pipeline technologies.
• Assist with migration efforts to transition Splunk infrastructure from legacy domains into modernized environments.
• Monitor and maintain SmartStore integrations utilizing Pure Storage S3 backends.
• Optimize performance across multi-homed indexer configurations.
• Perform system health monitoring, troubleshooting, capacity planning, and performance tuning.
• Develop and maintain technical documentation, architecture diagrams, and operational procedures.
• Collaborate with customer stakeholders to support modernization initiatives and future-state architecture planning.

Required Qualifications:

• Active Top Secret clearance
• U.S. Citizenship required
• Splunk Core Consultant Certification (Required)
• Experience supporting large-scale Splunk environments processing terabytes of daily ingest
• Strong expertise with:
  • Distributed Splunk architectures
  • Search Head Clustering
  • Indexer administration and management
  • Splunk Enterprise Security (ES)
  • Splunk configuration management and troubleshooting
• Advanced experience utilizing btool for configuration auditing and issue resolution
• Experience supporting VMware-based Splunk deployments
• Experience with Git-based configuration management and version control

Preferred Qualifications:

• Splunk Enterprise Security Certification
• Experience with Cribl deployment and migration activities
• Experience supporting SmartStore environments
• Experience integrating Splunk with S3-compatible storage solutions
• Experience supporting cloud migration or infrastructure modernization initiatives
• Experience supporting multi-domain federal environments

Technical Environment:

• Splunk Enterprise
• Splunk Enterprise Security (ES)
• Search Head Clustering
• Distributed Indexer Clusters
• VMware
• Cribl
• Git
• SmartStore
• Pure Storage S3
• Multi-Domain Architectures
• Monitoring Console
• License Server
• Linux Administration