Senior Site Reliability Engineer - Observability (FedRAMP)
Splunk
Software Engineering
Indiana, USA · Remote
Posted on Oct 31, 2024
Splunk is here to build a safer and more resilient digital world. The world's leading enterprises use our unified security and observability platform to keep their digital systems secure and reliable. While customers love our technology, it's our people that make Splunk stand out as an amazing career destination and why we've won so many awards as a best place to work. If you become a Splunker, we want your whole, authentic self, what we call your "million data points". So bring your work experience, problem-solving skills and talent, of course, but also bring your joy, your passion and all the things that make you, you.
Join us as we pursue our ground-breaking vision to make machine data accessible, usable, and valuable to everyone. We are a company filled with people who are passionate about our product and seek to deliver the best experience for our customers. At Splunk, we are committed to our work, customers, having fun, and most significantly to each other’s success.
The Splunk Observability Cloud provides full-fidelity monitoring and fixing across infrastructure, applications, and user interfaces, in real-time and at any scale, to help our customers keep their services reliable, innovate faster, and deliver great customer experiences. Infrastructure Software Engineers at Splunk are cloud-native systems engineers who use infrastructure-as-code, microservices, automation, and efficient design to build, operate, and scale our products.
Role Summary
You will help us run one of the largest and most sophisticated cloud-scale, bigdata, and microservices platforms in the world. You will be responsible for enabling developers to operate highly available, scalable, and cost-efficient applications with low operational burden by handling and improving the reliability and resiliency of SRE-managed services and infrastructure. You thrive on automation, infrastructure-as-code, reliability engineering, and getting rid of tedious, manual tasks.
Meet the Products and Technology Team
Want to build security and observability products people love AND work with people as smart (and humble) as you are? Our products and technology team delivers digital resilience at enterprise scale with a self-service Splunk portfolio that offers unified security analytics, full stack observability and real-time visibility of streaming data. Learn more about the team, meet our leaders, and hear from Splunk technologists and engineers at splunk.com/careers/products-and-technology.
You will:
- Design new services, tools, and monitoring to be implemented by the entire team.
- Analyze the tradeoffs of the proposed design and make recommendations based on these tradeoffs.
- Mentor new engineers to achieve more than they thought possible. You enjoy making other teams successful and are fulfilled through the success of others.
Work on reliability projects, including:
- HA, Business Continuity Planning, disaster recovery, backup/restore, RTO, RPO
- Chaos engineering
- Application uptime and performance
- Capacity management & planning
- SLIs, SLOs, error budgets, and monitoring dashboards
- Responsible for deployment and operations of large-scale distributed data stores and streaming services
- Establishing design patterns for monitoring and benchmarking
- Establishing and documenting production run books and guidelines for developers
- Tooling, toil reduction, runbooks & automation to handle production environments
- Incident management and improving MTTD/MTTR for services
- Cloud cost optimization
Must-have Qualifications
- 7+ years of SRE experience in handling large-scale cloud-native microservices platforms.
- 3+ years of strong hands-on experience deploying, handling, and monitoring large-scale Kubernetes clusters in the public cloud specifically AWS or GCP
- Experience with infrastructure automation and scripting using Python and/or bash scripting.
- Strong hands-on experience in monitoring tools such as Splunk, Prometheus, Grafana, ELK stack, etc. in order to build observability for large-scale microservices deployments.
- Experience with deployment, operations, and performance management of one or more of the following large-scale clusters such as Cassandra, Kafka, Elastic Search, MongoDB, ZooKeeper, Redis, etc.
- Excellent problem-solving, triaging, and debugging skills in large-scale distributed systems
- Candidate must be a US citizen or a permanent resident and must reside on US soil
Nice-to-have Qualifications
- AWS Solutions Architect certification preferred.
- Confluent Certified Administrator for Apache Kafka and/or Apache Cassandra Administrator Associate certifications are preferred
- Experience with Infrastructure-as-Code using Terraform, CloudFormation, Google Deployment Manager, Pulumi, Packer, ARM, etc.
- Experience with CI/CD frameworks and Pipeline-as-Code such as Jenkins, Spinnaker, Gitlab, Argo, Artifactory, etc.
- Proven skills to effectively work across teams and functions to influence the design, operations, and deployment of highly available software.
- Experience handling cloud infrastructure and operations in strict security, compliance, and regulatory environments such as FedRAMP.
Bachelors/Masters in Computer Science, Engineering, or related technical field, or equivalent practical experience.
Splunk is an Equal Opportunity Employer
At Splunk, we believe creating a culture of belonging isn’t just the right thing to do; it’s also the smart thing. We prioritize diversity, equity, inclusion, and belonging to ensure our employees are supported to bring their best, most authentic selves to work where they can thrive. Qualified applicants receive consideration for employment without regard to race, religion, color, national origin, ancestry, sex, gender, gender identity, gender expression, sexual orientation, marital status, age, physical or mental disability or medical condition, genetic information, veteran status, or any other consideration made unlawful by federal, state, or local laws. We consider qualified applicants with criminal histories, consistent with legal requirements.