Site Reliability Engineer


San Francisco, CA, US
  • Job Type: Full-Time
  • Function: IT
  • Post Date: 01/08/2021
  • Website:
  • Company Address: 436 Bryant St., Floor 2, San Francisco, CA, 94107

About Theorem

Theorem is an investment manager in the marketplace lending space. Our team uses machine learning and data science to identify the best, safest loans in the $100 billion marketplace lending space. We use these insights to manage capital and advise investors on purchasing the best risk-adjusted loans and create stable portfolios across the business cycle.

Job Description

Theorem ( is a San Francisco based company at the intersection of technology and finance.
Our mission is to make credit safe and available.
An innovative investment fund with over $1BN+ assets under management, we combine quantitative research -- machine learning and data science, software engineering, and rigorous scientific investigation to build credit portfolios that produce strong and consistent yields across business cycles. 
We started our company 6 years ago with seed funding and an investment fund of $50,000, and are now profitable and thriving with 30+ employees.
Role Mission: Theorem's Site Reliability Engineer will be responsible for accelerating the delivery of working, reliable software into production through the systematic application of software automation to the firm’s IT operations.  
Level: Senior engineer with previous career experience and domain expertise in engineering operations directly focused on cloud infrastructure. 

What You'll Be Doing

      • Serve as the technical operations lead for the following systems:
      • Cloud configuration (AWS + IONOS / Terraform)
      • Complete a turndown of Theorem's IONOS datacenter presence.
      • Theorem's cloud environment to be completely codified in Terraform and reconfigurable via a CD pipeline.
      • Reduce privileges of technical employees to require creation of production infrastructure via Terraform.
      • Container orchestration (our Kubernetes clusters)
      • Create an access audit log for our production Kubernetes clusters.
      • Perform cluster software upgrades.
      • Provision a high performance Ceph storage cluster.
      • Continuous integration & deployment (currently Jenkins)
      • Build a continuous delivery pipeline for deploying changes to alerting infrastructure.
      • Explore alternative cloud-native CI/CD offerings to potentially replace Jenkins
      • Monitoring & alerting (Prometheus / Alertmanager / PagerDuty)
      • Redefine our alert-handling policies to significantly reduce the frequency of unhandled alerts.
      • Creation of a cost allocation dashboard that associates cloud costs to business programs and reveals how Theorem spends its money on compute resources.
      • Creation of an alert SLO dashboard that reveals how much time we spend dealing with operational toil and which systems are being operationally neglected.
    • Support and assist the following data infrastructure objectives:
    • Complete a database migration away from AWS Redshift to Parquet in S3.
    • Deploy a distributed dataflow engine (e.g. Apache Spark) for use in the data pipeline
    • Streamline the data pipeline deployment process.
    • Ingestion of systems operation data into the data warehouse.


    • Experience with and knowledge of public cloud offerings such as AWS (preferred), GCP, Azure, etc.
    • Previous experience with infrastructure automation tools such as Terraform (preferred), CloudFormation, Ansible, Chef, Puppet, etc.
    • Experience with various flavors of the Linux operating system, including shell scripting in bash or zsh.
    • Working knowledge of Python.
    • Previous experience working with container orchestration systems, such as Kubernetes (preferred), Borg, Twine, Apache Mesos, etc.
    • Has a deep appreciation for continuous integration and deployment (CI/CD).Is familiar with standard practices in software monitoring and alerting.
    • Has a working knowledge of SQL, data warehousing, and ETL pipelines.
    • Understanding of staging environments, integration and release testing.
    • Attention to detail
    • Hardworking and gritty
    • Sharp and fast learner
    • Transparency & intellectual honesty
    • Welcomes and adapts behavior to feedback
    • Collaborative and team-oriented
    • Ability to communicate and escalate issues

Training & Experience

    • Previous experience and subject matter expert in aspects of build engineering. Experience using modern build systems such as Bazel, Buck, Pants, Please, etc. is desired, but not a hard requirement. If they have no previous experience with any of those tools, they should signal enthusiasm for learning to use and extend Bazel.
    • Has a deep appreciation for continuous integration and deployment (CI/CD). All engineers contribute CI/CD pipelines, but our hire will help us organize and streamline our CI pipelines to make it simple for other engineers to contribute to.
    • Has previous experience managing containerized workloads. Ideally previous experience with Kubernetes, but not required. Our hire will eventually become a primary owner for our Kubernetes clusters.
    • Is familiar with standard practices in software monitoring and alerting. They will be expected to create software monitors and configure alert routing so that the appropriate engineers are notified of failures.
    • Understands how to use staging environments and integration testing to create system release tests. All engineers are expected to create release tests; our hire will help define standard testing environments that makes it easier for others to write release tests.
    • Bachelor's degree or higher in computer science, software engineering, or related discipline.
Founded in 2014 by a Morgan Stanley quant trader and a Google software engineer, we are backed by some great investors, including Y Combinator and Two Sigma Ventures
We are profitable, managing over $1BN, including investment from non-profits and university endowments
We are a small, passionate, and collaborative team -- fewer than 40 research, engineering, and finance and operations professionals with a diversity of backgrounds and ideas
Every member of the team has a major impact on the company's success with visible contributions to the business
We deeply value intellectual curiosity, creative idea generation, and close collaboration. 
We seek talented individuals who are thoughtful, innovative, tenacious, and interested in a mission-focused team environment
We share ideas informally at team lunches, academic paper discussions, and through our bi-weekly All-Hands meetings
Company events are inclusive and fun, with expeditions to food trucks and Michelin-starred local restaurants, and annual trips to Montana and Stinson Beach featuring hiking and cooking
We are located in SoMa in San Francisco, a vigorous 4-minute walk from BART and 20-minute walk from CalTrain
We offer competitive salary and opportunity for equity ownership, generous benefits, and an inclusive and collaborative work environment

Related Jobs

No Jobs found with this criteria
Disclaimer: Local Candidates Only
This company does NOT accept candidates from outside recruiting firms. Agency contacts are not welcome.