Automating Jenkins Failure Diagnosis at Scale with Amazon Bedrock

TeraSky built Broadway Gaming, an agentic system that automatically diagnoses failed Jenkins builds. A LangGraph agent reasoning on Claude Sonnet via Amazon Bedrock reads each failed build’s log, searches a knowledge base of past failures and internal runbooks, and sends the responsible engineer the root cause, severity, and remediation path. A Microsoft Teams assistant answers follow-up questions against the same knowledge base.

Problem Statement

Broadway Gaming runs more than 30 bingo, slots, and casino brands alongside its Dragonfish white-label platform, supported by a CI pipeline that runs a significant number of builds per day. Roughly 25-30% of them fail to build a day and must be addressed by the engineering organization.

Many of those were diagnosed by hand. An engineer opened the console output, read through a log with thousands of lines to find the failing stage, and either recognized the cause or searched earlier builds and internal knowledge for something similar. The work was repetitive and varied depending on who picked up the build, so two engineers facing the same failure did not reliably arrive at the same explanation. New engineers depended on senior colleagues to interpret failures because the knowledge of what a given error actually meant was not written down anywhere they could find it.

The trend made it worse rather than steady. AI-assisted code generation is increasing commit and build volumes across the industry, and Broadway Gaming is seeing this firsthand. Failure counts rise with build counts, while manual triage is bounded by engineer headcount and cannot scale with them. Adding engineers adds builds.

Proposed Solution & Architecture

A Jenkins plugin on Broadway Gaming’s controller uploads each failed build’s log and its metadata to Amazon S3, authenticating through the host’s own instance role. The upload triggers an AWS Lambda function that runs a LangGraph agent that reasons with Anthropic’s Claude Sonnet via Amazon Bedrock.

The agent is not a fixed pipeline. It decides for itself how many times to search the knowledge base and what to search for, judges whether the results are sufficient, classifies severity, and writes the root cause, the failure stage, and a remediation path. Retrieval runs against an Amazon Bedrock Knowledge Base over an Amazon OpenSearch vector collection, embedded with Amazon Titan Text Embeddings V2, indexing both past build logs and Broadway Gaming’s own runbooks and best-practice documentation. The finished analysis is delivered by Amazon SNS to the engineers who own the pipeline.

A second surface runs off the same knowledge base. An assistant in Microsoft Teams answers questions about a failure interactively, served by a separate Lambda function, so a newer engineer can investigate a failure without first going to a senior colleague.

The whole system runs in Broadway Gaming’s own AWS account, with all three functions inside their VPC and no route to the internet, reaching every AWS service through VPC endpoints.

Outcomes & Success Metrics

Every failed build now receives a written analysis, in the same structure, whether or not an engineer had time to look at it. Previously, a build was written up only if someone picked it up, and nothing recorded how many went undiagnosed.

The pipeline analyzed failed builds, delivering each analysis in an average of 170 seconds. The average log it reads is 25K.

From an originally estimated 140 minutes of effort per build failure, the estimate was reduced to 45 minutes, resulting in a reduction of 106 minutes across the tested scenarios.