LLM-Powered Build Analytics

An automated, AI-driven approach to CI/CD failure triage and DevOps self-service – built on AWS Bedrock

Overview

A prominent international e-commerce enablement company set out to reduce the manual effort required to manage a large estate of Jenkins CI/CD pipelines. When build jobs failed, engineers had to manually gather logs, reconstruct pipeline context, and hunt through internal documentation before they could diagnose and remediate an issue – a slow, repetitive process that pulled skilled staff away from higher-value work.

TeraSky designed and delivered a generative-AI proof of concept that pairs an automated failure-analysis agent with a conversational assistant, both grounded in the organization’s own operational knowledge. The solution demonstrates how AWS Bedrock and a retrieval-augmented knowledge base can turn raw build telemetry and tribal DevOps knowledge into instant, context-aware guidance.

The Challenge

Managing Jenkins pipelines at scale had become an operational drag. The existing workflow depended heavily on manual intervention whenever a job failed:

• Logs, pipeline context, and troubleshooting data had to be retrieved and analyzed by hand.
• Engineers repeatedly re-solved similar failures because past resolutions were not easily discoverable.
• Institutional best practices are scattered across multiple documentation repositories, making them hard to apply in the moment.
• The cumulative delay and effort increased operational cost and slowed delivery.

The organization wanted to know whether a large language model, grounded in its own logs and documentation, could serve as the first line of triage and provide engineers a faster path to answers.

The Solution

TeraSky architected a Gen AI solution on AWS built around two complementary components, both drawing on a shared knowledge base of historical build logs and internal engineering documentation.

1. Automated failure-analysis agent

Whenever a Jenkins job fails, a webhook triggers the agent and passes along rich context about the failed build. The agent then:

• Pulls additional pipeline context, including the pipeline definition and related upstream and downstream jobs.
• Queries the knowledge base for previous, similar failures and relevant remediation guidance.
• Generates and sends a clear email containing a problem description, the surrounding pipeline context, and suggested remediation steps.

2. Conversational assistant in Microsoft Teams

A Microsoft Teams chatbot gives engineers a natural-language front door to Jenkins operations. Through simple conversation, users can:

• Request plain-language summaries of failed builds, enriched with context from the knowledge base.
• Ask about build statuses and retrieve relevant job details on demand.
• Get answers on building best practices and common troubleshooting procedures drawn from the organization’s own guidelines.

Grounded in a Bedrock knowledge base

At the heart of the solution is a retrieval-augmented knowledge base. Jenkins logs and internal documentation are collected into Amazon S3, parsed and split into semantically meaningful chunks, and converted into vector embeddings using foundation models on AWS Bedrock. The embeddings are stored in an OpenSearch vector collection, enabling fast semantic search so both the agent and the chatbot can surface the most relevant prior knowledge for any given query.

How It Works

The end-to-end flow ties Jenkins, AWS, and Microsoft Teams into a single intelligent loop:

• Jenkins streams build logs and documentation into an S3 bucket and emits failure events to an agent endpoint.
• The S3 content is vectorized into an OpenSearch-backed Bedrock knowledge base, keeping the AI grounded in current, organization-specific information.
• An AWS Lambda endpoint routes both user queries from the Teams bot and error events from Jenkins to the Bedrock-powered LLM agent.
• The agent reasons over the knowledge base to produce detailed, context-aware responses and recommendations.
• Follow-on Lambda tasks act on those responses — for example, sending remediation emails or performing Jenkins-related actions.

Cost and usage were kept transparent throughout, with Amazon CloudWatch dashboards and alerts tracking Bedrock and related service consumption. Response quality was assessed using an LLM-as-a-judge approach that compared generated answers against a reference Q&A set for accuracy and relevance.

The Outcome

The proof-of-concept demonstrated a working, end-to-end pattern for AI-assisted DevOps: an agent that automatically triages build failures and drafts remediation guidance, and a Teams assistant that puts pipeline knowledge at engineers’ fingertips. By grounding a large language model in the organization’s own logs and documentation, TeraSky showed how repetitive troubleshooting can be offloaded to an intelligent, self-service layer — freeing engineers to focus on building rather than firefighting.

Delivered entirely as Infrastructure as Code, the solution provides a consistent, reproducible foundation that the organization can build on as it continues to explore generative AI across its engineering operations.

Why TeraSky

TeraSky combines deep AWS expertise with hands-on experience delivering production-grade generative AI solutions. From architecture and knowledge-based design through agent development, conversational interfaces, and Infrastructure as Code, TeraSky helps organizations move from idea to working Gen AI capability — grounded in their own data and aligned to real operational needs.