Posting Performance Evidence

The ABCo Scenario

Acme Build Corporation (ABCo) provides cloud-based code build services with an SLA guaranteeing build completion within specified time thresholds. Their challenge: creating verifiable proof of performance for both internal monitoring and customer transparency.

Step 1: Create the BuildPerformance Repo

Step 2: Create the Articles

First, ABCo defines what they're making assertions about - the BuildJob:

To define the BuildJob They'll need to add the following properties and then save the changes.

Properties

• jobName

  • Type: Text

  • Description: Name of the build job

  • Required: Yes

  • List: No

• version

  • Type: Text

  • Description: Unique version number for the build job

  • Required: Yes

  • List: No

Next, they'll create an article that represents the set of build jobs which they'll call the BuildService. The only property that BuildService needs is the $id.

Step 3: Creating the Source

Before creating Evidence, ABCo needs to define where their evidence will come from, by creating a unique source that collects the build duration information, called BuildObserver.

Properties

• systemName

  • Type: Text

  • Description: The source of data about the performance of each build job

  • Required: Yes

  • List: No

Step 4: Creating Build Execution Evidence

Each build completion generates evidence about its duration which they will capture by creating a new Evidence type called BuildExecution, which is about the Article Type BuildJob

Properties

• duration

  • Type: TimeSpan

  • Description: Elapsed time for the build

  • Required: Yes

  • List: No

At this point ABCo have completed enough setup to begin capturing build execution data.

Their repo can capture duration evidence about build jobs using a build observer source.

However, before they begin posting data they should think about how to measure performance and make further claims about the BuildService

Advanced Pattern: Computed Claims for Performance Metrics

The Power of Claims Computed on WarmHub

Computed claims transform raw evidence into actionable performance metrics. WarmHub's computed claims automatically calculate percentiles, averages, and other statistics from accumulated evidence.

Creating SLA Compliance Claims

ABCo can take advantage of WarmHub Computed Claims by creating an SLACompliance Claim type and makig sure to check the Compute on WarmHub checkbox.

They then select the Article Type about which they're making this claim - in this case the BuildSystem Article Type that they defined earlier.

Next they select the Claim Type of 'Summary Computed Claim' - which means this is a claim about a collection of previous assertions (Evidence or Claims)

Then they define a Constant $id - this is a unique name within this repo that will allow them to reference it by name in the future. This can be any text but it can also be computed using variable data. In this case they want to make quarterly claims about SLA performance so they'll use some date related variables to specify QQQ which is the quarter of the year that this claim is about suffixed by the yyyy or four digit year - so an example value might be SLA-Q3-2025

Next they need to Create a Collector - the collector is responsible for gathering the data upon which the claim is based. In this case they are selecting the duration property of the BuildExecution Evidence Type.

• Name:slaComplianceCollector

• Assertion Type: BuildExecution

• Count: 1000

• Require exact amount: No

• Date Offset: CurrentQuarter

• Property to Collect: duration

Next they will Create Properties

Properties

• totalBuilds

  • Type: Integer

  • Description: Total number of builds executed

  • Required: Yes

  • List: No

• complianceRate

  • Type: Percentage

  • Description: Percentage of the builds compliant with SLA

  • Required: Yes

  • List: No

And now for each of the defined properties, they will bind it to the collector data by defining the expressions that will calculate the properties.

In this case, connecting totalBuilds to the collector using the count expression

and connecting complianceRate to the percentage less than a timespan expression

Threshold: Constant

Value: 500

Unit: Seconds (s)

Lastly, they'll create the trigger for the computation to execute - in this case, any time a BuildExecution evidence is created, WarmHub will run the SLACompliance computation and revise the previous claim with the updated computed values.

That's it - ABCo has defined a computed claim that will be updated every time a new build is executed, so they'll have a running calculation of SLA performance throughout the quarter.

Creating Performance Claims

Now they have set up calculation of the SLA, ABCo may decide it would be helpful to get a breakdown by month of how their performance is.

Since WarmHub includes a number of operators that calculate statistical results from collected data, ABCo decided to calculate the P95 or 95th percentile of BuildExecution

Creating a computed performance claim

To create the BuildPerformance Claim, they just need to repeat the step from above and create a new Warmhub Computed Claim.

Define the collector on the BuildExecution duration

• Name:durationCollector

• Assertion Type: BuildExecution

• Count: 1000

• Require exact amount: No

• Date Offset: CurrentMonth

• Property to Collect: duration

Define some properties on the claim - p95Duration

• p95Duration

  • Type: TimeSpan

  • Description: 95th percentile duration for this BuildJob

  • Required: Yes

  • List: No

Bind it to the duration Collector using the p95TimeSpan operator

Operator: P95TimeSpan

Variable (Collector): durationCollector

Create a trigger based on BuildExecution, so every time a BuildExecution evidence is created WarmHub will rerun this to update performance metrics.

And here's what the completed computed claim looks like

At this point ABCo has not only defined the types to capture build performance but they've defined two automatically updating claims that will track both SLA compliance on a quarterly basis and build performance on a monthly basis.

Seeing the results

After running on every build they can see a snapshot of how the builds are performing and where they're at on SLA compliance

50 builds have been executed. The P95 duration is 2285 seconds but with an SLA threshold of 500 seconds only 22% of the builds are within the SLA boundaries