OnPace Documentation

Quick Start with Sample Data

The fastest way to explore OnPace is to load the built-in sample data. This generates realistic mock ticket data across multiple projects so you can see the full analysis experience without uploading anything.

Steps:

1. Open OnPace in your browser.

2. Click the "Data" button in the top navigation bar.

3. Switch to the "Load Data" tab and click "Load Sample Data".

4. The app will auto-configure with sample projects (ENG, ITSM, CUST) and a Rovo implementation date of July 1, 2025.

5. Click "Analyze" to run the analysis and see results immediately.

The left-hand configuration panel controls every aspect of your analysis. Here’s what each setting does and how to get the most out of it.

Project & Issue Type Selection

Select one or more projects from the dropdown. When multiple projects are selected, their data is combined into a single analysis. Use issue type filters to focus on specific work types (e.g., only Bugs, only Service Requests).

Rovo Implementation Date

This is the most important setting. It defines the dividing line between your pre-Rovo baseline period and the post-Rovo impact period. Set this to the date when your team started actively using Rovo AI agents.

Important

If the date is too recent, there won’t be enough post-Rovo data for meaningful analysis. Aim for at least 3 months of post-Rovo data for reliable results. The Rovo effect ramps over the first 3 months in the model.

Analysis Window

Choose how far back to look for baseline data: 12, 24, or 36 months. A longer window gives the regression model more data to work with, which typically produces a more accurate baseline. However, if your team’s processes changed significantly in the past (e.g., major reorganization 2 years ago), a shorter window may be more representative.

Baseline Trend Adjustments

For each metric, OnPace computes a trend line from your pre-Rovo data using linear regression. The trend slider lets you fine-tune this if you believe the computed trend is too aggressive or too conservative.

• Computed value: What the regression found (e.g., "-3.2%/year" means resolution times were already improving at 3.2% per year before Rovo)

• Slider range: ±10% adjustment around the computed value

• Use case: If you know there was a one-time improvement (like a process change) that inflated the pre-Rovo trend, pull the slider back to get a fairer baseline

Seasonality Toggle

When enabled (default), OnPace decomposes monthly seasonal patterns from your data. This accounts for predictable fluctuations like Q4 surges or summer slowdowns. Turn it off if your data doesn’t have seasonal patterns or if the analysis window is too short (less than 12 months) for reliable seasonal estimation.

Survey / Model Weight Slider

This slider controls how much weight agent survey ratings get versus the pure statistical model when attributing improvements to Rovo.

• All Model (left): 100% of the gap between baseline and actual is attributed to Rovo. Agent ratings are ignored.

• All Survey (right): Attribution is entirely driven by agent feedback. A rating of 7/10 means 67% attributed to Rovo.

• Balanced (middle, default): A weighted blend of both signals, adjusted for survey coverage confidence.

The blending formula accounts for survey coverage: if only 40% of tickets have ratings, the survey signal carries less weight than if 80% of tickets are rated. This prevents low-coverage surveys from dominating the attribution.

OnPace’s analysis pipeline has four stages: data aggregation, baseline modeling, impact measurement, and attribution blending.

Stage 1: Monthly Aggregation

Raw issue data is grouped by resolution month. For each month, OnPace computes:

• Average time to close (mean of resolutionHours)

• Ticket count (number of issues resolved)

• Average activities per ticket (mean of activities field)

• Average first response time (mean of firstResponseHours)

• Reopen rate (percentage of issues where wasReopened = true)

• Unique agent count (distinct assignees, used for team size tracking)

• Average Rovo rating and coverage (from rovoRating field, post-Rovo only)

Stage 2: Baseline Regression

Using only pre-Rovo monthly data, OnPace fits a linear regression model for each metric:

y(t) = α + β·t + Σ[γₖ · Sₖ(t)] + ε

Where α is the intercept, β is the trend slope, Sₖ(t) is the seasonal component for each calendar month, and ε is residual noise.

The model first fits a linear trend using Ordinary Least Squares (OLS), then computes seasonal factors as the average residual per calendar month. This two-step approach captures both the long-term direction and monthly cycles.

The baseline is then projected forward into the post-Rovo period, giving you what the metric would likely have been without Rovo, based on existing trends and seasonal patterns.

R² (Goodness of Fit)

Each metric’s baseline model includes an R² score indicating how well the trend + seasonality explains the pre-Rovo data. Higher R² means a more reliable baseline:

• R² > 0.7: Strong fit — the baseline projection is reliable

• R² 0.4–0.7: Moderate fit — results are directionally useful but have uncertainty

• R² < 0.4: Weak fit — high variance in pre-Rovo data; consider adjusting the analysis window or trend sliders

Stage 3: Impact Measurement

For each post-Rovo month, OnPace computes the gap between the projected baseline and the actual metric value. This gap represents the total improvement beyond what was expected from organic trends alone.

For the Tickets Closed metric specifically, OnPace also factors in team size changes. If your team grew 20% post-Rovo, the baseline is scaled up proportionally (using a configurable sensitivity weight) so that the throughput increase from hiring is separated from Rovo’s contribution.

Stage 4: Attribution Blending

The final step determines how much of the measured gap to credit to Rovo versus other factors. OnPace uses a blended model with two signals:

Signal 1: Statistical Model (default 100%)

The regression model attributes the entire gap between baseline and actual to Rovo by default. This is the maximum possible Rovo credit — it assumes every improvement beyond the projected trend is Rovo-driven.

Signal 2: Agent Survey Ratings

If your data includes rovoRating fields (1–10 scale), agents’ own assessment of Rovo’s helpfulness is incorporated. The survey attribution percentage is calculated as:

Survey % = (avgRating - 1) / 9 × 100

So a rating of 7/10 yields 66.7% Rovo attribution, and a rating of 10/10 yields 100%.

Blending Formula

The two signals are combined using a weighted average, with the survey weight adjusted for coverage confidence:

effectiveWeight = (sliderWeight / 100) × coverageConfidence

blended = model × (1 - effectiveWeight) + survey × effectiveWeight

Coverage confidence reflects what percentage of post-Rovo tickets have ratings. If only 40% of tickets are rated, the survey signal is down-weighted because it may not be representative. At 80%+ coverage, the survey has strong influence.

Understanding the Charts

Actual vs Projected Baseline Tab

The primary chart shows monthly metric values as a line chart with stacked bars representing the gap between the projected baseline (dashed line) and actual performance (solid line).

• Purple bars: The portion of the gap attributed to Rovo

• Gray bars: The portion attributed to Other Factors (based on survey blend)

• Teal bars (Tickets Closed only): The portion attributed to team size changes

• Vertical dashed line: Marks the Rovo implementation date

Rovo Impact Ratings Tab

Shows agent survey data over time: monthly average Rovo rating (1–10 scale) and survey coverage percentage. This helps you understand whether agents are finding Rovo increasingly helpful and whether enough people are providing feedback for reliable attribution.

Key Insights

Below the KPI summary cards, OnPace generates context-aware insights based on your analysis results:

• Metric improvements: Specific percentage changes with before/after values

• Survey summary: Average agent rating, coverage, and trend direction

• Attribution credit: How much of the improvement is credited to Rovo (when survey data adjusts the model)

• Team size changes: Flags significant headcount growth or reduction and explains how it affects attribution

• Data quality warnings: Alerts when R² values are low, suggesting you adjust analysis parameters