Statistical Analysis with AI: A Worked Business Guide

01Define the question and what one row means

State the outcome, units, groups and observation period. Decide whether rows represent independent observations (different people, each measured once), repeated measurements or individual transactions. Choose the comparison and the smallest difference that would matter before inspecting the result.

Here, the outcome is an operator’s average handling time in minutes. Each operator receives equal weight, whether they handled 40 tickets or 50. Pooling all tickets would answer a different question and give busier operators more influence.

For real data, decide which wider workforce or period the sample should represent. A file containing every operator that week describes that week. A statistical test does not automatically justify extending its findings to other teams or future weeks. The table below matches each method in this guide to the question it answers.

For the comparison, define the difference as pilot minus current. Use a two-sided test, which counts a difference in either direction, and a 95% confidence interval. Use 0.05 as the illustrative significance threshold, the cut-off for the p-value explained in step 4. Suppose one minute is the smallest difference worth investigating operationally; that is a hypothetical business threshold, not a statistical rule.

The groups are observational (operators were not assigned to them as an experiment) and contain different operators. Their difference does not establish a workflow’s causal effect. Ticket difficulty, experience and staffing could also differ. This example’s artificial data show how the calculations work if the methods’ assumptions hold.

02Prepare the file and record which rows count

Inspect the imported columns. Remove exact repeated records, keep missing outcomes separate from zero and save the inclusion rule. Check the remaining row count and group sizes before running a test.

Upload the raw CSV or workbook to your Anomaly project. When the import finishes, open the imported table and inspect workflow, tenure_months and avg_handle_minutes. Confirm that handling time is numeric minutes. A blank cell means the value is missing, not that the operator completed work in zero minutes.

Open the imported table before asking for statistics

Then ask Anomaly to audit the rows with this prompt:

Audit this table before any analysis. How many rows does it have, how many exact repeated records (identical in all six columns), how many rows have a missing avg_handle_minutes, and how many usable operators are there per workflow? State the inclusion rule you use for the analysis.

The table below shows the row counts at each stage of the audit.

Check which rows count

Two people with the same handling time are still two observations. Conversely, the same person measured next month is a repeated observation, even when every cell differs. Removing duplicates cannot solve dependence between measurements, such as repeated measurements of the same person.

Keep these rules with the result so a colleague can reproduce the selection. For a larger cleanup job, use the CSV analysis workflow. When repeated records disagree, the guide to removing duplicates by key explains the separate decision about which record to retain.

03Describe the groups and plot the values

Calculate the count, mean, median and sample standard deviation for each group. Keep the number of missing values visible. Inspect the distribution for unusual values, and independently reproduce the summary statistics.

The mean is the arithmetic average. The median is the middle of the sorted values. Standard deviation describes spread around the mean; a larger value means more variation. The pilot has a lower average and greater spread in this file. Neither fact tells us whether it caused a change.

Ask for a box plot (a chart of each group’s middle, spread and extreme values) or a plot of individual values with workflow and handling time in minutes labeled. Show both groups on the same numeric scale. A mean-only bar chart hides the spread and unusual observations you need to examine. For research data, retain the sample size and units in every graph.

Inspect the distributions

Check the descriptive statistics in Excel or Google Sheets

Open the cleaned check workbook. On its Current sheet, the 48 handling times occupy E2:E49. In empty cells, enter =COUNT(E2:E49), =AVERAGE(E2:E49), =MEDIAN(E2:E49) and =STDEV.S(E2:E49). On Pilot, use E2:E73.

Excel’s STDEV.S uses the sample formula, which divides by n−1, where n is the number of included values. In Google Sheets, STDEV gives that sample standard deviation. These formulas check the selected numbers; they cannot tell whether you selected the right people. See the Excel analysis guide or Google Sheets analysis guide for the surrounding spreadsheet workflow.

04Compare the means with a specified test

For two independent groups with a numeric outcome, request a two-sided Welch t-test. Report the direction of subtraction, mean difference, 95% confidence interval, t statistic, degrees of freedom and p-value together; each is explained after the results.

Welch’s test allows the groups to have different spreads and sample sizes. It still needs independent observations and a sound basis for modelling how much the result would vary from sample to sample. A two-sided test considers differences in either direction. A strongly lopsided spread of values (skew) or single values that sway the result (influential values) deserve investigation. A paired test suits a different design, such as the same operators measured before and after a change.

Then ask for the test with this prompt:

Run a Welch two-sided independent-samples t-test on avg_handle_minutes, pilot minus current, on the same usable rows. Report the mean difference, 95% confidence interval, t statistic, degrees of freedom and p-value. Explain the result in plain language, including whether the difference is larger than a practical threshold of 1 minute, and make no causal claim.

The independent reference calculations give these results:

The negative difference means the pilot group’s average is lower. The interval runs from about 1.35 to 2.65 minutes lower, entirely beyond our hypothetical one-minute threshold. If the test’s assumptions hold, this is evidence of a difference that would merit investigation; it is not a rollout decision.

The t statistic expresses the estimated difference relative to its standard error, a measure of uncertainty in that estimate. Welch’s degrees of freedom adjust the uncertainty calculation for the groups’ sizes and spreads, so the value need not be a whole number. The R t-test documentation identifies these outputs and the setting that allows unequal spreads (variances).

The actual Anomaly test output, converted to CSV, preserves the unrounded values. We compared that downloaded result with the separate reference calculations, rather than relying only on the chat explanation.

Keep the test settings visible

What the p-value and interval do not say

The p-value describes how unusual a result at least this extreme would be if the true group means were equal (the null hypothesis) and the test’s assumptions held. It is not the probability that the null hypothesis is true, that the pilot works, or that the finding happened “by chance.” A large p-value would not prove the workflows equivalent.

Under its assumptions, the method behind a 95% confidence interval captures the population difference (the true difference in the wider group the sample represents) in about 95% of repeated samples. It does not give a 95% probability that this fixed population difference lies inside this particular interval. Read uncertainty alongside effect size (how large the difference is) and study design; the American Statistical Association’s guidance warns against basing decisions on a significance threshold alone.

05Explore a relationship with correlation

Choose one clearly defined subset and pair the two numeric measurements on the same rows. Plot them, then calculate Pearson correlation if a straight-line relationship is the question. Keep association separate from causation.

Now ask whether more experienced operators tend to have lower handling times within the current workflow. Use its 48 complete pairs: tenure in months on the horizontal axis and handling time in minutes on the vertical axis. Mixing workflows would introduce another difference into the relationship.

The checked Pearson correlation is r = −0.525. Higher tenure tends to accompany lower handling time, with substantial variation around that relationship. Pearson’s r ranges from −1 to +1 and measures straight-line association; the SciPy reference also illustrates why a small correlation can conceal a curved relationship.

Inspect the scatterplot before summarizing it. Unusual points can change r, and a pattern does not show that extra tenure causes faster work. A rank correlation, such as Spearman’s, asks whether the values rise or fall together in order, not necessarily along a straight line; it is not an automatic repair for every problem with the data or design.

Inspect the paired observations

06Fit and inspect an explanatory regression

Fit a simple linear regression on the same complete pairs. Report the equation, units, sample size and R-squared. Inspect residuals and influential observations before interpreting the slope, and stay within the observed input range.

Correlation describes direction and strength. Regression expresses the fitted relationship in units you can discuss. Ordinary least squares chooses the line that minimizes the squared gaps between recorded and fitted values. For the same 48 current-workflow operators, fitting this line with an intercept gives:

Fitted handling time = 14 − 0.10 × tenure in months

Within this constructed sample, one additional month of tenure corresponds to 0.10 minutes lower fitted handling time, or six seconds. The slope is descriptive, not a claim that another month of employment will produce that improvement.

Read the fitted line in business units

R² = 0.276: the fitted straight line accounts for about 27.6% of the variation in these handling times around their mean. It does not mean “27.6% accurate.” Here, R² equals Pearson r squared, as documented for simple linear regression in SciPy.

A residual is the observed value minus the fitted value. Plot residuals against fitted values: look for curvature, widening spread and individual points that dominate the result. The US National Institute of Standards and Technology (NIST) explains in its model-validation guidance why a high R² cannot replace those checks. The residual table in the calculation-check pack preserves every point used here.

Here, residuals form four bands from −1.5 to +1.5 minutes with consistent spread. That artificial pattern does not resemble a smooth, bell-shaped normal distribution.

Inspect the residuals

Observed tenure runs from 3 to 25 months. The intercept of 14 is the line’s value at zero months, outside that range; do not interpret it as measured performance for a new starter. This explanatory fit has not been validated for forecasting or staff assessment.

07Save the checked result and its limits

Keep the source file, inclusion rule, method settings, calculations and interpretation together. State what the evidence does and does not establish. Reconsider the study design when the next report includes repeated observations.

A useful decision note for the original sample reads:

After removing two exact repeats and excluding two missing outcomes, we compared 72 pilot operators with 48 current-workflow operators, equally weighted. Pilot minus current mean handling time was −2.00 minutes; the two-sided Welch 95% interval was −2.65 to −1.35. This synthetic observational example does not establish a workflow effect. For real data, investigate ticket mix and service quality before considering a controlled evaluation.

We asked Anomaly to save a decision note and downloaded this unedited report from the synthetic example. Its figures are correct, and so are its caveats that the data do not show cause. But it recommends continuing the pilot and does not label the data synthetic. Constructed teaching data cannot justify that business action. Review the recommendation as carefully as the arithmetic.

Review the saved report

For next month’s report, reuse the definitions but reconsider the design. A new within-month comparison of different operators can still use an independent-groups test. Comparing the same operators across months instead requires attention to their related measurements: paired differences for a before/after question, or a model accounting for repeated observations across several periods. Anomaly made this distinction when we asked about measuring the same operators again.