The article compares ten hands-on tools in descending provisional-score order, with ties retaining prior order. The eight original workbook ratings below are unchanged. Better Analyst (Formula Bot) adds a separate 21/25 workbook self-score. ThoughtSpot adds a newly assigned 16/25 editorial score for its converted-CSV run only, not an XLSX rating. Neither additional score is independently finalized; differing inputs and plans prevent a controlled cross-tool ranking. Domo's removed documentation-only entry remains below as history.
Evidence supporting AI data analysis tools. Tests took place on 7–8 October 2026. This is a reader-facing summary, not a replacement for original run logs or the canonical Tool Testing Repository.
Raw transcripts and original exports are retained for reviewers. They are not anonymously accessible. The assembled workbook is not redistributed here; its provenance and redistribution terms still need clearance. The answer key permits checking definitions and reconciliation, but cannot replace access to the source workbook.
File: adventureworks_ai_tool_benchmark.xlsx, assembled in-house from AdventureWorks sample CSVs. It contains six sheets: sales, returns, products, product_subcategories, product_categories, and territories. Size: 2,167,052 bytes; sales rows: 56,046; return rows: 1,809. No customer data was used.
SHA-256: e790641e7c080df4fb63de63a1964154fdab3508a244e4200f25f75aef598108.
Inspect every sheet in this workbook. Explain the table relationships, identify any data-quality or semantic risks, and state which business measures are and are not computable. Do not calculate revenue unless a sale-price field exists.
Define units sold as SUM(OrderQuantity). Calculate: (a) units sold by year, (b) units sold and share of total by product category, and (c) units sold by sales region. Show the calculations, code, SQL, formulas, or other reproducible logic you used.
Define return rate for each product category as 100 × SUM(ReturnQuantity) ÷ SUM(OrderQuantity). Join the necessary sheets, report units sold, units returned, and return-rate percentage for every category, and identify the highest return rate. Explain any limitations in this definition.
Create a monthly 2017 units-sold chart and a concise executive summary containing three numerically supported findings. Then explain how another analyst can verify each finding. If the product supports export or sharing, produce the most useful available artifact.
The eight original scored runs used browser interfaces, not API substitutes, and the original workbook was not split or converted. ThoughtSpot's newly scored adaptation used six sheet-level CSVs. Original instructions withheld the answer key until a run finished. Later rerun testers had already seen it, so those reruns were not evaluator-blind.
Ratings are provisional editorial judgments, not statistical estimates: correctness, data handling, output usefulness, verifiability, and cost and access each receive 0–5 points. Add the five ratings without hidden weighting. Different plans and retries prevent treating the totals as a universal product ranking.
Download the original computed answer key (JSON). The following values come from that file, not vendor claims.
| Check | Expected result / definition |
|---|---|
| Total units sold | 84,174 = 2,630 (2015) + 36,230 (2016) + 45,314 (2017) |
| 2017 monthly units | January 7,020; February 6,828; March 7,327; April 7,680; May 8,199; June 8,260. Sum: 45,314. July–December are missing, not zero. |
| Bikes return rate | 100 × 429 returned units ÷ 13,929 sold units = 3.0799%. Aggregate sales and returns separately by category before dividing. |
| All returned units | 1,130 Accessories + 429 Bikes + 269 Clothing = 1,828. |
| January-to-June change | 100 × (8,260 ÷ 7,020 − 1) = 17.6638%, shown as 17.66%. |
| Q2-to-Q1 change | Q1 21,175; Q2 24,139. 100 × (24,139 ÷ 21,175 − 1) = 13.9976%, shown as 14.0%. |
| Revenue | Not computable: there is no transaction sale-price field. A catalog product price is not proof of the price actually paid. |
The comparison selects completed 7 October runs for Anomaly, Claude, ChatGPT, and Julius, and recorded 8 October reruns for Deepnote, Gemini, Quick, and Hex. Selection followed troubleshooting, not a preregistered attempt limit. Earlier failures were retained rather than overwritten. The 8 October reruns now have their own canonical Confluence records; older run records remain historical evidence, not proof of these later outcomes.
| Tool / selected run | Tested account | Ratings in stated order | Result and prior attempt |
|---|---|---|---|
| Anomaly AI · 7 Oct | Production Free | 5,5,5,5,5 = 25 | 4/4 prompts; core key matched; CSV exported. No scored factual defect in this run. |
| Claude · 7 Oct | Free; UI label Sonnet 5.5 Medium | 5,5,5,5,5 = 25 | 4/4 prompts; core key matched; HTML exported and reopened. Public sharing unavailable. |
| ChatGPT · 7 Oct | Free; model not visibly identified | 4,5,5,5,4 = 23 | 4/4 prompts; core key matched; XLSX exported. StockDate range misstated. Separate isolated Prompt 1 retry ended incomplete. |
| Julius · 7 Oct | Free; UI label Julius 1.2 Lite | 5,5,5,4,4 = 23 | 4/4 prompts; core key matched; PDF exported. Displayed float("") is invalid Python. Observed 25-credit daily allowance exhausted. |
| Deepnote · 8 Oct | Existing Team workspace | 5,5,5,5,4 = 24 | 4/4 prompts; core key matched; two exports: CSV and PNG. On 7 Oct an upload never became a project file (4/25 diagnostic); the Files-sidebar rerun worked. |
| Gemini · 8 Oct | Personal account; exact plan not shown | 4,4,4,4,4 = 20 | 3/4 answers; Prompt 1 ended in an error despite a code trace. On 7 Oct only inspection completed (13/25 diagnostic); prompts 2–4 failed. |
| Amazon Quick · 8 Oct | 30-day trial; Space | 1,1,2,3,4 = 11 | 4 responses, 0/4 fully satisfied; incomplete source tables and an all-zero chart. Earlier direct chat rejected the file at 665,000 characters; Space remained Processing (5/25 diagnostic). |
| Hex · 8 Oct | Team trial | 0,1,0,0,3 = 4 | 0/4 answers. Native upload failed when Prompt 1 was submitted, including retries. Zero analytical points mean no output to assess, not wrong numerical answers. |
These runs used different inputs and sometimes paid plans. Testers were not uniformly blind to the key. The synthetic messy-data fixture deliberately inserted 295 duplicate rows adding 441 units. Clean units: 45,314; duplicate-retaining units: 45,755. Overstatement = 100 × 441 ÷ 45,314 = 0.9732%.
The clean-workbook run matched the key and exported CSV. The later messy-data response gave 43,506 units, not the clean 45,314. The clean-run 25/25 does not establish cleaning accuracy.
The original Free run exported a 5.7 KB standalone HTML brief. The separate Pro-account messy-data run reached 45,314. File inspection found six native chart objects in both its XLSX and PowerPoint exports; these are paid-plan observations, not proof of Free-plan support.
The selected rerun produced CSV and PNG, not three exports. The separate messy-data run returned 45,755, retaining the fixture's duplicate contribution.
The requested clean-workbook numbers matched, but StockDate was described as 2002–2003 rather than 2001-09-11–2004-06-15. The messy-data run returned 45,755. On a 616,506-row file its component totals were correct, but prose stated 1,025,155 rather than 925,914. A shared Free account hit a daily analysis limit.
Requested clean-workbook values matched, but the displayed float("") snippet cannot execute unchanged. Credits ended before the separate messy-data and large-file checks; those capabilities remain untested, not failed.
The selected rerun's prompts 2–4 matched the key at displayed precision. Its first answer failed. A generated XLSX had an unusable chart preview despite a correct in-chat chart; an unsupported financial-impact statement was also recorded. The observed account plan was not identified.
The upload showed Ready when revisited at an approximate age of 15 hours. The transition time was not observed, so this is not a measured 15-hour processing duration. The code environment lacked full source tables. No requested aggregate series was calculated; the generated chart showed twelve zero-valued months, not the six available values.
The original six-sheet workbook failed when submitting Prompt 1 in the selected 8 October retry; prompts 2–4 were not independently attempted. Separate Team-trial tests on 7 October used different XLSX files and made four correct charts and a dashboard with filter controls; filter interaction was not independently verified. That success preceded the selected failed retry and does not establish that the original workbook worked. The separate workflow refreshed after June data was added and exported PDF, HTML, PNG and CSV. Its messy-data response identified 295 duplicate rows, 884 mixed dates, 590 text quantities and 147 text territory codes, reaching 45,314.
On 8 October the account showed “Analyst Studio unavailable. Contact your org admin to know more.” Documented XLSX ingestion was not available to test. Six converted CSVs later produced correct core values, including 84,174 units. On 9 October, at the user's request, this saved CSV adaptation receives a separate provisional editorial score of 16/25. The original tester did not assign that score.
An independently recomputed raw-join SQL example gives a 100.57% Bikes return rate, not 3.08%; the growth headline said 27.9% rather than 14.0%. A messy CSV returned 45,755 rather than 45,314 (+0.97%). The original chart stayed blank; a later render required browser visibility, animation and observer workarounds. Normal rendering remains unverified. The documented 5 MB limit refers to chat attachments, not every ingestion path; Analyst Studio Excel and Liveboard/Answer exports remain untested.
Review by Marketing-Analytics-AnubhavGPT on 9 October 2026, using saved 8 October A1 four-prompt CSV evidence and its source-data checks. Same five equally weighted 0–5 dimensions; total = 3 + 4 + 3 + 3 + 3 = 16. These are judgment-based ratings, not measured pass percentages or an independently finalized audit. No new product prompts or purchases were made. It is neither a score for the unchanged workbook nor a whole-product rating.
| Criterion | Score /5 | Evidence and deduction |
|---|---|---|
| Correctness | 3 | Core yearly, category, regional, return-rate and six monthly values match the source; refused to invent revenue. Material executive-summary errors remain: Q2 headline 27.9% versus computed 14.0%, and claimed uninterrupted monthly acceleration despite February's decline. Four completed responses do not mean four fully correct responses. |
| Data handling | 4 | Read all six CSV tables, mapped keys and used separately aggregated Python sales/returns to obtain correct values. One deduction: the accompanying purported equivalent SQL joins raw fact rows and would multiply totals. Native Excel ingestion is outside this CSV score, not scored as a numerical failure. |
| Output usefulness | 3 | Readable tables, summary and CSV/PDF download controls were exposed. Requested chart stayed at “Rendering chart...” and the summary needs factual correction. A successfully reopened A1 PDF was not independently established, so advertised downloads are not treated as verified report delivery. |
| Verifiability | 3 | Visible Python/SQL, metric definitions and reconciliation steps support checking. SQL and Python are not equivalent; complete generated code and every A1 export have not been rerun/reopened unchanged. Independent local source checks do not certify all product code. |
| Cost/access | 3 | Restored authenticated trial completed the adapted prompts without purchase. Earlier reauthentication blocked a retry; tenant Analyst Studio was unavailable. Commercial trial edition and the paid plan required for the broader workflow were not established. This is observed access friction, not proof of whole-product unavailability or a paid-plan failure. |
Preserved primary evidence: A1 CSV-adapted report, full rendered transcript and five screenshots dated 8 October, plus the conversion manifest. The original report's decision not to assign a score remains historical evidence; this new user-requested assessment does not overwrite it. Separate A2 messy/large-file/dashboard checks and the browser-workaround chart are not used to raise or lower these five A1 scores. The A2 seven-criterion partial score is not converted into this five-criterion score. Numerical checks verify observations; they do not independently validate the editorial point assignments. All release holds remain unchanged.
Browser run on 8 October 2026, Free plan, visible model label Base Model. The unchanged workbook fingerprint is the one above. Prompts 1–4 were submitted at 13:38:39.841, 13:43:26.655, 13:45:01.383 and 13:46:23.711 UTC. Original response text, screenshots and product exports are retained. The tester had seen answer-key values: this was not a blinded test. No analytical code was supplied to the product. An internal chart error recovered automatically; the workbook test had no tester analytical retry.
| Check | Saved product result and verification |
|---|---|
| Clean workbook | All four responses completed. Year totals 2,630 + 36,230 + 45,314 = 84,174. Category/region values and return rates match at displayed precision. The six exported monthly chart values match the original source. Refused to present catalog price as actual revenue. CSV and SVG exports retained. |
| Verifiability limits | Join/grouping descriptions and checking steps were saved, but full generated analytical code was not saved and rerun unchanged. External sharing was not checked. |
| Messy CSV: first attempt | 45,755 instead of 45,314. The 441-unit excess is the retained duplicate contribution. It returned a combined category-region table, not the separate requested summaries. |
| Messy CSV: one fixed retry | 44,427 instead of 45,314. Calculation: 45,314 + 441 retained duplicate units − 1,328 valid mixed-date units excluded = 44,427. Valid DD/MM/YYYY dates were called invalid. Both attempts were preserved. |
| Free-plan limits | Observed one-file-per-chat restriction blocked adding the targets file. The original 41,955,071-byte large CSV was refused by the 5 MB file limit before any prompt. These are plan limits, not wrong-calculation results. No paid-plan run. |
Correctness 5/5 + data handling 5/5 + output usefulness 4/5 + verifiability 3/5 + cost/access 4/5 = 21/25. The saved review calls the fourth dimension transparency/reproducibility; it maps here to verifiability, not a new measurement. Independent numerical verification confirms the observations, not these subjective rating judgments. This is not a final independent score, and separate CSV failures do not silently change the clean-workbook score. No cross-tool overall ranking is asserted.
The source record is the frozen wave-2 run folder, with an independent 8 October 18:45 UTC numerical audit. A fresh read-only 9 October recalculation again matched all six workbook monthly exports and reproduced both messy-data errors from original input bytes. The saved 8 October 19:28 UTC registration receipt records workbook run 10158104 v2 and CSV run 9633823 v2 (workspace login required). A current live readback and reciprocal A1 article-record links have not been verified in this revision.
This summary and the answer-key asset accompany the article; private raw logs remain restricted. Original eight rerun records and the earlier CMO-signed ten-tool brief are complete; that earlier brief included Domo and is not claimed to register the new Better Analyst entry. The user subsequently approved replacing Domo and adding Better Analyst's score. No tool was retested while preparing this revision. Original captures remain untouched; framing-only crops exclude account details without changing results. Source screenshots use mixed themes and dimensions.
Publication was explicitly requested on 9 October after the provisional-rating limitations were disclosed. Publication does not independently validate those ratings, make unequal inputs comparable, or establish a fresh independent article-quality score. The source workbook is still not redistributed.
Official sources reviewed on 8 October 2026: Features for file uploads and connected data; Use AI Chat for questions over DataSets/cards, source references and generated-SQL process steps; Pricing for custom credit-based pricing and the advertised 30-day trial. These are vendor descriptions, not independently observed analytical results.
An Anubhav Chrome access check reached Domo's company-domain login; an authenticated workspace and AI Chat were not reached. No analysis prompts, dataset uploads, computed answers, exports, test-plan entitlement or performance screenshot were recorded. No trial was created and no score is assigned. This record is retained as history, not a tested recommendation.