How indexed. reads AI answers and turns them into a score

What the tool reads

Every week the same 15 buyer questions are put to ChatGPT, Perplexity and Google AI Overviews. These are the questions a fleet manager types when looking for software, from early research to comparing named companies.

15 questions × 3 AI tools × 2 runs = 90 answers a week

Each question is asked twice on each AI tool because the same question can get a different answer a few minutes later. Comparing the runs is how the tool tells a real change from chance. The data loaded now holds 507 answers from week 1 to week 6.

Some questions matter more than others. Each has a priority from 1 to 3, with 3 closest to a purchase: here 6 are priority 3, 7 priority 2 and 2 priority 1.

How a company is found

A company counts as mentioned when the answer itself names it, in any of these ways:

• Its name, in full or short (“Corvane Fleet”, “Corvane”).
• Known spellings, such as “CorvaneFleet”, “Corvain” and “Corvain Fleet”.
• Its website, such as corvanefleet.com.
• Close misspellings of longer names, which AI tools sometimes produce.

Corvane Logistics is a different company with a similar name and is never counted.

Links alone don't count. AI tools often list their sources under an answer; a company that appears only there, and not in the answer, isn't counted as mentioned.

Position and tone

Position is the order companies are first named in: 1 for the first, 2 for the next new one, and so on.

Tone is what the answer says about each company. There are four:

• Recommended“For a small trucking company, Corvane Fleet is a strong pick.”
• Mentioned“Corvane Fleet offers GPS tracking and a driver app.”
• Criticised“Some users complain that Corvane Fleet's reports are clunky.”
• Advised against“At your size I'd avoid Corvane Fleet.”

A company that is named but not judged is Mentioned. When an answer changes its mind, the last verdict in the answer wins, and within a sentence the part after “but” or “though” decides:

Trakvia is a strong pick for large fleets. For a team of five trucks, though, I'd avoid Trakvia. Advised against

The score

The score, from 0 to 100, is how strongly AI answers point buyers toward a company. Each answer gives each company points for its tone:

Being named early counts a little more: named first, full points; second, 90%; third or later, 80%.

The runs of each question on each AI tool are averaged. Those averages are then combined into one score for the week, with each question weighted by its priority: a priority 3 question counts three times as much as a priority 1 question.

Clear change or normal variation

Ask an AI tool the same question twice and the answer can differ. So the tool measures, for each company, how much its points usually differ between the two runs of the same question in the same week. That is the usual run-to-run difference. It is estimated only from the weeks up to the one you are looking at, so a later week never changes an earlier verdict.

A change between two weeks is called a clear change when it is more than twice the usual run-to-run difference for the questions compared, and at least one point. Anything smaller is normal variation: it may be real, but it could also be chance.

This is an estimate, not proof. It assumes the two runs of a question are independent and that a company's run-to-run difference is similar across questions. With only a few weeks and two runs, the estimate is itself rough. A small but real shift can read as normal variation until it lasts a few weeks, and a clear change says that something moved, not why.

Missing data

A request that fails, with an error or an empty answer, is left out. It doesn't count as zero, because a company can't lose points for an answer that never came back. Duplicate copies of the same answer are counted once.

Week-on-week changes compare like with like: only the questions and AI tools that both weeks have. If an AI tool is missing for a week, it can't look like a drop. For example, week 5 has no answers from Perplexity, so changes that week compare ChatGPT and Google AI Overviews only.

Screens with an incomplete week say so at the top. See what came in each week.

Wrong facts

Claims about a company are checked against the fact sheet, which covers Corvane Fleet, Trakvia, Routelyne and Gridwell Systems. Five kinds of claim are checked:

• starting price
• where the company is based
• the year it was founded
• features on the fact sheet, such as dashcams or ELD compliance
• integrations, such as QuickBooks

Anything not on the fact sheet, such as review ratings or customer numbers, is left alone. It isn't marked wrong, because there's nothing to check it against. A sentence that names two companies isn't checked either, since it's unclear which company the claim is about.

How accurate it is

15 answers were drawn at random from the sample data and labelled from the answer text by the AI assistant used to build this, not yet reviewed by a person. The tool agreed with those labels on all 90 company checks (whether each company was named), all 39 positions and all 39 tones. The rules were written after reading the sample data, so this is the best case.

On its first run, a set of sentences in new wording written before the run got tone right on 8 of 12. 4 of the 4 misses showed as Mentioned rather than a wrong verdict, and 0 turned praise into criticism or the reverse. Wrong facts were found in 3 of 4 contradicting sentences, with 0 false alarms, and 4 of 4 true or unchecked statements were left alone.

These are small samples (12 and 8 sentences), so they show the kind of mistakes to expect rather than a precise rate. The pattern so far: missing a verdict is more common than inventing one, so scores are more likely to understate strong opinions than overstate them.

What it can't tell you

• The history is short: 6 weeks so far. Trends get more trustworthy with every week added.
• 15 questions aren't the whole market. Buyers ask many other things, in other words, and AI answers to those may differ.
• The tool shows patterns, not causes. It can tell you that an AI tool stopped recommending a company, not why.