Field test · Norfolk Fire Department
Same Data, Different Answers: An AI Field Test
We asked AI the same four questions about Norfolk Fire Department's inspection records twice: once with the report files uploaded, once connected through FlorianAI. A two-page summary of what came back.
A two-page summary of a real-data test on Norfolk Fire Department's inspection records. With the report files uploaded, the AI matched the records on one of four questions and gave three different numbers in four runs on another. Connected through FlorianAI, the same AI matched the records on all four, with the same answer every run. Tested with Claude only; ChatGPT was not tested.
What you'll learn
- Why uploading report exports into an AI chat can return a different number every time you ask the same question
- How an AI working from uploaded files decided "open" was not recorded, invented its own rule, and reported 12 open violations when the records show 8
- What changes when definitions like "completed" and "open" are written once in code instead of guessed from column names on every run
- How answer speed compared on a one-file question and a two-file join of about 5,200 rows
- The two properties a number needs before you take it to a select board or city manager: it matches the records, and it comes back the same when someone asks again
The chief asks the question. FlorianAI keeps the rulebook.
What's inside
- Scorecard: four inspection questions graded against the department's own records
- Consistency: the same question asked four times on each side
- Speed: average answer time on a simple question and a complex one
- Why the answers differ: who defines the terms and who does the math
- How we tested: 674 inspections, 35 violations, and 4,543 buildings from 23 October 2026
Frequently asked questions
- What was tested?
- Norfolk Fire Department's inspection, violation, and building exports from 23 October 2026: 674 inspections, 35 violations, and 4,543 buildings. The same four questions were asked word for word in a fresh Claude chat on each side, once with the report files uploaded and once connected through FlorianAI, and every answer was graded against the department's own records.
- Did you test ChatGPT?
- No. The test used Claude on both sides. Chiefs often ask about ChatGPT, but we did not test ChatGPT or any other AI tool, and the results should not be read as a claim about them.
- Why did the uploaded files give different answers?
- With uploaded files, the AI has to work out what fields like "open" and "inspected" mean from column names on every run, and it sometimes works them out differently. It also does its own arithmetic without saying which figures were computed and which were estimated.
- What does connecting through FlorianAI change?
- Definitions are set once in code and FlorianAI computes every number, so the AI quotes and explains the result instead of rebuilding the rules each time. In the test, that produced the same answer on every run.
- What is FlorianAI?
- FlorianAI is an AI operations assistant built specifically for fire departments. It connects the assistant a department already uses to the department's own records, so chiefs and officers can ask one question and get an answer that spans their systems. It reads from the systems you already run rather than replacing them.