Where AI Scored Higher Than Search, and Where It Didn’t

Marketing Doctors Patient Discovery Study #1

An exploratory comparison of 25 healthcare-discovery questions informed by 50 coded observations

Review edition v0.4 | October 10, 2026 | Content review in progress

Author: Eric L. Clark, Marketing Doctors

Research and analysis collaborator: Chispa, with OpenAI GPT-5.6 Sol identified in the study record. Other AI contributions are disclosed below.

Abstract

Question. Where might conversational AI help people understand healthcare options, and where does traditional search already provide a strong route forward?

Approach. A purposively selected corpus of 50 coded observations informed 25 questions about unresolved symptoms, provider selection, pediatric neuropsychiatric conditions, cognitive health, and trust or financial decisions. Archived search assessments were compared with 75 answers from ChatGPT, Gemini and Kimi. Three AI raters scored the answers on synthesis, navigation, restraint and decision support, each from 0 to 3, for a total of 12. AI results are equal-weight means across raters and questions. Search was not rescored by that panel, so the comparison does not establish equivalent measurement across channels.

Results. On five unresolved-health questions, search averaged 5.40/12, compared with 10.07 for ChatGPT, 10.67 for Gemini and 8.93 for Kimi. On five cognitive-health questions, search averaged 11.40, matching ChatGPT and exceeding Gemini's 10.73 and Kimi's 11.20. The unresolved-health segment had the largest positive AI-minus-search difference for every generator across the five segments. Exact agreement among the AI raters was 55.3%; agreement within one point was 95.7%. Navigation produced the weakest agreement.

Interpretation. The score advantage varied with the question set. Low familiarity with medical terminology alone does not explain it: search also scored strongly on cognitive questions expressed without diagnostic terms. Vocabulary, the organization of available information, and fragmented care pathways offer an emerging explanation to test. This study measured rated responses, not patient decisions, diagnostic accuracy or clinical outcomes.

The decision before the diagnosis

One question in the study asked:

I've seen several specialists for symptoms affecting different parts of my body, but nobody has figured out what's wrong. What kind of doctor should I see?

This is a researcher-prepared study question, not a verbatim patient quotation. It asks for a route through care when the person does not yet have a useful name for the problem.

Marketing Doctors uses Patient Discovery to describe research into how people express health-related needs, assess information and identify possible next steps. It includes decisions about clinicians, credibility, cost and the limits of available advice. Functional medicine is one application examined here; it does not define the research program or represent an assumed destination for unexplained symptoms.

Study #1 examined whether differences in those questions were associated with differences in rated search and AI responses. Its most useful observation was the contrast between unresolved-health questions and cognitive-health questions. Both can begin in ordinary language. Their scores differed substantially.

How the comparison was assembled

The starting corpus contained 50 coded observations selected for their relevance to healthcare discovery: 27 patient-community entries, 11 parent-forum entries, six caregiver-community entries and six institutional-resource entries. The initial 20 pilot entries remained within the expanded corpus.

These are not 50 verified independent patient accounts. Some entries point to the same Reddit post or forum thread, and six concern institutional resources. Researcher summaries describe the needs represented in the material. They should not be read as quotations, a representative survey or direct observation of patient behavior.

Researchers coded such features as familiarity with a condition, familiarity with provider options, and proximity to a decision. Forty-two entries were coded as high decision proximity, 21 as low provider knowledge, and 18 had both characteristics. The last figure is 18 of 50 entries, not an estimate that 36% of patients face this situation. The coding itself has not undergone independent validation.

The research history records an early synthetic question set being discarded in favor of questions grounded in observed needs. The final 25 questions were normalized by researchers into consistent prompts. Their wording matches across the archived search scorecard and AI test packet. This verifies which questions were compared; it does not reconstruct every step from original source passage to final prompt.

Historical Semrush US keyword estimates provided context on the visibility of different formulations. They were not measures of patient numbers or clinical importance. Complete original exports and extraction settings are unavailable. The estimates therefore do not establish that people move through a particular sequence of search terms, or that a term recorded at zero volume represents no underlying need.

The final response sets were ChatGPT, Gemini and Kimi. An earlier collection template named Perplexity; the research history reports a quota interruption and substitution of Kimi. Perplexity responses are outside this scored comparison. Exact model builds and complete session conditions were not preserved. Eric L. Clark reported on October 10, 2026 that Claude declined the answer-generation test. Its reason was not independently established. Claude participated separately as a rater; refusal to generate and participation in scoring should not be conflated.

What a score means

The frozen rubric assigned 0–3 points to each of four dimensions:

The four scores were added with equal weight. A total of 12 is the top of this rubric, not a certification of medical accuracy. A score of 10 is not twice the clinical value of a score of 5.

GPT-5.6 Sol, Claude and DeepSeek served as AI raters. Each scored all 75 answers, producing 225 response ratings. The historical record describes neutral response labels and subsequent unblinding. Neutral labels are preserved, but full logs authenticating all blinding conditions are not. Writing style could also reveal a generator's identity. Separate AI scoring runs should not be described as three independent clinical experts.

The search baseline used the same four dimensions, but it represents an archived assessment of a search experience rather than a generated answer. Each question has a selected evidence URL; complete search-result captures are missing. The three-rater panel did not independently rescore search. These differences could influence the apparent gap, especially on a rubric that values organizing information into a next step.

The paired finding: unresolved health and cognitive health

All values are mean rubric totals out of 12. AI means include three raters; search values come from the archived baseline. These are descriptive comparisons under unequal evaluation conditions.

For unresolved-health questions, all three generators scored above the search baseline. The baseline records describe information that offered possible explanations but left the person to assemble a route forward. The AI ratings favored responses that organized the uncertainty and explained conditional next steps. Those judgments concern the form and perceived usefulness of the answer; the recommended pathways were not independently established as clinically appropriate.

For cognitive health, search matched or exceeded each generator's mean. Two questions in this segment were explicitly classified as low-vocabulary or caregiver questions, yet their individual search totals were 12 and 11. That counterexample matters. It prevents interpreting the unresolved-health result as evidence that unfamiliarity with diagnostic terminology generally gives AI an advantage.

The complete segment comparison also shows that the result was not uniform:

The pediatric segment concerns pediatric acute-onset neuropsychiatric syndrome (PANS) and pediatric autoimmune neuropsychiatric disorders associated with streptococcal infections (PANDAS). Inclusion of these questions does not establish the clinical correctness of any answer or endorse a particular treatment pathway.

Across these five segments, each generator's largest positive mean difference from search occurred in unresolved health. Overall AI means across all 25 questions were 10.91 for ChatGPT, 10.71 for Gemini and 10.29 for Kimi. Segment sizes differ, so those overall means cannot be obtained by simply averaging the five displayed segment means. Nor do the small differences establish a general ranking of healthcare AI systems.

Where the raters disagreed

Agreement was assessed separately for each answer and scoring dimension, giving 300 sets of three scores. Exact agreement meant all three scores were identical. Within-one-point agreement meant the highest and lowest differed by no more than one point; it includes exact agreement.

Navigation was the least consistently scored dimension. This identifies a measurement problem worth investigating with human clinical reviewers. It does not establish that navigation advice was right or wrong. These percentages describe agreement, not accuracy, and have not been adjusted for chance agreement.

Means also conceal meaningful differences among raters. For Kimi's unresolved-health answers, Claude's mean was 6.40, compared with 10.20 from each of the other two raters. The resulting three-rater mean of 8.93 should be read with that disagreement in view.

An emerging explanation, not a tested causal model

The pattern suggests that vocabulary may matter alongside two other features: how well existing information is organized around the question, and how fragmented the problem is across symptoms, specialties or possible next steps.

Someone can describe a memory concern without knowing a diagnostic term and still encounter an established information pathway. A question spanning unexplained symptoms and several specialists may require more work to connect the available pieces. This is an interpretation of the study pattern, not a measured explanation of it.

Our emerging hypothesis is that patient vocabulary, information-ecosystem maturity and problem fragmentation interact in shaping discovery difficulty. Here, information-ecosystem maturity means how clearly available sources explain a problem and its possible care pathways. The shorthand vocabulary × ecosystem maturity × fragmentation is conceptual; no statistical interaction was estimated. Study #1 did not manipulate these features separately, validate their measurement or establish that any one caused the score differences.

Other explanations remain plausible. The selected questions, search-source choices, response length, presentation and the rubric itself may favor one format. Cognitive-health scores were already near the maximum, leaving little room for improvement. A future study needs to separate these possibilities rather than treat the preferred explanation as settled.

A better-rated answer is not yet a better patient decision

In a randomized study using hypothetical medical scenarios, Bean and colleagues found that strong standalone AI performance did not translate into superior human performance against a control using usual resources. That experiment concerned condition identification and urgency of care, rather than our discovery rubric. It supports testing people using the tool, separately from scoring the tool's output; it does not validate Study #1 or establish clinical outcomes from our scores. Nature Medicine, 2026.

Study #1 included no experiment with patients using the answers. It did not measure whether a reader understood the advice, contacted an appropriate clinician, received a correct diagnosis or experienced a better outcome. High restraint scores also cannot substitute for a clinical safety review.

Corrections preserved with the evidence

Recalculation during the September 27, 2026 America/Chicago audit, corresponding to September 28 UTC for the frozen record, identified discrepancies in the historical summaries. The original artifacts were retained. The corrected values used here are:

DeepSeek's separate summary differed from its detailed matrix on 12 of 15 total or component means. The supplement preserves the complete reconciliation. Detailed scores were used for recalculation. The causes of the discrepancies are unknown. The corrected arithmetic preserves the principal segment pattern; this statement is descriptive, not a statistical test that the corrections were immaterial.

What can be reproduced, and what cannot

The preserved numerical artifacts allow reconstruction of the aggregates and agreement statistics reported here. That is stronger than relying on a historical narrative alone.

The original collection cannot be fully recreated from the archive. Session timestamps, model builds, browsing settings, personalization, location and conversation state are incompletely recorded. Planned fresh sessions are documented in a template, which is not proof that every original session met those conditions. No repeat collection tested stability over time or repeated generations.

Sampling and measurement impose further limits. This was a purposive, exploratory comparison with small topic segments, repeated source threads and researcher-generated prompts. It was not a randomized causal experiment or a preregistered confirmatory study. No power calculation or inferential significance claim is presented. Scores are bounded, subjective judgments on an ordinal scale; their sum and equal weighting are analytic choices, not validated measures of clinical utility.

The comparison also stops at the archived response. Some Gemini local-provider answers requested a location. A completed conversation could perform differently, but equivalent follow-up opportunities were not tested. AI raters may share preferences, and none substitutes for independent human clinical assessment.

Implications for healthcare communication

The practical question raised by this work is whether information helps someone move from an expressed concern to an understandable, conditional next step. A practice or health organization can investigate that question alongside the familiar question of which search terms attract traffic.

That is a proposed application, not a demonstrated marketing effect. Study #1 tested no website change, search-optimization intervention, conversion outcome or revenue result. It does not validate a Patient Discovery Audit as a commercial product. Any such service needs its own prospective validation and clear limits.

The paired result also leaves room for complementary roles: established search resources may serve some questions well, while conversational synthesis may help organize others. A combined workflow was not tested. The next research step should include human clinical review and people attempting realistic discovery tasks, with repeatable collection conditions and questions that separate terminology from the organization and complexity of the problem.

Conclusion

In this exploratory comparison, AI's largest rated advantage appeared on five unresolved-health questions. Search matched or exceeded AI on five cognitive-health questions. That contrast deserves further study, including the counterexample it provides to a vocabulary-only explanation. The evidence supports a task-specific observation and a testable hypothesis. Clinical benefit remains an unanswered question.

Authorship and contributions

Eric L. Clark is the author and accountable editor. Chispa is credited as an AI research and analysis collaborator, with OpenAI GPT-5.6 Sol identified in the study record. This attribution does not imply that every subsequent editing or auditing session used that same model. AI systems materially contributed to research design, analysis, scoring, auditing and manuscript development. Final editorial responsibility rests with Eric L. Clark and Marketing Doctors. Generator and rater roles are identified in the methods; the scoring panel consisted of AI systems rather than human clinical experts.

Funding and commercial interests

This study received no external funding. It was conducted using internal Marketing Doctors resources and researcher time. Marketing Doctors has a commercial interest in healthcare marketing and possible applications of Patient Discovery. The study does not validate a commercial service or establish a marketing benefit.

Ethics and privacy

The study involved secondary analysis of publicly accessible online material, keyword data, search results and AI-generated responses. Researchers did not recruit or interact with patients, access medical records, or intentionally collect personally identifiable private health information. Public patient and community material was coded into researcher summaries rather than reproduced as identifiable case histories. No institutional IRB determination was obtained. This was not a clinical study with enrolled patients.

The complete source-level corpus and community-source inventory will remain internal. Public accessibility does not remove the possibility of identifying a person through a linked discussion. The release supplement therefore excludes those links and source-level narratives. The statement above describes the reported activities; it does not assert a regulatory exemption or determination that review was unnecessary.

Data availability and reproducibility

The accompanying Study #1 reproducibility supplement is prepared for open access with the manuscript. It contains all 25 exact questions, the frozen rubric, segment definitions, search scores, the three AI-rater scoring matrices, aggregate calculations, agreement calculations and the corrections register. The package includes a calculation script so readers can reproduce the central numerical results from the disclosed score rows.

The complete 50-observation source-level corpus, community-source inventory, internal research procedures, query-generation workflow, internal prompts and full raw AI responses remain in the Marketing Doctors archive. The supplement supports independent arithmetic verification. These disclosure limits prevent readers from independently repeating all source coding or judging every response score from the public package alone. Full reconstruction of original collection conditions is also impossible because some session records were not preserved.

No bulk Semrush extract, patient-post excerpts or complete third-party publications are included. Historical keyword estimates are described as study context rather than offered as a downloadable database. Additional third-party excerpts or keyword-data releases would require their own permissions assessment.

Editorial status: this website edition is available for content review. Eric confirmed on October 5, 2026 that outside ethics review is not required before release. No outside review, institutional determination or regulatory exemption is claimed.

Reproducibility supplement

The accompanying questions, scoring data and calculations allow readers to reproduce the central numerical results. The disclosure limits described above still apply.