I fed the same ICU patients to two AI models, one that read numbers and one that read text, and found they fail on different people.
Try it: where each model goes blind
Every number in this demo is a published aggregate from the paper. No patient-level data is shown, and nothing here is drawn from invented points.
Step 1: the study as it was
22,591 ICU patients from MIMIC-III, each patient's first 24 hours of ICU data, represented two ways. Numbers: 458 human-engineered numeric features into XGBoost. Meaning: the same variables rewritten as text and embedded by a large language model, into the same XGBoost. Six prediction tasks.
In-hospital mortality, headline scores (AUROC)
On the headline score, the plain numbers win. That is not the finding.
Step 2: pick a task, see the result
Step 3: the disagreement explorer
The paper studied two tasks in depth, mortality and readmission. Pick one, then pick an error type, and see how much the two models' errors overlap. The numbers on screen are the paper's published aggregates.
The two models do not fail on the same patients. A leaderboard score cannot show you that. This can.
Step 4: what the embeddings kept
A probe asks the representation a question it was not trained on, and measures how much of the answer it kept (R²). Pick a lab and compare how well the embedding kept how often it was measured against the last value itself.
The embedding remembered the watching, not the number: it kept how closely a patient was being measured far better than the value that was measured.
Two models read the same 22,591 ICU patients from MIMIC-III, each patient’s first 24 hours: one as 458 engineered numeric features, one as the same variables rewritten as text and embedded by a large language model, both into the same XGBoost classifier.
Results
Numbers won five of six tasks: mortality AUROC 0.901 vs 0.835. Embeddings won 30-day readmission, 0.662 vs 0.590, a figure the paper flags as the best of a large search (108 configurations per task), a ceiling, not a typical result. The models failed on different patients: 80.7% of mortality false alarms were unique to one model. Probes showed the embeddings kept how often a lab was measured better than its last value (potassium count R² 0.496; last value 0.141). Paper: “Numeric versus Embedding Pipelines for Optimized Clinical Risk Prediction,” JAMIA Open. Code: github.com/Tensor-Lab-for-Computational-Medicine/EHR-Embeddings.