Dreamware / Work / COVID-19 Scientific Paper Dashboard
COVID-19 Scientific Paper Dashboard
Transformed an on-demand NLP search and analysis tool into a web-scale system that helped researchers and frontline doctors make sense of 200,000+ scientific papers during the pandemic.
The Challenge
Primer AI had a product called Science — a tool for exploring academic literature from PubMed, arXiv, and Crossref. Paper abstracts were ingested into Elasticsearch, and a linear processing pipeline handled analysis tasks using a homegrown queue built on top of Elasticsearch itself. The system was fragile — the queue mechanism was flakey, the pipeline was rigid, and under any real load it would fall over.
John Jansen was consulting at Primer as an engineer when John Bohannon, Primer's head of data science, tasked him with rebuilding Science from the ground up. The vision was ambitious: Sean Gourley wanted users to enter a query and watch results progressively appear as different analyses completed — entity extraction, topic modelling, classification — each populating its own panel in real time. The existing linear pipeline couldn't support this. It needed to be replaced with something fundamentally different.
Then, before the dust had settled on the rebuild, COVID-19 hit — and the same system needed to go from serving a handful of researchers to powering a public dashboard at web scale.
Our Approach
The core NLP engine was proven — entity extraction, topic modelling, text classification all worked. The challenge was architectural: how to turn an on-demand system into a web-scale public dashboard, essentially overnight.
John made the critical architectural decisions under extreme time pressure. An aggressive caching layer was added to the system so that pre-computed analysis results could be served instantly rather than computed on every request. Scheduled and repeating saved searches were implemented — the system would automatically re-run key COVID-19 queries on a schedule, updating the cached results as new papers were ingested. This meant the dashboard could serve fresh, rich analysis to thousands of users without the backend collapsing under the load.
The progressive loading architecture from Science was preserved — the same Turbo Frames and Hotwire patterns that made the on-demand tool feel responsive now powered a dashboard where different analysis panels updated independently as fresh data flowed in. The white-label architecture meant the same underlying engine could power both the COVID-19 Primer dashboard and other products without code duplication.
The Outcome
The COVID-19 Primer dashboard won 1st Prize in the Professional Track of the CGDV "Flattening the Curve: COVID-19 Data Challenge" (CGDV Data Challenge) — a global competition bringing together data scientists, economists, and public health experts. The dashboard launched in April 2020 and was immediately used by researchers and frontline medical staff. An emergency medicine doctor at UC San Francisco described it as "actually a really amazing way to cut through the noise" and used it to inform hospital treatment protocols.
The platform was featured in Axios, and the dashboard is still live at covid19primer.com — a system built under pressure in 2020 that continues to serve in 2026. The architectural decisions made overnight — the caching layer, the scheduled searches, the progressive rendering — proved to be exactly right. What started as a rebuild of a failing academic search tool became one of the most recognised COVID data resources in the world.
Key result
1st Prize (Professional) — CGDV Flattening the Curve Data Challenge, featured in Axios, used by UCSF doctors, dashboard still live at covid19primer.com
Tech Stack
Related Services
Interested in similar work?
Tell us about your project. We'll have an honest conversation about what's possible, what it costs, and how we'd approach it.