The landscape of modern healthcare is undergoing a tectonic shift. For decades, the Electronic Health Record (EHR) has served as the primary repository for clinical decision-making, documenting the "nurture" side of the human health equation: our medical history, lifestyle choices, and clinical encounters. Yet, these records offer only a monochromatic view of a patient’s life. To truly unlock the potential of predictive medicine, researchers must harmonize this lived experience with the "nature" of our biology—our genetic blueprint.
In a landmark collaboration, health technology leader Verily, in partnership with NVIDIA and the National Institutes of Health (NIH) All of Us Research Program, has unveiled a groundbreaking advancement in artificial intelligence: the first multimodal foundation model that successfully integrates EHR and genomic data. This development marks a critical step toward a future where disease prediction is not just reactive, but deeply, biologically personalized.
Main Facts: The Emergence of Forecast™ 1.0
The core of this innovation is Forecast™ 1.0, a multimodal foundation model now available to the global research community via GitHub. Unlike traditional AI models that rely on isolated data silos, Forecast 1.0 is engineered to synthesize disparate data structures. It bridges the gap between static genetic markers—the inherited risk factors present at birth—and the dynamic, evolving narrative of a patient’s EHR.
The technical achievement is significant. By training the model on the vast, longitudinal datasets provided by the All of Us Research Program, the team at Verily has demonstrated that AI can identify complex disease risk patterns that were previously invisible to clinicians. The model’s ability to process these high-dimensional data inputs—powered by NVIDIA’s advanced GPU infrastructure—allows it to forecast health trajectories over five to ten years, moving beyond the short-term, episodic predictions typical of current healthcare algorithms.
A Chronological Evolution of AI in Healthcare
The journey to this multimodal breakthrough did not happen overnight. It represents the culmination of a multi-year trajectory in health informatics:
- The EHR Era (2010s): Initial machine learning applications focused on cleaning and digitizing clinical data. AI was primarily used for administrative tasks and simple diagnostic support, such as spotting potential medication errors or identifying patients at risk for immediate hospital readmission.
- The Foundation Model Shift (2020–2022): The rise of large-scale foundation models in generative AI proved that "pre-training" on massive, diverse datasets could create models with a high degree of generalizability. Researchers began to apply these architectures to healthcare, recognizing their ability to learn broad patterns before being fine-tuned for specific tasks.
- The Multimodal Integration Phase (2023–Present): With the support of massive initiatives like the NIH’s All of Us Research Program, the data required to train complex, multimodal systems finally became available at scale. Verily and NVIDIA identified this as the "missing link" in precision medicine.
- The Launch of Forecast 1.0 (2024): The successful integration of genomic risk scores with clinical history marks the transition from theoretical research to an actionable, open-source tool for the global research community.
Supporting Data: Efficiency and Accuracy
The performance metrics of the Forecast 1.0 model underscore the efficacy of the multimodal approach. In testing, the researchers focused on Type 2 diabetes, a condition where both genetics and lifestyle play critical roles.
Enhanced Predictive Accuracy
When genomic data was integrated into the model, the performance metrics improved substantially. The model demonstrated a higher sensitivity and specificity, correctly identifying a larger proportion of at-risk patients while simultaneously reducing the rate of "false alarms" that plague clinical workflows. By projecting risk over a five-to-ten-year window, the model provides clinicians with a meaningful timeline to implement preventative interventions.
Computational Efficiency
The project faced significant hurdles in processing power. Integrating static genetic data with dynamic clinical histories creates a massive computational load that standard CPU-based systems cannot handle. Leveraging NVIDIA’s H100 GPUs, the team achieved a threefold increase in pre-training efficiency compared to standard frameworks like Hugging Face Accelerate. This acceleration is not merely a technical footnote; it represents a faster "time to insight," allowing research teams to iterate, experiment, and refine models in a fraction of the time previously required.
Official Perspectives: The Vision for Personalized Health
Jonathan Amar, senior manager of Verily Data Science, emphasizes that this project is a proof-of-concept for a much larger architectural shift in medical AI.
"Electronic health records capture an important piece of a patient’s story, but they’re only one chapter," says Amar. "To truly understand disease risk, AI needs to learn from multiple dimensions of human health, including both our biology at birth and our lived clinical experience."
Amar highlights that the primary challenge has long been the "mismatched" nature of the data. EHR data is messy, sparse, and time-dependent, while genomic data is structured and static. The success of the Verily-NVIDIA partnership lies in their ability to contextualize these two distinct data streams within a single, unified mathematical framework.
"By bringing together two types of typically mismatched data, we demonstrated how multimodal AI can improve disease prediction," Amar noted. "This provides a preview of what’s possible in areas such as pharmaceutical biomarker discovery, diagnostic risk stratification, and health system precision medicine programs."
Implications: The Future of Precision Medicine
The implications of the Forecast 1.0 framework extend far beyond diabetes. This project serves as a "blueprint" for the future of healthcare technology, setting the stage for several critical advancements.
1. The "Total Health" Portrait
The ultimate goal of multimodal AI is to incorporate every facet of a patient’s health, including medical imaging, wearables data (e.g., heart rate variability, sleep patterns), patient-reported outcomes, and even unstructured clinician notes. As these datasets continue to expand, models like Forecast will evolve to capture an increasingly high-resolution, dynamic picture of patient health.
2. Democratizing Research
By releasing Forecast 1.0 via GitHub, Verily is lowering the barrier to entry for research institutions worldwide. By providing access within secure Trusted Research Environments (TREs), the project ensures that sensitive data remains protected while enabling a decentralized, collaborative approach to solving the world’s most complex medical mysteries.
3. Precision Drug Discovery
For the pharmaceutical industry, this technology could revolutionize biomarker discovery. If a model can identify the specific genetic and clinical signatures that lead to disease in a specific patient subgroup, drug developers can design clinical trials that target those exact populations. This would significantly reduce the failure rates of new drugs and accelerate the path to market for targeted therapies.
4. Shifting to Proactive Health
Perhaps the most profound implication is the shift in the healthcare paradigm. Current medicine is largely designed to treat the "sick." A multimodal foundation model, however, enables a proactive approach. By identifying risk years before clinical symptoms manifest, healthcare systems can move from "sick care" to "preventative care," potentially saving millions of lives and reducing the economic burden on healthcare infrastructure.
Conclusion: A New Chapter in Human Health
The integration of EHR and genomic data via the Forecast 1.0 platform is more than a technical milestone; it is a fundamental re-imagining of how we document and understand the human condition. As we move forward, the success of this multimodal approach will depend on the continued collaboration between government initiatives like the NIH, industrial leaders like NVIDIA and Verily, and the global scientific community.
The barriers between data modalities are finally beginning to dissolve. As these systems become more sophisticated, we approach a future where a patient’s health is no longer a collection of disconnected records, but a continuous, understandable, and actionable story. The blueprint has been drawn; the era of multimodal precision medicine has officially begun.
For those interested in the technical methodology and the specific architectural breakthroughs of the Forecast project, the full whitepaper detailing the integration process and performance validation is available through the Verily Workbench.
