by Stuart Snydman, Associate University Librarian and Managing Director, Library Technology Services, Harvard University Information Technology
On Monday, September 15, Harvard Library and Harvard University Information Technology (HUIT) launched the MVP of Collections Explorer, the first working prototype of the Reimagining Discovery initiative. This marks a pivotal moment: for the first time, researchers, students, and the broader public can explore Harvard’s special and archival collections using natural language search powered by semantic retrieval and generative AI.
Library Discovery Reimagined
Unlike HOLLIS or HOLLIS for Archival Discovery, Collections Explorer does not require users to know the structure of library metadata or the names of repositories. A student might simply type, “What is the history of the MRI?” and the system will return relevant collection records—even if the metadata does not include those exact words. Results can be narrowed through filters for digitized items, date ranges, repository (“location”), and resource type (“collection” vs. “item”).
Three AI-driven features help guide exploration:
- Summaries of results, generated on demand, explain why an item is relevant.
- Suggested queries encourage users to branch into new directions.
- A “Try this in HOLLIS” option transforms natural language into a keyword search, preserving interoperability with other systems.
The design reflects months of user research: students and researchers told us they want more transparency (“About this material”), recommendations for next steps, and easier entry points when they don’t know how to begin.
What’s Inside Today
The MVP is intentionally constrained: it draws exclusively on archival descriptions, primarily finding aids published through ArchivesSpace and LibraryCloud. This represents nearly 13,000 finding aids and about 2.9 million child-level records, of which roughly 195,000 include digitized links that can be accessed directly.
While substantial, this is only one slice of Harvard’s distinctive collections. The largest contributors are Houghton (nearly a million items), Schlesinger (~500,000), and Baker (~320,000) libraries. Metadata is drawn from ArchivesSpace at the collection level and LibraryCloud at the item level, then merged into a single semantic index. Future phases will expand coverage to include rare books, images, full text, open access publications, research data, and born-digital materials—making the system far more expansive.
How It Works
At its heart, the system uses “hybrid search,” which combines keyword search with semantic search to capture meaning, allowing interpretation of natural language questions. Keyword search guarantees precision when a user looks for an exact title or identifier. Semantic search, by contrast, retrieves items with similar meaning, even if the wording differs. Each method produces a score, and the highest-scoring documents rise to the top. This hybrid approach ensures that deterministic matches are honored while still unlocking serendipitous discovery.
Semantic search is powered by open-source embedding models (currently Cohere multilingual v3), while generative AI features run through AWS Bedrock, which provides access to models such as Mistral and Claude. Bedrock also offers Guardrails for content safeguards, giving us additional tools to manage safety and reliability.
The underlying architecture has been deliberately built for scale. The first prototype relied on an in-memory FAISS index; production now uses Elasticsearch as the dedicated vector database to support millions of records with true metadata filtering. A new data pipeline coordinates ingest from LibraryCloud, tracks status in MongoDB, and uses RabbitMQ and Celery to update the index continuously.
Collections Explorer has been built with a commitment to responsible and ethical use of AI. Users are informed when summaries are generated by AI, and a dedicated “How this System Works” page explains the underlying technologies and safeguards. To foster accountability and improvement, the site also offers an embedded survey and feedback tools, enabling an ongoing feedback loop.
Lessons and Challenges
Bringing archival metadata into this environment has surfaced longstanding issues. Dates required extensive normalization, with support even for BCE dates. Broken or inconsistent identifiers, especially around digitized links, are common; the team has adopted a principle of pushing data quality fixes back upstream rather than “engineering around” these inconsistencies and errors.
Semantic search, by design, is not fully deterministic. The same vague query may yield slightly different results across runs. For exploratory research, this variability can be a feature, offering serendipity. But for known-item searching, it underscores the importance of retaining keyword matching alongside AI methods.
AI-based systems are also imperfect: at times they can generate odd or even misleading summaries or suggested queries. The hybrid design of Collections Explorer helps mitigate this, and the team has worked hard to minimize such results, but they can and will surface. Importantly, these moments are opportunities to learn more about both the technology and user expectations.
Finally, user feedback revealed that while AI summaries are valuable to many, others prefer to interpret records themselves or to minimize environmental costs of LLM calls. A “Hide Summaries” toggle was added to honor that choice.
Strategic Significance
Collections Explorer is more than a new interface: it is a proof of concept for the future of library discovery. It shows that Harvard can begin to integrate AI into scholarly infrastructure in a way that emphasizes trust, transparency, and rigor.
By starting with archival description, the team deliberately tackled one of the most challenging data types—deeply hierarchical, uneven, and complex metadata. This constrained scope allowed us to test the model in a demanding environment and learn what works well and where limits remain.
The MVP therefore serves two roles. It contributes to the broader field of library innovation, alongside similar experiments at other institutions, by offering a production system that is openly documented, modular, and designed to evolve. It also lays the groundwork for expansion. As new data sources are added—rare books, images, open access publications, research data, born-digital and geospatial materials—we expect the system’s reach and impact to grow. How effective it proves to be in practice will depend on ongoing use, feedback, and iteration.
The Road Ahead
The MVP is a beginning. Next steps include:
- Adding more data sources—rare books from Alma, digitized images from JSTOR Forum and Harvard Art Museums, and items represented in CURIOSity collections.
- Integrating HarvardKey authentication to expose restricted items and allow user features like saved searches and bookmarks.
- Iterating on relevance tuning, especially for non-English queries, historical events, and geographic searches.
- Scaling to full-text, open access publications, research data, born-digital, AV, and geospatial materials in years two and three.
Conclusion
The launch of Collections Explorer represents a vision for library discovery where anyone can explore Harvard’s extraordinary collections with the ease of natural language, supported by the precision of metadata and the context of AI. At the same time, important questions remain about its value for key audiences, especially researchers, instructors, and students. Will semantic search truly support scholarly workflows and teaching more effectively than traditional keyword and facet-based search? Will it change how collections are used in research and classroom contexts?
The coming months will provide an opportunity to learn how different users engage with the system, where it adds value, and where it needs refinement. In that sense, this launch is not only about what has been built, but about opening the door to test, adapt, and advance toward new pathways to knowledge.