An early goal of the Espy Foundation was to establish and maintain in Oysterville a library to house Espy's personal collection of reference books and his own works, totaling some 700 volumes. Work began on the library in the fall of 1999, and the small building was dedicated in April, 2000. The intimate and cozy facility provides an inspiring work space for writers and artists in the Residency Program. The collection is a valuable resource for research. The library is also open to the general public by appointment.

The collection consists largely of books on the lighter side of language: word-play and riddles, anagrams, puns and palindromes, tongue-twisters, slang, curious word orgins, limericks, and collections of light verse. Its many reference volumes range from general to specialized dictionaries, thesauruses, quotation anthologies, and readers' and writers' references. To celebrate the opening of the Library, Merriam-Webster donated a comprehensive set of its language related titles, nearly 50 volumes in print and on CD-ROM. The Library also includes a growing shelf of Southwest Washington history.

In 2001, the Library was enormously enriched with writer Paul Dickson's donation of 2,000 volumes, devoted primarily to American English. Mr. Dickson believes that the Espy Library may well become a major source for books on colloquial English usage and American humorous language, noting that the holdings of the Library of Congress in the area of "the rougher language" (the jargon and humor of the G.I., the logging camp or prison yard, for example) are relatively limited.

Theodor Schuchat, author of "The Library Book", donated a number of volumes to the Library, which are a welcome complement to the Library's collection.

Due to current space restrictions, Mr. Dickson's collection and the collections donated by Theodor Schuchat and James Townsend remain in storage, awaiting a planned larger library.

In July 2007, the Espy Foundation received word of three pending major book donations from Paul Dickson, who is contributing the rest of his collection; Laurence Urdang, of Verbatim Books; and John Kolars, who will donate his entire personal poetry library.  All books in the Espy collection are currently being entered into an online catalog so scholars from around the world may access this rare collection.