The library, or office, is missing two key elements: a collection of books and a billiards table. Knox, being mostly self-educated, was a great lover of books and he had a sizable collection of almost 1,600 volumes. After his death most of his books were given to the Boston Athenaeum, where they now reside.

We know from Knox's financial papers at the Maine Historical Society that he owned a billiards table made by Benjamin Frothingham, Jr., who had served under him in the Revolutionary War. The probate inventory taken after his death places the table and the books in the "office," implying that the library was used by the gentlemen for after-dinner diversions.