John Wise, who established Gadsby's Tavern, built this attractive late-Georgian brick edifice as his residence in 1797. In the early 19C a succession of prominent Alexandrians lived in the house, including Benjamin Hallowell, a Quaker schoolmaster who briefly tutored Robert E. Lee. In 1832 the house was purchased by John Lloyd, whose wife, Ann Lee Lloyd, was a first cousin of Robert E. Lee. The house remained in the Lloyd family until 1918. In 1976 it became part of the Alexandria library system.