The Newton D. Baker House was built in 1794 by architect Thomas Beall. The house reflects a New England style of architecture. At five stories high, the home includes six bedrooms, three fire places, and a library. Newton D. Baker, for who the house is named, lived in the house from 1916 to 1920. The house was also occupied by Jackie Kennedy after John F. Kennedy was assassinated in 1963. It was listed as a National Historic Landmark in 1976.

Dumbarton Oaks includes the historic Dumbarton Oaks House museum and its associated formal garden, a 27-acre wilderness area, and Harvard University's Dumbarton Oaks Research Library. Founded by Robert Woods Bliss and Mildred Barnes Bliss, the Dumbarton Oaks Research Library and Collection specializes in Byzantine, Garden and Landscape, and Pre-Columbian studies, and includes not only books, but also images, art, objects, and documents.