Prior to the Duncans, the Saltonstall family lived at this site for generations, from the 1660s to the American Revolution. Local legend states that Richard Saltonstall, the head of the family at that time, was loyal to the Crown while most of the Haverhill residents were in support of the Revolution. They threatened to burn down the Saltonstall house with the family inside until a quick-thinking Richard sent money down to the tavern. The mob ate and drank and forgot about their destructive plans while the Saltonstalls were able to slip away. Similar stories are recorded from Greater Boston during the years leading up to the war. Following the American Revolution, this property sat vacant until it was purchased by the Duncan family in 1814. It is unclear whether anything remained of the Saltonstall Seat, as the Saltonstall's had called it, by that time, or whether it had been demolished years earlier. Regardless, the Duncans set about building a new structure at the site.

The house was built to be a wedding present for Samual White Duncan and his bride, Mary White, from his father, James Duncan, Jr. It is built in a rural Federal style, meaning that it is symmetrical. There is a central hallway on the first and second floors and the same number of rooms on the left as the right and the front and the back. There are also an equal number of windows on either side of the front door.

The Duncan family had made their money as merchants. They owned ships that traveled the world bringing goods from far away, such as Cantonware from China and Staffordshire ware from England, examples of which are displayed inside the house in the dining room. The Duncans owned flat-bottomed cargo boats, called gundalows, that traveled up the Merrimack River bringing goods offloaded from larger ships in the deeper ports of Newburyport and Salem. They also owned stagecoaches that traveled from Haverhill up to Londonderry, New Hampshire and dry-good stores located along the way.

Samuel and Mary moved into the house shortly after their marriage in 1814. They never had any children and Samuel died from consumption fourteen years later. Mary never remarried and stayed in this house until her own death forty years later. She was not lonely in her widowhood, hosting many parties, holidays, and wedding celebrations for her nieces and nephews. 

After Mary's death, because she had no children, the house went to her husband's brother James H. Duncan. James owned the Eagle House Hotel, started the prestigious Pentucket Club, and had been a Massachusetts Representative in Congress for two terms. He died a year after Mary and the house went to his children, who added the addition at the back of the house in the 1870s. One of James' children, Mary Duncan Harris, bequeathed the house to the Haverhill Historical Society. The family had called the home the Buttonwoods Estate after the sycamore—also known as buttonwood—trees planted on the front of the property by Haverhill's first Irish immigrant, Hugh Tallant, in the 1730s, while the site was still under the Saltonstall family's care. The Society was formed in 1897 and moved into the house in 1903, opening the Buttonwoods Museum there.

Buttonwoods Museum. Tour Script, March 1st 2018. Accessed August 18th 2020.