Marblehead was a comparatively insignificant port when Jeremiah Lee came to town. At once he made a place for himself among the humble fishermen and other seafaring men of the place. He was a member of the Board of Firewards in the town’s first fire department, and he served on important committees.

When, in 1768, he built a wonderful mansion that cost more than ten thousand pounds, the most wonderful house in Massachusetts at the time, his townsmen knew him well enough to understand that he was their good friend, even if he did have much more money than any of them.

The Lee Mansion was a hospitable home. The Colonel and his wife Martha entertained lavishly, not only the people of the town but famous men from abroad. In 1789 Washington was entertained in the house. But it was one of the glories of the mansion that the humblest mariner in the place was not slow to go there if he wished to have a chat with the bluff owner or if he desired to go to the quaint cupola from which it is possible to look far out to sea. To this outlook Colonel Lee himself often went, for his ships were sailing to Marblehead from all parts of the world, and he was as eager as anyone to turn his eyes seaward.

The house is sixty-four feet by forty-six feet, and the walls are of brick, though they are covered with wooden clapboards two feet by one and a half feet. There are fifteen rooms, in addition to the great halls that make the house seem like a palace.