Joseph worked in the War Department, in Washington, in 1810.
Joseph became and Indian Agent and worked until after the War of 1812 ended. Joseph then worked for John Jacob Astor in the fur trade. He later became a dry goods merchant in New York in partnership with Graham, Bebb, and Graham.His other interests included banks, railroads, and disposing of large tracts of land he had received in payment for debts.
Joseph's father is quiet notable, as a U. S. Senator from Massachusetts, and the 7th Speaker of the House of Representatives.
Joseph and his wife Mary Graham Varnum, were originally interred in the Varnum vault at The Mable Cemetery, in New York City.