James married married Julia Matilda Graham in 1833. They purchased their home at 21 Washington Square in 1837. Although they had no children of their own, James and Julia played a huge roll in the Graham family, always opening their home to family members, some of whom stayed for long periods of time. Their home was lavishly furnished with art by the old masters, Van Dyke, Ruebens, Rembrandt and others.
James was an attorney, very successful and became the president of the Metropolitan Fire Insurance Company.