Nathaniel was the son of Mary Lincoln and Nathaniel Crosby. His father, a seafaring captain, died in Hong Kong and is buried there. Nathaniel also took to the sea and was a purser on a steamer. He later operated a book store in Olympia.
Nathaniel and his wife, Cordelia, had two sons: Frank and Harry. Harry was the father of famed singer/actor Harry Lillis "Bing" Crosby.