The 1860 census lists him as James H. Fish.