Jay was raised at Manhattan, New York where his father was a stock broker. He was the son of Jay Gould, II and Anne Douglas Graham Gould. Jay attended University of California at Los Angeles, California and then served in the military during World War II from 1942 to 1945. Jay married first to Jennifer Beryl Bruce on 21 September 1944 at Beverly Hills, California. He married next to Joan Elizabeth Blair Roemer on 27 November 1948 at Westwood, California. Jay married last to Elena Maria “Lina” Romay on 30 June 1953 at Los Angeles County, California. He had one child each by his first two marriages and three children with his third wife.