Robert Spencer was born in 1879 in Nebraska, the son of a minister who moved from parish to parish throughout Spencer's childhood. After studying medicine briefly, he decided to become an artist and moved to New York City, where he enrolled at the National Academy of Design in 1899. Later he studied with William Merritt Chase and Robert Henri at the New York School of Art. He moved to New Hope, Bucks County in 1906, and studied privately with the well-known Bucks County painter Daniel Garber. It was at the home of painter William L. Lathrop that Spencer met his future wife, Margaret Fulton, herself an accomplished artist and architect. In 1914, the same year they married, Spencer received two gold medals: the Jennie Sesnan Gold Medal from the Pennsylvania Academy of the Fine Arts and the Inness Gold Medal from the National Academy.

For the next 25 years Spencer lived and worked in Bucks County. By 1916 Spencer joined the exhibiting group of local landscape painters known as the New Hope Group. His work differed stylistically from his colleagues and was often somber in mood and palette. His canvases typically depicted buildings and scenes of common people accomplishing everyday tasks. As early as 1910 he was painting local tenements along the Delaware Canal. In the late 1910s and early 1920s, prior to his first trip to Europe, Spencer painted a number of European scenes having no factual basis.

Spencer suffered several nervous breakdowns in the 1920s, and in 1931 took his own life. Today his works hang in many private collections as well as national museums, including the Metropolitan Museum of Art and the James A. Michener Art Museum.