The house is the original English home of the painting The Toilet of Venus by Diego Velázquez, now known in English as The Rokeby Venus. The original now hangs in the National Gallery, London and a copy in the saloon at Rokeby Park. Sir Walter Scott was a regular visitor to the house and used it as the setting for his narrative poem Rokeby in 1812.
The building is of interest in its own right. Completed in 1735 (and known at the time as Rokeby Hall) by Sir Thomas Robinson, it is considered a fine example of the italianate Palladian style. Robinson owned it until 1769, when he sold it to JS Morritt, an ancestor of the current owner.
Much of the present interior was designed by John Carr. The house contains the collection of fine needlework pictures by Anne Morritt (1726–1797), the unmarried sister of JS Morritt. There is also a rare surviving print room wallpapered with 18th-century prints.