During Philipse's life, the building was used primarily as a stopover point on the long journey up and down the river between his home in New Amsterdam and the northern parts of his estate. His grandson, Frederick Philipse II, the second Lord, and his great-grandson, Frederick Philipse III the last, successively enlarged and enhanced the building, making it the primary family residence.
On November 28, 1776, nearly five months after the signing of the Declaration of Independence and the start of the American Revolution, Frederick Philipse III and over 200 of his contemporaries signed a document declaring their allegiance to the British Crown and their unwillingness to support the Revolutionary cause. Because of his Loyalism, Philipse was branded a traitor and placed under arrest on orders signed by General George Washington. He was held in Connecticut for a time, but was given special permission to travel back to Yonkers to settle his affairs on the condition he was not to aid the British cause. In violation of his parole, he and his family fled to British-occupied New York City and later to Great Britain, leaving their estate and Philipse Manor Hall behind to be attainted in 1779.
Philipse family holdings, which included the Philipse Patent, a 250 square mile tract that became today's Putnam and part of Dutchess counties, were sold at public auction by New York's Commissioners of Forfeitures during the Revolution. Philipse Manor Hall was occupied by various families throughout the 19th century. In 1868, the building became Yonkers' municipal center (as Village Hall, and later, as City Hall) and remained such until 1908. During this period, an elaborate monument to those Yonkers natives who had died during the American Civil War was installed on the east lawn (1891).
By 1908, the growing complexity of city government had made the building nearly obsolete as a government center. Public meetings were held, and options such as adding wings onto the building and tearing it down outright were discussed. The question became moot when Eva Smith Cochran, matriarch of a wealthy local carpet milling family, stepped in and donated $50,000 to the city as a nominal reimbursement for their care of the building during the previous 40 years. This allowed the city to turn ownership of the building over to the State of New York. Between that time and the 1960s, the building was owned by the state but cared for by the American Scenic and Historic Preservation Society. Since the dissolution of the Society, the building is owned, maintained and curated by the New York State Office of Parks, Recreation and Historic Preservation.
From 1911 to 1912, the most intense restoration project in the building's history brought the house back to a semblance of its colonial appearance. The building has been open as a museum of history, art and architecture since 1912.
The house is home to a ca. 1750 papier-mâché and plaster Rococo ceiling, one of two in-situ ceilings of its type in the United States. The elaborate ceiling is covered in designs and motifs relevant to Frederick Philipse III's lifestyle. For example, his love of music is represented by lute players, bagpipers and singers; his enthusiasm for hunting is represented by hunting dogs and game birds; and his education in the arts and sciences is represented by busts of Alexander Pope and Sir Isaac Newton.
Also of architectural significance is the 1868 City Council Chamber, designed by John Davis Hatch. The Chamber's high, vaulted ceiling and woodwork are intentionally reminiscent of a typical English manor house's great hall.
Throughout the house are paintings from the Cochran Collection of American Portraiture. This collection was put together by agents of Alexander Smith Cochran (son of Eva Smith Cochran and owner of the family's carpet mills) and features works by Charles Willson Peale and John Trumbull. Represented among the 60 paintings are nearly all of the presidents of the United States from Washington to Calvin Coolidge, as well as war heroes, historical figures, and members of the Philipse family.
^ ab"Philipse Manor Hall". National Historic Landmark summary listing. National Park Service. September 18, 2007. Archived from the original on July 21, 2011.