In 1907, Brownson was assigned as Chief of the Bureau of Navigation, which handled the U.S. Navy's personnel matters. In July 1907, he was transferred to the retired list, but continued to serve in his post at the Bureau of Navigation until December 1907. That month, President Theodore Roosevelt decided to give command of hospital ships to U.S. Navy doctors, against the advice of Brownson, who then resigned from the Navy. A storm of protest arose from within the Navy and from the public, but Brownson's active naval career was over.
Brownson married Isabella Robinson Roberts in July 1872. Their first child, Henry was born in 1874 and died in 1876 shortly after his father left for the Asiatic Station. The following spring his daughter Harriet was born. In 1878 another son, Roswell, was born and in 1894 a second daughter, Caroline Robinson, was born. Caroline Brownson married Thomas C. Hart, later an admiral and the last Commander-in-Chief of the Asiatic Fleet. Harriet married Admiral Charles Lincoln Hussey, who as an ensign received the Navy Cross during the Spanish–American War.