Brown was born in Greensboro, North Carolina, and was nicknamed "Skinny" by his parents because he was a chubby child.[4] Brown weighed 180 pounds (82 kg) and stood 6 feet 2 inches (1.88 m) tall during his active career. He was 26 years old when the White Sox purchased his contract from the Triple-ASeattle Rainiers and he made his major league debut with the team on April 19, 1951. He spent two years with the ChiSox before moving to the Red Sox, the team that had originally signed him to a pro contract in 1946. In 1953, Brown went 11–6 in 25 starts. He joined the Orioles in July 1955, winning 34 games for them from 1956 to 1959. In 1960, he compiled a 12–5 mark with a career-low 3.06 ERA for a contending Baltimore team that finished second in the American League. The next year, he went 10–6 and 3.19 He was sold to the pennant-bound Yankees in September 1962.
Brown worked in two late-season games for the Yankees, and was ineligible for the Bombers' 1962 World Series roster. He was purchased by the Colt .45s at the outset of the 1963 season. It was the third time that Houston general managerPaul Richards, who managed Brown in Seattle in 1950, had acquired the right-handed pitcher — he had done so in 1951 when Richards managed the White Sox and in 1955 when he was both general manager and field manager with the Orioles.
With Houston in 1963, Brown was a victim of poor run support, as he walked just eight batters in 141 innings and posted a 3.31 ERA, but tallied a 5–11 record. In 1964, his last major league season, he finished 3–15 with a 3.95 ERA.
