Located on Sixty-first Street off Madison Avenue, The Colony was founded in 1919 by Joseph Pani,, who sold it to employees Ernest Cerutti, Alfred Hartmann, and Gene Cavallero, Sr in 1922. At first, it was known for attracting playboys trolling for dates. The club featured a lesser known upstairs gambling club where men would often meet their mistresses; however, after Mrs. William K. Vanderbilt discovered it, the room became the fashionable haunt of New York high society. Mayor Jimmy Walker's victory celebration was held at the Colony in 1925.
The Colony served liquor during prohibition, serving it in cups rather than glasses, and keeping its liquor in a service elevator where it could easily be moved, though Mayor Walker protected the restaurant from raids. It was the first restaurant in New York to have air conditioning, which was installed in the late 1920s. The Colony became the first establishment in the U.S. to serve Dom Pérignon champagne.Sirio Maccioni was the bar captain at the Colony from 1960 to 1970.
Among its noted customers were Groucho Marx, Dick Cavett,the Vanderbilts, Preston Sturges, Mike Todd, Fulco di Verdura, Hattie Carnegie, Carmel Snow, Kitty Carlisle Hart, Elsie de Wolfe, Groucho Marx, Mrs. Irving Berlin, Millicent Rogers, Barbara Hutton, Doris Duke, Betsey Whitney, George Vanderbilt, Samuel Newhouse, Marlene Dietrich, Lucius Beebe, Rosalind Russell, Gary Cooper, Carol Channing, Richard Nixon, C.Z. Guest, Ernest Hemingway, Luis Miguel Dominguin, Walter Wanger, John Ringling North, Frank Sinatra, Aristotle Onassis, the Duke of Windsor, the Duchess of Windsor, Merle Oberon, Vincent Astor, Elsa Maxwell, Rex Harrison, Richard Widmark, Errol Flynn, Babe Paley, Frank Shields, Frank Costello, J. Edgar Hoover, Orson Welles, Gilbert Miller, Joan Crawford, Mia Farrow, Serge Obolensky, Charles Revson, Leona Helmsley, John Wayne, Oleg Cassini, Grace Kelly, Jean Howard, Clarence Brown, Walter Wanger, Jacqueline Kennedy Onassis, Lee Radziwill, Gloria Guinness, Betsy Bloomingdale, Amanda Burden, Jean Shrimpton, Jill St. John, Truman Capote, and Billy Baldwin.
When it closed on December 4, 1971, many of its faithful patrons attended. The building which housed the enterprise has since been demolished.