The Colored Orphan Asylum was an institution in New York City open from 1836-1946 that housed on average four hundred children annually and was mostly managed by women. Its first location was on Fifth Avenue between 42nd and 43rd Street, a four-story building with two wings.
The Colored Orphan Asylum was founded in 1836 by three Quakers; Anna and Hanna Shotwell and Mary Lindley Murray. It was one of the first of its kind in the United States to take in black children whose parents had died, or were not able to take care of them. Prior to its founding, orphaned black children were housed in jails or worked as beggars or chimney sweeps as orphanages refused to take them. The orphanage initially offered schooling only for infants, feeling that their wards would not advance far in society due to being Black and orphans. Older children were bound by indentured servitude in which they were contracted to families, both Black and white, to learn a trade or skill until age 21. The families, in turn, paid a small fee to the Colored Orphan Asylum for the services which were placed in the bank for when the child left the institution. By 1897, schooling was increased until grade six and sent several students to the Hampton Institute for further study. In 1918 schooling was increased until grade eight and the indenture system evolved into a loose foster care system in which the child was to be incorporated into the family and continue their studies. In 1846 Dr. James McCune Smith, the country's first licensed African American medical doctor, became the orphanage's medical director. The orphanage moved several times in Manhattan.
In March 1863, conscription in the United States became stricter, and the federal government used a lottery system to choose citizens for the draft. Those chosen could hire a substitute or pay the government, but most working-class men could not afford substitution, while black men were ineligible for the draft (they were not considered citizens of the United States at the time). Working-class white males, furious about the federal draft laws, rioted and attacked federal buildings and black neighborhoods. The Colored Orphan Asylum was burned down by Irish mobs on July 13, 1863, during the first day of the New York Draft Riots. A policeman was killed while leading the children out the back door to escape.
The asylum was rebuilt by the Quakers in 1867 on 143rd Street and Amsterdam Avenue. The operation moved to a new building in 1907, in Riverdale, Bronx.  At the new site, the orphanage adopted a new plan to house its wards in cottages, with 25 children and a housemother in each cottage. The new plan was received favorably and encouraged ownership and self respect in the children. In 1910, the asylum purchased a farm in Duchess County for boys to learn practical skills.