East Hampton in the early 19th century was lightly populated and one of the biggest slave holders were descendants of Lion Gardiner who owned Gardiner's Island. During the War of 1812 slaves from the island were used to ferry supplies back and forth because they could slip through British lines because it was "obvious" that they were "owned."
After slavery was abolished in New York State in 1827, freed slaves were relocated to small houses just north of East Hampton (village), New York. An 1858 Chace Map shows only 13 structures in town 
There is little direct connection to the neighborhood now. One of the last structures from the town is the St. Matthews Chapel which was moved in 1976 to what is now East Hampton Pointe.