The circular sandstone fort was named in honor of the sixth Governor of New York, Dewitt Clinton, who held office in the beginning of 19th century. The castle was built between 1808 and 1811 on a small artificial island located several meters off shore. Originally known as the West Battery, it was supposed to serve as a fortified emplacement for cannons in the War of 1812. However, the war somehow bypassed New York City; therefore the castle never saw any military action. When more landfills were made to expand what became known as Battery Park, the castle was integrated into the park.
The American Army ceased using the fort in 1821 and leased it to the city administration, which transformed it into a place of public entertainment, calling it Castle Garden. The fort became a garden, a promenade, a restaurant, an exhibition space, an opera house and a theater, all without a roof. All the major 19th century dancers, actors and opera singers, American and European, performed on the open-air stage.
In 1855 the castle became a landing ground for European emigrants. There they underwent police checks and medical exams. Over eight million foreigners had passed through the premises before the immigration station was moved to bigger facilities on Ellis Island in the bay in front of the castle.
At the end of 19th century Castle Garden was turned into the New York City Aquarium, and so it remained until 1941, when it was finally closed for entertainment purposes. After World War 2nd it became a national monument.