Main Street, from Municipal Building, 2004

Sag Harbor was back up the Bay, around past Cedar Island Light, a foreign place to us. The last whaler had sailed from there seventeen years before. People were talking about a revival, but none of them really believed it. Sag Harbor captains, most of them really from Bridgehampton or Southampton, still commanded whalers out of San Francisco or Honolulu for the Arctic fishery. You could still see Kanakas, and Indians, and Chinese too, on Sag Harbor Main Street, and you could see saloons and such social eyesores. Sag Harbor had Irish, too, working in the watchcase factory, and a Catholic church, a candle shop. Sag Harbor made me nervous.

Everett T. Rattray, The Adventures of Jeremiah Dimon: A Novel of Old East Hampton (1985)