Dead Castles — Exploring the Ruins of the Fortress That Abercrombie & Fitch Built

I’ve been traveling for a long time, and if there’s one thing I’ve learned, it’s to be skeptical about any place that is advertised as being “off the beaten path.” Sure, that is what I’m often looking for—the quieter, locals-frequented spots, not the tourist-clogged restaurants and museums. But I also know that the path has been pretty much totally beaten at this point, that Google Maps and Yelp and Instagram have conspired to ruin the charm of anything that was once mysterious or out-of-the-way. That all of those signs that read “Adventure awaits!” in some cutesy cursive font probably mean that, for a bunch of money, you can live out your wildest Ernest Hemingway fantasies in a safe and controlled environment.

What I want is to take a weird route to get somewhere not everybody knows about. I want the odd, the uncertain, even the uncomfortable. And that’s why, on an early winter’s morning as the temperatures reached a number high enough to melt the latest snowfall, I got in my car with my friend Isaac—who was going through a phase where he dressed like WWE legend The Undertaker, for reasons I still don’t quite understand—and drove north from Brooklyn to look for an abandoned castle in the woods. 

There aren’t many castles on the East Coast. Few of them are just sitting there lonely and derelict, magnets for local teens to hang out in and practice tagging and drink whatever booze they swiped from their parents. And there are even fewer abandoned castles that once belonged to David Abercrombie, the co-founder of Abercrombie & Fitch. 

I was never someone who wore Abercrombie & Fitch. In high school, my group of friends openly despised the preppy, fratty kids who did, one front in the bloody battleground that is high school society. But I was still fascinated with the brand. My grandpa had some pieces from way before the company became all softcore, black-and-white Bruce Weber photos of young people barely dressed in anything. I always looked for Abercrombie & Fitch items in antique shops or at tag sales, whether old cans of waterproof dressings or ashtrays.

So when I learned that David Abercrombie had a castle in Westchester County that he called Elda—an acronym of the first letters of the names of his four kids—I knew I had to see it. Part of the appeal was obvious: It’s an abandoned castle in the woods. But there is something about trespassing, about going on land I’m not supposed to walk on, land that’s just sitting there, that’s so satisfying. That, and this particular land was once owned by a man who looked at the outdoors—something that shouldn’t cost anything to enjoy—and figured out how to profit from it. The only way to do it was to trespass. 

THIS LAND IS OUR LAND

When I was a kid, I took part in a socialist summer camp singalong program where we learned a bunch of old songs originally sung by striking workers and later by generations of wannabe Bob Dylans. “If I had a Hammer”; a few songs about brothers and sisters tied together by toil and struggle; a take on “The Hearse Song” that I loved which replaced the worms crawling in and out with scab workers. But my favorite, and the most popular overall, was “This Land Is Your Land.” Everybody knew that one. 

For me and my peers—born on the Gen X-Millennial cusp—the ’60s didn’t look that far in the rearview mirror. Our minds were greenwashed pretty early on. If we gave a hoot and didn’t pollute, if we followed the three Rs (reducing, reusing, and recycling), the planet would be OK. We figured our politicians could be trusted, and that big corporations could be held accountable. 

“But there is something about trespassing, about going on land I’m not supposed to walk on, land that’s just sitting there, that’s so satisfying. That, and this particular land was once owned by a man who looked at the outdoors—something that shouldn’t cost anything to enjoy—and figured out how to profit from it. The only way to do it was to trespass.”

Soon enough, I started to figure out that a lot of what I was being taught didn’t exactly add up. But certain things made sense to me. Singing “This Land Is Your Land” for the first time with my third-grade teacher, who I remember as always wearing a baggy sweater and long skirt combo, was one of those things. I loved singing about walking past the redwood forest and Gulf Stream waters. I thought I knew the song by heart, but then that summer camp program scrambled my brain. There were other lyrics to “This Land Is Your Land,” I discovered. I listened and learned, specifically focusing on one part:

“Was a big high wall there that tried to stop me

A sign was painted said: ‘Private Property’

But on the back side it didn’t say nothing 

This land was made for you and me”

Huh, I thought. It’s weird that we didn’t learn that lyric when my teacher had us sing the song. And what was with the song’s last line, about the people standing around the welfare office and singing about how “I stood there wondering if God blessed America for me”?

I wasn’t even 10. I didn’t know Woody Guthrie wrote the song as a response to the bombast of Irving Berlin’s “God Bless America,” and even if I did, it wouldn’t have meant much at the time. I didn’t understand much of anything, besides Saturday morning cartoons and comic books. But that line about ignoring the private property sign said something to me. It emboldened me in small ways. Private property? Fences? Do not trespass? Why? I found myself wanting to do the opposite. I wanted to climb the chain-link, I wanted to walk deeper into the forest near my home, and I wanted to see what was on the other side of those signs. 

So that’s what I did, any chance I had. If something looked interesting to me and there was a fence, I’d figure out a way around or over it. If there was a path that said “Do not enter,” I more often than not ignored it. I wasn’t doing anything that dangerous. I wasn’t breaking anything or—like some of my friends—tagging stuff with graffiti. I just wanted to see what I thought I wasn’t supposed to see. 

HOW TO BUILD A CASTLE

David Abercrombie’s biography reads like a Gilded Age “follow your passion” story. His obituary states he was educated in Baltimore’s public schools, but by private tutors. He went to college, and eventually landed a job as a civil engineer and a topographer, though his first love was for the outdoors, mainly fishing and hunting. In 1892, he gave up working for somebody else and opened Abercrombie Co. on South Street, just across from the East River. Today, Abercrombie would look out his window and see the huge overpass leading cars onto FDR Drive, but back then, the area was one of the busiest and most prosperous in the city. It was the right place for Abercrombie to open a store for the “elite sportsman.” 

Ezra Fitch, who was born into a well-to-do New York family and spent his childhood playing along the Hudson River north of the city, was just that sportsman, lured to Abercrombie’s shop because of its status. Abercrombie Co. had marketed itself as a specialist retailer for men of means and good taste, and Fitch saw in it an opportunity to escape the drudgery of his already wealthy existence as a developer. He proposed to Abercrombie that they go into business together. 

“If something looked interesting to me and there was a fence, I’d figure out a way around or over it. If there was a path that said ‘Do not enter,’ I more often than not ignored it. I wasn’t doing anything that dangerous. I wasn’t breaking anything or—like some of my friends—tagging stuff with graffiti. I just wanted to see what I thought I wasn’t supposed to see.”

Abercrombie and Fitch founded their company in 1904. Just three years later, in 1907, Abercrombie was out, seemingly because he didn’t like the direction the company was going in. Fitch wanted bigger and better. He had dreams of A&F being in other cities long before chain stores or malls came and went. Abercrombie, meanwhile, hadn’t lost his desire to provide a place for sportsmen. Not just people who enjoyed nature—America’s most famous adventurers, from Theodore Roosevelt to Hemingway.

The sort of businesses that followed in the wake of David Abercrombie’s success—the L.L. Beans, North Faces, and Patagonias of the world—furthered the idea that you should live a life without limits, that you should explore, go off the beaten path to catch a fish or see the world from a different angle. But before you did that, of course, you had to spend a bunch of money on their sneakers, water bottles, windbreakers, ropes, backpacks, tents, and any other equipment you needed to “rough it” in the wilderness. 

Don’t get me wrong—I subscribe to the buying-a-tent-instead-of-building-one-out-of-whatever-you-find-on-the-forest-floor school of thought. But at some point I started thinking a lot more about how all these companies, no matter their good intentions (or their campaigns about conservation or ethical shopping), are still businesses. They still have to make a profit. They still need you and me to give them money to feel like we can access what is already ours. 

So when my friend and I made our way to Elda Castle, I had one small idea. And that idea was we were simply going on a hike. We were going to break the law by trespassing; we were doing it in broad daylight; and we didn’t really need any “gear” besides some extra bottles of water, a pocket knife, our phones and the hope that—if we needed it—we could get reception to make an emergency call. I took my friend Isaac because he’s down for whatever. I told him I wasn’t looking for anything—I just wanted to see it. The abandoned castle in the woods.

We set out around 6:30 in the morning. The goal was to be there no later than 8. It was a Saturday morning, and we wanted to be as inconspicuous as possible. I figured the people of one of the wealthiest communities in the country were probably lounging around at that hour, not giving much thought to what they had to do on a cold Saturday morning in the winter. We got as close as my GPS told me we’d get to the property without actually being on the property and parked my car. It was about half a mile from there to the first sign that told us to keep out, that we weren’t allowed on the 49.6 acres of land that just sits there with a decaying castle in the middle of it. I figured part of the reason for that was that you had to journey a little through the overgrowth to get to the castle, that the journey was a little treacherous. I was correct, I discovered, as I started sliding down a muddy hill, narrowly missing a bunch of thorny branches.

But we kept walking, using the compasses on our phones and rudimentary outdoor skills to not get too lost. We stepped in a lot of mud holes, had to clear away a lot of branches, and had a few moments where one of us wondered if the sound we heard was a bear, coyote, or, worse—a cop. 

Eventually, after about an hour of walking through hilly, wild terrain, one of us noticed something that didn’t quite fit the wooded landscape. It was a chimney. We’d found a little furnace, the first sign that the castle itself couldn’t be too far away. Nearby was an old wooden bathroom that was still mostly intact. And a few minutes after that, there it was: David Abercrombie’s castle. 

HOW A CASTLE FALLS

As I write this, Elda Castle is once again on the market, with an asking price of just over $3 million. The property has changed ownership a few times, including in 2001, when it sold for $1.5 million. A decade later, in 2011, it went for $3.75 million.

The thing about castles is that they’re built for the ages, for generations to live in. Americans like to move around. We typically see homes as investments, not places our kids and grandkids and their children will be living in, at least not anymore. Walking around the property Abercrombie built, I suspected he had envisioned a future that he knew he wouldn’t be around to see, a hope that his descendants would live on and enjoy the place he created. 

But just as he was starting his life in Ossining, tragedy struck. Lucy, Abercrombie’s oldest daughter and a partner in his company, was killed when a tub filled with paraffin wax and gasoline used for waterproofing exploded. The New York Times report on the accident tells a horrific story of a person being burned alive, and hanging on just long enough for her family to say goodbye. It broke her father. A little over two years later, he also died. In 1937, one of Abercrombie’s sons, David Jr., died after being kicked by a horse on his dude ranch in Idaho. Abercrombie’s widow, not wanting to live alone in a castle, moved in with the couple’s surviving daughter and lived until 1955. Whether or not she visited Elda in the 20 years after she moved out is unknown, but her death started the grand structure’s slide into dilapidation.

“We got as close as my GPS told me we’d get to the property without actually being on the property and parked my car. It was about half a mile from there to the first sign that told us to keep out, that we weren’t allowed on the 49.6 acres of land that just sits there with a decaying castle in the middle of it.”

Isaac and I looked around a bit. Elda has been abandoned for a few decades, and from an anthropological standpoint, it was fascinating to see the different kinds of graffiti on the walls, like a psychedelic mushroom with some initials and “1999” scrawled next to it. Somebody had recently sprayed “God might be dead but I’m not” in red where the living room once was, and another wrote “Respect this place” in lettering that looked like it had been done by a seasoned spraypaint user. There was furniture in the castle. Some of it was old and broken, but some of it looked as if it had been recently brought there for the purpose of hanging out a little more comfortably. There was a winding iron staircase that was still intact, but I decided not to test it out. A weathered basketball hoop that looked about 20 years old still stood outside. For the most part, it’s the shell of Elda that remains, and not much else. 

For about two hours we wandered the property. There was nothing we figured was going to surprise us unless we bumped into a crew of local teens doing the same thing we were doing. But that day, we were alone. It was an old castle in the woods and not much else, both strange to behold and ultimately anticlimactic. Seeing it for ourselves, discovering the history in front of us, was only part of the point. Some light law-breaking, ignoring a no-trespassing sign, was, too. Mostly, I wanted to see the place where the guy who figured out how to put a price on nature once lived—and to see nature’s decades-long attempts to take it back.

Words by Jason Diamond
Illustrations by David Alvarado