Often referred to as the "Castle on the Mountain," the Skene Manor was built in 1874-1875 by Judge Joseph H. Potter (1821-1902) at a cost of approximately $25,000, and was originally called Mountain Terrace. The Gothic-style building, located on Potter Terrace on the side of Skene Mountain (see map), was constructed using gray sandstone quarried from the mountain, with slate used in construction of the roof, dormers, and towers. The design and actual construction were the work of local architect A.P. Hopson, with the assistance of local Italian workers and slater Richard Lewis. The exterior included four chimneys and six dormers, and Mr. Hopson surrounded the building with fruit trees, evergreens, and flowering bushes.

The interior features three stories, each of which is 3000 square feet in area. There are 10 bedrooms, 3 dining rooms, and 7 fireplaces. The stair railing and banister are made of mahogany, walnut, oak, and birch. Stained-glass windows are located on the stair landing and over the front door.

Thanks to the efforts of Doris Begor Morton and other members of the Historical Society of Whitehall, the Skene Manor was placed on the National Register of Historic Places in 1974.

The land on which the Manor was constructed was purchased by Judge Potter in 1867 from Melancton Wheeler. The land had been obtained from heirs of John Williams, who had purchased some of the forfeited land of Whitehall's founder, Philip Skene.

Joseph Potter was born in Easton, Washington County, New York in 1821. He studied law and graduated from Union College. He moved to Whitehall in 1845 to enter a partnership with William Parker; he married Catherine Eights Boies the same year. Mr. Potter was elected Washington County judge in 1863, and New York State Supreme Court judge in 1871. He served 14 years in the latter position.

Judge Potter and his wife had three sons. J. Sanford Potter practiced law in the village and built a stone home just south of his father's. Henry Potter became an engineer and moved to Mexico. Rear Admiral William P. Potter served in the Spanish-American War; served on the USS Maine Court of Inquiry; and sailed with the Great White Fleet, a fleet of 16 new battleships sent around the world by President Theodore Roosevelt from December 16, 1907 to February 22, 1909.

Judge Potter died in 1902, and his widow sold the Manor to J. Lowenstein in 1906. Lowenstein had been in Whitehall since 1892, when he had come to work in his brother-in-law's silk mills. Mr. Lowenstein added gas fixtures and a carriage house to the Manor.

The building changed hands again in August, 1917, when the wife of Mr. Lowenstein's son sold it to Dr. Theodore Sachs (1876-1939), an optometrist and jeweler from Prattsville, NY. Dr. Sachs had moved to Whitehall in 1913, having been told there was a great deal of business checking watches for railroad men. Being a tall man, Dr. Sachs had been looking for a home in which he could enter without bumping his head.

In 1920, Dr. Sachs' son, Herman, moved the town clock from the Presbyterian church tower to the Manor tower and Dr. Sachs installed it. The clock had two 100-lb. weights that extended to the cellar, and a railroad bell served as the chimes. Each week, Dr. Sachs' two oldest daughters had the job of winding the clock.

After Dr. Sachs' death and the marriage of his daughters, the youngest resided in the home with her husband for three months. By then, World War II had broken out. To aid the war effort, the lead was removed from the tower clock, rendering it inoperable, and the copper was stripped from the attic cistern.

In 1946, a retired state trooper and his beautician wife, Clayton and Pauline Scheer from Schenectady, NY, purchased the Manor. They transformed the first floor into a bar and restaurant and lived upstairs. It was Mr. Scheer who was responsible for a popular, yet untrue legend surrounding the Manor. He placed an imitation grotto of garnet stone in the corner of the bar, and a lady's hand rose out of the grotto. Mr. Scheer told patrons that this was the hand of Philip Skene's wife, who had been interred in the basement for 100 years. He claimed to have carried her coffin from the basement and used it as a base for a stone waterfall in the corner of the taproom. Although this story wasn't true, it did make for interesting conversation.

The Manor was sold again in 1951 to Mr. and Mrs. J.R. Reynolds, who continued the restaurant business. The business grew over the next 17 years until the death of the general manager, Richard Reynolds, the son of the owners. Ownership again changed hands to Leo Mulholland, who lived in the Manor and ran the restaurant along with his nine children.

In 1983, Mr. Joel Murphy purchased the Manor from Mr. Mulholland. He ran the business for a few years before selling the it to someone else. The most recent owner was unsuccessful in running the restaurant business and allowed the building to fall into disrepair. Ownership was eventually transferred back to Mr. Murphy.

In the early- to mid-1990s, an out-of-state party wished to buy the Manor and move it to his state, removing forever one of Whitehall's most famous landmarks. Luckily, a group called Save Our Skene (SOS) was formed; the group was able to raise the money to pay the back taxes owed on the building and save it from being dismantled and moved.

SOS became Whitehall Skene Manor Preservation, Inc., which has worked at restoring and reopening the Manor for the past several years. From April to December, Friday through Sunday, the Manor is open for free tours and operates a Gift Shop and Tea Room (see full schedule). Skene Manor Preservation, Inc., still welcomes donations and volunteers.