The first Cliff House was a modest structure built in 1863 by Senator John Buckley and C. C. Butler. Captain Junius Foster eventually leased the Cliff House Restaurant from C. C. Butler and under his management wealthy San Franciscans flocked to the coast to enjoy the unique restaurant and wonderful views. The guest register bore the names of three U.S. presidents as well as prominent San Francisco families such as the Hearsts, Stanfords, and Crockers, who would drive their carriages out to Ocean Beach for horse racing and recreation.
Captain Foster renovated the Cliff House in 1868, adding a promenade and two new wings. It became the meeting place for local politicians as well as less savory citizens from the Barbary Coast. High society locals abandoned the Cliff House although it remained a favorite attraction for tourists and the less wealthy. It became known for scandalous behavior, which greatly disturbed one prominent and well-known San Franciscan. Adolph Sutro, a self-made millionaire, philanthropist, and later, mayor of San Francisco, had built his estate at Sutro Heights overlooking the Cliff House.
Sutro purchased the Cliff House in 1883 and tried unsuccessfully to manage it himself. He then leased it to Sroufe and McCrum, a local wholesale liquor company. In 1885 Sutro leased the Cliff House to J. M. Wilkins, directing him to clear out the riffraff and bring back the local families. In 1887, the Cliff House was severely damaged when the schooner Parallel, abandoned and loaded with dynamite, ran aground on the rocks below. The explosion was so powerful it was heard all over the Bay Area. A patched-up Cliff House continued to operate until 1889 when the exterior of the building was treated to a new paint job, and the interior received modern water closets and a new kitchen closer to the dining room. A chimney fire destroyed it on Christmas day in 1894.