Spencer Brydon returns to New York City after thirty-three years abroad. He has returned to "look at his 'property,'" two buildings, one his boyhood home on "the jolly corner." These properties have been the source of his income since the deaths of his family members. The second, larger structure is now going to be renovated into a big apartment building. Spencer finds he is good at directing this renovation, despite never having done this work before, suggesting that his innate ability for business was hiding deep within him unused. Spencer rekindles a relationship with an old friend, Alice Staverton. Both comment on his "real gift" for business and construction which he also finds "vulgar and sordid." He starts to wonder who he would be if he had stayed in the U.S.
He starts to prowl the house at night to try to meet his American alter ego. Brydon has begun to realize that he might have been an astute businessman if he hadn't forsaken moneymaking for a more leisurely life. He discusses this possibility with Alice Staverton, his woman friend who has always lived in New York.
Meanwhile Brydon begins to believe that his alter ego—the ghost of the man he might have been—is haunting the "jolly corner", his nickname for the old family house. After a harrowing night of pursuit in the house, Brydon finally confronts the ghost, who advances on him and overpowers him with "a rage of personality before which his own collapsed."Brydon eventually awakens with his head pillowed on Alice Staverton's lap. It is arguable whether or not Spencer had actually become unconscious or whether he had died and has awoken in an afterlife. She had come to the house because she sensed he was in danger. She tells him that she pities the ghost of his alter ego, who has suffered and lost two fingers from his right hand. But she also embraces and accepts Brydon as he is.