The story is told by Nathan Zuckerman, a writer who lives quietly in New England, where Coleman Silk is his neighbor. Silk is a former professor and dean of faculty at nearby Athena College, a fictional institution in the Berkshires of western Massachusetts. Silk is accused of racism by two African American students. The uproar leads to Silk's resignation. Soon after, his wife Iris dies of a stroke, which Silk feels is caused by the stress of his being forced out of the college.
Silk begins a relationship with Faunia Farley, a 34-year-old local woman who works as a janitor at the college and is illiterate. Silk is criticized by feminist scholars at the college for this.
It is slowly revealed through Zuckerman's musings that Silk is an African American who has been "passing" as Jewish (and white) since a stint in the Navy. He completed graduate school, married a white woman and had four children with her. He never told his wife and children of his African American ancestry. As Roth wrote in the novel, Silk chose "to take the future into his own hands rather than to leave it to an unenlightened society to determine his fate".