In 1964, Dr. David Henry is forced to deliver his wife Norah's twins with the help of a nurse, Caroline Gill. Their first child, a boy they name Paul, is born a healthy perfect child, but when the second baby is born, David notices she has Down syndrome. David, recalling the possibility of a heart defect and early death and decides that the baby girl will be placed in an institution.
Caroline was given the baby to take to the institution, but simply didn't like the conditions. She decides to keep and raise the baby, who is named, Phoebe. While Caroline is at the store, her car battery dies and she is stranded in the snow with Phoebe. She is picked up by a truck driver, Al Simpson, who drives them to Caroline's home. Meanwhile, David tells Norah that their daughter died at birth. After hearing that Caroline had kept Phoebe rather than take her to the institution, David bids her to do what she thinks is right. Caroline leaves for Pittsburgh to make a fresh start with Phoebe.
The "death" of their daughter has caused a rift in David and Norah's marriage. They move to a new home but continue to find it difficult to romantically connect. Meanwhile, Caroline begins working for Dorothy "Doro" March as a private nurse for her father, Leo. Caroline claims that Phoebe is her daughter and tells a half-true story of running away from Phoebe's father because he wanted to institutionalize Phoebe.
Caroline sends letters and pictures of Phoebe to David. David sends money to Caroline and makes a half-hearted attempt to find out where Caroline and Phoebe live. Meanwhile, Al, the truck driver who assisted Caroline on the night of Phoebe's birth, discovers her whereabouts. He and Caroline begin regular visits, and romance begins to bloom between them.
Six years later, the distance between the Henrys has increased. David is now an aspiring photographer with his own darkroom, where he keeps Phoebe's pictures and Caroline's letters locked away. He immerses himself in his work. Norah drinks secretly and becomes overprotective of Paul.
In Pittsburgh, Phoebe is growing up a healthy child despite her mental disabilities and David's negative predictions at her birth. Caroline and the Upside Down Society, a group of other parents of Down syndrome children, petition the school system to mainstream their children in public school.
Al still visits Caroline and has proposed to her twice. She continues to decline, although out of doubts for his fatherly love for Phoebe rather than his romantic intentions toward her. Phoebe is stung by a bee while at play and has a serious allergic reaction. Al helps Caroline take her to the hospital and steps in when a nurse's comment about Phoebe's condition outrages Caroline. Caroline realizes his genuine love for her and fatherly intentions for Phoebe. She tells Al she wants to marry him.
Paul and Phoebe are now thirteen, and Caroline and Al have been married five years. David sends Caroline a letter, asking her to let him meet Phoebe and allow her to know her twin brother, Paul. Caroline decides not to contact David again, worried he might unknowingly hurt Phoebe.
Paul is becoming an accomplished guitarist and dreams of attending Juilliard. David and Norah live almost completely separate lives and differ on what Paul should do when he's older. Norah simply wants her son to be happy, while David pushes Paul to follow a career path that will guarantee stability, money, and success.
While on vacation in Aruba, Norah has a love affair with Howard, who is also married with children at home. David blames the affair on himself and continues to spend more and more time in his darkroom with his photographs of Phoebe.
Paul and Phoebe are now twenty-five. Norah and David are divorced and Paul is traveling and studying music in France. Phoebe is in love with Robert, who also has Down syndrome, and wants to get married and live in a group home, but Caroline is scared to let Phoebe live an independent life.
David considers making a confession to Norah about Phoebe but can't bear to go through with it. Soon afterward, he dies from a heart attack. Later, when Norah sorts through David's collection of photographs, she begins to understand him in a way she never did when they were married. Caroline comes to visit Norah and explains that Phoebe never died at all and is living with her. Norah and Paul later visit Pittsburgh and meet Phoebe for the first time. Paul drives Phoebe to their late father's grave, despite her not knowing who he was to her. Paul thinks of what his twin sister might have been like if she had not been born with Down syndrome.