The book is set in the Tenley Heights section of Baltimore, Maryland. Little is known about this town except for the fact that Memorial Stadium, then-home of the Baltimore Orioles, is in the neighborhood. The time period in which this story takes place is not specified, but it is set some time in the 1980s.
Young adolescent Vernon Dibbs is the book's protagonist, who also narrates the story through the use of flashbacks. At the beginning of the story, Vernon is talking to his tutor, Miss Annie, who is a former teacher that helped Vernon with his schoolwork, about how many years have passed and how they can still remember what happened.
The flashback begins with Vernon in seventh grade. He is one of five children born to a functionally illiterate factory worker and his wife, who died three years earlier while at work and left Vernon's father to raise five children by himself. Vernon is the middle child with two older siblings; his sister Steph and his brother Tony (Vernon states in the beginning that Steph has since married and moved away and Tony is away at college after graduating with a scholarship), and his two younger siblings are his brother Ben and sister Sandra. Vernon's father works the second shift at his factory. As a result, all five of his children have to pitch in to help around the house, doing tasks such as cooking dinner and washing clothes.
Vernon is a year older than the rest of his classmates, due to being held back one grade earlier in his schooling. He is in danger of being held back again, because his schoolwork is suffering and so are his grades. Vernon also does not have any resources to help him get tutoring, as his father is usually working and none of his siblings are able to take him.
Vernon spends most of his time hanging out with his friends Bobby, Chris, and Jerry, and they get their kicks by causing trouble in town and harassing Maxine Flooter, an alcoholic dubbed the "Crazy Lady" by everyone in the neighborhood. Maxine is often seen dressed eccentrically and dragging her intellectually disabled son, Ronald, around with her wherever she goes. Ronald is also a target for the boys' ridicule, which further infuriates his mother as she is his only caregiver.
One day, things begin to change. On a trip to the local grocery store for potatoes, Vernon sees the store owner, an old man named Milt, arguing with a customer over the price of the potatoes. Vernon has the same argument with Milt, as Steph had gone to his store a few days before and bought potatoes for the same price. Milt kicks Vernon out of the store, and the customer interrupts him on the way out to tell him he had done well. Vernon turns to look at her and discovers that the customer was Maxine.
The two of them get to talking, and Vernon lets it slip about how much trouble he is having in school. Maxine decides to try to get Vernon help and enlists the help of Miss Annie, who is her neighbor. She agrees to help Vernon pull up his English grade, but eventually insists that as payment, Vernon must help Maxine out around the house and help with Ronald as well. While it is not easy for him, as Maxine is usually drunk and sometimes ends up in jail as a result, Vernon begins spending more and more time around Maxine and Ronald and begins to form a friendship with the young man.
Eventually Vernon is all but ignoring his family, which does not help his already tense situation at home. He finds himself up late at night on several occasions talking with his father, who likes to come home and sit in the kitchen listening to oldies (which he claims help him think about the past), and those conversations help to form a bond between the two. (Later in the story, Vernon's father asks him to teach him how to read, which is an important moment in their relationship.)
After a few weeks, Vernon meets Ronald's teacher, who suggests to Maxine that Ronald sign up for the local Special Olympics. Vernon decides to take charge of the efforts to send Ronald to the games, getting the help of his friends in the process—especially Jerry, who has an intellectually disabled brother that he visits on the weekends and that he never tells anyone about.
Things start to come to a head when Vernon organizes a block party to help raise funds for Ronald. It ends when Maxine, drunk as usual, storms onto the block in a drunken rage and tells everyone off. Vernon's frustrations finally boil over after Maxine interrupts a Sunday Mass at the local church and begins telling all the people in the church exactly what she thinks about them—specifically telling Vernon that he does not have "the brains God gave a stump." After that, he refuses to have anything to do with her, only going in and out of the house to check on Ronald. Maxine, however, makes several attempts to get in contact with Vernon as she has something important to tell him.
The day of the Special Olympics arrives, and Ronald and Vernon win several ribbons as a team in walking races. They run into Ronald's teacher at the games, who offers them a ride home and reveals to Vernon what Maxine had been trying to tell him—that Ronald was going to be moving to North Carolina to live with one of Maxine's family members who can better take care of him. Vernon is shaken by the news and storms out of the car on the side of the road. He then confronts Maxine about what she is about to do, with her telling him that she tried to help Ronald but that he will be better off in North Carolina.
Finally the day comes for Ronald to leave, and everyone in the neighborhood comes to bid him farewell (except for Maxine, who cannot bear to see her only child leave her). After everyone says their goodbyes, Ronald's new family drives away. The sight of Ronald leaving is too much for Vernon to handle, and he begins chasing after the car as fast as his legs can carry him. Unfortunately, not only can he not catch up to the car, he trips over a curb and is sent sprawling into a wall. As he lies on the hard ground, severely injured, his father comes over to him. The book ends with him saying to his middle son: "Vernon, I'm here."