Cold Mountain is the story of Confederate soldier W. P. Inman, a disaffected combat casualty who flees from a military hospital in Virginia to return to his family's farm at Cold Mountain. Hunted by the Confederate Home Guard, Inman seeks to return to his loving wife, Ada Monroe. Ada's attempts to keep the farm afloat constitute the other half of the book's narrative. Inman is killed by the Home Guard a short time after reuniting with Ada.
The novel opens in a Confederate military hospital near Raleigh, North Carolina, where Inman is recovering from battle wounds during the American Civil War. The soldier is tired of fighting for a cause he never believed in. After considering the advice from a blind man and moved by the death of the man in the bed next to him, he decides one nightfall to slip out of the hospital and return home to Cold Mountain, North Carolina.
At Cold Mountain, Ada's father soon dies. The farm, named Black Cove, that the genteel, city-bred Ada lives on is soon reduced to a state of disrepair. But she is saved from destitution by a resourceful-but-homeless young woman named Ruby, who soon moves in with her. Together, they clean the place up and return it to productivity. Ruby also teaches Ada how to survive in these very different times.
Inman soon becomes aware of the Confederate Home Guard, who hunt down military deserters from the Confederacy. He meets a preacher called Veasey, whom he catches in the act of attempting to murder the woman he has impregnated. After Inman dissuades him, they travel together. They butcher a dead bull that had fallen into a creek and the bull's owner, Junior, gives them away to the Home Guard. They are put into a group of other captured prisoners and march for days before the Home Guard decides to simply shoot them because they are "too much trouble". Veasey steps forward to try to stop them and is killed. Inman survives when he takes a graze from a bullet that has already gone through Veasey, and they think he is dead. The Guardsmen dig a shoddy mass grave and Inman pulls himself out, helped in part by some passing wild pigs. He cannot bury Veasey, so he turns him face down and continues on.
Inman's journey is rough. He faces hunger and an attempted armed robbery at a rural tavern, even though he carries a LeMat revolver for protection. Occasionally he is helped and sheltered by civilians who want nothing to do with the war. Through cunning ingenuity he helps one of them track and recover a hog, her only possession and source of food for the winter, which had just been seized from her by Union soldiers. He is also helped by a woman who owns goats, who gives him advice and medicines to finally heal his wounds.
Ruby's father, Stobrod, is caught stealing corn at Ada's farm. Ruby reveals he was a deadbeat who abused and neglected her when she was very young; he is also a Confederate deserter. Nevertheless, Ruby grudgingly feeds him. Soon he returns another day with a simple-minded friend named Pangle. Together they entertain everyone by playing the fiddle and banjo. However the Home Guard, led by the sadistic Captain Teague, eventually tracks them down and shoots them. A third companion, referred to as "Georgia," escapes the killing and goes off to alert Ada and Ruby. The two women ride and find Stobrod barely alive. Ada and Ruby pitch camp to give him a place to recover.
After Inman arrives at Black Cove to find it empty, he sets out to find Ada on the mountain. Unexpectedly he soon encounters her out hunting wild turkeys. Both have changed so greatly in their appearance and demeanor since they parted that it is some moments before they recognize one another. Inman takes up camp with Ada and Ruby. Ruby is afraid Ada will dismiss her now she has a husband, and Ada reassures her that she needs her as a friend and for her ideas and help. Ruby gives the pair her blessing. Later Ada and Inman make love. They happily begin to imagine the life they will have together at Black Cove and make plans for their future.
However, as the party begins the trek back to the farm, they encounter the Home Guard. A shootout commences in which Inman kills all the members of the Home Guard except for 17-year-old Birch, Teague's vicious protégé. Inman eventually corners the boy against a rock ledge but is reluctant to shoot him down in cold blood. However, after attempts fail to convince Birch to lay down his arms and leave, the boy shoots and kills Inman.
Ada is left a pregnant widow. She raises her daughter at Black Cove, where she lives with Ruby and Stobrod. Ruby has married the boy from Georgia, named Reid, and has three children.