The title of the book is taken from 15th-century Dutch painter Hieronymus Bosch's triptych "The Garden of earthly delights", where oranges and other fruits symbolize the delights of paradise. The book is dedicated to Miller's friend Emil White, who established the Henry Miller Memorial Library in his old cabin in Big Sur.
The first two main parts of the book are portraits of Big Sur, with descriptions of its inhabitants, including writers, mystics, and two of Miller's children, Tony and Val. There are also reflections on the benefits of solitude, and on events from Miller's past.
The third part tells the story of when Miller was visited by an old friend from Paris, the French astrologer Conrad Moricand, in 1947. Moricand had written Miller that he was penniless. Miller invited Moricand to live with him in Big Sur for the rest of his life. Moricand arrived at the end of the year. The arrangement quickly turned into a disaster. Although Miller had told Moricand about the isolated and rugged life of Big Sur, Moricand was unprepared and complained often about the weather, food, and his own poor health, among other things. Miller put Moricand in a hotel in Monterey, and arranged for him to return to France. Moricand did not immediately return to Europe, however, instead writing Miller angry letters about his perceived mistreatment. Miller wrote about this episode, which would be published in 1956 as A Devil in Paradise , and a year later as the third part of Big Sur, called "Paradise Lost."
The lengthy epilogue details the many letters he has received from readers, and concludes by asking the readers not to write to him anymore. It was originally written in Anderson Creek in 1946, and revised for the book's publication.