Harry Caul (Gene Hackman) is a surveillance expert who runs his own company in San Francisco. He is highly respected by others in the profession. Caul is obsessed with his own privacy; his apartment is almost bare behind its triple-locked door and burglar alarm, he uses pay phones to make calls, claims to have no home telephone and his office is enclosed in wire mesh in a corner of a much larger warehouse. Caul is utterly professional at work but finds personal contact extremely difficult because he is intensely secretive about even the most trivial aspects of his life. Dense crowds make him feel uncomfortable and he is withdrawn and taciturn in more intimate social situations. He is also reticent and obsessively secretive with colleagues. His appearance is nondescript, except for his habit of wearing a translucent grey plastic raincoat almost everywhere he goes, even when it is not raining.
Despite Caul's insistence that his professional code means that he is not responsible for the actual content of the conversations he records or the use to which his clients put his surveillance activities, he is wracked by guilt over a past wiretap job which resulted in the murder of three people. This sense of guilt is amplified by his devout Catholicism. His one hobby is playing along to jazz records on a tenor saxophone in the privacy of his apartment.
Caul, his colleague Stan (John Cazale) and some freelance associates have taken on the task of bugging the conversation of a couple (Cindy Williams and Frederic Forrest) as they walk through crowded Union Square in San Francisco, surrounded by a cacophony of background noise. Amid the small-talk, the couple discuss fears that they are being watched, and mention a discreet meeting at a hotel room in a few days. The challenging task of recording this conversation is accomplished by multiple surveillance operatives located in different positions around the square. After Caul has worked his magic on merging and filtering different tapes, the final result is a sound recording in which the words themselves become crystal clear, but their actual meaning remains ambiguous.
Although Caul cannot understand the true meaning of the conversation, he finds the cryptic nuances and emotional undercurrents contained within it deeply troubling. Sensing danger, Caul feels increasingly uneasy about what may happen to the couple once the client hears the tape. He plays the tape again and again throughout the movie, gradually refining its accuracy. He concentrates on one key phrase hidden under the sound of a street musician: "He'd kill us if he got the chance" . Caul constantly reinterprets the speakers' subtle emphasis on particular words in this phrase, trying to figure out their meaning in the light of what he suspects and subsequently discovers.
Caul avoids handing in the tape to the aide (Harrison Ford) of the man who commissioned the surveillance (Robert Duvall). Afterwards, he finds himself under increasing pressure from the client's aide and is himself followed, tricked, and bugged. The tape of the conversation is eventually stolen from him in a moment when his guard is down.
Tormented by guilt over what he fears will happen to the couple, Caul's desperate efforts to forestall tragedy fail. To Caul's surprise, it turns out that the conversation he had obsessed over might not mean what he thought it did: the tragedy he had anticipated is not the one which eventually occurs. He is led to believe that his own apartment has been bugged and goes on a frantic search for the listening device, tearing up walls and floorboards and destroying his apartment to no avail. He sits among the wreckage, playing the only thing in his apartment left intact: his saxophone.