Tom Reagan (Gabriel Byrne) is the advisor and right-hand man for Leo O'Bannon (Albert Finney), an Irish mobster and political boss who runs an unspecified northeastern U.S. city during Prohibition. When Leo's rival, the Italian gangster Johnny Caspar (Jon Polito) announces his intent to kill bookie Bernie Bernbaum (John Turturro), Leo goes against Tom's advice and extends his protection to Bernie. Bernie is the brother of Verna Bernbaum (Marcia Gay Harden), who has begun a relationship with Leo while carrying on an affair with Tom.
Tom tries everything he can to convince Leo to give Bernie up to Caspar to end the war; he attempts to convince Leo that Verna is playing him to protect her brother, but Leo will not be swayed. After an assassination attempt on Leo, Tom reveals his affair with Verna to Leo to prove that she is dishonest. Leo beats Tom, and turns his back on both of them. Tom then approaches Caspar looking for work, and Caspar commands him to kill Bernie in the woods at Miller's Crossing to prove his loyalty. Bernie pleads with Tom to spare him, saying "Look in your heart". Tom fires his gun to fake the killing and tells Bernie to run and hide.
Caspar assumes Leo's position as boss of the city, controlling the police and using them to destroy Leo's operations. Meanwhile, Tom begins sowing discord between Caspar and his trusted enforcer, the brutal, homosexual Eddie "the Dane" Dane (J. E. Freeman). Upon finding that his men didn't actually see Tom kill Bernie, Dane takes Tom back to Miller's Crossing to see if Bernie's body is there. Tom nearly cracks as they approach the location, but they find a body that had been shot in the face and disfigured by birds. Unknown to Tom, Bernie returned to town and killed Dane's lover Mink (Steve Buscemi), who was Bernie's lover too, and placed the body where his should have been. Bernie holds this over Tom's head and tries to blackmail Tom into killing Caspar.
Tom uses Mink's unknown whereabouts to convince Caspar that Eddie Dane has betrayed him. Dane denies it, and Caspar has to decide whom he believes, and whom he will kill. In a rage, he beats Eddie Dane before shooting him in the head. Tom then arranges a meeting with Bernie, but sends Caspar instead on the pretext that he will be meeting Mink. Bernie gets the jump on Caspar and kills him. Tom arrives and tricks Bernie into giving up his gun, saying they can blame Eddie Dane, then reveals that Dane is dead, and that he intends to kill Bernie in retribution for blackmailing him. Bernie again begs for mercy, saying "Look in your heart", but Tom asks "What heart?" and shoots him.
With Caspar and Eddie Dane dead, Leo resumes his post as top boss. Verna has won her way back into Leo's good graces, and she reacts coldly to Tom. On the day Bernie is buried, Leo announces that Verna has proposed to him, and offers Tom his job back. Tom refuses, and remains behind, watching as Leo departs.