Jesus asks the disciples if they want to depart from Him (John 6). It doesn't appear from Jesus' words that the disciples couldn't depart, but rather, that they had a free will choice and they could stay or go. When Peter, the leader of the group speaks, he asks Jesus in response to His question, 'Lord, to whom shall we go?' (John 6:66). Peter assumes that he can depart from Christ, and that the disciples can too. So, his question is the logical next step: "If we disciples can leave Jesus, where, or to whom, do we go? Who can we turn to if we turn away from Jesus?' This doesn't sound as though Peter believed he couldn't depart from Jesus. Rather, Peter believed that, if he departed, no one else had the words of eternal life. He couldn't have eternal life if he departed from Christ.
Deidre Richardson, “Lydia’s Heart: The Case for Prevenient Grace,” pp. 305-306