He states it in interviews often, but I couldn't tell you which ones off of the top of my head. I think I first learned of it because he appeared in a documentary on "chubby chasers" years ago. But here he explains it in his own words, at his keynote address in 2001:
"A lot of people ask how I got interested in the problems of fat people, especially since I was thin myself when I was growing up. Well, without knowing it, I was an admirer of the large figure from the age of 5 years old. I always liked girls, from kindergarten on, and somehow or other, they were always chubby. When I was nine, I developed a crush that lasted 5 years, on a girl who lived across the street who was eight years old. She was the kind of girl who caused mothers on the street, including mine, to say things like, 'Diana has such a pretty face--it's such a shame she has that weight problem.' I always thought to myself, 'What weight problem? She looks great to me!'
Around the age of 14, I came out of the closet as an FA when I went to my mother and pointed out a picture in a magazine of a woman who was perhaps a size 18. I told my mom that I thought the woman had a perfect figure by my standards. After she regained her composure, she explained to me that the woman was pretty but 'very overweight' and that if I thought that was attractive, I must be going through a phase. (The phase has so far lasted 54 years, and I'm still counting.)
After that, I came to face a total lack of acceptance on the question of my taste in women, from family, from friends, and from society. In fact, there was virtually no recognition in the media of the existence of others like myself, and I felt pretty isolated. I noted that there was plenty of books and articles written about gay men and women, but almost nothing about those whose sexuality involves attraction to a larger partner. I began to read anything I could find on the subject, and came to the opinion that there was a huge gap in the world's understanding of fat people and their admirers.
When I dated in high school and college, my dates were always plus-sized girls and women. The one exception was when a thin female friend and I both wanted to go to a certain prom, and neither of us had dates, so we asked each other to the prom! But it never would have occurred to me to think of her as anything other than a friend.
In 1964, I married a very plus-sized woman, named Joyce. In living with her and getting to know and understand her issues, I became very angry at the way that she was treated every day of her life. For example, when we were engaged, and applied for a marriage license, blood tests were required under New York State law. She went to a new doctor in her community to receive a blood test, and was shown into his office. He asked why she was there, not having been informed by his nurse. Joyce replied that she was engaged to be married soon, and needed to obtain a blood test. The astonished doctor replied, 'Who would want to marry you?' When Joyce, between her tears, told me the story on the phone, I don't need to tell you how I felt. Fortunately for both myself and the doctor, I was in a distant city, and did not own any firearms.
Events like these set the stage for an anger that built up inside me, anger that was later to be put to use productively."