I USE THIS MILL QUOTE IN MY SIGNATURE

Conservatives are not necessarily stupid, but most stupid people are conservatives.

John Stuart Mill
English economist & philosopher (1806 - 1873)

And it certainly annoys some conservatives. One rebuttal I often come across is that Mill wasn't referring to modern day conservatives. Thanks to the online Stanford Encyclopedia of Philosophy, I can say that Mill came pretty close:

For example, there is his remark in his speech on the Reform Bill of 1866, where he proposed and defended universal suffrage and votes for women; this remark is in response to a comment by Conservative MP Sir John Pakington on a remark Mill had made elsewhere:

"What I stated was, that the Conservative Party was, by the law of its constitution, necessarily the stupidest party. Now, I do not retract that assertion; but I did not mean to say that the Conservatives are generally stupid. I meant to say that stupid people are generally Conservative. I believe that is so obviously and universally admitted a principle that I hardly think any gentleman will deny it." (Public and Parliamentary Speeches, 31 May 1866, pp. 85-86.)