It's largely random, and dialect dependent. I suspect that the "the" is fixed primarily with older terms, from when there were relatively few diseases with commonly known and recognized names. In my idiolect, the article only occurs with a small number of conditions (flu, mumps, measles, plague, heebiejeebies), and it's always optional. Older speakers in my native dialect region use it much more regularly, even with cancer, sugar diabetes ("the sugar"). I've even heard it once or twice with AIDS. "They say he's got the AIDS."
Most European languages have similar peculiarities about the definite article usage, like German's infamous habit of using it with the names of some countries and not with others.