Remember the difference between acute (basically a single dose) and chronic (long term exposure) doses.
Vito's post indicates that one needs to eat grams of Selenium for an acute exposure. A brazil nut has 544 micrograms (1 millionth of a gram) of selenium so you'd have to eat some where around 10 thousand brazil nuts to recieve an acute toxic dose.
Or, you could probably eat 1000 brazil nuts a day for many months to get a chronic exposure.
Selenium is water soluble so it does not generally get stored in the body. You eat a bunch of brazil nuts, blood concentrations are a bit elevated for a while, and then the excess is excreted.
So as long as you are not living on brazil nuts exclusively (or have some other underlying health issue) enjoying a few from time to time is pretty harmless.
Selenium does, however, have some environmental implications: