Mold toxicity can and does affect most animals. Even to the point of death. Take a look at this article shared on the Journal of the American Veterinary Medical Association, Black Toxic Mold Deadly to Animals.
It is commonly accepted that mold symptoms are not like a cold that can be passed from person to person. A person only gets symptomatic from direct contact or exposure to the contaminated area of mold. If a pet were to carry mold spores containing the mold toxins on its fur or paws, etc it may be possible. However even that is unlikely. The pet would have to be carrying large amounts of such spores and you would need contact for a significant amount of time.
If anyone else has anything to add, please join in!
