By early 1843 there was much speculation in Nauvoo as to whether or not Joseph Smith was secretly practicing polygamy. More and more were learning that members of the Twelve and others had secret plural wives, and the saying became more popular, "There cannot be so much smoke without some fire." On February 21,1843, the Prophet spoke to those who were building the Temple ... a group consisting of both men and women, and told them that he knew what people were saying about him. According to Official Church History, Joseph told them: Smith tells temple workers to stop gossiping about polygamy . "There is a great noise in the city, and many are saying there cannot be so much smoke without some fire. Well, be it so. If the stories about Joe Smith are true, then the stories of John C. Bennett are true about the ladies of Nauvoo; and he says that the Ladies' Relief Society are all organized of those who are to be the wives of Joe Smith. Ladies, you know whether this is true or not. It is no use living among hogs without a snout. This biting and devouring each other I cannot endure. Away with it. For God's sake, stop it."
[Source: Joseph Smith Polygamy Timeline, http://www.i4m.com/think/polygamy/JS_Polygamy_Timeline.htm]