When baking a cake to be eaten with milk[or at a dairy meal], a pan which was used within the previous 24 hours for meat should not be used, even if the pan is completely clean. Likewise, Challah (or any other bread) which is being baked for use at a meat meal, may not be baked in a pan that was used for dairy within the previous 24 hours.
If the bread or Challah was already baked in a pan that was used for the opposite food within the past 24 hours, for Ashkenazic Jews it may not be eaten with food of the opposite class.
Jews who follow the Sephardic rulings of the Shulchan Aruch, may eat the cake/Challah with a food of the opposite class.
According to some Poskim, they may do so even L’Chatchilah. (See Shulchan Aruch, Rama, Shach and Taz Yoreh Deah Siman 95:1 and 2. See also Shu”t Yabia Omer Vol. 4 Yoreh Deah Siman 4)