The minhag to eat special fruits in Rosh HaShanah is from the Gemara in Horiyos 12a. While the gemara gives a list of recommended fruits dipping apples in honey is not mentioned.
The earliest source for the apple in honey is from the Abudraham as quoted by the Rema (OC 583:1). What is the significance of this famous minhag? The Moadim L’Simcha brings a number of reasons.
1. The Maharil says that Yitzchok smelled an apple orchard when he commented on the beautiful smell as Yaakov came in to him to receive the Brachos. The Vilna Gaon in the Biur HaGra says that this happened on Rosh HaShanah.
2. The Ben Ish Chai says that apples give three pleasures Smell, Appearance and Taste which correspond to three Brachos we’d like to receive for the next year Bini (Chidren and Nachas from Children), Chayai (Life/Health) and Mezoni (Wealth).
3. The Zohar says that wine symbolizes Din or judgment. Apples are eaten after strong wine in order not to be harmed by the strength of the wine. Therefore since Rosh HaShanah is a time of Din we eat apples in order to sweeten the judgment.
4. Rosh HaShanah we want Hashem to remember the ashes of Akeidas Yitzchok. The ashes that accumulated on the mizbei’ach from all the korbonos are called Tapuach or apple.
5. “Tapuach” has the same gematria as “Piru V’Rivu” and “Dvash” has the same gematria as “Isha”. Rosh HaShanah is a special time for barren women to be remembered by Hashem and blessed with children. We therefore eat Tapuach B’Dvash.