EDIT: Science adds a useful comment...
You actually do encounter a GCD from swapping, but on casts longer than the GCD (such as Starfire), there is no penalty, but on instant casts (Moonfire), you encounter a slightly longer GCD due to the idol swap.
So be a little more careful than I and the following post make it sound like you should be; there are times when the global cool down (GCD) will effect your spellcasting, mainly if you jump to Moongoddess without pausing in between (more important on raids, I suppose).
Re: Ivory Idol of the Moongoddess
Score 4.3 Vote: [-] [+] by Vartharazun, 5.0 months ago
You can use every single different Idol for every single different spell you cast without causing any sort of GCD delay.
/equip Idol of the Unseen Moon
/equip Ivory Idol of the Moongoddess
/equip Idol of the Avenger
Copy and Paste these macros for pve moonkin pwnage.