I think that is a known issue with the 4.0.4 update that SD card reading speed is slower than the app drawer display. Due to that even your drawer settings like separate pages, and also you home page icon keep on vanishing.
I suggest to move to some custom launcher till we get an update. I purchased Nova launcher in 25 Billion sale and now it is working amazingly well.
