The Brazilian site MSXPró have published an article on how to clean your Floppy Disk Drive head in true MacGyver spirit, using an ordinary disk, coffee filters and some basic household tools. The tutorial covers both 3.5" and 5.25" disks and comes with a lot of pictures, so you probably won't mind the content being in Portuguese only.
Relevant link: Drive head cleaning disk - DIY