I don't bother about it too much, LOL! Just use a good solvent based cleaner like JudiKins permanent fabric ink cleaner.
I have found it to be the best at getting off hard to remove junk like acrylic paint or totally dried on inks. It is also great for first time use of a stamp especially solid images. This cleaner removes the leftover silcone (from the vulcanizing) which is what causes ink to bead up so badly the first time you ink up. BEWARE, though you folks who use clear polymer stamps this is not the cleaner for you it will melt your clear stamps to jelly!