Apps can also use components of IE, not just the browser itself. So even if you don't use IE for the web it's worth treating it as another part of the OS. For example, programs that auto-update software may use the same HTTP routines as IE and share the same 'Internet Options'.
IMO, it would be wise to patch IE, and its components, so that other apps aren't vulnerable.