Inverse spin galvanic effect in the presence of impurity spin-orbit scattering: a diagrammatic approach