Modes of open layered anisotropic waveguides: a numerical method with exponential convergence