Theory and implementation of H-matrix based iterative and direct solvers for oscillatory kernels