There are two special values of the coupling constant for which there exist noncentral elements of SL(2,Z) that map N=4 Super Yang-Mills theory with gauge group U(n) to itself. At these values, the field theory can be compactified on a circle with duality-twisted boundary conditions. The low-energy limit of this model directly probes the S-duality operator. Augmented by an R-symmetry twist, and with additional restrictions on the rank n, this low-energy limit appears to be a nontrivial topological field theory. Upon further compactification on a torus, the Hilbert space of the low-energy theory can be mapped, using U-duality, to the finite dimensional space of minimal string states on a three-dimensional manifold that is a torus fibre-bundle over a circle. Using the string theory realization, I'll compare the low-energy theory with Chern-Simons theory. Also, compactification on a Riemann surface of higher genus suggests a relation between the dimension of the Hilbert space of certain Chern-Simons theories on the Riemann surface and the supertrace of the action induced by mirror symmetry on the appropriate cohomology of the appropriate Hitchin space.