The Rayleigh hypothesis is the assumption that the field in the region above (below) a rough surface can be expressed as a weighted sum of upwards (downwards) propagating scattered (transmitted) modes, and that these expressions can be used to satisfy the boundary conditions on the fields at the surface. This hypothesis is expected to be valid for surfaces of sufficiently small slopes. For one-dimensional sinusoidal surfaces, the region of validity is known analytically, while for randomly rough surfaces in one and two dimensions, the limits of validity of the Rayleigh hypothesis are not known. In this paper, we perform a numerical study of the validity of the Rayleigh hypothesis for two-dimensionally rough metal and perfectly conducting surfaces by considering the conservation of energy. It is found for a perfect electric conductor that the region of validity is defined by the ratio of the root-mean-square roughness,~$\delta$, over the correlation length,~$a$, being less than about $0.2$, while for silver we find $\delta/a \lae 0.08$ for an incident wavelength $\lambda=\unit{457.9}{\nano\meter}$. Limitations in our simulations made us unable to check the Rayleigh hypothesis for roughness where $\delta \gae 0.13\lambda$.