Brownian motion plays a crucial role in determining the structural stability and transport properties of colloidal dispersions. Accurately capturing these dynamics requires consistent treatment of both hydrodynamic interactions (HI) and thermal fluctuations, which are not explicitly resolved in standard Brownian dynamics simulations. Direct numerical simulation (DNS), when coupled with fluctuating hydrodynamics (FHD), provides a rigorous framework to incorporate these effects. In this study, we develop a DNS approach that integrates the smoothed profile method with FHD to simulate the Brownian motion of arbitrarily shaped rigid particles. To validate our approach, we consider two standard non-spherical shapes, for which analytical results are available: rods and disks. For rod-like particles, the short-time diffusion is anisotropic, with preferential motion along the long axis because of the reduced hydrodynamic resistance. On the other hand, disk-like particles exhibit dominant in-plane diffusion in the short-time regime, with out-of-plane motion significantly suppressed. At long times, rotational motions lead to isotropic behavior in both cases. Our results demonstrate that this DNS framework can accurately reproduce Brownian motion while consistently accounting for both HI and thermal fluctuations. This study contributes to understanding and controlling dispersions of non-spherical particles.