Selenium solar cells (SeSCs) are gaining renewed interest as wide band gap photovoltaic absorbers suitable for indoor energy harvesting and tandem applications. While significant progress has been made through extensive optimization of electron transport layers (ETLs), the role of hole transport layers (HTLs) has been comparatively less explored. In this work, we investigate the integration of inorganic transition metal oxides (TMOs), namely molybdenum oxide (MoO x ), tungsten oxide (WO x ), and vanadium oxide (V2O x ), as hole-selective contacts in SeSCs. We systematically optimize the TMO thicknesses and assess their effect on device performance under both standard AM1.5G and indoor illumination conditions. Our results demonstrate that incorporating optimized TMO layers substantially improves the fill factor (FF) and parasitic resistances of the device, leading to enhanced power conversion efficiencies (PCEs). The best outdoor performance is achieved with a 20 nm MoO x HTL, delivering a champion PCE of 5.5%. For indoor conditions, a 10 nm V2O x HTL enables PCE values exceeding 10% across a wide range of light intensities and spectra. Ultraviolet photoelectron spectroscopy and transmission electron microscopy-energy dispersive X-ray spectroscopy analyses reveal strong interfacial interactions between Se and the TMOs, including evidence of spontaneous MoSe2 formation at room temperature, which likely contributes to enhanced hole selectivity and suppressed recombination. Additionally, preliminary indications suggest the possible formation of VSe2 under similar conditions. These findings underscore the crucial role of inorganic HTLs in unlocking the full potential of SeSCs and highlight their suitability for emerging applications such as indoor photovoltaics and monolithic tandem architectures.