Sedimental microbial communities played an important role of driving biogeochemical cycles in river ecosystems. The Yellow River is one of the rivers with highest turbidity around the world. However, the impact of physiochemical and spatial factors on the microbial structure in the sediment of upper (up) and middle_lower (mid_lower) reaches remains unclear. In this study, we examined the microbial community, predicted the related ecological functions, and evaluated their response to the physiochemical and spatial factors. The results revealed that microbial structure was mostly affected by nutrients factors (NH4+, TN and TP) in the mid_lower reaches, while in the up reaches, it was mostly affected by the spatial factors. The most dominant phyla and family were Gammaproteobacteria and Hydrogenophilaceae respectively, while Deltaproteobacteria and Nitrospirae were more abundanct in the mid_lower reaches and higher abundance of Gammaproteobacteria and Verrucomicrobia were found in the up reaches. The dominant microbes in the up and mid_lower reaches were mostly showed correlations with COD, NH4+, TP and temperature, but they responded differently to the physiochemical factors. The predicted ecological function was mostly related with carbon, nitrogen and sulfur cycles, and they were active both in the up and mid_lower reaches. In summary, this study revealed the different responses of sedimentary microbial communities to the physicochemical and spatial factors between the up and mid_lower reaches, which will contribute to a better understanding of the ecological functions of microbial communities in the sediment of the Yellow River.