An OLED display substrate, a manufacturing method thereof and a display device are provided. The OLED display substrate includes a TFT array layer, a first electrode, a pixel definition layer, an OEL layer and a second electrode arranged on a base substrate. The pixel definition layer is configured to define a plurality of subpixel regions. A reflection structure surrounds each subpixel region and is capable of reflecting light beams from the OEL layer and beyond an escaping cone in such a manner as to enable at least parts of the light beams to enter the escaping cone.