In an example embodiment, data to be indexed in a distributed file system is received via a near real time publish application program interface (API). A shard responsible for the data to be indexed is determined Then a message is generated in a shard queue corresponding to the shard responsible for the data to be indexed, the message indicating that data needs to be urgently indexed, the detection of the message in the shard queue by a near real time manager corresponding to the shard responsible for the data to be indexed causing the near real time manager to cause the data to be indexed.