A lifting and lowering control system, which drives a carrier device of a stacker to perform lifting and lowering operations, includes a driving module, a position sensor, and a processor. The driving module is connected with the carrier device for driving the carrier device to move from the initial position to the lifting position, which is defined as a first travel, and to move from the lifting position to the placing position, which is defined as a second travel. The position sensor detects a first or a second displacement amount respectively generated by the first or the second travel. The processor receives and stores the first and the second displacement amounts. The processer is allowed to read the stored first or second displacement amount and accordingly control the driving module to trigger the carrier device to carry out the first or the second travel at non-constant speeds.