Video information defining video content to be encoded may be obtained. Scene composition information for the video content may be obtained. The scene composition information may be determined by a convolutional neural network based on visuals represented within the video content. The video content may be encoded based on the scene composition information. The encoding of the video content may generate encoded video information defining the encoded video content.