Background Suboptimal prenatal and early postnatal growths are associated with obesity in later life, but the underlying mechanisms are unknown. The aim of this study was to systematically review the literature that reports on the longitudinal association of (i) birth size or (ii) infant growth with later (i) energy intake, (ii) eating behaviors, (iii) physical activity or (iv) sedentary behavior in humans. Methods A comprehensive search of MEDLINE, EMBASE, PsycINFO and The Cochrane Library was conducted to identify relevant publications. We appraised the methodological quality of the studies and synthesized the extracted data through a best-evidence synthesis. Results Data from 41 publications were included. The quality of the studies was high in three papers, moderate in 11 and low in the large majority (n = 27) of papers appraised. Our best-evidence synthesis indicates that there is no evidence for an association of birth weight with later energy intake, eating behavior, physical activity or sedentary behavior. We found moderate evidence for an association of extreme birth weights (at both ends of the spectrum) with lower physical activity levels at a later age. Evidence for the association of infant growth with energy balance-related behavior was generally insufficient. Conclusions We conclude that current evidence does not support an association of early-life growth with energy balance-related behaviors in later life, except for an association of extreme birth weights with later physical activity.