Grow Deep Before You Scale Wide