This is a helpful guide to look through / watch the video running through each of the followers and assessing their relative strengths.
Updates to the game will shift the importance or relative value of different followers when prioritising who to spend food and gold on, but you will likely quickly hit the point where it is follower crystals that become your limiting factor rather than not wanting to upgrade them higher.
As of 0.36 we just really hope something is done to increase the ore rate, as there is currently not much point having 9 followers that can craft thing using ore, but have so little ore production that you can struggle to keep 1 or 2 active.