You should have more than 4 workers. Army related stuff, barracks, armory etc uses food. Library and temple gold.
So armory and temple can both be done. But Yellowmar is right, its frustrating hoarding the goodies to find out you do not have sufficient storage to do upgrades. I know since I have done this more than once.