Prayer is a petition or incantation to an entity (a god, spirit) to give you what you desire. In order to gain energy, you must expend energy in worship of the deity so he will grant you favor. The decision is based on the mood or desire of the deity.
Spell work is drawing energy from around the caster with objects, words and actions and then directing that energy in a specific direction to accomplish the caster’s desire. So the decision is based on the mood or desire of the caster.
So the difference is simple. If you want to succeed at you task, you can either depend on the temperament of the god or deity or you can use the energy that surrounds you and decide on your own what you want to do.
Prayer is a religious term and should be looks as such. Spell casting may have prayers or incantations involved but there is specific actions that direct the energy. Prayers alone without any action do little to resolve your needs.
Spell casting relies on trusting yourself. Prayer relies on trusting someone outside of yourself.
I hope this answers the question. It all boils down to a choice. You gather the power on your own or you rely on the power of an outside source for success.