As far as I know (having owned MPC1000, MPC2500, SP-202, SP-404SX, Electribe ESX-1) none of the devices mentioned so far can be used to "play my synthesizer or a guitar or a voice through but can save them so they are still there next time I turn the thing on as well as easy sampling and manipulating in live play". To do that you need either a looper that can save loops, preferably a multitrack one, or an Octatrack. Octatrack is $1200 or so, if that's too much have a look at a looper, maybe the Boss RC-30 or Digitech Stereo Jamman. A looper won't let you do a great deal of manipulation to what you're sampling, but you'll be able to record while you're playing back. Another option would by the Korg KP3.
But really the Octatrack is the best device for what you want to do, it does sequencing, sampling, looping and sample manipulation and does it all really well. I've seen them go for $800 second hand, would be worth it for you to save some money and get one of them rather than make do with things that don't quite do what you want in the meantime.