Removing the turbo requires the downpipe to be removed, plus the oil and water lines, heat shields, and stays, plus the manifold interface. So its a small savings in time to do both, but not huge. No need to replace the manifold unless you are going for some monster turbo with a different flange setup. If the stock turbo is quiet and not burning oil, and your not trying to increase power greatly, then no need to touch it in my opinion.
If you have the stock setup on there, it is likely the bolts are totally frozen and you will be having to do some cutting, breaker bar-ing, and swearing in the small area you have to work with. That's a good reason to just focus on the downpipe initially. Once you get some fresh hardware on the exhaust, swapping out the turbo later will go much more smoothly.