Yes, he returned to his holiday in Valle d'Aosta, but consider this:
What does Pope Benedict like to do on his holidays?
Write and play the piano.
Both of these are now pretty well shot to you-know-where for the next 4 weeks.
Add to that the general discomfort that comes with a cast (difficulty buttoning buttons, eating, and all sorts of other functions - if you've ever had a cast on you know what I mean), then add to that the difficulty he's going to have in saying Mass, and you see that he may be feeling a bit down lately.
So say a prayer for Pope Benedict.