The best place to start when building a collection of such apps is probably PortableApps.com Suite (PA), a preconfigured collection of programs that's a snap to unpack and get running. The default package of applications for the suite weighs in at about 256MB, and is like a free software greatest-hits collection, including mobile versions of OpenOffice, Firefox, Thunderbird, and Sunbird. You can also add other applications, including the 7-Zip archive manager, AbiWord (another open-source word processing app), the FileZilla FTP client, IM clients (Gaim and Miranda), the Sumatra portable PDF reader, and many more. You can elect to install as many or as few pieces as you'd like, and it comes with its own launch menu that you can customize with new apps of your own choosing.
A big reason to go with the PA suite rather than download productivity applications individually (although you can still do that) is that there is a fair degree of integrity within the suite. Everything in the PA suite has been pre-screened to work reliably in a standalone fashion, and each application is kept fresh and up-to-date. You don't have to worry about whether or not a given program will behave correctly; all that gruntwork has already been done for you.
Another point in favor of the PA suite is that it automatically saves user-created data in a directory tree that travels with the suite itself, so user data is never saved to the PC itself. PA even includes a backup utility which saves and restores user or application data (or both, or everything on the thumb drive) to an archive somewhere.
As tightly integrated as the PA suite is, you'll still need to keep an eye out for possible quirks. The antivirus program Clamwin, for instance, needed to be updated by hand after I'd downloaded the PA suite, but there were full instructions on how to do this on the PA site's page for Clamwin.