sure you can do this but when you backup your database all shit that was in there comes to the new installed forum... thing you can do is cleanup your old forum by uninstalling ALL plugins and products reinstall them on your new vbulletin. Now backup the most important tables in your database you wanna keep, like tables:
user
posts
threads
forum
than the whole structure of the forum will be save.... also do a complete backup of your database with all hacks and without all hack to be save nothing goes wrong.