The only downside I can think of would be that your character saves might not transfer (this is what's stopping my girlfriend from getting the Special Edition on Xbox), but Bethesda hasn't clarified whether saves will carry over yet. There's ultimately no reason not to take advantage of these free upgrades in this scenario, especially if the next-gen enhancements Bethesda has planned for the new edition are substantial. For PlayStation players who don't have the option to use mods and fans on both Xbox and PC who prefer the vanilla Skyrim experience, I think upgrading Skyrim Special Edition to the Anniversary Edition that features next-gen enhancements and the new fishing mechanic will be a no-brainer.
