Just like that, it seems Matthew Vaughn is off X-Men 3 and Joss Whedon might be up to bat as director.
According to Aintitcool.com they have confirmation straight from Avi Arad that Vaughn is off X-Men 3. There was no reason given except to say that it was not due to creative differences and that Matthew was "heartbroken".
And on Joss Whedon's Blog there's a post talking about the possibility that Whedon might be the new guy.
Nothing confirmed on the Joss Whedon side, of course.