Goku couldn't use his Ultra Instinct form in Dragon Ball Super: Broly, highlighting a wider issue with the franchise's power scaling. Dragon Ball Super: Broly follows on from the Dragon Ball Super anime series, the final arc of which saw Goku unearth a devastating new transformation - by far his most powerful form yet. Battling against Jiren from Universe 11, Goku is backed into a corner and utterly exhausted. Only then, does he manage to heed the instructions of his current master, the angel Whis, and completely synchronize his mind and body into reacting automatically, thus attaining the God-like power of Ultra Instinct.
However, this form was entirely absent in the recent Dragon Ball Super: Broly movie. Instead, Goku and Vegeta battled mostly using their Super Saiyan God forms, although there has been some speculation online that during one powering-up scene, Goku's hair turns the gray-white color associated with Ultra Instinct for a split second, hinting at the ability still residing deep within the hero. Regardless, Goku does not use this power in his fight against Broly.
The in-story reason for this is provided in the Dragon Ball Super manga series. Speaking with Vegeta, Goku reveals that ever since the Tournament of Power ended, he's been unable to repeat his Ultra Instinct transformation, implying that the power can only be used in the direst of situations. This explanation is problematic for several reasons. Firstly, it means that Goku wasn't fighting at full-strength for the entire Dragon Ball Super: Broly movie, taking away from the drama and intensity of the film's climactic battle. Secondly, this reasoning suggests that Jiren was a more powerful opponent than Broly, despite the latter being introduced more recently.
While the canon explanation for the absence of Ultra Instinct in Dragon Ball Super: Broly may have been addressed, the real-life reason for its omission is less clear. Why wouldn't Akira Toriyama want Goku's strongest form in the franchise's latest big movie, especially when he conceptualized the transformation himself?
The root of the issue goes back a number of years. In the Dragon Ball Z anime series, the power difference between each transformation is obvious, with clear upgrades between the various Super Saiyan forms. When the Dragon Ball story was revived in 2013's Battle of Gods, Goku attained a brand new power-up: Super Saiyan God mode. The improvement was made abundantly clear, as Goku managed to go toe-to-toe with Beerus in his new form, despite being completely outclassed while in Super Saiyan 3.
The problems began with the Resurrection F movie two years later, which introduced an advanced form of the Super Saiyan God form with blue hair. This new form was added even though the previous version had barely been explored and came largely devoid of any explanation or context, with Goku and Vegeta attaining "Super Saiyan Blue" off-screen in between films. Since almost no story was shown between those two transformations, the difference between regular Super Saiyan God and Super Saiyan Blue has felt non-existent at times. In the Dragon Ball Super anime, Goku even used the form against Android 17 and Krillin, both of whom shouldn't be anywhere near God-level territory.
The addition of Super Saiyan Blue triggered a gradual breakdown in Dragon Ball's already shaky power scaling, where there seemed to be no discernible difference between each transformation, but this was rectified by the arrival of Ultra Instinct, which firmly put Goku onto a completely higher level.
And this may be the very reason Ultra Instinct was left out of the Dragon Ball Super: Broly movie. Ultra Instinct has been established as the most powerful form in the franchise by some distance. However, using the form against Broly would've either made Ultra Instinct too overpowered (thus rendering Vegeta's presence pointless) or, if Broly had been a match for Goku's new form, Ultra Instinct would've instantly felt just like any other transformation. Perhaps, then, it could be said that leaving Ultra Instinct out altogether in Dragon Ball Super: Broly was the only real option Toriyama had available.