This wasn't an off day for the defense IMO. Brady is the caliber of QB who can rarely be stopped. Good enough to play a quick-tempo passing game that keeps pass rushers from ever getting to him, capable of exploiting even the slightest separation (or none at all), and WILL make you pay for a coverage mistake.
The only answer to that kind of QB is to keep up with your offense.