The Driver only saw one level of Agent Smith’s illusions

“Driver,” said Stealth, “what are you seeing right now?”
St. George swore the car lifted itself a little higher on the road. “What do you mean, ma’am?” asked the voice on the radio.
“What do you see on the road ahead of us? How many cars are there on this stretch of road with us?”
“Well … uhhh,” Cesar said, “there aren’t any.”
Freedom glanced at a glossy black Hummer as they swooped past it.
“Please explain,” said Stealth.
“I mean, there’s some wrecks and stuff. Abandoned cars. That what you meant?”
…
“It would seem,” said Stealth, “that the Driver’s unique senses in this state allow him a different perception of the world around us, much as Madelyn’s mind allowed her to resist the false memories.”

Excerpt From Ex-Purgatory, pg 418-419
Peter Clines
https://books.apple.com/us/book/ex-purgatory/id676938479