https://www.theinformation.com/the-reality-behind-magic-leap

The Reality Behind Magic Leap
By Reed Albergotti Dec. 08, 2016 2:46 PM PST · Comments by David Rodnitzky and Danny Sullivan

Plantation, Florida—In a massive, nondescript office complex far from the hustle of Silicon Valley, Magic Leap has become one of the hottest companies in the world of “mixed reality,” creating a headset that shows people images overlaid against the real world—like a cartoon-like firefly that controls a real-life lamp by tapping it with a wand. And while Magic Leap won’t be the first to market—Microsoft’s HoloLens is already there—Magic Leap executives long have said publicly they would use breakthrough technology to build a product far superior to rivals’.

That promise has been enough to lure in such brand name investors as Google, Alibaba and Andreessen Horowitz over the past three years. Florida-based Magic Leap has raised $1.4 billion, giving it a valuation earlier this year of $4.5 billion.

But Magic Leap may have oversold what it can do. Most investors put money in after seeing dazzling demonstrations put on by early prototypes and other technologies still in development. Much of that technology won’t be in the product now planned for commercial release, former employees say. Some technology the company invented couldn’t be applied to a consumer product, they say, and other pieces were so large and cumbersome they wouldn’t fit into a device designed like a pair of spectacles—which Magic Leap revealed to The Information is what it plans to release.

The first prototype was the size of a refrigerator, for instance, and was called the “Beast” by company employees, The Information has learned. It used a projector with a motorized lens that enabled images to have more depth and therefore look more realistic. The spectacles that Magic Leap plans to release will use a different kind of lens that aren’t likely to offer the same level of depth, for instance.

In an interview with The Information, Magic Leap CEO Rony Abovitz acknowledged that the prototypes used different technology. Prototypes were “big and they’re tethered and they’re not really what we’re ultimately going to be shipping but you were able to extract what was good about it, what was not.”

When asked about technology that was discarded, he said, “you can’t get precious about anything.” While the company has invented “so many novel technologies” in the process of building its ultimate device, he said, many of those technologies will never get used. He said the device that is released will still be better than rival products.

It’s not known what Magic Leap told investors about how much of the prototype technology would be in any product that was released. Magic Leap said all its investors were sophisticated.

Google, Alibaba and Andreessen Horowitz either declined to comment or didn’t respond to requests. Bing Gordon, a partner at Kleiner Perkins, said the firm invested in Magic Leap’s series B in late 2014 after seeing a demonstration of the Beast. He said he expected the device would get smaller, as is typically the case with digital technology.

Meanwhile, with the HoloLens already available, employees worry that Magic Leap is “getting late in the game.” The company said over the summer that it will release a product “soon.”

Mr. Abovitz professes to be unconcerned about the competition. He likened Magic Leap’s rivalry with Microsoft to that of Coke and Pepsi. “If we’re Coke and they’re Pepsi, it’s actually good, because you need people creating a market for soda,” he said. “They validated us and increased the number of investors banging on the door.”

He added that the “major question” from venture capitalists considering investing was “how much could we write a check for.” Some investors scrutinized the company more deeply, he said. Potential investors were “sending their brilliant professors from all the top schools to try and shoot us down,” he said.

Still, he acknowledged that the expectations of consumers might be too high. “I wonder if there are people” expecting something “50 years away from where we are,” he said.

Marketing Efforts

If expectations are too high, it is in no small part due to Magic Leap’s marketing efforts. Mr. Abovitz is prone to using fancy words to describe the technology he is developing, which has contributed to the mystique and misunderstanding. He calls a projector or a display a “spatial light modulator.” The optics used to reflect light into the eyes is a “photonic light field chip.” He refers to Magic Leap’s augmented reality or mixed reality as “cinematic reality.”

Magic Leap, former employees say, pushed the boundaries of marketing, releasing videos that purported to be Magic Leap technology that were actually created by special effects companies. For instance, in March of last year, it released a video online titled “Just Another Day in the Office at Magic Leap.” Shot from the perspective of one of its employees working at his desk, all appears normal until robots start falling from the ceiling and converging on the worker, who picks up a toy gun and starts blasting his enemies into tangled lumps of virtual metal.

The video, viewed 3.4 million times on YouTube, was meant to demonstrate a game people were playing with Magic Leap’s headset. It had been used for more than a year to recruit employees to South Florida. “This is a game we’re playing around the office right now,” Magic Leap wrote in the description of the video.

But no such game existed at the time, according to two former employees with direct knowledge. The video was not actually filmed using any Magic Leap technology. It was made by New Zealand-based special effects company Weta Workshop, which has worked on movies like “Mad Max: Fury Road” and “The Hobbit,” the employees said. One of them called it an “aspirational conceptual” video. The employees said some at the company felt the video misled the public. Magic Leap has since begun working on an actual game similar to the one in the video.
More recently, Magic Leap has released videos shot through its prototype devices.

In the interview, Mr. Abovitz said he had planned to keep the company secret, but that public interest was so great that he had no choice but to begin marketing its product publicly.

But it may have been the demonstrations of the product, to investors and others, that had the biggest impact. Until now, detailed descriptions of those demonstrations have been a closely guarded secret. Magic Leap showed a demonstration of the technology to The Information this week, using a bulky, helmet-like device connected by several cables to a desktop computer about six feet away. In one demonstration that took place on a set, a Star Wars-themed action scene unfolded. A droid appeared in the foreground and when the engines of a large space ship in the background turned on, fans blew air at the person wearing the Magic Leap device.

In another set of demonstrations, a small fairy buzzed toward a real-life lamp and turned it off with a magic wand. A green monster sat atop a table, and giant TV screens showing “Mad Max” and CNN appeared to float in the room. Miniature Star Wars spaceships battled as they buzzed around the room.

While it displayed images that look similar to those on Microsoft’s $3,000 HoloLens, the images were jittery and blurry when the headset moved around, a problem that wasn’t evident on the HoloLens. Magic Leap says it will fix that problem in the hardware that gets released.

In addition to the bulky demo connected to a computer, Mr. Abovitz showed The Information a prototype of the compact device it intends to build. It looked as if somebody fastened electronics to every inch of a pair of wire-framed glasses. It had a multi-layered, flat lens. He would not turn the device on, but assured a reporter that it worked just as well as the larger, helmet-like device.
Mr. Abovitz would not discuss details of the technology, repeatedly responding to probing questions with the phrase “Squirrels and Sea Monkeys.” That’s a phrase he has been known to use before to conceal the company’s secrets.

Furry Costumes

Before starting Magic Leap in 2011, Mr. Abovitz co-founded and served as chief technology officer of Mako Surgical, which made a robotic arm that aided in knee surgery. His penchant for grand promises was evident early on. He first made a splash in the tech scene with a five-minute TED X talk in Florida in December 2012. Dressed in a space suit and accompanied by people in bright, furry costumes, he said, “Greetings. A few awkward steps for me, a magic leap for mankind.” A heavy metal band began playing. There was no description of what Magic Leap was building and Mr. Abovitz never showed his face.

In the interview with The Information, Mr. Abovitz said he came up with the idea for the company while eating vegan pho balls at a Vietnamese-Cambodian restaurant with a physicist friend who dropped out of Caltech. “I came out with a screwball idea and he came out with an improved screwball idea.” The friend now works at Magic Leap, but Mr. Abovitz wouldn’t name him. He then teamed up with Brian Schowengerdt, a University of Washington professor who had been a leading researcher in augmented reality for decades, and NASA engineer Sam Miller.

The mystery surrounding the company has spawned countless blogs that analyze its patents and speculate on what the company might be building. Jono Macdougall, a U.K.-based engineer, has followed the company closely on his blog, GPU of the Brain. And Karl Guttag, an entrepreneur and expert in the field of augmented reality, has also devoted many hours to analyzing the company on his blog.
Fans and skeptics alike hang on every word Mr. Abovitz utters publicly, no matter how arcane or meaningless. For instance, at one tech conference in June, he described the Magic Leap device as “biomimetic,” or mimicking nature. To diagram this, he showed a glowing, blue animation of the human brain and the optical system. As light entered the eyeballs, a digitally rendered Pegasus appeared inside the brain. That was how Magic Leap worked, more or less, he explained.

Investor Demonstrations Starting around the fall of 2013, during fundraising for its $50 million series A round of financing, Magic Leap showed off the Beast prototype.

A rectangular, shoulder-width version of the Beast sat atop a bench or table. People stuck their faces into it like they were looking through a pair of coin-operated binoculars. The Beast dazzled some people who tried it. Projected onto a transparent, semi-reflective material in front of the user’s face, the images, like a firefly that landed in a person’s hand, were created so that they appeared to occupy space in the room, as opposed to looking like a two-dimensional screen floating in the air.

To avoid a common problem in 3-D and virtual reality headsets—discomfort and nausea—Magic Leap came up with something it called a “light field display” for the Beast. It used fancy hardware, including $1,000 lenses made of a flexible material that could rapidly refocus a projected image to give it a sense of depth.

After people got a demonstration of the Beast, they were shown a small mockup of a pair of sunglasses attached via cable to an iPhone-sized battery pack. Eventually, a device that looked like the glasses would do what the Beast could do, company officials said.
Magic Leap hoped to accomplish this with breakthrough technology such as a patented projector the size and shape of a piece of spaghetti it called a “fiber scanning display,” which would replace some of the components in the Beast.

The fiber scanning display, which co-founder Mr. Schowengerdt had worked on years earlier as a researcher at the University of Washington, involved shining a laser through a fiber optic cable and then rapidly shaking it back and forth to essentially draw an image on a surface. The fiber scanner would be tiny, but capable of doing the work of a much larger projector like the one on the Beast.

But Magic Leap hasn’t been able to get the fiber scanning display to work well enough, according to former employees.

Magic Leap still hadn’t gotten the fiber scanning display or its photonic light field chip working by the time it finished a new prototype in late 2014, when it was raising its series B round of financing. That prototype, a large helmet connected to a computer, was dubbed the WD3, for Wearable Demo Three. This was also the prototype used in the main demo shown to investors who put money into the company during its $793 million series C investment, which closed in February of this year valuing the company at $4.5 billion.
Instead, it used display technology similar to what was in the Beast.

After much deliberation, Magic Leap relegated the fiber scanning display to a long-term research project in hopes that it might one day work, and significantly paring back on its light field display idea.

“You ultimately in engineering have to make tradeoffs,” Mr. Abovitz said in the interview. “You have thousands of design parameters and go ‘what can I pick now?’” He said the company hopes to improve on the display technology in future devices over the next decade. “But we also realize if ‘product 1’ isn’t good, there isn’t 10 years.”

The latest prototype at Magic Leap has been kept under wraps and is referred to internally as the PEQ, or product equivalent, version because it is using hardware Magic Leap expects will be in its first release version. It is the spectacle-like device Mr. Abovitz showed to The Information, but would not turn on.

The PEQ is significantly sleeker than the WD3, but it uses a type of projection technology without moving parts, said Mr. Abovitz. Former employees say that approach is similar to what is in the Microsoft HoloLens. Mr. Abovitz disputes that, saying Magic Leap has created its own new technology. The PEQ prototype has a number of drawbacks compared to the earlier prototype devices. It doesn’t have the level of image depth as the Beast and the WD3. Mr. Abovitz likened the PEQ’s depth to compact discs, which isn’t the highest-quality audio format available, but is still pretty good.

It was so compact that Mr. Abovitz acknowledged it will be impossible to put the device over a pair of glasses, though he said there might be a solution for that problem. (The HoloLens and other headsets, in contrast, can be adjusted to accommodate glasses.)
Mr. Abovitz said the company’s PEQ already produces images with more depth that look better than the current Microsoft HoloLens. That will make the most difference when objects are up close to a user’s face.

Manufacturing

In a move that has befuddled industry experts and employees who worked at the company, Magic Leap decided to build a crucial component, the diffractive optics, in its factory in Florida. It’s a challenge for a startup because the technology is difficult and costly to manufacture. The microscopic structures must all be perfectly identical in each piece of glass. Any imperfection renders them useless, which is why the vast majority of diffractive optical elements used for this type of display technology are discarded at the factory.
Mr. Abovitz said manufacturing its own diffractive optics means Magic Leap can move faster in development of the product. He also said there was no other choice because Magic Leap is inventing something that is new.

The decision highlights questions from former employees about how Mr. Abovitz is spending the money Magic Leap has raised. Another issue many people raise is the sheer size of the workforce. Magic Leap has more than 800 employees spread out in far-flung offices in Israel, New Zealand, Dallas, Austin, Mountain View, San Francisco and Seattle.

The high-profile hardware company Nest, for instance, had its first product in stores with only 75 employees. Oculus had 75 employees when it was acquired by Facebook for $2 billion and was already selling the second version of its virtual reality headset to software developers. Mr. Abovitz says Magic Leap is leaner than other big companies currently working on the same type of technology.

If Magic Leap is successful in building a product, it could face an even bigger problem: How will it sell it and who will buy it?
Microsoft’s HoloLens retails for $3,000 and the company is not marketing it to consumers. Instead, it sells the device as a “developer kit,” meaning it is not considered a finished, polished product. It is selling the devices mainly to companies that are considering using it to increase productivity in employees, like in-the-field workers who can get instructions on how to fix machinery or assemble parts, such as elevator repair people. Medical students use it to study the human body in place of cadavers.

But Mr. Abovitz says Magic Leap’s headset is aimed at the consumer market. Its videos—some of them created by outside special effects companies—depict the device being used mainly for video games and entertainment. But it has also suggested people might use the device at the office instead of a computer screen.

Mr. Gordon of Kleiner Perkins said he thinks skepticism about Magic Leap’s market mirrors that of Nintendo’s Game Boy released in 1989. Because of the Game Boy’s small, black and white screen, it was met with doubts about its prospects, he said. But the Game Boy was a hit because it was fun to play. Magic Leap needs to create a consumer device at an affordable price tag with just one great piece of software that will drive people to buy the hardware, he said.

To get developers to create that software ahead of the product’s release, Magic Leap plans to create developer kits for software makers. But those early developers won’t be able to buy the devices and work on them at home. They’ll have to travel to specific locations, like San Francisco and Seattle, where the devices will reside, to create software. That is a departure from the way most companies operate, selling or handing out devices to lots of developers all over the world in hopes of creating good content at launch.
Microsoft, in this area, has perhaps the biggest advantage. It already owns the world’s biggest operating system and has adapted it for mixed reality with its “Windows Holographic.” Several manufacturers, like HP, Dell, Asus, Acer and Lenovo, are already using Windows Holographic to build virtual reality devices. In the same way, Microsoft aims to make the technology in the HoloLens available to other manufacturers in hopes that its operating system will become standard in virtual and augmented reality computers.

Mr. Abovitz acknowledges the company has taken on an immense challenge, but he projects unwavering optimism. He believes Magic Leap’s technology will replace phones, tablets, computers and televisions and people will use its glasses to customize the world with their own artwork and design. Mr. Abovitz says he believes he will prove skeptics wrong. “We’re entering the third act of the film—the one that has the cool ending and the Death Star explodes,” he says.