Microsoft's new flight Simulator is a technological marvel that sets a new standard for the genre. To create a world that feels real and alive, and has billions of buildings in all the right places, Microsoft and Asobo Studios have relied on the work of several partners.
One of them is the small Austrian startup blackshark.ai from Graz, which with a team of only about 50 employees has recreated every city around the world using AI and massive computer resources in the cloud.
Before launching the new flight simulator, we sat down with Blackshark Co-Founder and CEO Michael Putz to talk about working with Microsoft and the broader vision of the company.
Blackshark is actually a spin-off of the game studio Bongfish, the maker of World of Tanks: Frontline, Motocross Madness and the Stoked snowboard game series. As Putz told me, it was actually Stoked who got the company off to Blackshark.
“One of the first games we made in 2007 was a snowboard game called Stoked and S Stoked Bigger Edition. It was one of the first games with a full 360-degree mountain where you could fly around in a helicopter and get off and land anywhere and down, ”he explained. “The mountain itself was procedurally constructed and described – and the placement of vegetation obstacles, other snowboarders and small animals was also procedurally carried out. Then we got into racing, shooting and driving, but we still had the idea of positioning and descriptions in the back of our minds. "
Bongfish returned to this idea while working on World of Tanks simply because it takes so much time to create such a large map with each stone placed on it by hand.
Based on this experience, Bongfish began building an internal AI team. This team used a number of machine learning techniques to create a system that could learn from how designers create maps and then at some point create their own maps created by the AI. The team actually used this for some of their projects before Microsoft got into the picture.
“By chance, I met someone from Microsoft who was looking for a studio to help them with the new flight simulator. The main idea behind the new flight simulator simulator was to use Bing Cards as a playing field, as a card, as a background, ”explained Putz.
However, the photogrammetry data from Bing Maps only yielded exact 1: 1 replicas of 400 cities – but this data does not exist for the vast majority of the planet. Microsoft and Asobo Studios needed a system to create the rest.
This is where Blackshark comes in. For Flight Simulator, the studio reconstructed 1.5 billion buildings from 2D satellite images.
While Putz says he met the Microsoft team by chance, there's a little more to it than that. At that time there was a Bing Maps team in Graz that developed the first cameras and 3D versions of Bing Maps. And while Google Maps have won the market, Bing Maps has actually beaten Google with its 3D maps. Microsoft then opened a research center in Graz, and when it closed, Amazon and others came in to attract local talent.
“So it was easy for us to fill positions like a doctorate. when rebuilding on the roof, ”said Putz. "I didn't even know it existed, but that's what we needed – and we found two of them.
“It's easy to see why reconstructing a 3D building from a 2D map is difficult. Finding out the exact structure of a building is not easy.
"What we basically do in flight simulators is look at areas and 2D areas and then figure out building footprints, which is actually a computer vision task," said Putz. “But when a building is obstructed by a shadow from a tree, we actually need machine learning because then it is no longer clear what is part of the building and what is not, because the shadow overlaps – but then machine learning completes the rest of the part Building. This is a super simple example. "
While Blackshark could rely on some other data as well, including photos, sensor data, and existing map data, it needs very little information to make a determination of the height of the building and some of its characteristics.
The obvious next problem is figuring out the height of a building. With GIS data, this problem is easy to solve. However, in most regions of the world, this data is simply not present or available. For these areas, the team takes the 2D image and looks for clues such as shadows in the image. However, to determine the height of a building from a shadow, you need the time of day, and the Bing Maps images are not timestamped. For other use cases the company is working on, Blackshark has that and that makes things a lot easier. And that's where machine learning comes in again.
"Machine learning takes a slightly different approach," noted Putz. "It looks at the shadows too, we think – because it's a black box, we don't really know what it's doing. But also if you look at a flat roof, like a skyscraper versus a mall. Both of them have mostly flat roofs Roofs, but the roof furniture is different on a skyscraper than it is in a mall. This will help the AI learn if you label it correctly. "
And when the system knows that the average height of a mall in a given area is typically three stories, it can work with that.
One thing that Blackshark is very open about is that their system makes mistakes – and when you buy Flight Simulator you will find that there are obvious mistakes in the placement of some buildings. In fact, Putz told me that he believes one of the biggest challenges in the project was convincing the company's development partners and Microsoft to adopt this approach.
"You're talking about 1.5 billion buildings. You can't do traditional Q&A on those numbers. And traditional finger pointing like a halo layer or something where you say," This pixel is no good, fix it "doesn't really work when you're statistically evolving like you do with AI. So it might be that 20% of the buildings are down – and that's the case in flight simulator I think – but there is no other way to tackle this challenge, since the outsourcing for the hand-modeling of 1.5 billion buildings is only logistical level and also budget level, not feasible.
This system will improve over time. With Microsoft transferring a lot of data from Azure to the game, users are sure to see changes over time.
Labeling is still something the team must do to train the model, however, and that's actually an area Blackshark has made great strides in, although Putz wouldn't say too much about it because it's part of the secret sauce of the company and one of the main reasons why this is all possible with only about 50 people.
"Data labels were not a priority for our partners," he said. “And so we used our own live marking to mark the entire planet by two or three people (…). This gives the data analysts a very powerful tool and user interface. And if the data analyst wants to recognize a ship, he tells the learning algorithm what the ship is and then immediately receives an output of the recognized ships in an example image. "
From there the analyst can train the algorithm in order to be able to recognize a certain object like in this example a ship or a shopping mall in Flight Simulator even better. Other geospatial analysis companies tend to focus on specific niches, Putz also noted, while the company's tools are independent of the type of content being analyzed.
And this is where the larger vision of Blackshark comes in. While the company is now being recognized for its work with Microsoft, Blackshark is also working with other companies, for example to reconstruct city scenes for autonomous driving simulations.
“Our larger vision is a digital twin of our planet in real time, especially the surface of the planet, which opens up a trillion use cases in which conventional photogrammetry like Google Earth or Apple Maps does not help because it only simplifies photos on simple geometric structures. For this we have our cycle in which we extracted information from aerial data, which may be 2D images, but also 3D point counts that are already doing another project. And then we visualize the semantics. "
These semantics, which describe the building very precisely, have a major advantage over photogrammetry: shadow and light information is essentially burned into the images, which makes it difficult to realistically re-illuminate a scene. Because Blackshark knows everything about the building it is building, it can also place windows and lights in those buildings, which creates the surprisingly realistic night scenes in Flight Simulator.
Point clouds, which are not used in the flight simulator, are another area that Blackshark is currently focusing on. Point clouds are very difficult for humans to read, especially when you get very close. Blackshark uses its AI systems to analyze point clouds and find out how many floors a building has.
“The whole company was founded on the idea that we need to have a huge technological advantage to get there, especially from video games where big productions like Assassin's Creed or GTA are now reaching their limits with thousands of capacities People working on what is very difficult to scale, very difficult to manage across continents and make a product delivered on time. It was clear to us that there would have to be more automated or semi-automated steps. "
And while Blackshark got its start in the gaming space – and while it's working on it with Microsoft and Asobo Studios – it actually doesn't focus on gaming, but rather on things like autonomous driving and geographic analysis. Putz noted that another great example of this is the Unreal Engine, which started out as a game engine and is now everywhere.
"To me, having been in the game industry for a long time, it is very encouraging to see because when you develop games you know how groundbreaking the technology is compared to other industries," said Putz. “And when you look at simulators, military simulators or industrial simulators, they always look like shit compared to what we have in driving games. And it's time for gaming technologies to expand out of the game pile and help all of these other industries. I think Blackshark is one of those examples to make this possible. "