A place to create/learn with Unity, ARKit, and Mapbox!
Note: This library is experimental and will change. It is published to gather feedback from the community. Please use with caution and open issues for any problems you see or missing features that should be added.
We'd love to have you contribute to the project!
Also, see our related iOS library.
One limitation of ARKit is that it does not have any knowledge of where in the world a session was started (or ended), nor does it know the True North alignment with the real world. ARKit is great for location positional tracking, but suffers over distance, or when tracking is poor. My testing has shown that ARKit's position begins to drift after just a few meters and becomes very noticeable after 50 meters (average accumulated drift ~10 meters with GPS accuracy 5 meters).
This project aims to inform ARKit of global positioning using location data from the phone, which will enable Unity developers to "anchor" augmented reality experiences to real world coordinates. Given the sheer volume of data that Mapbox provides, the possibilities are endless.
For additional inspiration and reference, please see this library for iOS. Concepts and potential solutions should be similar, though implementation will vary greatly.
Please note: while it is possible to use Mapbox to display 3D maps on your coffee table, this project is aimed at building "world-scale" maps and augmented reality experiences.
AutomaticWorldSynchronizationscene OR drag
WorldAlignmentKitprefab into your scene.
Alignmentdepends on your settings.
If you're not familiar with the Mapbox Unity SDK, it may help to check out the built-in examples. You should be familiar with building a map and using
LocationProvider. For brevity, I will assume you know how these components work.
All relevant AR World Alignment components live in
At the core, we use a
SimpleAutomaticSynchronizationContext to align our map (world) to where the AR camera thinks it is in the world. This context uses the AR position deltas (a vector) and GPS position deltas (mapped to Unity coordinates) to calculate an angle. This angle represents the ARKit offset from True North. Note: I could have used
LocationService compass heading, but I've found it's often far more inaccurate than these manual calculations.
You can think of a synchronization node as a comparison of ARKit and location data. You are essentially capturing the position of both "anchors" at the same time. We use this information in various implementations to compute an
This event is sent when the context has successfully calculated an alignment for the world. This alignment can be used to manipulate a root transform so that it appears to be aligned with the AR camera.
Due to ARKit positional drift over distance, we need to constantly refresh our offset (and potentially our heading). To do so, we should consider the range at which ARKit begins to visually drift, as well as the accuracy of the reported location (GPS). You can think of this relationship as a venn diagram.
The center of the circles represent reported ARKit and GPS positions, respectively. The radius of the circles represent "accuracy." The intersection represents a potentially more accurate position than either alone can provide. The bias value represents where inside that "intersection" we want to be.
As previously mentioned, we use the delta between position updates to calculate a heading offset. Generally (depending on GPS accuracy), I've found this to be far more reliable and accurate than compass data.
The end result of a successful synchronization is an
Alignment, which offers a rotation and position offset. These values are meant to be used to modify the transform of a
world root object. We have to do this because ARKit's camera should not be modified directly.
Attempt to compute the bias (see below) based on GPS accuracy and
How much to compensate for drift using location data (1 = full compensation). This is only used if you are not using
The minimum distance that BOTH gps and ar delta vectors must differ before new nodes can be added. This is to prevent micromovements in AR from being registered if GPS is bouncing around.
This represents the radius for which you trust ARKit's positional tracking, relative to the last alignment. Think of it as
accuracy, but for AR position.
This class is mostly a monobehaviour wrapper around the context itself, which allows you to specify settings in the inspector. However, it also has knowledge of when ARAnchors are added, so as to offset the
Alignment height based on a detected anchor height.
This class is also responisble for listening to location updates from the
LocationProvider and adding synchronization nodes (gps + ar positions) to the context. Important: GPS positions must be converted to Unity coordinate space before adding to the context!
Lastly, this object needs an
AbstractAlignmentStrategy which is used to determine how an
Alignment should be processed. For example, you can snap, lerp, or filter and then lerp a transform (such as the
WorldRoot). I've had the best success and most stable results using the
AlignmentStrategy is responsible for averaging the previous alignment headings to determine a better heading match, over time. Additionally, it will not use
Alignments with reported rotations outside of some threshold to reposition the world root transform. This is important because sometimes a GPS update is wrong and gives us a bad heading. If we were to offset our map with this heading, our AR object would appear to be misaligned with the real world.
Note: this implementation is actually a bit of a hack. Ideally, filtering of this type should be done directly in an
ISynchronizationContext. I've used this approach in the interest of time and to keep the example context as simple as possible.
How many heading samples we should average. This is a moving average, which means we will prune earlier heading values when we reach this maxmimum.
We will not use any alignments that report a heading outside of our average plus this threshold to position or rotate our map root. This helps create a more stable environment.
When we get a new alignment (that should not be dismissed), this value represents how quickly we will interpolate from our previous world root alignment to our new world root alignment. Smaller values mean the transition will appear more subtle.
You will need to experiment with various
UpdateDistanceInMeters settings. I recommend keeping your update distance on the higher side to prevent unnecssary alignment computation. The tradeoff, of course, is that you may begin to drift. Which value you use depdends entirely on your application.
You can easily implement
ManualSynchronizationContext so that a user can manually calibrate their "session." One example is to use Mapbox Satellite data to build a map (as is done in the provided scene). You can drag this map or place pins on it to approximate your current physical location. You can also use an arrow to represent which direction are you facing. Note, however, that you will eventually see drift over time if no further calibration is done.
This has been developed with TDD (see
SimpleAutomaticSynchronizationContextTests). However, these tests are quite limited and do not yet fully encapsulate the different scenarios which will inevitably arise in reality. Additionally, while I have done extensive testing "on the ground," I've been in limited, specific locations, with ideal GPS accuracy. I make no guarantees that what is currently provided in this library will solve your problems or work in your area (please help me make it better).
There are various
FIXME tasks scattered around in the
Mapbox.Unity.Ar namespace. Please take a look at these to get a better idea of where I think there are some shortcomings. In general, my implementation so far is quite naive. Hopefully the community can help improve this with new context implementations or more sophisticated algorithms/filters.
There are some ignored tests in
SimpleAutomaticSynchronizationContextTests that suggest potential issues with the initial implementation. Some of these are (wrongly) addressed with the
AverageHeadingAlignmentStrategy, but should be moved to the context.
Solving for UX is not an easy matter. Manual calibration works great, but is not user-friendly (or immune to human error). Automatic calibration works, but still has shortcomings, such as requiring the user to walk x meters before acquiring a workable alignment.
There's a giant
Log button. Use this log to help diagnose issues. If you're seeing lots of red lines (or the alignment just doesn't seem to be working), then something is wrong. Search the C# solution to see what may be the cause of those. If you want, log your own data there, too! You can also use the map toggle to show your paths (AR = red, GPS = blue). If you are aligned properly, the two paths should nearly be on top of one another.
Other issues to note:
With access to Mapbox geospatial data, you can easily augment the AR experience to great effect. Here are some ideas:
roadlayer to construct navigation meshes for your zombie apocalypse simulation
buildinglayer to occlude AR elements or anchor AR billboards on building facades
landuseor custom data (or procedurally place gameobjects based on type)
labellayers to show street and water names
buildinglayer to raycast against for gameplay purposes (ARKit cannot detect vertical planes, but a building could subsitute for this)
What can we do to improve automatic alignment? Here are several ideas:
UpdateDistanceInMetersto more quickly find an initial alignment. Increase this value one calibration is achieved.