Build your own Augmented Reality Sneakers with Blender, Procreate and Lens Studio
In your quest of creating AR Fashion, you’ll probably be asked by clients to replicate physical fashion in AR. This article is meant to help you in doing so. Starting with a must: Sneakers.
In this article we’ll be covering how to create your own AR Sneaker and prepare a 3D model in Blender, paint its textures on Procreate 3D and make it into an AR filter using Lens Studio.
For our example, we’ll be making use of a 3D sneaker shoe we downloaded on CGTrader and turning it into the foot tracking AR filter you see below :
Here is the list of programs used to make this experience possible :
- Blender — free 3D modeling and animation software
- Procreate 3D —iPad only graphic design app
-important to get version 5.2 or above- - Lens Studio — free AR software by Snapchat
1° Preparing the 3D model in Blender
UV unwrapping
While this step may be optional, for models downloaded online it’s recommended that you import them in Blender first before importing in Procreate. The reason for this is to make sure the model is properly UV unwrapped as well as removing any potential junk in the scene (lights, cameras, etc…). Procreate will only be able to paint in 3D if the object has proper UV(s).
If the object doesn’t have a proper UV layout, a first option is to try the Smart UV Projection feature in Blender. You’ll be able to tell if your UV layout is useable by inspecting its corresponding 2D projection (seen on the left below).
Exporting the model
Once the UV unwrapping is done, we can export our model. Procreate supports .usdz and .obj file formats so we’re going to export our object as a .obj.
— Note : Why .obj and not .usdz format?
Currently our model does not have any materials or textures that we want it to carry over, so we have no need for the more complex .usdz format.
The .obj file format is an older format that is still widely used and accepted by most if not all programs, however it doesn’t transfer any texture or material properties, just the 3D mesh information. The .usdz file format on the other hand is a newer format used by companies like Apple and Pixar. This format properly exports not only the mesh but also its materials and corresponding textures. However very few programs support this new .usdz format natively (you’ll need a special add-on to export in this format with Blender for example).
— Tip 1 : Web converters such as https://spase.io/playground can be used to convert .glb files to the .usdz file format.
— Tip 2 : The latest version of Blender 3.0 can import .usdc files! If you have a .usdz file, simply change its file extension to .zip and it will automatically become a .usdc file.
Now that our model is ready and exported, let’s head over to Procreate for the texture painting.
2° Painting textures in Procreate 3D
Importing the model
There are different ways to transfer your model to your iPad. Here we’ve used the Google Drive app. Once uploaded from your computer, you can download you .obj file on the iPad, then drag and drop it in Procreate 3D. A project will open automatically.
3D texture painting
If you’ve ever used Procreate before, the steps from here on out will be very similar to the 2D painting process, with the difference that we can now choose to paint on the diffuse, the roughness or the metallic values of our materials!
The diffuse value of the material will determine its base color. The roughness and metallic values are both represented in black and white and will affect how the material reflects light.
Similarly to the 2D workflow, you can add and/or remove layers in the new 3D workflow, as well as change blend modes of each layer.
— Tip : There is unfortunately no option to mirror your drawings while painting in 3D yet. In order to do work around this, you’ll have to paint/duplicate the layer you want to mirror in the 2D view, with the 3D window serving as a preview.
If you had prepared artwork for your 3D model prior, no problem! You can insert images in the 2D view like before. From there you can manipulate the layer’s position, rotation and scale in the advanced options to make sure it’s applied where you want it to be.
With this knowledge regarding materials and layers, we can finally paint our model! In the timelapse below we’re mainly painting the roughness and metallic values to create more interesting surfaces. It’s very satisfying to paint a metallic shine on an otherwise non-reflective surface!
If your model ends up with several materials making use of their metallic values, it’s worth experimenting with the environment light options and the light sources. This will drastically affect the look of your object and has a big impact on look and feel.
— Tip : To move the lights in the scene, keep your pen pressed down on them and then move around
Previewing in AR
Before entering the AR logic work it might be nice to see how the textures you just painted looks in a real world setup with natural lights and environment. And Procreate 3D has a handy built-in AR preview for you to be able to do that.
Once saved to your device, you can send the file (.usdz format) to clients who can open it with any iPhone and view the model in AR themselves in order to validate the look before you start the AR logic work.
If you want to know more about Procreate features for professional AR design, here’s an article on the topic!
Exporting the model
Once the model has been previewed in different environments and in AR, we can finally think of exporting it for use elsewhere. For the purposes of this article we need to look at the import requirements for Lens Studio, which doesn’t accept .usdz file formats. We’ll need to export out our mesh as a .obj or .fbx file and export our textures separately.
Finetuning the model
Before rushing into importing our object into Lens Studio, a good idea would be to check its scale. In order to do so, we’ll download the feet mesh object found in the Foot Tracking template of Lens Studio and import it in Blender. We can now also import our shoe .obj/.fbx to compare the size.
— Note : the Foot Tracking template can be found here : [link]. Open the project, save it on your computer and you’ll find the model in the project’s folders
It’ll generally be easier to adjust this within Blender and re-export the mesh. As long as you don’t change your UV’s, there will be no additional texture work required.
— Note : If you want an effective occluder for your shoes, it’s worth creating one yourself. You could re-use the feet mesh from the template here and simply scale them accordingly
Once your model is scaled to size, export it as a .fbx and save it in the same folder where the textures lives. This will allow Lens Studio to import the object with its textures directly -you’d otherwise have to re-assign the textures to their corresponding materials-
— Tips :
- It’s recommended to compress your textures prior to importing them on programs such as Lens Studio. This is due to the low file size limit imposed for AR effects. A good online compression tool is TinyPNG.
- If online compression isn’t enough, you can always manually compress your images with 2D editing software like Photoshop and Affinity Designer. Another viable route is to reduce the canvas dimensions.
- Textures from Procreate can be quite heavy (especially if set at 4K). Sometimes it might be worth using a built-in material from Lens Studio’s Asset Library instead to save on performance.
3° Foot Tracking in Lens Studio
In this final part of the article we’ll be taking our 3D model as well as the hand-painted textures we made in the prior steps and creating an Augmented Reality filter that will tie the shoe to our user’s feet.
Importing in Lens Studio
In Lens Studio we’ll choose the Foot Tracking template as a starting point for our project.
From here we’ll import our .fbx into Lens Studio via the File > Import menu. Once our mesh is in the Scene (aka in the Objects panel), we can quickly work on repositioning it to fit around the pre-existing feet mesh from the template.
We can also notice that our sneaker doesn’t look quite like the preview we had in Procreate. This is because different software interpret material properties differently. Basically, we’re gonna have to finetune/recreate our material setup inside Lens Studio. This involves creating one material in Lens Studio per material in the original Blender file, assigning the proper textures to them and playing with the material properties to match what we had configured in Procreate as best we can.
Metallic and Roughness values in particular do not get transferred in Lens Studio which means we’re going to manually adjust them in the Material Editor.
In order to get the same look as in Procreate 3D, a good practice would be to have a rendered image of the shoe next to you while you play with the material settings. Try also matching the lighting setup from Procreate 3D inside Lens Studio as well as finding a matching HDRI map. As mentioned previously, these dramatically affect the look of the model.
— Tip : Check for free HDRI’s here
Below is a comparative picture of our AR shoe in the different environments :
A final step, only relevant to this specific AR shoe filter, is to position our AR shoe correctly in Lens Studio’s hierarchy. The template makes things very easy for us in that regard, as everything is named properly and there’s even a documentation for it here.
Once we’ve rebuilt our materials and repositioned our mesh in Lens Studio, it’s finally time we preview this in AR!
Let’s see how it fits!
Previewing your effect from Lens Studio is fairly straight-forward once you’ve paired your device with the program. To start, just hit “Preview in Snapchat” in the top right corner of Lens Studio. A QR code will show up that can be scanned with the Snapchat app, automatically redirecting to the Snapchat login. Once logged in, we’re good to go!
With our device paired, the option in the top right will have changed to “Send to Snapchat”. Whenever you feel like a testing round is needed, you can now hit the button to have our preview sent directly to our Snapchat app.
Ok seems like it works, let’s offer it to the world.
Publish your effect and share it!
The steps for publishing are also quite straightforward, you can find the Snap AR documentation for submitting an effect here.
Don’t overlook the confirmation step, the icon, demo video and tags of your effects will have a big role in how people find and share your effects!
Hit “Submit Lens” and there you go! Your AR Sneaker will be reviewed and if all goes well should be published live a bit later and visible on your Snap AR portfolio.
AR Fashionista Hint: You can also wear these or any piece of AR Fashion on your Zoom calls by using Snap Camera 😎
That’s it for our process to create an AR Sneaker, we hope you’ve enjoyed it.
Please let us know your thoughts for improving this article in the comment section!
.
.
.
What’s next?
We’re currently working on a bunch of new articles you might like!
- Three pieces covering an AR Fashion business case
- A tech case reviewing the Multipeer technology from Spark AR and the revolution it brought from the user’s psychology standpoint
- A review of a VR cognitive game spec'd by two famous European neuroscientists we worked on last year.
Another exciting news regarding this publication is that a couple of like minded creators from our community have rolled up their sleeves and will start sharing some of their magic in the tech neutral culture we’re building so you’ll start to see new faces over here!
We’ll also release a suite of 3 articles about AiR, a real-time air quality visualization tool touching how and why we made it, our technical roadtrip and a UX thinking walkthrough
AiR is our first ever piece of proprietary content, we made it to help kids and public’s understanding of how air pollution works hoping to spark sustainable oriented ideas :)
Don’t miss a piece of it by subscribing on the form below.
You’re an AR creator? Join a tech-neutral AR culture.
Get support from our team and like-minded creators while building your own AR Sneaker or any kind of AR work (Spark AR, Lens Studio, 8thWall or any AR/VR software) on the Lab.
Credits
Workflow research: Sarah Mayer
Article: Boris Josz, Laszlo Arnould
