Quain Courtyard pt 3, Unity

In my last post, I developed ten materials in Substance Alchemist. Now it's time to start putting everything together in Unity.

I should also mention I've switched from working on the big machine to my personal laptop, so my graphics processing power has been cut in three (at least, haha). Still, my 15in Spectre is not too bad. I'd rather be targeting midrange PCs anyways.

The first thing I need to do is export all of my Alchemist materials out to something useable. I notice right away that I can't seem to consistently export 4K texture maps... Alchemist will crash. Although I was able to export two out of my ten materials. Perhaps a bug because it is still in Beta, or maybe my laptop. I notice 4K maps in the viewport will also crash. Either way, I'm stuck at 2K textures for now.
Another frustration is that I have NO idea where Alchemist is saving my working files, so the only way to get assets OUT is through the exporter. So at the moment, I cannot transfer my working documents to a more powerful computer (to rule out the weaker processing issue). To be fair, Allegorithmic warned us not to use their beta release in production, so I kinda asked for this.

Jumping ahead, later on I realize that the Substance Plugin for Unity reads SBSAR files. This is essentially a package of your texture maps. Its much easier / more accurate to implement these instead of the image files. Still though, I am clamped at 2K exports for whatever reason.
--


So now, I have a folder full of 2K SBSAR files. I did some investigating into current Unity project templates. I've been out of the game (pun intended!) for a little while now, so I kind of wanted to work on something more modern (again, probably asking for trouble).

I first looked into the High Definition Render Pipeline, which was used on the Fontainebleu demo (the same demo that published the expert guide on Photogrammetry for games).

I also looked into the Light Weight Render Pipeline, which is not the legacy renderer but instead, a newer and more optimized scriptable pipeline which targets a wider range of platforms, including mobile. Their demos seemed quite pretty as well, plus with some initial testing, ran totally fine on my laptop and played nicely with the Substance shaders. I chose this over the legacy renderer because the lighting is more sophisticated.

I went back and forth a lot here, and I'm still not totally convinced I'm settled into HDRP. I actually have three WIP versions started. One thing I was realllyyy hoping for was to get the shaders as close to the Substance preview as possible. Those sneaky bastards, the height maps really displace the geometry, which is not the case (as far as I know) in realtime rendering elsewhere. HDRP does have a height map channel and claims vertex displacement, but just like normal maps, it does not seem to alter the silhouette


The deciding factor was ultimately, that I really wanted the lighting quality and to follow the expert guide.

So now, I drag drop my ten SBSAR materials into my Unity project.


Notice the pink? So, the Substance plugin brings in your material as a standard shader. The standard shader belongs to the legacy renderer, and canNOT be rendered with the HDRP. So I need to open up each one and change it to the HDRP "Lit" shader, set the quality to 2048, and double check the maps. Most cases I had to plug them all back in for some reason.

Unfortunately, doing this breaks the thumbnail in the asset bin. :-(
This does not affect the material.


Concrete is expanded to show the HDRP material nested in the SBSAR ^^



For the frosted glass, first the Material Type needs to be set to Translucent.
Then, the base color needs an alpha value of <1. Simply click on the color picker icon and slide the A slider down to taste.



Another setting I really like in the SBSAR properties is the ability to switch between UV and Triplanar projection. Triplanar works great for static meshes, and essentially takes away the need for UV unwrapping with the click of a button.

I mock up a quick greybox version of the space based on the layout geo (generated with the 360 camera)

Honestly I could have done this step in Unity, but I work faster in Maya. I limit myself to about 30 minutes on this step. I use Maya's Game Exporter to export all objects as seperate fbx's (check plugins list if you don't see it under File).

I import back into Unity.
Couple of noob notes for myself RE moving an entire scene into Unity

Maya to Unity:
1. do not freeze transforms in Maya (Modify>bake pivot to get transforms back)
2. export all separately using Game Exporter
3. import into Assets. select all and check the Convet Units checkbox
4. drag into hierarchy. Everything should be in its place


Again, first import the rough layout geo... (iterative design!)

Then everything else. I'm glossing over a few steps here, but I did a quick pass at applying all the materials and tweaking them a bit.

and this is a screengrab, without the layout geo.

A couple of obvious things stick out right away. 

First, there are obviously tiling materials. This is a little in part that they are placeholder, and I'm using them on objects that a much larger than what I intend for the final assembly. 

Still though, I had a lot of problems with scale. I had to manually tweak each material to look right. I realize that I really needed to take real world measurements and plan ahead for texel density. Unity says to take a 1 meter square of any material scan. I grabbed whatever I could get. 

Second, my placeholder geo kinda sucks. haha. I would have been better off with cubes I think? It's actually related to the scale issue, I added bevels without thinking about how noticeable they look.

The postives are, its starting to come together! The rendering and post processing are really really nice. It does't look like a typical "unity student project" that I have noticed is really common in Unity games (even Overcooked has symptoms of this... something about lacking antialiasing, color correction, dinky looking shadows and shaders that falloff to black? I can't totally put my finger on it but there's definitely a look to the legacy renderer.)

Walking around the space is starting to feel like the space. Getting up close to the bricks is actually kind of exciting. 

Next post I'm going to implement the scanned assets... stay tuned!

Quain Courtyard pt 2 scan material processing

I had a successful shoot day!
It was rather bright outside, so I had to deal with a lot of harsh sun and directional shadows.
I took a lot of notes as well, and recorded sound.

First I wanna talk about Photoshop's 'Photomerge' function... it's a tool I've been using more lately especially with Alchemist. Its a simple photo stitching feature. Under File>Automate.
It's gotten better than what I remember 5-10 years ago. It keeps all photos on separated layers with masks, and looking at the chunky shapes, the stitching is rather sophisticated.

I would like to, in the future, use a tripod and carefully measure out and scan a large area for material capture. However, I got successful results from just a handful of photos (5-8 or less). I can also repurpose the photogrammetry batches of photos (intended for geo reconstuction).

 

I was able to make this material kit in about a day. I love Alchemist's "Generate Board" feature, which quick exports your material collection into a thumbnail sheet.

The four slate textures were especially fun... in the courtyard, there are four types of slate tiles. I started with two types of slate, pulled from photos. Then used their "Pavement Generator" to define two types of tile patterns.

There are still some frustrations with this Beta release... it will crash on saving, and is a little slow on my laptop.

Something important to mention was that part of my capture day I took a couple of 'reference photos', wide shots of the area or in general, any photo that documented the qualities of the space and were not intended for reconstruction. These were especially valuable, as it helped inform some of my color choices.

I've been feeling a little overwhelmed about asset generating, but I recently rewatched the Star Wars Battlefront GDC talk. Their level kit contained like, 8 assets. They were simple enough that they could be scaled/rotated and repeated throughout the environment. Quality over quantity! And my favorite quote is still that the experience of going on location was more valuable to the artists than the data itself. It's time we stop making CG art based off of Google Images.

More on Material Scanning... with Substance Alchemist!

I'm very excited to talk about Substance Alchemist, it has really solved a huge chapter in my reality capture explorations.

The methods I had previously researched for material scanning were tedious and required a lot of manual labor. Out of all the scan based material authoring methods, I can generally group into two procedures: one is using only photo data, and the other is using geometry from photogrammetry.

Photo-only: https://www.youtube.com/watch?v=kDMz6djZCkc
Geo baking: https://www.youtube.com/watch?v=HMPZV_Se8W0



I had plans to compare geo based workflow to photo based workflows... and I was getting stuck on the amount of time needed. To pull maps off of a photogrammetry reconstruction, I need to re-topo and UV a clean mesh first. This is fine for one or two materials, but to process more than that would be a lot. 

The other documented method recommends building a scan box to control light, which is cool, and provides a lot of accuracy. If I was scanning a large batch of say, fabric samples that are all roughly the same size, this would be awesome. 

It's DIY but it's also pretty complicated. 
However, applying it to shooting on site, usually outdoors, sometimes vertically... I'd have to engineer some kind of portable scanbox and probably have an assistant to help with lighting. 

Throughout all this research, I came across Substance's announcement for their new Project Alchemist, which threw all the above out the window. 

I took my baked brick photo texture, literally drag and dropped it into a new document, and immediately got this:


A few minutes playing with the auto tile feature...



At around 20 minutes I had this. It's by no means perfect, but holy shit it was FAST. And easy to pick up.  Everything in the Alchemist document is parametric, so I could revisit and re export. (Like any Substance product, I can export all the PBR maps as image files.)


Here's the thing... one of the reasons it wasn't perfect was because I was using a re projected texture map rather than a straight photo (above) so the data already wasn't clean. If the bricks weren't distorted, I think the material would be ready to use.

So I took a few more minutes and generated more.





Here I'm correcting the perspective in Photoshop first.




Alchemist has a feature which generates variants based on color pallets pulled from images. I could see this potentially being used to color correct a large batch of materials with an existing pallette. It's also fun.


Here I gave myself a challenge to replicate one of the materials using NO imported images and only the built in, parametric tools. This is how far I got in about 10-15 minutes.


Lastly, I tested the maps in Unity with its standard shader.
 

It's never as pretty as in the Substance viewport, but the import works and is very straightforward. I'll have to play with Unity shaders a bit in the future.

Hologram Portraits

I got to show off my scan portraits the other day on the Looking Glass holo display. I used a pre built app to display the models, and paired with a Leap Motion, I could pinch and rotate/zoom in on the models. 

It was really great to share this with others, especially when it was someone interacting with their own portrait. Sometimes I get really lost in the bigger projects and research ideas, so it was a good feeling to come back to simpler ideas like these portraits. Everyone loves a portrait. They're so fundamental and human.

The only issue was running the display off of my laptop, which has an NVIDIA graphics card but still lagged. 

To prepare the models, I first decimate the mesh to under 1000 verts (about 300 polys). I do this right in Zephyr.

Quain Courtyard pt1 Scouting

Quain courtyard... a small outdoor space I found on UPenn's campus, hidden in their Levine building. I technically don't have access to it, which adds an extra layer of fun. I scouted this space because it's semi enclosed, contains a lot of different materials, foliage, and overall interesting atmosphere.

New Term: New goals and research

I wrapped up a few weeks ago after learning a whole bunch about scanning processes. I am actually in the process of condensing everything from my previous posts into a short writeup, which I have found quite therapeutic. I was a bit all over the place and it helps to summarize in a formal way.

One immediate goal I have in mind is to apply to the Thoughtworks / Scatter volumetric filmmaking residency. It is paid (!!!) and requires a project proposal that uses their tool Depthkit. The deadline is April 25th which is very soon.

I have been poking around and evaluating the tool https://www.depthkit.tv/faq

My first concern is, yes, it's volumetric and uses a depth sensor. However, it is not a fully 3D reconstruction. In fact I was a bit underwhelmed by their sample files... rotating more than 5-10 degrees will greatly break the illusion.

At the angle of capture...



oh no

oh noo

The support for multiple sensors is planned but not yet released.

That being said, their resume of artists / projects that used their tool have come up with some stunning works that work with the 'hologram' style capture. Usually by leveraging a tastefully stylized shader (they call "Looks", which are paid features).
There is also no other tool like Depthkit, which makes volumetric video so easy to obtain and use.

--

In other news, I have a rough overview of research goals.

Week 2	Project approved
Week 3	Scout Shoot locations Come up with a couple options, document each
Week 4	Material Scanning Tests Ground Vertical surface Complex materials Decide on location
Week 5	First Shoot Process Pointclouds
Week 6	Reshoot Process Pointclouds
Week 7	Optimize Major assets Skydome Ground planes Large static objects
Week 8	Cont Optimizing Optimize mid range objects Static Props
Week 9	Additional elements Atmospheric effects lighting
Week 10	Flex time/Polish

The Cemetery Hill - Final write up

Above: a screencap of the cemetery hill environment in Unity.

• The fence is a geo asset reconstructed from scanned object data
• The ground is a tileable material asset reconstructed scanned ground data
• Audio is white noise was recorded same day on location
• A particle emitter to mimic seedlings being blown off a tree (documented from that day) to demonstrate dynamic shadows
• Statue is a previously scanned asset (reconstructed from scanned object data) which is NOT true to the location to demonstrate combining scanned assets
• a default cube, for lighting comparison
• Lighting setup is a simple directional light and gradient skybox. 

Critical analysis:

Bad input = bad output

• There are obvious issues in model and texture quality, which can only be solved with better data capture. Admittedly, I began shooting this location in the late afternoon, which made me feel rushed. The benefit was that it was cloudier (therefore, more diffuse lighting) but I relied too much on burst shutter mode and was sloppy in photographing.
• This brings up an important point: I wish I spent more time in the data capture phase of my research for this term. I don't feel closer to perfecting the (already documented) methods, instead I felt rushed to output assets and get back into software side. In the end I didn't feel ready to talk about rendering because I was being sloppy upfront and not testing all the tools I have available to me. Bad input = bad output, and there are no good renders without good data.

The tileable material doesn't look like ground

• Yeah, see above. Need a tripod, manual shutter, and more time.
• There was also a lot of wind that day, which is not an ideal condition.

The statue looks out of place

• This is another case of bad input = bad output, but I can get a little more specific here. First, this was shot on my phone instead of a proper DSLR. The albedo was not completely clean and color matched to the scene, however with more time both of these can be finessed to perfection. In my last post I demonstrate how I can use Substance Designer to achieve a clean albedo, then reuse the template on different assets.

The seedlings are just planes

• I know, it's placeholder. This kind of asset is an example of something that would be better box modeled, and for this demo that's a skill I didn't think was necessary to demonstrate. Plus, the real reason this was included was to show something animated with dynamic shadows.

Successes:

I'm happy with the way the space is taking shape

• The organic feeling (that usually takes SO long to replicate) is almost instant in scanning. The space has a distinct 'character' to it that feels natural. Popping the white noise track in made the whole scene come alive. Plus, the process on this project has been FAST, this is the product of less than a total week of work hours. 

I'm happy with the direction of my research

• I am starting to see the path of how to get from where I currently am to here https://www.youtube.com/watch?v=hCeP_XUIB5U. I think it's important to emphasize that I am still replicating these scanning methods (which Unity has thoroughly documented and published). It turns out that there are already some pretty big shoulders to stand on. 
• I've greatly enjoyed working closer with Unity and realtime art-- something I was never formally trained in and want to continue learning. 
• Posting about my work has helped me get involved in other scanning projects around the school, and there does not seem to be any shortage of them. 

--

Looking ahead to next term, I hope to be able to increase the quality of scans and get results that are closer to the Unity Fontainebleu demo. 

I would also like to define a more specific and meaningful production that would fit into the thesis model.

I am not so worried about producing an effective tech demo as I am about making interesting interactive art. One fear I have is that I am entering my third term without having any discussions about art making and art direction. I hope this is to come before its time to propose our thesis. 

For that reason, next term I am hoping to continue researching best practices in scanning tech, but also look into areas such as installation art, interactive art, VR.