Quain Courtyard pt1 Scouting

Quain courtyard... a small outdoor space I found on UPenn's campus, hidden in their Levine building. I technically don't have access to it, which adds an extra layer of fun. I scouted this space because it's semi enclosed, contains a lot of different materials, foliage, and overall interesting atmosphere.

New Term: New goals and research

I wrapped up a few weeks ago after learning a whole bunch about scanning processes. I am actually in the process of condensing everything from my previous posts into a short writeup, which I have found quite therapeutic. I was a bit all over the place and it helps to summarize in a formal way.

One immediate goal I have in mind is to apply to the Thoughtworks / Scatter volumetric filmmaking residency. It is paid (!!!) and requires a project proposal that uses their tool Depthkit. The deadline is April 25th which is very soon.

I have been poking around and evaluating the tool https://www.depthkit.tv/faq

My first concern is, yes, it's volumetric and uses a depth sensor. However, it is not a fully 3D reconstruction. In fact I was a bit underwhelmed by their sample files... rotating more than 5-10 degrees will greatly break the illusion.

At the angle of capture...



oh no

oh noo

The support for multiple sensors is planned but not yet released.

That being said, their resume of artists / projects that used their tool have come up with some stunning works that work with the 'hologram' style capture. Usually by leveraging a tastefully stylized shader (they call "Looks", which are paid features).
There is also no other tool like Depthkit, which makes volumetric video so easy to obtain and use.

--

In other news, I have a rough overview of research goals.

Week 2	Project approved
Week 3	Scout Shoot locations Come up with a couple options, document each
Week 4	Material Scanning Tests Ground Vertical surface Complex materials Decide on location
Week 5	First Shoot Process Pointclouds
Week 6	Reshoot Process Pointclouds
Week 7	Optimize Major assets Skydome Ground planes Large static objects
Week 8	Cont Optimizing Optimize mid range objects Static Props
Week 9	Additional elements Atmospheric effects lighting
Week 10	Flex time/Polish

The Cemetery Hill - Final write up

Above: a screencap of the cemetery hill environment in Unity.

• The fence is a geo asset reconstructed from scanned object data
• The ground is a tileable material asset reconstructed scanned ground data
• Audio is white noise was recorded same day on location
• A particle emitter to mimic seedlings being blown off a tree (documented from that day) to demonstrate dynamic shadows
• Statue is a previously scanned asset (reconstructed from scanned object data) which is NOT true to the location to demonstrate combining scanned assets
• a default cube, for lighting comparison
• Lighting setup is a simple directional light and gradient skybox. 

Critical analysis:

Bad input = bad output

• There are obvious issues in model and texture quality, which can only be solved with better data capture. Admittedly, I began shooting this location in the late afternoon, which made me feel rushed. The benefit was that it was cloudier (therefore, more diffuse lighting) but I relied too much on burst shutter mode and was sloppy in photographing.
• This brings up an important point: I wish I spent more time in the data capture phase of my research for this term. I don't feel closer to perfecting the (already documented) methods, instead I felt rushed to output assets and get back into software side. In the end I didn't feel ready to talk about rendering because I was being sloppy upfront and not testing all the tools I have available to me. Bad input = bad output, and there are no good renders without good data.

The tileable material doesn't look like ground

• Yeah, see above. Need a tripod, manual shutter, and more time.
• There was also a lot of wind that day, which is not an ideal condition.

The statue looks out of place

• This is another case of bad input = bad output, but I can get a little more specific here. First, this was shot on my phone instead of a proper DSLR. The albedo was not completely clean and color matched to the scene, however with more time both of these can be finessed to perfection. In my last post I demonstrate how I can use Substance Designer to achieve a clean albedo, then reuse the template on different assets.

The seedlings are just planes

• I know, it's placeholder. This kind of asset is an example of something that would be better box modeled, and for this demo that's a skill I didn't think was necessary to demonstrate. Plus, the real reason this was included was to show something animated with dynamic shadows.

Successes:

I'm happy with the way the space is taking shape

• The organic feeling (that usually takes SO long to replicate) is almost instant in scanning. The space has a distinct 'character' to it that feels natural. Popping the white noise track in made the whole scene come alive. Plus, the process on this project has been FAST, this is the product of less than a total week of work hours. 

I'm happy with the direction of my research

• I am starting to see the path of how to get from where I currently am to here https://www.youtube.com/watch?v=hCeP_XUIB5U. I think it's important to emphasize that I am still replicating these scanning methods (which Unity has thoroughly documented and published). It turns out that there are already some pretty big shoulders to stand on. 
• I've greatly enjoyed working closer with Unity and realtime art-- something I was never formally trained in and want to continue learning. 
• Posting about my work has helped me get involved in other scanning projects around the school, and there does not seem to be any shortage of them. 

--

Looking ahead to next term, I hope to be able to increase the quality of scans and get results that are closer to the Unity Fontainebleu demo. 

I would also like to define a more specific and meaningful production that would fit into the thesis model.

I am not so worried about producing an effective tech demo as I am about making interesting interactive art. One fear I have is that I am entering my third term without having any discussions about art making and art direction. I hope this is to come before its time to propose our thesis. 

For that reason, next term I am hoping to continue researching best practices in scanning tech, but also look into areas such as installation art, interactive art, VR. 

The Cemetery Hill - proof of tech

One thing that I like about working with scanning is that it is easy to work iteratively and BIG to SMALL. It's easy to quickly generate a sparse pointcloud of the entire space (esp with 360 cameras), then refine per object to smaller, denser pointclouds. Then lastly, each asset can be optimized and made beautiful for a realtime environment. 

I feel good about ending out the term with a workflow that makes sense and can show results in just a couple of days. The Cemetery Hill project is just at the beginning, but the proof of tech can already be presented.

Putting it all together

On my fourth day working on the new dataset from the graveyard-- this process is so fast. 

In my last post I experimented with using control points to merge multiple chunks of data. There were two methods (same steps, different order of operations) and I predicted that recalculating the entire workspace with all cameras included would produce better results. I did that, turned up the settings to high details, and let it process overnight.

As you can see, the results are gorgeous 😍

To compare, here is the dense pointcloud processed with default settings and without recalculating the entire workspace. 

And from another angle

Even more on de-lighting

I wanted to build on my notes from before, to inlcude what I've worked with over the term.

Capture Best Practices

Good input = good output
It helps to shoot subjects on cloudy, overcast days. Or with diffuse lighting.
There is a tradeoff however, sometimes harsh lighting helps generate a better mesh or better surface details.

The stitching software does some amount of color correction. This is all under the hood.

Geometry Dependent Methods

The 'traditional' and more accurate way to delight a base color map would be to totally recreate the lighting conditions, to match all the realworld shadow data, then cancel them out. However, researching other methods has led me to believe that not only is this tedious, it is no longer industry standard.

Image only Methods - stripping light/shadow from base color map

Material Scanning

Notes from this tutorial:
https://www.youtube.com/watch?v=kDMz6djZCkc
https://www.youtube.com/watch?v=dcswz_vYjiQ
https://www.youtube.com/watch?v=1W2VWCehgGg

The end results are technically correct-- tileable PBR maps all exported. They're even de-lit and color corrected.
However, the input data was sloppy. Would like to shoot again with a tripod.

See my notes after the jump.