ISSUE 192: Data Fabric for the Movie Factory

“Kubernetes is all about making software more elastic, and elastic software requires elastic observability; yet we are still stuck using the same old debugging and logging approaches of decades past. We need a new, better approach.”

VMware has certainly seen some changes in its corporate structure during the past few years — but today, VMware’s business moves have been especially relevant for developers. Following its agreement to purchase Pivotal in August, for example, VMware continues to try to adapt its open source contributions and services for the ever-evolving needs of the community, with a greater emphasis on Kubernetes. This means VMware is increasingly engaged in offering and creating the tools and platforms developers need for Kubernetes, as well as for other open source projects.

In this edition of The New Stack Makers podcast: Kubernetes, open source, the developer experience and new technologies. VMware’s role during the upcoming KubeCon + CloudNativeCon conference was also discussed. The guests from VMware on hand to discuss these topics were:

• Bryan Liles, senior staff engineer.
• Tim Pepper, senior staff engineer.
• Tasha Drew, product line manager.

All three guests are actively engaged in the open source community, and particularly, in improving the Kubernetes experience for DevOps engineers.

Data Fabric for the Movie Factory

When we go to a movie theater to enjoy an animated motion picture, such as DreamWorks’ “Abominable” or “How to Train Your Dragon,” we are in effect watching data, pointed out Kate Swanborg, DreamWorks’ senior vice president of technology, speaking in a keynote at last week’s NetApp Insight conference, held in Las Vegas. 

“Behind the scenes, and under the hood, fundamentally DreamWorks Animation is a digital manufacturer. There’s only one thing we make: Data. We make lots and lots and lots of data. It’s the only thing we make,” she said.

And its Big Data at that. Her talk revolved around the importance of data for the movie company. She noted that NetApp storage technologies have been instrumental in helping the company produce every one of their pictures in the past two decades. 

For each character in “Abominable,” for instance, DreamWorks created a “character rig” with up to 5,000 unique animation control points (think of digital marionette strings). Each film will also need dozens and dozens of landscape sets. “Everything you see in those has to be crafted from the ground up as well,” she said. One sequence in Abominable had 3 million plants, every one of them had to be shaded, composited, lit and textured.

Each film, which runs on average about 90 minutes, has 24 frames per second. This adds up to about 130,000 frames, with each frame going through the hands of 12 unique creative departments. The upshot? The process of creating a film spurs the creation of about half a billion files. Keep in mind, it takes the company about four years to make a single animated movie, and DreamWorks must release two films a year.

With these numbers, you can guess how seriously DreamWorks takes its data fabric, the common infrastructure used by everyone in the company to store and share all these files.  In the upcoming months, we will be taking a harder look at data fabrics and DataOps. Share with us your stories about how you manage your own data — the movies of your business.

Take Our Reader Survey and Help Determine 2020 Coverage

The New Stack has continued to grow in 2019 along with the cloud native ecosystem we cover. It’s been a banner year as Kubernetes and TNS turned five years old! Looking ahead, we’re excited for continued growth in 2020 as we expand our coverage into new areas and dig deeper into our established expertise around at-scale application development and management. Your feedback will help us make 2020 even better. 

With your input, we can further improve our coverage and deliver the highest quality articles, podcasts, ebooks, livestreams, videos and more. We want to hear from you!

Extend Your Organization’s Reach with Software-Defined Wide Area Networking

By connecting multiple branches back to the center via low-cost services and utilizing centralized, policy-driven management for dataflows, the emerging software-defined wide-area networking (SD-WAN) will fundamentally change the way your organization operates. Building a business case to adopt SD-WAN requires taking into account the savings realizable not only in costs but benefits in terms of agility, flexibility and simplicity.

Primer: What Is Container Security?

According to the RightScale 2019 State of the Cloud Report, software containers, and containers-as-a-service, are one of the most sought-after new uses of the cloud — and their popularity is growing year after year. If you or your team is interested in using software containers, you’ll need to know about the best security practices you should use to keep the containers and the platforms they run on secure. This post is in a multipart series examining the challenges of securing containers. Check back throughout the month for additional installments.

VMware Bridges Kubernetes and vSphere with Launch of Pacific Beta

At VMworld 2019 Europe in Barcelona, VMware announced an advanced beta program providing Project Pacific to select customers. Project Pacific is designed to enable administrators and developers to design, deploy and manage Kubernetes and container infrastructure in any environment using tools they already know. With the company’s acquisition of Pivotal and the size of its own customer base, CEO Pat Gelsinger told the crowd in San Francisco, “We will be the largest enabler of Kubernetes.”