Can you talk about the various trade-offs between working with the private sector first and the government second?
The corporate contacts we've had in the last three decades have been very focused on research and development. Basically, the agreement is that there's a certain topic and expertise we can deliver; then we can develop software for their needs. It’s a matter of ability integration. They recognize they don't have the innate human resource talent and skill set so it's efficient for them to delegate these missing components to us. These research contracts work pretty well but it's a buckshot approach because sometimes we get a contract, sometimes we don't. We can't have any kind of reliable long-term funding.
It is very difficult to get reasonable funding from the entertainment industry. They will argue, and probably honestly, that the computer graphics practitioners at their studios are actually overworked relative to what they're paid.
So, by the time you look at a blockbuster movie that makes mega dollars the graphics has been done essentially on a fixed fee. It's done. There’s some argument that there’s no leeway, there’s nothing to give away from any profit. Whether that's true or not, we never received funding from Pixar and Disney or any other studio.
Are some of the things you develop sold to private industry? I know, for example, that there's a technology transfer department at Penn. What are some of the logistics? How do you balance the creative and education side with the possibility of making money for the university?
Our intention is not to make money. That’s not our goal. Rather, for us, computer graphics is a door that leads to interdisciplinarity. I’m pretty certain about this claim: That computer graphics, as a technology, has the shortest lifecycle from generation to deployment amongst any industry. That, coupled with the fact that computer graphics people love to show off their work, means that as soon as it’s published, anyone can recode and use it.
The cycle time from publication to deployment of computer graphics in somebody else's software in a movie is about six months. That doesn’t leave any room in the cycle for the traditional technology development that's patented or protected. Trying to keep the technology to ourselves so we charge money just doesn't work.
My best example of that was a student here in the early 2000s. The student, Nick Foster, did his Ph.D. on fluid simulation - simulating water in tanks. So, he gets his Ph.D. and moves to DreamWorks. DreamWorks is in the middle of making the movie Antz where the climax of the movie is a flood in the ant colony. The original script called for artists to draw the water with traditional methods. They hired Nick and they said: hey Nick, you know - fluid simulation might actually work here, right? Nick recodes his fluid solver for DreamWorks. It looks great. It appears in the movie Antz six months later.
A year later, Nick gets a Motion Picture Academy Technical Achievement Award. So, here's a guy who was literally rocketed to the pinnacle with his own technology in one year. That's not atypical.
So, back to the question. Yes. We work with the tech transfer office. When we think we have good ideas, we file disclosures. But 90 percent of those disclosures never amount to anything.
We've been very fortunate. At least two of our lab products have become corporate products. The first is the Jack human simulation software that was developed here years ago. The second is our MarchingOrder graduation software concept.
When I became Associate Dean of the Engineering school, I had a great fear (full circle here) that I now had to face going to every graduation ceremony the Engineering school had. Since high school, I hadn’t gone to any graduation ceremonies.
I feared I couldn’t sit through these things. I then used fear to think that maybe the ceremonies could be more interesting if we had a big screen that would display a picture and a message and maybe the major every time a student went up to receive the diploma. This way, 99% of families who wait for their kid’s name to be called could be entertained seeing who the other students were.
My younger son David, who was a student here, and a couple of his Wharton colleagues put together the software. They started running it here at Penn.
Then my son decided to opt out of the company. But one of the colleagues, Tyeler Mullins, still runs the company. The software, called MarchingOrder, is now used at graduation ceremonies all over the country.
Can you talk about your involvement with the military and government?
Our first real government contract was with NASA. I worked with NASA for almost 30 years. They were fantastic. Everyone always felt that with NASA there was some humanistic and scientific good because we were actually involved in the space program. The reason we were involved is that the shuttle program had just started in the late 70s. They were mandated by Congress to make the shuttle flights accessible to nearly every American who was qualified to fly.
Well, when you have that kind of criteria, all of a sudden you need to cover a full range - from very small men and women to big athletic tall guys. That's not how aircraft are designed. They’re designed for a very tight fit. So, they came to us and we worked to create a system to try to check out the ergonomics of these shuttle spaces.
After about a year, we delivered the first workable product. There were rough edges but they actually used it for the shuttle toilet redesign. The last time I visited NASA they were still using a future generation of the software because it was really solving problems on the spaceflight side.
On the military side, the work we did for NASA was noticed through our publications. It came to the attention of folks in the Army, Navy, Air Force, and their contractors. They gradually all realized that our human modeling software, called “Jack”, could help to design their vehicles and work places. Our Jack software was supported by all these organizations.
There was a problem, though. It was good to work with the military because there was steady research income. But, sometimes, they would say – if I pay for it, how come I don't own it? I'd have to say - well the Army is paying for this feature but the Air Force is paying for that feature and the Navy is paying for this feature. And, oh, by the way - you get all their features. That's where Jack had its origins.
We then started to use our Jack software to accumulate some industrial money. We got to work with some very good companies - Lockheed Martin, John Deere, Caterpillar, General Motors, and Ford I think. They were all paying license fees though we weren’t getting rich off of them.
By 1996, Jack finally had to leave the university. If you asked me in 1996 what I did, I would honestly answer: I run a small business inside a university. Jack had become a massive undertaking. We had four full-time staff, 24 Ph.D. students, three international distributors and an annual user’s meeting. Even though we were getting good money, it just became a software enterprise and the university was not the place for it anymore.
I think you sort of answered this question before – but what are the corporations and government looking for when they hire students?
Students with advanced degrees end up going into academia or into companies that actually do have a research enterprise. The companies in that category have changed a lot over the years. It used to be Bell Labs and IBM were big consumers of Ph.D. graduates. Now, companies like Google or Apple are hiring them. Many Ph.D. candidates do want to teach. They’re in it for the long run.
Do you work with any Philadelphia institutions such as the Franklin Institute Science Museum or the Philadelphia Museum of Art.
No. We had one joint project with the Franklin Institute many many years ago. We have an open mind about those things but there are problems such as working through the intellectual property issues.
It’s easier to say - look I've got a whole pool of great students here. If you pay them, we'll provide expertise and then you can own everything.
That's worked very successfully with Children's Hospital. One of our students, Warren Longmire, went to develop games for autistic children. They just hired the guy and were really happy with his work. That kind of model works better for us.
When I first started working with computer programs, I began with a broad outline and then filled in the details. Are there current computer programming methodologies?
I think there are probably three standard ones. The first one is if someone has written the software you need to go and buy it or if they publish it then reproduce it. Is there a solution that one can use or adapt?
The second is that most computer graphics programming is done in C++. Our DMD students need to learn C++ while regular computer science students may not need to learn that particular language. C++ is kind of the lingua franca for computer graphics.
The third paradigm is very interesting. It’s called visual programming. The best example of that is the interface system called Blueprint to the Unreal game engine. Blueprint uses a visual paradigm where you have blocks and literally connect them with curved lines. You build your program visually by laying out blocks that generate data, provide timing, give you color. Then you wire those blocks together - in a dataflow organization - to actually generate animated 3D graphics on the screen. It’s very different than the linear textual programming paradigm that we teach in computer science.
Can you explain the interplay between software, hardware, electronics, and computer theory?
I will restrict my response to computer graphics. The technical folks in computer graphics have pushed very hard for efficiency. If you can create one image in five minutes, that’s not good enough. You want to create that image in five seconds. And even that's not good enough. You want to create it a 50th of a second so you can start to animate it. There's a constant pressure to make things faster.
Concomitant with that is sort of the architecture of computer graphics. The algorithms that make images have become much more widely understood and have coalesced in a few different paradigms. The principal paradigm actually ends up looking like a pipeline. Very early on – 1981, over 25 years ago – embedding computer graphics algorithms in hardware started to be a thing. That was carried forward by Silicon Graphics for a long time.
Then someone smart said well we don't need to buy all this other stuff. Why don't we just embed the computer graphics on a board? The first computer graphics boards were built and slotted into IBM PCs: a self-contained board that you could just slide in.
That spelled the death knell for special purpose graphics hardware. It led to the development of graphics processing units (GPUs). They were initially optimized just to do graphics but now they have been generalized in ways where they do all sorts of other computations.
A principal company that does these graphic boards is NVIDIA, which now thinks of itself more as an AI company. Their hardware is used in self-driving vehicles because they’re processing images. They do deep learning. The hardware is used to mine cryptocurrencies, too.
But graphics was the driving reason for this hardware. The graphics processing unit as a piece of hardware is programmable. Its architecture is different from a CPU (Central Processing Unit), used for general computation.
You do have to learn how to talk to it in the right way, but the capabilities are so phenomenal. It's truly a parallel processor. It's the most powerful computer in your workstation; more powerful than the CPU. Many supercomputers are actually built by just taking hordes of graphical GPUs to gain parallelism. So, we've impacted the field that way.
The answer to your question is, therefore, it is important for our students to understand that there are CPUs and GPUs, and they need to communicate and interact. We have at least one course here at Penn, taught by Patrick Cozzi, that really gets deeply into programming the GPU.
Can you explain your work with crowd simulation? What applications are there for it?
We did work in this area for about 10 years but we’ve moved on. We moved on because crowds are too homogeneous. We’re more interested in places where people are more differentiated.
The crowd simulations were generally used for evacuation and disaster panic. That's the most common real application of crowd simulation.
Crowds are an interesting problem from a computer science perspective. You don’t want individuals bumping into each other or passing through walls or each other. So, there are lots of computational problems that can blow up when you have large numbers of individuals.
After producing four books on this topic, I decided to leave it aside.
Was your software ever used for the medical school?
Almost. What we had briefly was a project from the Advanced Research Projects Agency to build a simulated medic. The idea was to help train medics for their first encounter with a real injury. We collaborated with a top Philadelphia trauma surgeon, John Clarke. He was our expert. We actually built a physiological model of a human to allow that model to respond to injury and interventions.
The set up was basically - you would see this guy lying on the ground. The medic would have to figure out what tests to do and what interventions to apply in order to stabilize him. You get five minutes. And if you don't do the right test or you don't do the right stabilization, your patient will die.
It was a real nail biter for people who tried it. We did speed it up, however, to make the experience even more compelling. For example, if you failed to notice that the guy had a tension pneumothorax, he would suffocate. You needed to put that chest tube in to let the air out. We could even change the skin color, show distended veins, and animate the chest breathing abnormally.
That project lasted for about three years and then it was moved to the Sandia National Laboratory. We were then no longer part of it though they continued it for years.
How much of what you do with computer graphics impacts or is impacted by other computer fields? Let’s start with artificial intelligence.
My own position in the world has often been in-between computer graphics and artificial intelligence.
There's a long history of Penn’s computer science department working in collaboration with other departments like psychology, linguistics and philosophy to do what's generally called Cognitive Science work. For many years, computer graphics was part of that broader community.
Many of my closest collaborators have been Natural Language understanding people: Bonnie Webber, Mark Steedman, and Martha Palmer. I maintain an existing collaboration with one of them as well, Ani Nenkova.
We are simply interested ultimately in why humans do what they do. You can't study that in a vacuum.
So even though I'm not into formal reasoning and deep learning (a new thing in AI), throughout my whole career we often had papers in what's called animated agents or embodied conversational agents. These are human- or character-like virtual beings that you could talk to or interact with. We have a long and steady thread in that domain. It got us involved in facial animation, eye movement, micro-expressions, and understanding gesture.
So, yes, I think that AI connects really well but it's so big and machine learning-oriented that I do keep at a distance.
What about computer graphics and 3D printing?
I don't do anything in that vein. Another faculty member at Penn does do 3D printing. In my opinion, 3D printing has been greatly benefited by the research done in computer graphics. The difference is that our products are just images, while they print 3D “images” using mechanical processes. Some of the nicest work that we see at the annual computer graphics conference involves fabrication. There’s a tight link with the broad computer graphics 3D modeling community though I don’t work with 3D printing myself.
Are there any other computer fields where computer graphics and that other field intersect?
Computer graphics and computer vision overlap a lot. The primary reason is that computer graphics at its core is about building 3D representations of stuff. A big component of computer vision is having sensors like cameras look at the world and try to build 3D models of stuff. So, computer vision becomes a very convenient though challenging way of getting complicated stuff into a graphics representation for further reasoning and use. Those two areas are very close.
How many women are now in the program?
That's a great question. DMD is two-thirds women. Our master’s program is about 35% women. Of the Ph.D. students I’ve worked with, about 20% have been women.
Our DMD Associate Director Amy Calhoun is a great asset both as a former Penn admissions officer and as our liaison to industry recruiters.
We're very encouraged that the DMD numbers have a good majority of women. I think it helps that being artistic is an important part of our program. It’s important enough to motivate learning the technology. What we say with DMD is that there's a main course of computer science but you get a side dish which is the art. It’s a nice way to show off their artistic and creative sides.
Is the Digital Media Design program still working with the communications school?
That's an historical artifact. We still suggest that people take a communications course. This goes back to something I said earlier which is if you just do computer graphics for its own sake, it doesn't really have any storytelling power. If you're going to do computer graphics, there ought to be a reason and that reason is to communicate something.
Any advice for high school students who want to get in your programs?
As I said at the start - figure out what you're afraid of. I’m encouraged that the exposure to computer graphics now is fairly ubiquitous.
When I speak about the difference between computer engineering and computer science, I say well look at your phone. If you're more interested in running apps on it and figuring out how the apps work, then you're probably going to be a computer scientist. If you're more interested in how that thing works at all, you’re a computer engineer.
Do you have any thoughts on using computer graphics to envision a future 30-50 years down the road?
When we had the turn of the millennium, I was asked to write a short piece for the major computing magazine: The Communications of the Association for Computing Machinery (ACM). They were doing a series of articles. They asked me to write on what computer animation would be like a thousand years. That's impossible to answer, so I cut it back to like 50 years.
But there is a vision and it's a vision that I see each year being more and more realistic though we’re not quite there yet. It's my belief, though not my idea, that at some point displays will become just about as cheap as construction sheetrock. At that point, there'll be no reason to build buildings and walls the way we do now. Instead of paying for painting and design, we'll just put up display screens everywhere.
First, we can constantly change our surroundings - changing decor as our moods or seasons change. I want a window there, no I want that window here. The interesting part of that is not actually the technology. The interesting part is how do I make all this work so that it becomes a livable environment. I don't quite want to do Minority Report stuff where I have to go up to a display wall and start moving things around manually. And it's not quite the Alexa model where I just speak - let's do this and it just happens.
The environment itself must be much more awareness of my everyday activities - where I am and what I'm doing - so that the environment can be responsive to my needs too.
How many times have you taken a call and walked into another room and realized you know you left behind the information you were looking at? Well why doesn’t that information just follow you as you take your call? These are all user interface issues and I think they're interesting. My colleague Stephen Lane and I have talked about setting up a room like this but it's still too expensive. It would be a lot of fun to realize that.
So, I think that in the future we will have all this graphics technology in our households and it will be the 3D embodiment of Alexa anywhere and everywhere. Maybe she (or he) will be a virtual embodied being as well.
Do you talk to colleagues throughout the world?
We are an international community. Many of the published papers cross international boundaries. I have colleagues in London, Brazil, Europe, and Asia.