Epic v Apple Daily Pool Report Day #13
Subject: Epic v Apple Daily Pool Report Day #13
EXTERNAL SENDER: Use caution with links and attachments.
Here's the sum of the reports. Tim Cook testifying first thing Friday. Tomorrow's reporters will be Liz Lopatto from The Verge and Amy Miller with MLex.
Good morning! Josh Sisco here with The Information. I'm sitting on a very uncomfortable courtroom bench (Kouchtown anyone?) next to Bobby Allyn from NPR.
Judge Yvonne Gonzalez Rogers (YGR) is wearing a black mask. I can't compete with Leah's fashion sense. Everyone else is dressed like lawyers.
These are the names I know, will add later if/when I get them.
Katie Adams (general counsel)
First up on the stand today: Apple's Michael’s Schmid, head of game business development for the App Store.
Next up per an Epic spox: Craig Federighi, Apple's Sr VP of software engineering, followed by Dominique Hanssens, a marketing professor at UCLA.
KF wants to know when Epic can respond to Apple's motion from yesterday that the court should rule in its favor that Epic has not met its burden to prove the the App Store is an essential facility. Epic will respond Sunday. Motion here: https://storage.courtlistener.com/recap/gov.uscourts.cand.364265/gov.uscourts.cand.364265.708.0.pdf
YGR: I will go through on monday how much evidence I will have to consider in this case. You will not get a ruling on Monday, or on Tuesday. I believe Judge Koh's ruling in the FTC's Qualcomm case was 250 pages long (It was 233 pages). "I have a very tiny team"
Apple's Richard Doren says Tim Cook will be it's final witness first thing on Friday. Epic says they may call rebuttal witnesses that will follow Cook.
Schmid testimony starts at 8:08
Michael Schmid, Apple’s head of game business development for the App Store
Resumes direct examination at 8:08 by Apple attorney Jay Srinivasan with Gibson Dunn
He was on the stand yesterday only for about 15 minutes and left off discussing games and other apps available on multiple platforms, as well as “cross wallet” games
Since these are all “friendly” questions from Apple counsel, I’m not including the questions. If YGR asks a question I’ll note it.
Schmid says he works with hundreds of developers around the world.
Schmid discusses differences between game developers and other app developers, game developers are at the “bleeding edge” of graphics processing and other tech.
Game developers are also different in how they monetize apps, such as through IAP, there's also a premium market for games with a purchase up front.
Other app developers are increasingly focused on subscriptions, but less so for games.
Public line is down, trial paused to fix it, YGR talks about the Golden State Warriors with the lawyers, your pool reporter is not familiar enough to follow.
YGR: will be interesting to see what happens when we return to normal, there has been much more public access to the courts in general bc of the pandemic
Epic attorney Katherine Forrest agrees
YGR says she never wanted lawyers to appear remotely prior to covid, “cant get the same kind of information when I have people on a telephone line” once a relationship is established its easier, “I don't mind zoom as much”
Schmid says developers can allow game players to avoid IAP by making in-game currency purchases in mobile and desktop browsers; Hearthstone, Roblox and Candy Crush are all examples. It’s at the discretion of the developer, then users decide where to purchase in-game currency.
Schmid narrates a visual presentation of making Hearthstone purchases in a mobile browser instead of inside the app. That allows the developer to avoid paying a commission to Apple. “Then I get to experience the joy of what’s inside.” says something about a game card related to the plague?
Goes through a similar exercise with Candy Crush, this time purchasing gold bars with paypal. “As you can see it’s very sad, I’m only level two.” Schmid is making these demos in his own game accounts on his phone.
Schmid says there are other games w/ the same functionality and it's at the discretion of the developer. Apple does nothing to prevent or discourage this. “Absolutely not”
Says some game developers like Fortnite, Roblox and Minecraft also sell physical gift cards in stores.
“Invest a tremendous amount of time and money and energy” to support game developers. Engineering, marketing and business, and developer tools and products.
“Silly to suggest that a dev looking to grow a biz wouldnt be thinking about mobile.”
"We want to be the best platform to develop a game or app in the world," but we also compete with Google Play, and other android stores, consoles, PC stores, and cloud gaming.
Developer tools include APIs and AppStore Connect to help build the business on the App Store. Former includes tools to test apps prior to launch.
Schmid says Epic took advantage of these tools including Test Flight, App Analytics, to manage sales and revenue, ARkit to experiment with augmented reality in their games
Schmid says From day one since epic was in the app store there was a tremendous amount of support for Epic within Apple.
Calls, emails, in person support with engineers from Apple going to work onsite Epic. Says Apple was able to solve significant Epic-specific problems.
Apple also offered a great deal of marketing support for Fortnite.
Apple aggregated all of its marketing communications about Fortnite and it involved 500 million communications, mostly emails, but also tweets
Epic questions Schmid’s direct knowledge of that number, and YGR presses him on it, but he seems to satisfy her that he knows directly. Schmid says they track this info in the ordinary course of business.
Schmid: “I was certainly the most familiar with the [Epic] account and I dealt with them more than anyone else at Apple.”
In the 11 months prior to Fortnite getting kicked out of the app store Apple spent just over $1 million on marketing on behalf of Epic.
“Neverending crescendo of support”
Not a 9-5 engagement. "It was quite intense, there were 3 am phone calls, 5am phone calls, christmas phone calls it was a pretty demanding relationship."
Eventually brought in people in Australia to provide 24-hr support.
Relationship was “fairly tumultuous, a lot of really good times, a lot of stressful times”
It was a “net positive” to have Fortnite in the App Store
It was an amazing game, generating lots of revenue
Fortnite was reviewed over 200 times in the App Store, not part of the App Review time, but would push on behalf of Epic to expedite the app review process
There were more than 80 plus requests for expedited review in 2020.
Displayed Mar 27 2019 email from a colleague describing how Epic is “abusing” the expedited review process, doesnt agree with the abuse, but says there were “systemic” issues that Epic wasnt addressing that caused the repeated need for quick reviews: “It almost feels like they’re abusing expedite requests due to a systemic issue on their end in the development/QA/submission process,” the Apple official wrote.
Never felt like he could push back against Epic’s requests because Apple feared missing out on being able to offer its users the most updated versions of Fortnite.
Mark Hutchinson was first primary contact at Epic Games, then Brad Cummings, then Haseeb Malek, then Alex Shobin. All but Alec have left Epic.
Displays several email threads about Epic’s emergency problems and effusive praise from Epic for Apple, including for their help setting up a collaboration between Fortnite and Houseparty, Epic’s social media app.
Direct ends at 9:06, Now up cross by Epic lawyer Lauren Moskowitz.
Cross of Schmid by Lauren Moskowitz started at 9:06, morning break until 10:37, will continue cross after the break.
With a lot of heavy lifting from Bobby.
Epic lawyer Lauren Moskowitz doing cross-examination of Mike Schmid, Apple’s head of game business development.
Q: You said Apple competes with consoles and PCs. Has Apple ever offered a better commission rate?
A: I can only speak to the game development side, and they have not.
Q: Apple mandates all apps that require any other third-party option must require sign-in with Apple. Not required when a party doesn’t offer third-party log-ins, right?
Q: Users don’t know about any commission that’s going to Apple, right?
A: I can’t tell you what users are broadly aware of
Q: Are you aware that Facebook publicly stated that Apple was not to tell the public that they were being charged a 30% commission?
A: I’m actually not aware of that
Moskowitz grilled Schmid about the videos played for the court, arguing that the videos skipped log-in steps in a way that made the process of circumventing the App Store seem more seamless than it actually is.
To demonstrate, Moskowitz walks court through trying to make Candy Crush purchases by projecting an iPhone on screen to the court, showing the log-ins required (with an Apple log-in option) and pointing out that Candy Crush can’t tell users to go to a website to make purchases, since that violates Apple’s anti-steering provisions. Additionally, a user has to request a desktop version of the Candy Crush website, which she suggests may not be so intuitive to every user.
YGR asks: remind me, was the price the same for the consumer? (Wondering if the price was the same for making a Candy Crush purchase as an Apple in-app purchase versus on the developer’s website.
Schmid said, yes, both $2.99.
However, Mosokowitz points out that the purchase on the website did not include a 30% commission to Apple and Apple prohibits Candy Crush from telling consumers to go to the developer’s website to make purchases. Schmid agrees.
Q: Are you aware that the tech and gaming press is covering this trial?
Q: Are you aware of the testimony involving Roblox?
Q: Do you know that since then Roblox made changes to its website no longer describing itself as a game?
There’s a discussion about whether Roblox has changed its self-label from a game to an experience. Schmid says Apple doesn’t review user generated content within Roblox, and says Apple has no pending inquiries into rule App Store rule violations.
LM going over an Apple termination clause that says “if at any time Apple determines or suspects that you or any developer with which you are affiliated have engaged in or encouraged or participated with other developers to engage in any suspicious misleading, fraudulent, improper, unlawful, or dishonest act or omission, Apple may without payment due to you or such other developers.”
Schmid resisted specific questions about this clause, saying he is not a lawyer nor on the Apple App Review team.
YGR is concerned about Schmid testifying about material without direct knowledge.
Epic lawyers appear to be on high alert, passing notes around their table.
LM asking questions about Apple’s App Store revenue, but Schmid has no knowledge about how Apple records profits.
Brief digression about playing candy crush, Schmid asks Moskowitz about her level in Candy Crush. LM says “Embarassingly high”
LM asks about Apple’s help with fixing tech issues for Epic
Schmid can’t say whether Apple has offered similar support for other developers.
Schmid said Apple made over $100 million on App Store commissions on Fortnite.
LM asks if it’s actually closer to $200 million, and Schmid pauses to think before saying: “It would be inappropriate to say.”
LM: so they paid $1m [in marketing costs] and made over $100 million?
Schmid says the $1m was only for the last 11 months and doesn’t know much more than that.
Schmid says his team tracks app revenue after a game is featured in the App Store, but is then confronted with his deposition where he said his team doesn’t track that.
LM: asks about developers bidding on keywords to advertise in the App Store.
Schmid says that’s a different team, and doesn’t know how the process works. He generally understands that Apple offers paid search results in the App Store.
LM: would it surprise you if pandora came up as a paid search result when users search for Spotify?
Schmid: yes, but again doesnt have a clear understanding of the process
LM: Apple benefited substantially from having Fortnite on the App Store?
LM: Apple music also benefited from the DJ Marshmallow concert? And other concerts? There were ads for Apple music in Fortnite at those concerts?
LM asked Schmid if he ever talked to Epic about creating a Steve Jobs outfit on Fortnite called The Innovator. He said he indeed did follow up about it, but that it was not his idea. Adds that Phil Schiller would have “hated” it. Phil Schiller, in the courtroom, nods his head.
There was a discussion about the “rogue agent” skin being removed and then put back in Fortnite.
Schmid confirms that Apple asked Rogue Agent to be exclusive to Fortnite on iOS.
Back from break at 10:38, cross of Schmid by Lauren
When Apple update its iOS to version 12, there were some memory related issues. Moskowitz pulls up an email in which it says one Apple engineer was working with Epic and 8 Epic engineers to resolve the issues.
every build for every app has to go through app review?
LM: pulls up an email from April 2019 with a colleague complaining about the technical aspects of Apple’s app propagation process, the process of pushing out the app to users.
the propagation often took 24 hours
consoles propagated games faster than Apple?
ends at 10:50
RE-DIRECT of Mike Schmid by Apple lawyer Jay Srinivasan.
Q: Did you edit those videos in any way? [refers to the videos by buying in game currency outside of the app]
A: I Did not
They had a back-and-forth about Epic’s Candy Crush iPhone example that showed how a user needs to enter a third-party login-system in order to make a digital purchase.
User can choose to save their log-in and password in apps and websites, which Schmid says is typical.
asks about propagation process between iphone and consoles
JS passes the witness
LM: has nothing for recross
YGR jumps in with a series of questions.
there has been lots of discussion about gamers in this trial. Has Apple done any studies or profiles about gamers as “a separate and distinct customer base”?
of all app users how many would you put in a gamer category?
YGR: so in the context of revenue, what are you finding?
A: We analyze the revenue from games but don't look at the revenue gamers on our platform generate.
any analysis of IAP revenue v initial downloads
Moskowitz comes back for a couple questions.
you were asked by the court about where revenues were generated from, can you say how much revenue generated by categories of apps?
Schmid testimony ends at 11:03
Apple calls to the witness stand Craig Federighi, Apple's senior VP of software engineering. Takes the stand at 11:05, break for lunch at 12:35.
Again thanks to Bobby for the help.
He is being questioned by Apple lawyer Jason Lo.
Federighi mainly works on engineering of the operating systems, including iOS, iPadOS and MacOS.
Federighi says MacOS and iOS have some small similarities, but "there are tremendous differences”.
CF said when Apple created the iPhone, it saw it as a “once-in-a-generation opportunity” in terms of building “security architecture.”
At first, he said, iPhones did not allow third-party apps, but eventually opened iPhones up to native third-party apps. Did that present security threats?
“Oh, tremendously,” he said. “We also wanted to radically rethink peoples’ relationship to apps”
For iOS, he envisioned users would download lots of apps to solve all types of problems. “We wanted to make that something users can do very easily without having to be very thoughtful about the security consequences of those downloads,” so they built an end-to-end security review system.
Apple displays a “Threat Model Considerations” exhibit with these three categories: Number of devices. Number of opportunities. Value of access -- all things, CF said, are major attacker considerations.
CF on why attackers are interested in targeting iPhones:
“iPhones are very attractive targets. They are very personal devices that are with you all the time. They have some of your most personal information,” he said. “They have cameras and microphones. They are capable of knowing your location. Because they’re always with you, they can be used as a key to get into your palace of business, or a token to unlock your bank account,” he said. “All of these things make an iPhone incredibly valuable to an attacker.”
Another exhibit displayed to the court, common attacks
Apple sees across the computing ecosystem: Scams, Ransomware, Surveillanceware, Vandalism and Sabotage, Info Stealing.
Federighi says Apple has many layers of security to protect users. "We try to stack up many layers of defense," he says. There are three broad categories: malware scans, signatures/certifications, sandboxing
Federighi says iOS has many additional security features beyond those in MacOS.
Displays an image of a Mac next to an iphone. The Mac is surrounded by 8 red human figures, the iPhone is surrounded by dozens, says the red people are “bad attacker people.” Indicative of the many more security threats faced by iPhones v Macs.
Federighi says The App Store review guidelines is the “single most important” line of defense against security threats. The automated processes and human app reviewers are making sure the app is what it says it is and does what it says it does.
Federighi says developers have to ask for “entitlement” permissions. “Why is this calculator asking for permission to access health information?”
Sideloading apps would “dramatically” increase the security threats faced by iPhone users. It would completely remove app remove from the security checks. Users could download a fraudulent microsoft app for example
“If policy could be signed and downloaded directly, you could put an unsafe app up and nobody would check that policy,” he said. “You could say, ‘I’m Microsoft Word, here I am,” he said. “You could think you’re downloading legitimate software and you’re downloading a Trojan.”
YGR: “There are multiple [app] stores on the mac, why should we not allow the same stores to exist on the phone?”
CF: “it is regularly exploited on the Mac.” iOS has a much higher security level “the mac is not meeting that bar today” and “If you took mac security techniques.. iOS would get run over”
CF: “It would be a very bad situation for our customers”
YGR: What about android?
CF: says android has a bad malware problem, “it is a problem for android. With iOS we have aspired to create something far more secure… dont want to slide back” to the level of security of a Mac.
YGR: Mickens [Epic’s security expert] said otherwise, that MacOS and iOS are comparable.
CF: says every report he’s read said iOS is much more secure. Said Android has something like 30x more attacks than iOS. “The results in the real world are dramatically different”
CF: Says with centralized distribution, we can be pretty
confident that bad actors will get shut down. “This is dramatically different from what we see on the mac unfortunately”
CF: Mac users also expect a greater degree of flexibility
that is useful for what they do”
CF: “There is a significant large malware problem on the Mac”
CF: Removing the App Store would as the sole means of software distribution on iPhones “Would subject iOS users to a huge decrease in their safety”
CF says Apple occasionally deals with well-funded “nation-state-level attacks” using “previously unknown zero-day exploits” that are “extremely targeted,” perhaps against a political rival or a dissident journalist. Such attack attempts are rare.
CF: The primary security concern Apple has about most users is protecting their privacy. Apple wants to give users as much control over their own data as possible.
CF: While apps can ask for certain information, they cannot withhold functionality if a user doesn’t give up information. “Incentivized access,” when an app asks a user basically “are you sure?” showing what a user gets in exchange for access to, say, a user’s contact list. This practice, which CF calls a trend among developers, is banned by the App store, because he says it compromises a user’s privacy.
CF talks about Apple’s enterprise program which allows
companies to develop and install internal apps on work phones outside of the App Store. Says those stores are ”Absolutely full of virus and Trojan software.”
CF says microsoft is still the most exploited platform because they have the most “motivated” group of attackers.
Back from lunch at 1:17, End of direct exam of Federighi by Jason Lo, ends at 1:32
CF: continues testifying that third-party app distribution on iPhones would open up phones to malware. Says it could also cause users to lose trust in less known, yet reputable developers.
CF says security and app reviews would be difficult to hand off to third parties. Fear many companies would not enforce the same sorts of guidelines that our customers and we care about. "I would have grave concerns"
I don't think anyone cares as much about maintaining that promise” as Apple does.
Q: Does iOS support streaming games?
A: We do
Q: Streaming games have to comply with human interface requirements?
Q: Are streaming games on iOS devices required to be individualized?
A: Apps are the center of the iOS experience. And so, for streaming apps, our policy is, each streaming game be distributed as a distinct app that they can download and launch independently from the streaming service.
Q: Any privacy control reasons for this?
A: Yes. There are privacy and management reasons. It can help with setting parental controls. When it comes to privacy, for each app, the user is given what they allow the app to access. If I was playing Pokemon Go, I might allow that app access to the camera. Or I might be playing Assassin's Creed, and I wouldn’t give that kind of access.
Q: Are you familiar with games that have user-created content?
A: Some games, like Minecraft, let you modify the world, creating another place to play.
Q: Are you aware of apps in the iOS store that allow for streaming media, like Netflix? How does iOS view that?
A: Media experiences are not different application experiences from the user’s point of view. The user often doesn’t have with a single episode the kind of 30 to 100 hour relationship that they might with a given game title that they’re coming back to continually.
Q: When something like Netflix have different movies and different shows, does Apple make them have separate apps?
A: No, that wouldn’t make sense.
Direct ends at 1:32, Cross starts by Epic lawyer Yonatan Even.
Directed to deposition: Asked if he has any numbers on the prevalence of malware downloaded from third-party stores? During deposition, he said he did not have any numbers. Now he does, CF said.
CF: says he wasn't at Apple when the first iPhone or App Store launched. When he took over iOS in 2012, he wasn't on the Executive Review Board [for app review].
YE: You have no first-hand knowledge about the design that went into App Store?
Federighi and Even are discussing a Nov 2007 document about allowing third party apps on iOS. Document, prepared by Apple’s security team, discusses giving Apple absolute control over app distribution. Says “code-signing” will prevent malicious apps.
They go over one portion of the document apparently entitled “guy in his basement,” which refers to a small developer v a large software company.
YE points out that the security team who wrote the 2007 white paper never said distribution has to be limited to Apple in order to keep the apps safe.
“I don’t think it draws a conclusion one or or another in that regard,” CF says. “You haven’t pointed me to any lines that said that.” Tries to say that code-signing is not a full security solution, but Even cuts him off.
[The whole discussion is difficult to follow since the documents they are discussing are not being displayed to the gallery]
Now discussing MacOS and how it allows third-party app stores.
YE: asks if CF knows that if Epic get’s its way and a third-party store is allowed, would that remove Apple’s store from the phone.
CF: says he doesn’t know what remedy Epic is seeking but it's possible that Apple’s store would still be there.
YE: if third party stores are allowed, and users still only used the App Store, they would have same level of protections that the user enjoys today?
CF: “no the environment would change.”
YE: pulls up CF’s earlier depo where CF says that the same protections would still exist in Epic’s alternate world.
YE: Asks if Apple can offer a much bigger audience to developers than any other company since its App Store is preinstalled?
CF: There's a web browser installed, Federighi says, they can search that way.
CF is appearing exacerbated with YE’s questioning, repeatedly shaking his head and looking puzzled.
They are running through questions about Apple’s security processes, and what Epic perceives as the limits of those processes.
For example Even asks if Apple could use PhotoDNA to look for pornography. CFsays that wouldn't work. Even says YouTube and TikTok do this now.
The exchange is getting testy. CF wants to explain why certain things are difficult or not possible. YGR tells him he can explain himself once Apple’s attorney takes over the questioning again.
YE mentions Parler’s removal from the App Store as an example of Apple booting out apps that don’t follow the rules.
YE: says there’s nothing that prevents other stores like Epic and Steam from having their own set of policies and security checks
YE: asks about the user base for the Mac, says there were record sales in 2020.
CF: agrees but says he’s not sure of the precise numbers
YE: tries to be truthful with its customers?
YE: seeks out a very diverse array of customers?
YE: used to store a lot of information about users lives, web searches, telemedicine visits, education
There’s a line of questions about how many different types of people and uses for which Apple offers the Mac
YE: is photoshop available on MacOS?
YE: nothing suggests, in the marketing material they are looking at, that getting photoshop from Adobe is a security threat?
CF: agrees, but adds that they are not looking at a security related document
This is the site they are looking at: https://www.apple.com/education/k12/
YE: nothing that says “if you are a novice, stay away, buy an iOS device”?
Even is taking a very long time to make the point that if MacOS is so bad, Apple would send all but the most experienced users to iOS.
YE: if the court were to open up iOS tomorrow, Apple could continue with sandboxing? [a means of analyzing code for security threats.]
YE asks if CF knows how long Apple staff spend reviewing each app?
“It’s changed over time,” CF said, estimating that it’s “minutes.”
He acknowledges that he is not directly part of the day-to-day app review process.
Cross ends at 3:01
Redirect by Jason Lo
JL: asked in depo if third party stores existed, would the same protections currently in the App Store still exist?
CF: the sophistication of attacks would be greater, so malware risks would increase, even if the original protections were still in place in the App Store.
Asked about enterprise program, and says no matter how thoroughly it was reviewed upfront, it wouldn’t prevent that enterprise security certificate from falling into the wrong hands later. “Many things can happen after the fact that can cause that certificate to become abused.”
JL: has anything changed with security threats to iOS devices since 2007?
CF: absolutely. And we’ve seen how successful iOS has become and how that’s made it a larger target.
CF says centralized app distribution and review “is critical to the safety of our users”
JL: asked about app privacy and “nutrition labels”
CF: developers must submit accurate information about data collection and if its not accurate we will work with them to fix it. However we some auditing on our own to make sure its accurate
There’s a discussion about Jekyll and Hyde apps that change their behavior after they are approved. Says “sophisticated attackers could bypass” security measures. “Its a bit of a cat and mouse game.”
Direct ends at 3:24
Recross by Even ends at 3:26
You received this message because you are subscribed to the Google Groups "Epic-apple-pool" group.
To unsubscribe from this group and stop receiving emails from it, send an email to epic-apple-pool+unsubscribe@....
To view this discussion on the web visit https://groups.google.com/d/msgid/epic-apple-pool/CANE5d9SbgAjNtUWfU32fRSXY6Ly7VuoDA19qXZLOG%3DPipWfz7A%40mail.gmail.com.