AI is reopening a core question from the development of the web: how to preserve the freedom to learn while giving publishers meaningful control over how their works are made available.
In this episode, Matt Perault is joined by Derek Slater, cofounder of Proteus Strategies and an expert on information access, copyright, and free expression, to examine this question and explain why it matters for the future of AI.
The web worked because people could access, read, analyze, and build on lawfully available information. Standards like robots.txt helped manage the balance between openness and control at scale, giving publishers a way to express preferences without requiring every builder to negotiate permission website by website.
AI is now testing that equilibrium. Some proposals would restrict not only unlawful access, like when AI developers circumvent paywalls to get data, but also lawful learning from public information through expanded copyright theories, terms of service, technical barriers, or licensing requirements. Derek and Matt separate those issues, including the difference between training models on lawfully accessed data, producing infringing outputs, using AI tools to summarize content a user can already access, and breaking through access controls.
For Little Tech, this question is fundamental. Access to data operates as a form of startup capital, allowing new companies to develop products and compete. But if AI companies can’t learn without negotiating expensive licenses or if large pools of data are entirely off limits to AI learning, then only the biggest, most-resourced companies will be able to survive.
Topics covered:
00:00: Introduction
01:34: The freedom to learn and AI data access
04:05: What the early web can teach us about openness, control, and contested norms
08:37: How robots.txt helped publishers express preferences at scale
11:32: Why voluntary standards worked for search engines and publishers
13:39: How freedom to learn applies beyond technology and copyright debates
15:51: The publisher POV on traffic, monetization, and value exchange
18:25: Why AI agents are raising new questions about user control
20:25: The difference between protecting publishers and limiting lawful AI-assisted reading
21:31: How copyright law applies to AI training inputs and model outputs
27:57: How contracts and terms of service can attempt to restrict lawful learning
31:01: The limits of “learning” as a defense
33:42: How licensing markets are evolving around data access and AI outputs
37:01: Technical collaboration and the future of robots.txt-style standards for AI
39:32: Why data access is a Little Tech issue
42:14: Public policy guidance for preserving the freedom to learn
45:18: Closing thoughts
Disclosure: Derek Slater has previously provided consulting support to Andreessen Horowitz. The views expressed here are his own, and this conversation was not part of a paid engagement.
This transcript has been edited lightly for readability.
Derek Slater (00:00)
When it comes to things like intellectual property, what is ownable by somebody and what belongs to us all, fair use is not antithetical, right?
Copyright’s limits are not antithetical, rather, they help create new markets. And when people reach for copyright, they are entrenching big tech. They are reinforcing monopoly, right? Limits and exceptions to copyright have always been a core anti-monopoly lever. And I think when people miss that and you know start to say, “well, we should just propertize everything,” they don’t get how Google and Facebook are gonna be the biggest winners in that.
Matt Perault (00:31)
Derek, welcome to the AI Policy Brief.
Derek Slater (00:41)
Great to be here. Thanks for having me.
Matt Perault (00:42)
So you are an expert on all things information access. And you and I have known each other for a long time. I’ve come to you over a long time on those topics, free expression, copyright. So you’re the perfect person, I think, to talk to us today about freedom to learn principles. And the thing I want to start with is how the conversation that you and I had been having evolved to this topic. So we were originally, I think, discussing what is the approach that a firm like a16z should take on this important copyright debate that’s happening now, which is much narrower than freedom to learn. And over time, as we were talking about the concept here, it seemed like the appropriate place to go was to actually step further and further and further back to this broad idea about a freedom to learn. Why do you think that’s the right lens for this conversation?
Derek Slater (01:34)
The great thing in this space is there are lots of different lenses we can apply, but I do think the freedom to learn is attractive in two ways.
First, I think one way of thinking about this issue is where do my rights as a reader, a learner, a consumer, somebody who’s using a book, music, a website, whatever, where do my rights begin and the publisher’s rights end? Right. And what should I get to do with the thing that I have lawful access to that I bought, and so on and so forth?
That’s both sort of, literally, how can I access it? Can I take a website and turn on a read aloud function and make it more accessible? But I think also this other layer of when it comes to things like intellectual property, what is ownable by somebody and what belongs to us all. Right. And that’s where that’s a copyright issue, but I think that’s also a broader issue to think about when it comes to information access and knowledge. Right. When we grant property rights to somebody, say you own this expression or you own this thing.
That could be for good reason to incentivize people to create, to help them make money, and so on and so forth. And we want there to be this commons of knowledge that belongs to all of us. Ideas, facts, lots of stuff that we rely on all the day, the way language works, where we all rely on the fact that you can’t get a property right and then say, no, you can’t do this with that. You can’t use my fact, you can’t use my idea. It belongs to everybody. And that’s, I think, something that’s really important to be reckoning with.
I think a lot of these issues can so easily be taken as, there’s this company and this company who are fighting. That’s true. There’s a lot of that. And there’s a lot to tease out there. And we need to step back and think about what are the things that should belong to each of us and to all of us together. And that’s where public policy really has a strong role to play.
Matt Perault (03:20)
Can you walk through some other venues where we confront some of these conversations? Like as you’re describing, I think a lot of people come into the AI copyright debate with some preconceived notions based on sometimes like, do they like one side of the debate or the other? Like it’s so crazy that companies would use copyrighted work as training, or it’s so crazy that doing this really innovative thing of training on publicly available copyrighted work would not be fair use.
And I wonder if it’s helpful, if we could just step back a little bit and look at how for something like the web, for instance, how do these general principles operate? Because that’s a world where I think like some of these things have been more resolved and we have a sense of like, what are the rights that should belong to publishers and what are the rights that should belong to people who are roaming around the web?
Derek Slater (04:05)
Going back to the beginning, the web is good. And there are other technologies too. And I think the key thing here is that even though things can look like on the web, things have been working in a certain way and they’ve always worked that way. Actually, at the beginning, things were very contested. When it comes to what does it mean to have the freedom to learn or to have an open web, what does that mean? So I think it really came down to two things that went on in the early days of the web. In the early days, people created websites. The web was a standard that was open.
What did that mean? That meant anybody could build a browser or piece of software to go and access your site and render it in whatever way was best for them. You know, if that meant blowing up the font, if that meant reading it aloud, if that meant doing it in different colors, you as the reader had that control because of that open standard. That was the bargain. And particularly as automated processes for accessing websites came about, what we would call bots or web crawlers, search engines were one type, right?
Search engines crawl the web to index it, create the index, but there are lots of types of bots and crawlers. As those started to develop, websites said, hold on a second, I didn’t know you were going to do this, or this is a lot of traffic. I wasn’t ready for this. My site’s going down. So a bunch of different standards and norms and practices had to be developed through bodies like the Internet Engineering Task Force, which is the home for robots.txt. So the small text file you anybody can put on their website and say, hey, you know, you can crawl this part, but not that part.
There are other standards that are like, well, you can crawl my site, but only at the speed that works for me because I only have this one server, not 50 million servers. I don’t have a data center. Right. And those things had to evolve over time. And I think there were mutual benefits that led them, but it’s worth not losing sight of. These were contested. These were fought. Or rather, there were both fights in the standards processes about that. There was, you know, legal threats and litigation, both about search engines. And about things like deep linking, where instead of linking to the front page of a website, people chose to link to some subsidiary page. And there were websites that said in their terms of service, that’s not okay. Go through the front door. I want to control the interface and your path through my site so that I present the information the way I want to. And so there are always these tussles that go on when you have this sort of interface between publisher, reader, you know, software and website.
And so one way that these things got reconciled was norms and standards, like at the IETF. And part of it was law and the balance that copyright has helped set, which is to say when search engines were sued in the sort of early days of the web, late 90s, early 2000s, courts said, Look, you own this expression. Sure. But what this thing is doing is transformative. It’s analyzing this expression to derive facts and relationships and things that aren’t copyrightable. And it shows this little snippet of text, but that’s so small that it’s fair use. So the index, this is sort of, you know, in line with what AI training does. It analyzes all this data, not for the copyrightable expression to replicate it and substitute for it, but rather to create something new out of the uncopyrightable elements such as ideas, facts, patterns, you know, relationships between websites, who links to whom, right?
That’s an uncopyrightable fact, not part of the expression. Courts had to sort a lot of that out and and you know, as we see today, are continuing to wrestle with this, not just in AI, but with other sorts of cases where, you know, someone wants to access data on a site and the site doesn’t like it.
To give another example, you know, Facebook was faced with and LinkedIn too and other social media sites faced some service providers that wanted to help customers, wanted to help users, consumers, take the Facebook social media feed and combine it with their Twitter and all their other things into one interface to make them interoperable. So consumers had more choice. And Facebook said, Hey, this is a bot that doesn’t have authorized access to my site. This is equivalent to hacking. And Facebook won. You can look at that and say, it’s Facebook’s property, they have property rights, they have contract rights, you know, they have these anti hacking rights, they should win. On the other side, it’s how you get a walled garden and lack of interoperability.
Those are real like there’s real trade-offs there. And I don’t mean to oversimplify to say, you know, this good, that bad, but these are the things we have to be wrestling with here and they’re sort of evergreen.
Matt Perault (08:37)
So robust.txt seems like an interesting effort at a compromise. I’m wondering if you can talk about a couple of different components of it.
One is the desire or the equities involved in offering it to begin with. So, why is it useful for either an entity to say, I want to be crawled or I want to not be crawled? How would that decision be made by a website operator? Why might they opt in one direction or the other?
And then robust.text is not a formal legal regime, right? It’s a voluntary standard. So, I think a lot of people would say, well, that seems fairly meaningless if there’s no legal teeth behind it. How has that aspect of it played out in practice?
Derek Slater (09:14)
I think from the site owner’s perspective, the piece of this is control. And then also, I think what a lot of publishers would describe as maybe value exchange or who gets the value from this? Right. So what’s the bargain? If you’re coming with your crawler, what do I get? And what do you get? Okay. You get data, you get to create your search index. And there, well, you’re going to send me traffic? Great. That’s helpful to me. And that could sort of be the bargain.
And on top of that, you know, robots.txt has evolved to be granular in certain ways. So again, you can say you’re allowed to crawl this part of the site, but this other part, no. So to your point, why would you know a search engine or others follow that? I think they have an interest in working with publishers collaboratively and constructively. And you know, if somebody doesn’t want to be in the index, fine. There’s lots of other sites that do, we will work with them. The key thing about it being voluntary now a law is if at the beginning of things, you know, Yahoo or other search engines had had to go and negotiate individually with each site and ask permission first and negotiate a separate contract first, that never would have scaled to the you know billions of websites that come out there. So robots.txt both allowed people to just sort of continue operating as normal. And then when somebody puts up a hand and says, actually, I have a different preference, a simple way to respect it. And I think the fact that it was simple, machine readable and sort of fit within the broader construct of like, okay, we if you don’t want to be crawled, fine. You don’t like this traffic, fine. It’s sort of, you know, landed as a reasonable norm.
And to your point to just underscore it, yes, it’s a norm. And that means sure, there are people who will not respect it. And some of them, you know, maybe less sympathetic, you know, bad actors of some mind. Others, it’s for, you know, good reasons. So you look at the, you know, the Internet Archive. There are some types of sites, certain, you know, government sites, I think in particular, where they might not respect robots.txt because they’re an archive. They’re a library. Their whole job is capturing the historical record. And, you know, libraries have a job to do too. That’s very different from, say, a commercial search engine or AI or some other sort of data collector.
Matt Perault (11:23)
So you used to work for a small little commercial search engine called Google. Why would that search engine respect robots.txt?
Derek Slater (11:32)
It’s a great question. And I think we could sort of characterize it over time. If I roll the clock back, I would say that it was in recognition of this need to reach an equilibrium of sorts. Right. That is, if you take the bright line hard roll all or nothing, that could end up worse. People start locking up their sites behind more restrictive things and paywalls, right? That would sort of be the inevitable end. And that’s sort of what we see today.
Okay, let’s imagine they decided not to respect robots.text. Then people put stuff behind paywalls or behind logins. And that’s even worse. Right. So it’s about finding that equilibrium that mediates this relationship so that there are many winners. It’s not just, you know, the search engine wins, the publisher loses.
Now, I think today one of the questions, and now this is, you know, why do they follow it? Now there are ways in which this is more of a regulatory issue for a company like Google. There are more obligations for them in part because it’s different. They have a different amount of power than in those days. But I think if you go to those early days, it was really about that recognition of what’s a happy medium. And just to sort of, add another layer on, it wasn’t just in search.
So if you look at Google Book Search, which is, you know, where Google, they both worked with publishers to license some books, but they also worked with libraries to digitize their collections. If it was public domain, make the whole book available. If it was in copyright, they’d just search, show it in the index, but just small snippets, not full text, not full pages and the like. From the very beginning of that, Google also had an opt out.
You know, rights holders come forward and say, hey, I don’t want my book even indexed. And Google would respect it. That wasn’t required by the law. It wasn’t required by the fair use decision that the court ended up rendering. And because Google is in repeat relationships with publishers, it makes sense to do something that strikes a medium, recognizes that both sides have an interest in making this work well. So that’s what things look like when they’re working well-ish. Now again, there was litigation over the web. There was litigation over Google Book Search because of the author’s dilemma, and so on and so forth.
So none of this is easy. And I think that’s still in the background of how do we have something that leads to many winners rather than just a winner-take-all situation.
Matt Perault (13:39)
So we’ve talked a lot about the freedom to learn in the context of the tech universe, in the context of development of the web. But could we back up even further and talk about freedom to learn generally? How do these principles govern how humans learn?
Derek Slater (13:53)
You know, going back to where we started, you know, remember buying a book, a physical book. I don’t know if you still do that. I still do it from time to time. You actually get a set of rights when you have that book.
The person can’t say to you, you’re not allowed to give this away, or you’re not allowed to sell it to somebody else. Right. And that’s how we have libraries, right? They’re not allowed to do that with the book. You can quote from it. You can generate summaries generally to a point. You can learn from it in the sense of, hmm, the author’s style. They write in this way, or these are the themes they’re using, or this is how they structure the novel. You can learn facts, right? You buy a book on World War II, you get a bunch of facts from them. You can reuse those. You can also take that physical book and you can use it as a doorstop, you can go to the park and use it as a pillow. It’s yours. And so that has always been sort of a foundational thing, both in property, but also in copyright law. That the user, the reader, the consumer has certain rights too.
And I think what we’re seeing now is a debate over how these translate to AI. And I would say generally the parallel is there. I think people get tripped up over the idea of, well, humans are allowed to learn, but this is about machines using the stuff. And machines don’t have rights. Okay, fine. But I’m the one using the machine. I should have the right to use a machine to help me learn in ways that are in line with law and norms and so on and so forth, right? And that’s what’s happening with AI models.
I could watch and have watched, you know, 30 sci-fi films and try to figure out the different concepts and themes. George Lucas can draw on Kurosawa, and then people can draw on George Lucas, right? All that’s your game. If I use, you know, an algorithm, a machine to do that same sort of pattern matching and learning, we call it data science. It doesn’t change that I’m the one executing it. I’m the one learning. And I think that’s something that we should be celebrating people’s ability to do and to use that freedom to learn in new ways with machines as their assistant.
Matt Perault (15:51)
Can you talk a little bit about the arguments that publishers are making on the other side? Like they understand and recognize freedom to learn principles generally as well. What is the concern for them about AI training, how does that relate to these principles?
Derek Slater (16:04)
I think there are sort of three buckets of concerns and the way people have come at this.
One is really around traffic, traffic to websites and the costs of traffic. This is less, I think, with large commercial publishers, they can scale their servers and it’s probably a rounding error. I don’t mean to say that generally. I’ve seen it as a bigger issue for other types of publishers, like a Wikimedia, right? Wikimedia, although they say, our content is free, our infrastructure is not, right? Our content is licensed under Creative Commons license. Take it, remix it, re-share it, all those great things. And we got to pay for the servers. And if people aren’t sent back to the site, right, and they only sort of get access to Wikimedia content through an AI overview and search or some other AI model that’s ingested it and used it and so on and so forth, people never come to Wikimedia itself or Wikipedia itself.
Then how are we going to get new users and new members and new contributors? And then that hurts our sustainability. And that, you know, you can think of that as another cost. So that’s one slice of things.
And,I think there are a lot of important things to be done of, okay, how do we, like we did with robots.txt and other standards, create the right ways of saying, hey, if you’re going to run a bot, it should behave kind of like this so it doesn’t crash my server.
And you know, if you would, can you validate with this third party that you’re a good bot, not a bad bot, and show me a credential? There are lots of interesting ideas like that we can play with, but that’s one, which is really about traffic costs.
And then that merges in this competitive one. And this is where I think you’re going with the publishers of this and the value exchange, the bargain. Wait a second. You’re taking my content, you’re creating these trillion dollar companies. And not only am I not necessarily getting some benefit of that trillion dollars. You’re making a dollar, I’m not even getting a dime, and I should get a dime. But also, wait a second, the fact you’re displaying these facts. And now that if I’m a newspaper and I’m expecting people to come to me for the facts, and you’re taking these uncopyrightable facts and showing them they’re not going to come to my site. And then I lose traffic and I lose ad revenue and so on and so forth. There’s a real competitive angle there.
And you know, to round it out because it’s easy to just think about news publishers, but t’s a broader issue with web publishers, right? Just last week, Amazon and Perplexity, I think there’s another hearing in that litigation where Amazon has sued Perplexity for giving users a bot, an agent that at the user’s direction will navigate the Amazon site and help them do their shopping. So that instead of dealing with the Amazon listings, which prioritize ads and things that might maximize their profit, it has a duty to the user. Do what the user wants. And Amazon said, I don’t like that. I want to control the interface of my site and use that same anti hacking statute that I mentioned earlier. And so that’s another case, like the news publishers, where they’re saying, I want control over how my content is displayed. You’re competing with me, you’re taking some of that value. And I think some of these disputes, it’s about finding that equilibrium, the win-win.
Really, like at some level, back up from Amazon Perplexity, if you’re a commerce site, a shopping site, and you’re a commerce agent, you have some shared interests. Commerce sites want to sell stuff, you want to help consumers buy stuff. There’s reasons to come together, make agreements, make APIs, come up with standards. And I think we’re starting to see some of that with news publishers, too, of how that value exchange happens, right? As much as the AI developers think, yes, if your information is public, you allow me to crawl it, I can use it for training. I can use it for these purposes. And if you have a paywall, well, you can say to an AI developer, you have to pay me to acquire that content, even if it’s fair use to train on it. Or the developers have said, we will respect certain sorts of opt-outs, like the search engines. If you want to opt out of foundation model training or use these things like AI overviews, as you know, Google announced last week, we’ll respect those too. So we’re seeing to see that new option set come about of how do we both come up with the right business models that help both parties.
The right options, the right level of control, that’s markets, that’s standards, and there’s still gonna be legal fights. And I think that last part then, so we said traffic costs, competition, the last one is this one about the consumer, the reader, the user. I really think it’s important to come back to that. You know, I get when people look at the equities of AI companies and news publishers say, this is a fight over money and they should contribute to the publishers. The equities feel very different to me when it’s a publisher saying, sure, you paid for access to my newspaper, but I want you to read it in this particular way. And I’m gonna have a technical control that blocks you from using these, from using an assistant to help you read it. Right. So in my browser today and many other browsers, there are AI tools, you know, that will let you summarize a page. Should a website publisher be able to tell you, the user, no, you may not summarize. You must read the full article the way I wrote it. I mean, we could have a world that works that way.
That wouldn’t be the freedom to learn that either serves the individual and I don’t think really serves society ultimately. Now, there are lines to be drawn here, but I think that angle is really important to put in the front.
Matt Perault (21:31)
Can we run through how existing law applies here in your view? You’re talking about several different bodies of law. Maybe we can just kind of tick through them one at a time. First of all, copyright. I mean, obviously this is hotly contested. There are a bunch of court cases that are gonna have a lot to say about this and will give us a lot of guidance. But just in terms of how you size it up, given your expertise in copyright, what do you think copyright law has to say on this question?
Derek Slater (21:57)
Yeah, I mean it certainly varies around the world and governments around the world are looking at how to amend, change, and reconcile AI with their copyright law. You know, just starting in the US, I think it’s very important to separate inputs from outputs. That is to say, training a model on public information, you know, lawfully accessed, either as a website or information online elsewhere, or, you go and you get a book and you scan it as Anthropic and others have done. That’s the input side.
And if you’re using that to train a model, which means to derive those uncopyrightable ideas and facts and so on, and not output the expression, then that’s consistent with that’s highly transformative. It’s not substitutional for the original work. It helps people create new works, but it’s not replicating the old work. There are corner cases here, like what if the model memorizes stuff? What if it’s a copyrightable character and it outputs that? You know, you say, give me a picture of Iron Man and it outputs Iron Man.
There are some interesting corner cases here, but I think the input and output side are very distinct there. I think copyright law in the US will have a very clear answer on the input one. And I think even around the world, right? Japan has an exception like this. Singapore, the EU has an exception that is subject to an opt-out. And I think as countries around the world think about both sovereignty, AI sovereignty, and about what it means to have competition and choice across all the layers of the AI stack, they’re going to have to come to grips with.
The idea that they will need some sort of way of doing, you know, allowing this sort of use of public lawfully accessed content for training on the input side.
Matt Perault (23:37)
The input output distinction seems really important. So the input side is you’re using copy publicly available copyrighted work to train a model. And then the outputs of the model might look nothing like or certainly not exactly like the underlying work. Can you say a little bit more about the specific use case you have in mind on the output side? Like what would a potential infringing output look like?
Derek Slater (23:58)
Yeah, I’ll start with the non-infringing, because I think there’s so much that is about, particularly the large language models, that has been bound up in copyright and certain types of creators. And that’s fair enough. And those objections and those rights holders are a vanishingly small portion of all rights holders and creators talking about a vanishingly small percentage of uses. Most people are not using their LLMs to go and try and recreate an article to spit out a New York Times article. They’re creating new stuff, as you said. They’re writing their own stuff. They’re, you know, they’re getting an educational tutor. They’re doing scientific research, right? All sorts of sectors across the economy are using these large language models for things that have nothing to do with creative works. And even when they do have to do with creative works, it’s generally about creating a new, novel thing.
Okay, so how would it be infringing? Let’s say the model did memorize a full work, the expression of a work, and then outputted it. Which is alleged in both some of the newspaper cases, but also in the case brought by music publishers against Anthropic who have said your model memorizes full song lyrics and outputs them readily. That’s going to be something to wrestle with. And I think there will be questions about hmm, let’s look at what did the model developer do to impede that outcome? You know, did they do any deduplication of their training set so it would be less likely to memorize?
Did they use any output filters? How did that look? That might be part of it.
Matt Perault (25:28)
But that’s still on the input inquiry, right? Infringing output would be cognizable under copyright law.
Derek Slater (25:34)
Right. If it outputted that article, the question is, is the AI model developer responsible for that? And I and I just bring that up because look, remember, Google Books is a database of books. It’s like it has memorized those books, but there are output filters that limit what it outputs. So in the Google Books case, there was both us, you know, scanning them, but also creating the index and the output. And they’re all, you know, different, but they have these linkages.
Another one that may be clearer is that is sort of again the Iron Man example. Let’s say I say to my image generator, create a movie that is, you know, Avengers Doomsday, my own version of it. And then you go and you sell it. I think because characters can be copyrightable, there will be questions about specific outputs and then what are the developers’ responsibility. You know, they’re gonna have to work through that. My own perspective on it is it’s going to come down to questions, not of what’s called direct liability, but secondary liability.
That is, the user is the one making the thing. They’re the one who type the prompt, hit enter, go. The developer, if they, you know, had actual knowledge and contributed in certain ways or benefited in certain ways. I think that inquiry might also look at did they try and impede infringement in one way or another? Those are the factors the courts I think may look at over time.
Professor Matthew Sag is one of leading writers on this topic wrote a really interesting article about fair use and AI’s jagged frontier, saying this uncertainty in these issues means that there’s still an incentive to cut certain types of deals, not necessarily for the input side, but for this output side. And that reminds me a lot, reminded Professor Sag a lot of YouTube, right? Where YouTube in the early days copied stuff. There’s a notice and takedown process, but they went further and said, hey, instead of just notice and takedown, what if we add this option where we leave this stuff up and we run ads against it and you, the rights holder, get the majority of the money? And that created a new revenue stream for rights holders. It was that win-win-win. The user gets to leave this stuff up, the rights holder gets money, YouTube makes money too, and that keeps going. And I think those sorts of things are, I think, on the output side, part of what we’ll see develop. We’ll have some protection, but there will also be areas that are clearly off limits if you’re really outputting something that is infringing work and commercializing it and so on and so forth.
Matt Perault (27:57)
Let’s talk about contract law, which I have found to be really complicated. So there’s all the copyright related stuff, which I think is probably like the headline issue here. But then increasingly, rights holders have said, and I and I see this almost every time I pick up an ebook, the book will say like the content of this book cannot be used in AI training any form, which is an attempt to impose a unilateral contract and to hold any potential AI developer liable if they violate that provision in the ebook. How should we think of that?
Derek Slater (28:29)
That’s a great question and a really tricky one. Cause I think one of the key questions it gets at is when should you get to do, if ever, through contract, what you can’t do through copyright. So, you know, go back. If I buy a book in the bookstore, they can’t simply put something in the book that says you may not resell and make that a binding contract. And, you know, copyright would preempt that. You know, similarly if they said no quoting, you know, no way. And if you know, think about it. Newspapers could never live in that world. Newspapers are the greatest beneficiaries in many ways of copyrights limitations and exceptions, that they can reuse facts they get from third parties, they can quote from third parties as fair use, right? So that’s something we’ve always relied on. And we’re seeing more and more attempts where these things are coming into conflict, where you know, sites are saying in their terms of service, you may not crawl my site to do X, Y, and Z. There have been a set of cases that have started to reckon with that and say, you’re not allowed to try and create an information monopoly through contracts that copyright prohibits, where copyright gives these rights to the public, you’re not allowed to strip them from the public. They stay with the public.
Now I’m sort of riffing on X v Bright Data, which is a case of you know about X, and then Bright Data is a data collector that helps with a lot of different data analysis. You know, there are a set of cases like that. And this is still a moving target in the law. I think it’s still very much evolving. And there are these other tools that pop up that are contract, but then are used to sort of accomplish the same thing. So with your ebook, they don’t just put a contract in front of you that says you may not give this away. They use digital rights management, digital locks that prevent you. And if you were to circumvent those, you could be liable. And if somebody creates a tool that helps you circumvent, they could be liable.
Similarly, going back to Amazon and Perplexity. There are contract elements there, but the core claim from Amazon is, you’re hacking my site. You’re violating this anti-hacking law. And both of these sorts of claims are being used. You know, in some cases, news publishers have said robots.txt is the same as that technological protection measure that’s on your ebook. And so if we say no AI training in our robots.txt, then dot dot then you shouldn’t be able to… Or all these things. We put in our terms of service. That’s a technological protection, all these sorts of things.
So a lot of claims are being thrown at the law, but I think it comes back to what rights stay with the public because copyright gives it to them and what can be revoked through other means.
Matt Perault (31:01)
What about unlawful access issues generally? So like in the bookstore example, you can buy a book and learn from it. The book publisher can’t prevent resale, but you can’t break into a bookstore and steal a book and then if you’re apprehended, say, well, I was doing it to learn.
Derek Slater (31:16)
Yeah, and I think that is sort of tried and true and holds here. Rather, there are allegations in various, you know, cases about circumvention of a paywall and so on. And I think all of the major AI developers say that they respect paywalls, login walls, they do not try to circumvent those. Now, if the user has credentials to the site, right? I have a New York Times login and subscription and then I use my AI, well, that’s a different story, right?
So you have to sort of tease those apart. But generally I think the AI developers, I don’t think that is the real challenge here. And that’s why we do see deals evolving for access to data. I think that people have ways of putting their data behind walls and locking it up already, and then, arranging with AI developers, if you want to acquire this, here’s what it costs. What’s missing in that is that the vast majority of all works that have ever been created are not managed in any way? Right? There are like over a billion websites. Most of them are not managed, but are still up there. You know, books from the 20th century. Most are not managed or licensed in any way, never will be. The rights holders cannot be easily contacted. They’re out of commerce, or they’re just really old.
So anyway, back to lawful access, I think the other place where this has come up is what if you lawfully access the data, but the data was books that someone else had pirated. This is what came up in the Anthropic case. And I think that case, in my reading, took a narrower line and looked at, okay, it wasn’t just the acquisition of the books, but you kept the books and maybe you didn’t secure them well enough. And that’s different from Google Books, which puts really good security controls around its books. Other people read that case differently and say, no, you are not allowed to use pirated material. And if you had knowledge of some sort that this was from a pirate site, don’t use it. And so there are all these now tons of lawsuits about the books database, LibGen, these other databases that, you know, were up there on the web, just like any other website, but people are saying, those should have been off limits. I think that’s going to be a trickier issue because I don’t see the easy distinction between these books that are publicly available in a you know a BitTort file and an HTML file. That’s sort of an artificial distinction.
Matt Perault (33:42)
So you kind of mapped out like the issues, the underlying bodies of law that apply. Looking forward to the piece, we wrote about market evolution, technical collaboration, and public policy guidance as three ways to think about addressing various different components here. You’ve kind of teased each of these, I think, in your comments. But let’s go into them a little bit in more detail. So market evolution, what do you think are the approaches there that would have a meaningful effect on preserving the freedom to law.
Derek Slater (34:13)
What we’re seeing happen in the market already are again both deals around access to data that people would not otherwise be able to, you know, they’re not on the public web. They’re behind a paywall, they’re locked in somebody’s archives or whatever. We’re seeing deals already for that sort of data. I think that will continue. But again, that’s fine. It’s just important to realize that most information ever produced will never be licensed and acquired in that way. And in the sort of the sum of human knowledge and history, everything is copyrighted as soon as it’s fixed in a tangible medium. You write a letter, you know, write some letters on a napkin, copyright. And nobody’s licensing and managing all of those different scratches and scribbles and websites. But that will still be a valuable and meaningful business model.
And then I think on the output side, things get really interesting. Again, like with YouTube, where rights holders and service providers and users come together to create something that wouldn’t be possible otherwise just under fair use and whatnot. You know, I think while Sora itself, the OpenAI product, is gone, that was the nature of the deal that they cut with Disney, to say, okay, we can use Disney characters and outputs in this way that otherwise we might not be able to. I imagine a lot of things like that will evolve over time. So that’s I think part of the business model evolution.
I think there will be other things that involve publishers that could involve more of the sort of real-time information for sort of facts and that come in what’s called grounding of models with certain factual information. I think that will continue to evolve. And we’re certainly seeing lots of different marketplaces and mechanisms to attribute value. And I don’t have a firm view of what will be best, but I think that you know experimentation can help lead the way.
And one thing to underscore here is those markets would not be existing but for fair use. That is, the ability to use stuff for free under fair use enabled this new market to be created around AI, which is leading to new revenue sources. That is a tale as old as time in media and information. It’s what we saw with the, you know, early internet platforms and search engines, is what we saw with the VCR. I was recently reading a paper that was written at the dawn of cable television. It’s called The Television of Abundance, right? An abundance agenda for the 70s. And it was. They were wrestling with these same issues. This infrastructure is so important. It’s going to lead to all these things in creativity. And how do we figure this out? So, you know, I think recognizing the free use which cable TV had of broadcasting back in the day opened up new markets, new opportunities. And I think we’ll continue to see that in the marketplace. Fair use is not antithetical, right? Copyright’s limits are not antithetical, rather, they help create new markets.
Matt Perault (37:01)
What about technical collaboration? I think the main concept there is thinking about ways to implement some version of robot.text for an AI world. And I think there are considerations there that are most important to the part of the tech ecosystem that we’re focused on, Little Tech, which is you talked about cases of like a provider burying a do not train provision and terms of service. That obviously seems like it’s the kind of thing that for small companies with small legal teams, it’s gonna be really hard for them to implement in practice. So what are the ways that you would think about making robots.txt workable for an AI ecosystem that also work for Little Tech.
Derek Slater (37:42)
Tomorrow, there is a meeting of the Internet Engineering Task Force AI Preferences Working Group, which for the last year and a half has been working to wrestle with this question of how might we create a standardized vocabulary about different sorts of AI use cases, foundation model training, decisions about whether or not to be included in search versus AI overviews that are in Google search, what to do about other sorts of inference or use, you know, should I be able to tell a browser you may not summarize this for a user, even if it’s the user clicking the summarize button? Those have all been live issues. And on the one hand, standards work is really hard and frustrating. And if you go through the mail list, it’s a mess. And through this process, I’m optimistic we’re getting somewhere on the core issues of training and around search. There are thorny issues still to come.
But I think there’s been a lot of collaboration among publishers, AI providers, and civil society. It’s not perfect, but I do think we can get there. And I think I’ve seen a lot of willingness to arrive at that, as you were saying, machine readable, standardized in ways everybody can work with and is feasible. So I have some amount of optimism.
There’s a bunch of other things at IETF too around identity. Should bots have to identify themselves? If so, how and why? Is it because you want to know specifically which company it is? Or are you just trying to figure out are you a well-behaved bot or a bot that’s going to take down my servers? And I want to segment those. So there’s lots of other stuff going there. It is tough. And with all these standards processed, no one is ever perfectly happy. And it’s left me with optimism that at least on the training and the search side, we can start to square that circle.
Matt Perault (39:32)
We’ve been talking about this issue generally, but our focus is really on smallest startups and entrepreneurs who are trying to build AI tools. What are the equities here from a startup perspective?
Derek Slater (39:45)
Let’s start with access to data is so crucial for the development and use and refinement of AI systems. Right. So that means that getting access to data and being able to use data is an asset and a barrier to entry. Right. And let’s remember like the companies we’re seeing today that are new, OpenAI, Anthropic, all these ones had access to public data to train their stuff. That was startup capital that put them on a more level playing field with giants, whether that’s Google or Facebook or whomever.
And if you flip it around, you think about a world where you have to get a license to do AI training and so on and so forth. That means a huge new barrier to entry for everyone. And that is the hardest hit for Little Tech who doesn’t have big pockets or an existing store of data to put a fine point on that. Think about Google that has, you know, both the search index, which they can access more websites than anyone else, any other crawler, but also YouTube. YouTube is this huge video resource.
To give you a sense of that scale, I think when I looked at the data the other day, every five hours, there’s as much video uploaded to YouTube as the entire Warner Brothers catalog all time. And Warner Brothers has one of the deepest catalogs in the history of Hollywood. Every five hours, right? There are like 500 million hours every, you know, added to YouTube. It’s crazy.
So the idea that, you’re Little Tech, you’re gonna go compete with Google on their models and both the transcripts for the language models and the video itself. Like, get out of here. There’s no replicating that. Similarly, books. Google has access to all these books. Yeah, I mean, they can cut easier deals. And if they didn’t have fair use, they couldn’t use their library collection. But you’re not gonna cut deals with all these libraries that already digitize their collection. They already have their copy. They don’t need to work with you. So you’re screwed.
That is so huge for every sort of new entrant and startup. I think to me, this is one of the most challenging things in this debate, where everything does have sort of, okay, what do we do about Big Tech? And when people reach for copyright, they are entrenching Big Tech. They are reinforcing monopoly, right? Limits and exceptions to copyright have always been a core anti-monopoly lever. And I think when people miss that and start to say, “well, we should just propertize everything,” they don’t get how Google and Facebook are going to be the biggest winner in that.
Matt Perault (42:14)
And then last but not least, the law. And that’s often where I think people start as opposed to thinking about market evolution and technical collaboration. But on the public policy side, what’s the roadmap that you think works best here?
Derek Slater (42:27)
Said, I think copyright law is consistent with the way AI is trained. Both copyright’s function and purpose, right? Copyright is there to incentivize creativity. These are tools used to create new stuff. And the training is on the un you know, deriving these uncopyrightable elements, which is exactly the sort of innovation and creativity that copyright was meant to support. And so, yes, I think they’re having fair, clear, but broad exceptions for those sorts of transformative uses to train models makes a lot of sense. And to do text and data mining more generally or computational analysis more generally. And I think that’s consistent with a flourishing market.
By the same token, as you said, breaking into a site to get the data, not okay. Right. Outputting and selling, you know, full copies of somebody else’s stuff and just saying, well, AI. There’s no AI get out card for that either.
So if your model is spitting out and selling, you know, copies of the latest Stephen King novel or the latest Marvel movie or whatever, yeah, that’s not learning. That’s just copying it and forwarding along the copy. So I think you know, there are some lines to be drawn there. And I think the basic line of you should be, I, a person, should be able to use a machine to learn.
To analyze and study and derive these uncopyrightable elements and make other lawful uses, right? Quote from it, summarize it in ways that are lawful. I should be able to do that. I think the the harder challenge here is going to come with where a lot of debate has moved, these other elements of the law that we talked about, contract, anti-hacking law, anti-circumvention law, and reckoning with how those laws go far beyond those other laws, beyond what we would typically think of, of those claims we think of like your property. Like I’m the reader. Once I get your website, shouldn’t I be able to have some rights to do what I want? That’s not an all or nothing. And I think consumer rights are a fundamental part of backstops about other types of bad market failures where consumers lose out, where competitors lose out, where innovation loses out. And so there’s a lot of room in between where there’ll be litigation, there’ll be negotiation standard setting, I’m not saying that, we should force everybody to be interoperable and open. No, but still, I think we should live in a world where when I buy an ebook, I should be able to use a read aloud function and they shouldn’t be able to lock me out of that. Unfortunately, that’s not the world we live in today because of anti-circumvention law. But I hope that this is actually a moment to try and reckon with some of that rather than expand those protections.
Matt Perault (45:18)
Derek, this is super fun. Thanks for coming on the podcast.
This newsletter is provided for informational purposes only, and should not be relied upon as legal, business, investment, or tax advice. Furthermore, this content is not investment advice, nor is it intended for use by any investors or prospective investors in any a16z funds. This newsletter may link to other websites or contain other information obtained from third-party sources - a16z has not independently verified nor makes any representations about the current or enduring accuracy of such information. If this content includes third-party advertisements, a16z has not reviewed such advertisements and does not endorse any advertising content or related companies contained therein. Any investments or portfolio companies mentioned, referred to, or described are not representative of all investments in vehicles managed by a16z; visit https://a16z.com/investment-list/ for a full list of investments. Other important information can be found at a16z.com/disclosures. You’re receiving this newsletter since you opted in earlier; if you would like to opt out of future newsletters you may unsubscribe immediately.
