Thoughts on Union Square Venture’s “research” on online education

I am not sure how this only came across my radar this morning given its publication 2 weeks ago, but Union Square Venture’s “research”/investment thesis on online education was an interesting development for me to observe given my previous background in finance. The “research,” which has been widely noted as rather slim (and not so much research of any sort but rather a scattered collection of media articles), was not all that original given the existing chatters of the space— but obviously it provided a good anchor point for the “tech/product/investment” community to share their thoughts a la Techcrunch style. Rarely would one see 100+ comments on any of the popular education tech news and blog sites, when 10+ already signifies unusual momentum. So, there is some silver lining to this thing: at least it got some people talking and perhaps thinking a bit more about the space, even if it can be uninformed enthusiasm layered on top of an uninformed starting point.

Audrey Watters was kind in lauding USV’s effort:

Kudos to USV because as is the case in many, many sectors, there are plenty of folks who argue that you must keep this sort of knowledge locked up and proprietary.

I think that is generous, and perhaps also a little naive. There are two things to note here 1) USV is not doing this out of some altruistic purpose to inform others; there is a very strong PR element to what they are doing and also in shaping how they want the sector to develop. 2) An investment thesis in a few dozen words is not truly proprietary; an investment thesis exposited with data and knowledge that only you have access to is truly proprietary. On this, USV divulged little (and I doubt they would share much); this is where the true value of their knowledge lies and it will not be given away easily.

I am sure that educators and perhaps even entrepreneurs, heck, anyone that hasn’t been in the industry, would be appalled to learn what really goes on in financial industry to “evaluate” and “predict” investments. There is a lot of art that we comfort ourselves as science: a weak linear regression based on <10 data points, a heavily guesstimated financial model, some words extracted from periodic publications. But even that presents the more effortful side of the “research” that takes place; on most days it is an opinion disguised as some revelatory trend that can only be derived from “industry experience”. Worse still, even when there is much research effort that takes place at times (usually in consulting, not as much in finance), the research is either forcibly interpreted to fit the bill rather than fully exposited for its ambiguity, or built upon a scientifically questionable methodology. But consulting houses and banks and funds will continue to publish their research: because they want to be seen as thought leaders and indirectly have a hand in influencing the way the sector forms (through partially self-fulfilling prophesies; psychology and PR drive more Wall Street fluctuations than actual financial happenings).

Finally, I want to comment on Christina Caciopo’s response to Greg Wilson on the quality of their research. To USV’s credit, they responded very politely and constructively to these criticisms. One thing stood out to me in their defense of why the research consisted mostly of popular media articles.

Busy people won’t read narrowly-focused papers reporting micro-progress on nano-topics — they need overviews they can absorb in a hurry, because they have nine other meetings this week to prepare for.

This is true, but this does not logically prevent a research house to do the proper research then digest it for people to read. You can sift through papers and still produce informed, succinct summaries. What she really means (but perhaps doesn’t realize) is that “busy” people have a hard time tolerating ambiguity. It doesn’t make for feel-good investment thesis, and it doesn’t make for the emphatic positing of a thought leader. (And on the nomenclature of “busy”: not for a second do I believe that a partner at a VC fund actually produces substantially more work than, say, a Georgia Tech professor. I am not saying he produces less. But certainly the two are on par).

What I am left wondering by myself is— why is it that it is the “busy” but “uninformed” people that get to shape the biggest decisions? We cannot always rely on “them” to be informed; perhaps what we should be asking ourselves is why we try not to ask for a larger voice in choosing what gets support—or not, in the sector.

Advertisements

4 comments

  1. Hi Jasmine — Christina from USV here. Thanks for your feedback on the post.

    You’re right there’s a PR element to what we posted, but we want much more than website hits and kudos. We’re looking for 1) feedback on what we got wrong and what we should be looking at (your point about 100+ comment threads is spot-on); and 2) entrepreneurs working in spaces we find compelling to know that we’re interested.

    We’re not experts, and I hope we didn’t come off as trying to be such. We’re learning, very actively and very much in public. That means getting things wrong in public too. So if there’s resources, publications, studies, or anything else you think we should be looking at — even if they’re academic and dense — please do let us know.

  2. Hi Christina,

    Thanks for reaching out. I do really appreciate that you have been following on on the feedback received various blog posts. It’s gracious and constructive of you guys and that is something admirable.

    Let me explain a few things about my post
    1) The primary intended audience of this post is actually not USV, or anyone in the finance industry. That is probably hard to believe given that you are the subject matter of the post, but my intended readership was actually the people who are interested in education technology but did not know much about the inner workings of the finance industry. Of course, every firm conducts its due diligence differently– but what I wanted to bring attention to was that this is mostly an art and not a science. I am not advocating that things must be done differently (after all, businesses operate under their own priorities and deadlines), but it’s important that people understand that the judgment arrived at is not always some peer-reviewed, neutral scientific discovery but rather a product comprising subjective views and leaps of logic (at times, for lack of data). This is something that people not privy to the scene are not always aware of. I felt very compelled to write the post when I felt that Audrey Watters’ statement indicated such lack of awareness.

    2) I had no problem with your investment theses, in fact I found them pretty interesting. As with all investment theses, they are predictions that are researched to various degrees— there isn’t a requirement for there to be research at all, in my opinion, if it is backed by well-informed personal experiences. I feel that a lot of people actually missed the point that your theses is not always related to the research that they have been looking at; your sector hypotheses in fact do not directly relate to discussions on pedagogy.

    That said, I think the trouble started when you juxtaposed your theses with the “research” that you have done. It immediately becomes contextual and people start to think about how you got from point A (the research) to point B (your theses); you get an unfair disadvantage because when you talk about education, you enter a realm occupied by many academics who are necessarily sensitive about the approach of these things, first the methodology/sources behind the research, second the actual conclusions that you are drawing. There is a gap between your theses and your research, whether perceived (you are not releasing other data informing your conclusions, rightly so), or actual (you are basing it purely on superficial newspaper articles and videos, which worries the academics).

    My guess is that the gap is perceived rather than actual. Your theses is in fact informed a lot by other metrics and offline conversations which you cannot release to the public. I am sure the collection of articles also formed a backdrop of what’s going on generally in the space: the chatters, the aspirations, but they alone would not constitute the decision process. What is driving the real bulk of your investment decisions is the information you received from the startups directly, whether that is data or informal conversations.

    My small advice would be:
    1) Be careful when you publish the research and the hypotheses together. If your hypotheses are being driven by other offline sources too, make it clear that the “research” that you are presenting is actually only a partial picture of your thought process.

    2) Don’t call it “research”, which is a loaded word that invites scrutiny, esp when juxtaposed with the word hypothesis! Call it background information that you want to share about the sector, or even more plainly, a collection of articles that may be useful to understand popular trends in the sector.

    3) Dig a little deeper, if you are up for it. Your hypotheses at the moment do not actually relate to pedagogy; they are much more business/market-oriented hypotheses, which is why I felt that a lot of the discussions is unrelated to what should be included in the context of your current hypotheses.

    That said, I am sure people want to see you think about those issues too. The obvious questions left out of your theses include: what content delivery format works (questions on pedagogy), and what effective methods of credentialing there can be. One of the biggest elephant in the room surrounding online education is the dropout rate: does it matter for the business model? If these are questions that you want to tackle, then you can delve into the wealth of academic research that has been done on the subject.

    I am not an academic, so I cannot offer you the best advice on this topic, but if you regularly read EdSurge or Mark Guzdial’s Computing Education blog, they usually link to pretty good research reports that come out on the subject.

    Here are a few reports that are not as “hardcore” as traditional academic papers but still more balanced starting points than newspaper articles. There are many more of them out there, and you can usually keep a good list of them if you follow the two above sources that I mentioned.

    Digital Faculty: Professors, Teaching and Technology (http://www.insidehighered.com/news/survey/digital-faculty-professors-and-technology-2012_)
    Conflicted: Faculty and Online Education
    (http://www.insidehighered.com/news/survey/conflicted-faculty-and-online-education-2012)
    Innovating Pedagogy (Open University)– this one is particularly great because it links to other primary research sources
    (http://www.open.ac.uk/personalpages/mike.sharples/Reports/Innovating_Pedagogy_report_July_2012.pdf)

    Anyway, good luck with it! Every industry has its own workings and limitations. It’s a good thing that USV has gotten a wealth of response on this subject, which is the most important thing.

  3. Thanks very much for the thoughtful reply and for the concrete suggestions. They’re already being folded into our upcoming work.

    Your conclusion about “busy” but “uninformed” people shaping the biggest decisions is frustrating for both of us, I suspect. I believe, perhaps naively, that there’s goodwill and a desire to learn on both sides, which means there’s much that experts can do to drive “the biggest decisions.”

    The EdTech Primer, about which I now know through Greg Wilson (http://www.third-bit.com/educate/index.php?title=Main_Page) seems an initiative in this direction. I hope it succeeds.

    I suspect there’s other, similar initiatives — your blog, Audrey’s blog — here too. One of the brilliant things about the internet is that it gives anyone a voice and quickly elevates the best. We’ve now got a better sense of where to “hang out” and overhear these conversations, which is exciting for us.

  4. Hi Christina,

    You are right on the frustrations. I knew that what I wrote was emotionally loaded even as I wrote it, and debated whether to include it. I guess my frustration came from the idea that because people are “busy” they were therefore exempt from absorbing research in a thoughtful, balanced way. I am sure it was not your intention to present it in this light, but the quote that I saw above from you gave me that impression— probably unfairly and distorted by the static nature of the medium.

    There is a gaping hole in the general conversation about edtech, which is the gap between the education-oriented and the business-oriented mind. A lot of the discussions you see from Audrey’s blog and the likes don’t effectively address issues on product development, market analysis, etc. There are many generalizations being thrown around (on both sides) on issues that we are not experts on— the educators make erroneous assumptions about business models (http://gasstationwithoutpumps.wordpress.com/2012/07/26/mooc-humor/) , and Techcrunch-esque people make overly exalted claims about “disrupting” education when similar models of education have been in discussion for decades.

    And of course, that gaping hole is a great opportunity for the likes of USV to start initiating more balanced, informed discussions. It’s an exciting place to be in.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s