Embed Video by URL / Web Scraping (C#)
I'm sure most if not all of you are on Facebook. You'll notice that when you copy a url into your status, Facebook is able to generate an image/video/link based on that url and post it in your status. This can be achieved by a method known as data/web scraping. Scraping is basically extraction of data from another page - as long as you have access to the markup, you can scrape it. There are tools out there that make this easy, one of the most popular being the HtmlAgilityPack. This tool allows you to parse an HTML page into nodes similar to XML, and while that opens the door to many possibilities we are only going to use part of it in this post. With that being said, let's get started.
First, create a new website project. Download the HtmlAgilityPack and add a reference to the dll in your project. Next, create a new class called OutputVideo.cs and in this class create a Get() function that accepts a string (for the URL). So far what you have should look like this:
<![CDATA[ public class OutputVideo { public static string Get(string url) { } } ]]>
Now when it comes to video, we are fortunate that many sites use Facebook's OpenGraph feature. OpenGraph (OG) is basically a series of meta tags containing information about the content of a page. In the function we are about to create, we are going to use mainly OG and then add support for a few other sites without OG. Let's begin by supporting the sites that use OG. From here on out, all the code we write will be inside of our Get() function.
We'll start off by creating a new HtmlDocument, load in the desired URL and loop through the meta tags to find the OG specific ones.
<![CDATA[ string output; HtmlWeb web = new HtmlWeb(); HtmlDocument doc = web.Load(url); Dictionary<string, string> openGraph = new Dictionary<string, string>(); foreach (HtmlNode meta in doc.DocumentNode.SelectNodes("//meta")) { var property = meta.Attributes["property"]; var content = meta.Attributes["content"]; if (property != null && property.Value.StartsWith("og:")) { openGraph[property.Value] = content != null ? content.Value : String.Empty; } } ]]>
The next step is to retrieve the video from the page and use the source to create an embed tag that we can use. Most pages that use OpenGraph (and contain a video) will include an og:video meta tag that contains the source of the video file. This makes our job very easy, as we can just look for that tag and plug it in to our embed code.
<![CDATA[ if (openGraph.ContainsKey("og:video")) { // Get the MIME Type string mime; if (!openGraph.TryGetValue("og:video:type", out mime)) { mime = "application/x-shockwave-flash"; } // Build the embed tag (our output) output = String.Format("<embed src=\"{0}\" type=\"{1}\" width=\"{2}\" height=\"{3}\" />", openGraph["og:video"], mime, "600", "400"); } ]]>
All is well as long as this tag exists, but what if it doesn't? There are a few popular video hosting sites that do not use the og:video meta tag, so I looked for ways that I could support those sites also.
Almost all YouTube videos have the og:video tag, but some of them don't - these are generally the ones that say they don't allow embedding. Here is an addition to our if statement that will cover most of them:
<![CDATA[ else { //Alternative - for vids without og:video if (openGraph.ContainsKey("og:site_name") && openGraph.ContainsKey("og:url")) { //YouTube - alternative if (openGraph["og:site_name"].ToLower() == "youtube") { string prefix = "http://www.youtube.com/v/"; string suffix = "?version=3&autohide=1"; Uri uri = new Uri(openGraph["og:url"].ToString()); string uriQuery = uri.Query; string YouTubeID = HttpUtility.ParseQueryString(uriQuery).Get("v"); string resolved = prefix + YouTubeID + suffix; output = String.Format("<embed src=\"{0}\" type=\"{1}\" width=\"{2}\" height=\"{3}\" />", resolved, "application/x-shockwave-flash", "600", "400"); } ]]>
In the example above, I've taken the prefix/suffix of a working YouTube link and determined the video ID by the query string parameter "v". Concatenate these and you have a resolved link which you can input into your embed tag. Next example is Photobucket, and we are basically doing the same thing - we know what format our url needs to be in, so we are stripping out the variables we need and creating a usable src string for our embed tag.
<![CDATA[ //Photobucket else if (openGraph["og:site_name"].ToLower() == "photobucket") { string src = "http://static.photobucket.com/player.swf"; Uri uri = new Uri(openGraph["og:url"].ToString()); string uriQuery = uri.Query; string videoID = HttpUtility.ParseQueryString(uriQuery).Get("current"); string preUrl = openGraph["og:url"].ToString().Substring(0, openGraph["og:url"].ToString().LastIndexOf("/") + 1); string vars = "file=" + preUrl + videoID; output = String.Format("<embed src=\"{0}\" type=\"{1}\" width=\"{2}\" height=\"{3}\" flashvars=\"{4}\" />", src, "application/x-shockwave-flash", "600", "400", vars); } ]]>
The last supported type I have is DailyMotion, which doesn't allow direct embedding of videos. A workaround to this one is to use an iframe. Demonstrated below:
<![CDATA[ //DailyMotion else if (openGraph["og:site_name"].ToLower() == "dailymotion") { //just need to convert URL to embed URL string src = openGraph["og:url"].ToString().Replace("/video/", "/embed/video/"); output = String.Format("<iframe frameborder='0' width='600' height='400' src='{0}'></iframe>", src); } ]]>
With this function you can allow users to embed videos from the most popular video hosting sites. More importantly you are seeing ways that you can make it work for all sites. Say you want to scrape all img tags, or all embed tags - now you know how to do it, and if anything you can provide the option (like facebook does) to scroll through each scraped element and let the user choose the one they want to use.
Below is the full OutputVideo class that we've written, hopefully it will be a good starting point for integrating video embedding / web scraping into your project!
<![CDATA[ public class OutputVideo { //CURRENTLY SUPPORTED VIDEO TYPES - Others may work, but these are guaranteed // 1. YouTube // 2. Vimeo // 3. Myspace Video // 4. Photobucket // 5. DailyMotion // Sites that utilize Facebook's OpenGraph video feature. public static string Get(string url) { string output = ""; var doc = new HtmlAgilityPack.HtmlDocument(); string output; HtmlWeb web = new HtmlWeb(); HtmlDocument doc = web.Load(url); Dictionary<string, string> openGraph = new Dictionary<string, string>(); foreach (HtmlNode meta in doc.DocumentNode.SelectNodes("//meta")) { var property = meta.Attributes["property"]; var content = meta.Attributes["content"]; if (property != null && property.Value.StartsWith("og:")) { openGraph[property.Value] = content != null ? content.Value : String.Empty; } } // Supported by: YouTube, Vimeo, CollegeHumor, etc if (openGraph.ContainsKey("og:video")) { // Get the MIME Type string mime; if (!openGraph.TryGetValue("og:video:type", out mime)) { mime = "application/x-shockwave-flash"; // should error } // Build the embed tag (our output) output = String.Format("<embed src=\"{0}\" type=\"{1}\" width=\"{2}\" height=\"{3}\" />", openGraph["og:video"], mime, "600", "400"); } else { //Alternative - for vids without og:video if (openGraph.ContainsKey("og:site_name") && openGraph.ContainsKey("og:url")) { //YouTube - alternative if (openGraph["og:site_name"].ToLower() == "youtube") { string prefix = "http://www.youtube.com/v/"; string suffix = "?version=3&autohide=1"; Uri uri = new Uri(openGraph["og:url"].ToString()); string uriQuery = uri.Query; string YouTubeID = HttpUtility.ParseQueryString(uriQuery).Get("v"); string resolved = prefix + YouTubeID + suffix; output = String.Format("<embed src=\"{0}\" type=\"{1}\" width=\"{2}\" height=\"{3}\" />", resolved, "application/x-shockwave-flash", "600", "400"); } //Photobucket else if (openGraph["og:site_name"].ToLower() == "photobucket") { string src = "http://static.photobucket.com/player.swf"; Uri uri = new Uri(openGraph["og:url"].ToString()); string uriQuery = uri.Query; string videoID = HttpUtility.ParseQueryString(uriQuery).Get("current"); string preUrl = openGraph["og:url"].ToString().Substring(0, openGraph["og:url"].ToString().LastIndexOf("/") + 1); string vars = "file=" + preUrl + videoID; output = String.Format("<embed src=\"{0}\" type=\"{1}\" width=\"{2}\" height=\"{3}\" flashvars=\"{4}\" />", src, "application/x-shockwave-flash", "600", "400", vars); } //DailyMotion else if (openGraph["og:site_name"].ToLower() == "dailymotion") { //just need to convert URL to embed URL string src = openGraph["og:url"].ToString().Replace("/video/", "/embed/video/"); output = String.Format("<iframe frameborder='0' width='600' height='400' src='{0}'></iframe>", src); } } // Else - you can try to scrape other objects from the page } return output; } } ]]>