Featured Post

Man behind the Curtain for al-Qaeda in Syria is Assad

Syrian President Bashar al-Assad wanted the recent Geneva II peace conference to focus on terrorism. He says terrorism is the main problem a...

Friday, June 21, 2013

Trust but Verify: How to Download YouTube Videos

YouTube videos are playing an important role in documenting the social news of our time.

This is the way news breaks today, whether it is on the battlefield in Aleppo or a protest in São Paulo, someone uploads a video to YouTube and then Tweets it out. Often it is completely unedited recordings from a cell phone or camera and not minutes old.

I have used hundreds of these YouTube videos in my blog posts at WL Central, the Daily Kos and here, on the revolutions in Libya, Syria and elsewhere in MENA and over time I have noticed a troubling phenomenon: Some of them have disappeared.

Most often they are replaced by a black screen with the notice "This video is no longer available because the YouTube account associated with this video has been terminated."

Whenever I see that, I think "terminated," that's a bad word to use in connection with activists and journalist in a revolutionary situation. It brings me visions of someone sitting in front of a computer and deleting their YouTube, Facebook, Gmail and Twitter accounts at gunpoint just before they, themselves are "terminated."

Hopefully, the reality is not that grim, but the fact remains YouTube videos have occasionally been known to disappear for whatever reason and unless there is a downloaded copy elsewhere, a piece of recorded history may be lost forever.

We must also guard against the future possibility that a change in management at Google or a change in ownership of YouTube could lead to a change in policy and the wholesale deletions of material that, shall we say, may prove an embarrassment to the powers that be. That is why I say "Trust but verify." So far, I think YouTube has been a pretty good steward of the people's video history but we need a backup.

I am writing this post in the hopes that many activists will start creating their own collections of YouTube videos from the many blogs, tweets and other sources covering the people's revolutionary struggles around the world. If it's important to you, download it and help insure that it will never disappear.

With the right tools it is easily. I use Linux and the Linux command line or terminal interface, so I will show you how I do it. No doubt, there are other ways to do this on a Windows box or a Mac, and if you know them, please share what you know in comments.

Linux and especially the command line interface [CLI] may seem a bit intimidating at first, but as you will see, its really not that hard and can be very powerful. And look at it this way, if you use Google, you already use Linux. It runs their servers. Hell, If you use the Internet or own an Android, you already use Linux.

Moreover, if you think yourself progressive or revolutionary, or if you want the best possible foundation for keeping an intrusive government out of your computer files, you really should get to know Linux. On the other hand, if you want to make it easy for the NSA to just browse your hard drive at will, just keep using Windows.

And as a bonus, since all the software we will use to download YouTube is Free Software, this solution won't cost you anything.

If you don't already have Linux I suggest you download the latest DVD image of Ubuntu Linux with this link. There are more favors of Linux than there are of socialism. Ubuntu is the world's favorite, mind too, so we will work with this. Ubuntu is an ancient African word meaning 'humanity to others'. It also means 'I am what I am because of who we all are'.

You will them need to burn it to a DVD. The same page that the link above leads to will also have help in how to burn a DVD from Windows, OS X, and Ubuntu in the lower right-hand column. There are also instructions for creating a bootable USB stick for those that don't have a DVD drive.

These instructions were tested on Ubuntu Desktop version 12.10.

Once you reboot from the DVD or USB stick, you will be given the option to run Linux without installing it, most likely this is what you want at first. Click on the [Try Ubuntu] button.

Once the Linux Graphic User Interface [GUI] is up, you will need to open a terminal, either with the keyboard combination Alt-Ctrl-T or by clicking Dash Home [Ubuntu logo, upper left corner], typing term and then clicking on the terminal icon.

Either way, a terminal will open and you can now type commands into the command line interface.

The key to our ability to download YouTube videos from the Linux CLI is a little program called youtube-dl, if you follow that link you will see that they also have a Windows version. We are interested in the Linux version.

That doesn't come with the standard Ubuntu install but it is easy enough to add. First we have to enable the "Universe" Ubuntu software repository and rather that repeat what has already been well said, I will refer you to this link: How do I enable the “Universe” repository?

Once you have done that, you can return to the terminal (click the mouse cursor anywhere in the terminal window.) and type:

sudo apt-get update

this will update your local software list:

Then type:

sudo apt-get install youtube-dl

Once you answer by pressing Y, it will download and install youtube-dl and a number of other packages it requires.

You can read more about the sudo and apt-get commands by clicking the names.

If you install Ubuntu Linux on your hard disk, you will have to perform the above steps only once, because the LIVE Version runs entirely in memory, all changes will be lost as soon as you reboot. Also because the LIVE system runs on a ram disk, you will need change to a suitable directory on a mounted drive so that the downloaded YouTube videos will be saved to the hard drive.

Most likely at suitable drive was mounted automatically and mounted with write permission. Otherwise you may have to do a little extrema work and get familiar with the mount command.

Look at the bottom of the column on the left and you should see the icons of your mounted drives. You may browse these to find the drive/partition and directory you want to save the YouTube videos to. Remember the PATH to that directory given at the top of the file manager. For example if it says media windows pictures
you are going to want to type
cd /media/windows/picture
in the terminal window to change to that directory. You can read about the cd command here
Here is a helpful hint: Open this page in the Linux Desktop and then you can simply cut-n-paste these command line instructions to the terminal command line. That makes this stuff real easy. You can open Firefox with the logo button on the left side.
youtube-dl is an amazing little script. You feed it the URL of a YouTube video and it retrieves the page, captures certain meta-data about the video and downloads the file to your computer. The command you might type into the terminal could look something like this:
youtube-dl -t http://www.youtube.com/v/YncVKgpTd6E

The -t option tells the program to include the video's title in the filename and is useful in identifying files. In this case: Google_s_domination_of_the_world_and_loss_of_mojo_谷歌征服世界-YncVKgpTd6E.flv

The .flv extension indicates that it is a flash video file. YouTube videos are generally either flash [.flv] or mpeg4 [.mp4] and to view them under Windows you may need to install a new video player or new video codecs.

Mplayer is a good video player for them under Linux and ffmpeg will allow you to convert them to other formats such as those used on DVDs or suitable for editing.

You can capture the URL of an embedded YouTube video by hovering over it, clicking the right mouse button and choosing "Copy video URL" at the top of the list.

This brings us nearer to the real power of using youtube-dl from the command line because the key to that power is not the ease with which it can download a single video but the ease with which it can download many videos.

Because the terminal understands the bash scripting language natively a simple command like
for f in `cat YT-blog.lst`; do youtube-dl -t $f; done
will take a list of YouTube URLs, each one on a separate line, and one by one, download them all to your hard drive. In this case the list is in a file named YT-blog.lst in the working directory and looks something like this
You can read about the cat command here, the for loop is an internal bash command.

Such a list can be created in the Ubuntu environment by opening a text editor [ click on Dash Home (Ubuntu logo in upper left corner, them type 'text' and click on the text editor icon when it comes up. ] and pasting the

copied URLs to it, one per line and then saving it to the filename you will use later.

This file could also be created beforehand in the Windows environment with Notepad and Windows Exploder and there are even Unix like environments for Windows that might allow you to use these commands under Windows, but I won't go there.

A more advanced example, using other tools in the Linux environment, such as grep, sed, sort, and uniq and by piping '|' a text stream between them I can create a filter that can read the entire xml download of my blogspot blog [filename: blog-06-07-2013.xml ] and create a file named YT-blog.lst

The command that did this look like this:
grep "http://www.youtube.com/" blog-06-07-2013.xml|sed "s|http://www.youtube.com/|\nhttp://www.youtube.com/|g"|sed 's|"|\n|g'|sed 's|?|\n|g'|sed 's|&|\n|g'|grep "http://www.youtube.com/"|sort|uniq>YT-blog.lst
So by going to my Blogger dashboard->Settings->Other

choosing Export Blog and exporting my blog to the above file and them running the above command on it to create the video URL list and then running this command on that list:
for f in `cat YT-blog.lst`; do youtube-dl -t $f; done
I was able to download the over 900 YouTube videos that I used in over 600 Linux Beach blog posts that still remain on YouTube. It took 2 days on my slow ass DSL line and takes up 17GB on my hard drive but I will no longer have to look at a video has been terminated notice and feel powerless to do anything about it.

I only hope this blog post will encourage other people to do the same and between all of us, we can preserve this video history of the people's struggles to such a depth that it can never be erased

Click here for a list of my other blogs on Syria
Click here for a list of my other diaries on Libya

1 comment: