Recently, there has been a lot of hype about the HTML5 video. No wonder why - Google thinks the format is hot enough it is worth encoding thousads of the YouTube videos as webm and make them available through new HTML5 version of youtube: http://www.youtube.com/html5.

Apple thinks HTML5 video is hot, so it decided to ditch flash player long time ago and embrace the featues of the open source web.

I think HTML5 video is hot, because it allows us - developers, to take full control over the media we are playing in the web browser.

I decided to put this article together to demostrate how easy it is to manipulate video in the real time and perform transformations that were previously reserved only for Flash developers or people actually creating the video.

Playing with pixels

Working example above demostrates several simple real-time filters I applied to the video showing fun trailer of the movie "Madagascar 3: Europe's Most Wanted" (Copyright DreamWorks Animation).

Set up above uses hidden <video> tag that actually plays the video. Each frame is then redrawn on the <canvas> element on the left, and then redrawn again on the <canvas> on the right after applying adequate transformation/filter.

Each filter works in a very similar fashion - it iterates through the pixels of each frame of the video and updates RGB values and stores them back on the canvas. The only difference (apart from Pixelate example which is using slightly different technique) is a color setting applied to every pixel.

For instance, code below demostrates pixel manipulation applied to achieve black&white effect. Firstly, I draw video frames taken from <video> element into a canvas and store pixel information for later.

bcv.drawImage(video, 0, 0, w, h);
var apx = bcv.getImageData(0, 0, w, h);
var data = apx.data;

Next, I interate through the pixel collection and update RGB values of the pixel data object to average value to achieve desaturated black&white effect. Finally, I put all updated data back on to the canvas and display it on the page.

for(var i = 0; i < data.length; i+=4)
{
	var r = data[i],
		g = data[i+1],
		b = data[i+2],
		gray = (r+g+b)/3;
	data[i] = gray;
	data[i+1] = gray;
	data[i+2] = gray;
}
apx.data = data;
ctx.putImageData(apx, 0, 0);

Entire code, together with the canvas object set up:

var video = document.getElementById('canvasVideo'),
	canvas = document.getElementById('canvasVideoCvs'),
	backcvs = document.getElementById('canvasVideoBcvs'),
	ctx = canvas.getContext('2d'),
	bcv = backcvs.getContext('2d'),
	w = canvas.width,
	h = canvas.height;

function drawBlackWhiteFrame() {

	bcv.drawImage(video, 0, 0, w, h);
	var apx = bcv.getImageData(0, 0, w, h);
	var data = apx.data;

	for(var i = 0; i < data.length; i+=4)
	{
		var r = data[i],
			g = data[i+1],
			b = data[i+2],
			gray = (r+g+b)/3;
		data[i] = gray;
		data[i+1] = gray;
		data[i+2] = gray;
	}
	apx.data = data;
	ctx.putImageData(apx, 0, 0);

	if (video.paused || video.ended) {
	  return;
	}
	setTimeout(function() {
		drawBlackWhiteFrame();
	}, 0);
}

Performance, performance

Obviously there are some limitations to what we can do when it comes to real-time video processing in JavaScript. Best example to demostrate it is Pixelate filter that does not perform that great on slower machines. This filter is doing additional processing to draw squares (pixels) on the canvas where color is corresponding to the actual color of the pixels behind.

function drawPixelFrame(blocksize) {
	bcv.drawImage(video, 0, 0, w, h);

	//apply pixalate algorithm
	for(var x = 1; x < w; x += blocksize)
	{
		for(var y = 1; y < h; y += blocksize)
		{
			var pixel = bcv.getImageData(x, y, 1, 1);
			ctx.fillStyle = "rgb("+pixel.data[0]+","+pixel.data[1]+","+pixel.data[2]+")";
			ctx.fillRect(x, y, x + blocksize - 1, y + blocksize - 1);
		}
	}

	if (video.paused || video.ended) {
	  return;
	}
	setTimeout(function () {
		drawPixelFrame(blocksize);
	}, 0);
}

Nested loop and performance hit related to drawing shapes on the canvas really slows things down but I am sure as browsers get faster and JavaScript proves its superiority gets faster, we will see more cool effects.

You can access JavaScript code used to generate each of the filters here: https://gist.github.com/1888841