Extract website data using php

Many time web programmer needs to get some data from other website. Extraction of particular data from other website is also known as web scraping or Web Harvesting. In this tutorial i will explain how to extract data from website using php.

First extract complete html source of webpage

php has inbuilt function file_get_contents to do this

$html=file_get_contents("http://www.somewebsite.com")

But i will not recommend it because we can not set timeout parameter in function. So instead of this i will use curl library.

Here is php function to get html source of webpage using curl

function getHTML($url,$timeout)
{
       $ch = curl_init($url); // initialize curl with given url
       curl_setopt($ch, CURLOPT_USERAGENT, $_SERVER["HTTP_USER_AGENT"]); // set  useragent
       curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); // write the response to a variable
       curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true); // follow redirects if any
       curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout); // max. seconds to execute
       curl_setopt($ch, CURLOPT_FAILONERROR, 1); // stop when it encounters an error
       return @curl_exec($ch);
}

Now use regular expression match to extract particular data from source

Here is php code to extract title of website

$html=getHTML("http://www.website.com",10);
preg_match("//i", $html, $match);
$title = $match[1];

If you don’t know how to use regexp then there is a php class simplehtmldom to parse html.

include_once("simple_html_dom.php");
//use curl to get html content
function getHTML($url,$timeout)
{
       $ch = curl_init($url); // initialize curl with given url
       curl_setopt($ch, CURLOPT_USERAGENT, $_SERVER["HTTP_USER_AGENT"]); // set  useragent
       curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); // write the response to a variable
       curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true); // follow redirects if any
       curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout); // max. seconds to execute
       curl_setopt($ch, CURLOPT_FAILONERROR, 1); // stop when it encounters an error
       return @curl_exec($ch);
}
$html=str_get_html(getHTML("http://www.website.com",10));
// Find all images on webpage
foreach($html->find("img") as $element)
echo $element->src . '
';
// Find all links on webpage
foreach($html->find("a") as $element)
echo $element->href . '
';

SimpleHtmlDom has selectors just like jquery. So it is very easy to use.