Kalyan Varma : Google SOAP API

Did you always want to have a little search engine in your site and never had enough resources ? Why not let google do the work for you ? Google provides a very neat SOAP interface for it's search engine. So you can query google via the SOAP interface and get the results.

Google also has a nifty feature where you can limit the search to a specific domain. For example, if you want to search for a string only at say apple.com site or domain, then you can say "site:apple.com ". This will only return search results from apple.com website and from no where else.

So, to get search results only from your site, search at google for the what I have mentioned above and pull the results via the SOAP interface provided by google. Instead, you could have a search box and send the query to google : http://www.google.com/search?q=site:apple.com+linux, but this will take the user to google, which is not what you want.

Anyway here are the 5 easy steps to have a search engine for your site.

1. Before you go ahead, make sure of the following :

You have atleast more then 10 - 15 html pages in your site

All the pages are indexed by google

All of them have suitable titles.

There will be less then 1000 queries per day.

2. On your web server, install the perl module SOAP::Lite. Do do this, do the following

$ perl -MCPAN -eshell
cpan> install SOAP::LIte

3. Now you have added all you need on the webserver end. To query google via the SOAP interface you should create an account with google and obtain a license key. ( Fill up your email address, and the key will be mailed to your mail address). Remember that google lets you make only 1000 queries per day for free. If you need to make more than that, then you will have to pay google. Google does not let you query them without the license key, which is also used to keep track of how many queries you have made per day.

4. You will also need to get the google WSDL file. This comes as part of the Google developer kit. Once you unzip the file, you will find GoogleSearch.wsdl. Copy this file over to your webserver.

5. Fetch the results using the SOAP interface , as shown below.


#!/usr/bin/perl
use SOAP::Lite;
my $key='000000000000000000000000';
my $query="foo";
my $googleSearch = SOAP::Lite -> service("file:GoogleSearch.wsdl");
my $result = $googleSearch -> doGoogleSearch($key, $query, 0, 10, "false", ""
, "false", "", "latin1", "latin1");
print "About $result->{'estimatedTotalResultsCount'} results.\n";

Don't forget to fill in the $key variable in this script, with the key that you receive. The above script returns the number of results that were found on google. To get the actual results, use the method, 'resultElements' instead of 'estimatedTotalResultsCount'. This returns a reference to an array, which has each search element as an element of the array. Each of the elements is actually a Hash table, with various values. Here is my cgi script, which implements the search engine. ( remember to fill in the $key )


#! /usr/bin/perl -w
use SOAP::Lite;
use strict;
use CGI qw(:standard -no_debug);

my $CGI = new CGI; 
if ($CGI->param("action") and ($CGI->param("action") eq "search")) {
   &Search();
}
else  {
   &Display();
} 

sub Display() {
    print "Content-Type: text/html\r\n";
    print "\r\n";
    print "<html>";
    print "<form method=post action=\"/cgi-bin/search.pl\" name=s autocomplete=off> ";
    print "<input maxLength=256 size=25 name=q value=>";
    print "<input type=hidden name=action value=\"search\">";
    print "<input type=submit value=\"search\">";
    print "</form></html>";
    
}

sub Search() {

    my $query = $CGI->param('q');

    my $key='xxxxxxxxxxxx';
    $query="site:kalyanvarma.net " . $query;

    my $googleSearch = SOAP::Lite -> service("file:GoogleSearch.wsdl");
    my $result = $googleSearch -> doGoogleSearch($key, $query, 0, 10, "false", "",
    "false", "", "latin1", "latin1");

    my @results = $result->{'resultElements'};

    print "Content-Type: text/html\r\n";
    print "\r\n";

    foreach my $dummy_array(@results)
    {
        if (defined($dummy_array)) { 
        foreach my $ResultElement(@{$dummy_array}){
            if (defined ($ResultElement) && defined($ResultElement->{'URL'})) {
                my $url = $ResultElement->{'URL'};
                my $snippet = $ResultElement->{'snippet'};
                my $title = $ResultElement->{'title'};

                if (defined($url)){ 
                    print "<a href=\"$url\">$title</a><br>\n";
                    print "$snippet<br>\n";
                    print "$url<br>\n";
                    print "<br>\n";
                } else {
                    print "Could not find anything";
                }
            }
        }
        } else {
            print "Could not find anything";
        }
    }
}
exit(0);

You can see the search engine of my site here.

Using Google SOAP API for fun and profit