How do you find information on the web? it's highly likely that you turn to one of the many search engines to cut through the Billions of pages of information to find what you want. But do you ever give a second thought to all the pages that were rejected by this 3rd party on your behalf?

It's widely known that the big search engines like Google, Yahoo and now MSN use intricate algorithms to trawl through billions of indexed pages to narrow down results based on your keywords, but who chooses how those algorithms behave and therefore what you don't see?

To understand the selection process we first need to comprehend the way a search engine functions, the first stage of any search engine is to index content, a spider or web based application starts from one page and follows the links contained within it to move to the next page. Each page is indexed by the spider and all text information is stored within the search engines index or database, overtime a spider will find every page on the internet that is linked from another website.

Having obtained this snapshot of the web one thing is already certain, its out of date! so the indexing process is a continual 24 / 7 process with frequently updating websites being indexed more often building a more accurate repository of pages.

Now that the search engine has all this data it needs a way to present the most relevant sites to you through their search tool and that's where algorithms come in. When you enter a phrase or keyword's as they are known the search engine rapidly selects the best matches according to a very complex set of mathematical rules or algorithms. In a sense this is that part where different search engines start to take on a personality and what makes Google different from MSN or Yahoo, some engines are very open about this for example focusing on a country or topic by weighting the algorithm to favour the Keyword "England" within the web page.

Anyone who runs a website will understand the importance of being ranked highly in major search engines, the average web surfer will favour one engine and even then will only check the first page of results presented to them, perhaps they will venture as far as the third page but if you are outside the top 30 results for a keyword you might as well be invisible.

The big boys in search engines constantly tweak their algorithms to deliver "relevant" results, so in effect they are trying to decide what is relevant to you and me, on the whole they do a good job if you enter Ford then you will most likely find the website of the Ford Motor company.

But here lies the conundrum, the internet is fast becoming one of the most powerful forces in modern life and access to the web is often mentioned in political circles, we've all heard of the information rich and the information poor. This term is normally reserved for describing how the poor are least likely to have access to internet capacity and therefore miss out on a vast library of information which could materially effect their lives.

Is there a deeper problem brewing? just having access to the web is one thing but searching for information is in effect being filtered, if we went to the British library today we would have access to all of the books contained within. The library employs a computer system to aid our search for information but every item is catalogued and importantly those that catalogue the British library have no commercial or personal interests in ensuring you find any individual book.

Search Engines on the other hand are generally commercial profit making organisations, they create revenue by selling advertising contained within the results pages, some differentiate this "paid for" content clearly like Google's sidebar, others just mix it all into the top ranking pages.

This raises key questions, can we have gateway to the worlds greatest library having a possible commercial interest in effecting the results, thus turning them from gateway to gatekeeper? Who watches the search engines? who ensures that commercial pressure does not become the major weighting factor in the algorithms? and do we care if it is after all its a commercial world we all live in.

We are not going to suggest that the current major search engines tweak the results they provide to favour their own interests or commercial partners, however the mere possibility would lead some to call for regulation of search providers. Hardly a day passes without more regulation in the UK but the information superhighway seems to escape tighter scrutiny, it does pose serious legal issues as search technology is commonly located across the world in data centres that would fall under different jurisdictions.

However we are still amazed that no political party in the UK is seen at technologically aware, we can all see the likely direction  of the UK's service industries and how heavily IT reliant they are becoming, yet it is difficult to see the parties policies on this topic or indeed if the have any views.

Can regulation be the answer to ensuring that the information resource of the Web remains fair, balanced and free from overly commercial pressures? in short no, but more could be done to ensure that we the consumers are aware that search results may depend on more than keywords and how popular a website is.

Perhaps the community that gave birth to the search engines are already creating the antidote to commercialism on the internet, through Open Source search engines with clear and transparent search algorithms. Maybe the sheer choice of search engines becomes the webs own natural defence against any one engine or commercial organisation slewing results?

Whatever the outcome be it politically driven or taken up by the online community, you should always be aware that hitting that little button marked search does not necessarily present you with the results you want as you are not the Gatekeeper.