Company Overview:

Is the Internet a good venue for getting online leads and sales? Oftentimes, online merchants are disappointed or disheartened and end up reverting to traditional media for their advertising needs. Creating a website is not enough. Submitting the URL (with appropriate Meta tags and keywords to make the site searchable) is not enough. Having a merchant account on your site is not enough. The best way to get the most traffic to go to your site is through the forging of relationships with other companies. Short of pounding the pavement or writing emails to anonymous companies who may not give you the time of day, the best solution offered by the internet today are — affiliate programs.

Commission Junction is a one of three ValueClick companies. Together they provide complete Internet marketing solutions to its clients and partners. Commission Junction’s expertise lies in forging strategic relationships between publishers and advertisers; thereby creating an open marketplace low risk and high reward for its clients.

Commission Junction is one of the largest affiliate programs on the Internet today with over three hundred advertisers in the UK, France, and Germany alone and over 1,700 merchants and growing daily. Their partners and clients include leading big name Internet and traditional companies such as Air France, Ebay, Microsoft, Palm, Razorfish, Thomas Cook, and Yahoo. Commission Junction ranks as one of the largest ad networks in the world.

Program Details and Benefits:

How does it work? Once upon a time we had a simple term for it. We called it “banner swap”. Another was direct ad placement. Publishers would include ads or links to advertisers in the content of their website (oftentimes in the form of banners). This helps advertisers in building their brand, drive up their sales, and decrease the acquisition cost of customers.

Income and commissions are generated by what we call “click through”s. This means that the publisher makes money every time the ad is seen or clicked by a person. The commission rate is often set in batches, for example, every 1,000 clicks, etc.

There are several reasons why Commission Junction is considered one of the best affiliate programs on the Internet today.

1) Commission Junction has a very competitive pricing scheme for their merchants

2) They pay out commissions to their members quickly and on time

3) They constantly look for ways to improve their services by increasing the benefits of their clients (or affiliates) and encourage them to help in expanding the Commission Junction network.

Commission Junction employs traditional affiliate commission models such as “pay per click”, “pay per lead” and “pay per sale”.

Commission Junction’s affiliate program offers a variety of benefits to its affiliates. The first is the two tier commission plan.

1) An affiliate can earn a referral commission ($2 up-front) by encouraging other webmasters or companies into joining the Commission Junction network (assuming they are not yet members of the Commission Junction network) once they sign up. For example, if ten (10) people click through from your site and all of them sign up. That means that the affiliate automatically earns twenty dollars ($20) in referrals.

2) An affiliate can earn a five percent (5%) commission on all commissions generated by their referrals. This only applies to new accounts, and not existing members of Commission Junction. For example, if all ten become active members and each one generates twenty dollars ($20) in commissions each month, then the first tier affiliate will earn an additional ten dollars ($10) monthly income. This translates into an annual income of around one hundred twenty dollars ($120) — all from referrals!

Second, commissions are pooled together in one check paid directly by Commission Junction, on a monthly basis, not by the merchants. There is no set minimum commission per merchant. This means that if you earn five dollars ($5) each from ten merchants in a single month, Commission Junction will remit fifty dollars ($50) to you immediately. Also, Commission Junction offers a centralized online commission report for all its merchant programs.

Another benefit that makes Commission Junction so attractive to merchants and publishers is that they also offer their merchants the same two tiered program they offer their affiliates. This means that all merchants are also entitled to the five percent (5%) second tier commissions.

Commission Junction also offers a variety of affiliate programs that you can promote on your site. Aside from the big name companies, Commission Junction has attracted the attention of many smaller merchants that are highly targeted and are focused on a particular niche. This helps in targeting your site as well and will increase the opportunities of finding a merchant appropriate for your target audience.

Visitor Feedback:

I’ve read several comments about Commission Junction on the Internet from forums and review sites and there seems to be mixed feedback, depending on whether you are a merchant or an affiliate.

Merchants have given glowing reports all around. According to the merchants, Commission Junction offers them a well-designed and well-implemented solution at a reasonable price. From the affiliates, though, there have been a few glowing reports and quite a few not so desirable ones. Here are some comments posted:

· (August 23, 2001)I could spend all day on their site trying to figure out which programs to choose from the thousands on offer. Don’t make much money from them despite the programs looking good.

· (February 25, 2003) Hardly makes you any money even if you try hard. They used to be a good and reliable bunch, but since they’ve left the pay per click system, they’ve gone downhill in many respects. When I requested payment, next thing I knew was that they terminated my account, accusing me of cheating! This was supposedly detected by their infallible anti-cheating software… I subsequently emailed them several time asking for an explanation, and they never replied once! What can you do! I’d say skip them if you’re looking for a company with integrity. There are many around, like Tradedoubler, Fastclick and Focusin.

· (February 4, 2004 – from a 2-year Commission Junction member) Commission Junction offer a fantastic business opportunity from people working from home (who have a website) wanting to earn an extra income brilliant companies on offer easy to join and add links to – Their payment structure is second to none – I now receive a guaranteed monthly check which is always more every month!

· (April 16, 2004) Commission Junction seemed to be good, but I have just recently requested payment and found my account has been put on payment review for 30 days and then if they think I have increased clicks they will terminate my account. This is very annoying and I don’t think I’m the only one who has had this problem. My advice would be to look elsewhere or deal directly with the companies of which you want to become an affiliate.

· (October 18, 2005) Commission Junction interface is hard to navigate, keeps crashing. Stay well clear.

As you can see the comments are mixed. In the early stages of Commission Junction’s history there were reports of undelivered commissions and inaccurate tracking. On the up side, the last negative comment I’ve read was the one listed above (October 18, 2005). I assume this means that Commission Junction has ironed out their bugs and has definitely moved forward and been able to supply its merchant partners and affiliates with the quality service they promised. With nine months of glowing reviews, I think it is safe to recommend Commission Junction to merchants and publishers all over the net.

High placement in a search engine is critical for the success of any online business. Pages appearing higher in the search engine results to queries relevant to a site’s business will get higher targeted traffic. To get this kind of competitive advantage Internet companies employ various SEO techniques in order to optimize certain factors used by search engines to rank results.

In the best case SEO specialists create relevant well-structured keyword rich pages, which not only please the eyes of a search engine crawler but also have value to the human visitor. Unfortunately it takes months for this strategic approach to produce feasible results, and many search engine optimizers use so-called “black-hat” SEO.

‘Black Hat’ SEO and Search Engine Spam

The oldest and simplest “black SEO” strategy is adding a variety of popular keywords into web pages to make them rank high for popular queries. This behavior is easily detected since generally such pages include unrelated keywords that lack topical focus. With the introduction of the term vector analysis search engine became immune to this sort of manipulation. However “black-hat’ SEO went one step further creating the so-called “doorway’ pages – tightly focused pages consisting of a bunch of keywords relevant to a single topic. In terms of keyword density such pages are able to rank high in search results but never seen by human visitors as they are redirected to the page intended to receive the traffic.

Another trend is the abusing the link popularity based ranking algorithms, such as PageRank with the help of dynamically-generated pages. Such pages receive the minimum guaranteed PageRank and the small endorsements from thousands of these pages are able to produce a sizeable PageRank for the target page. Search engines constantly improve their algorithms trying to minimize the effect of “black-hat”‘ SEO techniques, but SEOs also persistently respond with new more sophisticated and technically advanced tricks so that this process bears a resemblance to an arms race.

“Black-hat” SEO is responsible for the immense amount of search engine spam-pages and links created solely to mislead search engines and boost rankings for client web sites. To weed out the web spam search engines can use statistical methods that allow computing distributions for a variety of page properties. The outlier values in these distributions can be associated with web spam. The ability to identify web spam is extremely valuable to search engine not just because it allows excluding spam pages from their indices but also using them to train more sophisticated machine learning algorithms capable to battle web spam with higher precision.

Using Statistics to Detect Search Engine Spam

An example of an application of statistical methods to detect web spam is presented in the paper “Spam, Damn Spam and Statistics” by Dennis Fetterly, Mark Manasse and Marc Najork from Microsoft. They used two sets of pages downloaded from the Internet. The first set was crawled repeatedly from November 2002 to February 2003 and consisted from 150 million URLs. For each page the researches recorded HTTP status, time of download, document length, number of non-markup words, and a vector indicating the changes in page content between downloads. A sample of this set (751 pages) was inspected manually and 61 spam pages were discovered, or 8.1% of the set with a confidence interval of 1.95% at 95% confidence.

Another set was crawled between July and September 2002 and comprises 429 million pages and 38 million HTTP redirects. For this set the following properties were recorded: URL, URLs of outgoing links; for the HTTP redirects – the source and the target URL. 535 pages were manually inspected and 37 of them were identified as spam (6.9%).

The research concentrates on studying the following properties of web pages: – URL properties, including length and percentage of non-alphabetical characters (dashes, digits, dots etc.). – Host name resolutions. – Linkage properties. – Content properties. – Content evolution properties. – Clustering properties.

URL Properties

Search engine optimizers often use numerous automatically generated pages to massively distribute their low PageRank to a single target page. Since the pages are machine generated we can expect their URLs to look differently from those created by humans. The assumptions are that these URLs are longer and include more non-alphabetical characters such as dashes, slashes or digits. When searching for spam pages we should consider the host component only, not the entire URL down to the page name.

The manual inspection of the 100 longest hostnames had revealed that 80 of them belong to adult site and 11 refer to the financial and credit related sites. Therefore in order to produce a spam identification rule the length property has to be combined with the percentage of non-alphabetical characters. In the given set 0.173% of URLs are at least 45 characters long and contain at least 6 dots, 5 dashes or 10 digits-and the vast majority of these pages appear to be spam. By changing the threshold values we can change the number of pages flagged as spam and the number of false positives.

Host Name Resolutions

One can notice that Google, given a query q, tends to rank a page higher if the host component of the page’s URL contains keywords from q. To utilize this search engine optimizers stuff pages with URLs containing popular keywords and keyphrases and set up DNS servers to resolve these URLs to a single IP. Generally SEOs generate a large number of host names to rank for a wide variety of popular queries.

This behavior can also be relatively easy detected by observing the number of host name resolutions to a single IP. In our set 1,864,807 IP addresses are mapped to only one host name, and 599,632 IPs-to 2 host names. There are also some extreme cases with hundreds of thousands host names mapped to a single IP, and the record-breaking IP referred by 8,967,154 host names.

To flag pages as spam a threshold of 10,000 name resolutions was chosen. About 3.46% of the pages in the Set 2 are served from IP addresses referred by 10,000 and more host names and the manual inspection of this sample proved that with very few exceptions they were spam. Lower threshold (1,000 name resolutions or 7.08% pages in the set) produces an unacceptable amount of false positives.

Linkage Properties

The Web consisting of interlinked pages has a structure of a graph. Therefore in graph terminology the number of outgoing links of a page can be referred to as the out-degree, while the in-degree equals to the number link pointing to a page. By analyzing out- and in-degrees values it is also possible to detect spam pages which would represent the outliers in the corresponding distributions.

In our set for example there are 158,290 pages with out-degree 1301, while according to the overall trend only 1,700 such pages are expected. Overall 0.05% of pages in the Set 2 have out-degrees at least three times more than suggested by the Zipfian distribution, and according to the manual inspection of a cross section, almost all of them are spam.

Similarly the distribution for in-degrees is calculated. For example 369,457 pages have the in-degree of 1001, while according to the trend only 2,000 such pages are expected. Overall, 0.19% of pages in the Set 2 have in-degrees at least three times more common than the Zipfian distribution would suggest, and the majority of them are spam.

Content Properties

Despite the recent measures taken by search engines to diminish the effect of keyword stuffing, this technique is still used by some SEOs who generate pages filled with meaningless keywords to promote their AdSense pages. Quite often such pages are based on a single template and even have the same number of words which makes them especially easy to detect using statistical methods.

For Set 1 the number of non-markup words in each page was recorded, so we can draw the variance of word count in pages downloaded from a given host name. The variance is plotted on the x-axis and the word count is shown on the y-axis, both axes are drawn on a logarithmic scale. Points in the left side of the graph marked with blue represent cases where at list 10 pages from a given host have the same word count. There are 944 such hosts (0.21% of the pages in Set 1). A random sample of 200 these pages was examined manually: 35% were spam, 3.5% contained no text and 41.5% were soft errors (a page with a message indicating that the resource is not currently available, despite the HTTP status code 200 “OK”).

Content Evolution

The natural evolution of the content in the Web is slow. In a period of a week 65% of all pages will not change at all, while only 0.8% will change completely. In contrast many spam SEO web pages generated in response to an HTTP request independent of the requested URL will change completely of every download. Therefore by looking into extreme cases of content mutation we search engines are able to detect web spam.

The outliers represent IPs serving the pages that change completely every week. Set 1 contains 367 such servers with 1,409,353 pages (97.2%). The manual examination of a sample of 106 pages showed that 103 (97.2%) were spam, 2 were soft errors and 1 adult pages counted as a false positive.

Clustering Properties

Automatically generated spam pages tend to look very similar. In fact, as already said above, most of them are based on the same model and have only minor differences (like inserting varying keywords into a template). Pages with such properties can be detected by applying clustering analysis to our samples.

To form clusters of similar pages the ’shingling’ algorithm described by Broder et al. [2] will be used. Figure 7 shows the distribution of the cluster sizes on near duplicate pages in Set 1. The horizontal axis shows the size of the cluster (the number of pages in the near-equivalence class), and the vertical axis shows how many such clusters Set 1 contains.

The outliers can be put into two groups. The first group did not contain any spam pages, pages in this group are more related to the duplicated content issue. In the same time the second group is populated predominantly by spam documents. 15 of 20 largest clusters were spam containing 2,080,112 pages (1.38% of all pages in Set 1)

To Sum Up

The methods described above are the examples of a fairly simple statistical approach to spam detection. The real life algorithms are much more sophisticated and are based on machine learning technologies which allow search engine to detect and battle spam with a relatively high efficiency at an acceptable rate of false positives. Applying the spam detection techniques enables search engine to produce more relevant results and ensures a more fair competition based on the quality of web resources and not on technical tricks.

References:

1. Dennis Fetterly, Mark Manasse, Marc Najork. “Spam, Damn Spam, and Statistics: Using statistical analysis to locate spam web pages” (2004). Microsoft Research.

2. A. Broder, S. Glassman, M. Manasse, and G. Zweig. “Syntactic Clustering of the Web”. In 6th International World Wide Web Conference, April 1997.

1.0 Introduction

This 21st century has become an age of recycling where a lots of emphasize is placed on reuse of material to curb current environmental problems and maximize use of depleting natural resources and energy conservation. Modern day sustainable use and management of resource recommend need to incorporate recycling culture in our ways of life including technological process. Biomass is not left behind in this; the use of biomass energy resource derived from the carbonaceous waste of various natural and human activities to produce electricity is becoming popular. Biomass is considered as one of the clean, more- efficient and more-stable means of power generation. And it has become imperative for marine industry to tap this new evolving power generation mode especially the use of micro generation approach considering the mobile nature of ships.

 

Biofuels exist in solid, liquid or gas form thereby potentially affecting three of our core markets. Solid biofuels or biomass tend to be used in external combustion, however its use in the shipping industry has been limited to liquid biofuel due to lack of appropriate information economics forecasts, Sources of biomass include by-products from the timber industry, agricultural crops, raw material from the forest, major parts of household waste, and demolition wood, all things being equal using pure biomass that do not affect human and ecological chain make it suitable energy source. Biomass has low sulfur content means biomass combustion therefore considered much less acidifying than with coal, for example. Also, the ashes from biomass consumption, which are very low in heavy metals, can be recycled.

One advantage of biomass compared to other renewable-based systems that require costly advanced technology (such as solar photovoltaics) is that biomass can generate electricity with the same type of equipment and power plants that now burn fossil fuels. Many innovations in power generation with other fossil fuels may also be adaptable to the use of biomass fuels. Various factors have hindered the growth of the renewable energy resource, however. Most biomass power plants operating today are characterized by low boiler and thermal-plant efficiencies; both the fuel’s characteristics and the small size of most facilities contribute to these efficiencies. In addition, such plants are costly to build.

Biomass remains potential renewable energy contributor to net reduction in greenhouse gas emissions by offsetting CO2 from fossil generation. The current method generating biomass power is biomass fired boilers and Rankine steam turbines. Recent research work in developing sustainable, and economic biomass focus on high-pressure supercritical steam cycles , use of feedstock supply system, and conversion of biomass to a low or medium gas that can be fired in combustion turbine cycles, resulting in efficiencies one-and-a-half times that of a simple steam turbine. biofuels has potential to influence marine industry, and it as become importance for designers and ship owners to accept their influence on the world fleet of the future especially the micro generation concept with co generation for cargo and fuel for  ships.

 

The paper discuss conceptual work, trend , sociopolitical driver, economic, development, and future of biomass with hope to bring awareness to local, national and multinational bodies making biofuels policies as well as maritime multidisciplinary expertise in regulation, economics, engineering, and vessel design and operation. The paper also discusses how the shipping industry can take advantage of growing tide to tap benefit promised by waste use power generation system.

 

 

2.0 Biomass developmental trend

 

The concept of use of Biofuels for energy generation has has been existing concept, and in the face of challenges posed by environmental need, its growth is likely to dominate renewable energy market. Following the advent of peanut oil diesel engines developed by Rudolf Diesel in 1911 the production and use of biofuels worldwide has grown significantly in recent years. The current world biofuels market is focused on: Bioethanol blended into fossil motor gasoline (petrol) or used directly and biodiesel or Fatty Acid Methyl Ester diesel blended into fossil diesel. However the use of The Fischer-Tropsch model that involve catalyzed chemical reaction to produce a synthetic petroleum substitute, typically from coal, natural gas or biomass, for use as synthetic lubrication oil or as a synthetic fuel seem promising and negate risk posed by food based biomass. This synthetic fuel runs only in diesel engines and some aircraft engines. Oil, product and chemical tankers being constructed now are likely to benefit much more from use of biomas. However use on gasoline engines ignites the vapors at much higher temperatures, which pose limitation to inland water craft.

 

Biomass generation and growing trend can be classified into 3 generation types:

first generation’ biofuels relate to biofuels made from sugar or starch, producing bioethanol, and vegetable oil or animal fats producing biodiesel. First generation biofuels provoke increasing criticism through their dependence on food crops and issues over biodiversity, land use and human rights. Hybrid technology for percentage blending is being employed to mitigate food production impact. Second generation biofuels mitigate problem posed by the first generation biofuels. They do not affect food crops because they are made from waste biomass from agricultural and forestry, fast-growing grasses and trees specially grown as so-called “energy crops”. With technology, sustainability and cost issues to overcome, second-generation biofuels are still several years away from commercial viability and many second generation mass produced biofuels are still under development including the biomass to liquid. Fischer-Tropsch production technique. third generation biofuels are green fuels like algae biofuel made from energy and biomass crops that have been designed in such a way that their structure or properties conform to the requirements of a particular bioconversion process. They are made from such as sewage, and grown on ponds.

 

Just like tanker revolution influence on ship type, demand for biomass will bring, will bring capacity, bio -material or completed product from source to production area and then to the point of use, will bring technological, environmental change will require ships of different configuration, size and tank coating type. As well as impact on the tonne mile demand will change accordingly.

 

Effect on shipping is likely to follow shipping large scale growth on exports and seaborne trade from key exporting regions, particularly South America. Brazil has a key role. Brazil has already been branded to be producing en-mass ethanol from sugar cane since the 1970s with a cost per unit reportedly the lowest in the world. And it is currently exploring ethanol

 

Table 1 – World ethanol consumption 2007

Consumption

 

World ethanol consumption -

51 million tones, 2007

Us and brazil

68%

EU and China –

17% – surplus of 0.1 million tones

US deficit –

1.7mt

EU deficit -

1.3 mt

World – deficit

1mt

 

Recent year is also witnessing  emerging trade on biofuel product between the US, EU, and Asia and whilst Brazil exports the most ethanol globally at about 2.9 million tonnes per year, the top importers of the US, EU,Japan and Korea have increasing demand that will have to be satisfied by increased shipping capacity. Seaborne vegetable oil supply is increasingly growing

 

 

Table 2 – Biofuel growth

 

 

 

Vegetable oil

33 mt in 2000 to 59 mt in 2008

 

Palm oil

13 mt in 2000 to 32 mt forecast in 2008.

 

a 7.5% p.a growth rate

Soya bean

7 mt to some 11.5 mt in 2008,

 

EU

imports – 5.7 mt in 2001 to an expected 10.3 mt for 2008

8.9%.

 

3.1 mt in 2001 to 5.2 mt forecast for 2008

39%

 

Production capacity- 1.9 mt in 2002 to 11 mt in 2007, with 2007.

 

50% of total capacity.

 

 

Recently biofuel is driving a new technology, Worldwide; the use of biofuels for cars and public vehicles has grown significantly. With excess capacity waiting for source material it seems inevitable that shipping demand will increase.

 

3.0 Inter industry Best Practice

 

3.1 Land based use - 

 

UK pumps mandate at least 2.5% biofuels. This target will rise to 5% by 2010. Also in the UK, the first train to run on biodiesel went into service in June 2007 for a six month trial period. The train uses a blended fuel, which is 20% biodiesel and the operator, Virgin Trains, is confident the mix can be increased to at least a 50% mix with the further possibility to run trains on fuels entirely from non-carbon sources. On January 15, 2006- Central Ohio Transit Authority (COTA lunch a program to test a 20% blend of biodiesel (B20) in its buses. In two months they used approximately 45,000 gallons of B20. As a result of the test, in April 2006 they began using biodiesel fleet-wide. In addition to using B20 in the winter months, COTA has committed to using 50-90% biodiesel blends (B50 – B90) during the summer months. This is projected to decrease regular diesel fuel consumption by over one million gallons per year. 26th of October 2007. buses in the UK running on B100 was launched on  In a pilot project. Argent Energy (UK) Limited is working together with Stagecoach to supply biodiesel made by recycling and processing animal fat and used cooking oil. For power stations, B&W have orders in the EU for 45 MW of two-stroke biofuel engines with a thermal efficiency of 51-52%. Specifically, these operate on palm oil of varying quality, and in the future, it is expected that more engines, whether stationary or marine, will be developed to run on biofuels.

·         US DOE has funded five new advanced biomass gasification research and development projects beginning in 2001(Vermont project)

·         2008 – Ford announced a £1 billion research project to convert more of its vehicles to new biofuel sources. The first trial oft, Last year. BP Australia has now sold over 100 million liters of 10% ethanol content fuel to Australian motorists, and Brazil sells both 22% ethanol petrol nationwide and 100% ethanol to over 4 million cars, It is a trend that is gathering momentum.

In a program initiated by the Swedish National Board for Industrial and Technical Development in Stockholm, several Swedish universities, companies, and utilities are collaborating to accelerate the demonstration of the advanced EVGT for natural-gas firing, especially in small-scale units. A natural-gas-fired EVGT pilot plant (0.6 megawatts of power output for a simple gas-turbine cycle) should start operation in Lund, Sweden, in 1998.

·         AES Corporation is a leading company in biomass conversion internationally. At AES Kilroot in Northern Ireland, the team recently completed a successful trial to convert the plant to burn a mixture of coal and biomass. With further investment in the technology, nearly half of Northern Ireland’s 2012 renewable target could be met from AES Kilroot alone.

3.2 Aero industry–

 

Virgin Atlantic – Air transport is receiving increasing attention because of environmental concerns linked to CO2 emissions, air quality and noise. Virgin Atlantic in collaboration with Boeing and General Electric aircraft alternative fuels project for aircraft. A successful test flight from London to Amsterdam flight took place on 24th February of this year, running one of the four jumbo jet engines on a mixture of 20% coconut oil and babassu nut oil, with 80% conventional jet fuel. This fuel was specifically chosen due to its performance at low operating temperatures. The test was successful, with no noticeable difference in performance. Except that; imitation that biofuel mix used was in no way sustainable in the quantities required by the demands of the aviation industry. In a way to mitigate this Virgin is looking to us use of Algae based fuels as it is predicted that they may be suitable for use at low temperature.

 

3.3  Maritime industry 

 

The use of land based transportation, is growing, however the use for sea based transportation need to be explored. Biofuels  for ship will be advantageous. In recent UK pilot project where Buses are run on B100 Argent Energy (UK) Limited is working together with Stagecoach to supply biodiesel made by recycling and processing animal fat and used cooking oil. Marine engines with their inherent lower speed and more tolerant to burning alternative fuels than smaller, higher speed engines tolerance will allow them to run on lower grade and cheaper biofuels. Royal Caribbean Cruise Lines (RCCL) unveiled a palm oil-based biodiesel since 2005.Optimistic outcome of the trial made RCCL confident enough to sign a contract in August 2007 for delivery of a minimum 15 million gallons and for the four years after, a minimum of 18 millions gallons of biodiesel for its cruise ships fleet. The contract marked the single largest long-term biodiesel sales contract in the United States. In early 2007, United States Coast Guard indicated that their fleet will augment increase use of biofuels by 15% over the next four years. In the marine industry, beside energy substitute advantage, biolubricants and biodegradable oil  are particularly advantageous from an environmental and pollution perspective. Bio lubrication also offer higher viscosity, flash point and better technical properties such as increased sealing and lower machine operating temperature advantageous use in ship operation.

 

Time has gone when maritime industry could afford nitty gritty in adopting technology, other industry are already on a fast track preparing themselves technically for evitable changes driven by environmental problem, Global energy demands and political debate add further pressures to find alternative energy especially bio energy  because of hybridization of old and new system advantage it offer. The implication is that shipping could be caught ill prepared for any rapid change in demand or supply of biofuel. Thus this technology is in the early stages of development but the shipping industry need top be prepared for the impacts of its breakthrough because Shipping will eventually required be at the centre of this supply and demand logistics chain again. Table 3 shows the projection for the main present players.

 

Table3  – projection

 

Region

Growth (1990-1994)

Projection (2020)

United states

7%

15%

Europe

2%

15%

 

4.0 Sources of biomass

North American Electric Reliability Council (NERC) region. Supply has classified biofuel into the following four type’s vizs: agricultural residues, energy crops, forestry residues, and urban wood waste/mill residues. A brief description of each type of biomass is provided below:

Agricultural residues from the remaining stalks and biomass material left on the ground can be collected and used for energy generation purposes this include residues of wheat straw and corn stover. Energy crops are produced solely or primarily for use as feedstocks in energy generation processes. Energy crops includes hybrid poplar, and switchgrass, grown on idled, or in pasture, and in the Conservation Reserve Program (CRP). Forestry residues are composed of logging residues, rough rotten salvageable dead wood, and excess small pole trees. Urban wood waste/mill residues are waste woods from manufacturing operations that would otherwise be landfilled. The urban wood waste/mill residue category includes primary mill residues and urban wood such as pallets, construction waste, and demolition debris, which are not otherwise used.

The most important agricultural commodity crops being planted in the United States are listed in Table 4. Corn, wheat, and soybeans represent about 70 percent of total cropland harvested.

 

 

Table 6 shows representative characteristics for different subcategories of urban wood waste and mill residues.

 

5.0 Risk and Uncertainties

Although a significant amount of effort has gone into estimating the available quantities of biomass supply, the following risk and uncertainties that need to be incorporated into design and decision work on biodiesel use are:

Risk to land use – Our planet only have 295 land, for example Brazil has some 200 million acres of farmland available, more than the 46 million acres of land,  required to grow the sugarcane needed to satisfy the projected 2022 Evolving competing uses of biomass materials, the large market consumption, pricing and growing need. In agricultural waste, the impact of biomass removal on soil quality pose treat to agricultural residues that need to be left on the soil to maintain soil quality could result in significant losses of biomass for electric power generation purposes. Impact of changes in forest fire prevention policies on biomass availability could cause vegetation in forests to minimize the potential for forest fires could significantly increase the quantity of forestry residues available. Potential attempt to recycle more of the municipal solid waste stream might translate into less available biomass for electricity generation. \ Impact on the food production industry as witness in recent food scarcity crisis

5.1 Regulatory impact

 

The EU has stated that by 2020 a target of 20% of community wide energy will be renewable. Further to this, all member states are to achieve a mandatory 10% minimum target for the share of biofuels in transport petrol and diesel consumption by 2020.. The legislation provides a phase-in for biofuel blends, including availability of high percentage biofuel blends at filling stations.  The United States Congress passed the Renewable Fuels Standards (RFS) in February 2008, which will require 35 billion gallons of renewable and alternative fuels in 2022. In parallel to this, work is continuing to reduce emissions further in vehicles. Political drivers in Asia vary according to region. In Southeast Asia, the centre of world production for palm oil, coconut oil, and other tropical oils, political support for farming is the key driver.

 

The issue affecting shipping is whether to refine and use biodiesel locally, or export the unrefined oil for product production elsewhere. In the short term the economics have favored the exports of unrefined oil – which is good news for us. Over the next ten years, with the cost of oil rising, and strict emission reductions in place, the need for increased biofuel production is likely to increase. as well as creating a net positive balance fuel. According to the IEA, world biofuels demand for transport could increase to about 3% of overall world oil demand in 2015 and double by 2030 over the 2008 figure. This does not sound so significant but as we show later it has a significant impact on the specialist fleet capacity demand. As we said before, predicting the trade pattern of biofuels adds a layer of complexity to the overall  nergy supply picture and our oil distribution system.

 

We also believe that this forecast will be the minimum seen as the political pressures will cause the level to rise beyond 3%. To put the scale in context, the current oil tanker fleet of vessels 10,000 dwt or larger comprises of some 4,600 vessels amounting to 386 million dwt. These include about 2,560 Handysize tankers. Additionally, there are some 4,400 more small tankers from 1,000 to 10,000 dwt accounting for 16 million dwt. Our projections show a significant role for seaborne transport, even using conservative bases with high proportions of locally supplied biofuels. This is a significant fleet segment that poses technical and regulatory challenges. As we have discussed, the requirements cannot be fully defined because many market factors remain uncertain, but ship owners who are building new vessels or operating existing vessels should consider this future trade through flexible design options that we will introduce later.

 

 

5.3  Potential Impacts to Shipping

 

The key political drivers for biofuels are environmental concerns, energy security and agricultural policy. The tonne mile demand for future tankers will be greatly affected by national, regional or global policy and political decision making in these areas. There is a greater flexibility in the sourcing of biofuels than there is in hydrocarbon energy sources and this may be attractive to particular governments. Once the regulatory framework is clear, economics will determine how the regulations will best be met and seaborne trade will be at the centre of the outcome. In many parts of the world, environmental concerns are the leading political driver for biofuels. Reflecting these concerns, the global Kyoto Protocol, was negotiated in 1997, and this further provides a driver for the use of biofuels.

 

 

 

 

5.4  Shipping Routes and Economics Impacts

 

The above trend analysis discussed indicate potential capacity requirement from shipping, so far  North America, Europe and South East  Asia are the key importing regions where this growth is concentrated. This includes the Latin American counties of Brazil, Argentina, Bolivia, and Paraguay and Southeast Asia’s Indonesia and Malaysia will remain key suppliers for the palm oil, Philippines and Papua New Guinea have potentials for vegetable oil and agricultural while Thailand has potential for sugarcane. This trade potential will determine future trade route from Malacca Straits to Europe, ballast to Argentina, to load soybean oil to China, and then make a short ballast voyage to the Malacca Straits, where the pattern begins again, a typical complicated fronthaul / backhaul combinations that can initiate, economies of scale need top reduce freight costs and subsequent push for bigger ship production and short sea services like recent experience of today’s tankers.  According to plateau case study the following regional impact can be deduced for shipping.

 

 

 

Biofuel

Demand

North America

ethanol

33 million tons

Europe

ethanol and biodiesel.: 50:50

30 million tons

Asia

ethanol and biodiesel.: 50:50

18 million tons

 

North America demand – policy work support biofuel use in the us and 32 Handysize equivalent tankers will be needed to meet US demand in 2015. with technological breakthrough there will be need for 125 vessel 2030.

 

European demand – Due to environmental requirement and energy security believed to be politically acceptable in the EU but economics may drive a different outcome.80 Handysizes with some due to the growth in trade and longer voyage distance.  With technological breakthrough for 2nd and 3rd generation biofuel growth will need growing to 145 in 2030 Aframax vessels if the technical issues can be overcome.

 

Asia demand  - In plateau case  50 Handysize equivalents are required in 2015 and 2030 with forecast vessel sizes being Handysizes with some Panamax vessels 162 vessels total in the three regions.

 

By adding up all the regions, with biofuels as only 3% of world transport demand, we are looking at a fleet of about 400 Handysize vessels to accommodate the demand and supply drivers by 2030 and 162 by 2015. The total vessel forecast for 2030 could means 2,560 vessels of 81 million deadweight tons.

 

As regions identify these growth markets and recognize the economies of $/tonne scale that can be achieved, as shown here, with bigger tonnage, we are seeing natural investment occurring. New port developments in concerned trade rout will be required to accommodate large Panamax vessel and parcel size for palm oil exports. on the long haul routes.

 

5.5  Biomass  Ship Technologies Impacts

Generation

A variety of methods could turn an age-old natural resource into a new and efficient means of generating electricity. biomass in large amounts is available in many areas, and is being considered as a fuel source for future generation of electricity. Biomass is by its nature both bulky and widely distributed and electricity from conventional, centralized power plants requires an extensive distribution network. Traditionally power is generated through centralized, conventional power plant, where biomass is transported to the central plant, typically a steam or gas turbine power plant, and the electricity is then distributed through the grid to the end users. Costs include fuel and transportation, power plant construction, maintenance, and operation, and distribution of the electric power, including losses in transmission.

 

 

Electrical efficiency

capacity

 biomass

thermal efficiency -40 %

$2,000 per kilowat

 

coal

45 %

$1,500 per kilowatt,

 

However, micro-biomass power generators located at the site of end-use seem to offer a path for new solution for energy. Recent development in towards use of micro biomass will equally offer best practice adaptation for marine power. Biomass is used at or near the site of end-use, with heat from external combustion converted directly to electricity by a biomass fired free-piston genset . Costs include fuel and acquisition and maintenance of the genset and burner. Since the electricity is used on site, both transmission losses and distribution costs are minimal. Thus, in areas without existing infrastructure to transmit power, there are no additional costs. In this case it is also possible to cogenerate using the rejected heat for space or hot water heating, or absorption cooling. Previously, option two has not been feasible, since there have been no small (less than ~50 kW) devices for directly and efficiently converting biomass energy to electricity. Micro-biomass power generation is a more cost-effective means of providing power than central biomass power generation. In particular, areas where there is a need for both power and heat – domestic hot water and space heat and absorption chilling – are attractive for cogeneration configurations of this machine. Biomass can be generated using single or ganged free-piston Stirling engines gensets. These micro-biomass generators offer a number of advantages over centralized biomass fueled power plants. They can be placed at the end-user location taking advantage of local fuel prices and do not require a distribution grid. They can directly provide electrical output with integral linear alternators, or where power requirements are larger they can be ganged and drive a conventional rotary turbine. They are hermetically sealed and offer long lives through their non-contact operation.

Biomass for electricity generation is treated in four ways in NEMS: (1) new dedicated biomass or biomass gasification, (2) existing and new plants that co-fire biomass with coal, (3) existing plants that combust biomass directly in an open-loop process,18 and (4) biomass use in industrial cogeneration applications. Existing biomass plants are accounted for using information such as on-line years, efficiencies, heat rates, and retirement dates, obtained through EIA surveys of the electricity generation sector.

Emissions offsets and waste reduction could help enhance the appeal of biomass to utilities  An important consideration for the future use of biomass-fired power plants is the treatment of biomass flue gases. Biomass-combustion flue gases have high moisture content. When the flue gas is cooled to a temperature below the dew point, water vapor starts to condense. By using flue-gas condensation, sensible and latent heat can be recovered for district heating or other heat-consuming processes; this increases the heat generation from a cogeneration plant by more than 30 percent.  Flue-gas condensation not only recovers heat but also captures dust and hazardous pollutants from flue gases at the same time. Most dioxins, chlorine, mercury, and dust are removed, and sulfur oxides are separated out to some extent. Another feature of flue gas condensation is water recovery, which helps solve the problem of water consumption in evaporative gas turbines.

 

Biomass open door for another way rather than competing with fossil fuel plants a substantial opportunity exists to generate micro-biomass electric power, at power levels from fractions of a kilowatts through to tens or hundreds of kilowatts, at the point of en d use. At these power levels neither small internal combustion engines, which cannot use biomass directly, nor reciprocating steam engines, with low efficiency and limited life, can offer the end user economic electric power. Free-piston Stirling micro biomass engine engines are an economic alternative. Stirling offers the following advantages over significantly larger systems:

Stirling machines have reasonable overall efficiencies at moderate heater head temperatures (~600ƒC) cogeneration is simple large amounts of capital do not have to be raised to build a single evaluation plant with its associated technical and economic risks A large fraction of the value of the engine alternator can be reused at the end of its life Stirling systems can be ganged with multiple units operating in parallel.

 

United States: 1996, P1-R96-STAB-00-NTH (Washington, DC, November 1996). l.

Search engine optimization this and search engine optimization that. You read and hear about it all day, but what about your site? While there are plenty of articles providing useful information, this article shows you how a real world example met with success. The point of this article is to emphasize creativity when approaching tough optimization situations.

Problems for BusinessTaxRecovery.com

In November of 2004, our firm took on the seo marketing for BusinessTaxRecovery.com. The site was being promoted through offline activities and pay-per-click campaigns. No effort had been made to achieve high rankings in Google, Yahoo or MSN.

Keyword analysis revealed that combinations of the root keywords, “business” and “tax” were going to be difficult to attack. The primary problem concerned government agencies with web sites. The IRS site, for instance, had roughly 9,680 inbound links and an absolute ton of content. State agencies weren’t far behind. The California tax agency site had roughly 7,000 inbound links and, again, tons of content.

For a final nail in the coffin, the client informed us the business was cyclical with the busiest months being January through April when people focused on taxes. The site absolutely had to rank highly during this period. We had two months to achieve results.

Gulp!

The Solution for BusinessTaxRecovery.com – 140,000 Hits

After staring at a Salvador Dali painting for a few hours, we came up with a solution. It involved a combination of internal site page focus, meta tag optimization, link exchange and massive article promotion. The results produced 145,828 hits from January through April, with only 5,000 coming from the pay-per-click program.

The first step was to change the focus of the site from the home page to the article page. Jumping the tax agency sites on keywords such as business tax and taxes was impossible in two months, so we didn’t even try. Instead, we decided to focus on the keyword phrase “business tax articles” and bring people into the site through the article page. Meta tags were optimized and a link exchange program undertaken. The key to campaign, however, was a strong article promotion campaign.

Since taxes are confusing, it seemed obvious that an article campaign focusing on tax information would meet with success. Boy, did it. Approximately 35 articles were written, published and submitted to article directories. Since the articles were timely, they were snapped up and published. The articles produced direct traffic to the site as well as numerous inbound links because of the link created in the article byline.

As for the search engines, we focused on everything but Google. We expected nothing from Google because the major content and meta tag changes would take six to eight months to show results per the usual practices of Google. In reality, it didn’t matter. The Yahoo and MSN search engines produced big time.

In mid-January, the site went to number 1 on MSN under “business tax articles.” By the end of January, Yahoo was also listing it as number 1. MSN started listing it at number 1 for “tax articles” in February. The combination of these listings produced a significant amount of traffic, conversions and a very happy client.

Can we go on cruise control now? No! With the end of the tax season, the traffic to the articles page of the site has dropped by 75%. Nobody is looking for tax information after April 15th, so this is hardly surprising. The promotion of the article page was simply a short-term solution to a difficult situation. While it should produce traffic during the first quarter of each subsequent year, it is not a year-around solution.

Over the next six months, we will focus on the long-term goal of jumping over the tax agency sites for keywords such as “business taxes”, etc. It is going to take a lot of patience, but will eventually produce a significant amount of business for the client.

Creativity is often the key to conquering seo situations. Blindly slapping up new meta tags and links isn’t always the best answer. Sometimes, a little pre-emptive consideration can yield amazing results. It did in this case.

Ok, so over the past month or so I’ve been collecting various search engine optimization questions from all of you. Today, I’m going to answer what was the most frequently asked question over the past month.

You guessed it… What is the Google Sandbox Theory and how do I escape it? When you finish reading this lesson, you’ll be an expert on the good ‘ole Google Sandbox Theory and you’ll know how to combat its effects. So, pay close attention. This is some very important stuff.

Before I start explaining what the Google Sandbox theory is, let me make a few things clear:

The Google Sandbox theory is just that, a theory, and is without official confirmations from Google or the benefit of years of observation.

The Google Sandbox theory has been floating around since summer 2004, and has only really gained steam after February 4, 2005, after a major Google index update (something known as the old Google dance).

Without being able to verify the existence of a Sandbox, much less its features, it becomes very hard to devise strategies to combat its effects.

Almost everything that you will read on the Internet on the Google Sandbox theory is conjecture, pieced together from individual experiences and not from a wide scale objective controlled experiment with hundreds of websites (something that would obviously help in determining the nature of the Sandbox, but is inherently impractical given the demand on resources).

Thus, as I’ll be discussing towards the end, it’s important that you focus on ·good’ search engine optimization techniques and not place too much emphasis on quick ·get-out-of jail’ schemes which are, after all, only going to last until the next big Google update.

What is the Google Sandbox Theory?

There are several theories that attempt explain the Google Sandbox effect. Essentially, the problem is simple. Webmasters around the world began to notice that their new websites, optimized and chock full of inbound links, were not ranking well for their selected keywords.

In fact, the most common scenario to be reported was that after being listed in the SERPS (search engine results pages) for a couple of weeks, pages were either dropped from the index or ranked extremely low for their most important keywords.

This pattern was tracked down to websites that were created (by created I mean that their domain name was purchased and the website was registered) around March 2004. All websites created around or after March 2004 were said to be suffering from the Sandbox effect.

Some outliers escaped it completely, but webmasters on a broad scale had to deal with their websites ranking poorly even for terms for which they had optimized their websites to death.

Conspiracy theories grew exponentially after the February 2005 update, codenamed ·Allegra’ (how these updates are named I have no clue), when webmasters began seeing vastly fluctuating results and fortunes. Well-ranked websites were losing their high SERPS positions, while previously low-ranking websites had gained ground to rank near the top for their keywords.

This was a major update to Google’s search engine algorithm, but what was interesting was the apparent ·exodus’ of websites from the Google Sandbox. This event gave the strongest evidence yet of the existence of a Google Sandbox, and allowed SEO experts to better understand what the Sandbox effect was about.

Possible explanations for the Google Sandbox Effect

A common explanation offered for the Google Sandbox effect is the ·Time Delay’ factor. Essentially, this theory suggests that Google releases websites from the Sandbox after a set period of time. Since many webmasters started feeling the effects of the Sandbox around March-April 2004 and a lot of those websites were ·released’ in the ·Allegra’ update, this ·website aging’ theory has gained a lot of ground.

However, I don’t find much truth in the ·Time Delay’ factor because by itself, it’s just an artificially imposed penalty on websites and does not improve relevancy (the Holy Grail for search engines). Since Google is the de facto leader of the search engine industry and is continuously making strides to improve relevancy in search results, tactics such as this do not fit in with what we know about Google.

Contrasting evidence from many websites has shown that some websites created before March 2004 were still not released from the Google Sandbox, whereas some websites created as late as July 2004 managed to escape the Google Sandbox effect during the ·Allegra’ update. Along with shattering the ·Time Delay’ theory, this also raises some interesting questions. This evidence has led some webmasters to suggest a ·link threshold’ theory; once a website has accumulated a certain amount of quantity/quality inbound links, it is released from the Sandbox.

While this might be closer to the truth, this cannot be all there is to it. There has been evidence of websites who have escaped the Google Sandbox effect without massive link building campaigns. In my opinion, link-popularity is definitely a factor in determining when a website is released from the Sandbox but there is one more caveat attached to it.

This concept is known as ·link-aging’. Basically, this theory states that websites are released from the Sandbox based on the ·age’ of their inbound links. While we only have limited data to analyze, this seems to be the most likely explanation for the Google Sandbox effect.

The link-ageing concept is something that confuses people, who usually consider that it is the website that has to age. While conceptually, a link to a website can only be as old as the website itself, yet if you have don’t have enough inbound links after one year, common experience has it that you will not be able to escape from the Google Sandbox. A quick hop around popular SEO forums (you do visit SEO forums, don’t you?) will lead you to hundreds of threads discussing various results · some websites were launched in July 2004 and escaped by December 2004. Others were stuck in the Sandbox even after the ·Allegra’ update.

How to find out if your website is sandboxed

Finding out if your website is ·Sandboxed’ is quite simple. If your website does not appear in any SERPS for your target list of keywords, or if your results are highly depressing (ranked somewhere on the 40 the page) even if you have lots of inbound links and almost perfect on-page optimization, then your website has been Sandboxed.

Issues such as the Google Sandbox theory tend to distract webmasters from the core ·good’ SEO practices and inadvertently push them towards black-hat or quick-fix techniques to exploit the search engine’s weaknesses. The problem with this approach is its short-sightedness. To explain what I’m talking about, let’s take a small detour and discuss search engine theory.

Understanding search engines

If you’re looking to do some SEO, it would help if you tried to understand what search engines are trying to do. Search engines want to present the most relevant information to their users. There are two problems in this · the inaccurate search terms that people use and the information glut that is the Internet. To counteract, search engines have developed increasingly complex algorithms to deduce relevancy of content for different search terms.

How does this help us?

Well, as long as you keep producing highly-targeted, quality content that is relevant to the subject of your website (and acquire natural inbound links from related websites), you will stand a good chance for ranking high in SERPS. It sounds ridiculously simple, and in this case, it is. As search engine algorithms evolve, they will continue to do their jobs better, thus becoming better at filtering out trash and presenting the most relevant content to their users.

While each search engine will have different methods of determining search engine placement (Google values inbound links quite a lot, while Yahoo has recently placed additional value on Title tags and domain names), in the end all search engines aim to achieve the same goal, and by aiming to fulfill that goal you will always be able to ensure that your website can achieve a good ranking.

Escaping the sandbox…

Now, from our discussion about the Sandbox theory above, you know that at best, the Google Sandbox is a filter on the search engine’s algorithm that has a dampening influence on websites. While most SEO experts will tell you that this effect decreases after a certain period of time, they mistakenly accord it to website aging, or basically, when the website is first spidery by Googlebot. Actually, the Sandbox does ·holds back’ new websites but more importantly, the effects reduce over time not on the basis of website aging, but on link aging.

This means that the time that you spend in the Google Sandbox is directly linked to when you start acquiring quality links for your website. Thus, if you do nothing, your website may not be released from the Google Sandbox.

However, if you keep your head down and keep up with a low-intensity, long-term link building plan and keep adding inbound links to your website, you will be released from the Google Sandbox after an indeterminate period of time (but within a year, probably six months). In other words, the filter will stop having such a massive effect on your website. As the ·Allegra’ update showed, websites that were constantly being optimized during the time that they were in the Sandbox began to rank quite high for targeted keywords after

the Sandbox effect ended.

This and other observations of the Sandbox phenomenon · combined with an understanding of search engine philosophy · have lead me to pinpoint the following strategies for minimizing your website’s ·Sandboxed’ time.

SEO strategies to minimize your website’s “sandboxed” time

Despite what some SEO experts might tell you, you don’t need do anything different to escape from the Google Sandbox. In fact, if you follow the ·white hat’ rules of search engine optimization and work on the principles I’ve mentioned many times in this course, you’ll not only minimize your website’s Sandboxed time but you will also ensure that your website ranks in the top 10 for your target keywords. Here’s a list of SEO strategies you should make sure you use when starting out a new website:

Start promoting your website the moment you create your website, not when your

website is ·ready’. Don’t make the mistake of waiting for your website to be ·perfect’.

The motto is to get your product out on the market, as quickly as possible, and then

worry about improving it. Otherwise, how will you ever start to make money?

Establish a low-intensity, long-term link building plan and follow it religiously. For

example, you can set yourself a target of acquiring 20 links per week, or maybe

even a target of contacting 10 link partners a day (of course, with SEO Elite, link

building is a snap). This will ensure that as you build your website, you also start

acquiring inbound links and those links will age properly · so that by the time your

website exits the Sandbox you would have both a high quantity of inbound links

and a thriving website.

Avoid black-hat techniques such as keyword stuffing or ·cloaking’. Google’s search

algorithm evolves almost daily, and penalties for breaking the rules may keep you

stuck in the Sandbox longer than usual.

Save your time by remembering the 20/80 rule: 80 percent of your optimization can

be accomplished by just 20 percent of effort. After that, any tweaking left to be done

is specific to current search engine tendencies and liable to become ineffective

once a search engine updates its algorithm. Therefore don’t waste your time in

optimizing for each and every search engine · just get the basics right and move on

to the next page.

Remember, you should always optimize with the end-user in mind, not the search engines.

Like I mentioned earlier, search engines are continuously optimizing their algorithms in order to improve on the key criteria: relevancy. By ensuring that your website content is targeted on a particular keyword, and is judged as ·good’ content based on both on-page optimization (keyword density) and off-page factors (lots of quality inbound links), you will also guarantee that your website will keep ranking highly for your search terms no matter what changes are brought into a search engine’s algorithm, whether it’s a dampening factor a la Sandbox or any other quirk the search engine industry throws up in the future.

Have you taken a look at SEO Elite yet? If not…

What’s stopping you?

Now, get out there and start smoking the search engines!

A lot of my subscribers have written to me wanting to know how i have raised my website from a Google rank “0″ in November 2007 to Google rank “4″ in April 2008, that is, in exactly 6 months! I always say to them that it is all about what you want for your site and no more. You Really Have To Want to raise your site up the rankings. As I have mentioned before in my many articles, unless you really want to generate a lot of traffic and make money online, you might end up creating a brand new website, submitting it to thousands of search engines and waiting for customers to pop up, click your links and buy whatever it is that you are selling only to no avail.

This is my testimony, If you optimize your site like the way thousands of webmasters are doing today, chances are that you might never rank highly for any keyword term. Sites do not rank well for any keyword term by chance, you may know that already, since you might have done a little reasearch and may be come to that conclusion. You should also be able to decipher that i want to rank well and get traffic for the keyword make money online through this article. At this point in time, what you need to do is not to jump into a different niche, no! It almost cost me a fortune and a lot of time doing that.

I use articles to build my website, and this is the secret that i will let you into today. You can do that too, by using the right words, the right word length, the right word positioning and the right word density to build long tail keyword article based sites that bring huge profits through the Google Adsense Program, Yahoo Publisher Program, Chitika, Affiliate Programs as well as Information Product Sales. This is what is referred to as web publishing: It involves building original content pages and knowing how to generate links and traffic to these pages.

I want you to launch your web publishing empire today. It took me one year to write my first article, four months to write my second one. Yet today am writing ten of them daily. Not getting started today will definitely make you fail. Am not saying that you should not plan on how you will get started though, what am saying is that you should get started today since there is no perfect way to get started anyway. The secret to the success of a web publisher is so simple, it hurts my fingers writing it down! Its about putting the right words on your web pages, getting free search engine traffic or people for free to your website and earning money. That is it. Does it sound simple enough?

Maybe you are asking now, how will i generate links pointing to my website and make money online? Simple, search for “Google keyword tool” on your browser, you will get it listed on the first position, its a database of what Google adword customers use to find keywords to bid on Google advertiser program. Type in your basic keyword to find keyword phrases you should target as a publisher. It will come up with all the associated information including search volumes and relevancy. It will also tell you which one of those keywords generated will bring you more traffic to your website.

So far so good, now write an article based on that keyword, it should have the proper keyword density, i use 1% as my keyword density. i.e. for every 100 words i place my keyword only once. If my article has 500 words i will only repeat it 5 times, no more no less. It works for me. But i also make sure that my article is understood by the normal person looking for that information. My keywords have to have very low competition, they should be high traffic keywords and have high payment rates. How i determine this is a topic for another day though.

Go to your website and publish your article there. This will ensure that you have a url related to your article and a space to put your Google code. Later on i will show you how, and what makes me manage my site so well and how to interact with your customers. When you are done go to ezinearticles.com, open a free account and post your article there. Include an authors bio provided at the end of your posting. It should be a little bit about yourself and a link pointing to the website url that you created for people to get more information about the particular keyword, like i have done here. It takes about a week to have your article approved at ezine articles for the first time author, but its worthy the wait. Once its approved then the process of getting traffic and links to your website starts. Thousands of people visit this site looking for information just like what you have written, others want to publish it on their sites, thereby giving you the much needed backlinks. Stay tuned for part 2 of how i make money online.

How to get your website listed on Google

There are numerous ways to get your website listed on the great Google search engine. But…and there is always a big but, before continuing I’ve got to make one thing perfectly clear.

Do NOT pay any submission service to get your website or blog listed. Unless you have deep pockets and thousands of big bucks to throw away…then go ahead and waste your money. I guarantee that paying any submission service to get your website listed on google is a complete waste of time and money!

Free submission services are also a waste of time…don’t do it. These submission services make their money from unsuspecting new Internet marketing folks who have no idea what they are doing.

Do it yourself and I will show you how right here…right now! And the price for this information is FREE!

What is your website or about?

Ok…here is how you get your website listed without all the hassles.

1. Write an article about whatever your website or blog is about. Then submit it to the article directories. Just make sure that you insert your website URL or Blog URL in the bio page of each directory. Be sure to add some of the same keywords that you use on your website.

2. The absolute best article directories are ezinearticles.com and goarticles.com Remember to add you bio in the resource box provided by all the article directories.

The resource or bio box is a short paragraph about the author that publishers usually want with each article submissions. It’s included at the bottom of the published article. You should have your name, and web site address. You should also add your tagline and email address.

3. Remember I write articles about what I know. Do not copy someone Else’s work…it’s ok to read an article to get new ideas and write something similar that will interest your readers. Never copy anything word for word!

Ok…now if you don’t believe me here is proof on how this works. Try Googling “Southwestern trees” you will see my gardening website listed on the first page and top billing. I actually have hundreds of webpages that are listed at the top of Google SERP’s. I did this by writing articles about southwestern trees and submitting them to the articles directories.

Get your new website listed in weeks.

Ok…I know many of you are saying…what about a brand new website?

Before we continue I want to set the record straight about “Google’s Sandbox Theory”. IT is just that “theory”. In my opinion there is no sandbox unless the google gods have cats as pets!

Lets continue… try googling “How to start a website”. You will see my artricle that I submitted to ezinearticles.com around the first week of April. This is an ezinearticles.com article. If you click on the link it will take you to my article and my bio page that contains my website URL.

Go ahead do it right now I’ll wait…

This will generate some traffic, and will also build links to your site which in turn helps your website page rank. My IM website is only about 6 months old and it already has a page rank of 2. I do almost zero promotion other than artricle marketing.

By the way page rank is a system for ranking web pages/sites/blogs/pdf files according to link popularity and text-matching. The higher your page rank the better your chances you will be listed on the first page of your chosen keywords.

I never submitted my marketing website to any search engine…they actually found it thru article marketing. However if you want to submit anyway just follow the instruction provided on the next paragraph.

Submitting to the search engines yourself

OK…as you know by now I hate submission services. The only submissions you should do can be done yourself. And in my opinion you should only submit to the top three search engines, Google, Yahoo and MSN.

GOOGLE Traffic – Everyone knows or should know that the top search engine is Google. Did you know that once you have your website published or online google will eventually crawl it. This is done by way of your webhosting company. All webhosting websites are already indexed by google and if you have your website uploaded on their server Google will eventually find it and index it. However the best way to get your site notice is to submit it yourself. Add your website (http://www.google.com/addurl/?continue=/addurl) it’s is free. Remember it will take sometime before your website will appear for the “keywords” that you have on your content.

Yahoo Traffic – The next Search Engine is Yahoo they are somewhat harder to get listed but there is a way to do it free. It will take longer than google but again it is free. They also have a paid service which will guaranteed inclusion within 2-3 weeks, but the cost is around 300.00 bucks (I wouldn’t Pay)! Please make sure to use the Suggesting a Site to the Directory link. Here is the link to submit free: (http://help.yahoo.com/l/us/yahoo/directory/)

MSN Traffic – Now the thrid Search Engine is MSN. If you have good content on your website it should get indexed in about 3-4 weeks. By the way MSN and Live.com are one and the same. Here is the link (http://search.msn.com.sg/docs/submit.aspx?FORM=WSDD)

Do a little SEO yourself. Put your keywords in your page heading meta tags.

(meta http-equiv=”Content-Type” content=”text/html; charset=UTF-8″ /)

(title>Your title (/title)

(meta name=”Description” content=”description of your website”)

(meta name=”KeyWords” content=”your keywords shouldn’t be more than eight”)

(h1)Your heading(/h1)

The heading should be what your web page is about. Also sprinkle some of the same keywords (a little seo yourself) on your text. And more importanly use a variation of those same keywords (e.g. “Self SEO”, How to do your own SEO” etc…)

Example: (h1)A little SEO yourself(/h1)

Put your keywords into the description tag:

Example: (meta name=”Description” content=”How to do SEO yourself”)

Remember the description tag is what will be displayed on the search engines. So you will want something that will make the searcher click on your link.

ADD your keywords in on your alt tags but not all of them.

If you’ve got graphics on your page, considering writing a natural piece of unique, keyword rich text to apply to the alternative text (alt tag) on your image. Example: (img src=”yourimage.jpg border=”0″ alt=”a little seo yourself”)

Put your keywords into the page name Example: www.yourwebsite.com/do-seo-yourself.htm Just don’t over do it. Stay away from do-it-your-self-search-engine-optimizaion.htm

Got questions? You can always email me by visiting my bio resource page.

Below you’ll find twelve ideas to drive qualified leads to your Web site over the next year. Even if you don’t follow this exact order, it might make sense to start with January and February’s tasks because they lay the foundation for the rest of the year.

January: Set Up Google Analytics. The reason why this is first on your month-by-month plan is because you should have a benchmark to compare to your year-end results. The reason I suggest Google Analytics over any other traffic report program is the price–free–and the quality, comprehension, and usability of the program.

February: Do a Keyword Analysis. If you use the language of your prospects they’ll find you at the search engines. You can uncover the words they use through an in-depth keyword analysis.

We used to recommend keyword analyses to businesses in competitive industries. Now every industry is competitive on the Web, and chances are your competition has already had a keyword analysis done.

Use a tool like Keyword Discovery or WordTracker to help you discover what your prospects are Googling, or have a professional do an in-depth analysis.

March: Review Your Web Site. Take what you learned from your keyword analysis and rewrite the copy on your Web site. Focus on page titles, headers, and your body copy. It’s also a good idea to read through your site and excise any out-of-date copy.

April: Gather Information. There are so many amazing resources out there for keeping up with what’s going on in your industry…or–better yet–your customers’ industries. One of the best ways to manage all of this incoming information is through RSS: Real Simple Syndication.

Blogs, podcasts, search engines and many Web sites can generate newsfeeds that you can subscribe to and manage in one place. Set up a free account at Bloglines or a personalized Google page and start subscribing. Then watch targeted news and information flow your way.

May: Get Social with Social Media. Social media Web sites are places where your customers and prospects congregate. You should be there. Don’t look at this as a place to push your products and services, but rather to listen and learn. B2C companies should check out MySpace, while any business can benefit from a presence on Facebook or LinkedIn.

June: Focus on Your Blog. With your keyword analysis under your belt and a fresh delivery of news and information on your chosen areas of expertise, posting regularly to your blog should be a snap.

I can’t think of anything that can jumpstart your organic search engine visibility and establish your expertise and credibility quicker than a focused, content-rich blog.

Beginners: If you don’t have a blog and aren’t ready to take the leap, consider finding the influential bloggers in your niche, then read and comment intelligently on their blog. It will be a good first step into the blogosphere.

Experts: Set a goal of doubling the number of subscribers to both your RSS and email feeds. Put your feed signup “above the fold,” if it’s not there already.

July: Get Ready for Your Close Up. Most Web marketers agree: video is the killer app. Videos engage people in ways that words often fail to do. Search engines such as Google and Ask are returning video with traditional Web pages.

If you already have videos on your site consider setting up a free account at YouTube, starting your own channel, and uploading your videos there. A single video might bring you thousands of visitors.

August: Spread the Word with Article Marketing. Article marketing is an underutilized method of reaching thousands of prospects that would have never heard of you otherwise.

By sending out short, targeted articles through an article distribution service such as iSnare or The Phantom Writers, you can drive links and traffic back to your Web site or blog.

Use your completed keyword research to help you write compelling, click-worthy articles that can’t be ignored.

September: Dust Off Your Email Newsletter. Email may be getting a bad rap these days due to low delivery rates, but it’s still a cost-effective way of reaching a wide swath of Internet users. Remember: no one’s ever coming back to your Web site. Ever. Not even your Mom. So get over it.

Instead, provide a compelling reason for them to subscribe to your email newsletter so that you can stay in contact with them and retain front-of-mind real estate for when they’re ready to make a buying decision.

October: Publish Your eBook. Remember when I told you to set up Google Analytics? Well, now it’s time to review your stats and find out which keyword phrases are driving the most traffic to your site.

Maybe it’s “deep fry a turkey.” Or “spanking alternatives.” Or “buying carbon emission credits.”

Whatever’s driving traffic to your site is an opportunity for an eBook topic. It’s an itch that needs scratching. Now, start writing.

November: Give an Ear (or a Voice) to Podcasts. You don’t have to host your own podcast to realize and leverage the power of podcasting.

You can find podcasts in your niche or industry by browsing the podcast directory in the Apple iTunes store. (Don’t worry: they’re almost invariably free, and no, you don’t need an iPod to listen to a podcast, just a computer.)

Listening may give you the impetus to start your own or to at least contact some podcasters in your industry and see if they want to interview you.

December: Put on a Webinar. The cost of putting on Webinars has come down in recent years. Yugma.com offers free and inexpensive methods to create your own Webinars, then you can record them and sell them as digital downloads.

Whatever you method for delivery, Webinars allow you to communicate with many people at once anywhere in the world.

In Conclusion: 2008 is going to be your best year ever…if you start planning now. Don’t try and do everything at once, but budget time every month to accomplish your goals for that month.

Do you need Internet marketing for your website? Many people do not think so. They believe that having a good quality website is enough to get them the kind of traffic that they need. They could not be further from the truth.

You may have the best site on the net, but its not getting noticed because it lacks visibility. Visibility is a key ingredient in getting your site working for you. If you are not as visible as your competitors despite having quality content and the best features around, you will lose out on the kind of traffic that will turn a buck.

Think about Internet marketing as a way to get your name noticed. It is just like having a product that is so good and while there are people who swear to it being good and come back over and over again to buy it, you may not be generating enough of an income to keep your production up.

Sure, you can say that word of mouth will get people to come and visit your site and maybe they will, but until they do, will you suffer the consequences of low revenues simply because you did not use an effective internet marketing strategy?

What is Internet Marketing?

In a nutshell, internet marketing is an advertising strategy that gets your site noticed by a lot of people who visit a lot of other sites and search on search engines.

How can Internet Marketing get my Site Noticed?

There are a lot of ways. The most commonly used and probably one of the most effective strategies used by websites and SEO companies is pay per click advertising.

Pay per Click Advertising

This strategy gets your website advertised and shown every time somebody types in a keyword that you chose to include in your list of active keywords. With the right bid and the right ad formulation, you can get people to come to your site simply because of these pay per click ads.

How much does Pay per Click Advertising cost?

Pay per click ads may cost as little as a few cents per click to over fifty dollars, depending on the popularity of the keyword you are targeting. Since you only pay for the clicks that your ad gets from people who might be interested in what your site has to offer, you are getting the most for your money.

Google has released a great tool to search for the current CPC for keywords which can be found at https://adwords.google.com/select/KeywordToolExternal

Where are pay per clicks ads displayed?

Aside from these ads showing on search engines every time someone types in a keyword that is relevant to what you are selling, by choosing to have these ads shown on certain content based sites that are related to your product, you also get an additional chance to attract customers that are already on a site that is similar to yours.

Always be careful when opting to advertise on certain content based sites. The website publisher receives a percentage of the cost of the click. The temptation will be there for the website publisher to click on your ad to increase their revenue from the pay per click ads on their sites.

Does Internet Marketing Work?

It may be pretty competitive but it does work. Search-related ad spending in North America neared $10 billion in 2006, growing 62% over 2005. [Source: Search Engine Marketing Professional Organization (SEMPO), February 2007]

Six out of 10 marketers are planning to increase their paid search and natural search budgets over the next 12 months, a higher percentage than for any other digital marketing channel. [Source: E-consultancy UK Search Engine Marketing Report 2007, April 2007]

* Half of respondents are getting an ROI of more than 300% from PPC. For natural search, two thirds of respondents are getting returns of more than 300%.

* For return on investment, Google is rated as the best search engine by 80% of company respondents compared to 11% for MSN and 8% for Yahoo!.

In the US, 65% of advertisers said they expect to increase investment in search, while a mere 7% were likely to cut spending. The rest expected search spending to remain constant. [Source: Jupiter Research, April 2007]

I want to try Pay per Click Advertising, where do I start?

If you are planning on starting your own pay per click advertising campaign, Google Adwords (http://adwords.google.com) or Yahoo! Search Marketing (http://searchmarketing.yahoo.com/) is a good place to start.

Try not to waste your recious advertising budget by making sure to implement the return on investment (ROI) tracking codes on your site. The ROI tracking quickly identifies which ads are working and those that are just wasting money.

Is there anyone who will do my Internet Marketing for me?

You could use an Internet Marketing Agency but expect to spend around 15% of your budget on their management fee. Their expertise is usually worth paying for as they can use their expertise to make your budget go much further.

You can get the quality traffic you want by using smart Internet Marketing strategies. With a little help from an SEO team that knows what needs to be done within a certain budget; you can start generating the kind of income you want.

Good luck with your Internet Marketing!

 

Critical Analysis of Web Crawlers’ Algorithms

 Minou Parhizkar 0527553

Abstract- A web crawler is a program or automated script which browses the World Wide Web in a methodical, automated manner. The objective of the paper is to make a make a critical analysis of the algorithms used by Web Crawlers. It intends to review and evaluate the different and various approaches to the methods used by the different web search engines to catalog the information.

 

 

Index Terms-

Web Crawler, Search Engines, WWW, SEO

 

•I.     INTRODUCTION

 

The software that searches for information and returns sites which provide that information is referred to as a search engine or web crawler. Everyone uses web crawlers-indirectly, at least! Every time you search the Internet using a service such as Alta Vista, Excite, or Lycos, you’re making use of an index that’s based on the output of a web crawler. Web crawlers-also known as spiders, robots, or wanderers-are software programs that automatically traverse the Web. Search engines use crawlers to find what’s on the Web; then they construct an index of the pages that were found.

 

Search Engines use spiders to index websites. When you submit your website pages to a search engine by completing their required submission page, the search engine spider will index your entire site. A ‘spider’ is an automated program that is run by the search engine system. Spider visits a web site, read the content on the actual site, the site’s Meta tags and also follow the links that the site connects. The spider then returns all that information back to a central depository, where the data is indexed. It will visit each link you have on your website and index those sites as well. Some spiders will only index a certain number of pages on your site.

A spider is almost like a book where it contains the table of contents, the actual content and the links and references for all the websites it finds during its search, and it may index up to a million pages a day.

 

 

Example: Google spider

 

When you ask a search engine to locate information, it is actually searching through the index which it has created and not actually searching the Web. Different search engines produce different rankings because not every search engine uses the same algorithm to search through the indices.

One of the things that a search engine algorithm scans for is the frequency and location of keywords on a web page, but it can also detect artificial keyword stuffing or spamdexing. Then the algorithms analyze the way that pages link to other pages in the Web. By checking how pages link to each other, an engine can both determine what a page is about, if the keywords of the linked pages are similar to the keywords on the original page. Most of the top-ranked search engines are crawler based search engines while some may be based on human compiled directories. The people behind the search engines want the same thing every webmaster wants – traffic to their site. Since their content is mainly links to other sites, the thing for them to do is to make their search engine bring up the most relevant sites to the search query, and to display the best of these results first. In order to accomplish this, they use a complex set of rules called algorithms. When a search query is submitted at a search engine, sites are determined to be relevant or not relevant to the search query according to these algorithms, and then ranked in the order it calculates from these algorithms to be the best matches first.

Search engines keep their algorithms secret and change them often in order to prevent webmasters from manipulating their databases and dominating search results. They also want to provide new sites at the top of the search results on a regular basis rather than always having the same old sites show up month after month. An important difference to realize is that search engines and directories are not the same. Search engines use a spider to “crawl” the web and the web sites they find, as well as submitted sites. As they crawl the web, they gather the information that is used by their algorithms in order to rank your site.

This paper aims at critically analyzing various search engineers, how they work and comparing their algorithms.

•II.     Working of web crawlers – a detailed look up

Let us now look at a more detailed explanation on how Search Engines work. Crawler based search engines are primarily composed of three parts.

A search engine robot’s action is called spidering, as it resembles the multiple legged spiders. The spider’s job is to go to a web page, read the contents, connect to any other pages on that web site through links, and bring back the information. From one page it will travel to several pages and this proliferation follows several parallel and nested paths simultaneously. Spiders frequent the site at some interval, may be a month to a few months, and re-index the pages. This way any changes that may have occurred in your pages could also be reflected in the index. The spiders automatically visit your web pages and create their listings. An important aspect is to study what factors promote “deep crawl” – the depth to which the spider will go into your website from the page it first visited. Listing ’submitting or registering’ with a search engine is a step that could accelerate and increase the chances of that engine “spidering” your pages.

The spider’s movement across web pages stores those pages in its memory, but the key action is in indexing. The index is a huge database containing all the information brought back by the spider. The index is constantly being updated as the spider collects more information. The entire page is not indexed and the searching and page-ranking algorithm is applied only to the index that has been created. Most search engines claim that they index the full visible body text of a page. In a subsequent section, we explain the key considerations to ensure that indexing of your web pages improves relevance during search. The combined understanding of the indexing and the page-ranking process will lead to developing the right strategies. The Meta tags ‘Description’ and ‘Keywords’ have a vital role as they are indexed in a specific way. Some of the top search engines do not index the keywords that they consider spam. They will also not index certain ’stop words’ (commonly used words such as ‘a’ or ‘the’ or ‘of’” so as to save space or speed up the process. Images are obviously not indexed, but image descriptions or Alt text or “text within comments” is included in the index by some search engines.

The search engine software or program is the final part. When a person requests a search on a keyword or phrase, the search engine software searches the index for relevant information. The software then provides a report back to the searcher with the most relevant web pages listed first. The algorithm-based processes used to determine ranking of results are discussed in greater detail later.

These directories compile listings of websites into specific industry and subject categories and they usually carry a short description about the website. Inclusion in directories is a human task and requires submission to the directory producers. Visitors and researchers over the net quite often use these directories to locate relevant sites and information sources. Thus directories assist in structured search. Another important reason is that crawler engines quite often find websites to crawl through their listing and links in directories. Yahoo and The Open Directory are amongst the largest and most well known directories. LookSmart is a directory that provides results to partner sites such as MSN Search, Excite and others. Lycos is an example of a site that pioneered the search engine but shifted to the Directory model depending on AlltheWeb.com for its listings.

Hybrid Search Engines are both crawler based as well as human powered. In plain words, these search engines have two sets of listings based on both the mechanisms mentioned above. The best example of hybrid search engines is Yahoo, which has got a human powered directory as well as a Search toolbar administered by Google. Although, such engines provide both listings they are generally dominated by one of the two mechanisms. Yahoo is known more for its directory rather than crawler based search engine.

Search engines rank web pages according to the software’s understanding of the web page’s relevancy to the term being searched. To determine relevancy, each search engine follows its own group of rules. The most important rules are.

- The location of keywords on your web page; and – How often those keywords appear on the page ‘the frequency’

For example, if the keyword appears in the title of the page, then it would be considered to be far more relevant than the keyword appearing in the text at the bottom of the page. Search engines consider keywords to be more relevant if they appear sooner on the page (like in the headline) rather than later. The idea is that you’ll be putting the most important words – the ones that really have the relevant information – on the page first.

Search engines also consider the frequency with which keywords appear. The frequency is usually determined by how often the keywords are used out of all the words on a page. If the keyword is used 4 times out of 100 words, the frequency would be 4%. Of course, you can now develop the perfect relevant page with one keyword at 100% frequency – just put a single word on the page and make it the title of the page as well. Unfortunately, the search engines don’t make things that simple.

While all search engines do follow the same basic rules of relevancy, location and frequency, each search engine has its own special way of determining rankings. To make things more interesting, the search engines change the rules from time to time so that the rankings change even if the web pages have remained the same. One method of determining relevancy used by some search engines ‘like HotBot and Infoseek’, but not others ‘like Lycos’, is the Meta tags. Meta tags are hidden HTML codes that provide the search engine spiders with potentially important information like the page description and the page keywords.

Meta tags are often labeled as the secret to getting high rankings, but Meta tags alone will not get you a top 10 ranking. On the other hand, they certainly don’t hurt. Detailed information on meta-tags and other ways of improving search engine ranking is given later in this chapter.

In the early days of the web, webmasters would repeat a keyword hundreds of times in the Meta tags and then add it hundreds of times to the text on the web page by making it the same color as the background. However, now, major search engines have algorithms that may exclude a page from ranking if it has resorted to “keyword spamming”; in fact some search engines will downgrade ranking in such cases and penalize the page.

Link analysis and ‘clickthrough’ measurement are certain other factors that are “off the page” and yet crucial in the ranking mechanism adopted by some leading search engines. This is quickly emerging as the most important determinant of ranking, but before we study this, we must first look at the most popular search engines and then look at the various steps you can take to improve your success at each of the stages – spidering, indexing and ranking.

For March 2003, according to a study by Jupiter Media Metrix, there were an estimated 114 million Internet users online in the US at work or at home, 80 percent of whom are estimated to have made some type of search request during the month.

•III.     a summarised comparison OF SEARCH engines

Yahoo!

been in the search game for many years. is better than MSN but nowhere near as good as Google at determining if a link is a natural citation or not. has a ton of internal content and a paid inclusion program. both of which give them incentive to bias search results toward commercial results things like cheesy off topic reciprocal links still work great in Yahoo!

MSN Search

new to the search game is bad at determining if a link is natural or artificial in nature due to sucking at link analysis they place too much weight on the page content their poor relevancy algorithms cause a heavy bias toward commercial results likes bursty recent links new sites that are generally un-trusted in other systems can rank quickly in MSN Search things like cheesy off topic reciprocal links still work great in MSN Search

Google

has been in the search game a long time, and saw the web graph when it is much cleaner than the current web graph is much better than the other engines at determining if a link is a true editorial citation or an artificial link looks for natural link growth over time heavily biases search results toward informational resources trusts old sites way too much a page on a site or sub-domain of a site with significant age or link related trust can rank much better than it should, even with no external citations they have aggressive duplicate content filters that filter out many pages with similar content if a page is obviously focused on a term they may filter the document out for that term. on page variation and link anchor text variation are important. a page with a single reference or a few references of a modifier will frequently outrank pages that are heavily focused on a search phrase containing that modifier crawl depth determined not only by link quantity, but also link quality. Excessive low quality links may make your site less likely to be crawled deep or even included in the index. things like cheesy off topic reciprocal links are generally ineffective in Google when you consider the associated opportunity cost

Ask

looks at topical communities due to their heavy emphasis on topical communities they are slow to rank sites until they are heavily cited from within their topical community due to their limited market share they probably are not worth paying much attention to unless you are in a vertical where they have a strong brand that drives significant search traffic

•IV.     Detailed Analysis of Search Engines

Now that we have understood the working and basics of web crawlers and reviewed a summarized comparison of a few major search engines out in the market, now we are in a position to have a detailed analysis and comparison between these and get into nitty gritty technical details. The sections below will deal with each of these engines one by one with a detailed analysis.

•V.     Yahoo!

 

Yahoo! was founded in 1994 by David Filo and Jerry Yang as a directory of websites. For many years they outsourced their search service to other providers, but by the end of 2002 they realized the importance and value of search and started aggressively acquiring search companies.

Overture purchased AllTheWeb and AltaVista. Yahoo! purchased Inktomi (in December 2002) and then consumed Overture (in July of 2003), and combined the technologies from the various search companies they bought to make a new search engine.

•a)                   On Page Content

Yahoo! offers a paid inclusion program, so when Yahoo! Search users click on high ranked paid inclusion results in the organic search results Yahoo! profits. In part to make it easy for paid inclusion participants to rank, I believe Yahoo! places greater weight on on-the-page content than a search engine like Google does.

Being the #1 content destination site on the web, Yahoo! has a boatload of their own content which they frequently reference in the search results. Since they have so much of their own content and make money from some commercial organic search results it might make sense for them to bias their search results a bit toward commercial websites.

Using descriptive page titles and page content goes a long way in Yahoo!

In my opinion their results seem to be biased more toward commerce than informational sites, when compared with Google.

•b)                   Crawling

Yahoo! is pretty good at crawling sites deeply so long as they have sufficient link popularity to get all their pages indexed. One note of caution is that Yahoo! may not want to deeply index sites with many variables in the URL string, especially since

Yahoo! already has a boatload of their own content they would like to promote (including verticals like Yahoo! Shopping) Yahoo! offers paid inclusion, which can help Yahoo! increase revenue by charging merchants to index some of their deep database contents.

You can use Yahoo! Site Explorer to see how well they are indexing your site and which sites link at your site.

•c)                   Query Processing

Certain words in a search query are better at defining the goals of the searcher. If you search Yahoo! for something like “how to SEO ” many of the top ranked results will have “how to” and “SEO” in the page titles, which might indicate that Yahoo! puts quite a bit of weight even on common words that occur in the search query.

Yahoo! seems to be more about text matching when compared to Google, which seems to be more about concept matching.

•d)                   Link Reputation

Yahoo! is still fairly easy to manipulate using low to mid quality links and somewhat to aggressively focused anchor text. Rand Fishken recently posted about many Technorati pages ranking well for their core terms in Yahoo!. Those pages primarily have the exact same anchor text in almost all of the links pointing at them.

Sites with the trust score of Technorati may be able to get away with more unnatural patterns than most webmasters can, but I have seen sites flamethrown with poorly mixed anchor text on low quality links, only to see the sites rank pretty well in Yahoo! quickly.

•e)                   Page vs Site

A few years ago at a Search Engine Strategies conference Jon Glick stated that Yahoo! looked at both links to a page and links to a site when determining the relevancy of a page. Pages on newer sites can still rank well even if their associated domain does not have much trust built up yet so long as they have some descriptive inbound links.

•f)                    Site Age

Yahoo! may place some weight on older sites, but the effect is nowhere near as pronounced as the effect in Google’s SERPs.

It is not unreasonable for new sites to rank in Yahoo! in as little as 2 or 3 months.

•g)                   Paid Search

Yahoo! prices their ads in an open auction, with the highest bidder ranking the highest. By early 2007 they aim to make Yahoo! Search Marketing more of a closed system which factors in clickthrough rate (and other algorithmic factors) into their ad ranking algorithm.

Yahoo! also offers a paid inclusion program which charges a flat rate per click to list your site in Yahoo!’s organic search results.

Yahoo! also offers a contextual ad network. The Yahoo! Publisher program does not have the depth that Google’s ad system has, and they seem to be trying to make up for that by biasing their targeting to more expensive ads, which generally causes their syndicated ads to have a higher click cost but lower average clickthrough rate.

•h)                   Editorial

Yahoo! has many editorial elements to their search product. When a person pays for Yahoo! Search Submit that content is reviewed to ensure it matches Yahoo!’s quality guidelines. Sites submitted to the Yahoo! Directory are reviewed for quality as well.

In addition to those two forms of paid reviews, Yahoo! also frequently reviews their search results in many industries. For competitive search queries some of the top search results may be hand coded. If you search for Viagra, for example, the top 5 listings looked useful, and then I had to scroll down to #82 before I found another result that wasn’t spammy.

Yahoo! also manually reviews some of the spammy categories somewhat frequently and then reviews other samples of their index. Sometimes you will see a referral like http://corp.yahoo-inc.com/project/health-blogs/keepers if they reviewed your site and rated it well.

Sites which have been editorially reviewed and were of decent quality may be given a small boost in relevancy score. Sites which were reviewed and are of poor quality may be demoted in relevancy or removed from the search index.

Yahoo! has published their content quality guidelines. Some sites that are filtered out of search results by automated algorithms may return if the site cleans up the associated problems, but typically if any engine manually reviews your site and removes it for spamming you have to clean it up and then plead your case.

•i)                    Social Aspects

Yahoo! firmly believes in the human aspect of search. They paid many millions of dollars to buy Del.icio.us, a social bookmarking site. They also have a similar product native to Yahoo! called My Yahoo!

Yahoo! has also pushed a question answering service called Yahoo! Answers which they heavily promote in their search results and throughout their network. Yahoo! Answers allows anyone to ask or answer questions. Yahoo! is also trying to mix amateur content from Yahoo! Answers with professionally sourced content in verticals such as Yahoo! Tech.

•j)                    Yahoo! SEO Tools

Yahoo! has a number of useful SEO tools.

Overture Keyword Selector Tool – shows prior month search volumes across Yahoo! and their search network. Overture View Bids Tool – displays the top ads and bid prices by keyword in the Yahoo! Search Marketing ad network. Yahoo! Site Explorer – shows which pages Yahoo! has indexed from a site and which pages they know of that link at pages on your site. Yahoo! Mindset – shows you how Yahoo! can bias search results more toward informational or commercial search results. Yahoo! Advanced Search Page – makes it easy to look for .edu and .gov backlinks Yahoo! Buzz – shows current popular searches

•k)                   Yahoo! Business Perspectives

Being the largest content site on the web makes Yahoo! run into some inefficiency issues due to being a large internal customer. For example, Yahoo! Shopping was a large link buyer for a period of time while Yahoo! Search pushed that they didn’t agree with link buying. Offering paid inclusion and having so much internal content makes it make sense for Yahoo! to have a somewhat commercial bias to their search results.

They believe strongly in the human and social aspects of search, pushing products like Yahoo! Answers and My Yahoo!.

I think Yahoo!’s biggest weakness is the diverse set of things that they do. In many fields they not only have internal customers, but in some fields they have product duplication, like with Yahoo! My Web and Del.icio.us. 

•l)                    Search Marketing Perspective

I believe if you do standard textbook SEO practices and actively build quality links it is reasonable to expect to be able to rank well in Yahoo! within 2 or 3 months. If you are trying to rank for highly spammed keyword phrases keep in mind that the top 5 or so results may be editorially selected, but if you use longer tail search queries or look beyond the top 5 for highly profitable terms you can see that many people are indeed still spamming them to bits.

As Yahoo! pushes more of their vertical offerings it may make sense to give your site and brand additional exposure to Yahoo!’s traffic by doing things like providing a few authoritative answers to topically relevant questions on Yahoo! Answers.

•VI.     Msn Search

MSN Search had many incarnations, being powered by the likes of Inktomi and Looksmart for a number of years. After Yahoo! bought Inktomi and Overture it was obvious to Microsoft that they needed to develop their own search product. They launched their technology preview of their search engine around July 1st of 2004. They formally switched from Yahoo! organic search results to their own in house technology on January 31st, 2005.

•a)                   On Page Content

Using descriptive page titles and page content goes a long way to help you rank in MSN. I have seen examples of many domains that ranked for things like

state name+ insurance type + insurance

on sites that were not very authoritative which only had a few instances of state name and insurance as the anchor text. Adding the word health, life, etc. to the page title made the site relevant for those types of insurance, in spite of the site having few authoritative links and no relevant anchor text for those specific niches.

Additionally, internal pages on sites like those can rank well for many relevant queries just by being hyper focused, but MSN currently drives little traffic when compared with the likes of Google.

•b)                   Crawling

MSN has got better at crawling, but I still think Yahoo! and Google are much better at crawling. It is best to avoid session IDs, sending bots cookies, or using many variables in the URL strings. MSN is nowhere near as comprehensive as Yahoo! or Google at crawling deeply through large sites like eBay.com or Amazon.com.

•c)                   Query Processing

I believe MSN might be a bit better than Yahoo! at processing queries for meaning instead of taking them quite so literally, but I do not believe they are as good as Google is at it.

While MSN offers a tool that estimates how commercial a page or query is I think their lack of ability to distinguish quality links from low quality links makes their results exceptionally biased toward commercial results.

•d)                   Link Reputation

By the time Microsoft got in the search game the web graph was polluted with spammy and bought links. Because of this, and Microsoft’s limited crawling history, they are not as good as the other major search engines at telling the difference between real organic citations and low quality links.

MSN search reacts much more quickly than the other engines at ranking new sites due to link bursts. Sites with relatively few quality links that gain enough descriptive links are able to quickly rank in MSN. I have seen sites rank for one of the top few dozen most expensive phrases on the net in about a week.

•e)                   Page vs Site

I think all major search engines consider site authority when evaluating individual pages, but with MSN it seems as though you do not need to build as much site authority as you would to rank well in the other engines.

•f)                    Site Age

Due to MSN’s limited crawling history and the web graph being highly polluted before they got into search they are not as good as the other engines at determining age related trust scores. New sites doing general textbook SEO and acquiring a few descriptive inbound links (perhaps even low quality links) can rank well in MSN within a month.

•g)                   Paid Search

Microsoft’s paid search product, AdCenter, is the most advanced search ad platform on the web. Like Google, MSN ranks ads based on both max bid price and ad clickthrough rate. In addition to those relevancy factors MSN also allows you to place adjustable bids based on demographic details. For example, a mortgage lead from a wealthy older person might be worth more than an equivalent search from a younger and poorer person.

•h)                   Editorial

All major search engines have internal relevancy measurement teams. MSN seems to be highly lacking in this department, or they are trying to use the fact that their search results are spammy as a marketing angle.

MSN is running many promotional campaigns to try to get people to try out MSN Search, and in many cases some of the searches they are sending people to have bogus spam or pornography type results in them. A good example of this is when they used Stacey Kiebler to market their Celebrity Maps product. As of writing this, their top search result for Stacey Kiebler is still pure spam.

Based on MSN’s lack of feedback or concern toward the obvious search spam noted above on a popular search marketing community site I think MSN is trying to automate much of their spam detection, but it is not a topic you see people talk about very often. Here are MSN’s Guidelines for Successful Indexing, but they still have a lot of spam in their search results.

•i)                    Social Aspects

Microsoft continues to lag in understanding what the web is about. Executives there should read The Cluetrain Manifesto. Twice.Or maybe three times.

They don’t get the web. They are a software company posing as a web company.

They launch many products as though they have the market stranglehold monopolies they once enjoyed, and as though they are not rapidly losing them. Many of Microsoft’s most innovative moves get little coverage because when they launch key products they often launch them without supporting other browsers and trying to lock you into logging in to Microsoft.

•j)                    MSN SEO Tools

MSN has a wide array of new and interesting search marketing tools. Their biggest limiting factor with them is that they have limited search market share.

Some of the more interesting tools are

Keyword Search Funnel Tool – shows terms that people search for before or after they search for a particular keyword Demographic Prediction Tool – predicts the demographics of searchers by keyword or site visitors by website Online Commercial Intention Detection Tool – estimates the probability of a search query or web page being commercial, informational-transactional, or Search Result Clustering Tool – clusters search results based on related topics

You can view more of their tools under the demo section at Microsoft’s Adlab.

•VII.     Google Search

Google sprang out of a Stanford research project to find authoritative link sources on the web. In January of 1996 Larry Page and Sergey Brin began working on BackRub.

After they tried shopping the Google search technology to no avail they decided to set up their own search company. Within a few years of forming the company they won distribution partnerships with AOL and Yahoo! that helped build their brand as the industry leader in search. Traditionally search was viewed as a loss leader.

Google did not have a profitable business model until the third iteration of their popular AdWords advertising program in February of 2002, and was worth over 100 billion dollars by the end of 2005.

•a)                   On Page Content

If a phrase is obviously targeted (ie: the exact same phrase is in most of the following location: in most of your inbound links, internal links, at the start of your page title, at the beginning of your first page header, etc.) then Google may filter the document out of the search results for that phrase. Other search engines may have similar algorithms, but if they do those algorithms are not as sophisticated or aggressively deployed as those used by Google.

Google is scanning millions of books, which should help them create an algorithm that is pretty good at differentiating real text patterns from spammy manipulative text (although I have seen many garbage content cloaked pages ranking well in Google, especially for 3 and 4 word search queries).

You need to write naturally and make your copy look more like a news article than a heavily SEOed page if you want to rank well in Google. Sometimes using less occurrences of the phrase you want to rank for will be better than using more.

You also want to sprinkle modifiers and semantically related text in your pages that you want to rank well in Google.

Some of Google’s content filters may look at pages on a page by page basis while others may look across a site or a section of a site to see how similar different pages on the same site are. If many pages are exceptionally similar to content on your own site or content on other sites Google may be less willing to crawl those pages and may throw them into their supplemental index. Pages in the supplemental index rarely rank well, since generally they are trusted far less than pages in the regular search index.

Duplicate content detection is not just based on some magical percentage of similar content on a page, but is based on a variety of factors. Both Bill Slawski and Todd Malicoat offer great posts about duplicate content detection. This shingles PDF explains some duplicate content detection techniques.

•b)                   Crawling

While Google is more efficient at crawling than competing engines, it appears as though with Google’s BigDaddy update they are looking at both inbound and outbound link quality to help set crawl priority, crawl depth, and weather or not a site even gets crawled at all. To quote Matt Cutts:

The sites that fit “no pages in Bigdaddy” criteria were sites where our algorithms had very low trust in the inlinks or the outlinks of that site. Examples that might cause that include excessive reciprocal links, linking to spammy neighborhoods on the web, or link buying/selling.

In the past crawl depth was generally a function of PageRank (PageRank is a measure of link equity – and the more of it you had the better you would get indexed), but now adding in this crawl penalty for having an excessive portion of your inbound or outbound links pointing into low quality parts of the web creates an added cost which makes dealing in spammy low quality links far less appealing for those who want to rank in Google.

•c)                   Query Processing

While I mentioned above that Yahoo! seemed to have a bit of a bias toward commercial search results it is also worth noting that Google’s organic search results are heavily biased toward informational websites and web pages.

Google is much better than Yahoo! or MSN at determining the true intent of a query and trying to match that instead of doing direct text matching. Common words like how to may be significantly deweighted compared to other terms in the search query that provide a better discrimination value.

Google and some of the other major search engines may try to answer many common related questions to the concept being searched for. For example, in a given set of search results you may see any of the following:

a relevant .gov and/or .edu document a recent news article about the topic a page from a well known directory such as DMOZ or the Yahoo! Directory a page from the Wikipedia an archived page from an authority site about the topic the authoritative document about the history of the field and recent changes a smaller hyper focused authority site on the topic a PDF report on the topic a relevant Amazon, eBay, or shopping comparison page on the topic one of the most well branded and well known niche retailers catering to that market product manufacturer or wholesaler sites a blog post / review from a popular community or blog site about a slightly broader field

Some of the top results may answer specific relevant queries or be hard to beat, while others might be easy to compete with. You just have to think of how and why each result was chosen to be in the top 10 to learn which one you will be competing against and which ones may perhaps fall away over time.

•d)                   Link Reputation

PageRank is a weighted measure of link popularity, but Google’s search algorithms have moved far beyond just looking at PageRank.

As mentioned above, gaining an excessive number of low quality links may hurt your ability to get indexed in Google, so stay away from known spammy link exchange hubs and other sources of junk links. I still sometimes get a few junk links, but I make sure that I try to offset any junky link by getting a greater number of good links.

If your site ranks well some garbage automated links will end up linking to you weather you like it or not. Don’t worry about those links, just worry about trying to get a few real high quality editorial links.

Google is much better at being able to determine the difference between real editorial citations and low quality, spammy, bought, or artificial links.

When determining link reputation Google (and other engines) may look at

link age rate of link acquisition anchor text diversity deep link ratio link source quality (based on who links to them and who else they link at) weather links are editorial citations in real content (or if they are on spammy pages or near other obviously non-editorial links) does anybody actually click on the link?

It is generally believed that .edu and .gov links are trusted highly in Google because they are generally harder to influence than the average .com link, but keep in mind that there are some junky .edu links too (I have seen stuff like .edu casino link exchange directories).

When getting links for Google it is best to look in virgin lands that have not been combed over heavily by other SEOs. Either get real editorial citations or get citations from quality sites that have not yet been abused by others. Google may strip the ability to pass link authority (even from quality sites) if those sites are known obvious link sellers or other types of link manipulators. Make sure you mix up your anchor text and get some links with semantically related text.

Google likely collects usage data via Google search, Google Analytics, Google AdWords, Google AdSense, Google news, Google accounts, Google notebook, Google calendar, Google talk, Google’s feed reader, Google search history annotations, and Gmail. They also created a Firefox browser bookmark synch tool, an anti-phishing tool which is built into Firefox and have relationships with the Opera (another web browser company). Most likely they can lay some of this data over the top of the link graph to record a corroborating source of the legitimacy of the linkage data. Other search engines may also look at usage data.

•e)                   Page vs Site

Sites need to earn a certain amount of trust before they can rank for competitive search queries in Google. If you put up a new page on a new site and expect it to rank right away for competitive terms you are probably going to be disappointed.

If you put that exact same content on an old trusted domain and link to it from another page on that domain it can leverage the domain trust to quickly rank and bypass the concept many people call the Google Sandbox.

Many people have been exploiting this algorithmic hole by throwing up spammy subdomains on free hosting sites or other authoritative sites that allow users to sign up for a cheap or free publishing account. This is polluting Google’s SERPs pretty bad, so they are going to have to make some major changes on this front pretty soon.

•f)                    Site Age

Google filed a patent about information retrieval based on historical data which stated many of the things they may look for when determining how much to trust a site. Many of the things I mentioned in the link section above are relevant to the site age related trust (ie: to be well trusted due to site age you need to have at least some link trust score and some age score).

I have seen some old sites with exclusively low quality links rank well in Google based primarily on their site age, but if a site is old AND has powerful links it can go a long way to helping you rank just about any page you write (so long as you write it fairly naturally).

Older trusted sites may also be given a pass on many things that would cause newer lesser trusted sites to be demoted or de-indexed.

The Google Sandbox is a concept many SEOs mention frequently. The idea of the ‘box is that new sites that should be relevant struggle to rank for some queries they would be expected to rank for. While some people have debunked the existence of the sandbox as garbage, Google’s Matt Cutts said in an interview that they did not intentionally create the sandbox effect, but that it was created as a side effect of their algorithms:

“I think a lot of what’s perceived as the sandbox is artefacts where, in our indexing, some data may take longer to be computed than other data.”

•g)                   Paid Search

Google AdWords factors in max bid price and clickthrough rate into their ad algorithm. In addition they automate reviewing landing page quality to use that as another factor in their ad relevancy algorithm to reduce the amount of arbitrage and other noisy signals in the AdWords program.

The Google AdSense program is an extension of Google AdWords which offers a vast ad network across many content websites that distribute contextually relevant Google ads. These ads are sold on a cost per click or flat rate CPM basis.

•h)                   Editorial

Google is known to be far more aggressive with their filters and algorithms than the other search engines are. They are known to throw the baby out with the bath water quite often. They flat out despise relevancy manipulation, and have shown they are willing to trade some short term relevancy if it guides people along toward making higher quality content.

Short term if your site is filtered out of the results during an update it may be worth looking into common footprints of sites that were hurt in that update, but it is probably not worth changing your site structure and content format over one update if you are creating true value add content that is aimed at your customer base. Sometimes Google goes too far with their filters and then adjusts them back.

Google published their official webmaster guidelines and their thoughts on SEO. Matt Cutts is also known to publish SEO tips on his personal blog. Keep in mind that Matt’s job as Google’s search quality leader may bias his perspective a bit.

Google Sitemaps gives you a bit of useful information from Google about what keywords your site is ranking for and which keywords people are clicking on your listing.

•i)                    Social Aspects

Google allows people to write notes about different websites they visit using Google Notebook. Google also allows you to mark and share your favorite feeds and posts. Google also lets you flavorize search boxes on your site to be biased towards the topics your website covers.

Google is not as entrenched in the social aspects of search as much as Yahoo! is, but Google seems to throw out many more small tests hoping that one will perhaps stick.They are trying to make software more collaborative and trying to get people to share things like spreadsheets and calendars, while also integrating chat into email. If they can create a framework where things mesh well they may be able to gain further marketshare by offering free productivity tools.

•j)                    Google SEO Tools

Google Sitemaps – helps you determine if Google is having problems indexing your site. AdWords Keyword Tool – shows keywords related to an entered keyword, web page, or web site AdWords Traffic Estimator – estimates the bid price required to rank #1 on 85% of Google AdWords ads near searches on Google, and how much traffic an AdWords ad would drive Google Suggest – auto completes search queries based on the most common searches starting with the characters or words you have entered Google Trends – shows multi-year search trends Google Sets – creates semantically related keyword sets based on keyword(s) you enter Google Zeitgeist – shows quickly rising and falling search queries Google related sites – shows sites that Google thinks are related to your site related:www.site.com Google related word search – shows terms semantically related to a keyword ~term -term

•k)                   Business Perspectives

Google has the largest search distribution, the largest ad network, and by far the most efficient search ad auction. They have aggressively extended their brand and amazing search distribution network through partnerships with small web publishers, traditional media companies, portals like AOL, computer and other hardware manufacturers such as Dell, and popular web browsers such as Firefox and Opera.

I think Google’s biggest strength is also their biggest weakness. With some aspects of business they are exceptionally idealistic. While that may provide them an amazingly cheap marketing vehicle for spreading their messages and core beliefs it could also be part of what unravels Google.

As they throw out bits of their relevancy in an attempt to keep their algorithm hard to manipulate they create holes where competing search businesses can become more efficient.

In the real world there are celebrity endorsements. Google’s idealism associated with their hatred toward bought links and other things which act similarly to online celebrity endorsements may leave holes in their algorithms, business model, and business philosophy that allows a competitor to sneak in and grab a large segment of the market by factoring the celebrity endorsement factor into being part of the way that businesses are marketed.

•VIII.     Ask Search

Ask was originally created as Ask Jeeves, and was founded by Garrett Gruener and David Warthen in 1996 and launched in April of 1997. It was a natural query processing engine that used editors to match common search queries, and backfilled the search results via a meta search engine that searched other popular engines.

As the web scaled and other search technologies improved Ask Jeeves tried using other technologies, such as Direct Hit (which roughly based popularity on page views until it was spammed to death), and then in 2001 they acquired Teoma, which is the core search technology they still use today. In March of 2005 InterActive Corp. announced they were buying Ask Jeeves, and by March of 2006 they dumped Jeeves, changing the brand to Ask.

•a)                   On Page Content

For topics where there is a large community Ask is good at matching concepts and authoritative sources. Where those communities do not exist Ask relies a bit much on the on page content and is pretty susceptible to repetitive keyword dense search spam.

•b)                   Crawling

Ask is generally slower at crawling new pages and sites than the other major engines are. They also own Bloglines, which gives them incentive to quickly index popular blog content and other rapidly updated content channels.

•c)                   Query Processing

I believe Ask has a heavy bias toward topical authority sites independent of anchor text or on the page content. This has a large effect on the result set the provide for any query in that it creates a result set that is more conceptually and community oriented than keyword oriented.

•d)                   Link Reputation

Ask is focused on topical communities using a concept they call Subject-Specific PopularitySM. This means that if you are entering a saturated or hyper saturated field that Ask will generally be one of the slowest engines to rank your site since they will only trust it after many topical authorities have shown they trusted it by citing it. Due to their heavy bias toward topical communities, for generic search they seem to be far more biased on how many quality related citations you have than looking as much at anchor text. For queries where there is not much of a topical community their relevancy algorithms are nowhere near as sharp.

•e)                   Page vs Site

Pages on a well referenced trusted site tend to rank better than one would expect. For example, I saw some spammy press releases on a popular press release site ranking well for some generic SEO related queries. Presumably many companies link to some of their press release pages and this perhaps helps those types of sites be seen as community hubs.

•f)                    Site Age

Directly I do not believe it is much of a factor. Indirectly I believe it is important in that it usually takes some finite amount of time to become a site that is approved by your topical peers.

•g)                   Paid Search

Ask gets most of their paid search ads from Google AdWords. Some ad buyers in verticals where Ask users convert well may also want to buy ads directly from Ask. Ask will only place their internal ads above the Google AdWords ads if they feel the internal ads will bring in more revenue.

•h)                   Editorial

Ask heavily relies upon the topical communities and industry experts to in essence be the editors of their search results. They give an overview of their ExpertRank technology on their web search FAQ page. While they have such limited distribution that few people talk about their search spam policies they reference a customer feedback form on their editorial guidelines page.

•i)                    Social Aspects

Ask is a true underdog in the search space. While they offer Bloglines and many of the save a search personalization type features that many other search companies offer they do not have the critical mass of users that some of the other major search companies have.

•j)                    Ask SEO Tools

Ask search results show related search phrases in the right hand column. Due to the nature of their algorithms Ask is generally not good at offering link citation searches, but recently their Bloglines service has allowed you to look for blog citations by authority, date, or relevance.

•IX.     Technical Working of a Search Engine – Taking Google as example

•1)     Google Architecture Overview

 

In this section, we will give a high level overview of how the whole system works as pictured in Figure below. Further sections will discuss the applications and data structures not mentioned in this section. Most of Google is implemented in C or C++ for efficiency and can run in either Solaris or Linux.

 

 

In Google, the web crawling (downloading of web pages) is done by several distributed crawlers. There is a URLserver that sends lists of URLs to be fetched to the crawlers. The web pages that are fetched are then sent to the storeserver. The storeserver then compresses and stores the web pages into a repository. Every web page has an associated ID number called a docID which is assigned whenever a new URL is parsed out of a web page. The indexing function is performed by the indexer and the sorter. The indexer performs a number of functions. It reads the repository, uncompresses the documents, and parses them. Each document is converted into a set of word occurrences called hits. The hits record the word, position in document, an approximation of font size, and capitalization. The indexer distributes these hits into a set of “barrels”, creating a partially sorted forward index. The indexer performs another important function. It parses out all the links in every web page and stores important information about them in an anchors file. This file contains enough information to determine where each link points from and to, and the text of the link.

The URLresolver reads the anchors file and converts relative URLs into absolute URLs and in turn into docIDs. It puts the anchor text into the forward index, associated with the docID that the anchor points to. It also generates a database of links which are pairs of docIDs. The links database is used to compute PageRanks for all the documents.

The sorter takes the barrels, which are sorted by docID, and resorts them by wordID to generate the inverted index. This is done in place so that little temporary space is needed for this operation. The sorter also produces a list of wordIDs and offsets into the inverted index. A program called DumpLexicon takes this list together with the lexicon produced by the indexer and generates a new lexicon to be used by the searcher. The searcher is run by a web server and uses the lexicon built by DumpLexicon together with the inverted index and the PageRanks to answer queries.

 

•2)     Major Data Structures

 

Google’s data structures are optimized so that a large document collection can be crawled, indexed, and searched with little cost. Although, CPUs and bulk input output rates have improved dramatically over the years, a disk seek still requires about 10 ms to complete. Google is designed to avoid disk seeks whenever possible, and this has had a considerable influence on the design of the data structures.

•a)                   BigFiles

 

BigFiles are virtual files spanning multiple file systems and are addressable by 64 bit integers. The allocation among multiple file systems is handled automatically. The BigFiles package also handles allocation and deallocation of file descriptors, since the operating systems do not provide enough for our needs. BigFiles also support rudimentary compression options.

•b)                    Repository

  

The repository contains the full HTML of every web page. Each page is compressed using zlib. The choice of compression technique is a tradeoff between speed and compression ratio. We chose zlib’s speed over a significant improvement in compression offered by bzip. The compression rate of bzip was approximately 4 to 1 on the repository as compared to zlib’s 3 to 1 compression. In the repository, the documents are stored one after the other and are prefixed by docID, length, and URL as can be seen in Figure below. The repository requires no other data structures to be

 

 

 

used in order to access it. This helps with data consistency and makes development much easier; we can rebuild all the other data structures from only the repository and a file which lists crawler errors.

•c)                   Document Index

 

The document index keeps information about each document. It is a fixed width ISAM (Index sequential access mode) index, ordered by docID. The information stored in each entry includes the current document status, a pointer into the repository, a document checksum, and various statistics. If the document has been crawled, it also contains a pointer into a variable width file called docinfo which contains its URL and title. Otherwise the pointer points into the URLlist which contains just the URL. This design decision was driven by the desire to have a reasonably compact data structure, and the ability to fetch a record in one disk seek during a search

Additionally, there is a file which is used to convert URLs into docIDs. It is a list of URL checksums with their corresponding docIDs and is sorted by checksum. In order to find the docID of a particular URL, the URL’s checksum is computed and a binary search is performed on the checksums file to find its docID. URLs may be converted into docIDs in batch by doing a merge with this file. This is the technique the URLresolver uses to turn URLs into docIDs. This batch mode of update is crucial because otherwise we must perform one seek for every link which assuming one disk would take more than a month for our 322 million link dataset.

•d)                   Lexicon

 

The lexicon has several different forms. One important change from earlier systems is that the lexicon can fit in memory for a reasonable price. In the current implementation we can keep the lexicon in memory on a machine with 256 MB of main memory. The current lexicon contains 14 million words (though some rare words were not added to the lexicon). It is implemented in two parts — a list of the words (concatenated together but separated by nulls) and a hash table of pointers. For various functions, the list of words has some auxiliary information which is beyond the scope of this paper to explain fully.

•e)                   Hit Lists

A hit list corresponds to a list of occurrences of a particular word in a particular document including position, font, and capitalization information. Hit lists account for most of the space used in both the forward and the inverted indices. Because of this, it is important to represent them as efficiently as possible. We considered several alternatives for encoding position, font, and capitalization — simple encoding (a triple of integers), a compact encoding (a hand optimized allocation of bits), and Huffman coding. In the end we chose a hand optimized compact encoding since it required far less space than the simple encoding and far less bit manipulation than Huffman coding. The details of the hits are shown in Figure below.

 

 

Our compact encoding uses two bytes for every hit. There are two types of hits: fancy hits and plain hits. Fancy hits include hits occurring in a URL, title, anchor text, or meta tag. Plain hits include everything else. A plain hit consists of a capitalization bit, font size, and 12 bits of word position in a document (all positions higher than 4095 are labeled 4096). Font size is represented relative to the rest of the document using three bits (only 7 values are actually used because 111 is the flag that signals a fancy hit). A fancy hit consists of a capitalization bit, the font size set to 7 to indicate it is a fancy hit, 4 bits to encode the type of fancy hit, and 8 bits of position. For anchor hits, the 8 bits of position are split into 4 bits for position in anchor and 4 bits for a hash of the docID the anchor occurs in. This gives us some limited phrase searching as long as there are not that many anchors for a particular word. We expect to update the way that anchor hits are stored to allow for greater resolution in the position and docIDhash fields. We use font size relative to the rest of the document because when searching, you do not want to rank otherwise identical documents differently just because one of the documents is in a larger font.

 

The length of a hit list is stored before the hits themselves. To save space, the length of the hit list is combined with the wordID in the forward index and the docID in the inverted index. This limits it to 8 and 5 bits respectively (there are some tricks which allow 8 bits to be borrowed from the wordID). If the length is longer than would fit in that many bits, an escape code is used in those bits, and the next two bytes contain the actual length.

•f)                    Forward Index

 

The forward index is actually already partially sorted. It is stored in a number of barrels (we used 64). Each barrel holds a range of wordID’s. If a document contains words that fall into a particular barrel, the docID is recorded into the barrel, followed by a list of wordID’s with hitlists which correspond to those words. This scheme requires slightly more storage because of duplicated docIDs but the difference is very small for a reasonable number of buckets and saves considerable time and coding complexity in the final indexing phase done by the sorter. Furthermore, instead of storing actual wordID’s, we store each wordID as a relative difference from the minimum wordID that falls into the barrel the wordID is in. This way, we can use just 24 bits for the wordID’s in the unsorted barrels, leaving 8 bits for the hit list length.

•g)