Forum: Behavior Issues ]]> http://parchmenthill.websitetoolbox.com/post?id=3975855 Fri, 04 Dec 2009 16:47:30 GMT http://parchmenthill.websitetoolbox.com/post?id=3705553

Forum: General ]]> http://parchmenthill.websitetoolbox.com/post?id=3705553 Fri, 09 Oct 2009 02:54:24 GMT http://parchmenthill.websitetoolbox.com/post?id=3703588


I saw you in my log by chance. I run a spider that crawls the Gnutella (Limewire) network. Quite a bit different than what you're doing. What language is your crawler written in?



-Adam

Forum: General ]]> http://parchmenthill.websitetoolbox.com/post?id=3703588 Thu, 08 Oct 2009 05:49:31 GMT http://parchmenthill.websitetoolbox.com/post?id=3699687
I really loved this Message Board style. Where can I download this PHP script to use it in m own website?

Thank you,

Alexandre

Forum: General ]]> http://parchmenthill.websitetoolbox.com/post?id=3699687 Tue, 06 Oct 2009 12:55:47 GMT http://parchmenthill.websitetoolbox.com/post?id=3689237
In general, we consider a website not to be legitimate if an organization has multiple websites that would best be handled as one website.  In other words, where an organization is taking advantage of the low cost of a domain to help improve their search engine rankings.  The worst offenders are those that have identical wording on multiple domains as well.

Why is this so bad?  Let's say that there are two domains in the world that mention both 'widgets' and 'Acme'.  We typically have an easy time determining which website is run by the Acme Corporation and sells widgets, and which is the blog of a happy customer of theirs that happens to mention Acme widgets.  Then, someone comes along and creates 1,000 domains that all have the words 'widgets' and 'Acme' (perhaps as random words, perhaps mentioning Acme widgets for a seemingly valid reason).  We now have 1,002 domains to sort through (rather than 2), which takes up a lot more resources.

Plus, as we are analyzing words on the entire web and comparing it to individual domains, if we aren't careful, we might think that there is something 'wrong' with the real Acme widgets website, since we see that 99% of all websites that mention 'Acme' and 'widgets' also mention 'gizmo' (but the real Acme widgets website does not).  So we get a false view of what words belong where, and truly legitimate websites get hurt.

So to produce decent results, we have to hunt down lots of the duplicated domains.  Ideally, we'll have them turn up in search results when appropriate (e.g. someone searching for 'Acme', 'widgets', and 'gizmo', where no other domains match), but also mark them for exclusion when doing certain analyses of the web.

For those that have registered multiple domains for a single website (e.g. variations of spelling, an abbreviation of the company name, different TLDs, etc.), you really need to redirect to the 'proper' website.  In other words, you want browsers (and search engines) to know that you'd prefer for people to use one specific domain name.
                                          -Scott

Forum: Technical Details ]]> http://parchmenthill.websitetoolbox.com/post?id=3689237 Tue, 29 Sep 2009 16:55:56 GMT http://parchmenthill.websitetoolbox.com/post?id=3689211
This is the forum for discussing any technical details about the search engine.  This would be the place for SEO (Search Engine Optimization) questions and the like.  To start a new thread, just click the 'New Topic' button.
               -Scott

Forum: Technical Details ]]> http://parchmenthill.websitetoolbox.com/post?id=3689211 Tue, 29 Sep 2009 16:34:41 GMT http://parchmenthill.websitetoolbox.com/post?id=3674029


Just letting you know that your bot trawled my site yesterday. I was somewhat surprised to see it there, but that being said, it is most welcome.



Feel free to hit my domain as many times as you require.

Forum: General ]]> http://parchmenthill.websitetoolbox.com/post?id=3674029 Sat, 19 Sep 2009 14:53:50 GMT http://parchmenthill.websitetoolbox.com/post?id=3671254 Hello,

did you have an icon for your ParchBot ? Then we can use it in out stats ;-)

thx,
françois.

Forum: General ]]> http://parchmenthill.websitetoolbox.com/post?id=3671254 Thu, 17 Sep 2009 18:15:41 GMT http://parchmenthill.websitetoolbox.com/post?id=3669175
This forum is for giving us suggestions for our project.  Do you have a neat idea for how we could improve our searching?  An idea for how the searching could be used?  This is the place!
                                  -Scott

Forum: Suggestions ]]> http://parchmenthill.websitetoolbox.com/post?id=3669175 Wed, 16 Sep 2009 13:55:21 GMT http://parchmenthill.websitetoolbox.com/post?id=3667775
Our search engine is designed to get around a problem with most search engines -- it can sometimes be very difficult to find a website (as opposed to a webpage).  If you're looking for the website of a large company, it's usually easy to find.  But for smaller organizations (especially those with few incoming links, like restaurants), and those with common names, it can be next to impossible.

This involves using some similar techniques that are used by other search engines (e.g. having an index of words that appear in webpages).  One of the main ways that search engines rank websites is by their popularity -- the number of other websites linking to them.  But we can't do that, since a small website (that you are trying to find!) may have very few, if any, links pointing to it.

If you're looking for information (such as the year something occurred, or how to fix something), our search engine isn't likely to be very helpful.  But if you are looking for (or have) a website, domain, business, or organization, our search engine should come in handy.
                                          -Scott

Forum: General ]]> http://parchmenthill.websitetoolbox.com/post?id=3667775 Tue, 15 Sep 2009 17:15:16 GMT http://parchmenthill.websitetoolbox.com/post?id=3667747
This forum is used for discussions regarding how our search engine behaves.

We value your resources, so we want to make sure that we aren't causing any problems with your site (e.g. slowing things down for other visitors, costing you money, etc.).  Due to the methodology of our search engine, it is not likely to access many pages at your site, so it is unlikely to cause any problems.  But if it does, please do let us know.

Just click on 'New Topic' to start a new thread.
                               -Scott

Forum: Behavior Issues ]]> http://parchmenthill.websitetoolbox.com/post?id=3667747 Tue, 15 Sep 2009 17:01:51 GMT http://parchmenthill.websitetoolbox.com/post?id=3667743
This is the forum for discussing the robots.txt file and how we interact with it.  Robots.txt is a de-facto standard for letting search engines (and other bots, crawlers, etc.) know whether or not they are welcome at your site, and if so, what pages are off-limits.

Before downloading pages from a website, we check for a robots.txt file in the root directory of your domain (e.g. 'http://www.example.com/robots.txt').  If the file is not there, we assume the search engine is welcome.  If the file is there, we check to see what it says about what pages we may and may not download from your site.

Just click on 'New Topic' to start a new thread.
                                     -Scott

Forum: Robots.txt ]]> http://parchmenthill.websitetoolbox.com/post?id=3667743 Tue, 15 Sep 2009 16:58:41 GMT http://parchmenthill.websitetoolbox.com/post?id=3667734
These message boards are for anyone who wants to discuss the search engine that we are developing.  At this point, it is likely only going to be webmasters that are here, as the search engine isn't available for testing as of this writing.  But it is crawling!

Feel free to post any general questions or comments in the 'General' category.
                         -Scott

Forum: General ]]> http://parchmenthill.websitetoolbox.com/post?id=3667734 Tue, 15 Sep 2009 16:54:51 GMT