Posted by on May 13, 2011 in Ancestry.com Site

Last October we launched Ancestry Labs to test a few new ideas, and we’d like to thank all those members who contributed a lot of great feedback and discussion around these.  Today we’re excited to announce the introduction of one of the ideas, Web Search, into the main Ancestry.com search.

Why are we launching Web Search?

We’ve heard from many members that although Ancestry.com has the broadest collections of historical records available, it certainly isn’t completely comprehensive. Every day, digital records are being published on sites across the web, many of which are free to access.

These sites can be a great resource in helping break through brick walls, however, it can be hard to know where to find sites that are relevant to your ancestors, and it also takes time to work out the best way to search them once you do manage to track them down.

To help solve this, we are launching a new feature that searches select genealogy websites and brings back any matching results we find, along with a link to the site to enable you to go straight to the original record. Where relevant, we will include these results into your main search results. We will also list each collection we have within our card catalog, which will allow you to search those collections directly from within Ancestry.com.

Principles we will follow

As we’ve been developing Web Search, we’ve spoken with many members within the genealogical community to try to make sure we approach this in the right way. We recognize that this content is really important to every single site owner, and we believe it’s important we respect the wishes of these owners regarding links to their content.

From these discussions, we have developed some principles on how this service will work:

  • Free access to Web Records – Users do not have to subscribe or register with Ancestry.com to view these records.
  • We will always strive to follow web standards for web crawling permissions.  For example, some websites have a robots.txt file that instructs search engines (like Google) to not crawl the site, or to only crawl certain areas.
  • Proper attribution of Web Records to content publishers – we will link prominently to the original site within the search experience.
  • We have in place processes to remove content from the index if a website owner requests us to do that and we will publish how to contact our team to do this.  Website owners can also contact us to ask questions or to request their site be indexed – see this page to learn how to contact us: http://www.ancestry.com/websearch.
  • Ancestry.com users will be able to save key information to their trees but it will list the website as the source and will have an easy way to link back to the original site.

How does it work?

When you do a search on Ancestry.com, if we find a relevant match in a record on a site we have indexed, we will include that match in your search results.

You can view the essential information in the search results list, click on a link to visit the website, or click on the information to view a little more and save it to your tree.  You don’t have to subscribe or even have a guest account with Ancestry.com to find and view this information or to get to the source website.  Of course, if you want to save it to your online tree, you will need to login or create a free account so we know where to save the information.

To designate which of these records are from Ancestry.com and which are found via web search, we are prefacing the source information with “Web:” and then the name of the collection.  For example: “Web: Rootsweb Obituary Index”. There is also an icon next to the name indicating this is a web record.

You will also see a link under the title that will go straight to the website.  If you are using new search, you will also find a special icon next to the title.

When you click on the name of the collection, you will get to this page:

Web Search - Index Page

This page only contains the essential information needed to find the information and to make sure it is the person for whom you are searching.

In the same way you should always check the image when you look at an index, make sure you go to the web site to see what other information is there.  You will usually find additional information – reference and publication information, grave site locations, ways to order the original record, notes, and sometimes even images.

Going to the original website

The first time you click on a link to go to an external website, you will see a message that looks like this:

We want to make sure users know a few things about this experience.  First of all, the content is not on Ancestry.com and the site will open in a new tab, or a new window (depending on your browser.)  Secondly, when you get to the website, you may have to search for the record.  Some sites have static URLs so we can help you get straight to the record.   Other sites dynamically generate the page for the record so you have to search when you get there.

If you don’t want to see this message in the future, just check the box in the lower right hand portion of the screen “Don’t show this message again.”

Can I search just one collection of web records?

Yes, you can do this.  There are a few ways to search just one set of records from a website.

If you are looking at a single record on Ancestry.com, just click the title of the collection and it will take you to a search page for that collection.

Web Search - Click on Title

This will take you to a search page that will look something like this:

Web Search - Search one collection

You can also find the individual search forms by going to the card catalog and searching on the name of the collection.  If you don’t know, just search on the word “web” in the title since we will include the word “web” to denote they come from an external source and not from Ancestry.com.

How fast will we grow the Web Search service?

We plan to add sites to our index gradually so we can learn about how useful they are to users. Today, we are launching Web Search with select websites but expect to add many more websites to help our users find more information about their ancestors.

Go try it out

The best way to understand this is to try it out.  Since we are starting out small, you may not see these results in your search results for your ancestors, yet.  (If you do, please let us know if it was helpful.)

You can see it in action by doing a search for Louise M Chrisman, died in Indiana, USA or click here to see the search results.

In the example above, the first result is for Louise in “Web: Allen County, Indiana Deaths 1870-1920.”

We would like your feedback

We hope this answers many of your questions.  We have appreciated all of the feedback we have gotten so far and look forward to continuing to learn from our users, from web publishers, and from others in the community.

We have a page with general information on this service at www.ancestry.com/websearch.  On this page, you will find a summary of some of this information as well as ways to contact us.

If you own a website or publish content, we have a special message board where you can ask questions and share your suggestions with us at http://boards.ancestry.com/content-publishers-feedback/mb.ashx.

If you publish family history records on the web, you can ask your site to be searched, or request your site not be searched, by emailing us at websearch@ancestry.com.

We also invite any of our users to share their thoughts and suggestions with us at the same email address – websearch@ancestry.com.

Happy Searching!

29 Comments

MikeF 

The Internet Biographical Collection rides again!

Some questions Brian since they are not listed in your principles to follow.

1) Are going to only link to a website and not store its content on Acom servers other than perhaps a brief nutshell? While you speak of an index, this point needs to be explicit.

2) Will users clicking on such links be taken from the Acom site to that one (whether in a new tab or whatever), rather than being stuck in an Acom frame?

3) Are you going to allow non-profit volunteer websites to opt-out globally if they wish? For example, if an entire state’s USGenweb project wants to opt-out, rather than forcing opt-outs on a county-by-county basis?

4) Will updating of the index also reflect later implementations of the robots.txt exclusion protocol by a site that initially did not have one? That is you add a site which then de facto opts out by adding the robots.txt protocol and your indexing crawler sees this and removes the previous entry?

May 13, 2011 at 11:38 am
Deb H 

Will we be able to limit searches solely to records that are on Ancestry.com? As it is, an Ancestry search brings up way too many records. I’m not sure I want more miscellaneous web records cluttering up the already over-populated search results currently generated by Ancestry.

May 13, 2011 at 11:57 am
BCarol 

Familysearch.org has a similar feature that will take you to websites that might be relevant. I’ve always liked that feature. I’ll give this a whirl and let you know what I think.

May 13, 2011 at 2:20 pm
Brian Edwards 

Mikef:

To get an idea of what is being indexed, please take a look at the examples in the post and try it out. We are aiming to index enough to make it findable but still leave value on the individual websites so that people have an incentive to go to the website.

The site opens in a new tab with no frame.

The opt-out needs to come from the owner/publisher of the content. If someone has rights to decline to be indexed for a group of content, that’s great. You can email us directly if you have questions about this.

We haven’t set a re-crawling schedule so, if a publisher finds their content indexed and wants it removed, it’s best to just let us know and we will remove the index right away.

We really want to make sure it is a win-win scenario for everyone. Please feel free to ask further questions on the publisher message board at http://boards.ancestry.com/content-publishers-feedback/mb.ashx.

May 13, 2011 at 2:50 pm
Brian Edwards 

Deb H: We don’t have the ability to filter these out at this point but we will monitor feedback and look at that if it becomes an issue for people.

BCarol: Thanks for taking a look at it. I am looking forward to hearing your thoughts on it.

May 13, 2011 at 2:53 pm
BEE 

Two questions come to my computer/electronic ignorant “senior” brain, although I think I know the answer to my first question.
Is this feature only available with “new search”?
I the information on my “private” trees still private?

May 13, 2011 at 3:28 pm
long time user 

#6 Bee:

I tried the “CLICK HERE” on the above text in the blog, and yes, it puts you into new search and then, after multiple clicks, you get the option to enter the name again in the Allen County database. That’s as far as I went

Getting back will put you in new search so you will have to go through the necessary clicks to get into old search. I don’t think it has the flexibility of Google, outside of Ancestry.

Google would be my choice for a search as it is possible to change the search parameters to anything you can think of. The oddball words you can use with Google give you more chances for finding an obscure document. But that will require you to put on your thinking cap, which I wear all the time anyway!

I don’t know the answer to your second question for sure, but you know even if your tree is private and not indexed, the member connect system will notify someone searching the same name (or even a similar name) that “so-and-so” has added a record to their tree. My guess is yes, that would happen even if you add/merge a source outside of Ancestry to your tree.

The only way to stay absolutely private is not to put a tree on Ancestry. You have to make that decision for yourself.

May 13, 2011 at 6:47 pm
Gentry Davies 

Bee:

These web search results will appear in old search as well as new search – try searching for “Louise M Chrisman,” the sixth and seventh results are from web records collections.

Also, this won’t have any effect on the privacy of your tree, web search is for pointing people to web records external to Ancestry.

Long time user:

The first time you open a web record, you will see a warning that you are leaving Ancestry (which you can then opt to not have show up in the future). However, when you see a search result in global search you can get to the source website with one click by following the “Go to website” link.

I agree that Google is a great tool for finding records on the web about your ancestors, and web search certainly doesn’t seek to replace that. It does allow a lot of flexibility for doing proximity searches and searching text you don’t generally find indexed in records. However, records that can be searched through Ancestry.com (including the web records in this project) can be hinted, easily merged to your tree, and searched using Ancestry’s filters.

Also, Google usually doesn’t index records that are found through a search form, which means that many records will be accessible through web search that wouldn’t ever appear in Google results.

Thanks for the great questions and comments!

May 13, 2011 at 10:01 pm
Download keylogger 

This feature on this website, will make it more user friendly and easy accessibility to search items. Thanks for the info.

May 14, 2011 at 6:49 am
Kirk Sellman 

I don’t see that http://www.findagrave.com is included in this search. Will that be added soon?

May 14, 2011 at 10:15 am
Tammy 

There appears to be no way to do a general web search, just a search of specific web databases. One must first locate a web database in the card catalog (good luck!), then search from there. What were you thinking? This is extremely limiting. If that’s the way it must be done, it would make more sense for the researcher to go to the original website and search from there.

May 14, 2011 at 11:04 am
Andy Hatchett 

Tammy Re: #11

Finding the web databases is easy.
1) Go to Card catalog
2) Type “web” in title box (without quotes)
3) Hit Return

All web databases will be displayed (only three at present).

May 14, 2011 at 11:13 am
Brian Edwards 

Hi Tammy,
These results will show up in general search results. The method I mentioned from card catalog is only if you want to search a specific collection. The primary way we expect people to use these is from general and category searches.

May 14, 2011 at 9:41 pm
Brian Edwards 

Hi Kirk,

Thanks for the question. In general, I don’t expect that you will see us publish a schedule for when specific sites will be indexed for a couple of reasons.

(1) Since this is a new service, making sure we provide a quality service that really adds value to our users and, at the same time, is respectful to publishers is more important than the speed of indexing new sites.

(2) At any time, a publisher could request us to not index their site.

When we feel we have the right experience for our users and balance for publishers, I think you will see the speed of indexing new sites increase substantially.

May 14, 2011 at 9:47 pm
Patsy West 

I’m a subscriber.

I have a couple of suggestions:

1. Are you paying people when you use their materials on the web which they offer for free? Find-A-Grave is a perfect example. You require a subscription to use your site, so you might consider making the websearch free to everyone and it might encourage new subscribers.

2. The ancestry site search function has improved, but it is still terrible. I would estimate that no matter how much accurate information I enter for a search, about 80% of the search results are just laughable. I get so disgusted with search results that are 100 years past the time of the death I have entered for the person I’m searching. I also can’t find ways to filter out locations. I tell your search engine the birth and death locations and add where locations lived. I still get results from everywhere, including the UK. I think your resources would be better spent fixing your own search engine. Finding a database is also a HUGE pain, particularly when I have no idea how a database is indexed – by state, by county, etc.

May 14, 2011 at 10:09 pm
Brian Edwards 

Hi Patsy,

The web search service is entirely free.

In terms of the search, we are certainly continuing to learn and improve. I have spent that last several days here at NGS (The National Genealogical Society) conference meeting with customers and experts, demonstrating our products and new search capabilities, and answering questions. From my conversations with people here, I heard both praise and criticisms. We will continue to learn from our customers and continue to improve the search experience.

May 14, 2011 at 10:44 pm
Ancestry.com Adds Web Search 

[...] make sure you go to the web site to see what other information is there,” advises Ancestry.com in its announcement. “You will usually find additional [...]

May 15, 2011 at 12:02 am
May 15, 2011 at 11:17 am
Pat Haley 

I don’t like your new format and prefer the old one. How do I get back to it?
Pat

May 15, 2011 at 4:43 pm
Carol A. H. 

#19 Pat:

I’m assuming you dont’t like the NEW search and want to go back to the OLD search.

Click “Search” from the main menu bar.
On the far, far right near the top, in a teeny tiny font, will be “Go to Old/New Search.”
Click that and you will change to the other search mode.

May 15, 2011 at 5:38 pm
Ken Hinds 

#15 Patsy:

Re your second issue:

The reason you get so many ridiculous results to your searches is that it is not matching on everything you enter.

On the search page, click “Advanced Search”. This gives you a new option: “Match all terms exactly”. Click this option and try your search again.

Without this option, if you search for Wilson Rigsby born in Kentucky, you get:

a: Everyone named Wilson Rigsby, regardless of birth place
b: Everyone named Wilson born in Kentucky, regardless of last name
c: Everyone named Rigsby born in Kentucky, regardless of first name

With the option, you get only Wilson Rigsby born in Kentucky.

Why “match all exactly” is not the default I simply cannot comprehend.

May 16, 2011 at 5:59 am
Pat Haley 

THANK YOU Carol A. H.!!!! No, I don’t like the new format. I was “raised” with the old one so I prefer it.

May 16, 2011 at 4:03 pm
download keylogger 

It is always nice to read your post as having unique and informative content to read. Can you please tell this will really helpful for showing fast indexing of new site?

May 17, 2011 at 4:27 am
Tony Cousins 

We have our own family tree web site and publish at least twice a year. When we do the process regenerates all the HTML pages which means that any indexing can not be set in stone. How would Ancestry handle changes like this – if the site isn’t reindexed after a major update then people would be directed to the wrong pages. That would be just another snafu for Ancestry and the subscribers.

TonyC

May 18, 2011 at 8:06 am
Andy Hatchett 

Tony Re: #24

From what I have seen, if a site is using non-static pages then the link just goes to the home page of the site and then the user has to use that site’s search engine to find the person.

If you use the example person and do a search you will see that is what happens when you go to the Allen County website.

Brian made a post about it on one of the boards but I don’t remember which one.

May 20, 2011 at 8:42 am
Tony Cousins 

Thanks Andy
– we use Gedhtree and like I said it regenerates the HTML pages and we don’t have a search function on our home page.

TonyC

May 20, 2011 at 1:27 pm
Andy Hatchett 

Tony Re: #26

I’m curious, with no search function on the home page, how does one go about finding people once they are on the site?

May 20, 2011 at 8:03 pm
Gentry Davies 

Tony and Andy,

This is an important issue we’re aware of and will try to deal with as best we can. I think they are actually two separate issues.

Andy, you alluded to the first one which is “dynamic” URL’s that are generated when the main way to access a record is via a search engine. Allen County Deaths is a good example of this – everything is session based, the URL doesn’t change from search to search, so there is no way for us to send people to the record directly. We realize that sending someone to a search page is not the ideal user experience, but it’s the best we can do for now and it does make data searchable that Google doesn’t index. In the case that each record has a static/permanent URL we’ll of course try to send people to that page rather than the search form.

Tony the other issue which is what I assume you’re talking about, which is more about site reorganization, right? We’re working on ways to monitor whether links are still good so we can fix them as necessary. In my experience the sites that change the most often are smaller (less than 100,000 records) and html based, while the larger search engine based datasets are somewhat more constant, although this is definitely a generalization.

Great questions, thanks!

May 23, 2011 at 9:14 am
Tanya 

For the problems of duplicates: I go to the list of people and look for duplicates. Click on each name and see if which one you wish to keep. Go to quick edit and add a keyboard symbol (I use *) before or after the first name.
This way I know which one to keep and can delete the duplicates. Remove the (*) when finished with your tree.
I have problems getting duplicate census dates on an ancestor even when I don’t check the box on hints. I look at the profile and have the same date 2 or more times with the same source.

May 26, 2011 at 8:37 pm