Insights into Project Testing


I think it’s important to address the issue of why things sometimes change in a project.  But before I start that discussion it’s important to know how the overall process works…

Before being released every project goes through the following processes – it is imaged (the original documents are scanned), defined/setup (where we figure out what should be keyed to create the index, how many fields and form types, etc), and then it is tested (to make sure everything is working in the tool, that the Field Helps are appropriately instructive and concise, etc).  When setting up and testing the project there is a test group consisting of 10+ people who review a good portion of the images and come to an agreement on how the majority of the images will be keyed, then we look for anomalies – but the issue with anomalies is that don’t always encounter all of them before we release the project to be keyed.

And this is where I want to focus.  There are times when situations arise while you are keying that we didn’t anticipate, and when you bring these to our attention we discuss it and come to a conclusion on how this image/situation should be keyed.  When this happens we will adjust the Field Help, help articles and post a message on the message boards to let you know about this situation so if you encounter it you will know how to key the record(s).  This is the main reason why instructions are adjusted.

Another reason we adjust or change the instructions is if we receive quite a few questions about how to key a certain field or a certain form type.   In this situation there are a few more factors in play regarding whether we will adjust the instructions but if needed we will adjust the instructions so they are more clear – based on feedback from you, the community of contributors. 

Something else to keep in mind when you run across something that you may not be sure how to key is to look at the instructions and see if there is something that mostly fits the record you are keying.  There are times when using your best judgment will come into play.  This can be illustrated by using the NSW Police Gazettes as an example.  If the event that occurs is listed as a “larceny” or “robbery” (but the only options for Event Type are Desertion, Assault,  Missing Person, Murder, Police Business, Theft, and Other) choose the option that best fits – Theft.   

So although we extensively test a project before it is released there are situations that arise where it makes sense to add to or adjust the instructions.  We realize that this can be frustrating, but we hope you can also see where these changes add to the quality of the indexes and how they improve the ease of keying.

Information and Links

Join the fray by commenting, tracking what others have to say, or linking to it from your blog.


Other Posts

Write a Comment

Take a moment to comment and tell us what you think. Some basic HTML is allowed for formatting.

Reader Comments

Thanks for this explanation.
Considering the huge amount of data that we are dealing with, I think you all deserve a great big BRAVO!

My biggest gripe, as an arbitrator, is people do not check what they key before they submit it. It only takes a few seconds to catch typo’s etc and can avoid a lot of problems.

Again, thanks for all your hard work, and for the opportunity to participate in such a worthy project.

The 10 people who ‘test’ the project – who exactly are they?

10 people at Ancestry or 10 experienced keyers?

I am happy to be involved in this project overall and as Ms Mason commented considereing the sheer volume overall I think you folks at adm do a good job. I have been keying for six or eight months. I’ve only done 8 or 9 thousand records because I am not very fast. To me it is interesting to note what it is that I am keying and in so doing find all kinds of interesting things about people and places. History comes to life.
How you set up these projects is interesting. I must say from the onset there have been some frustrating issues in every different type of document I’ve worked on. And, to this day, after reading numerous boards and all the available instructions I am still not sure about many things I am keying. The learning experience has been very hit and miss and I am sure many share my sentiments.
The comment about making set instructions concise and to the point is my point. They are too concise and never address half the issues the keyers have concerns about. It seems to me that the instructions should be more instructive and better thought out. This would help avoid the keyers having to post to boards and read many, many posts to maybe, possibly get some kind of answer. If you are going to grade key work then the playing field should be level…instructions could be a great deal more informative.
Again, thanks for your hard work and effort to always improve the project. I will remain involved from my humble perspective. John Nowlin

Claire,
The people testing the project are Ancestry employees who are very familiar with keying – many of them have years of experience and some have a little less experience. We also run instructions by others who aren’t as familiar with keying so we can get their perspective as this helps us make sure that the instructions are thorough.
I think the main difference between how we key and how many community members key is that we read everything before we key, then we do our best – we are willing to make judgment calls about how something should be keyed which can be a little uncomfortable for many keyers.

John, I appreciate your comments. I think we have improved a lot over the past year and a half and we are still striving to do better.

It is very useful to have the above insight into project testing; and an explanation for changes that might be experienced in instructions and field helps.

I don’t find changes in instructions all the threatening. It indicates that some notice is being taken of the comments on the blogs.

I also think that there is a different perspective between those keyers who are new to the system and those that have used it for sometime.

To me, one of the problems as a new keyer has been the apparent “inconsistences” in the various instructions. These tend to suggest to the newcomers that the instructions are not thorough, which is probably not what is intended. Just to take one (minor) example, the general Keying Standards advise not to enter periods (full stops) after initials whereas the field help states to “type as seen” if the name does not appear on the list of suggestions – which tends to suggest that periods should be added to initials. The field help already gives some exclusions like no parentheses; so if there is to be no periods after initials the field help could easily state “type as seen” but no parentheses and no periods after initials.

Judgement calls is another one of those difficulties for the “newcomer”. The impression given is that judgment calls should not be taken over names and places. The instructions are clear: type as seen, no converting “Thos” to “Thomas”, etc. The only leaway appears to be in interpreting letters, e.g. is that circle an O or an S, etc. Perhaps it needs to be made clearer what judgement calls the keyer should make, if they are different to my interpretation above.

Much of this these concerns are part of the “learning process” and later on these things will cease to be so troubling, but they are fairly troubling at the start.

Checking is one of those difficult areas: finding my own mistakes is not all that easy, but I do try; however, I find it much easier to find other people’s mistakes.

Your explanation is most useful. I have found that the people keying the NSW Police Gazettes are amazing helpful and prompt. There are a range of emerging issues as new items come to light, and Ive found that our community of collaborators always has a quick answer.

This project is bringing history to life and its a great way to share in building our historical database. Its a case of learning on the job and certainly the WAP team is to be congratulated for their processes and striving for improvement.

I think the whole archives project is fascinating. My only comment is the keying in of dates of birth which are often shown in a different way in England.

To be honest I also have problems with the instructions (mostly too vague and I end up rejecting a project that I might really want to try because I have so many questions) but sometimes I wonder if I (as a keyer) over think a question. I have found that reading and rereading the instructions have helped (more now than in the past) but they leave too many questions in their wake for the keyer. But they have gotten better than before…..keep it up

A little bit more….
Here is the perfect example…doing the NSW Police Gazettes (which are great) you say to key in all names does that include the arresting officer or just the victim and suspect? Why do I have to list the victims the same as the suspects as to crime…and when event location say “near” wagga wagga…do I put Wagga Wagga…I could just follow instruction without thought…but I do have thoughts and valid questions

I have the same issue as Michelle.
In the NSW Gazettes I am keying the ‘crim’ and the arresting officers (all the names). I am also keying the same crime for all named persons.
The suffix does not allow for the police officer’s title,thus if not overidden does not distinguish between the individuals. Mr Mrs etc is OK but what about Constable or Police Sergeant?

Wow! You ancestry.com folks have done so much excellent work on all of these projects. I am so grateful you have opened them up to members to help with the keying. I know I am exceedingly careful in my keying, but am as human as you are. I find no fault in any confusion — it could as easily be on my end as yours. I am ecstatic to be a part of these projects, to be invited in to them, to find you being transparent about the process, and see the progress.

My wondering? I love getting a statistical accuracy rate. Just my logical side and perfectionism squeaking out! Is it cumulative, however, from day one when I made more mistakes as I learned how to manage the forms to today when I hopefully have a better handle? Can we ever start the % over? Could we some how see our accuracy on different projects? That would give you better insight into your projects as well.

Thank you. Thank you. Thank you!

As a relatively new keyer, although no longer brand new, my personal frustration is that if I want to try keying a different project, I would first have to read through a lot of posts in the hope to find a general idea of the established rules for a given project. The posts aren’t really organized in a way to make it easy to find everything related to a specific project, either. This could take several hours, when I’d rather spend the time actually getting to keying, and at the end, I’m still not confident that I actually have located all of the material.

I think it’s completely reasonable to change the rules. What would be helpful to me as a keyer, and actually as an arbitrator, would be to have a complete list of all rules and decisions that have been made for a specific project, available to just sit down and read through, and to print out as reference.

Then, as changes occur, somehow flag these updates, so we know something has changed – maybe track each time a person downloads a specific project, and alert them if anything has changed since the last time they keyed that project. Or, a slightly more low-tech solution, just provide a list of changes for each project in a central place that’s easy to access, and list the decisions in order, starting with most recent. Then we can quickly get up to speed.

I know this approach would be helpful for me, and I am a Certified Usability Analyst, so I can say with some level of authority that I think it might help others as well 🙂

Reply to Christine #12.

You asked “Is it cumulative, however, from day one when I made more mistakes as I learned how to manage the forms to today when I hopefully have a better handle?”

No, Accuracy is only for the last 90 days. So as you continue to key the “older” or first records you entered will drop off.

For example, I have been keying/arbitrating for the past 14 months, however, only the last 3 months go towards my accuracy. (Which is good because when I first started my numbers wern’t so good.)

I agree with Theresa. Wading thru the bulletin boards and trying to determine which are the “official” rule changes and which are other keyers and arbitrators sharing what they think should be the way to do things can be frustrating, time-consuming and just plain difficult.

I’d prefer the project popup when I open a project to highlight the changes to a project’s instructions right up front, with a date of the change as well. Some of these projects have been active for months, and it’s easy to just keep doing things the way we’ve been doing them from the start, without realizing a rule may have changed.

Amen, Wayne! I tend to stick to certain projects. If any rule changes have been made, I would never know because I have given up going into the bulletin boards. They are unorganized and more confusing than anything I have ever seen. If a highlighted note were to show up on the instruction pop-up, I would certainly pay attention!

I just read the explanation and think it’s completely wrong to put what fits best. We should put down what is there. I am working on the Lubeck census and at the moment I have some pages where none of the cities are in the database. Same for a lot of the occupations. I have been transcribing for nearly 10 years for different project now, I am pretty good at reading. I am also wondering if my missing cities will ever be put into the database, it’s a real hazzle to have to type them every time for pages and pages.
Ines

I’ve always had a problem about rating our keying performance or accuracy, since the project information changes on a daily basis. My experience has been that the instructions are vague from the get go. When I’ve had questions about how to key certain information, I’ve posted them right away, only to find the next morning 5 to 6 different ways to key the same information. Nothing is more frustrating than finding that your rating has dropped from 98% to 80% overnight, just because the instructions were vague to begin with and the “advice” we receive from others is oftentimes contradictory. That’s my two-cents regarding rating accuracy. Do away with it please. We’re all just trying to help out.

Thank you all for your great comments! I appreciate hearing about your experiences as it gives us specific areas to work on.

I have not been keying long yet but find I stick to the easy projects. The accuracy listing does not give me any hint what I might be doing wrong. I would appreciate some kind of feedback – if not specifically for my keying at least for the project overall. For instance I was showing a record as duplicate and found out I should not do that if it winds up in a different set (which it often does. The last record of one set is the same as the first record of another set. Another question is the typing sometimes does not match the signature. The signature is often very clear as to the spelling on the name and the typing often is obviously wrong. I have keyed in records belonging to my relatives with the name spelled wrong because that is how the record is and that is extemely frustrating. I find it difficult to work through this site and have yet to figure out how to contact someone for specific questions that do not have to do with the records I am keying. I love doing this keying but sometimes it is frustrating and I don’t have many hours per week to work on it.

Anna, another important aspect of this amazing project should be verifying the data once it is uploaded onto the Ancestry.com website. For example, some of the Nebraska census records have “precinct” listed as many as 3 times as the place the person was living at the time of the census – of course, it is not a place at all, but merely a term of art describing the area of a larger census tract. Similarly, I just looked at the Gretna Green registers, one of my favorite projects. In one record, the groom signed his name “Beard” but the record is listed as Baird, in both the scrap of paper, and the beautifully-written journal (which duplicates the records on the scraps of paper and is thus a secondary source.) I looked at only one other record, and in it the scrap of paper clearly shows a date of 1830, while the journal records it as 1831. The person who created the journal entries may have had a reason to suspect the date was incorrect, but shouldn’t the primary record, the scrap of paper, rule? Was it not understood when the project was set up that the journal duplicated the information in the scraps of paper? I realize this is much like complaining that census enumerators had bad handwriting – much too late to do anything about it! – but I think the description of the data sets should clearly state their respective worth as historical records. These are quibbles, however, in light of the huge contribution so many are making to preserve this information.

I have to agree with Diann. I double check my work, when the writing is difficult to read I try to check elsewhere in the document to verify spelling and dates. Yet I frequently come up with errors dropping my percentages. Having some sort of feedback would help to know exactly what areas I need work on. If it’s typos or just not understanding the directions.

Anna, I have come up with another problem with data that have been released – names with one or more question marks. An example is
“P??n”. I think this means two keyers entered the same name with question marks in the same place, so the arbitrator was not alerted to look at it. Would names with question marks even come up in a search? It seems to me that it might be better to have a field for a keyer’s best guess as to the name – in this case, perhaps “Penn,” so that the arbitrator would at least have something to look at. Along the same lines, an earlier commenter said she was forced to enter some of her family’s names incorrectly, because the enumerator (or immigration clerk) entered the names with bad spellings. I have encountered many situations in which I thought a name was one thing, but was required to key something else – usually because of handwriting rather than a variant of a name. It would have been very helpful to have a field to enter what I thought the name was – the way those of us who send in a correction can. Would that not improve the data?

(1) I am perlexed when following my Dean Family Tree – I came to a point where through the Webster tree line where I was presented with a Webster Esquire (unsubstantiated as far as I can research) many family tress have crossed this pathway into an amazing history of continuing Kings and Queens of Wales back to forever and a day.

What to do with this amazing point of no return in my family tree???

I can see the distinct possibly we all have genetic links back to these people. Current science in DNA tests have proved this has happened.

Happily I feel I was led into these ancestors and amazed at the amassed cellular history that is available.

(2) It confuses me that people still living, are linking into the old ancestral names in a religious or maybe egoistic vernacular. Why????

Would appreciate your comments.

helenatlakes

Can you please explain why my path has crossed this

It is very frustrating when, having adjusted the image (by size and/or position on screen) so that we can see the required data, it shifts its position on its own when a record is keyed. Is it not possible to freeze the image window while the record is completed? (It’s something I do every day when copying into a Word document from a scanned image, so it surely shouldn’t be too difficult.)