How to evaluate machine translation: A review of automated and human metrics - Eirini Chatzikoumi
This article presents the most up-to-date, influential automated, semiautomated and human metrics used to evaluate the quality of machine translation (MT) output and provides the necessary background for MT evaluation projects. Evaluation is, as repeatedly admitted, highly relevant for the improvement of MT. This article is divided into three parts: the first one is dedicated to automated metrics; the second, to human metrics; and the last, to the challenges posed by neural machine translation (NMT) regarding its evaluation. The first part includes reference translation–based metrics; confidence or quality estimation (QE) metrics, which are used as alternatives for quality assessment; and diagnostic evaluation based on linguistic checkpoints. Human evaluation metrics are classified according to the criterion of whether human judges directly express a so-called subjective evaluation judgment, such as ‘good’ or ‘better than’, or not, as is the case in error classification. The former methods are based on directly expressed judgment (DEJ); therefore, they are called ‘DEJ-based evaluation methods’, while the latter are called ‘non-DEJ-based evaluation methods’. In the DEJ-based evaluation section, tasks such as fluency and adequacy annotation, ranking and direct assessment (DA) are presented, whereas in the non-DEJ-based evaluation section, tasks such as error classification and postediting are detailed, with definitions and guidelines, thus rendering this article a useful guide for evaluation projects. Following the detailed presentation of the previously mentioned metrics, the specificities of NMT are set forth along with suggestions for its evaluation, according to the latest studies. As human translators are the most adequate judges of the quality of a translation, emphasis is placed on the human metrics seen from a translator-judge perspective to provide useful methodology tools for interdisciplinary research groups that evaluate MT systems.
This weeks homework was about determining ones’ personality type, which would also determine ones’ work habits.
* I’ve also compiled, in another post, my analysis about the test, how it works and the definitions here.
What personality type are you?
My personality type is ISTJ. This means I’m a bit of an introverted type ( I ) person who perceives and makes decisions by relying on information presented (S)(T) and prefers to lead a lifestyle that is more structured and planned (J).
People with ISTJ type personality are called Inspectors and are defined as people who:
have a keen sense to right and wrong
are devoted (especially to their work and interests) and will see it through, even at the expense of their own health
perform better on the basis of doing things step-by-step
are keen learners and learn best by practical method
are heartfelt expressing people but do so with large amount of energy expenditure (can be emotionally draining)
appear to be hard to approach
keep to themselves but can also open up if asked i.e. feelings
prefer to be honest
are somewhat perfectionists.
People of ISTJ personalities suit the roles of being someone who leads or is a role model to others.
Who else (famous people) shares this personality type with you?
Here are some famous people (that I have heard of) whom share the same personality type as me.
W.Disney, Princess Diana, George W Bush, Albert Einstein, Harrison Ford and Lewis Carol.
I also happened to find a character of W.Disney’s creations that shared the same personality type as me.
(Sebastian from the Little Mermaid)
Did the results of this test surprise you?
I would say yes, I was surprised as many of those attributes do relate or define me.
For starters, I can say I am a introverted type person since I prefer to work independently or amongst a few people. I’m not a very big socializing type person and have some social anxiety issues.
Then there is my perfectionism and self-confidence problem where I can’t trust my own instincts and am meticulous but am a person who is devoted and aims to get work done.
I read up on how the personality type ‘learning style’ and this was very accurate!
ISTJ learns best through practical ways and when material is presented in many ways through examples and is set in orderly fashion. However, ISTJ learners do not adhere to learning well when material is overly theoretical (and I for one suck at theoretical based areas) and lacks practical application.
So, for me to achieve, I learn best through practical methods but in instructed order.
Its almost accurate that it’s scary. An example of this is shown with someone famous who shares the same personality type. Princess Diana! I share my birthday with hers.
Do you agree with your personality assessment (why/why not)?
I would say that I do agree. After All, its near accuracy to my personality was surprisingly uncanny.
How do you think your personality would fit into the workplace (would your work better in a small studio or a large global agency? would be a team player or a leader?)?
This particular question, in all honesty, makes me doubt myself.
ISTJ personalities are found in occupations that involve control over production, require responsibility and being efficient working. They have been linked to careers where success takes place such as the military, police force, engineers, lawyers and surgeons. They also make good personality types for higher ranking positions such as management, leading and head of a department.
Now, letting my self-confidence speak here. As for as the personality type being accurate at defining my persona, I feel like ISTJ careers contrast. To elaborate, I’m someone who’d prefer to follow (so I’d want to be a team player) rather than lead when it comes to the industry area as I lack confidence to be a lead or even manager.
I do see myself as a role model though and have a tendency to help and guide though.
I know for a fact that I stress out quite easily. So if I’m loaded with a huge responsibility, I’ll take any little negative thing or problem, in relation to the responsibility, overly seriously. Also, having social anxiety, I would prefer to work, and I know I would work better in, in a place that is small, such as a small studio.
For me and my personality type in a workplace but based on the facts I have jotted down, I would be someone who would fit into the workplace as an team player or assistant mananger(?) as I’m a person who supports. I’m not a big socialiser and I work best in an environment with few people but given I am a devoted type person, I would be quite reliable.
I saw a post going around a while back rating the various testing sites. It gave keys2cognition five stars, 16 personalities three, and human metrics one. I’m not familiar with the other two sites it named and rated so I can’t speak for them.
I’ve used keys2cognition with a 33% success rate, 16 personalities with a 50-60% success rate, and human metrics with a 90-95% success rate.
I’m not sure why other people have had difficulty, but I have to recommend the test that I’ve had extraordinary success with: human metrics.
INFJs are gentle, caring, complex and highly intuitive individuals. Artistic and creative, they live in a world of hidden meanings and possibilities. Only one percent of the population has an INFJ Personality Type, making it the most rare of all the types.
INFJs place great importance on havings things orderly and systematic in their outer world. They put a lot of energy into identifying the best system for getting things done, and constantly define and re-define the priorities in their lives. On the other hand, INFJs operate within themselves on an intuitive basis which is entirely spontaneous. They know things intuitively, without being able to pinpoint why, and without detailed knowledge of the subject at hand. They are usually right, and they usually know it. Consequently, INFJs put a tremendous amount of faith into their instincts and intuitions. This is something of a conflict between the inner and outer worlds, and may result in the INFJ not being as organized as other Judging types tend to be. Or we may see some signs of disarray in an otherwise orderly tendency, such as a consistently messy desk.
INFJs have uncanny insight into people and situations. They get "feelings" about things and intuitively understand them. As an extreme example, some INFJs report experiences of a psychic nature, such as getting strong feelings about there being a problem with a loved one, and discovering later that they were in a car accident. This is the sort of thing that other types may scorn and scoff at, and the INFJ themself does not really understand their intuition at a level which can be verbalized. Consequently, most INFJs are protective of their inner selves, sharing only what they choose to share when they choose to share it. They are deep, complex individuals, who are quite private and typically difficult to understand. INFJs hold back part of themselves, and can be secretive.
But the INFJ is as genuinely warm as they are complex. INFJs hold a special place in the heart of people who they are close to, who are able to see their special gifts and depth of caring. INFJs are concerned for people's feelings, and try to be gentle to avoid hurting anyone. They are very sensitive to conflict, and cannot tolerate it very well. Situations which are charged with conflict may drive the normally peaceful INFJ into a state of agitation or charged anger. They may tend to internalize conflict into their bodies, and experience health problems when under a lot of stress.
Because the INFJ has such strong intuitive capabilities, they trust their own instincts above all else. This may result in an INFJ stubborness and tendency to ignore other people's opinions. They believe that they're right. On the other hand, INFJ is a perfectionist who doubts that they are living up to their full potential. INFJs are rarely at complete peace with themselves - there's always something else they should be doing to improve themselves and the world around them. They believe in constant growth, and don't often take time to revel in their accomplishments. They have strong value systems, and need to live their lives in accordance with what they feel is right. In deference to the Feeling aspect of their personalities, INFJs are in some ways gentle and easy going. Conversely, they have very high expectations of themselves, and frequently of their families. They don't believe in compromising their ideals.
INFJ is a natural nurturer; patient, devoted and protective. They make loving parents and usually have strong bonds with their offspring. They have high expectations of their children, and push them to be the best that they can be. This can sometimes manifest itself in the INFJ being hard-nosed and stubborn. But generally, children of an INFJ get devoted and sincere parental guidance, combined with deep caring.
In the workplace, the INFJ usually shows up in areas where they can be creative and somewhat independent. They have a natural affinity for art, and many excel in the sciences, where they make use of their intuition. INFJs can also be found in service-oriented professions. They are not good at dealing with minutia or very detailed tasks. The INFJ will either avoid such things, or else go to the other extreme and become enveloped in the details to the extent that they can no longer see the big picture. An INFJ who has gone the route of becoming meticulous about details may be highly critical of other individuals who are not.
The INFJ individual is gifted in ways that other types are not. Life is not necessarily easy for the INFJ, but they are capable of great depth of feeling and personal achievement.
So my friend finally persuaded me to take a personality test, and tbh as an INTJ ((introversion, intuition, thinking, judgement) I don't feel too great knowing only 2% of the population share my traits... What type are you guys?
Test: http://www.humanmetrics.com/cgi-win/JTypes2.asp
