• News
  • Spirituality
    • Dream Interpretation
    • Angel Numbers
    • Tarot
    • Prayers
    • Spells
  • Health
  • Science
  • Celebs
  • Betting

Scientists Tested Natural Language Models To Predict Human Language Judgments

30Shares
29.6KViews

Natural language models may test computational assumptions about how people understand language. A group of scientists from Columbia University in New York, coordinated by Tal Golan and Matthew Siegelman, assessed the model human consistency of several language models using a unique experimental approach: problematic sentence pairs. Two language models differ regarding which sentence is more likely to appear in the real test for each contentious sentence pair. Taking into account nine language models (including n-gram, recurrent neural networks, and transformer models),

The researchers generated hundreds of such contentious sentence pairings by picking phrases from a corpus or synthetically optimizing sentence pairs to be highly controversial and controversial. Human volunteers subsequently made evaluations indicating which of the two terms was more plausible for each couple. Controversial phrase pairings successfully highlighted model flaws and found models most closely matched with human assessments. GPT-2 was the most human-consistent model studied. However, testing showed severe deficiencies in its alignment with human perception.

Natural Language Models

These researchers put nine models from three different classes to the test: n-gram models, recurrent neural networks, and transformers. The Natural Language Toolkit's open source code was used to train the n-gram models. The recurrent neural networks were trained using PyTorch designs and optimization processes. HuggingFace, an open-source repository, was used to build the transformers. They gathered opinions from 100 native English speakers who took an online exam. Participants in each experimental session were asked to determine which statements they would be "more likely to encounter in the world, as either speech or written text" and rate their confidence in their response on a 3 point scale.

Despite the consistency in model ranking between our findings and earlier work, GPT-2's severe failure in predicting human reactions to natural vs. synthetic contentious pairings reveals that GPT-2 does not adequately imitate the computations used in human processing of even short words. This result is somewhat predictable because GPT-2 is an off-the-shelf machine learning model that was not created with human psycholinguistic and physiological features in mind. Even though we found a lot of human inconsistency, a recent GPT-2 study found that almost all of the variations in how people responded to actual words could be explained.

COPYRIGHT_SZ: Published on https://stationzilla.com/natural-language-models/ by Dr. Cooney Blades on 2022-05-13T06:05:07.531Z

Natural And Synthetic Sentences Pairs

The researchers arranged 90 sentence pairings into ten sets of nine sentences each and gave each set to a different group of ten individuals. They calculated the percentage of trials in which the model and the person agreed on which phrase was more likely to assess model-human alignment. All nine language models outperformed chance by predicting human choices for randomly generated natural phrase pairings (50% accuracy). They gave each group of ten individuals a different set of phrase pairs. We statistically analyzed between-model differences while accounting for both people and sentence pairs as random variables using a simple Wilcoxon signed-rank test across the ten participant groups.

A process for synthesizing contentious sentence pairs was created, in which naturally existing sentences serve as initializations for synthetic phrases and reference points that drive sentence synthesis. They started with a naturally occurring statement. They then keep replacing words in the sentence with comments from a predefined vocabulary to make the synthetic sentence less likely to be correct by one language model while ensuring that the synthetic sentence is at least as possible to be accurate by another model.

Human participants rated ten contentious synthetic-sentence pairings for each model pair. They assessed how well each model predicted human sentence choices in all of the controversial synthetic-sentence combinations in which it was one of two models tested.

Conclusion

The tests proved that:

  • There are many ways natural language processing models can generate controversial sentence pairs. They can pick pairs of sentences from a corpus or change natural sentences to make controversial predictions.
  • The contentious sentence pairs make it easy to quickly compare models that seem the same in terms of human consistency.
  • All of the existing natural language processing model classes mistakenly give a high probability to the following non-natural sentences: A simple statement may be modified such that its likelihood according to a specific model does not diminish. Still, as per human judgments, the phrase becomes much less likely.
  • This method of comparing and testing models may give new ideas about which types of models work best with human language perception, and which types of models need to be made in the future.
Share: Twitter | Facebook | Linkedin

About The Authors

Dr. Cooney Blades

Dr. Cooney Blades

Recent Articles

  • Khushi Mukherjee - An Indian Actress And Model, Who Works In Hindi TV Shows And Web Series

    Celebs

    Khushi Mukherjee - An Indian Actress And Model, Who Works In Hindi TV Shows And Web Series

    Actress Khushi Mukherjee hails from India. Modeling was where Khushi's profession got its start. Anjalaturai, the film in which she first appeared, was released in 2013. In 2017, she made her debut in the reality program MTV Splitsvilla Season 10, which was her first appearance on television.

  • Rudhraksh Jaiswal An Indian Film And Television Actor

    Celebs

    Rudhraksh Jaiswal An Indian Film And Television Actor

    In the realm of Indian entertainment, the young performer Rudhraksh Jaiswal is quickly becoming a prominent figure. His most notable collaboration was with the well-known Hollywood actor Chris Hemsworth in the most recent movie to be released on Netflix, which was titled "Extraction."

  • What Does A Red Moon Symbolize? Time To Let Go And Heal

    Spirituality

    What Does A Red Moon Symbolize? Time To Let Go And Heal

    Today in this article you will discuss what does a red moon symbolize. In essence, the term "blood moon" is just a poetic moniker for what is known as a "total lunar eclipse." Normally, the moon receives its shine from sunlight bouncing off of its surface, but during a lunar eclipse, the moon passes into the Earth's umbra, which is the region of shadow that is the deepest and most central.

  • What Does The Peacock Represent? Symbol For Power, Strength, Confidence, And Even Divinity

    Spirituality

    What Does The Peacock Represent? Symbol For Power, Strength, Confidence, And Even Divinity

    In this article you will learn about what does the peacock represent. People have long admired peacocks because they are flamboyant, captivating, and breathtakingly beautiful. These elaborate and vibrant birds are frequently connected to splendor, grandeur, and luxury.

  • Getting Fired In A Dream - A Sign Of Dissatisfaction

    Dream Interpretation

    Getting Fired In A Dream - A Sign Of Dissatisfaction

    Getting fired in a dream from a job entails losing credibility and self-assurance. Something you adore so much will be taken from you. To get there, you'll have to put in more effort. You could experience stress. Getting fired in a dream signifies rejection. In your romantic life, rejection will happen. The rejection of a proposal will put the love life on hold.

  • Dreams Of Being Paralyzed And Unable To Speak - Means That Your Freedom Is Threatened

    Dream Interpretation

    Dreams Of Being Paralyzed And Unable To Speak - Means That Your Freedom Is Threatened

    Your potential is represented by the dreams of being paralyzed and unable to speak. You must address a problem or issue destroying your sense of equilibrium and harmony while being more hospitable and less judgmental of others. Being immobilized in a dream represents your difficulties in achieving a goal.

  • Court Sentences Life Imprisonment To Indian Guru Asaram In Second Rape Case

    News

    Court Sentences Life Imprisonment To Indian Guru Asaram In Second Rape Case

    According to his attorney, on Tuesday, the Indian court sentences life imprisonment to Indian Guru Asaram for rape and sodomy. In his ashram in the western Indian state of Gujarat, Asaram was found guilty of abusing the lady on many occasions between 2001 and 2006.

  • What Do Hiccups Symbolize? Mean You're Under A Curse Or Spell

    Spirituality

    What Do Hiccups Symbolize? Mean You're Under A Curse Or Spell

    In this article, you will learn about what do hiccups symbolize. The worst are the hiccups. You appear to have halted for a brief moment, remaining calm and steady. Your stomach reacts abruptly by contracting in some way. You feel it before you hear it: a swift, uncontrollable jerk followed by a loud popping noise made by your vocal cords vibrating as a result of your diaphragm.

  • Dream About Being Pregnant Christian Meaning - A Sign Of The Love Of God

    Dream Interpretation

    Dream About Being Pregnant Christian Meaning - A Sign Of The Love Of God

    In general, dream about being pregnant christian meaning is positive. While it might be a positive omen for all expectant mothers. One of every woman's regular prayers is to have a child. In actuality, the child is the soul of a marriage. To some extent, dreaming of becoming pregnant can have a positive meaning if you are currently pregnant.

  • What Does It Mean To See A Black Squirrel?

  • What Do Bananas Symbolize In The Bible?

  • Coping With Insomnia And Why Does It Happen?

  • Raajeshwari Sachdev - An Indian Television And Film Actress

  • Alice Kaushik - An Indian Model And Actress Who Is Known For Playing The Role Of Ravi In Television Series