Our results strongly suggest that grammatically unbiased sentence embeddings and contextualized word embeddings have non-trivial implicit knowledge of grammar before supervised training on CoLA.

Still, we maintain that our approach is a valuable step in the direction of evaluating the APS. We find that our models do not show evidence of learning non- local dependencies related to agreement and questions, but do appear to acquire knowledge about basic subject-verb-object word order and verbal argument structure. These experiments illustrate how acceptability classification and CoLA can give detailed insights into what grammatical knowledge typical neural network models can acquire. (iv) We assess our models’ performance on acceptability classification of specific linguistic phenomena. (iii) We analyze the impact of supervised training on acceptability classifiers by varying the domain and quantity of training data. Our best model outperforms unsupervised baselines, but falls short of human performance on CoLA by a wide margin. (ii) We train several semi-supervised neural sequence models to do acceptability classification on CoLA and compare their performance with unsupervised models from Lau et al. In this paper we make four concrete contributions: (i) We introduce the Corpus of Linguistic Acceptability (CoLA), a collection of sentences from the linguistics literature with expert acceptability labels which, at over 10k examples, is by far the largest of its kind.

Sandra model set 233