An Experiment Based Evaluation of Logical Reasoning Abilities of GPT-3.5 and GPT -4

Employing logical logic capability is a comprehensive natural language understanding bid. With the release of Generative Pretrained Transformer 4( GPT- 4), stressed as” advanced” at logic tasks, we’re eager to learn the GPT- 4 performance on colorful logical logic tasks. This report analyses multiple logical logic datasets, with popular marks like LogiQA and ReClor, and recently- released datasets like AR- LSAT. We test themulti-choice reading appreciation and natural language conclusion tasks with marks taking logical logic. We further construct a logical logic out- ofdistribution dataset to probe the robustness of ChatGPT and GPT- 4. A comparison in terms of performance has also been been made between ChatGPT and GPT-4 . Trial results show that ChatGPT performs significantly better than the RoBERTa forfeiture- tuning system on utmost logical logic marks. With early access to the GPT- 4 API we’re suitable to conduct violent trials on the GPT- 4 model. The results show GPT- 4 yields indeed higher performance on utmost logical logic datasets. Among marks, ChatGPT and GPT- 4 do fairly well on well- known datasets like LogiQA and ReClor. still, the performance drops significantly when handling recently released and out- of- distribution datasets. Logical logic remains grueling for ChatGPT and GPT- 4, especially on outof- distribution and natural language conclusion datasets. We release the prompt- style logical logic datasets as a standard suite and name it LogiEval.