On December 6, Google introduced its latest artificial intelligence (AI) model called Gemini, asserting that it outperforms OpenAI's GPT-4, positioning itself as the most advanced AI model presently available. Gemini is designed to be multimodal, capable of comprehending and amalgamating diverse types of information. It is available in three versions – Ultra, Pro, and Nano – catering to various applications, with a notable strength in executing complex mathematical operations and professional coding tasks, an aspect where it appears to surpass GPT-4.
Upon its launch, Google presented several benchmarks comparing Gemini against GPT-4. Notably, Gemini Ultra excelled in achieving "state-of-the-art performance" in 30 out of 32 academic benchmarks commonly used in the development of large language models (LLMs). However, skeptics have emerged across various online platforms, criticizing Google's methodological approach in showcasing Gemini's supremacy, questioning the benchmarks utilized, and expressing skepticism about Google's promotional tactics for the product.
Criticism on social media pointed out potential flaws in Google's marketing strategy and presentation of Gemini's capabilities. Some users criticized Google's choice of benchmarks, suggesting that they may not provide an accurate portrayal of Gemini's true performance compared to the current state of GPT-4. Others expressed disappointment, feeling misled by Google's promotional content that may have overemphasized Gemini's capabilities compared to GPT-4.
Some users highlighted discrepancies in the comparison between Gemini and GPT-4, noting differences in the cues and versions utilized for evaluation. This discrepancy might significantly impact the outcome of the comparison. Moreover, the model used for testing Gemini's capabilities is not currently available publicly, prompting concerns about the transparency of the comparison and its relevance to practical applications.
Individual experiences using Gemini, as opposed to relying solely on benchmarks, have also surfaced. Several users reported their experiences using Gemini through Google's Bard tool, sharing mixed reviews. Anne Moss, an online publishing services worker and regular AI user, expressed being underwhelmed by Gemini/Bard's performance compared to GPT-4, citing instances where Gemini appeared reluctant to answer political questions and seemingly misrepresented personal information. Other users shared similar sentiments, providing examples where Gemini/Bard's output seemed inferior compared to GPT-4.
Google has indicated plans for a broader public release of Gemini in early 2024, aiming to integrate the model into various Google applications and services.
