Large Language Model (LLM) Training Data Is Running Out. How Close Are We To The Limit? In the quickly developing fields of Artificial […]

IBM AI Team Releases an Open-Source Family of Granite Code Models for Making Coding Easier for Software Developers IBM has made a […]

OpenBezoar: A Family of Small, Cost-Effective, and Open-Source AI Models Trained on Mixed Instruction Data The recent success of […]

CodeEditorBench: A Machine Learning System for Evaluating the Effectiveness of Large Language Models (LLMs) in Code Editing Activities Coding-related jobs have led to the rapid advancement of Large Language Models (LLMs), with a focus on code editing. LLMs created specifically for coding jobs are applied to a variety of activities, including code optimisation and repair. As programming tools, they are becoming more and more popular, but most evaluation techniques concentrate on code production, ignoring the crucial role that code editing plays in software development. In recent research, a team of researchers from the Multimodal Art Projection Research Community, University of Waterloo, HKUST, University of Manchester, Tongji University, and Vector Institute has introduced CodeEditorBench, an assessment system that has been designed to evaluate LLMs’ effectiveness in a range of code editing activities, such as requirement switching, debugging, translating, and polishing.  In contrast to other benchmarks that primarily concentrate on code creation, CodeEditorBench emphasises real-world applications and pragmatic elements of software development. The team has selected a variety of coding scenarios and challenges from five distinct sources, covering a broad spectrum of programming languages, degrees of difficulty, and editing assignments. By doing this, they have made sure that the evaluation takes into account the variety and complexity of difficulties found in actual coding environments. The team has found some intriguing trends in their review, which included 19 distinct LLMs. In the CodeEditorBench framework, closed-source models, specifically, Gemini-Ultra and GPT-4 have demonstrated better performance than open-source models. This emphasises how important model architecture and training data are to deciding performance, particularly when varying prompt sensitivity and problem categories.  The team has summarized their primary contributions as follows. The goal of CodeEditorBench is to offer a uniform approach for evaluating LLMs. Tools for additional analyses, training, and visualisation have been included in this framework. To promote more research into LLM features, the team has shared that all evaluation-related data will be openly accessible. To improve the assessment’s comprehensiveness, more evaluation measures will be added in the future.  The main aim is to map the current state of LLMs. OpenCIDS-33B is the most effective base model available to the public, followed by OpenCI-DS-6.7B and DS-33B-INST. Models like Gemini, GPT, and GLM that are not publicly accessible usually perform better than those that are. OpenCIDS-33B and DS-33B-INST, two instruction-tuned models with over 30 billion parameters, close this performance difference.  The goal of CodeEditorBench is to draw attention to the shortcomings of LLMs, especially when it comes to rewriting and revising code. Though it performs admirably in three of the four categories, GPT4’s code-polishing abilities are noticeably lacking. In a similar vein, Gemini Ultra is not up to the challenge of changing code requirements. The team has recognized these constraints to tackle these particular issues in LLM training and development. In conclusion, CodeEditorBench’s main objective is to spur advances in LLMs by providing a strong platform for thoroughly assessing code editing capabilities. Check out the Paper, Project, and Github. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter. Join our Telegram Channel, Discord Channel, and LinkedIn Group. If you like our work, you will love our newsletter.. Don’t Forget to join our 40k+ ML SubReddit [1/n]🚀🚀🚀 Excited to share our latest work: “CodeEditorBench:Evaluating Code Editing Capability of Large Language Models”! https://t.co/GckeztzIbT### 🧐 Highlights of the CodeEditorBench:> 8K meticulously collected code editing questions from five sources: namely… pic.twitter.com/BUaN6v99BM— Ge Zhang (@GeZhang86038849) April 5, 2024 Tanya Malhotra is a final year undergrad from the University of Petroleum & Energy Studies, Dehradun, pursuing BTech in Computer Science Engineering with a specialization in Artificial Intelligence and Machine Learning.She is a Data Science enthusiast with good analytical and critical thinking, along with an ardent interest in acquiring new skills, leading groups, and managing work in an organized manner. 🐝 Join the Fastest Growing AI Research Newsletter Read by Researchers from Google + NVIDIA + Meta + Stanford + MIT + Microsoft and many ot […]

CodeEditorBench: A Machine Learning System for Evaluating the Effectiveness of Large Language Models (LLMs) in Code Editing Activities Coding-related jobs have led to the rapid advancement of Large Language Models (LLMs), with a focus on code editing. LLMs created specifically for coding jobs are applied to a variety of activities, including code optimisation and repair. As programming tools, they are becoming more and more popular, but most evaluation techniques concentrate on code production, ignoring the crucial role that code editing plays in software development. In recent research, a team of researchers from the Multimodal Art Projection Research Community, University of Waterloo, HKUST, University of Manchester, Tongji University, and Vector Institute has introduced CodeEditorBench, an assessment system that has been designed to evaluate LLMs’ effectiveness in a range of code editing activities, such as requirement switching, debugging, translating, and polishing.  In contrast to other benchmarks that primarily concentrate on code creation, CodeEditorBench emphasises real-world applications and pragmatic elements of software development. The team has selected a variety of coding scenarios and challenges from five distinct sources, covering a broad spectrum of programming languages, degrees of difficulty, and editing assignments. By doing this, they have made sure that the evaluation takes into account the variety and complexity of difficulties found in actual coding environments. The team has found some intriguing trends in their review, which included 19 distinct LLMs. In the CodeEditorBench framework, closed-source models, specifically, Gemini-Ultra and GPT-4 have demonstrated better performance than open-source models. This emphasises how important model architecture and training data are to deciding performance, particularly when varying prompt sensitivity and problem categories.  The team has summarized their primary contributions as follows. The goal of CodeEditorBench is to offer a uniform approach for evaluating LLMs. Tools for additional analyses, training, and visualisation have been included in this framework. To promote more research into LLM features, the team has shared that all evaluation-related data will be openly accessible. To improve the assessment’s comprehensiveness, more evaluation measures will be added in the future.  The main aim is to map the current state of LLMs. OpenCIDS-33B is the most effective base model available to the public, followed by OpenCI-DS-6.7B and DS-33B-INST. Models like Gemini, GPT, and GLM that are not publicly accessible usually perform better than those that are. OpenCIDS-33B and DS-33B-INST, two instruction-tuned models with over 30 billion parameters, close this performance difference.  The goal of CodeEditorBench is to draw attention to the shortcomings of LLMs, especially when it comes to rewriting and revising code. Though it performs admirably in three of the four categories, GPT4’s code-polishing abilities are noticeably lacking. In a similar vein, Gemini Ultra is not up to the challenge of changing code requirements. The team has recognized these constraints to tackle these particular issues in LLM training and development. In conclusion, CodeEditorBench’s main objective is to spur advances in LLMs by providing a strong platform for thoroughly assessing code editing capabilities. Check out the Paper, Project, and Github. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter. Join our Telegram Channel, Discord Channel, and LinkedIn Group. If you like our work, you will love our newsletter.. Don’t Forget to join our 40k+ ML SubReddit [1/n]🚀🚀🚀 Excited to share our latest work: “CodeEditorBench:Evaluating Code Editing Capability of Large Language Models”! https://t.co/GckeztzIbT### 🧐 Highlights of the CodeEditorBench:> 8K meticulously collected code editing questions from five sources: namely… pic.twitter.com/BUaN6v99BM— Ge Zhang (@GeZhang86038849) April 5, 2024 Tanya Malhotra is a final year undergrad from the University of Petroleum & Energy Studies, Dehradun, pursuing BTech in Computer Science Engineering with a specialization in Artificial Intelligence and Machine Learning.She is a Data Science enthusiast with good analytical and critical thinking, along with an ardent interest in acquiring new skills, leading groups, and managing work in an organized manner. 🐝 Join the Fastest Growing AI Research Newsletter Read by Researchers from Google + NVIDIA + Meta + Stanford + MIT + Microsoft and many othe […]

Meet Sailor: A Family of Open Language Models Ranging from 0.5B to 7B Parameters for Southeast Asian (SEA) Languages […]

Meet Sailor: A Family of Open Language Models Ranging from 0.5B to 7B Parameters for Southeast Asian (SEA) Languages […]

Meet Sailor: A Family of Open Language Models Ranging from 0.5B to 7B Parameters for Southeast Asian (SEA) Languages […]

'Think-and-Execute': A Machine Learning Framework that Encapsulates the Common Logical Structure of a Job Using Pseudocode for Efficient Reasoning in Large Language Models (LLMs) In Large Language Models (LLMs), reasoning involves dissecting a problem’s logical structure and turning it into a sequence of logical steps that lead to a solution. For LLMs, this procedure has proven difficult, particularly in algorithmic reasoning where intricate logical patterns must be interpreted and transformed into a series of processes. Understanding patterns inside an issue and decomposing them into a series of logical stages to arrive at a solution are key components of algorithmic thinking. Although a variety of reasoning tasks have demonstrated the potential of LLMs, algorithmic reasoning remains difficult because of its complex structure.  In order to convey the reasoning required to solve a particular instance or topic, recent studies have tried to address this challenge by employing programming languages like Python. It is challenging to write executable code that faithfully captures the reasoning in a single inference call and does it in real-time. Even if two instances need the same logic, the code created for one cannot be utilized for another. In recent research, a team of researchers from Yonsei University and KAIST AI has presented THINK-AND-EXECUTE, a unique architecture that splits the language model reasoning process into two parts to get over the limitations. The two parts are as follows.  THINK: The framework looks for a task-level logic in this phase that is shared by all instances of a certain task. Next, pseudocode, which offers a more adaptive and flexible representation than programming languages like Python, has been used to express the shared logic.  EXECUTE: The framework adapts the task-level logic to each unique instance after it has been defined and stated in pseudocode. Subsequently, it emulates the pseudocode execution for every occurrence, efficiently utilizing the found logic to resolve the issue. The effectiveness of THINK-AND-EXECUTE has been shown through comprehensive trials on seven algorithmic thinking tasks. The framework beats multiple robust baselines, including Program-of-Thought (PoT) and Chain-of-Thought (CoT), which rely on instance-specific reasoning techniques. This implies that learning task-level logic can help LLMs become more proficient reasoners. Even though these models have been trained to follow instructions in regular language, the results have demonstrated that pseudocode is a more useful tool for directing LLM thinking than natural language.  The team has summarized their primary contributions as follows. A new and unique thinking paradigm known as THINK-AND-EXECUTE has been suggested. This framework encapsulates the common logical structure of a given job using pseudocode. The method allows for more efficient reasoning in LLMs by utilizing pseudocode, which provides flexibility and adaptability.  The team has shown that THINK-AND-EXECUTE outperforms well-established baselines like Chain-of-Thought and Program-of-Thought prompting, based on substantial research on a variety of algorithmic tasks inside the Big-Bench Hard dataset. This demonstrates how well the system works to improve reasoning abilities in a variety of issue domains. Utilizing THINK-AND-EXECUTE, the team has demonstrated the effectiveness of the method by effectively transferring the pseudocode produced by an LLM to smaller language models. This indicates that the approach is both generalizable and scalable, meaning it can be applied to a variety of model topologies and sizes. Check out the Paper. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter. Join our Telegram Channel, Discord Channel, and LinkedIn Group. If you like our work, you will love our newsletter.. Don’t Forget to join our 40k+ ML SubReddit Tanya Malhotra is a final year undergrad from the University of Petroleum & Energy Studies, Dehradun, pursuing BTech in Computer Science Engineering with a specialization in Artificial Intelligence and Machine Learning.She is a Data Science enthusiast with good analytical and critical thinking, along with an ardent interest in acquiring new skills, leading groups, and managing work in an organized manner. 🐝 Join the Fastest Growing AI Research Newsletter Read by Researchers from Google + NVIDIA + Meta […]

IsoBench: An Artificial Intelligence Benchmark Dataset Containing Problems from Four Major Areas: Math, Science, Algorithms, and Games The fields of Natural Language Processing (NLP) and Natural Language Generation (NLG) have undergone amazing transformations since the introduction of Large Language Models (LLMs) and multimodal foundation models. These models, which include GPT4V, Claude, and Gemini, combine visual encoders and LLMs.  Present-day foundation models have shown remarkable performance when presented with text-only or combined image and text inputs. However, an important question arises: Will their capacities change according to the kind of input they are served? In order to answer this question, a team of researchers has presented IsoBench, a benchmark dataset containing challenges from four important domains: games, science, mathematics, and algorithms. There are several isomorphic representations for every problem in IsoBench, including textual, mathematical, and graphic formats. Because of this diversity, performance disparities resulting from different forms of representation can be thoroughly examined. The team has shared that IsoBench can be used as a tool to diagnose discrepancies in model performance caused by the input representation by giving detailed feedback. A recurring pattern is seen in a variety of foundation models as models show a predilection for textual representations on the same topic. For example, Claude-3 Opus performs 28.7 points lower when given photos instead of text when assessed on all issues in IsoBench. When presented with image inputs instead of text, GPT-4 Turbo and Gemini Pro both exhibit performance decreases of 18.7 and 14.9 points, respectively. Two prompting strategies, IsoCombination and IsoScratchPad, have been proposed to mitigate this reported bias and enhance model performance. IsoScratchPad focuses on enabling translations between multiple input forms, whereas IsoCombination considers combinations of diverse input representations.  By utilizing the advantages of various input modalities, these strategies can lessen the performance disparities between foundation models. The team has shown through experiments that IsoCombination and IsoScratchPad both improve model performance, presenting intriguing directions for further study and advancement in multimodal AI systems. The team has summarized their primary contributions as follows. IsoBench, an extensive test dataset with 1,630 samples has been introduced that spans a number of topics, including chess, physics, chemistry, and discrete and applied mathematics. Comprehensive multimodal performance evaluations are made possible by the many isomorphic input representations that each sample has, including textual formats specific to the domain and visual formats.  Using IsoBench, the team has evaluated eight well-known foundation models and found a recurring pattern, which is multimodal models outperform image-based prompts when it comes to text-only prompts.  The team has also suggested two methods to bridge the performance gaps between various input modalities. While IsoScratchPad (IsoSP) translates visual inputs into textual representations during inference, IsoCombination (IsoCB) mixes input modalities. Based on their research, the team has found that in some cases, IsoCB and IsoSP can improve multimodal foundation models’ performance by almost ten percentage points. By using these strategies, the observed bias towards textual representations is lessened, and the model performs better with a variety of input modalities. Check out the Paper and Project. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter. Join our Telegram Channel, Discord Channel, and LinkedIn Group. If you like our work, you will love our newsletter.. Don’t Forget to join our 39k+ ML SubReddit Tanya Malhotra is a final year undergrad from the University of Petroleum & Energy Studies, Dehradun, pursuing BTech in Computer Science Engineering with a specialization in Artificial Intelligence and Machine Learning.She is a Data Science enthusiast with good analytical and critical thinking, along with an ardent interest in acquiring new skills, leading groups, and managing work in an organized manner. 🐝 Join the Fastest Growing AI Research Newsletter Read by Researchers from Google + NVIDIA + Meta + Stanford + MIT + Microsoft and many other […]