Elon Musk-Led xAI Open Sources Its AI Chatbot Grok For Developers | All You Need To Know

Available on GitHub, the open-release protocol will enable developers to experiment with the model and help xAI improve Grok in the future. The model is Musk’s answer to take on AI products from Google, Meta and AI company OpenAI.

According to xAI, Grok is trained from scratch and is a 314 billion parameter Mixture-of-Experts model. The released version is the “raw base model” of the pre-training phase, which dates back to October 2023 and is “not fine-tuned for any specific application, such as dialogue”.

“We are releasing the base model weights and network architecture of Grok-1, our large language model. Grok-1 is a 314 billion parameter Mixture-of-Experts model trained from scratch by xAI,” xAI said. The weights and architecture have been released under the Apache 2.0 license. The company noted that due to the large size of the model, a system with sufficient GPU is a prerequisite to test the model.

As per xAI’s blog post, while the model has been trained via large amounts of textual data, it has not been fine-tuned for any specific purpose or task. It noted that Grok was “trained using a custom training stack on top of JAX and Rust” in October last year and “314 billion parameter Mixture-of-Experts model with 25 per cent of the weights active on a given token”.

According to VentureBeat, while the license allows commercial use, access to training data and real-time connection aided by X is not included. Meanwhile, according to the company’s announcement post from November 2023, powered by the Grok-1 large language model (LLM), Grok’s launch version was developed after four months of training. It can help with tasks including text generation, coding, solving queries and more.