Most companies and use cases do not care about PhD-level capabilities. They just want an AI model to depend on with absolute reliability, for the simple tasks in life.

Imagine a highly competent, dependable, reliable assistant, who does all your chores–instead of the caffeine-overloaded genius, who sometimes works when it “feels like it”--but otherwise causes more harm than good half the time.

That's what we just built with the Featherless.ai Action-R1 model & agent, which achieved SotA (State-of-the-Art) in the REAL (Realistic Evaluations for Agents Leaderboard) benchmark.

Our AI agent (test result at link) achieved a 65% success rate on the REAL benchmark. This makes it nearly 50% better than Anthropic computer use (result link), the next best model & framework by a major lab, at 42%.

What is the REAL benchmark?

What is interesting about the REAL benchmark is that it tests over 110 practical real-world tasks. These are not university-level knowledge tasks, but real-world chores that reflect what people do online every day, like booking flights, organizing and replying emails, and shopping for groceries. Basically, office desk work.

These are tested using a controlled test replica, which mirrors 11 major websites, including Airbnb, Amazon, Gmail, LinkedIn, and Uber.

The paper for the benchmark can be found here.

Why does reliability matter?

Frontier AI agents have been plateauing at around 43% overall task completion. We hit 65%, leapfrogging it by 22 percentage points.

More importantly, on certain sites, such as the Omnizon (aka Amazon clone), our agent reached a 100% success rate, while the next best managed 60%.

And the 99%+ distinction matters.

Because you can now rely on this model and agent for tasks in this domain.

If a model is only at 50% success rate at a category of task, you spend more time and energy “babysitting” the models. In several cases, the model will require human intervention which can take more bandwidth than the task itself.

Raising reliability means that you can hand off tasks to the AI, eliminating the frustration many workers face with AI in the enterprise.

For the first time, we have built an AI agent you can rely on for tasks within a handful of platforms with 99%+ success. We will eventually expand to 99%+ reliability for all web platforms.

How we did it

We partnered with the team at UI-licious, who automate end-to-end testing for their customers with AI at scale, building on their experience and knowledge in UI test automation and their proprietary PetaByte scale dataset which they have built up over the years on UI testing.

Together, we co-built a specialized AI action model and agent harness. This helps the AI understand instructions and how to navigate complex and dynamic UIs.

As a bonus, these AI agents are not just navigating and completing tasks. They are finding and issuing bug reports, which we will be following up with the eval maintainers on 😉

The bigger picture: attacking multi-billion dollar markets

The $3B+ Robotic Process Automation (RPA) market and the $20B+ UI testing market are dominated by brittle tools that break easily and require constant maintenance, where AI agents achieve about 60% reliability.

Today’s RPA tools like UIPath operate like glorified macros, breaking when user interfaces change or when variability is introduced. Businesses spend millions maintaining these systems, fixing broken RPA scripts.

Agentic AI is different. It learns. It adapts. It interacts with interfaces the way humans do: observing, interpreting, and acting based on context. It allows automation over dynamic interfaces.

And we’re just getting started.

As we scale our AI research in the open source space, we will be working alongside industry partners to build reliable AI Agents and models for their own industry specific use, with their own proprietary datasets.

So we will be increasing reliability, not just in automated UI testing, but in all future domains as well, with industry partners from accounting to legal.

Raising reliability means unlocking the 90% of AI projects that fail to enter production within enterprises, as the AI was just “not reliable enough” for day-to-day office tasks.

Raising reliability means bringing into production your AI agent into the market.

If you are interested in partnering with us, and making your AI more SoTA-reliable, please reach out to us.

How can we try this new action model & agent?

Look out for a follow-up launch announcement with our design partners at UI-licious.

This AI action model and agent will be co-launched on both platforms.

For early access, you can sign up here:
https://forms.gle/wxwQ2z12xf1KzFvPA

Priority access will be given to portfolio companies among our investors and users.

• What works small often fails big — Architectural innovations that show promise at 1M parameters may break down at 1B or 50B.

• Validating at scale is expensive — Training from scratch to test a new architecture at meaningful scale can cost at least $5–10M.

• High risk, uncertain reward — You’re just as likely to degrade performance as improve it—making architecture exploration financially unsustainable for most labs.

Training a state-of-the-art language model from scratch costs roughly $5-10M—just to validate a new attention mechanism, recurrence scheme, or memory system.

From our team experience, it typically takes 20–80 architecture iterations to achieve a 10%+ improvement. We've done this four times over the past two years.

For most AI labs, that level of experimentation would cost around $250 million in research GPU time. From that perspective, it's often more rational to invest in scaling model parameters and datasets for a near-guaranteed performance gain of ~10%.

At Featherless, we believe this bottleneck in architecture validation has slowed progress—not only in capabilities but in reliability.

But what if the cost to validate an architecture dropped from $5 million to $20K?

With that same $250 million, we could run over 12,500 iterations, uncovering 100+ architecture improvements, each with 10%+ gains. Compounded, that’s a theoretical 1,378,000% improvement in performance.

That’s why we’re excited about RADLADS.

Introducing RADLADS

RADLADS (Rapid Attention Distillation to Linear Attention Decoders at Scale) is a new method for converting massive transformer models (e.g., Qwen-72B) into new AI models with alternative attention mechanisms—at a fraction of the original training cost.

• Total cost: $2,000–$20,000

• Tokens used: ~500 million

• Training time: A few days on accessible cloud GPUs (8× MI300)

• Cost reduction: ~250× reduction in the cost of scientific experimentation

Instead of training from scratch, we convert existing models to new attention architectures in three steps:

1. Align hidden states between the original transformer and the target attention architecture

2. Distill output behavior (logits) from the original model

3. Fine-tune for long-context performance

You can read about the process details from our paper review on huggingface and arxiv. This is the same technique that allowed us to train our latest 72B1 attention-free, with only 8 GPU’s.

What does this mean for research?

RADLADS is already changing how we explore AI architecture. We can now:

• Rapidly test novel attention mechanisms and hybrid designs

• Iterate on model structures in days, not months

• Validate alignment and interpretability hypotheses at scale

This isn’t just about RWKV—it opens doors for advancing Transformers, State Space models, xLSTMs, and architectures yet to be imagined. Its about accelerating our pace of research.

And we’re not doing it alone. Since announcing our work, we've collaborated with other researchers to validate multiple attention mechanisms, including Transformer-based variants.

Reach out to us if you have any attention alternative your research team or university lab is working on and looking to validate in collaboration.

It’s all part of our mission to make personalized reliable AI — and eventually AGI — a reality

Featherless AI - recursive dev blog

🪿QRWKV-72B and 32B : Training large attention free models, with only 8 GPU's

We are proud to announce the updated QRWKV-72B and 32B…

Read more

a year ago · 14 likes · 3 comments · Eugene Cheah

In the past quarter, we have seen the following breakthroughs in open-source

• GPT-4o Mini class open models (deepseek, qwen-32b) on our platform

• 72B Attention-Free QRWKV model built on 8 GPUs

• Both with more knowledge and capability than an average human

• A tipping point with RWKV v6 and v7 improvements

As such, my prediction (Eugene Cheah) for 2025 is that we are at the inflection point.

• Where we scale towards model reliability, with better memories

• Instead of scaling a trillion parameters

Our vision for AGI in summary

The video above, is the condensed version of our vision for personalized AI & AGI
The writing below, is the longer form with some additional details

The scaling wall for bigger models

The problem with scaling today is within the fundamental promise is for a step-increased improvement in capability, for every 10x in parameter size.

While it has remain somewhat true, that there is a step-up improvement in capability. The problem is in the diminishing returns, on both training and inference costs.

We are now within Billion’s of investment dollar range for the current GPT 4.5 model, with no clear answer if OpenAI Super AGI goal is achievable with another 10x, or 100x, or 1000x. After which, we are starting to talk in Trillions of dollars. Just for training. With even more dollars required to actually run the model.

While this path might still make sense, potentially for “Super AGI”.

It makes no sense for “Human-level AGI”. Because ….

Short Context AGI - is already here (sort of)

For the vast majority of users on applications like Character.AI, within that context window of approximately 8,000 tokens.

These AI characters, within the window, is AGI, with perhaps some flaws, but a unique character experience regardless. It’s what drive the 28 Million plus users to keep engaging with the platform.

The limits are however clear

• How they are unable to reliably follow instructions within memory

• Nor are they able to gracefully handle memories beyond their context length

The lack of reliable AI memory

Because here lies an irony, these models are no doubt knowledgeable and capable.

Today’s best models for both open source (ie. DeepSeek R1) and close source, for example, is no-doubt capable of doing PhD level math and physics, with some degree of reliability (let say 1-out-of-30 times)

But yet they lack the reliability to do basic college-level task (30-out-of-30 times), be it as a cashier, or any simple agent needing to do a long multi step process.

This is also known, as the “Compounding AI Agent Error” problem

And fixing the reliability problem does not need a bigger model. We already have proof of this as observed across our platform…

How reliability is being solved in production today?

One of the interesting patterns we have observed on our platform is the workload differences between individual and scaled-up commercial use cases at featherless

For example, in the above, we show, how our individual users are running thousands of fine-tuned model - with a bias towards the top models like the DeepSeek R1, or LLaMA3 70B as expected.

But this dramatically changes when we see the models running by commercial users in production at scale.

The graph is not a mistake, the vast majority of the production workload we see at scale (by request or token count). Is the 1-year-old mistral Nemo-12b (including both base and finetuned)

• Not the latest model

• Not the biggest model

• That is good enough, for predictable prompt engineering or finetuning

This is consistent with AI systems and agents in production at scale, where either or both of the following solutions are used.

• AI Engineering: large problems are scoped, and broken into smaller tasks, solvable via prompt engineering

• Finetuning: specialized domain-specific model is finetuned for specific tasks

In both cases, significant engineering effort is required

• to design the AI agent, and its workflow, with prompt engineering

• to calibrate the dataset for finetuning, requiring dedicated specialized talent. Due to the limitations of catastrophic forgetting

The later, is typically reserved for larger teams, due to the difficulty in scaling the resources required to “get it right”. Due to the highly trial-and-error nature of the process involved in both tasks.

Catastrophic forgetting: Is the challenge faced by all existing model, where existing knowledge and capabilities is lost. While adding in new knowledge.

How RWKV memory module can solve this

If all AI models are already “capable” enough to potentially handle the vast majority of commercial tasks. Especially when they are finetuned for high reliability.

A recurrent model, with memories being part of its core design. Dramatically simplify the process of “finetuning”. In particular around the topic of catastrophic forgetting

For example, it is possible to “memory tune” the memory module state (Time Mix, in the diagram), which by default is left as blank in our base models.

This will not induce any Catastrophic forgetting, as none of the initial model weights is modified. A process we been testing, and implementing for a few pilot customers.

Recurrent models, also by their nature/design, is trained to handle memories, much closer to how we humans do so. Which makes it naturally more scalable for solving the reliable memory problem transformers face.

It also brings about multiple additional benefits, such as 100x lower inference cost (due to its linear scaling nature).

We are by no means perfect today (or ever),
it is why we iterate, more than anyone else

The RWKV team, is arguably the only team today, who has been consistently be making step function improvements in our architecture every half a year.

This can be best seen by our model performance chart across the past 2 years, from version 4 to version 7.

A process that we intend to accelerate, moving forward.

How to achieve Personalized AI

With the appropriate resource support. We expect to iterate into a reliable version of personalized AI within the next year or two.

We define this, as being able to “Memory Tune” without Catastrophic forgetting, and with high accuracy in narrow tasks. With 100 Million tokens or less. Into production use cases.

Quickly, and without the need for highly specialized professionals.

This is not considered “AGI”, because it will be constrained up to what it was “designed to be trained on”. As it would not permanently be learning new knowledge.

However, this is not an issue for most commercial use cases, as it would become the workhorse that drives the AI agent adoption cycle.

And make it → Personalized AGI

Once personalized AI is mastered, it would allow us to focus on the next step - to automate the process for the model to collect and reflect on their “day-to-day” interaction. For further memory tuning “in the night”.

For most part, this would be done incrementally, instead of a giant leap.

This is due to how its an extension of the Personalized AI memory tuning process.

The main differentiator, however, is we would now allow the model to decide on new knowledge and information from its day to day experiences to “learn from”.

However, we do not expect this automated tuning process to be infinitely stable. Due to potential memory loss in between past a certain scale. Be it a billion or a trillion tokens.

However, what this will simply translate to, is a lifespan for a continuous Personal AGI agent. One that can span days, to weeks, months, and eventually years for each iterative improvement.

Why all of this is inevitable

It is either a binary outcome, can recurrent models, be iterated and scaled (not just by param size), to be more reliable, and have better memories.

• We have provided evidence that the above statement is true, on a progressive trend for 2 years, despite the limited resources invested in the team.

• We have provided evidence that recurrent models can scale up to match the capability of some of the largest open transformer models.

Assuming the above is true

• The high-level roadmap is here, in public.

• Better and more capable open transformer models will arrive,
  which will speed up the process for us to iterate on.
  (little to no training from scratch required)
  

• We have enough resources to keep scaling our inference platform,
  while slowly iterating on this roadmap, with a handful of researchers.

• Even if we Featherless were to suddenly disappear. Enough resources and momentum within the open model space, will let future teams slowly eventually follow this same roadmap laid out.

Overall, we expect the following timeline, if featherless AI is properly supported in this journey, to seize the opportunity window

• < 2 Years for personalized AI

• < 4 Years for personalized AGI

Left on its own, with Moore’s law level of improvement in computing, we expect the hardware requirement to iterate on this roadmap to enter the “personal computing” space in 4 years. Once that is reached, we expect a rapid innovation cycle that leads to the same result, be it within or outside the USA.

We are proud to announce the updated QRWKV-72B1 and 32B.

Both models are available on huggingface and featherless.ai

• 32B | Hugging Face Link | Featherless AI Link

• 72B | Hugging Face Link | Featherless AI Link

The largest model to date - that is not based on the transformer attention architecture.

Surpassing existing transformer models in several benchmarks, while following right behind in others.

This builds on our previous experiments in converting the QRWKV6, where we converted the previous Qwen 2.5 32B model to RWKV. And the previous 72B preview.

Which we applied instead for the Qwen-QwQ-32B model and the Qwen-72B model respectively.

But lets take a step back at what this means …

We now have a model far surpassing
GPT-3.5 turbo, without QKV attention

While slowly closing in on GPT-4O-mini

With lower inference cost, param size, and better performance.

We need no super cluster - only a single server.

Because we were keeping most of the feed forward network layer the same.
We can perform the conversion, (barely) within a single server of 8 MI300 GPU’s

Requiring the full 192GB VRAM allocation per GPU

How the conversion is done: A summary

While more details will be revealed in an upcoming paper. The core idea is similar to the previous QRWKV6 conversion , but this time we apply it to the Qwen-72B and QwQ-32B models

At a high level, you take an existing transformer model

Freeze all the weights, delete the attention layer, replace it with RWKV, and train it through multiple stages

All while referencing the original model logits as a “teacher model”

More specifically it would be the following

• Train the RWKV layer individually, referencing the individual teacher blocks

• Train the RWKV layer, with the whole model, training on teacher logits

  • At this point the model is “usable” but has much more to improve on

• Train all the layers (both FFNN and RWKV), on teacher logits

• Train all the layers with longer context length

Unfortunately, due to the limitation of VRAM, our training was limited to 8k context length. However we view this as a resource constraint, and not a method constraint.

Implication:
AI knowledge, is not in attention, but FFN

Due to the limited token training of 200-500M, of the converted layers. We do not believe that the newly trained RWKV layers, is sufficiently trained for “knowledge/intelligence” at this level.

In other words, the vast majority of an AI model knowledge, is not in the attention but the matrix multiplication FFN (Feed-Forward-Network) layer.

It would be more accurate, to view the Attention mechanisms, be it transformer based, or RWKV. As a means of guiding the model to focus on “what the model thinks” about in the FFN layer.

Benefits: Ideal for large scale application

Additionally, with the shift towards inference-time-computing.

Linear architectures represents a dramatic reduction in both compute and vram requirement cost. Allowing us to scale hundreds to thousand requests per GPU.

We can now rapidly iterate new RWKV architectures on <100B scales

By dramatically reducing the compute requirement for scaling and testing a new RWKV attention architecture. To a small number of GPU’s

We will be able to test, iterate, and validate newer architecture changes faster, taking experiments what previously took weeks (or even months), to days.

Historically, the RWKV group has been averaging 4 major versions across 2 years. With improvement to both model architecture accuracy and memories at every step.

A trend which we plan to accelerate moving forward.

As we work on our roadmap to Personalized AI and eventually Personalized AGI, which you can see more in our following article …

Featherless AI - recursive dev blog

🛣️ Our roadmap to Personalized AI and AGI

If you want to find out more about the latest QRWKV model, that makes all of this possible, it is recommended to read this first…

Read more

a year ago · 4 likes · Eugene Cheah

"AI has the power to transform industries and empower individuals around the globe. However, accessing and utilizing a diverse range of models can be cost-prohibitive and logistically challenging for many users, particularly those in emerging markets, where there's no shortage of demand for AI solutions," notes our Founder and CEO Eugene Cheah. "Featherless.ai is changing the game by providing an affordable and scalable solution for using open source AI models in production. I don't want a future where AI is controlled by the few. I want to empower individuals globally."

What We Offer: The World's Largest Collection of Open-Source Models

At Featherless.ai, we provide instant and affordable access to over 4,000 open-source AI models - including popular options like DeepSeek and LLama - and we're continuously onboarding new models every week. Our flat capacity pricing model ensures cost predictability and scalability, allowing businesses to dynamically scale AI usage without worrying about unexpected charges or rate limits.

Instead of dealing with infrastructure headaches, our users can focus their energies on what matters: running experiments to develop, test, and fine-tune their models. With this new funding, we're planning to support all major AI modalities — including embeddings, vision, and speech.

For casual AI users, our serverless platform offers a low entry point of just $10 per month, providing access to a wide range of the latest open-source models without the need for expensive high-end GPUs. For businesses and enterprises, we deliver scale and significant cost efficiencies through dynamic AI workload scaling and flat capacity pricing.

Our Breakthrough Technology

With our new funding, we're advancing research and development into next-generation AI architectures that can dramatically lower inference costs, making AI deployment feasible on lower cost hardware rather than requiring high-end GPUs.

Our architecture enables seamless support for over 100 languages without any performance degradation. This breakthrough removes a major barrier to global AI deployment, where existing systems struggle to maintain consistency across multiple languages.

One of our proudest innovations is our proprietary hot-swapping technology, which allows AI models to be switched in under 5 seconds, compared to the 30 minutes typically required to load a new AI model onto a standard GPU. This innovation optimizes GPU utilization, eliminates costly downtime, and dramatically reduces operational costs.

We're also actively involved in the development of the open-source RWKV foundation model project, the first AI model under the Linux Foundation, which has been deployed on Windows to billions of devices1. This effort is part of our broader commitment to develop better and more accessible AI models across languages, and reduce the reliance on closed-source models from major tech companies.

What Our Investors Are Saying

"The biggest hurdle to scaling AI is compute cost. Featherless.ai tackles this head-on by providing the trifecta of efficient, effective and affordable AI deployment to businesses globally. Our portfolio companies are avid users of Featherless.ai, and we look forward to expanding that across all the regions we invest in," shares Vishal Harnal, Managing Partner at 500 Global.

Yuichiro Hikosaka, Principal at Airbus Ventures, explains: "Our research into foundational model development revealed that while much of the AI industry has focused on scaling existing transformer models, fundamental inefficiencies in the architecture have remained unaddressed. As most foundational models struggle to maintain performance across multiple languages, creating accessibility barriers in global markets, these structural inefficiencies have threatened to stifle competition and innovation across the broader global ecosystem. Now enter Featherless.ai - a serverless platform that empowers users to run, test, and fine-tune the latest models seamlessly and affordably. Featherless.ai is ready and primed to address a global market."

"We see Large Language Models being experimented and integrated into applications everywhere, Featherless.ai plays a unique role in enabling organizations to test, integrate and run any and every open source LLM easily and cost-effectively. Their innovative approach and focus on serving diverse markets make them a standout in this rapidly growing space," says Chee-We Ng, Managing Partner at Oakseed Ventures.

Our Team

Featherless.ai is co-founded by Eugene Cheah (CEO), Harrison Vanderbyl (CTO), and Wesley George (COO), who bring combined experience of 30 years building software and leading engineering teams.

Join us!

Featherless.ai is hiring! If you're passionate about making AI open and accessible, send your cover letter to hello@featherless.ai. For more information about our platform and services, visit www.featherless.ai.

Together, we can democratize access to AI and ensure that these powerful technologies benefit everyone, not just those with unlimited resources.

AI development continues to advance through diverse approaches to model design and optimization. DeepSeek-R1, with its impressive 671B parameters, has established itself as one of the most capable reasoning-focused models on the market. Its remarkable capabilities have set new benchmarks for what models in this space can achieve.

Meanwhile, efficiency and adaptability continue opening new frontiers, and this is where QwQ-32B, Qwen's latest release, makes its mark.

QwQ-32B: Efficient Reasoning Power

QwQ-32B delivers high-level reasoning, problem-solving, and strong coding/math capabilities in a lightweight package. With just 32B parameters, early benchmarks show impressive performance, making it an attractive option for those looking for strong reasoning capabilities in a more efficient format.

With AI applications diversifying, the demand for models that deliver excellent performance with a smaller resource footprint continues to grow. QwQ-32B exemplifies how well-optimized models can achieve remarkable results.

The Evolution of Reasoning Models

The AI field is evolving rapidly. While early models focused heavily on generative fluency and knowledge retrieval, today's most exciting breakthroughs are in models that can reason, plan, and solve complex problems. DeepSeek-R1 has been instrumental in this evolution, demonstrating the power of advanced reasoning capabilities.

Now, Qwen is expanding possibilities further, showing that reasoning power can be delivered in different formats to meet diverse needs.

How Does QwQ-32B Perform?

Testing indicates that QwQ-32B:

• Excels in logical reasoning tasks with impressive structured problem-solving

• Performs well in math and coding, key benchmarks for reasoning capability

• Offers strong efficiency, delivering high performance with reduced compute requirements

For users seeking high-quality reasoning in an efficient package, QwQ-32B presents an exciting option.

What This Means For You

With both QwQ-32B and DeepSeek-R1 available on Featherless.ai, you now have multiple excellent options for advanced reasoning capabilities. Our ongoing optimization efforts ensure that you'll benefit from continuous improvements in both performance and functionality.

We're committed to making advanced open AI models accessible and practical for everyone. Each model in our lineup offers unique advantages to suit different use cases and requirements.

Try Our Models and Share Your Thoughts

QwQ-32B is now available alongside DeepSeek-R1, and we want to hear about your experiences with both models. Each excels in reasoning tasks while offering different profiles in terms of scale and efficiency.

Leave a review on the Featherless.ai model page: QwQ-32B

Have questions about integrating these models into your workflow? Reach out to us on Discord or check our documentation for implementation guidelines and best practices.

IMPORTANT NOTICE: This preview model has been replaced entirely by the final QRWKV-72B1 model found here:

Featherless AI - recursive dev blog

🪿QRWKV-72B and 32B : Training large attention free models, with only 8 GPU's

We are proud to announce the updated QRWKV-72B and 32B…

Read more

a year ago · 14 likes · 3 comments · Eugene Cheah

At Featherless.ai , we’re thrilled to announce the launch of QRWKV-72B , a revolutionary hybrid model combining the computational efficiency of linear transformers with the precision of attention mechanisms. This breakthrough architecture reduces GPU compute costs by over 50% compared to traditional transformers, making it one of the most cost-effective large language models available today, costing less than $100k to build.

QRWKV-72B sets a new standard for scalability and accessibility, enabling real-time applications across industries.

Why QRWKV-72B Matters

The introduction of QRWKV-72B marks a pivotal moment in AI development. By merging the strengths of linear transformers and attention transformers, QRWKV-72B achieves unparalleled efficiency without sacrificing performance.

Traditional transformer-based models require ~100GB of VRAM (excluding model weights) to handle a single 72B parameter model at a 16k context length. For each additional concurrent request, the VRAM demand increases significantly due to the attention mechanism's quadratic scaling with sequence length.

These innovations unlocks several critical benefits:

• Lower inference costs: Businesses can deploy advanced AI solutions at a fraction of the cost, particularly for tasks requiring long-context processing on affordable GPUs.

• Global Accessibility: Reduced hardware requirements enable smaller organizations and developing nations to leverage state-of-the-art AI technologies.

• Higher Concurrency: Serve more users simultaneously on the same hardware.

• Scalability: Handle longer context lengths without exponential increases in resource usage.

This launch aligns with our mission to make AI accessible to everyone regardless of language or nation. QRWKV-72B runs at a fraction of the inference cost of current models, especially at larger context length, by merging the computational efficiency of linear transformers with the precision of attention mechanisms. This is a key multiplier unlock for not only making AI accessible for the world but for the recent test-time compute style models.

Real-time Voice: Instant, Natural AI Conversations

Alongside QRWKV-72B, we’re proudly introducing Real-time Voice, an ultra-low latency AI speech solution designed for seamless human-computer interaction. Built on our serverless infrastructure, it delivers fast speech processing and generation time, minimizing delays for a more natural conversation experience. This allows individuals and businesses to build interactive voice applications that respond instantly and accurately. Whether for virtual assistants, global call centers, or interactive voice response systems, Real-time Voice ensures fast, reliable, and cost-efficient AI-powered speech applications.

Private-cloud beta

Featherless.ai now offers a Private Cloud solution for organizations that need full control over their AI deployments. Our dedicated, secure environments allow businesses to run open models with the ease of serverless infrastructure, zero maintenance, pay-per-use pricing, and complete data sovereignty. With Private Cloud, sensitive data stays protected while maintaining the scalability and flexibility required for modern AI applications. Whether you're an enterprise prioritizing compliance and security or a developer needing custom AI deployment, Featherless.ai’s Private Cloud delivers seamless, cost-efficient AI hosting with full control over where and how your data is processed.

We invite you to experience the future of AI with QRWKV-72B, our real-time voice capabilities and private cloud solutions. Together, let’s make AI more accessible to everyone regardless of language or nation. Visit our website to access the model via our API or download the model directly from HuggingFace:

• Featherless

• Huggingface

Featherless.ai

Featherless.ai is a serverless inference platform. Our goal is to make all AI models available for serverless inference. We provide inference via API to a continually expanding library of open-weight models, including the most popular models for role-playing, creative writing, coding assistance, and more. Our solutions enable enterprises and individuals to harness the full potential of artificial intelligence without worrying about underlying infrastructure. Featherless.ai offers scalable, secure, and easy-to-use tools that empowers businesses and individuals alike to accelerate their AI initiatives. For more information, visit www.featherless.ai

Over the past year, Recursal.AI has served us well as a name and identity for our development team and our various projects. It has been a banner under which we’ve worked to push the boundaries in AI.

However overtime, we are now more commonly identified by our platform Featherless.AI, which has grown to be the biggest provider of HuggingFace LLMs, striving to make them more accessible, and free from the constraints of closed ecosystems.

It has become the identity most closely associated with our work, our vision, and the principles that drive us forward.

So today, we’re making it official: Recursal.AI is now Featherless.AI. This change isn’t just about branding—it’s a reaffirmation of our commitment to the usage of open-source AI.

We remain dedicated to bringing powerful language models to the wider world for the benefit of the many.

Our mission remains the same: to develop and support AI tools that are transparent, accessible, and unshackled from proprietary control. Here’s to the next chapter—under a name that truly reflects the community and future we’re building together.

— The Featherless.AI Team

Happy December Neurips,

We are proud to announce the triple model weights release of a charm of finches

Q-RWKV-6 32B Instruct Preview

Our latest frontier model.

A variant of RWKV-6, converted from an existing Qwen 32B model.

This is our strongest linear model to date, beating out all previous RWKV, State Space and Liquid AI models, smashing all previous key english benchmarks and evals.

Excitingly, this unlocks the option of converting existing transformer models to more efficient RWKV linear architecture.

Its limitation however, is how it inherits its knowledge training, and tokenizer, from the parent model. Which in this case is limited to approximately 30 languages (compared to RWKV 100+ languages)

See more info: Announcement article
Try the model on our: Featherless.ai inference

RWKV-6 Finch MoE 37B

Our first RWKV MoE model, for RWKV-6, with 11B out of 37B active parameters. Currently provides one of strongest multi-lingual model

See more info: Announcement article

RWKV-6 Finch 7B World 3

An overall multi-lingual upgrade of our v6 7B base models, that is a major bump up from our previous 7B models for multi-lingual and mixed use cases.

This was developed and released under the RWKV foundation. With various contributors from Eleuther AI and RWKV open source group.

See more info: Announcement article

The strongest linear model to date, beating out all previous RWKV, State Space and Liquid AI models, smashing all previous key english benchmarks and evals.

You can find this model available on both

• Hugging Face:
  https://huggingface.co/recursal/QRWKV6-32B-Instruct-Preview-v0.1

• Featherless.ai: https://featherless.ai/models/recursal/QRWKV6-32B-Instruct-Preview-v0.1

Note: that as an instruction preview, the model is not considered final

Trained by converting the weights of the Qwen 32B Instruct model, into a customized QRWKV6 architecture. We were successfully able to replace the existing transformer attention heads with RWKV-V6 attention heads, through a groundbreaking new conversion training process.

This unique training process was developed by the team at Recursal AI, in joint collaboration with the RWKV and EleutherAI open source community.

Benchmarks

We compared QRWKV6 against existing open weights models, both transformer based and linear-architecture based.

In overall, what is most exciting is how the QRWKV6 converted model, perform similarly to its original 32B model.

GPU Sponsor: TensorWave

The conversion process for QRWKV6, was done on 16 AMD MI300X GPUs kindly donated by TensorWave. Each MI300X comes with a whopping 192GB of VRAM, while having comparable H100 level of compute performance.

This allowed us to reduce the minimum number of nodes required for our training process, and simplify our overall training and conversion process.

The conversion process took about 8 hours

The Exciting

Linear models hold promise in substantially lower compute cost at scale. Delivering over a 1000x compute efficiency in inference cost, especially over large context length. A key multiplier unlock for both O1 style inference time thinking, and making AI more accessible for the world.

This technique is also scalable to larger transformer based models. Which we have since started.

The Good

The benefit of this process is that we are able to convert any previously trained QKV Attention based model, such as Qwen and LLaMA based models, into a variant of RWKV. Without needing to retrain the model from scratch.

This allows us to quickly test and prove out the significantly more efficient RWKV Linear attention mechanic at a larger scale, with a much smaller budget, without training from scratch. Proving out the architecture design and scalability of RWKV.

Once again proving, QKV attention is not all you need.
( Someone ping @jefrankle and @srush_nlp )

The Bad

The disadvantage of this process is that the model inherent knowledge and dataset training, is based on its “parent” model. Meaning unlike previous RWKV models trained on over 100+ languages. The QRWKV model is limited to the approximate 30 languages supported by the Qwen line of models.

Additionally, instead of RWKV based channel mix and feedforward network layers, we retain the “parent” model feed forward network architecture design. This means there will be incompatibility with existing RWKV inference code.

Separately, due to our compute budget, we were only able to do the conversion process up to 16k context length. While the model does exhibit stability beyond the given context length, the following model may need additional training to accurately support larger context length

Future Followups

Currently Q-RWKV-6 72B Instruct model is being trained

Additionally with the finalization of RWKV-7 architecture happening soon, we intend to repeat the process and provide a full line up of

• Q-RWKV-7 32B

• LLaMA-RWKV-7 70B

We intend to provide more details on the conversion process, along with our paper after the subsequent model release.

References

• Model weights and code here

Acknowledgements

• Special thanks to TensorWave and AMD for sponsoring the MI300X training

• EleutherAI for support and guidance, especially on benchmarks and publishing research papers about the RWKV architecture

• Linux Foundation AI & Data group for supporting and hosting the RWKV project

• Recursal AI for its commitment to providing resources and development for the RWKV ecosystem - you can use their featherless.ai platform to easily run RWKV and compare to it other language models

And of course a huge thank you to the many developers around the world working hard to improve the RWKV ecosystem and provide environmentally friendly open source AI for all.

We’re excited to release the latest addition to the RWKV family of model releases: Flock of Finches 37B-A11B v0.1!

This is an experimental model that uses 11 billion active parameters, and despite our new flock having been trained on only 109 billion tokens, roughly matches our recently released Finch 14B model on common benchmark evaluation scores. You can find the model and code at huggingface here , or try it on featherless AI platform here

We leveraged an efficient Sparse Mixture of Experts (MoE) method to supply a higher total parameter count while activating only a fraction of those parameters for any given token. This saves time and uses less compute during both training and inference. As with most architectural choices, there is a tradeoff; increased efficiency comes in exchange for higher VRAM usage.

From our perspective, the ability to inexpensively train and run a model with greater powers seems very much worth that cost.

GPU Sponsor: TensorWave

We trained Flock of Finches on 16 AMD MI300X GPUs kindly donated by TensorWave, over a period of nearly four weeks. Each MI300X comes with a whopping 192GB of VRAM, which easily accommodated the added VRAM requirements we had for MoE.

This allowed us to use our limited time efficiently, finding the best hyper-parameters and doing training instead of spending days or weeks developing software workarounds.

MoE Overview

A large part of the knowledge and intelligence of LLMs come from a component known as the Feed Forward Network (FFN), sometimes called the Channel Mixer. We added a flock of eight new Feed-Forward Network (“FFN”) “experts” to a Finch 7B checkpoint that had been trained on around 2 trillion tokens, then continued training it for only 109 billion more.

The original Finch FFN in Flock of Finches is always evaluated like usual, acting like the leader of the flock, and we call it the “shared expert”. Alongside this shared expert one additional expert from the flock is chosen for each token, and the results are added together.

This forms the mathematical equivalent of a double-width dynamically chosen FFN. The shared expert contributes the shared intelligence learned during the original 2 trillion tokens of training, while the new experts in the flock selectively contribute new information depending on context.

MoE Shared Expert

A few choices we made were unusual, and make Flock of Finches a bit different from other MoE architectures you may have encountered in the wild. One such choice was to use a shared expert and add eight fresh experts, instead of replacing the original FFN with eight cloned copies and continuing training from there.

We found that this setup learned much faster, even when accounting for the extra width and therefore computation it adds. We also discovered that with this setup we were able to use an extremely high initial learning rate for the new experts, eventually annealing it down to the original model’s learning rate as training progressed.

MoE Hash Routing

Another unusual choice we made was to use hash routing instead of a trained top-k gated router. We chose this partly for simplicity and speed, but also because it gives us a naturally even token-to-expert routing distribution, which we hope will improve inference efficiency. Hash routing is extremely simple; we take the token index fed into the model plus a prime number and use that result modulo eight as the index of the expert to which that token is sent for processing. Many other MoE models use a learned gating function, which is trained instead of being fixed in advance of training.

And one final very RWKV-specific quirk was our use of token-shift with these new experts. Ordinarily, RWKV does a unique kind of 1D convolution as part of its FFN called token-shift, which mixes parts of the current and prior token together. This allows the model to perform some kinds of operations in a single layer that a traditional transformer would require two layers to accomplish. We tried various ways of applying token-shift to our new experts, and in the end we found that the most efficient way was to perform the same shift on the input that goes to both the shared and new experts. The gate applied to FFN outputs is also generated from a single token-shift and applied uniformly to the combined output.

Benchmark

We evaluated Flock of Finches across a range of common industry standard benchmarks using EleutherAI’s lm-eval-harness. While some benchmarks got higher scores and some lower, it was generally around the same level as our recently released Finch 14B model. This is an interesting result for us, as this model has significantly fewer active parameters (11B versus 14B), and those parameters are more concentrated in the Feed Forward Network portion of the model.

The Takeaway

Flock of Finches 37B-A11B features a new Mixture of Experts RWKV-6 architecture with 11 billion active parameters and 37 billion parameters total. It’s the largest RWKV MoE model yet, but it’s just our first step combining MoE with the RWKV architecture. We’re excited to expand the use of MoE to the Time Mixer portion of RWKV, and to try more complex MoE ideas like employing expert parameter sharing across breadth and depth, and combining a larger number of narrower experts.

We hope you’ll give Flock of Finches a try, and see how the RWKV ecosystem is growing with new more powerful models.

References

• Weights & Code: https://huggingface.co/recursal/Finch-MoE-37B-A11B-v0.1-HF

Acknowledgements

• Special thanks to TensorWave and AMD for sponsoring the Flock of Finches MI300X training run

• Recursal AI for its commitment to providing resources and development for the RWKV ecosystem - you can use their featherless.ai platform to easily run RWKV and compare to it other language models

• EleutherAI for support and guidance, especially on benchmarks and publishing research papers about the RWKV architecture

• Linux Foundation AI & Data group for supporting and hosting the RWKV project

And of course a huge thank you to the many developers around the world working hard to improve the RWKV ecosystem and provide environmentally friendly open source AI for all.

This is a game with elements that are quite challenging for “traditional” software development that are remarkably easy with an LLM. If you want to try what we built before reading further, you can do so here!

The Challenges

Building a game requires solving lots of problems around input, output, UI, visuals, latency, the list goes on. We’ll be focussing on the content related challenges that are not easy for the “rules-based” nature of typical software.

When we think about building Family Feud - or Featherless Feud as we call this version - we hit two problems:

1. question and answer generation

2. fuzzy answer matching

i.e. how do come up with quiz questions, and when given answers that aren’t exactly on the list, how do we match them?

For the actual show, the question bank is built with the help of surveys of the general public, meanwhile identifying variation of official answers are done by show staff live, a natural language task which is simple for humans, but challenging for software.

In both cases, we’ll be prompting an LLM to generate a certain kind of text and interpreting the results, but beyond just prompting, how we run the inference also matters. Let’s dive in.

Part 1 - Question/Answers Generation

To generate questions, we might prompt an LLM like so

You are the producer of the game show Family Feud.

Your job is to devise a question and a list of common answers to this question. For each answer, output a number between 0 and 100 which is how common you think the answer would be when given by a member of the general public. The sum of the scores should not exceed 100.

Now that content has structure (e.g. question versus answers) that we need to know about so that the various pieces can be handled appropriately by the different parts of our app.

The LLM “knows” about the relationship between the questions and answers in an output, so we can ask it to include the annotations in the output. We really could ask the LLM to output any kind of structured format but the most common format to request output in is JSON.

As for getting a model to formulate it’s output as JSON there are typically two parts:

1. prompting for the answer to be in JSON with a description of desired structure

2. requesting JSON from your inference API call

For 1., you might update your prompt as follows

You are the producer of the game show Family Feud.

Your job is to devise a question and a list of common answers to this question. For each answer, output a number between 0 and 100 which is how common you think the answer would be when given by a member of the general public. The sum of the scores should not exceed 100.

Be very concise, present only the question and answers, and don't add anything else.

Output valid JSON with the following format: two keys, `question`, which should be a string, and `answers` should be an array of objects. Each object in the `answers` array should have two keys, `answer` and `score`, each of which should be a string.

e.g.
```json
{
  "question": "Name something that might be wobbly",
  "answers": [
    { "answer": "Furniture" , score: 64 },
    { "answer": "Person/A Drunk", score: 15 },
    { "answer": "Spinning toy/top", score: 5 },
    { "answer": "Shopping Cart", score: 3 }
  ]
}
```

The sum of the scores should not exceed 100.

Being prescriptive in structure is important; if the output description isn’t clear enough to the model, you’ll get variation in structure and keys used, which likely cause an exception when the part of your software operating on the JSON output tries to separate the question from the answers.

Though even with an explicit description of output structure, the model to output syntactically valid JSON is it is unlikely that if you prompting only will have your output will be valid JSON. Here’s a sample of the output from above:

Here is a question with a list of answers and scores:

{
  "question": "Name something you might find in a purse or wallet",
  "answers": [
    { "answer": "Money", score: 70 },
    { "answer": "Phone", score: 20 },
    { "answer": "Credit Card", score: 5 },
    { "answer": "Candy", score: 2 },
    { "answer": "Makeup", score: 3 }
  ]
}

Pretty good right? It came up with a question, and in the format as requested (and needed for the game). It also outputted the `score` values as Numbers even though I asked, in the prompt, for their input to be strings! The model overrode my instructions, but for the better. However despite the request to give nothing but the output, there still is some pre-amble which requires it’s own intelligence to parse. How do we get rid of this?

The other half the equation is an inference technique known as “guided decoding” in it’s most general form, but known as JSON mode. Here we guarantee the output is syntactically valid JSON by manipulating token probabilities during sampling to prevent syntactically invalid JSON from appearing (e.g. the string {‘‘} is not valid JSON).

Of course these two modes are in tension with another - if you don’t ask for JSON in your prompt (or specify structure clearly enough), and put on JSON mode, it’s possible that your output bears little relationship to your input. In an early version of the prompt above, I got a number of questions.

How you request JSON mode varies by your API provider (this is a specific type of guided decoding which is a big topic), but many (most?) providers that have an API will accept the same format to the request as OpenAI, which is to include a

"request_format": { "type": "json_object" }

in the body of the inference POST request. Once this is included, my output is exactly how I hoped: exactly the JSON and only the JSON.

{
  "question": "Name something you might find in a garage",
  "answers": [
    { "answer": "Tools", "score": 40 },
    { "answer": "Bike", "score": 20 },
    { "answer": "Work Bench", "score": 12 },
    { "answer": "Vacuum", "score": 8 },
    { "answer": "Lawn Mower", "score": 5 },
    { "answer": "Storage Containers", "score": 5 }
  ]
}

Note generation takes longer when JSON mode is enabled. This is a consequence of how that guided decoding works, which is worth a blog post in it’s own right.

We haven’t talked about the Featherless API we’ve been using for inference, but hold on to that thought - more on that in a bit.

Part 2 - Fuzzy Answer Matching

The other challenge is matching players answers to the list of official answers, as they don’t always match, in our running example from the video above, the question “Name something that might be wobbly” had the following answers

Where “A stool” was accepted in place of “Furniture” and “A baby” was accepted as a version of “Person / a drunk”.

It’s relatively easy to devise a prompt and post-processing that captures the gist of the idea. Here’s a short python program we used during testing to let you iterate through prompt and model choice by letting an answer you supply on the command line is judged.

But catching the comedic subtlety of the game is something else entirely. Despite many prompt iterations and many different models I tried (including Llama 3.1 405B!), I couldn’t get the judge to accept “A baby” as a variation of the answer “Person / A drunk”. If you have a different result, I’d love to hear it in the comments!

Choosing an inference provider

There is a growing list of inference providers to choose from. Every foundation model company provides API access to their models (OpenAI, Anthropic, etc.), companies focussed just on inference provision (friendli.ai, replicate.ai, together.ai), and aggregators (e.g. openrouter.ai).

What’s unique about featherless.ai is the number of models available serverlessly, i.e. without paying for dedicated GPUs. With the exception of featherless, all other serverless providers only make the most popular models are available, since, behind the scenes, there is dedicated infrastructure to those models, those costs are just being amortized across a large enough group of consumers. Featherless is different. Every fine-tune of a large class of models (i.e. all fine-tunes of a specific set of base architectures) is available serverlessly. Our goal is to make every public model on HuggingFace available for serverless inference and we currently have over 2k models available.

The next closest service in model count is openrouter.ai, which aggregates providers, and has less than 10% of the available models.

Model Selection and Testing

Play testing is an important part of any game development, and as we were playing with Featherless Feud, we tried a number of different models. The primary issues were a lack of diverse answers or answers that were too ambiguous for the game format. We’ve left the utility box that overrides the model in the lower-right hand corner of the game; this showcases some of the strength of the featherless platform: take any of our 2k models from our models page, drop it into that selector, and the game is instantly updated. When contrasted that just model download times are tens of minutes for small models, you might appreciate this feat.

We tested a range of model architectures, sizes, and data-sets, including

• failspy/Meta-Llama-3-8B-Instruct-abliterated-v3 - we chose this model to explore if the abilteration approach would increase the variety of questions in an interesting way (inconclusive)

• qwen/qwen1.5-32b - The Qwen models feel very different than the Llama models - maybe that’s the culture of Alibaba versus Meta coming through …

• meta-llama/Meta-Llama-3.1-70B-Instruct: While this 70B model offered superior judgment in evaluating user answers, its slower response time made it less suitable for rapid iteration during gameplay. However, it was useful in scenarios where answer accuracy was critical.

• alpindale/magnum-72b-v1: Initially, this 72B model seemed promising due to its creative output. However, its tendency for erratic responses made it unsuitable for the structured requirements of Featherless Feud.

• Kooten/Mistral-Nemo-Instruct-2407-norefuse-OAS

• Sao10K/L3-8B-Stheno-v3.2: This 8B model was selected for its optimal balance between speed and output quality. It consistently produced valid JSON responses, making it a reliable choice for generating game content efficiently.

• anthracite-org/magnum-32b-v2 - a qwen2-32b fine-tune intended for role-play

At a high-level, the more RP-focussed models tended to generate more variety of questions, and the bigger models didn’t generate content with a sufficient jump in quality to justify the longer running time. So we settled on Stheno as it provided a balance of variety of output with fastest inference time.

Try running the game with some of the models and let us know in the comments what you differences you notice!

Wrapping up

We learned a ton in this journey, and hope you did too. Perhaps most surprising is that capturing the comedic subtlety of Family Feud escaped us even when using the more capable (i.e. larger) open-source LLMs.

Thanks for reading! Let us know how you enjoyed this article, either in the comments, or with an email to hello@featherless.ai, and head on over to featherless.ai to experiment with the models listed here and more.

And again, check out the running game here

In April we launched our RWKV-based model, EagleX v2. EagleX goes toe-to-toe with modern transformers on performance, while being much cheaper to run, and with an infinite context limit. The most common question I have personally seen about EagleX since then however has been, "How do I run it?".

Minmodmon is a small self-contained tool that lets you easily and quickly run RWKV-based models on Windows, on your GPU, locally. No dependencies required! Just download the latest release ZIP and run it!

AI for everyone

Our number 1 goal at Recursal is to make sure *everyone* receives the benefits of AI. We don't want you to have to be a technology expert to start running EagleX. There are already plenty of ways to run RWKV, but these tend to be aimed at more expert users.

Minmodmon was made to be used by anyone wanting to try out AI models, regardless of computer skill. It runs on Windows, requires no separate installs (no python, pip, etc), and needs no command-line expertise.

However, by design minmodmon is very limited. For a more feature-complete setup, the library web-rwkv that is used by minmodmon is also used in the excellent project ai00_server.

What you can do with it

Minmodmon is for use with other applications. You can't talk to a model directly through its web interface, but it integrates with common standards.

In particular, I recommend trying out minmodmon with SillyTavern. SillyTavern is an amazing AI chat application that lets you load in AI personas to talk with locally, completely free and open source.

This is still an early release. If you encounter any issues head over to our issue tracker and report them to us!

What's next

This release is just one step in our plans. We want both local and remote AI to be a seamless experience, putting you in control of what you use and where your data goes. The user experience still leaves much to be desired, but we have big plans in the works.

If you do not have a powerful GPU necessary to run local models, or just want better performance and larger models, we also recently launched a privacy-focused remote AI service, Featherless.

Since our initial launch in June, we’ve been adding architectures, with fanfare only in our discord. But with Mistral Nemo 12B and Qwen2 32B becoming supported this week, taking the # of supported architectures to 12 and the total inferencible model count* to nearly 2k (1,922 at time of writing), this post seems overdue.

The full list of supported architectures is available on our about page, but the timeline is this

When you break that model registration out over time, it looks something like this

That smaller models are easier (take less time and money) to fine-tune accounts for a larger proportion of 7B and 8Bs (versus 70B, 72B). Likewise the older the model is, the more time folks have had to fine tune (hence many Llama 2 tunes, but few Nemos and Qwens).

If you want to weigh in on what model architecture we’re supporting next, join our discord.

Also a plug for fine-tuners: we’re working on a set of features that will be of benefit to model creators. If you are doing some fine-tuning, we’d love to connect with you for feedback on these upcoming features.

Like being able to access over 450+ models from huggingface and know someone who would love to try?

Refer them to featherless.ai and both of you get the $10 OFF your next monthly bill!

Refer 12 of your friends and you can have a full year off our basic plan! (The discount stacks!)

Enjoy 🤗 your models with featherless.ai !

Reminder: All our models do not log any of your messages prompt or completion 😉

Earlier this year, we took the world by storm when we announced that our Eagle model had beaten Meta’s Llama-2 while taking less training time, being the world’s most efficient model.

While Eagle still packs a powerful punch, and has been helping diverse use-cases from multi-lingual, to content moderation, gaming, and role-play, we’ve been working on something new, to bring our insights on efficiency to a much broader realm.

Just this Friday, we launched Featherless AI, which enables serverless inference of every Llama-3 8B and 70B model on Hugging Face we grabbed our hands.

That’s over 475 models. With many more being added daily.

Allowing anyone to quickly experiment, try, and choose the latest and best models, from huggingface. Starting from $10 / month.

Previously, to use the even the smallest fine-tunes requires dedicated hardware, which translates to real hosting costs, whether you’re experimenting with a model or ramping up production use. This is a barrier to a host of use cases particularly agents where each step in the agent computation might benefit from a particular model.

The goal of featherless is to make every model on HuggingFace available serverless and with these Llama & RWKV based models, we’re a big step of the way there.

With featherless, you can experiment with an entirely new range of models at completely different economics.

• Check out the site: featherless.ai

• Retweet our launch tweet,

• Support our product hunt launch

There are more than 100,000 distinct language models on the hugging face hub.

This is the output of an enormous amount of creative energy: built by over 10k AI enthusiasts, these models include impressive attempts to improve upon the best known language models like ChatGPT.

A lot of airtime goes to innovations on technical elements of language models (e.g. context length). And while important, that a huge part of what left with a collection there are a great many of domain specific LLMs like

• for specific languages (e.g. for German, Russian or Chinese)

• for creative-writing (e.g tdrussell’s Llama-3-70b-Instruct-Storywriter)

• with detailed medical knowledge (e.g. BioMistral/BioMistral-7B)

• can understand SEC filings, (e.g. arcee-ai/Llama-3-SEC)

• legal (e.g. umarbutler/open-australian-legal-llm - a model trained on a dataset of Australian Law curated by the Australian Attorney General’s office!)

• novelty / character (e.g. MopeyMule)

but it’s hard to use

So how do you use these things?

Despite that HuggingFace is the defacto place to host models, you’re hard pressed to use them there. If you’ve spent time on the site, you may have forgotten that there is a specific part of the model card designed to let you test the model: it’s typically disabled for models 8B and up, which is the vast majority of models.

This kind of UX is a fact of that running these models requires operating expensive hardware (i.e. GPUs). You can rent these GPUs, but you’re looking at at least $2 / hour, and that would only cover you for the smaller models.

If you have a budget to experiment with, you can try launching the model on a dedicated service. However this will also require your patience; the most natural service to do this is HuggingFace’s inference endpoints service. Which I haven’t gotten it to work, despite the suggestion I should be able to launch a model in a few clicks.

Openrouter is probably the cloud service with the most options and offers per-token pricing. but it doesn’t have any of the models listed above. Nor does it let you bring your own model.

You can try and run it locally (and there are a host of tools that have significantly simplify the process - gpt4all, ollama and cortex are the more popular. But they still require technical orientation, patience, and, most importantly, powerful computing hardware.

Experiment Faster with Featherless

The goal of featherless is to make every LLM on hugging face available serverlessly. Right now, our collection is up to 1,501 models, making it the largest collection of models available for inference from any inference provider.

You can test any of the models on the site, but we’re expecting that you’ll plug this directly into a client, whether that’s to chat with it as a human e.g. via Typing Mind, Jan, or Silly Tavern, or you’ll use the API directly, e.g. in a raw python program, or in some higher-level framework like Lang Chain or Llama Index

Checkout our terms of service and privacy policy.

If you are fine-tuning, we recommend waiting for the full EagleX 2T model coming out later this month instead, unless you are doing so for research purpose.

This model is released for research purposes, as it represents the major checkpoint that surpasses LLaMA2 7B, as part of our current training to 2T tokens and beyond.

EagleX 1.7T - in short

EagleX 1.7T is a early research release of our 7.52B parameter model training that:

• Is part of a larger 2T model training

• Is built on the RWKV-v5 architecture
  (a linear transformer with 10-100x+ lower inference cost)

• Is continuation based on the original Eagle 7B model

• Ranks as the world’s greenest 7B model (per token)

• Trained on 1.7 Trillion tokens across 100+ languages

• Outperforms all 7B class models in multi-lingual benchmarks

• Passes LLaMA2 (2T) in multiple English evals, approaches Mistral (>2T?)

• All while being an “Attention-Free Transformer”

We are releasing RWKV-v5 EagleX 1.7T, licensed under Apache 2.0, which can be used personally or commercially without restrictions.

• Download from HuggingFace

• Try it online today on

  • our hugging face

  • our new cloud platform

• Use our reference pip inference package, or any other community inference options (Desktop App, RWKV.cpp, etc) , and use it anywhere (even locally)

• Fine-tune using our Infctx trainer

• [Pending PR] Get support merged into Huggingface transformers!

• All eval data can be found in the google sheet here

What does it mean to fly past LLaMA 7B?

It is a definitely a very big claim to say you have caught up and pass the “Gold Standard” of the 7B weight class from scratch, which nearly every other major open access model is built on (allegedly even Mistral). Even more so given that this is done with a comparatively lower dataset token count of 1.7 trillion token (vs. 2 trillion tokens).

Going big on eval data

As this is a entirely different model, trained from scratch, there will be evals that we win and we lose, which we are fully transparent about, in showing how we are ahead of LLaMA 7B on average.

Instead of simply cherry picking 14 different evals which we won and calling it a day with a victory, we ran ALL the benchmarks in EleutherAI `lm-eval-harness`, at commit `f78e2da` that we could do, with the following limitations:

• It has to complete in under 30 minutes on 8x4090 (we were running lots of evals)

  • This rules out some of the rather more expensive long chain of thought evals

• We excluded all the personality / alignment evals

• Eval has to be executable across a wide variety of models, via lm-eval-harness

• All evals are 0 shot (no 5 shot-ing an MCQ question)

• We limited comparison to other models within the 7B weight class

These resulted into running 60+ major eval groups, which generated over 1,000+ data points per model. A data point count so high, that we had to drop standard error deviations, just to ensure the raw CSV file can be loaded in MacOS numbers.

Whew, that’s a crazy number of data points to digest. Let me break it down to more digestible parts:

• English perplexity

• Multi lingual performance

• 21 English Eval Focus

• 183 English Evals

All data shown here is made available in the Google Sheet over here:

We included explanations of what several of the evals mean, which you can keep in mind in future eval results you see (demystify what those numbers mean!)

Improved English Perplexity

We start with basics: Perplexity. This is the loss value against the test dataset (lower score = better), i.e. how good the model is with next token prediction.

In general, with the perplexity improvements, the EagleX model outperforms LLaMA2-7b, ranking between Falcom/LLaMA2-7b and Mistral.

Leading Multi-lang Perplexity & evals

EagleX maintains the lead for best in class multi-lingual performance, with the incremental improvements we’re making to the Eagle line of models.

Most of the tasks here are common sense reasoning tests of wide variety of formats, across languages including 23 of the world’s most widely used languages.

For the remaining languages, we urge the community to test and judge it themselves, over a 100+ languages was trained. Over time, we would want more languages to be added into evals.

21 English Evals

Nevertheless, English is still important. We reduced the evals down to 21 of the argubly most popular English evals, such as Lambada, Glue, Swag, Winogrande, TruthfulQA, MMLU:

Narrowing it down to the 4 models that most of us actually care about - LLaMA, Mistral, EagleX and Eagle-7b - the new EagleX model outperforms LLaMA-2-7b on average across the 21 evals, and lags not far behind Mistral.

The Good

Now, let’s look at where our model is blowing the rest of the models out of the water.

First, the big stand out is the first 6 evals, which even our small 1.7T trained model beats out even Mistral 2T++ trained model (sciq, glue, anli, mmnli, swag), across multiple tasks focused around either contextual based simple Q&A with common sense reasoning, or deductive logic. EagleX also performs better than LLaMA-2-7b in wingrade and wnli evals, which also involves contextual common sense reasoning as well. This implies that the EagleX model would be applicable in RAG use cases, which are mainly contextual Q&A, with the right prompt engineering.

Finally, for truthfulqa, while it outperforms LLaMA, but in my opinion, this is still indicative of how vulnerable all models are from learning common human misconceptions from the web, seeing how bad the scores are across all models.
(to be fair, this is hard for most humans as well)

PS: The jump for glue/mnli was high enough, that we needed to check the dataset specifically for contamination. Which we were not be able to find any. This jump is currently being attributed to multiple training datasets, along with data augmented / machine rewritten instruct dataset following a similar structure.

The Mixed

Next: the eval sets with mixed results. Here, we have very similar evals with 2 major variants. The results between EagleX and LLaMA are close enough, that it’s hard to say which model is clearly better between the two for these evals.

What’s interesting, is that even though logiqa can be seen as form of “common sense” reasoning test, the EagleX model scored much lower compared to the 6 evals (sciq, glue, anli, mmnli, swag). This could mean that while the model is better at reasoning given a context, but it lacks the depth of knowledge compared to other models with more token training.

The “Not too bad“ and the “Really Bad”

These are the evals the EagleX performs worse on compared to both Mistral and LLaMA. However, for the evals that we’ve lost to LLaMA, it’s by a narrow margin. But we’ll be keeping track of these as we train past 2T tokens.

Let’s look what went really badly: Math.

The results for arithmetic eval sank drastically, like a rock, even compared to our original Eagle model.

What went wrong?

We dug through the dataset we used for training, and realized we missed out the entire math dataset (along with a few others) due to an error. Oops.

This emphasize the importance of maintaining the dataset composition over the training run. We’re adding math back for future runs.

We expect overall math score to rise back up as the training continue, however realistically IMO - no one should be depending on a 7B model for math (just saying)

183 English Evals

We do not simply want to cherry pick 9 or 21 evals and claim victory over LLaMA, or even Mistral. So, let’s zoom out, and look at it holistically across 183 English evals.

You can view the full results here

Although using the overall averages across all the evals does have a bias the results towards larger eval sets (due to double counting, e.g. mmlu overall and many indivudall mmlu test), it does not change the ranking among the EagleX, Mistral, LLaMA and the original Eagle models.

However these results is extremely useful for smaller insights, for example

The EagleX model lost to LLaMA2 when it comes to US history, but won in world history. This makes sense, given the broader approach we took to making the dataset from a more inclusive, more global view, instead of a US centric one.

The detailed insights will be used by our dataset team to iterate and improve on our future datasets.

Perhaps a good dataset + Scalable architecture: is all you need?

One of the biggest change we did was to change the dataset for the current 1T tokens, which now uses a cleaner filtered set of data with careful considerations to ensure permissible licensed content sources used.

There are also huge implications on the fact, the model crossed the llama2 line earlier then the plan schedule. That either the architecture is more efficient in training, or that the improvements in dataset quality has a large impact in model performance.

The following is a summary of the dataset used, its public release will be made available next month after the current 2T training is completed.

## 15% Code 

Contains code/programming related topics
- the-stack
- codeparrot
- devopedia
- mdn

## 15% Multi lang

Generally multi-lang webtext
- sea-lion (Singapore)
- madlad
- culturax
- multi lang wiki

## The giant soup

Creative content
- fandom (only sites with permissive licenses, and low spam)
- scp-foundation

Wikipedia
- Various Permissively licensed wikis.
- wikipedia

Papers:
- Mainly arxiv (Permissive Licenses) and pes2o

Books:
All the books contained in out train sets are public domains books.
- gutenberg, 
- standardebooks

Webtext
- webtext
- refinedweb (Note: This chunk made the model worse, we recommend against refinedweb in future trains)
- slimpajama
- europarl
- eurlex.
- stackexchange

Various
- aya (multilang convo)
- some system prompt, instruct
- long list of sub 100B training datasets on HF
- rewritten text !!! (splicing in, to replicate the rewritten web paper)

Why does the architecture matter?

We are over a 100x more scalable then the transformer architecture.
Transformers became the most prominent architecture in AI, not because it was the best, but it was the first to successfully scale to billion of parameters in training.

Till today

The Milestone

In overall, the release of this model marks an important milestone and transition for many of us, within both the commercial team within Recursal AI, and the open source team in the RWKV group.

• Its the first major training done by the Recursal AI team, in partnership with AWS as our main compute provider

• This model is being released under Apache 2 licensing

• The fully trained 2T model will be released under the RWKV group, under the Linux Foundation

• The first Non-Transformer Architecture to pass LLaMA2 in evals

• The strongest Linear Transformer to date

• Proof you can have both strong multi-lingual and english performance

What’s next?

Similar to the original Eagle 7B announcements, the following is the revised goals for the model training

• [April 2024] Completion of the 2T Eagle 7B models

• [March-May 2024] Training of our v6 “Finch”line of models

• [June 2024] v6 MoE model, for GPT 3.5 class performance

Disclaimer: All dates are approximate, and is heavily subjected to compute availability from our sponsors/compute-provider/investors

Want more?

If you want find more about the RWKV opensource Project at

• Wiki: https://wiki.rwkv.com/

• Discord: https://discord.gg/bDSBUMeFpc

If you like to try the model today, you can do so on our platform at recursal.ai - the best place host, run, and create finetunes of the Eagle line of RWKV models.

The Eagle-7B model launch has been a great success!
More details can be found here: https://substack.recursal.ai/cp/141146731

As we are working behind the scenes for our cloud platform launch - we have decided to avoid an eagle-and-egg situation. And decided to provide our latest 7B model for free on both our chat demo and open router endpoints

• https://openrouter.ai/models/recursal/eagle-7b

• https://rwkv-demo-api.recursal.ai/

These models will be provided for free, with rate limits until the launch of our cloud platform in March - stay tuned!

We are proud to announce the public release of RWKV 3B models on open router

• RWKV v5 world 3B

• RWKV v5 3B AI Town

This is in line with our previous 2 public API release for RWKV world, and AI town respectively,

• Public RWKV v5 3B OpenAI endpoint

• Public AI town 3B OpenAI endpoint

All of which to make it easier for you to switch existing experiments for testing.

We hope, by giving such public access, we get to see more interesting and application built on RWKV. Ping @picocreator on twitter or RWKV discord, to show us what you built with it 😉

Recursal.AI intend to keep the RWKV 3B models, open to the public, with some resonable anti-abuse / rate limits - till the end of 2024, as we explore means of optimizing our inference infrastructure at scale under load, for our commercial cloud platform.

Dedicated RWKV inference cloud service will be out soon.

Disclaimer: This API service is provided as it is, without any warranties or guarantees.