• What works small often fails big — Architectural innovations that show promise at 1M parameters may break down at 1B or 50B.

• Validating at scale is expensive — Training from scratch to test a new architecture at meaningful scale can cost at least $5–10M.

• High risk, uncertain reward — You’re just as likely to degrade performance as improve it—making architecture exploration financially unsustainable for most labs.

Training a state-of-the-art language model from scratch costs roughly $5-10M—just to validate a new attention mechanism, recurrence scheme, or memory system.

From our team experience, it typically takes 20–80 architecture iterations to achieve a 10%+ improvement. We've done this four times over the past two years.

For most AI labs, that level of experimentation would cost around $250 million in research GPU time. From that perspective, it's often more rational to invest in scaling model parameters and datasets for a near-guaranteed performance gain of ~10%.

At Featherless, we believe this bottleneck in architecture validation has slowed progress—not only in capabilities but in reliability.

But what if the cost to validate an architecture dropped from $5 million to $20K?

With that same $250 million, we could run over 12,500 iterations, uncovering 100+ architecture improvements, each with 10%+ gains. Compounded, that’s a theoretical 1,378,000% improvement in performance.

That’s why we’re excited about RADLADS.

Introducing RADLADS

RADLADS (Rapid Attention Distillation to Linear Attention Decoders at Scale) is a new method for converting massive transformer models (e.g., Qwen-72B) into new AI models with alternative attention mechanisms—at a fraction of the original training cost.

• Total cost: $2,000–$20,000

• Tokens used: ~500 million

• Training time: A few days on accessible cloud GPUs (8× MI300)

• Cost reduction: ~250× reduction in the cost of scientific experimentation

Instead of training from scratch, we convert existing models to new attention architectures in three steps:

1. Align hidden states between the original transformer and the target attention architecture

2. Distill output behavior (logits) from the original model

3. Fine-tune for long-context performance

You can read about the process details from our paper review on huggingface and arxiv. This is the same technique that allowed us to train our latest 72B1 attention-free, with only 8 GPU’s.

What does this mean for research?

RADLADS is already changing how we explore AI architecture. We can now:

• Rapidly test novel attention mechanisms and hybrid designs

• Iterate on model structures in days, not months

• Validate alignment and interpretability hypotheses at scale

This isn’t just about RWKV—it opens doors for advancing Transformers, State Space models, xLSTMs, and architectures yet to be imagined. Its about accelerating our pace of research.

And we’re not doing it alone. Since announcing our work, we've collaborated with other researchers to validate multiple attention mechanisms, including Transformer-based variants.

Reach out to us if you have any attention alternative your research team or university lab is working on and looking to validate in collaboration.

It’s all part of our mission to make personalized reliable AI — and eventually AGI — a reality

Featherless AI - recursive dev blog

🪿QRWKV-72B and 32B : Training large attention free models, with only 8 GPU's

We are proud to announce the updated QRWKV-72B and 32B…

Read more

a year ago · 14 likes · 3 comments · Eugene Cheah

In the past quarter, we have seen the following breakthroughs in open-source

• GPT-4o Mini class open models (deepseek, qwen-32b) on our platform

• 72B Attention-Free QRWKV model built on 8 GPUs

• Both with more knowledge and capability than an average human

• A tipping point with RWKV v6 and v7 improvements

As such, my prediction (Eugene Cheah) for 2025 is that we are at the inflection point.

• Where we scale towards model reliability, with better memories

• Instead of scaling a trillion parameters

Our vision for AGI in summary

The video above, is the condensed version of our vision for personalized AI & AGI
The writing below, is the longer form with some additional details

The scaling wall for bigger models

The problem with scaling today is within the fundamental promise is for a step-increased improvement in capability, for every 10x in parameter size.

While it has remain somewhat true, that there is a step-up improvement in capability. The problem is in the diminishing returns, on both training and inference costs.

We are now within Billion’s of investment dollar range for the current GPT 4.5 model, with no clear answer if OpenAI Super AGI goal is achievable with another 10x, or 100x, or 1000x. After which, we are starting to talk in Trillions of dollars. Just for training. With even more dollars required to actually run the model.

While this path might still make sense, potentially for “Super AGI”.

It makes no sense for “Human-level AGI”. Because ….

Short Context AGI - is already here (sort of)

For the vast majority of users on applications like Character.AI, within that context window of approximately 8,000 tokens.

These AI characters, within the window, is AGI, with perhaps some flaws, but a unique character experience regardless. It’s what drive the 28 Million plus users to keep engaging with the platform.

The limits are however clear

• How they are unable to reliably follow instructions within memory

• Nor are they able to gracefully handle memories beyond their context length

The lack of reliable AI memory

Because here lies an irony, these models are no doubt knowledgeable and capable.

Today’s best models for both open source (ie. DeepSeek R1) and close source, for example, is no-doubt capable of doing PhD level math and physics, with some degree of reliability (let say 1-out-of-30 times)

But yet they lack the reliability to do basic college-level task (30-out-of-30 times), be it as a cashier, or any simple agent needing to do a long multi step process.

This is also known, as the “Compounding AI Agent Error” problem

And fixing the reliability problem does not need a bigger model. We already have proof of this as observed across our platform…

How reliability is being solved in production today?

One of the interesting patterns we have observed on our platform is the workload differences between individual and scaled-up commercial use cases at featherless

For example, in the above, we show, how our individual users are running thousands of fine-tuned model - with a bias towards the top models like the DeepSeek R1, or LLaMA3 70B as expected.

But this dramatically changes when we see the models running by commercial users in production at scale.

The graph is not a mistake, the vast majority of the production workload we see at scale (by request or token count). Is the 1-year-old mistral Nemo-12b (including both base and finetuned)

• Not the latest model

• Not the biggest model

• That is good enough, for predictable prompt engineering or finetuning

This is consistent with AI systems and agents in production at scale, where either or both of the following solutions are used.

• AI Engineering: large problems are scoped, and broken into smaller tasks, solvable via prompt engineering

• Finetuning: specialized domain-specific model is finetuned for specific tasks

In both cases, significant engineering effort is required

• to design the AI agent, and its workflow, with prompt engineering

• to calibrate the dataset for finetuning, requiring dedicated specialized talent. Due to the limitations of catastrophic forgetting

The later, is typically reserved for larger teams, due to the difficulty in scaling the resources required to “get it right”. Due to the highly trial-and-error nature of the process involved in both tasks.

Catastrophic forgetting: Is the challenge faced by all existing model, where existing knowledge and capabilities is lost. While adding in new knowledge.

How RWKV memory module can solve this

If all AI models are already “capable” enough to potentially handle the vast majority of commercial tasks. Especially when they are finetuned for high reliability.

A recurrent model, with memories being part of its core design. Dramatically simplify the process of “finetuning”. In particular around the topic of catastrophic forgetting

For example, it is possible to “memory tune” the memory module state (Time Mix, in the diagram), which by default is left as blank in our base models.

This will not induce any Catastrophic forgetting, as none of the initial model weights is modified. A process we been testing, and implementing for a few pilot customers.

Recurrent models, also by their nature/design, is trained to handle memories, much closer to how we humans do so. Which makes it naturally more scalable for solving the reliable memory problem transformers face.

It also brings about multiple additional benefits, such as 100x lower inference cost (due to its linear scaling nature).

We are by no means perfect today (or ever),
it is why we iterate, more than anyone else

The RWKV team, is arguably the only team today, who has been consistently be making step function improvements in our architecture every half a year.

This can be best seen by our model performance chart across the past 2 years, from version 4 to version 7.

A process that we intend to accelerate, moving forward.

How to achieve Personalized AI

With the appropriate resource support. We expect to iterate into a reliable version of personalized AI within the next year or two.

We define this, as being able to “Memory Tune” without Catastrophic forgetting, and with high accuracy in narrow tasks. With 100 Million tokens or less. Into production use cases.

Quickly, and without the need for highly specialized professionals.

This is not considered “AGI”, because it will be constrained up to what it was “designed to be trained on”. As it would not permanently be learning new knowledge.

However, this is not an issue for most commercial use cases, as it would become the workhorse that drives the AI agent adoption cycle.

And make it → Personalized AGI

Once personalized AI is mastered, it would allow us to focus on the next step - to automate the process for the model to collect and reflect on their “day-to-day” interaction. For further memory tuning “in the night”.

For most part, this would be done incrementally, instead of a giant leap.

This is due to how its an extension of the Personalized AI memory tuning process.

The main differentiator, however, is we would now allow the model to decide on new knowledge and information from its day to day experiences to “learn from”.

However, we do not expect this automated tuning process to be infinitely stable. Due to potential memory loss in between past a certain scale. Be it a billion or a trillion tokens.

However, what this will simply translate to, is a lifespan for a continuous Personal AGI agent. One that can span days, to weeks, months, and eventually years for each iterative improvement.

Why all of this is inevitable

It is either a binary outcome, can recurrent models, be iterated and scaled (not just by param size), to be more reliable, and have better memories.

• We have provided evidence that the above statement is true, on a progressive trend for 2 years, despite the limited resources invested in the team.

• We have provided evidence that recurrent models can scale up to match the capability of some of the largest open transformer models.

Assuming the above is true

• The high-level roadmap is here, in public.

• Better and more capable open transformer models will arrive,
  which will speed up the process for us to iterate on.
  (little to no training from scratch required)
  

• We have enough resources to keep scaling our inference platform,
  while slowly iterating on this roadmap, with a handful of researchers.

• Even if we Featherless were to suddenly disappear. Enough resources and momentum within the open model space, will let future teams slowly eventually follow this same roadmap laid out.

Overall, we expect the following timeline, if featherless AI is properly supported in this journey, to seize the opportunity window

• < 2 Years for personalized AI

• < 4 Years for personalized AGI

Left on its own, with Moore’s law level of improvement in computing, we expect the hardware requirement to iterate on this roadmap to enter the “personal computing” space in 4 years. Once that is reached, we expect a rapid innovation cycle that leads to the same result, be it within or outside the USA.

We are proud to announce the updated QRWKV-72B1 and 32B.

Both models are available on huggingface and featherless.ai

• 32B | Hugging Face Link | Featherless AI Link

• 72B | Hugging Face Link | Featherless AI Link

The largest model to date - that is not based on the transformer attention architecture.

Surpassing existing transformer models in several benchmarks, while following right behind in others.

This builds on our previous experiments in converting the QRWKV6, where we converted the previous Qwen 2.5 32B model to RWKV. And the previous 72B preview.

Which we applied instead for the Qwen-QwQ-32B model and the Qwen-72B model respectively.

But lets take a step back at what this means …

We now have a model far surpassing
GPT-3.5 turbo, without QKV attention

While slowly closing in on GPT-4O-mini

With lower inference cost, param size, and better performance.

We need no super cluster - only a single server.

Because we were keeping most of the feed forward network layer the same.
We can perform the conversion, (barely) within a single server of 8 MI300 GPU’s

Requiring the full 192GB VRAM allocation per GPU

How the conversion is done: A summary

While more details will be revealed in an upcoming paper. The core idea is similar to the previous QRWKV6 conversion , but this time we apply it to the Qwen-72B and QwQ-32B models

At a high level, you take an existing transformer model

Freeze all the weights, delete the attention layer, replace it with RWKV, and train it through multiple stages

All while referencing the original model logits as a “teacher model”

More specifically it would be the following

• Train the RWKV layer individually, referencing the individual teacher blocks

• Train the RWKV layer, with the whole model, training on teacher logits

  • At this point the model is “usable” but has much more to improve on

• Train all the layers (both FFNN and RWKV), on teacher logits

• Train all the layers with longer context length

Unfortunately, due to the limitation of VRAM, our training was limited to 8k context length. However we view this as a resource constraint, and not a method constraint.

Implication:
AI knowledge, is not in attention, but FFN

Due to the limited token training of 200-500M, of the converted layers. We do not believe that the newly trained RWKV layers, is sufficiently trained for “knowledge/intelligence” at this level.

In other words, the vast majority of an AI model knowledge, is not in the attention but the matrix multiplication FFN (Feed-Forward-Network) layer.

It would be more accurate, to view the Attention mechanisms, be it transformer based, or RWKV. As a means of guiding the model to focus on “what the model thinks” about in the FFN layer.

Benefits: Ideal for large scale application

Additionally, with the shift towards inference-time-computing.

Linear architectures represents a dramatic reduction in both compute and vram requirement cost. Allowing us to scale hundreds to thousand requests per GPU.

We can now rapidly iterate new RWKV architectures on <100B scales

By dramatically reducing the compute requirement for scaling and testing a new RWKV attention architecture. To a small number of GPU’s

We will be able to test, iterate, and validate newer architecture changes faster, taking experiments what previously took weeks (or even months), to days.

Historically, the RWKV group has been averaging 4 major versions across 2 years. With improvement to both model architecture accuracy and memories at every step.

A trend which we plan to accelerate moving forward.

As we work on our roadmap to Personalized AI and eventually Personalized AGI, which you can see more in our following article …

Featherless AI - recursive dev blog

🛣️ Our roadmap to Personalized AI and AGI

If you want to find out more about the latest QRWKV model, that makes all of this possible, it is recommended to read this first…

Read more

a year ago · 4 likes · Eugene Cheah

Happy December Neurips,

We are proud to announce the triple model weights release of a charm of finches

Q-RWKV-6 32B Instruct Preview

Our latest frontier model.

A variant of RWKV-6, converted from an existing Qwen 32B model.

This is our strongest linear model to date, beating out all previous RWKV, State Space and Liquid AI models, smashing all previous key english benchmarks and evals.

Excitingly, this unlocks the option of converting existing transformer models to more efficient RWKV linear architecture.

Its limitation however, is how it inherits its knowledge training, and tokenizer, from the parent model. Which in this case is limited to approximately 30 languages (compared to RWKV 100+ languages)

See more info: Announcement article
Try the model on our: Featherless.ai inference

RWKV-6 Finch MoE 37B

Our first RWKV MoE model, for RWKV-6, with 11B out of 37B active parameters. Currently provides one of strongest multi-lingual model

See more info: Announcement article

RWKV-6 Finch 7B World 3

An overall multi-lingual upgrade of our v6 7B base models, that is a major bump up from our previous 7B models for multi-lingual and mixed use cases.

This was developed and released under the RWKV foundation. With various contributors from Eleuther AI and RWKV open source group.

See more info: Announcement article

The strongest linear model to date, beating out all previous RWKV, State Space and Liquid AI models, smashing all previous key english benchmarks and evals.

You can find this model available on both

• Hugging Face:
  https://huggingface.co/recursal/QRWKV6-32B-Instruct-Preview-v0.1

• Featherless.ai: https://featherless.ai/models/recursal/QRWKV6-32B-Instruct-Preview-v0.1

Note: that as an instruction preview, the model is not considered final

Trained by converting the weights of the Qwen 32B Instruct model, into a customized QRWKV6 architecture. We were successfully able to replace the existing transformer attention heads with RWKV-V6 attention heads, through a groundbreaking new conversion training process.

This unique training process was developed by the team at Recursal AI, in joint collaboration with the RWKV and EleutherAI open source community.

Benchmarks

We compared QRWKV6 against existing open weights models, both transformer based and linear-architecture based.

In overall, what is most exciting is how the QRWKV6 converted model, perform similarly to its original 32B model.

GPU Sponsor: TensorWave

The conversion process for QRWKV6, was done on 16 AMD MI300X GPUs kindly donated by TensorWave. Each MI300X comes with a whopping 192GB of VRAM, while having comparable H100 level of compute performance.

This allowed us to reduce the minimum number of nodes required for our training process, and simplify our overall training and conversion process.

The conversion process took about 8 hours

The Exciting

Linear models hold promise in substantially lower compute cost at scale. Delivering over a 1000x compute efficiency in inference cost, especially over large context length. A key multiplier unlock for both O1 style inference time thinking, and making AI more accessible for the world.

This technique is also scalable to larger transformer based models. Which we have since started.

The Good

The benefit of this process is that we are able to convert any previously trained QKV Attention based model, such as Qwen and LLaMA based models, into a variant of RWKV. Without needing to retrain the model from scratch.

This allows us to quickly test and prove out the significantly more efficient RWKV Linear attention mechanic at a larger scale, with a much smaller budget, without training from scratch. Proving out the architecture design and scalability of RWKV.

Once again proving, QKV attention is not all you need.
( Someone ping @jefrankle and @srush_nlp )

The Bad

The disadvantage of this process is that the model inherent knowledge and dataset training, is based on its “parent” model. Meaning unlike previous RWKV models trained on over 100+ languages. The QRWKV model is limited to the approximate 30 languages supported by the Qwen line of models.

Additionally, instead of RWKV based channel mix and feedforward network layers, we retain the “parent” model feed forward network architecture design. This means there will be incompatibility with existing RWKV inference code.

Separately, due to our compute budget, we were only able to do the conversion process up to 16k context length. While the model does exhibit stability beyond the given context length, the following model may need additional training to accurately support larger context length

Future Followups

Currently Q-RWKV-6 72B Instruct model is being trained

Additionally with the finalization of RWKV-7 architecture happening soon, we intend to repeat the process and provide a full line up of

• Q-RWKV-7 32B

• LLaMA-RWKV-7 70B

We intend to provide more details on the conversion process, along with our paper after the subsequent model release.

References

• Model weights and code here

Acknowledgements

• Special thanks to TensorWave and AMD for sponsoring the MI300X training

• EleutherAI for support and guidance, especially on benchmarks and publishing research papers about the RWKV architecture

• Linux Foundation AI & Data group for supporting and hosting the RWKV project

• Recursal AI for its commitment to providing resources and development for the RWKV ecosystem - you can use their featherless.ai platform to easily run RWKV and compare to it other language models

And of course a huge thank you to the many developers around the world working hard to improve the RWKV ecosystem and provide environmentally friendly open source AI for all.

We’re excited to release the latest addition to the RWKV family of model releases: Flock of Finches 37B-A11B v0.1!

This is an experimental model that uses 11 billion active parameters, and despite our new flock having been trained on only 109 billion tokens, roughly matches our recently released Finch 14B model on common benchmark evaluation scores. You can find the model and code at huggingface here , or try it on featherless AI platform here

We leveraged an efficient Sparse Mixture of Experts (MoE) method to supply a higher total parameter count while activating only a fraction of those parameters for any given token. This saves time and uses less compute during both training and inference. As with most architectural choices, there is a tradeoff; increased efficiency comes in exchange for higher VRAM usage.

From our perspective, the ability to inexpensively train and run a model with greater powers seems very much worth that cost.

GPU Sponsor: TensorWave

We trained Flock of Finches on 16 AMD MI300X GPUs kindly donated by TensorWave, over a period of nearly four weeks. Each MI300X comes with a whopping 192GB of VRAM, which easily accommodated the added VRAM requirements we had for MoE.

This allowed us to use our limited time efficiently, finding the best hyper-parameters and doing training instead of spending days or weeks developing software workarounds.

MoE Overview

A large part of the knowledge and intelligence of LLMs come from a component known as the Feed Forward Network (FFN), sometimes called the Channel Mixer. We added a flock of eight new Feed-Forward Network (“FFN”) “experts” to a Finch 7B checkpoint that had been trained on around 2 trillion tokens, then continued training it for only 109 billion more.

The original Finch FFN in Flock of Finches is always evaluated like usual, acting like the leader of the flock, and we call it the “shared expert”. Alongside this shared expert one additional expert from the flock is chosen for each token, and the results are added together.

This forms the mathematical equivalent of a double-width dynamically chosen FFN. The shared expert contributes the shared intelligence learned during the original 2 trillion tokens of training, while the new experts in the flock selectively contribute new information depending on context.

MoE Shared Expert

A few choices we made were unusual, and make Flock of Finches a bit different from other MoE architectures you may have encountered in the wild. One such choice was to use a shared expert and add eight fresh experts, instead of replacing the original FFN with eight cloned copies and continuing training from there.

We found that this setup learned much faster, even when accounting for the extra width and therefore computation it adds. We also discovered that with this setup we were able to use an extremely high initial learning rate for the new experts, eventually annealing it down to the original model’s learning rate as training progressed.

MoE Hash Routing

Another unusual choice we made was to use hash routing instead of a trained top-k gated router. We chose this partly for simplicity and speed, but also because it gives us a naturally even token-to-expert routing distribution, which we hope will improve inference efficiency. Hash routing is extremely simple; we take the token index fed into the model plus a prime number and use that result modulo eight as the index of the expert to which that token is sent for processing. Many other MoE models use a learned gating function, which is trained instead of being fixed in advance of training.

And one final very RWKV-specific quirk was our use of token-shift with these new experts. Ordinarily, RWKV does a unique kind of 1D convolution as part of its FFN called token-shift, which mixes parts of the current and prior token together. This allows the model to perform some kinds of operations in a single layer that a traditional transformer would require two layers to accomplish. We tried various ways of applying token-shift to our new experts, and in the end we found that the most efficient way was to perform the same shift on the input that goes to both the shared and new experts. The gate applied to FFN outputs is also generated from a single token-shift and applied uniformly to the combined output.

Benchmark

We evaluated Flock of Finches across a range of common industry standard benchmarks using EleutherAI’s lm-eval-harness. While some benchmarks got higher scores and some lower, it was generally around the same level as our recently released Finch 14B model. This is an interesting result for us, as this model has significantly fewer active parameters (11B versus 14B), and those parameters are more concentrated in the Feed Forward Network portion of the model.

The Takeaway

Flock of Finches 37B-A11B features a new Mixture of Experts RWKV-6 architecture with 11 billion active parameters and 37 billion parameters total. It’s the largest RWKV MoE model yet, but it’s just our first step combining MoE with the RWKV architecture. We’re excited to expand the use of MoE to the Time Mixer portion of RWKV, and to try more complex MoE ideas like employing expert parameter sharing across breadth and depth, and combining a larger number of narrower experts.

We hope you’ll give Flock of Finches a try, and see how the RWKV ecosystem is growing with new more powerful models.

References

• Weights & Code: https://huggingface.co/recursal/Finch-MoE-37B-A11B-v0.1-HF

Acknowledgements

• Special thanks to TensorWave and AMD for sponsoring the Flock of Finches MI300X training run

• Recursal AI for its commitment to providing resources and development for the RWKV ecosystem - you can use their featherless.ai platform to easily run RWKV and compare to it other language models

• EleutherAI for support and guidance, especially on benchmarks and publishing research papers about the RWKV architecture

• Linux Foundation AI & Data group for supporting and hosting the RWKV project

And of course a huge thank you to the many developers around the world working hard to improve the RWKV ecosystem and provide environmentally friendly open source AI for all.

In April we launched our RWKV-based model, EagleX v2. EagleX goes toe-to-toe with modern transformers on performance, while being much cheaper to run, and with an infinite context limit. The most common question I have personally seen about EagleX since then however has been, "How do I run it?".

Minmodmon is a small self-contained tool that lets you easily and quickly run RWKV-based models on Windows, on your GPU, locally. No dependencies required! Just download the latest release ZIP and run it!

AI for everyone

Our number 1 goal at Recursal is to make sure *everyone* receives the benefits of AI. We don't want you to have to be a technology expert to start running EagleX. There are already plenty of ways to run RWKV, but these tend to be aimed at more expert users.

Minmodmon was made to be used by anyone wanting to try out AI models, regardless of computer skill. It runs on Windows, requires no separate installs (no python, pip, etc), and needs no command-line expertise.

However, by design minmodmon is very limited. For a more feature-complete setup, the library web-rwkv that is used by minmodmon is also used in the excellent project ai00_server.

What you can do with it

Minmodmon is for use with other applications. You can't talk to a model directly through its web interface, but it integrates with common standards.

In particular, I recommend trying out minmodmon with SillyTavern. SillyTavern is an amazing AI chat application that lets you load in AI personas to talk with locally, completely free and open source.

This is still an early release. If you encounter any issues head over to our issue tracker and report them to us!

What's next

This release is just one step in our plans. We want both local and remote AI to be a seamless experience, putting you in control of what you use and where your data goes. The user experience still leaves much to be desired, but we have big plans in the works.

If you do not have a powerful GPU necessary to run local models, or just want better performance and larger models, we also recently launched a privacy-focused remote AI service, Featherless.

If you are fine-tuning, we recommend waiting for the full EagleX 2T model coming out later this month instead, unless you are doing so for research purpose.

This model is released for research purposes, as it represents the major checkpoint that surpasses LLaMA2 7B, as part of our current training to 2T tokens and beyond.

EagleX 1.7T - in short

EagleX 1.7T is a early research release of our 7.52B parameter model training that:

• Is part of a larger 2T model training

• Is built on the RWKV-v5 architecture
  (a linear transformer with 10-100x+ lower inference cost)

• Is continuation based on the original Eagle 7B model

• Ranks as the world’s greenest 7B model (per token)

• Trained on 1.7 Trillion tokens across 100+ languages

• Outperforms all 7B class models in multi-lingual benchmarks

• Passes LLaMA2 (2T) in multiple English evals, approaches Mistral (>2T?)

• All while being an “Attention-Free Transformer”

We are releasing RWKV-v5 EagleX 1.7T, licensed under Apache 2.0, which can be used personally or commercially without restrictions.

• Download from HuggingFace

• Try it online today on

  • our hugging face

  • our new cloud platform

• Use our reference pip inference package, or any other community inference options (Desktop App, RWKV.cpp, etc) , and use it anywhere (even locally)

• Fine-tune using our Infctx trainer

• [Pending PR] Get support merged into Huggingface transformers!

• All eval data can be found in the google sheet here

What does it mean to fly past LLaMA 7B?

It is a definitely a very big claim to say you have caught up and pass the “Gold Standard” of the 7B weight class from scratch, which nearly every other major open access model is built on (allegedly even Mistral). Even more so given that this is done with a comparatively lower dataset token count of 1.7 trillion token (vs. 2 trillion tokens).

Going big on eval data

As this is a entirely different model, trained from scratch, there will be evals that we win and we lose, which we are fully transparent about, in showing how we are ahead of LLaMA 7B on average.

Instead of simply cherry picking 14 different evals which we won and calling it a day with a victory, we ran ALL the benchmarks in EleutherAI `lm-eval-harness`, at commit `f78e2da` that we could do, with the following limitations:

• It has to complete in under 30 minutes on 8x4090 (we were running lots of evals)

  • This rules out some of the rather more expensive long chain of thought evals

• We excluded all the personality / alignment evals

• Eval has to be executable across a wide variety of models, via lm-eval-harness

• All evals are 0 shot (no 5 shot-ing an MCQ question)

• We limited comparison to other models within the 7B weight class

These resulted into running 60+ major eval groups, which generated over 1,000+ data points per model. A data point count so high, that we had to drop standard error deviations, just to ensure the raw CSV file can be loaded in MacOS numbers.

Whew, that’s a crazy number of data points to digest. Let me break it down to more digestible parts:

• English perplexity

• Multi lingual performance

• 21 English Eval Focus

• 183 English Evals

All data shown here is made available in the Google Sheet over here:

We included explanations of what several of the evals mean, which you can keep in mind in future eval results you see (demystify what those numbers mean!)

Improved English Perplexity

We start with basics: Perplexity. This is the loss value against the test dataset (lower score = better), i.e. how good the model is with next token prediction.

In general, with the perplexity improvements, the EagleX model outperforms LLaMA2-7b, ranking between Falcom/LLaMA2-7b and Mistral.

Leading Multi-lang Perplexity & evals

EagleX maintains the lead for best in class multi-lingual performance, with the incremental improvements we’re making to the Eagle line of models.

Most of the tasks here are common sense reasoning tests of wide variety of formats, across languages including 23 of the world’s most widely used languages.

For the remaining languages, we urge the community to test and judge it themselves, over a 100+ languages was trained. Over time, we would want more languages to be added into evals.

21 English Evals

Nevertheless, English is still important. We reduced the evals down to 21 of the argubly most popular English evals, such as Lambada, Glue, Swag, Winogrande, TruthfulQA, MMLU:

Narrowing it down to the 4 models that most of us actually care about - LLaMA, Mistral, EagleX and Eagle-7b - the new EagleX model outperforms LLaMA-2-7b on average across the 21 evals, and lags not far behind Mistral.

The Good

Now, let’s look at where our model is blowing the rest of the models out of the water.

First, the big stand out is the first 6 evals, which even our small 1.7T trained model beats out even Mistral 2T++ trained model (sciq, glue, anli, mmnli, swag), across multiple tasks focused around either contextual based simple Q&A with common sense reasoning, or deductive logic. EagleX also performs better than LLaMA-2-7b in wingrade and wnli evals, which also involves contextual common sense reasoning as well. This implies that the EagleX model would be applicable in RAG use cases, which are mainly contextual Q&A, with the right prompt engineering.

Finally, for truthfulqa, while it outperforms LLaMA, but in my opinion, this is still indicative of how vulnerable all models are from learning common human misconceptions from the web, seeing how bad the scores are across all models.
(to be fair, this is hard for most humans as well)

PS: The jump for glue/mnli was high enough, that we needed to check the dataset specifically for contamination. Which we were not be able to find any. This jump is currently being attributed to multiple training datasets, along with data augmented / machine rewritten instruct dataset following a similar structure.

The Mixed

Next: the eval sets with mixed results. Here, we have very similar evals with 2 major variants. The results between EagleX and LLaMA are close enough, that it’s hard to say which model is clearly better between the two for these evals.

What’s interesting, is that even though logiqa can be seen as form of “common sense” reasoning test, the EagleX model scored much lower compared to the 6 evals (sciq, glue, anli, mmnli, swag). This could mean that while the model is better at reasoning given a context, but it lacks the depth of knowledge compared to other models with more token training.

The “Not too bad“ and the “Really Bad”

These are the evals the EagleX performs worse on compared to both Mistral and LLaMA. However, for the evals that we’ve lost to LLaMA, it’s by a narrow margin. But we’ll be keeping track of these as we train past 2T tokens.

Let’s look what went really badly: Math.

The results for arithmetic eval sank drastically, like a rock, even compared to our original Eagle model.

What went wrong?

We dug through the dataset we used for training, and realized we missed out the entire math dataset (along with a few others) due to an error. Oops.

This emphasize the importance of maintaining the dataset composition over the training run. We’re adding math back for future runs.

We expect overall math score to rise back up as the training continue, however realistically IMO - no one should be depending on a 7B model for math (just saying)

183 English Evals

We do not simply want to cherry pick 9 or 21 evals and claim victory over LLaMA, or even Mistral. So, let’s zoom out, and look at it holistically across 183 English evals.

You can view the full results here

Although using the overall averages across all the evals does have a bias the results towards larger eval sets (due to double counting, e.g. mmlu overall and many indivudall mmlu test), it does not change the ranking among the EagleX, Mistral, LLaMA and the original Eagle models.

However these results is extremely useful for smaller insights, for example

The EagleX model lost to LLaMA2 when it comes to US history, but won in world history. This makes sense, given the broader approach we took to making the dataset from a more inclusive, more global view, instead of a US centric one.

The detailed insights will be used by our dataset team to iterate and improve on our future datasets.

Perhaps a good dataset + Scalable architecture: is all you need?

One of the biggest change we did was to change the dataset for the current 1T tokens, which now uses a cleaner filtered set of data with careful considerations to ensure permissible licensed content sources used.

There are also huge implications on the fact, the model crossed the llama2 line earlier then the plan schedule. That either the architecture is more efficient in training, or that the improvements in dataset quality has a large impact in model performance.

The following is a summary of the dataset used, its public release will be made available next month after the current 2T training is completed.

## 15% Code 

Contains code/programming related topics
- the-stack
- codeparrot
- devopedia
- mdn

## 15% Multi lang

Generally multi-lang webtext
- sea-lion (Singapore)
- madlad
- culturax
- multi lang wiki

## The giant soup

Creative content
- fandom (only sites with permissive licenses, and low spam)
- scp-foundation

Wikipedia
- Various Permissively licensed wikis.
- wikipedia

Papers:
- Mainly arxiv (Permissive Licenses) and pes2o

Books:
All the books contained in out train sets are public domains books.
- gutenberg, 
- standardebooks

Webtext
- webtext
- refinedweb (Note: This chunk made the model worse, we recommend against refinedweb in future trains)
- slimpajama
- europarl
- eurlex.
- stackexchange

Various
- aya (multilang convo)
- some system prompt, instruct
- long list of sub 100B training datasets on HF
- rewritten text !!! (splicing in, to replicate the rewritten web paper)

Why does the architecture matter?

We are over a 100x more scalable then the transformer architecture.
Transformers became the most prominent architecture in AI, not because it was the best, but it was the first to successfully scale to billion of parameters in training.

Till today

The Milestone

In overall, the release of this model marks an important milestone and transition for many of us, within both the commercial team within Recursal AI, and the open source team in the RWKV group.

• Its the first major training done by the Recursal AI team, in partnership with AWS as our main compute provider

• This model is being released under Apache 2 licensing

• The fully trained 2T model will be released under the RWKV group, under the Linux Foundation

• The first Non-Transformer Architecture to pass LLaMA2 in evals

• The strongest Linear Transformer to date

• Proof you can have both strong multi-lingual and english performance

What’s next?

Similar to the original Eagle 7B announcements, the following is the revised goals for the model training

• [April 2024] Completion of the 2T Eagle 7B models

• [March-May 2024] Training of our v6 “Finch”line of models

• [June 2024] v6 MoE model, for GPT 3.5 class performance

Disclaimer: All dates are approximate, and is heavily subjected to compute availability from our sponsors/compute-provider/investors

Want more?

If you want find more about the RWKV opensource Project at

• Wiki: https://wiki.rwkv.com/

• Discord: https://discord.gg/bDSBUMeFpc

If you like to try the model today, you can do so on our platform at recursal.ai - the best place host, run, and create finetunes of the Eagle line of RWKV models.

The Eagle-7B model launch has been a great success!
More details can be found here: https://substack.recursal.ai/cp/141146731

As we are working behind the scenes for our cloud platform launch - we have decided to avoid an eagle-and-egg situation. And decided to provide our latest 7B model for free on both our chat demo and open router endpoints

• https://openrouter.ai/models/recursal/eagle-7b

• https://rwkv-demo-api.recursal.ai/

These models will be provided for free, with rate limits until the launch of our cloud platform in March - stay tuned!

We are proud to announce the public release of RWKV 3B models on open router

• RWKV v5 world 3B

• RWKV v5 3B AI Town

This is in line with our previous 2 public API release for RWKV world, and AI town respectively,

• Public RWKV v5 3B OpenAI endpoint

• Public AI town 3B OpenAI endpoint

All of which to make it easier for you to switch existing experiments for testing.

We hope, by giving such public access, we get to see more interesting and application built on RWKV. Ping @picocreator on twitter or RWKV discord, to show us what you built with it 😉

Recursal.AI intend to keep the RWKV 3B models, open to the public, with some resonable anti-abuse / rate limits - till the end of 2024, as we explore means of optimizing our inference infrastructure at scale under load, for our commercial cloud platform.

Dedicated RWKV inference cloud service will be out soon.

Disclaimer: This API service is provided as it is, without any warranties or guarantees.

Recursal.ai - is now launching a free public preview, of our OpenAI compatible chat / completion API, for RWKV models

This API currently supports, the following openAI format

• chat completion endpoint (with aitown 3B model)

• completion endpoint

• rate limited to 10 requests per 10 seconds

With the following two models

• RWKV-World-v5-3B

• recursal-aitown-3B

• recursal-pygmalion-chat-3B (coming soon)

All other API requests, will be rerouted to OpenAI endpoint, which will require an OpenAI authentication key

Enjoy playing with the RWKV 3B world model

We intend to keep this server up, to the public, with some sensible anti-abuse / rate limits - throughout the year, as we explore means of optimizing our inference infrastructure at scale under load, for our commercial cloud platform.

Dedicated RWKV inference cloud service will be out soon.

Disclaimer: This API service is provided as it is, without any warranties or guarantees.

Now you can - just by pointing to our dedicated AI town servers at

Without quotes or ending backslashes in the settings.

This API currently supports, the following openAI format

• chat completion endpoint (with aitown 3B model)

• completion endpoint

• embedding endpoint (passthrough to openAI, using your openAI key)

You will still need an OPENAI_API_KEY for the embeddings. AI town models at scales beyond 100+++

Alternatively to run locally, you can use the github repo here:

https://github.com/recursal/ai-town-rwkv-proxy

We intend to keep this server up, to the public, with some sensible anti-abuse / rate limits - throughout the year, as we explore means of optimizing our inference infrastructure at scale under load, for our commercial cloud platform.

Dedicated RWKV inference cloud service will be out soon.

Disclaimer: This API service is provided as it is, without any warranties or guarantees.

Harrison & Eugene (Picocreator) from the RWKV community (+ a few others), is teaming up to build the cloud and enterprise serverless platform for RWKV.

• Supports OpenAI compatible inference endpoints

• Finetuning of RWKV models

• Fully automated distillation, and offloading of existing OpenAI workloads

  • Lower your existing OpenAI cost, without performance compromise

• More stuff in the wraps

TLDR: Everything you love about the OpenAI/Claude, RWKV edition, and more

We have raised our seed round, with a large percentage of it set aside specifically to support the RWKV open source model training and development process (~500k for this alone).

What does this mean for the OpenSource RWKV community and group?

Mostly positives, more resources will be raised and allocated for the RWKV project.

There is no license changes to the open source model and code. It will remain apache2. Locked safely within the Linux Foundation

On a more immediate basis,

• more GPUs for running RWKV-X experiments, and dev work

• salaries for open source dev hires, fully focused on just building on RWKV

• funding to support RWKV auxiliary activities (from events, to paying for gmail)

Additionally, like other linux foundation projects, we expect over time, that more companies will commercialize RWKV. And work together on the common open source offerings. With the core RWKV project remaining neutral under the linux foundation.

In addition, the vast majority of Recursal AI models and dataset work (besides customer specific fine tunes, etc), will remain in the Opensource Apache 2 license.

Meaning we have to make sure its cheaper, and more efficient to use us, then to DIY inference on your own with a cloud provider. As such, ensuring RWKV stays open source is aligned with us. As the growth of the open source models, also means the increase in usage of our platform from commercial / enterprise customers.

A win-win structure that has worked for other opensource software

Major examples of this OSS and Commercial split in practice

• Many open source database, with multiple providers

• Selenium browser automation project

• RISC-V

Open Job Roles (as of 30th Nov 2023)

For all roles the following generally applies

• Its Full time, and remote

• Includes standard leave, vacation, and insurance packages (depending on country)

• ESOP for recursal.ai (details to be finalized closer to april 2024)

AI / Software Dev - For RWKV opensource projects

• Work directly with BlinkDL, PicoCreator and HarrisonV, on improving the RWKV group collections of various opensource projects or demos. This covers software development

  • For RWKV group inference / training projects

  • Other RWKV related opensource projects

  • Standardizing and development of Benchmarks

  • RWKV-X architecture itself

  • and public demos

• Highly preferred to be an existing active member of the RWKV community

• Coordinate with the recursal team and RWKV OSS team, on devrel and marketing matters.

DevOps & System Administrator

• Help coordinate our growing racks of GPUs, and dev servers

  • Including racking and stacking physical servers

• Proficient in:

  • PFSense networking

  • Linux Administration

  • Docker containers build process

  • Github & Gitlab for CI/CD build pipelines

• Coordinate with the recursal and RWKV oss team, on ensuring the dev and production environment uptime, and the CI/CD build processes to support them

Senior Frontend / Fullstack developer

• Build the Frontend for the Recursal AI platform

• Work on the user facing API services

• Work with the rest of the Recursal team, in developing the platform and its various public demos

• Be a strong UX / DX advocate, our team is too heavily biased to the infra/model side of things

• Have strong attention to detail for the user facing UI clients.

• Coordinate user feedback with the rest of the team

Senior Backend / Fullstack developer

• Build and scale the backend for the Recursal AI platform

• Work on the user facing API services, and backend internal services

• Work on internal UI for administration / etc, and CLI tooling

• Have a strong attention to detail for the development API platforms, its scaling design, performance, and its unit tests

• Coordinate user feedback (for our CLI / APII users) to the rest of the team

• Work with the rest of the Recursal team, in developing the platform and its various public demos

If any of the roles excites you, ping Picocreator on the RWKV discord.
Or drop an email with your resume to hr @ recursal.ai

Do not apply if it does not excite you, to be building a platform, for …
truely Open Source AI

RWKV v5 1.5B achieves SOTA status with

• Industry leading multi-lingual performance (across xLBD, xSC, xWG, xCOPA benchmarks) by significant margins, against all existing models

• Comparable performance to falcon-rw-1b in english based benchmark

  • We win out in LAMBDA, StoryCloze16, arch_challenge, arc_easy, headQA_en, openbookQA, sciq, COPA

  • but looses out very slightly on PIQA, Hellaswag, WinoGrade,ReCoRD, COPA

For nearly all use cases under the 2B param model class, RWKV V5 now represents either the best model for multi-lingual use, or a tied 1st place model with falcon-rw-1b

Making this a strong default model of choice within its weight class.

A pattern we intend to repeat in the 3, 7, and 14B weight classes respectively. We expect the 3B model to be out by first week december.

You can access the model today via the following options

• Public Demo: https://huggingface.co/spaces/BlinkDL/ChatRWKV-gradio

• Recursal AI Chat Demo: https://1b5-demo.recursal.ai/

• Model Download : https://huggingface.co/BlinkDL/rwkv-5-world/tree/main

Github repo is here: https://github.com/recursal/ai-town-rwkv-proxy

Working together with the AI town team @ a16z - we have fine-tuned a tiny, highly efficient RWKV 3B and 1.5B model, for the purpose of AI town use cases. All running locally on a macbook pro.

This has exciting potential both for gaming industry as a large, where we can simulate a large number of NPCs, with believable character chatter.

However what was more exciting to the team here at recursal, was the process behind it, as the impact extends beyond gaming.

As this can be replicated on any existing AI agents deployment, while taking advantage of RWKV lower cost advantages.

Where it is able to run on less then 1/10th the cost of existing GPT3.5 models pricing (with more potential to go lower).

Automated distillation with Recursal

The above AI town model was distilled from the original AI agent usage of Open AI, with a process that can be fully automated.

Our team didn’t need to build the dataset required for the fine tune by hand. All we simply did was collect the required data by using a proxy in between the AI agents and the OpenAI backend.

Using the data collected, we then fine-tuned a model respectively, and slowly begun offloading requests to the our RWKV optimized model.

While at the beginning the RWKV model may not be able to cover all use cases, overtime as the dataset is being built up, the model is incrementally finetuned to cover a wider array of capabilities.

Effectively, allowing drastic reduction in OpenAI 3.5 / 4 bills, at no performance compromises.

This process is not new, in itself, and is commonly known as AI distillation.

However what is new is the usage of

• Automated processes to simplify the distillation process, and dataset cleaning.

• Usage of smaller / more efficient RWKV models in the process, previous industry attempts at distilling LLaMA2 70B models, has proven to be not price competitive

• The usage of a smaller model in the router, to decide which requests gets to be routed to the RWKV model, and the openAI platform

All of which would be the focus for our upcoming recursal AI platform launch. Of which we would kick start our closed beta by Mid-December.

If the above process excites you, on the potential of drastic cost savings on your existing AI work load, you can signup for our closed beta pilot at the following form.

https://docs.google.com/forms/d/e/1FAIpQLSekNp_npm7unSmlfWsUsGs3aaBrplgKE8sLiHLoyeJaqvj5bQ/viewform

Ensuring that RWKV continues to grow as a true OSS model (Just Apache 2 license) By the community, for the world

Thanks @LFAIDataFdn for welcoming us on board OSS summit

More information can be found here on the official press briefing:
https://lfaidata.foundation/blog/2023/09/21/lf-ai-data-launches-generative-ai-commons/

This is a repost of a past event, prior to the setup of this blog

In that regard, we are proud to say that RWKV (the model which our team works on), has been independently benchmarked as the world's greenest and most energy-efficient AI model/architecture, on a per token output basis, for models of the same param sizes (7B params).

The energy efficiency of the RWKV architecture is derived from the 10-100 times compute efficiency of our linear transformer architecture vs the quadratic scaling of transformer architectures. A benefit we expect to scale better as our models get larger.

Combined with how RWKV models scales similarly to transformers in evals, against other models with the same dataset.

The industry wide benefits for scaling more energy efficient architecture, like RWKV, will be significant for our industry as a whole.

And we, the team at Recursal AI, are excited to be involved in helping spearhead this charge towards more energy efficient models with RWKV, and potentially other future models/architectures as our industry develops.

Towards a future with more, not less alternatives to AI,
with the various unique benefits each architecture will bring us.