TinyZero \u2013 Researchers Replicated DeepSeek\u2019s R1-Zero Model for Just $30
\n \t
\n
<\/BR>
\n
\n \t
\n
<\/BR><\/p>\n
In an impressive demonstration of cost-effective AI research, a group of researchers has successfully replicated DeepSeek\u2019s R1-Zero<\/a> model for just $30. <\/p>\n Dubbed TinyZero<\/strong>, this project focuses on countdown and multiplication tasks, leveraging reinforcement learning (RL) to enable a 3-billion-parameter (3B) base language model (LM) to develop self-verification and search abilities autonomously.<\/p>\n Built on the veRL framework<\/a>, TinyZero showcases how reinforcement learning can help large language models (LLMs) evolve reasoning capabilities independently. <\/p>\n The researchers behind this project highlight an \u201cAha!\u201d moment<\/strong> that users can experience firsthand with minimal computational costs.<\/p>\n For those interested in exploring the methodology, a detailed experiment log is available on Weights & Biases<\/a>, with further insights shared in a Twitter thread<\/a>. The team has also confirmed that a formal research paper is forthcoming.<\/p>\n The research team<\/a> selected the \u201ccountdown game\u201d as their test environment, a mathematical challenge where the AI generates equations from a set of numbers to reach a specific target. <\/p>\n This game is ideal for testing problem-solving capabilities, as it requires logical reasoning and strategic trial-and-error to improve over time. Initially, the model produced random outputs with no clear strategy. <\/p>\n However, through reinforcement learning, it gradually refined its approach<\/a>, developing logical reasoning skills independently.<\/p>\n To replicate TinyZero, users can follow a straightforward setup process:<\/p>\n Activate the environment and preprocess the dataset:<\/p>\n For models up to 1.5B<\/strong> parameters, a single GPU setup works effectively:<\/p>\n For larger models that exhibit more advanced reasoning skills, a two-GPU configuration is recommended:<\/p>\n The team also experimented with an instruction-tuned version of Qwen-2.5-3B. This requires additional data preprocessing:<\/p>\n Training follows a similar two-GPU setup:<\/p>\n TinyZero was developed based on the veRL framework and employs the Qwen2.5 series base models. The research team, comprising Jiayi Pan, Junjie Zhang, Xingyao Wang, Lifan Yuan, Hao Peng, and Alane Suhr, has made the project open-source, accessible on GitHub here<\/a>.<\/p>\n With the success of TinyZero, this experiment demonstrates that state-of-the-art AI capabilities can be developed and studied on a remarkably small budget,<\/strong> potentially paving the way for more affordable AI research.<\/p>\n The post TinyZero \u2013 Researchers Replicated DeepSeek\u2019s R1-Zero Model for Just $30<\/a> appeared first on Cyber Security News<\/a>.<\/p>\n<\/div>\nRunning TinyZero: Installation and Setup<\/strong><\/h2>\n
Installation Steps<\/strong><\/h4>\n
\n
conda create -n zero python=3.9<\/code>\n<\/li>\npip install torch==2.4.0 --index-url https:\/\/download.pytorch.org\/whl\/cu121<\/code>\n<\/li>\npip3 install vllm==0.6.3<\/code>\n<\/li>\npip install -e . pip3 install flash-attn --no-build-isolation pip install wandb IPython matplotlib<\/code>\n<\/li>\n<\/ol>\nCountdown Task: Training TinyZero<\/strong><\/h2>\n
Data Preparation<\/strong><\/h4>\n
conda activate zero\npython .\/examples\/data_preprocess\/countdown.py --local_dir {path_to_your_dataset}<\/code><\/pre>\nTraining on a Single GPU<\/strong><\/h4>\n
export N_GPUS=1\nexport BASE_MODEL={path_to_your_model}\nexport DATA_DIR={path_to_your_dataset}\nexport ROLLOUT_TP_SIZE=1\nexport EXPERIMENT_NAME=countdown-qwen2.5-0.5b\nexport VLLM_ATTENTION_BACKEND=XFORMERS\n\nbash .\/scripts\/train_tiny_zero.sh<\/code><\/pre>\nScaling Up: Training a 3B+ Model<\/strong><\/h4>\n
export N_GPUS=2\nexport BASE_MODEL={path_to_your_model}\nexport DATA_DIR={path_to_your_dataset}\nexport ROLLOUT_TP_SIZE=2\nexport EXPERIMENT_NAME=countdown-qwen2.5-3b\nexport VLLM_ATTENTION_BACKEND=XFORMERS\n\nbash .\/scripts\/train_tiny_zero.sh<\/code><\/pre>\nInstruct Ablation: Experimenting with Qwen-2.5-3B<\/strong><\/h2>\n
conda activate zero\npython examples\/data_preprocess\/countdown.py --template_type=qwen-instruct --local_dir={path_to_your_dataset}<\/code><\/pre>\nexport N_GPUS=2\nexport BASE_MODEL={path_to_your_model}\nexport DATA_DIR={path_to_your_dataset}\nexport ROLLOUT_TP_SIZE=2\nexport EXPERIMENT_NAME=countdown-qwen2.5-3b-instruct\nexport VLLM_ATTENTION_BACKEND=XFORMERS\n\nbash .\/scripts\/train_tiny_zero.sh<\/code><\/pre>\nFind this News Interesting! Follow us on\u00a0Google News<\/a>,\u00a0LinkedIn<\/a>, and\u00a0X<\/a>\u00a0to Get Instant Updates<\/strong><\/code><\/strong><\/p>\n