This blog post explores the performance of various consumer hardware setups for running local machine learning models, focusing on speed and efficiency. It compares an Intel NUC, an M2 Pro Mac Mini, and a Mini PC with an RTX 490 GPU, analyzing their power consumption, processing speed, and overall cost-effectiveness.
In recent years, the ability to run machine learning models locally has gained significant importance. This shift is particularly relevant for those looking to avoid the ongoing costs associated with cloud services. In this post, we will explore various consumer hardware options for running local machine learning models, focusing on speed and efficiency.
For this challenge, we have a diverse range of consumer hardware:
While these setups represent a wide range of options, they do not encompass all available hardware, especially in the Mac ecosystem, where options like the Mac Studio with the M2 Ultra chip can handle larger models due to its unified memory architecture.
For our tests, we will be using a 7 billion parameter model. This size is manageable across all tested hardware, allowing us to evaluate their performance effectively. However, it is essential to note that smaller models tend to run faster, albeit at the cost of quality. Achieving results comparable to ChatGPT at home is not feasible with larger models.
Before running the models, we observed the idle power consumption of each machine:
Upon powering on, the consumption increased significantly:
As we initiated the model runs, we noted the following power consumption:
The RTX 490 setup demonstrated impressive speed, completing tasks faster than the other machines. However, the initial transfer of the model from system RAM to VRAM was a bottleneck, impacting overall performance.
After running the models, we gathered performance metrics:
The RTX 490 outperformed the others in terms of raw speed, but the Intel NUC and M2 Pro provided faster response times for smaller queries due to their quicker time to first token.
Despite the RTX 490's high power consumption during operation (averaging 320 watts), the Intel NUC ultimately consumed more energy overall due to its longer processing time. The M2 Pro was the most energy-efficient, using the least power during operation.
The initial costs for the hardware setups are as follows:
Considering the average electricity costs in the U.S. (approximately 16 cents per kWh), we calculated the annual operating costs for each machine based on running 200 calculations per day. The results indicated that the M2 Pro was the most cost-effective option, while the RTX 490 could lead to significantly higher electricity bills, especially in countries with higher energy costs.
In conclusion, the choice of hardware for running local machine learning models depends on various factors, including speed, efficiency, and cost. The RTX 490 offers unparalleled performance for extensive tasks, while the M2 Pro Mac Mini provides a balanced approach with lower operational costs and noise levels. The Intel NUC, while not the fastest, can still be a viable option for specific use cases.
Ultimately, the decision should be based on the specific needs of the user, whether that be speed, efficiency, or cost-effectiveness. As technology continues to evolve, we can expect even more advancements in local machine learning capabilities.
