modeler@lemmy.worldtoTechnology@lemmy.world•An angry admin shares the CrowdStrike outage experienceEnglish
1·
4 months agoI think that’s a better plan than physically printing keys. I’d also want to save the keys in another format somewhere - perhaps using a small script to export them into a safe store in the cloud or a box I control somewhere
Typically you need about 1GB graphics RAM for each billion parameters (i.e. one byte per parameter). This is a 405B parameter model. Ouch.
Edit: you can try quantizing it. This reduces the amount of memory required per parameter to 4 bits, 2 bits or even 1 bit. As you reduce the size, the performance of the model can suffer. So in the extreme case you might be able to run this in under 64GB of graphics RAM.