Dran

Very cool project!

What was the motivation to make it cross platform? On the Linux side I see immediate value but I’m struggling to understand the usecase for win/Mac.

You can’t just write off capital expenditure though. The hardware, even for “effecient” MOE inference is still very expensive to buy, house, run, and cool. Even assuming open-weight model serving at $0 r&d for the models themselves, mixing high-prefill workloads doesn’t batch well with decode heavy concurrency (or other prefill-heavy jobs). The moment you do anything nontrivial you start running into very complicated architectural problems to efficiently solve at scale.

Hardware that is useful for 5-10 years at most, plus development and support for the inference workflows, doesn’t leave a lot of margin on the table.

My gut, along with basically everything I read, suggests that not most (even pure inference) shops are not profitable and are still floating on loans or vc money.

Can you cite your source on the claim that “inference is currently insanely profitable”? Everything I read suggests that openai and anthropic lose money on their plans.

For music enthusiasts plexamp is also basically unbeatable. I welcome the day open source catches up.

People shit on it but there’s a lot of good open-source tooling that supports it.

There are nist l1 profiles

Tutorials and guides for everything

etc

Part of being a good sysadmin is knowing when not to reinvent the wheel. Ubuntu has a lot of options for vetted, hardened, “other people’s wheels.”

Also, for posterity, the competent ones are running the headless, server version of Ubuntu. (As opposed to the bloated mess that is Ubuntu Desktop). The server version catches a lot of flack it doesn’t deserve.

Code rewrites are always going to have growing pains. Rewriting gnu-corrutils in rust is a noble effort.

Microvms or containers could give you external control of the networking. Then you would put whatever you want behind warp inside the warp container/vm.

+1 to this observation. I run zfs arrays at both home and work and it’s way more likely that your controller is flaking than you have that many simultaneous drive failures.

The unfortunate reality though is that you can’t trust the current copy of this data, even the snapshots, unless the restore passes a scrub post-restore.

Inference is dirt cheap in comparison. Hundreds to thousands of concurrent users can be served by hardware costing in the high-thousands to low-ten-thousands.

Training those same foundational models is weeks to months of time on tens to hundreds of millions worth of hardware.

If caddy is acting as a proxy for anything, you should not need to forward that port externally. Local host firewalls allowing traffic on your local network is sufficient.

Depending on your physical host layout you may be looking at an issue with nat reflection.

You have not given us enough about your topology to assist in troubleshooting.

There are server chips like the E7-8891 v3 which lived in a weird middle ground of supporting both ddr3 and ddr4. On paper, it’s about on par with a ryzen 5 5500 and they’re about $20 on US eBay. I’ve been toying with the idea of buying an aftermarket/used server board to see if it holds up the way it appears to on paper. $20 for a CPU (could even slot 2), $80 for a board, $40 for 32gb of ddr3 in quad chanel. ~$160 for a set of core components doesn’t seem that bad in modern times, especially if you can use quad/oct channel to offset the bandwidth difference between ddr3 and ddr4.

I think finding a cooler and a case would be the hardest part

+1 for Niagra. It takes a few days to get used to but it’s the launcher every power user didn’t know they wanted. Lifetime purchase options and a very responsive/passionate dev

https://en.wikipedia.org/wiki/Markov_chain

Before the advent of AI, I wrote a slack bot called slackbutt that made Markov chains of random lengths between 2 and 4 out of the chat history of the channel. It was surprisingly coherent. Making an “llm” like that would be trivial.

It definitely can be disabled post-install but is much simpler to install without it at install-time, and has the added benefit of not pulling 2-5gb of other things that won’t be relevant to your use case. It’s not that the disk waste is that big of a deal, but any issues you run into will be that much easier to troubleshoot with fewer moving parts.

That wasn’t quite the takeaway I was going for. You can get a lot done on 8gb of ram. I was just trying to point out that it would probably be your first bottleneck as you started to scale out, and that you should consider using the server headless to make the ram you have go that much further.

All of those would be perfectly cromulent nodes for small containers. The first issue you’ll run into is the low ram. Some homelab projects would cause you to exceed 8gb, but the good news is if you’re using an external backend via NFS, you can always scale out (more nodes) or up(more compute per node,) later with minimal headache.

If you’re going to be memory constrained, don’t waste 1-2gb on a gui, install Ubuntu/Debian/whatever headless

Not the same chips, but ddr5, gddr7, and hbm2 are made off the same wafers in the same plants. The issue is allocation in wafer and production time skewing towards the higher-margin items. DDR5 additionally is being made more into the server ecc variant, which companies are buying in droves for cost-efficient MOE inference.

What is the flag for this?

CGNAT does have a designated range by spec. 100.64.0.0/10, which covers addresses from 100.64.0.0 to 100.127.255.255. Technically they could be using any other private address space but it would be very uncommon in a modern ISP.

No offence: but the problem is an app forces me to trust you; a website does not. I have toghter and easier control over a web request than I do over an app, and even if an app doesn’t have these permissions today, an update or an update after a sale could trivially and silently introduce them.

A website is obvious if the deal changes-- you put up a login wall to harvest data; I stop using the site. You put trackers and ads into the UI; I block it at the DNS level.